Opened 8 years ago
Last modified 7 years ago
#12215 closed defect
Memleak in UniqueRepresentation, @cached_method — at Version 95
Reported by: | vbraun | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-5.7 |
Component: | memleak | Keywords: | UniqueRepresentation cached_method caching |
Cc: | SimonKing, jdemeyer, mhansen, vbraun, jpflori | Merged in: | |
Authors: | Simon King | Reviewers: | |
Report Upstream: | N/A | Work issues: | Rebase wrt #7980 |
Branch: | Commit: | ||
Dependencies: | #11115 #11900 #12645 #11599 #12808 #7980 | Stopgaps: |
Description (last modified by )
The documentation says that UniqueRepresentation? uses weak refs, but this was switched over to the @cached_method decorator. The latter does currently use strong references, so unused unique parents stay in memory forever:
import sage.structure.unique_representation len(sage.structure.unique_representation.UniqueRepresentation.__classcall__.cache) for i in range(2,1000): ring = ZZ.quotient(ZZ(i)) vectorspace = ring^2 import gc gc.collect() len(sage.structure.unique_representation.UniqueRepresentation.__classcall__.cache)
Related tickets:
- #11521 (needs review, introducing weak references for caching homsets), and
- #715 (needs review, using weak references for caching coerce maps).
- #5970 (the polynomial rings cache use strong references, which may now be a duplicate, as I introduce the weak cache in #715)
Further notes:
- not everything in Python can be weakref'ed, for example
None
cannot.
- some results that are expensive to compute should not just be cached by a weak reference. Perhaps there is place for a permanent cache, or maybe some minimal age before garbage collecting it.
Apply
Change History (95)
comment:1 Changed 8 years ago by
- Cc SimonKing added
comment:2 Changed 8 years ago by
- Description modified (diff)
comment:3 Changed 8 years ago by
comment:4 Changed 8 years ago by
- Dependencies set to #11115
comment:5 Changed 8 years ago by
Here is a patch. It isn't tested yet.
comment:6 Changed 8 years ago by
- Dependencies changed from #11115 to #11115 #11900
... and I immediately updated the patch: Join categories were not using unique representation but cached_function (by #11900). So, that had to change.
comment:7 Changed 8 years ago by
Sorry, it was impossible to use weak_cached_function on the join function in sage.categories.category, since it may return a list (not weakly referenceable). Hence, I had to work around. With the attached patch (applied on top of #11900 and its dependencies), sage at least starts...
comment:8 Changed 8 years ago by
It turns out that all the patches can still not fix the problem. We also have to deal with sage.structure.factory.UniqueFactory
.
I suggest to add an option to UniqueFactory
, that decides whether a strong or a weak cache is used. And I suggest to do this here, because I don't want to create yet another ticket.
The applications of UniqueFactory
should mainly be in cases where weak references work. Therefore I suggest to use the weak cache by default - I am curious how many doc tests will fail...
Coercion sucks.
comment:9 Changed 8 years ago by
It turns out that UniqueFactory
already was somehow using weak references, but in an improper way. The new patch version replaces that by WeakValueDictionary
.
It doesn't solve the problem, though.
comment:10 Changed 8 years ago by
I have slightly updated my patch, so that there is no conflict with #11935.
comment:11 Changed 8 years ago by
There is yet another location where it makes sense to use @weak_cached_function: For the cache of dynamic classes!
Namely, dynamic classes are frequently used in the category framework, they have a strong cache, and the parent/element classes keep a pointer to the category they belong to. So, that's preventing categories from being garbage collected.
I think that my patches from here, #715, and #11935 (which reduces the number of dynamic classes created) might actually be enough to fix the problem. When I run
sage: for p in primes(2,1000000): ....: R = GF(p)['x','y','z'] ....: print get_memory_usage()
then one initially still sees an increased memory usage. But after a while it seems to stabilise.
comment:12 Changed 8 years ago by
- Description modified (diff)
- Status changed from new to needs_review
I have updated the patch. It documents the changes, and at least the tests in sage/misc/cachefunc.pyx, in sage/categories/..., in sage/rings/... and in sage/structure/unique_representation.py pass.
Hence, needs review!
comment:13 Changed 8 years ago by
comment:14 Changed 8 years ago by
- Description modified (diff)
comment:15 Changed 8 years ago by
- Status changed from needs_review to needs_work
- Work issues set to segfaults for elliptic curves
While the tests in sage/categories, sage/rings and sage/structure/unique_representation.py pass, I get some segfaults for the elliptic curve tests. Thus, needs work.
comment:16 Changed 8 years ago by
I did sage -t --verbose "devel/sage-main/sage/schemes/elliptic_curves/ell_point.py"
, and it did not reveal a segfault while running the tests. The test process itself crashed:
830 tests in 54 items. 830 passed and 0 failed. Test passed. The doctested process was killed by signal 11 [23.8 s] ---------------------------------------------------------------------- The following tests failed: sage -t --verbose "devel/sage-main/sage/schemes/elliptic_curves/ell_point.py" # Killed/crashed
Strange.
comment:17 Changed 8 years ago by
I think I found the problem.
Some doctest of the form
sage: K.residue_field() <expected answer>
segfaults. But when the result is assigned to a variable, like this
sage: RF = K.residue_field(); RF <expected answer>
then everything works.
Is it perhaps the case that garbage collection of the residue field (that was enabled by my patch) happens between the creation and the computation of the string representation of the object?
But that is strange. There are variables _
and __
, which are supposed to provide strong references to the last two results - hence, there should be no garbage collection.
comment:18 Changed 8 years ago by
sage.structure.factory.UniqueFactory
did use weak references before. But it did so - I think - improperly, namely without using weakref.WeakValueDictionary
. The new patch version changes that.
It isn't ready for review, yet, because of the segfaults.
comment:19 Changed 8 years ago by
Some old code is not using the cache: There was some coerce map created in sage/rings/residue_field.pyx, whose parent was not created by Hom(domain,codomain)
, but directly by RingHomset(domain,codomain)
.
Changing it fixed at least one segfault. I wish all segfaults would go away so easily...
comment:20 Changed 8 years ago by
Fortunately, I now have a short example that triggers a memory access error when leaving Sage:
sage: E = EllipticCurve('15a1') sage: K.<t>=NumberField(x^2+2*x+10) sage: EK=E.base_extend(K) sage: EK.torsion_subgroup() Torsion Subgroup isomorphic to Z/4 + Z/4 associated to the Elliptic Curve defined by y^2 + x*y + y = x^3 + x^2 + (-10)*x + (-10) over Number Field in t with defining polynomial x^2 + 2*x + 10 sage: quit Exiting Sage (CPU time 0m1.98s, Wall time 0m52.03s). local/bin/sage-sage: Zeile 303: 30045 Speicherzugriffsfehler sage-ipython "$@" -i
However, I wonder how I can trigger the error without leaving Sage, and how I can trace what is going on.
comment:21 Changed 8 years ago by
Actually EK._torsion_bound(number_of_places=20)
is enough to trigger the memory access error.
comment:22 Changed 8 years ago by
Here is the stack:
Program terminated with signal 11, Segmentation fault. #0 cgetg (y=22, x=<optimized out>) at ../src/kernel/none/level1.h:114 114 ../src/kernel/none/level1.h: No such file or directory. in ../src/kernel/none/level1.h Traceback (most recent call last): File "/usr/share/gdb/auto-load/usr/lib64/libstdc++.so.6.0.16-gdb.py", line 59, in <module> from libstdcxx.v6.printers import register_libstdcxx_printers File "/usr/lib64/../share/gcc-4.6.2/python/libstdcxx/v6/printers.py", line 19, in <module> import itertools ImportError: No module named itertools Missing separate debuginfos, use: debuginfo-install atlas-3.8.4-1.fc16.x86_64 expat-2.0.1-11.fc15.x86_64 fontconfig-2.8.0-4.fc16.x86_64 keyutils-libs-1.5.2-1.fc16.x86_64 krb5-libs-1.9.2-4.fc16.x86_64 libcom_err-1.41.14-2.fc15.x86_64 libselinux-2.1.6-5.fc16.x86_64 ncurses-libs-5.9-2.20110716.fc16.x86_64 openssl-1.0.0e-1.fc16.x86_64 (gdb) bt #0 cgetg (y=22, x=<optimized out>) at ../src/kernel/none/level1.h:114 #1 convi (x=0x288b2a8, l=0x7fff6a2f8a38) at ../src/kernel/gmp/mp.c:1288 #2 0x00007f11fb1637ec in itostr_sign (x=<optimized out>, sx=1, len=0x7fff6a2f8b48) at ../src/language/es.c:500 #3 0x00007f11fb167b4f in str_absint (x=0x288b2a8, S=0x7fff6a2f8cb0) at ../src/language/es.c:1778 #4 bruti_intern (g=0x288b2a8, T=<optimized out>, S=0x7fff6a2f8cb0, addsign=1) at ../src/language/es.c:2557 #5 0x00007f11fb168453 in bruti_intern (g=0x288b2d8, T=0x7f11fb4b27a0, S=0x7fff6a2f8cb0, addsign=<optimized out>) at ../src/language/es.c:2730 #6 0x00007f11fb1679ae in GENtostr_fun (out=0x7f11fb16a7b0 <bruti>, T=0x7f11fb4b27a0, x=0x288b2d8) at ../src/language/es.c:1645 #7 GENtostr (x=0x288b2d8) at ../src/language/es.c:1651 #8 0x00007f11f5ae5c44 in gcmp_sage (y=0x583d1b8, x=<optimized out>) at sage/libs/pari/misc.h:60 #9 __pyx_f_4sage_4libs_4pari_3gen_3gen__cmp_c_impl (__pyx_v_left=<optimized out>, __pyx_v_right=<optimized out>) at sage/libs/pari/gen.c:8513 #10 0x00007f11f8663227 in __pyx_f_4sage_9structure_7element_7Element__richcmp_c_impl (__pyx_v_left=0x5780e10, __pyx_v_right=<optimized out>, __pyx_v_op=2) at sage/structure/element.c:7775 #11 0x00007f11f86875ec in __pyx_f_4sage_9structure_7element_7Element__richcmp (__pyx_v_left=0x5780e10, __pyx_v_right=0x5863f70, __pyx_v_op=2) at sage/structure/element.c:7498 #12 0x00007f11f5ae045b in __pyx_pf_4sage_4libs_4pari_3gen_3gen_44__richcmp__ (__pyx_v_left=<optimized out>, __pyx_v_right=<optimized out>, __pyx_v_op=<optimized out>) at sage/libs/pari/gen.c:8475 #13 0x00007f1208b32e6a in try_rich_compare (v=0x5780e10, w=0x5863f70, op=2) at Objects/object.c:619 #14 0x00007f1208b3518d in try_rich_compare_bool (op=<optimized out>, w=<optimized out>, v=<optimized out>) at Objects/object.c:647 #15 try_rich_to_3way_compare (w=0x5863f70, v=0x5780e10) at Objects/object.c:681 #16 do_cmp (w=0x5863f70, v=0x5780e10) at Objects/object.c:834 #17 PyObject_Compare (v=0x5780e10, w=0x5863f70) at Objects/object.c:863 #18 0x00007f1208af5ae5 in PyObject_Cmp (o1=<optimized out>, o2=<optimized out>, result=0x7fff6a2f8f0c) at Objects/abstract.c:41 #19 0x00007f1208b879d4 in builtin_cmp (self=<optimized out>, args=<optimized out>) at Python/bltinmodule.c:422 #20 0x00007f1208b917fd in call_function (oparg=<optimized out>, pp_stack=0x7fff6a2f9000) at Python/ceval.c:3706 #21 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2389 #22 0x00007f1208b934d9 in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968 #23 0x00007f1208b1f7f6 in function_call (func=0x1fec9b0, arg=0x5864518, kw=0x0) at Objects/funcobject.c:524 #24 0x00007f1208af97a3 in PyObject_Call (func=0x1fec9b0, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #25 0x00007f1208b0667f in instancemethod_call (func=0x1fec9b0, arg=0x5864518, kw=0x0) at Objects/classobject.c:2579 #26 0x00007f1208af97a3 in PyObject_Call (func=0x579d0f0, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #27 0x00007f1208b545c6 in half_compare (self=<optimized out>, other=<optimized out>) at Objects/typeobject.c:5253 #28 0x00007f1208b547a5 in _PyObject_SlotCompare (self=0x5713af0, other=0x5866af0) at Objects/typeobject.c:5278 #29 0x00007f1208b35260 in do_cmp (w=0x5866af0, v=0x5713af0) at Objects/object.c:817 #30 PyObject_Compare (v=0x5713af0, w=0x5866af0) at Objects/object.c:863 #31 0x00007f1208af5ae5 in PyObject_Cmp (o1=<optimized out>, o2=<optimized out>, result=0x7fff6a2f955c) at Objects/abstract.c:41 #32 0x00007f1208b879d4 in builtin_cmp (self=<optimized out>, args=<optimized out>) at Python/bltinmodule.c:422 #33 0x00007f1208af97a3 in PyObject_Call (func=0x7f120903e2d8, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #34 0x00007f11e5b541fc in __pyx_pf_4sage_5rings_13residue_field_20ResidueField_generic_8__cmp__ ( __pyx_self=<optimized out>, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at sage/rings/residue_field.c:7317 #35 0x00007f1208af97a3 in PyObject_Call (func=0x22e7200, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 ---Type <return> to continue, or q <return> to quit--- #36 0x00007f1208b0667f in instancemethod_call (func=0x22e7200, arg=0x586d320, kw=0x0) at Objects/classobject.c:2579 #37 0x00007f1208af97a3 in PyObject_Call (func=0x4e32aa0, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #38 0x00007f1208b545c6 in half_compare (self=<optimized out>, other=<optimized out>) at Objects/typeobject.c:5253 #39 0x00007f1208b547a5 in _PyObject_SlotCompare (self=0x57f2410, other=0x5906e90) at Objects/typeobject.c:5278 #40 0x00007f1208b35260 in do_cmp (w=0x5906e90, v=0x57f2410) at Objects/object.c:817 #41 PyObject_Compare (v=0x57f2410, w=0x5906e90) at Objects/object.c:863 #42 0x00007f1208af5ae5 in PyObject_Cmp (o1=<optimized out>, o2=<optimized out>, result=0x7fff6a2f99bc) at Objects/abstract.c:41 #43 0x00007f1208b879d4 in builtin_cmp (self=<optimized out>, args=<optimized out>) at Python/bltinmodule.c:422 #44 0x00007f1208b917fd in call_function (oparg=<optimized out>, pp_stack=0x7fff6a2f9ab0) at Python/ceval.c:3706 #45 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2389 #46 0x00007f1208b92593 in fast_function (nk=<optimized out>, na=2, n=<optimized out>, pp_stack=0x7fff6a2f9c10, func=0x3b0a410) at Python/ceval.c:3792 #47 call_function (oparg=<optimized out>, pp_stack=0x7fff6a2f9c10) at Python/ceval.c:3727 #48 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2389 #49 0x00007f1208b934d9 in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968 #50 0x00007f1208b1f7f6 in function_call (func=0x3af28c0, arg=0x588d7a0, kw=0x0) at Objects/funcobject.c:524 #51 0x00007f1208af97a3 in PyObject_Call (func=0x3af28c0, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #52 0x00007f1208b0667f in instancemethod_call (func=0x3af28c0, arg=0x588d7a0, kw=0x0) at Objects/classobject.c:2579 #53 0x00007f1208af97a3 in PyObject_Call (func=0x4e29be0, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #54 0x00007f1208b545c6 in half_compare (self=<optimized out>, other=<optimized out>) at Objects/typeobject.c:5253 #55 0x00007f1208b547a5 in _PyObject_SlotCompare (self=0x579f908, other=0x58c5528) at Objects/typeobject.c:5278 #56 0x00007f1208b34dad in PyObject_RichCompare (v=0x579f908, w=0x58c5528, op=2) at Objects/object.c:967 #57 0x00007f1208b3505f in PyObject_RichCompareBool (v=<optimized out>, w=<optimized out>, op=<optimized out>) at Objects/object.c:1001 #58 0x00007f1208b49264 in tuplerichcompare (op=2, w=0x5898a70, v=0x578bf38) at Objects/tupleobject.c:546 #59 tuplerichcompare (v=0x578bf38, w=0x5898a70, op=2) at Objects/tupleobject.c:517 #60 0x00007f1208b34d71 in PyObject_RichCompare (v=0x578bf38, w=0x5898a70, op=2) at Objects/object.c:958 #61 0x00007f1208b3505f in PyObject_RichCompareBool (v=<optimized out>, w=<optimized out>, op=<optimized out>) at Objects/object.c:1001 #62 0x00007f1208b49264 in tuplerichcompare (op=2, w=0x5898ab8, v=0x579c368) at Objects/tupleobject.c:546 #63 tuplerichcompare (v=0x579c368, w=0x5898ab8, op=2) at Objects/tupleobject.c:517 #64 0x00007f1208b34d71 in PyObject_RichCompare (v=0x579c368, w=0x5898ab8, op=2) at Objects/object.c:958 #65 0x00007f1208b3505f in PyObject_RichCompareBool (v=<optimized out>, w=<optimized out>, op=<optimized out>) at Objects/object.c:1001 #66 0x00007f1208b2f305 in lookdict (mp=0x14c51d0, key=<optimized out>, hash=-1399715627429533172) at Objects/dictobject.c:351 #67 0x00007f1208b3087c in PyDict_DelItem (op=0x14c51d0, key=0x5898ab8) at Objects/dictobject.c:742 #68 0x00007f1208b8e924 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:1555 #69 0x00007f1208b934d9 in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kws=0x0, kwcount=0, defs=0x1458be8, defcount=1, closure=0x0) at Python/ceval.c:2968 #70 0x00007f1208b1f7f6 in function_call (func=0x1464320, arg=0x7f1208fdf510, kw=0x0) at Objects/funcobject.c:524 #71 0x00007f1208af97a3 in PyObject_Call (func=0x1464320, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #72 0x00007f1208afa1e0 in PyObject_CallFunctionObjArgs (callable=0x1464320) at Objects/abstract.c:2723 #73 0x00007f1208bc4146 in handle_weakrefs (old=0x7f1208e52b40, unreachable=0x7fff6a2fa700) ---Type <return> to continue, or q <return> to quit--- at Modules/gcmodule.c:607 #74 collect (generation=2) at Modules/gcmodule.c:859 #75 0x00007f1208bc4b04 in PyGC_Collect () at Modules/gcmodule.c:1292 #76 0x00007f1208bb6d73 in Py_Finalize () at Python/pythonrun.c:424 #77 0x00007f1208bb5c38 in Py_Exit (sts=0) at Python/pythonrun.c:1714 #78 0x00007f1208bb5d2f in handle_system_exit () at Python/pythonrun.c:1116 #79 0x00007f1208bb5fc5 in handle_system_exit () at Python/pythonrun.c:1078 #80 PyErr_PrintEx (set_sys_last_vars=1) at Python/pythonrun.c:1126 #81 0x00007f1208bb643e in PyRun_SimpleFileExFlags (fp=<optimized out>, filename=<optimized out>, closeit=1, flags=0x7fff6a2fa9e0) at Python/pythonrun.c:935 #82 0x00007f1208bc35a3 in Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:599 #83 0x00007f1207e7569d in __libc_start_main (main=0x400620 <main>, argc=3, ubp_av=0x7fff6a2fab08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff6a2faaf8) at libc-start.c:226 #84 0x0000000000400651 in _start ()
For the record, I enabled coredumps and then ran gdb --core core.4522 local/bin/python
comment:23 Changed 8 years ago by
Thank you! How does one enable coredumps?
comment:24 Changed 8 years ago by
If I understand correctly, the coredump says that it occurs while doing c = cmp(self.p, x.p)
, where x and self are residue fields.
comment:25 Changed 8 years ago by
Yep, I just inserted a print statement before and after the "cmp" line. When leaving sage, the first line was printed, and the segfault happened before printing the second line. Hence, the problem occurs when comparing fractional ideals.
comment:26 Changed 8 years ago by
To enable coredumps (at least with bash):
ulimit -c unlimited
comment:27 Changed 8 years ago by
Aha! A comparison of two sage.libs.pari.gen.gen happens after _unsafe_deallocate_pari_stack is called, which closes pari. That is, of course, bad.
Only I wonder how the order can be changed. Alternatively, it could be tested before comparison whether pari is still alive. But that would result in a slow-down.
comment:28 Changed 8 years ago by
I guess one must make sure that there is a strong reference to the (unique?) pari instance until all sage.libs.pari.gen.gen are deallocated.
comment:29 Changed 8 years ago by
pari._unsafe_deallocate_pari_stack
is called in sage.all.quit_sage. It does not help to move it to the end of quit_sage. I wonder why it is not put into a proper __del__
method of PariInstance
? Is it really needed to be in quit_sage??
comment:30 Changed 8 years ago by
Yessss! When removing _unsafe_deallocate_pari_stack
from quit_sage and renaming it into a __dealloc__
method, then the segfault vanishes!
comment:31 Changed 8 years ago by
Too bad. It fixes the segfault of sage -t sage/rings/number_field/number_field_ideal.py
, but it doesn't help for sage -t sage/schemes/elliptic_curves/heegner.py
.
Why is it always the elliptic curves code that causes trouble for my patches?
comment:32 Changed 8 years ago by
I have attached a second patch, that fixes two or three segfaults - which isn't enough.
comment:33 follow-up: ↓ 34 Changed 8 years ago by
- Cc jdemeyer added
In Python one must not use dealloc() to free C resources, at least not unless you are absolutely certain that the Python object does not participate in circular references.
Does it help to do an explicit gc.collect()
at the end of quit_sage
and only then deallocate Pari? If not we might have to give up clearing the Pari stack...
comment:34 in reply to: ↑ 33 Changed 8 years ago by
Replying to vbraun:
In Python one must not use dealloc() to free C resources, at least not unless you are absolutely certain that the Python object does not participate in circular references.
Do you mean __del__
? If I remember correctly, __dealloc__
is Cython, has nothing to do with the ability of the garbage collector to deal with circular references, and it is what one must have if there are C-resources to free after deleting all Python stuff. So, from all what I know, using __dealloc__
(not __del__
!) is a clean solution.
Does it help to do an explicit
gc.collect()
at the end ofquit_sage
and only then deallocate Pari? If not we might have to give up clearing the Pari stack...
I didn't try.
comment:35 Changed 8 years ago by
Yes, you are right: __dealloc__
is ok, __del__
is not.
But the problem seems to be that we finalize Pari before finalizing all Pari elements. Ideally, elements keep their parent alive because they hold a reference but I think GENs are often used in an ad-hoc way in Sage. So moving the Pari finalizer to __dealloc__
just makes it run later, but still gives no guarantees about finalizer ordering.
comment:36 Changed 8 years ago by
Pari elements have no parent, if I am not mistaken. Adding a parent means: Create an overhead, namely an additional pointer as part of all Pari elements. I am not sure if the number theorists would like that - one might ask on sage-nt.
comment:37 Changed 8 years ago by
There is sage.rings.pari_ring
which implements parents and elements. But when Pari is used in the Sage library its usually directly via its C API.
Looking at Python's C API, it seems that Py_AtExit()
is what we want: A callback for a cleanup function that is run after Python is finalized. In fact anything from quit_sage()
that just finalizes a C library should probably be moved there. See http://docs.python.org/c-api/sys.html
comment:38 follow-up: ↓ 39 Changed 8 years ago by
Not sure why I was added to "cc". But the newly added doctest in trac12215_segfault_fixes.patch looks bad because there really should be only one running PariInstance
, since global variables are used for the PARI stack (this is the fault of PARI, not of Sage).
comment:39 in reply to: ↑ 38 ; follow-up: ↓ 40 Changed 8 years ago by
Replying to jdemeyer:
But the newly added doctest in trac12215_segfault_fixes.patch looks bad because there really should be only one running
PariInstance
How else could one test that __dealloc__
works?
comment:40 in reply to: ↑ 39 ; follow-up: ↓ 41 Changed 8 years ago by
Replying to SimonKing:
How else could one test that
__dealloc__
works?
By Sage not crashing upon exit. I don't see any other way here.
comment:41 in reply to: ↑ 40 Changed 8 years ago by
comment:42 Changed 8 years ago by
- Work issues changed from segfaults for elliptic curves to fix it...
With sage-5.0.prealpha0 plus #11780 plus #715 plus #11521 plus #12290, all tests pass. But if one adds the two patches from here, one gets
sage -t -force_lib devel/sage/sage/combinat/combinatorial_algebra.py # 4 doctests failed sage -t -force_lib devel/sage/sage/combinat/partition.py # 3 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/kschur.py # 17 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/sfa.py # 284 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/macdonald.py # 107 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/hall_littlewood.py # 61 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/llt.py # 50 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/monomial.py # 16 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/orthotriang.py # 25 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/elementary.py # 9 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/homogeneous.py # 9 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/dual.py # 87 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/schur.py # 13 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/ns_macdonald.py # 2 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/powersum.py # 17 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/classical.py # 9 doctests failed sage -t -force_lib devel/sage/sage/combinat/sf/jack.py # 35 doctests failed sage -t -force_lib devel/sage/sage/combinat/species/product_species.py # 1 doctests failed sage -t -force_lib devel/sage/sage/combinat/species/composition_species.py # 2 doctests failed sage -t -force_lib devel/sage/sage/combinat/species/functorial_composition_species.py # 3 doctests failed sage -t -force_lib devel/sage/sage/combinat/species/generating_series.py # 44 doctests failed sage -t -force_lib devel/sage/sage/combinat/species/library.py # 4 doctests failed sage -t -force_lib devel/sage/sage/combinat/species/species.py # 2 doctests failed sage -t -force_lib devel/sage/sage/libs/pari/gen.pyx # Killed/crashed
Hopefully most of these errors have a common root.
comment:43 Changed 8 years ago by
It seems that a good deal of the errors comes from a method sage.combinat.sf.sf.SymmetricFunctions.register_isomorphism
: It registers a coercion, but this is only possible when no coercion has been established for that object before.
What should one do: Catch the error and 'not registering the coercion? Or wipe the registered coercions, by calling sage.structure.parent.Parent.unset_coercions_used
?
comment:44 Changed 8 years ago by
I have to slightly modify my preceding statement. The error is raised not if there has been established a coercion for that object before, but if there has been a coercion registered between the two objects before. Anyway, the problem remains the same.
comment:45 Changed 8 years ago by
Here is a very short example triggering the error:
sage: P = JackPolynomialsP(QQ,1) sage: P([2,1])^2
Hopefully this is short enough for debugging - I find it quite mysterious, so far.
comment:46 Changed 8 years ago by
- Cc mhansen added
Something else is interesting: The error changes when repeating it.
sage: P = JackPolynomialsP(QQ,1) sage: p = P([2,1]) sage: p^2 ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (56, 0)) ERROR: Internal Python error in the inspect module. Below is the traceback from this internal error. Traceback (most recent call last): ... /home/simon/SAGE/sage-5.0.prealpha0/local/lib/python2.7/site-packages/sage/combinat/sf/sf.pyc in register_isomorphism(self, morphism) 324 mathematically wrong, as above. Use with care! 325 """ --> 326 morphism.codomain().register_coercion(morphism) 327 328 _shorthands = set(['e', 'h', 'm', 'p', 's']) /home/simon/SAGE/sage-5.0.prealpha0/local/lib/python2.7/site-packages/sage/structure/parent.so in sage.structure.parent.Parent.register_coercion (sage/structure/parent.c:11955)() /home/simon/SAGE/sage-5.0.prealpha0/local/lib/python2.7/site-packages/sage/structure/parent.so in sage.structure.parent.Parent.register_coercion (sage/structure/parent.c:11889)() AssertionError: coercion from Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis to Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis already registered or discovered sage: p^2 ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (56, 0)) --------------------------------------------------------------------------- KeyError Traceback (most recent call last) ... /home/simon/SAGE/sage-5.0.prealpha0/local/lib/python2.7/site-packages/sage/combinat/sf/sfa.pyc in _from_cache(self, element, cache_function, cache_dict, **subs_dict) 631 if sum(part) not in cache_dict: 632 cache_function(sum(part)) --> 633 for part2, c2 in cache_dict[sum(part)][part].iteritems(): 634 c3 = c*c2 635 if hasattr(c3,'subs'): # c3 may be in the base ring KeyError: [2, 1]
Cc to the author of sage/combinat/sf/jack.py.
comment:47 Changed 8 years ago by
I inserted some print statement into the register_isomorphism
method of symmetric functions. I found that with or without the patch, the isomorphisms are registered both during initialisation of the JackPolynomialsP
and before raising an element to a power for the first time:
sage: P = JackPolynomialsP(QQ,1) registering Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Schur symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Power symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Schur symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Schur symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Homogeneous symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Power symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Schur symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Homogeneous symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Homogeneous symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Power symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Homogeneous symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Power symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Homogeneous symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Power symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Schur symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Schur symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Elementary symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Homogeneous symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Power symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Homogeneous symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Schur symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Power symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Homogeneous symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Schur symmetric functions as basis registering Symmetric Function Algebra over Rational Field, Power symmetric functions as basis TO Symmetric Function Algebra over Rational Field, Monomial symmetric functions as basis sage: p = P([2,1]) sage: p^2 registering Symmetric Function Algebra over Integer Ring, Monomial symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Elementary symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Monomial symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Schur symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Power symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Schur symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Schur symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Homogeneous symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Elementary symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Power symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Schur symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Elementary symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Homogeneous symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Elementary symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Monomial symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Homogeneous symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Power symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Homogeneous symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Power symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Elementary symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Homogeneous symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Power symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Elementary symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Monomial symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Elementary symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Schur symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Schur symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Monomial symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Elementary symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Homogeneous symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Monomial symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Power symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Homogeneous symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Monomial symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Schur symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Power symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Homogeneous symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Schur symmetric functions as basis registering Symmetric Function Algebra over Integer Ring, Power symmetric functions as basis TO Symmetric Function Algebra over Integer Ring, Monomial symmetric functions as basis JackP[2, 2, 1, 1] + JackP[2, 2, 2] + JackP[3, 1, 1, 1] + 2*JackP[3, 2, 1] + JackP[3, 3] + JackP[4, 1, 1] + JackP[4, 2] sage: p^2 JackP[2, 2, 1, 1] + JackP[2, 2, 2] + JackP[3, 1, 1, 1] + 2*JackP[3, 2, 1] + JackP[3, 3] + JackP[4, 1, 1] + JackP[4, 2]
This gives rise to some questions:
- Why are the symmetric functions registering the isomorphisms twice, even without my patch?
- Why is there no error without my patch? There should be, since double-registration of a coercion is illegal!
I guess, the best solution would be to address the first question: Registering the same thing twice is a waste or resources anyway.
comment:48 Changed 8 years ago by
Sorry, I was mistaken! It is not two times the same! The first time it is over the rational field, the second time over the integer ring. So, forget my previous questions.
comment:49 Changed 8 years ago by
Now I understand the problem:
SymmetricFunctions
is a subclass of UniqueRepresentation
. By my patch, UniqueRepresentation
is using weak references. Apparently SymmetricFunctions
therefore can be garbage collected, but - and now comes the strange point - the coercion system still recalls that a coercion to them has already been registered.
Anyway, with my patch, SymmetricFunctions(ZZ)
and SymmetricFunctions?(QQ)` are created repeatedly, and that's bad.
comment:50 Changed 8 years ago by
- Status changed from needs_work to needs_review
- Work issues fix it... deleted
I updated the second patch, which should solve the problem!!
First of all: The segfault in the tests of sage/libs/pari/gen.pyx was due to my test for the new dealloc method. Following Jeroen's advice, I removed it and stated in the docs that Sage not crashing at exit is an indirect doctest.
Then, all failures in sage/combinat could be fixed by using a strong cache for SymmetricFunctions(...)
. So, I simply overrode the __classcall__
method inherited from UniqueRepresentation
.
I just tested that with the new patch all tests in sage/combinat and sage/libs/pari/gen.pyx pass. The others passed even with the old patch version, so that I am confident that they will pass as well (of course, one must try!).
comment:51 Changed 8 years ago by
FWIW, make ptest
succeeded.
comment:52 Changed 8 years ago by
As I said in my previous post, the tests pass with this patch. The tests also pass with the patch from #12313. However, there are three segfaults that occur when both patches are applied. I have difficulties to trace it down.
comment:53 Changed 8 years ago by
- Cc vbraun added
Cc to Volker, because I expect he has enough knowledge to give me some advice on how I could trace down the following segfault.
With #12313 and the patch from here, sage -t -verbose -force_lib "devel/sage/doc/en/bordeaux_2008/half_integral.rst"
segfaults. By inspection of the core file, I found that the segfault occurs during deallocation of a functor.
For debugging, I added a __dealloc__
method to sage.categories.functor.Functor
that writes the type and the address of self and of the two cdef attributes __domain
and __codomain
to some file. The same is done during initialisation of the functor.
And the last lines of the resulting file (before the segfault) are:
Dealloc Functor <type 'sage.structure.coerce_actions.LeftModuleAction'> at 71023056 Domain <class 'sage.categories.groupoid.Groupoid'> at 75636560 Codom. <class 'sage.categories.commutative_rings.CommutativeRings'> at 15429144 Dealloc Functor <type 'sage.structure.coerce_actions.LeftModuleAction'> at 71023056 Domain <type 'NoneType'> at 140661532564960 Codom. <type 'NoneType'> at 140661532564960
In other words, the functor is deallocated twice, which is a legitimate reason to segfault.
How can I find out why Sage tries to deallocate it twice?
comment:54 follow-ups: ↓ 55 ↓ 56 Changed 8 years ago by
Is it actually being finalized twice? To me, it seems that just the malloc bin was reused for a second LeftModuleAction
instance. In particular, why would domain and codomain be different in the second destructor call.
comment:55 in reply to: ↑ 54 Changed 8 years ago by
Replying to vbraun:
In particular, why would domain and codomain be different in the second destructor call.
Because domain and codomain were deleted the first time. The second time, they already are NoneType
.
comment:56 in reply to: ↑ 54 Changed 8 years ago by
Replying to vbraun:
Is it actually being finalized twice? To me, it seems that just the malloc bin was reused for a second
LeftModuleAction
instance.
And I do believe it is the same instance. Namely, if what you say was right, then we should see a call to "init" between the two deallocations (I made both init and dealloc write to the same log file). But the two deallocations followed directly: No initialisation and no other deallocation in between.
comment:57 Changed 8 years ago by
No progress on my side. For my project, it probably means that I have to pick between two evils: Either live with the memleak that would be fixed in #12313, or live with the memleak that would be fixed here. Bad.
comment:58 Changed 8 years ago by
- Milestone changed from sage-5.0 to sage-4.8
Now that's weird:
When I define
def __dealloc__(self): if self.__domain is not None: Py_INCREF(self.__domain) if self.__codomain is not None: Py_INCREF(self.__codomain)
for sage.categories.functor.Functor, then the segfault disappears.
Can this be a solution? It looks weird.
comment:59 Changed 8 years ago by
- Milestone changed from sage-4.8 to sage-5.0
comment:60 Changed 8 years ago by
- Description modified (diff)
I have updated the second patch, which was about fixing segfaults anyway.
As I already stated: I find it weird that the problem is solved by incrementing the reference count of the domain and codomain of an action when the action is deallocated. But it works, i.e., the doctests that used to segfault with #12313 and the old version of the patches run fine with the new patch version.
I need an expert opinion, though, and the full test suite is also to be run.
Concerning memleaks, here is the example from the ticket description.
With #12313 and the patches from here:
sage: import sage.structure.unique_representation sage: len(sage.structure.unique_representation.UniqueRepresentation.__classcall__.cache) 135 sage: sage: for i in range(2,1000): ....: ring = ZZ.quotient(ZZ(i)) ....: vectorspace = ring^2 ....: sage: import gc sage: gc.collect() 16641 sage: len(sage.structure.unique_representation.UniqueRepresentation.__classcall__.cache) 227
With #12313 only:
sage: import sage.structure.unique_representation sage: len(sage.structure.unique_representation.UniqueRepresentation.__classcall__.cache) 151 sage: sage: for i in range(2,1000): ....: ring = ZZ.quotient(ZZ(i)) ....: vectorspace = ring^2 ....: sage: import gc sage: gc.collect() 3805 sage: len(sage.structure.unique_representation.UniqueRepresentation.__classcall__.cache) 5142
So, it is a clear progress, and IIRC the patches comprise tests against at least one memory leak that is fixed. Needs review!
Apply trac12215_weak_cached_function.patch trac12215_segfault_fixes.patch
comment:61 Changed 8 years ago by
- Status changed from needs_review to needs_work
- Work issues set to Fix two tests
With sage-5.0.prealpha0 plus #11780, #11290, #715, #11521, #12313 and the patches from here, make ptest results in
sage -t -force_lib devel/sage/sage/combinat/sf/sf.py # 1 doctests failed sage -t -force_lib devel/sage/sage/categories/category.py # 1 doctests failed
So, it needs work (because all tests pass when the patches from here are not applied), but it should hopefully be easy to fix.
comment:62 Changed 8 years ago by
I tried the following in cdef class Action
:
def __cinit__(self): print 'Action __cinit__ ' + str(id(self)) def __dealloc__(self): print 'Action __dealloc__ ' + str(id(self))
then I do get occasionally reused id (=memory address in CPython), for example
Action __cinit__ 105376976 Action __dealloc__ 105376976 Action __cinit__ 105376976 Action __dealloc__ 105376976
But I don't see any double finalizers without the object being constructed in-between. I also don't get any segfault in bordeaux_2008/half_integral.rst
.
comment:63 follow-up: ↓ 64 Changed 8 years ago by
For the record, I have these patches applied on top of sage-4.8.rc0:
12221_debug.patch trac_12247_var_construction.patch 9138_flat.patch trac11900_category_speedup_combined.patch trac11900_only_fix_singleton_hash.patch trac11900_doctest.patch 11115_flat.patch trac_11115_docfix.patch trac12215_weak_cached_function.patch trac12215_segfault_fixes.patch
removed the Py_INCREF(self.__domain)
and Py_INCREF(self.__codomain)
bandaid. Still no segfault.
comment:64 in reply to: ↑ 63 Changed 8 years ago by
Replying to vbraun:
For the record, I have these patches applied on top of sage-4.8.rc0:
12221_debug.patch trac_12247_var_construction.patch 9138_flat.patch trac11900_category_speedup_combined.patch trac11900_only_fix_singleton_hash.patch trac11900_doctest.patch 11115_flat.patch trac_11115_docfix.patch trac12215_weak_cached_function.patch trac12215_segfault_fixes.patchremoved the
Py_INCREF(self.__domain)
andPy_INCREF(self.__codomain)
bandaid. Still no segfault.
Sure. As I stated in some post above, the segfault only results when applying both #12313 (hence, its dependency #715 as well) and the (old) patches from here.
If you only have the (old or new) patches from here or only have #715+#12313 then there is no segfault.
comment:65 follow-up: ↓ 67 Changed 8 years ago by
I ran all doctests and there are a few crashes in functor.so
elsewhere. I didn't have to apply any additional patches. It dies with
Action __cinit__ 84546128 Action __dealloc__ 84546128 Action __cinit__ 84546128 Action __dealloc__ 84546128 Action __cinit__ 84628736 Action __cinit__ 84546128 Action __dealloc__ 84546128 Action __dealloc__ 84546128 /home/vbraun/opt/sage-4.8.rc0/local/lib/libcsage.so(print_backtrace+0x31)[0x7fcc0db1adf6] /home/vbraun/opt/sage-4.8.rc0/local/lib/libcsage.so(sigdie+0x14)[0x7fcc0db1ae28] /home/vbraun/opt/sage-4.8.rc0/local/lib/libcsage.so(sage_signal_handler+0x20c)[0x7fcc0db1aa76]
It seems that its just memory corruption that manifests itself by freeing the object twice. But the error is presumably elsewhere. Also the gdb stack trace is completely corrupted.
comment:66 Changed 8 years ago by
Here is the stack trace:
#0 0x00007ffaadb88511 in __pyx_tp_dealloc_4sage_10categories_7functor_Functor (o=0x63ed250) at sage/categories/functor.c:2845 #1 0x00007ffaad970cc8 in __pyx_tp_dealloc_4sage_10categories_6action_Action (o=0x63ed250) at sage/categories/action.c:5943 #2 0x00007ffaad5485a0 in __pyx_tp_dealloc_4sage_9structure_14coerce_actions_ModuleAction (o=0x63ed250) at sage/structure/coerce_actions.c:7505 #3 0x00007ffabbcf8f0c in type_call (type=<optimized out>, args=0x63e09e0, kwds=0x0) at Objects/typeobject.c:748 #4 0x00007ffabbca27a3 in PyObject_Call (func=0x7ffaad754ec0, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #5 0x00007ffaad53ffbb in __pyx_pf_4sage_9structure_14coerce_actions_1detect_element_action (__pyx_self=0x0, __pyx_args=0x63fbb40, __pyx_kwds=0x0) at sage/structure/coerce_actions.c:4616 #6 0x00007ffabbca27a3 in PyObject_Call (func=0x2683dd0, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #7 0x00007ffaaeb0ea32 in __pyx_f_4sage_9structure_6parent_6Parent_discover_action (__pyx_v_self=0x644ab00, __pyx_v_S=0x6448770, __pyx_v_op=0x7ffab525aea8, __pyx_v_self_on_left=1) at sage/structure/parent.c:16618 #8 0x00007ffaaed48057 in __pyx_f_4sage_9structure_10parent_old_6Parent_get_action_c_impl (__pyx_v_self=0x644ab00, __pyx_v_S=0x6448770, __pyx_v_op=0x7ffab525aea8, __pyx_v_self_on_left=1) at sage/structure/parent_old.c:3312 #9 0x00007ffaaed47ea2 in __pyx_pf_4sage_9structure_10parent_old_6Parent_4get_action_impl (__pyx_v_self=0x644ab00, __pyx_args=0x63fb910, __pyx_kwds=0x0) at sage/structure/parent_old.c:3258 #10 0x00007ffabbca27a3 in PyObject_Call (func=0x636a5a8, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2492 #11 0x00007ffaaed46ee7 in __pyx_f_4sage_9structure_10parent_old_6Parent_get_action_c (__pyx_v_self=0x644ab00, __pyx_v_S=0x6448770, __pyx_v_op=0x7ffab525aea8, __pyx_v_self_on_left=1, __pyx_skip_dispatch=0) at sage/structure/parent_old.c:2935 #12 0x00007ffaaed4f19d in __pyx_f_4sage_9structure_10parent_old_6Parent__get_action_ (__pyx_v_self=0x644ab00, __pyx_v_other=0x6448770, __pyx_v_op=0x7ffab525aea8, __pyx_v_self_on_left=1, __pyx_skip_dispatch=0) at sage/structure/parent_old.c:6228 #13 0x00007ffaaeb0b17c in __pyx_f_4sage_9structure_6parent_6Parent_get_action (__pyx_v_self=0x644ab00, __pyx_v_S=0x6448770, __pyx_skip_dispatch=0, __pyx_optional_args=0x7fff38b8e2f0) at sage/structure/parent.c:15635 #14 0x00007ffaae1fa2e6 in __pyx_f_4sage_9structure_6coerce_24CoercionModel_cache_maps_discover_action (__pyx_v_self=0x26286d0, __pyx_v_R=0x644ab00, __pyx_v_S=0x6448770, __pyx_v_op=0x7ffab525aea8, __pyx_skip_dispatch=0) at sage/structure/coerce.c:12473 #15 0x00007ffaae1f6564 in __pyx_f_4sage_9structure_6coerce_24CoercionModel_cache_maps_get_action (__pyx_v_self=0x26286d0, __pyx_v_R=0x644ab00, __pyx_v_S=0x6448770, __pyx_v_op=0x7ffab525aea8, __pyx_skip_dispatch=0) at sage/structure/coerce.c:11424 #16 0x00007ffaae1e64e2 in __pyx_f_4sage_9structure_6coerce_24CoercionModel_cache_maps_bin_op (__pyx_v_self=0x26286d0, __pyx_v_x=0x6354b48, __pyx_v_y=0x63e36b0, __pyx_v_op=0x7ffab525aea8, __pyx_skip_dispatch=0) at sage/structure/coerce.c:6583 #17 0x00007ffaae448f03 in __pyx_pf_4sage_9structure_7element_6Vector_1__mul__ (__pyx_v_left=0x6354b48, __pyx_v_right=0x63e36b0) at sage/structure/element.c:16130 #18 0x00007ffabbc9dc5f in binary_op1 (v=0x6354b48, w=0x63e36b0, op_slot=16) at Objects/abstract.c:917 #19 0x00007ffabbca0cc8 in PyNumber_Multiply (v=0x6354b48, w=0x63e36b0) at Objects/abstract.c:1188 #20 0x00007ffa9be33b68 in __pyx_f_4sage_5rings_13residue_field_12ReductionMap__call_ (__pyx_v_self=0x63f10e8, __pyx_v_x=0x6405108, __pyx_skip_dispatch=0) at sage/rings/residue_field.c:8140
within coercion_model.bin_op()
(frame 17) there are calls to Python methods (PyObject_Call
), and in there the garbage collector is free to run. I suspect that this is what is happening somewhere...
comment:67 in reply to: ↑ 65 Changed 8 years ago by
Replying to vbraun:
I ran all doctests and there are a few crashes in
functor.so
elsewhere. I didn't have to apply any additional patches.
What exactly do you mean? Do you have the old patches from here applied (i.e., without the new __dealloc__
method), or does the segfault even occur with the new patches?
Is it normal that both you and me see segfaults, and it seems to be analogous problems (namely double deallocation), but we see it in different examples and with different patches (namely, even with the old patches from here, all tests pass for me)?
It dies with ... It seems that its just memory corruption that manifests itself by freeing the object twice.
So, you can confirm that it is the same object.
But the error is presumably elsewhere. Also the gdb stack trace is completely corrupted.
That sounds like one should write a complete log of all python code executed - according to your suggestion that the error somewhere occurs during a Python method.
comment:68 Changed 8 years ago by
Here is some more info on the segfault.
Setting: I have sage-5.0.prealpha0 plus #11780, #11290, #715, #11521, #12313 and the patches from here, removing the __dealloc__
method introduced by the last patch.
The segfault is triggered by doing
sage: half_integral_weight_modform_basis(DirichletGroup(16,QQ).1, 3, 10) [] sage: half_integral_weight_modform_basis(DirichletGroup(16,QQ).1, 5, 10) /home/simon/SAGE/sage-5.0.prealpha0/local/lib/libcsage.so(print_backtrace+0x31)[0x7fe047add9c6] /home/simon/SAGE/sage-5.0.prealpha0/local/lib/libcsage.so(sigdie+0x14)[0x7fe047add9f8] /home/simon/SAGE/sage-5.0.prealpha0/local/lib/libcsage.so(sage_signal_handler+0x20c)[0x7fe047add646] /lib64/libpthread.so.0(+0xfd00)[0x7fe04cd80d00] ...
When I revert the lines, that's to say, if I do
sage: half_integral_weight_modform_basis(DirichletGroup(16,QQ).1, 5, 10) [q - 2*q^3 - 2*q^5 + 4*q^7 - q^9 + O(q^10)] sage: half_integral_weight_modform_basis(DirichletGroup(16,QQ).1, 3, 10) [] sage: quit Exiting Sage (CPU time 0m2.02s, Wall time 0m20.49s). ********************************************************************** Oops, Sage crashed. We do our best to make it stable, but... A crash report was automatically generated with the following information: - A verbatim copy of the crash traceback. - A copy of your input history during this session. - Data on your current Sage configuration. It was left in the file named: '/home/simon/.sage/ipython/Sage_crash_report.txt' If you can email this file to the developers, the information in it will help them in understanding and correcting the problem. You can mail it to: sage-support at sage-support@googlegroups.com with the subject 'Sage Crash Report'. If you want to do it now, the following command will work (under Unix): mail -s 'Sage Crash Report' sage-support@googlegroups.com < /home/simon/.sage/ipython/Sage_crash_report.txt To ensure accurate tracking of this issue, please file a report about it at: http://trac.sagemath.org/sage_trac Press enter to exit:
I was tracing all python commands for the first variant of the segfault. The last few lines of the log are as follows:
sage.categories.pushout:__call__:2125 if self.p == other.p: sage.categories.pushout:__call__:2126 from sage.all import Infinity sage.categories.pushout:__call__:2127 if self.prec == other.prec: sage.categories.pushout:__call__:2128 extras = self.extras.copy() sage.categories.pushout:__call__:3102 except CoercionException: sage.categories.pushout:__call__:3104 except (TypeError, ValueError, AttributeError, NotImplementedError), ex: sage.categories.pushout:__call__:3108 raise CoercionException(ex) weakref:__call__:49 self = selfref() weakref:__call__:50 if self is not None: weakref:__call__:51 del self.data[wr.key] sage.rings.power_series_ring:__call__:556 s = "Power Series Ring in %s over %s"%(self.variable_name(), self.base_ring()) sage.rings.power_series_ring:__call__:557 if self.is_sparse(): sage.rings.power_series_ring:__call__:562 return self.__is_sparse sage.rings.power_series_ring:__call__:559 return s
So, indeed it seems that the problem has something to do with weak references. There is an item of a weak value dictionary deleted right before segfaulting.
To do: Find out what item of what dictionary is deleted, why it is deleted, and how deletion can be prevented.
comment:69 Changed 8 years ago by
I was also tracing the deletion of items of weak value dictionaries: I was writing the key to a log file whenever an item was deleted.
Already when starting sage, we see that the same key (and presumably the same value as well) is deleted repeatedly:
... ((<class 'sage.categories.category.JoinCategory'>, (Category of semirings, Category of infinite enumerated sets)), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Integer Ring), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Rational Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Rational Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Rational Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Complex Lazy Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Rational Field), ())
When issuing the first line of the crashing example and repeating it, we see something like
... ((<class 'sage.categories.groupoid.Groupoid'>, Complex Lazy Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Complex Lazy Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Cyclotomic Field of order 4 and degree 2), ()) ((<class 'sage.matrix.matrix_space.MatrixSpace'>, Rational Field, 0, 0, False), ()) ((<class 'sage.matrix.matrix_space.MatrixSpace'>, Rational Field, 10, 0, False), ()) ((5, 0, 'prealpha0'), (Rational Field, 0, False, None)) ((<class 'sage.matrix.matrix_space.MatrixSpace'>, Rational Field, 0, 0, False), ()) ((<class 'sage.matrix.matrix_space.MatrixSpace'>, Rational Field, 10, 0, False), ())
And at crashing, one has
((<class 'sage.matrix.matrix_space.MatrixSpace'>, Ring of integers modulo 46337, 4, 10, False), ()) ((<class 'sage.categories.vector_spaces.VectorSpaces'>, Ring of integers modulo 46337), ()) ((<class 'sage.matrix.matrix_space.MatrixSpace'>, Integer Ring, 4, 10, False), ()) ((<class 'sage.matrix.matrix_space.MatrixSpace'>, Rational Field, 4, 10, False), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Power Series Ring in q over Rational Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Power Series Ring in q over Rational Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Power Series Ring in q over Rational Field), ()) ((<class 'sage.categories.groupoid.Groupoid'>, Power Series Ring in q over Integer Ring), ())
Conclusion:
The occurring keys indicate that the deletions occur in UniqueRepresentation
. While using weak references for UniqueRepresentation
fixes memory leaks, it seems that far too often stuff is removed that would actually still be needed. Certainly it is bad for speed, and it seems that it is also responsible for the segmentation faults.
I am not sure how that problem should best be addressed.
comment:70 Changed 8 years ago by
- Work issues changed from Fix two tests to Fix a coercion problem in sage.combinat.sf.sf
I think I have not properly stated that with the latest patches applied to sage-5.0.prealpha0, the segfault is gone. However, at least when I also have a couple of other tickets (#11780, #12290, #715. #11521, #12313, #12357, #12351, #7797), I get one coercion error in sage.combinat.sf.sf.
To be precise, I do not get that error when I only have all the other patches. So, it really seems to be caused by the patches from here. Trying to track it down...
comment:71 Changed 8 years ago by
That's odd. The failing test is from the __call__
method in sage.combinat.sf.sf. When I execute things in the command line, I get the following:
sage: Sym = SymmetricFunctions(QQ[x]) sage: p = Sym.p(); s = Sym.s() sage: P = p[1].parent() sage: S = s[1].parent() sage: P.coerce_map_from(S) Generic morphism: From: Symmetric Function Algebra over Univariate Polynomial Ring in x over Rational Field, Schur symmetric functions as basis To: Symmetric Function Algebra over Univariate Polynomial Ring in x over Rational Field, Power symmetric functions as basis sage: S.coerce_map_from(P) Generic morphism: From: Symmetric Function Algebra over Univariate Polynomial Ring in x over Rational Field, Power symmetric functions as basis To: Symmetric Function Algebra over Univariate Polynomial Ring in x over Rational Field, Schur symmetric functions as basis
However, when the same is executed as a doctest, then there is no coercion map between S and P. Could it be that some other doctest is messing with the coercion maps, and my patch (perhaps in combination with #715 and #11521) reveals it?
comment:72 Changed 8 years ago by
- Dependencies changed from #11115 #11900 to #11115 #11900 #12645
- Status changed from needs_work to needs_review
- Work issues Fix a coercion problem in sage.combinat.sf.sf deleted
That's even odder. With #11780, #12290, #715. #11521, #12313, #12357, #12351, #7797 and #12645 (so, adding #12645, which only changes the rst markup in sage/combinat/sf/sf.py), all tests in sage/combinat pass.
Anyway. Since the second patch is in conflict with #12645 anyway, I am rebasing it. Since the doctest error has vanished, I put it back to "needs review", even though I wish I knew what was the reason for the temporary problem.
comment:73 Changed 8 years ago by
- Status changed from needs_review to needs_work
- Work issues set to coercion in symmetric function algebras
Bad. Meanwhile I work on top of sage-5.0.beta7. This time, it is the first patch that creates a coercion error in sage/combinat/sf/sf.py. Needs work.
comment:74 Changed 8 years ago by
comment:75 Changed 8 years ago by
- Dependencies changed from #11115 #11900 #12645 to #11115 #11900 #12645 #11599
- Work issues changed from coercion in symmetric function algebras to Rebase wrt #11599. Coercion in symmetric function algebras
It comes from #11599, which fixes the same docstring misformattings that I fix in my patch as well...
comment:76 Changed 8 years ago by
Arrgh. With #715, #11521, #12313, #11943, #11935, #12357 and #7797 on top of sage-5.0.beta13, all tests pass. But adding the (rebased) patch from here, I get failures in
sage -t -force_lib "devel/sage/sage/structure/coerce_dict.pyx" sage -t -force_lib "devel/sage/sage/combinat/sf/macdonald.py" sage -t -force_lib "devel/sage/sage/combinat/sf/llt.py" sage -t -force_lib "devel/sage/sage/combinat/sf/jack.py" sage -t -force_lib "devel/sage/sage/combinat/sf/kschur.py" sage -t -force_lib "devel/sage/sage/combinat/sf/hall_littlewood.py" sage -t -force_lib "devel/sage/sage/combinat/sf/sfa.py" sage -t -force_lib "devel/sage/sage/combinat/sf/multiplicative.py" sage -t -force_lib "devel/sage/sage/combinat/sf/schur.py" sage -t -force_lib "devel/sage/sage/combinat/species/library.py" sage -t -force_lib "devel/sage/sage/combinat/combinatorial_algebra.py" sage -t -force_lib "devel/sage/sage/categories/homset.py"
That's not good.
comment:77 Changed 8 years ago by
Oops, I had only the first of the two patches from here applied. Nevertheless, it doesn't look good.
comment:78 Changed 8 years ago by
- Work issues changed from Rebase wrt #11599. Coercion in symmetric function algebras to Coercion in symmetric function algebras
I have rebased the first patch relative to #11599.
With both patches, one "only" has errors in
sage -t -force_lib "devel/sage/sage/structure/coerce_dict.pyx" sage -t -force_lib "devel/sage/sage/combinat/sf/sf.py" sage -t -force_lib "devel/sage/sage/categories/homset.py"
So, it still needs work, but it is less bad than I thought...
Apply trac12215_weak_cached_function.patch trac12215_segfault_fixes.patch
comment:79 Changed 8 years ago by
A bit more detail: The tests in coerce_dict.pyx and homset.py fail even if only the first patch is applied. But the tests in sf.py pass if only the first patch is applied.
comment:80 Changed 8 years ago by
- Work issues changed from Coercion in symmetric function algebras to Keep the fix from #12313. Coercion in symmetric function algebras
I tested whether the problem comes from the combination of this ticket with #12357. But it turns out that the following test
sage: K = GF(1<<55,'t') sage: for i in range(50): ... a = K.random_element() ... E = EllipticCurve(j=a) ... P = E.random_point() ... Q = 2*P sage: import gc sage: n = gc.collect() sage: from sage.schemes.elliptic_curves.ell_finite_field import EllipticCurve_finite_field sage: LE = [x for x in gc.get_objects() if isinstance(x, EllipticCurve_finite_field)] sage: len(LE) # indirect doctest 1
still fails. The test has been introduced in #12313. And of course it is not acceptable that #12313 makes a memory leak disappear, but #12215 makes it show up again.
comment:81 Changed 8 years ago by
I think I located the problem. By some patch, I had introduced a weak dictionary in sage.structure.factory. But somehow I managed to remove the corresponding hunk from the patch. Now, I need to find out where that has happened...
comment:82 Changed 8 years ago by
Aha! It turns out that I introduced the WeakValueDictionary
in the first patch from here, but somehow I managed to delete it. Now the leak remains fixed, the patch is updated.
Apply trac12215_weak_cached_function.patch trac12215_segfault_fixes.patch
comment:83 Changed 8 years ago by
comment:84 Changed 8 years ago by
- Owner changed from rlm to (none)
What exactly is the problem?
It is
sage: S = SymmetricFunctions(ZZ) sage: S.inject_shorthands() doctest:...: RuntimeWarning: redefining global value `e` doctest:...: RuntimeWarning: redefining global value `m` sage: s[1] + e[2] * p[1,1] + 2*h[3] + m[2,1] s[1] - 2*s[1, 1, 1] + s[1, 1, 1, 1] + s[2, 1] + 2*s[2, 1, 1] + s[2, 2] + 2*s[3] + s[3, 1]
The last line fails with an error when doctesting, but works fine when doing the same in an interactive session.
comment:85 Changed 8 years ago by
The failure is really strange. If one does
sage: S = SymmetricFunctions(ZZ) sage: S.inject_shorthands() sage: e.has_coerce_map_from(m)
on the command line, then one gets the answer "True". Doing the same in a separate doctest, one still gets "True". But doing the same in line 384 of sage.combinat.sf.sf.py, one gets "False". So, there seems to be a nasty diffcult-to-debug side effect, which apparently was introduced by the second patch.
comment:86 Changed 8 years ago by
The error disappears if one does not override that __classcall__
method of symmetric function algebras. However, by the first ticket, it uses a weak cache, which results in many errors elsewhere...
But if I recall correctly, there has been a recent ticket dealing with coercion for symmetric functions. Perhaps a miracle occurs and the strongly cached custom __classcall__
can be cancelled (count the words that start with "c"...)?
comment:87 Changed 8 years ago by
How unfortunate. If I remove the custom (strongly cached) __classcall__
of symmetric function algebras, I get
The following tests failed: sage -t -force_lib "devel/sage/sage/combinat/sf/macdonald.py" sage -t -force_lib "devel/sage/sage/combinat/sf/llt.py" sage -t -force_lib "devel/sage/sage/combinat/sf/jack.py" sage -t -force_lib "devel/sage/sage/combinat/sf/kschur.py" sage -t -force_lib "devel/sage/sage/combinat/sf/hall_littlewood.py" sage -t -force_lib "devel/sage/sage/combinat/sf/classical.py" sage -t -force_lib "devel/sage/sage/combinat/sf/sfa.py" sage -t -force_lib "devel/sage/sage/combinat/sf/elementary.py" sage -t -force_lib "devel/sage/sage/combinat/sf/multiplicative.py" sage -t -force_lib "devel/sage/sage/combinat/sf/schur.py" sage -t -force_lib "devel/sage/sage/combinat/sf/homogeneous.py" sage -t -force_lib "devel/sage/sage/combinat/species/library.py" sage -t -force_lib "devel/sage/sage/combinat/combinatorial_algebra.py"
But if one has a custom strong cache for symmetric function algebras, then one has the single failure in
sage -t -force_lib "devel/sage/sage/combinat/sf/sf.py"
comment:88 Changed 8 years ago by
Hooray! It turns out that one can fix the failing doctest in sf.py by not only providing a strongly cached sf.SymmetricFunctions.__classcall__
, but by additionally providing a strongly cached sfa.SymmetricFunctionAlgebra_generic.__classcall__
.
It is a bit unfortunate that a strong cache creeps back in, but apparently the assumption of strong caching is extensively used in sage.combinat.sf. Having weakly cached unique representation everywhere except in sage.combinat.sf is at least something...
I am now running the full testsuite with the new modification.
comment:89 Changed 8 years ago by
- Status changed from needs_work to needs_review
- Work issues Keep the fix from #12313. Coercion in symmetric function algebras deleted
Now the second patch has been updated as well. As I have announced, the second patch is now not only introducing a strong custom cache for SymmetricFunctions
, but also for the different representations, i.e., for SymmetricFunctionAlgebra_generic
. It is certainly not ideal that symmetric functions need to be strongly cached, but I think that there may be a different ticket to resolve this special case.
Anyway, with sage-5.0.beta13 plus #715, #11521, #12313, #11943, #11935 and the two patches from here, all doctests pass. I am confident that they would also pass when one only has 5.0.beta13 plus the two patches from here.
Needs review, then!
comment:90 follow-up: ↓ 91 Changed 8 years ago by
- Dependencies changed from #11115 #11900 #12645 #11599 to #11115 #11900 #12645 #11599 #12215
- Status changed from needs_review to needs_work
- Work issues set to rebase wrt #12808
There is a conflict with #12808, which has a positive review. Hence, we need to rebase it!
comment:91 in reply to: ↑ 90 Changed 8 years ago by
comment:92 Changed 8 years ago by
- Dependencies changed from #11115 #11900 #12645 #11599 #12215 to #11115 #11900 #12645 #11599 #12808
- Work issues rebase wrt #12808 deleted
Thank you, Nicolas! I planned to do the change today.
comment:93 Changed 8 years ago by
- Status changed from needs_work to needs_review
... and I guess it needs review, again?
comment:94 Changed 8 years ago by
- Dependencies changed from #11115 #11900 #12645 #11599 #12808 to #11115 #11900 #12645 #11599 #12808 #7980
- Status changed from needs_review to needs_work
- Work issues set to Rebase wrt #7980
Anne mentioned on combinat-devel that the patch needs to be rebased because of #7980.
comment:95 Changed 8 years ago by
- Description modified (diff)
- Status changed from needs_work to needs_review
The patches are now rebased rel #7980. I have not been able to replace my first patch, because trac believes that it isn't my patch but Nicolas'...
Apply trac12215_weak_cached_function-sk.patch trac12215_segfault_fixes.patch
See my comment at #5970: It seems that having a weak version of cached_function (which is used to decorate
UniqueRepresentation.__classcall__
is the missing bit (in addition to #11521 and #715 and a two-line change in the polynomial ring constructor) for fixing the issues at #5970.I think this should be done on top of #11115, which rewrites cached methods and already has a positive review.