Configure Python with pydebug when SAGE_DEBUG is set

[jpflori]

Instructions for getting a debug version of Python:

• Download and untar sage-5.7.beta2
• Copy the Python spkg from #13867 to spkg/standard and (to be on the safe side) remove the previously present Python spkg
• Run

  export SAGE_DEBUG=yes
  make
  

  and wait.
• (optionally) install the gdb spkg from #13866
• Apply the patches from #13878, #14059 and #14067 to the Sage library
• Run sage -br

This gives you a Sage version that passes all tests, even with MALLOC_CHECK_=3.

[vbraun]

Looks good to me! I'm adding my proposed change to the documentation as a prerequisite to not deviate from whats documented...

[SimonKing]

Sage claims that the spkg built successfully. However, I see things like

/home/simon/SAGE/debug/sage-5.5.rc0/spkg/build/python-2.7.3.p4/src/Modules/_curses_panel.c:17:19: fatal error: panel.h: Datei oder Verzeichnis nicht gefunden
compilation terminated.

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             bsddb185           dbm             
dl                 gdbm               imageop         
sunaudiodev                                           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

in the install log. Is that normal?

[jpflori]

Yes it is (or at least I always had this behavior). I think these "optional" modules are only built if the needed prerequisites are present, but are not really required to get a working Python.

[jdemeyer]

Replying to jpflori:

I think these "optional" modules are only built if the needed prerequisites are present, but are not really required to get a working Python.

Exactly. Some of these optional modules (like "crypt" for example) are required for Sage, that's why we try to import them in Python's spkg-install.

[SimonKing]

One observation, potentially related with what we discussed on #715:

Starting python and importing sage works:

> ./sage -python
Python 2.7.3 (default, Dec 26 2012, 19:09:45) 
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sage
[39290 refs]

However, importing sage.categories.map.Map does not work:

>>> from sage.categories.map import Map
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name Map
[102454 refs]

I don't know whether this *should* work, but anyway: It doesn't.

[jpflori]

I'd say (although I don't know why) it is because it is a Cython module. Did you try with a plain Python build?

[vbraun]

I made a gdb spkg at #13866 so you can get a working debugger with the Python debug build.

[vbraun]

Some debugging hint: Use python -v to print imports as they happen:

[vbraun@volker-desktop sage-5.6.beta0]$ ./sage -python -v
...
>>> import sage.all_cmdline
...
import sage.rings.padics.padic_ext_element # dynamically loaded from /home/vbraun/opt/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ext_element.so
Fatal Python error: UNREF invalid object
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(print_backtrace+0x31)[0x7f0c88f3120d]
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(sigdie+0x14)[0x7f0c88f3123f]
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(sage_signal_handler+0x1da)[0x7f0c88f30deb]

[jpflori]

Replying to vbraun:

Some debugging hint: Use python -v to print imports as they happen:

[vbraun@volker-desktop sage-5.6.beta0]$ ./sage -python -v
...
>>> import sage.all_cmdline
...
import sage.rings.padics.padic_ext_element # dynamically loaded from /home/vbraun/opt/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ext_element.so
Fatal Python error: UNREF invalid object
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(print_backtrace+0x31)[0x7f0c88f3120d]
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(sigdie+0x14)[0x7f0c88f3123f]
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(sage_signal_handler+0x1da)[0x7f0c88f30deb]

Could you add this to the dev doc in #13865, or here?

[SimonKing]

Thank you! I was just about to ask how one can see what stuff is being imported. So, -v it is...

[vbraun]

Well ideally we'd ship Sage in a state where it at least starts up with a debug build. If I find something thats particularly useful in debugging then I'll document it. For now:

(sage-sh) vbraun@volker-desktop:sage-5.6.beta0$ gdb python
...
(gdb) run -v
...
>>> import sage.all_cmdline
...
Fatal Python error: UNREF invalid object

Program received signal SIGABRT, Aborted.
0x0000003da9835ba5 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003da9835ba5 in raise () from /lib64/libc.so.6
#1  0x0000003da9837358 in abort () from /lib64/libc.so.6
#2  0x00007ffff7d1a63b in Py_FatalError (msg=0x7ffff7d63eb1 "UNREF invalid object")
    at Python/pythonrun.c:1677
#3  0x00007ffff7c49f18 in _Py_ForgetReference (op=0x21194c0) at Objects/object.c:2223
#4  0x00007ffff7c49f6b in _Py_Dealloc (op=0x21194c0) at Objects/object.c:2242
#5  0x00007fffd9de6e7c in initpadic_ZZ_pX_element () at sage/rings/padics/padic_ZZ_pX_element.cpp:9634
#6  0x00007ffff7d0a496 in _PyImport_LoadDynamicModule (
    name=0x7ffffffe4360 "sage.rings.padics.padic_ZZ_pX_element", 
    pathname=0x7ffffffe3260 "/home/vbraun/opt/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ZZ_pX_element.so", fp=0x20d23b0) at ./Python/importdl.c:53
...
(gdb) frame 5
#5  0x00007fffd9de6e7c in initpadic_ZZ_pX_element () at sage/rings/padics/padic_ZZ_pX_element.cpp:9634
9634	  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
(gdb) l
9629	  __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
9630	  __pyx_t_2 = __Pyx_GetName(__pyx_m, __pyx_n_s__one); if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 34; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
9631	  __Pyx_GOTREF(__pyx_t_2);
9632	  __pyx_t_3 = PyNumber_Add(__pyx_t_1, __pyx_t_2); if (unlikely(!__pyx_t_3)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 34; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
9633	  __Pyx_GOTREF(__pyx_t_3);
9634	  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
9635	  __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
9636	  if (PyObject_SetAttr(__pyx_m, __pyx_n_s__big, __pyx_t_3) < 0) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 34; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
9637	  __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;

[jpflori]

See my long lonely rant at sage-devel ​https://groups.google.com/d/topic/sage-devel/Wt7uxbDkh_A/discussion, especially the links:

• post about weakref problem solved here ​https://groups.google.com/d/topic/cython-users/4es75DeacRA/discussion
• and your ticket for updating cython here #13832

Hopefully that will be enough to get Sage to start (and finally get a working #715?).

[SimonKing]

Interestingly, the error on my laptop looks different from what Volker reported in comment:15:

(sage-sh) simon@linux-sqwp:sage-5.5.rc0$ gdb python
...
(gdb) run -v
...
>>> import sage.all_cmdline
...
import sage.libs.pari.gen # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/gen.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/integer.so", 2);
python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6

Hence: No Fatal Python error: UNREF invalid object. Further:

(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff6d8e8fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6d8e9a2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7b12a19 in visit_decref (op=0x1462f40, data=0x0) at Modules/gcmodule.c:326
#5  0x00007fffeecdcfe1 in __pyx_tp_traverse_4sage_9structure_15category_object_CategoryObject (o=0x14b7d30, v=0x7ffff7b1295e <visit_decref>, a=0x0) at sage/structure/category_object.c:8971
#6  0x00007fffeef4724f in __pyx_tp_traverse_4sage_9structure_6parent_Parent (o=0x14b7d30, v=0x7ffff7b1295e <visit_decref>, a=0x0) at sage/structure/parent.c:21293
#7  0x00007fffef18a97c in __pyx_tp_traverse_4sage_9structure_10parent_old_Parent (o=0x14b7d30, v=0x7ffff7b1295e <visit_decref>, a=0x0) at sage/structure/parent_old.c:7253
#8  0x00007fffef39ab57 in __pyx_tp_traverse_4sage_9structure_11parent_base_ParentWithBase (o=0x14b7d30, v=0x7ffff7b1295e <visit_decref>, a=0x0) at sage/structure/parent_base.c:1892
#9  0x00007ffff7a4c572 in subtype_traverse (self=0x14b7d30, visit=0x7ffff7b1295e <visit_decref>, arg=0x0) at Objects/typeobject.c:838
...
#32 0x00007fffecc2bcbe in __Pyx_ImportModule (name=0x7fffecc2fc6c "sage.rings.integer") at sage/rings/complex_double.c:18370
#33 0x00007fffecc2b951 in __Pyx_ImportType (module_name=0x7fffecc2fc6c "sage.rings.integer", class_name=0x7fffecc2fc64 "Integer", size=64, strict=1) at sage/rings/complex_double.c:18319
#34 0x00007fffecc25b46 in initcomplex_double () at sage/rings/complex_double.c:16572
#35 0x00007ffff7ae9282 in _PyImport_LoadDynamicModule (name=0x7ffffffed660 "sage.rings.complex_double", 
...

So, "Volker's" error is in padic_ZZ_pX_element, but mine is in sage.rings.integer. Moreover, my gdb backtrace is a lot shorter, just 109 lines.

By the way:

>>> from sage.rings.integer_ring import IntegerRing

fails with an import error -- but there is no crash! So, there seems to be something else wrong, not just the refcount.

[jpflori]

What you get is similar to what I get (without gdb). Although the failing place is different as I call ./sage -c "1+1" which must import something else (I'd say sage/all.py) as all_cmdline.py.

Not sure why Volker's output differs.

Anyway, I think we should now focus on trying all of this with #13832. (Don't forget to rebuild the Sage library after upgrading Cython.)

[SimonKing]

Replying to jpflori:

See my long lonely rant at sage-devel ​https://groups.google.com/d/topic/sage-devel/Wt7uxbDkh_A/discussion, especially the links:

• post about weakref problem solved here ​https://groups.google.com/d/topic/cython-users/4es75DeacRA/discussion
• and your ticket for updating cython here #13832

Hopefully that will be enough to get Sage to start (and finally get a working #715?).

I don't find these links in the discussion at sage-devel, but indeed the announcement on cython-users looks promising! I am now trying to install the cython spkg from #13832. I guess after installing it I have to do sage -ba, right?

[jpflori]

Replying to SimonKing:

Replying to jpflori:

See my long lonely rant at sage-devel ​https://groups.google.com/d/topic/sage-devel/Wt7uxbDkh_A/discussion, especially the links:

• post about weakref problem solved here ​https://groups.google.com/d/topic/cython-users/4es75DeacRA/discussion
• and your ticket for updating cython here #13832

Hopefully that will be enough to get Sage to start (and finally get a working #715?).

I don't find these links in the discussion at sage-devel, but indeed the announcement on cython-users looks promising! I am now trying to install the cython spkg from #13832. I guess after installing it I have to do sage -ba, right?

Not really sure :) I touched everything in sage/structure hoping it will be enough.

If the bug persists, I'll report my minimal example on cython-users.

[jpflori]

Ok now my first errors with the pexpect interfaces are gone! Great!

But I now get an error similar to Volker's one. I hope that it is just some files in sage/rings/which were not rebuilt.

[SimonKing]

I'll need a bit longer to test, because I did sage -ba. I thought that, if Cython gets updated, then one certainly needs to rebuild all pyx files.

[jpflori]

You're right, I just did not remember about the -a flag.

[vbraun]

Replying to SimonKing:

Interestingly, the error on my laptop looks different from what Volker reported in comment:15: ... dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/integer.so", 2); python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. ...

You die in a garbage collection cycle, presumably because you have < 32 GB RAM (= my desktop). So the garbage collector runs earlier.

We both get killed by integer arithmetic, which points to a circular import problem. The Integer class is a particularly bad case of stuff that tries to define and compute everything upon import. At the same time it is interleaved with the rationals class, so it is full of circular imports. Which almost certainly equals half-constructed objects during the circular import fest.

[jpflori]

Ok now I also get:

./sage -sh
... gdb python
... r -v
...
import sage.rings.padics.padic_ext_element # dynamically loaded from /home/jp/boulot/sage/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ext_element.so
Fatal Python error: UNREF invalid object

Program received signal SIGABRT, Aborted.
0x00007ffff6cdfdd5 in raise () from /lib/x86_64-linux-gnu/libc.so.6

with bt

(gdb) bt                                                                      
#0  0x00007ffff6cdfdd5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff6ce2efb in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff7afa73c in Py_FatalError (
    msg=0x7ffff7b441d1 "UNREF invalid object") at Python/pythonrun.c:1677
#3  0x00007ffff7a2c31e in _Py_ForgetReference (op=0x1e18440)
    at Objects/object.c:2223
#4  0x00007ffff7a2c371 in _Py_Dealloc (op=0x1e18440) at Objects/object.c:2242
#5  0x00007fffde01b3ac in initpadic_ZZ_pX_element ()
    at sage/rings/padics/padic_ZZ_pX_element.cpp:9650
#6  0x00007ffff7aea57e in _PyImport_LoadDynamicModule (
    name=0x7ffffffe4910 "sage.rings.padics.padic_ZZ_pX_element", 
    pathname=0x7ffffffe3810 "/home/jp/boulot/sage/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ZZ_pX_element.so", fp=0x1e13800)
    at ./Python/importdl.c:53
#7  0x00007ffff7ae64a5 in load_module (
    name=0x7ffffffe4910 "sage.rings.padics.padic_ZZ_pX_element", 
    fp=0x1e13800, 
    pathname=0x7ffffffe3810 "/home/jp/boulot/sage/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ZZ_pX_element.so", type=3, 
    loader=0x0) at Python/import.c:1831
#8  0x00007ffff7ae8776 in import_submodule (mod=0x1acc600, 
    subname=0x7ffffffe4922 "padic_ZZ_pX_element", 
    fullname=0x7ffffffe4910 "sage.rings.padics.padic_ZZ_pX_element")
    at Python/import.c:2595
#9  0x00007ffff7ae7c9a in load_next (mod=0x1acc600, altmod=0x1acc600, 
    p_name=0x7ffffffe48f8, 
    buf=0x7ffffffe4910 "sage.rings.padics.padic_ZZ_pX_element", 
    p_buflen=0x7ffffffe4908) at Python/import.c:2415

and seems to be caused by

big = two**128 + one
#this should not fit in a long, since it's supposed to be bigger than any valid absolute precision.

[jpflori]

Replying to vbraun:

Replying to SimonKing:

Interestingly, the error on my laptop looks different from what Volker reported in comment:15: ... dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/integer.so", 2); python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. ...

You die in a garbage collection cycle, presumably because you have < 32 GB RAM (= my desktop). So the garbage collector runs earlier.

Are you sure? I'd say this really looks like the weakref mess fixed up in 0.17.3.

We both get killed by integer arithmetic, which points to a circular import problem. The Integer class is a particularly bad case of stuff that tries to define and compute everything upon import. At the same time it is interleaved with the rationals class, so it is full of circular imports. Which almost certainly equals half-constructed objects during the circular import fest.

[jpflori]

According to ​http://realmike.org/blog/2012/07/08/embedding-python-tutorial-part-1/ we might need to compile files by passing -DPy_DEBUG. Does this really changes anything in python headers? grep indeed finds occurences of it in $SAGE_LOCAL/include/python2.7

[SimonKing]

Replying to jpflori:

Replying to vbraun:

Replying to SimonKing:

Interestingly, the error on my laptop looks different from what Volker reported in comment:15: ... dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/integer.so", 2); python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. ...

You die in a garbage collection cycle, presumably because you have < 32 GB RAM (= my desktop). So the garbage collector runs earlier.

Are you sure? I'd say this really looks like the weakref mess fixed up in 0.17.3.

After upgrading to Cython 0.17.3, I get an error that seems to confirm what Volker stated.

So, now we have to try and find out where the padics problem comes from.

Note, however, that in pure python I still get a "harmless" import error when I want to do

Python 2.7.3 (default, Dec 26 2012, 19:09:45) 
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.rings.integer_ring import IntegerRing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "integer.pxd", line 9, in init sage.rings.integer_ring (sage/rings/integer_ring.c:12178)
  File "integer.pyx", line 170, in init sage.rings.integer (sage/rings/integer.c:38390)
  File "/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/infinity.py", line 200, in <module>
    import sage.rings.rational
  File "fast_arith.pxd", line 5, in init sage.rings.rational (sage/rings/rational.c:27828)
  File "fast_arith.pyx", line 51, in init sage.rings.fast_arith (sage/rings/fast_arith.c:8060)
ImportError: cannot import name ZZ
[133346 refs]
>>> quit()

Hence, no crash, but something is wrong with the ring of integers,

[SimonKing]

Question about the way to proceed:

This ticket provides the debug version of the python spkg, and it has a positive review. However, in the debug version, Sage does not even start.

So, shall we open a new ticket "Make Sage start when Python is build under SAGE_DEBUG", or shall we try to resolve the starting problem here?

I suggest to keep it here, thus changing the status into "needs work".

[SimonKing]

For the record: These imports occur when one tries to import the integer ring, as reported by python -v:

>>> import sage
import sage # directory /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/__init__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/__init__.py
import sage # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/__init__.pyc
[39292 refs]
>>> from sage.rings.integer_ring import IntegerRing
import sage.rings # directory /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/__init__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/__init__.py
import sage.rings # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/__init__.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/integer_ring.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/ring.so", 2);
import sage.structure # directory /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/__init__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/__init__.py
import sage.structure # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/__init__.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/dynamic_class.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/dynamic_class.py
import sage.structure.dynamic_class # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/dynamic_class.pyc
import sage.misc # directory /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/__init__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/__init__.py
import sage.misc # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/__init__.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/cachefunc.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/function_mangling.so", 2);
import sage.misc.function_mangling # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/function_mangling.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sageinspect.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sageinspect.py
import sage.misc.sageinspect # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sageinspect.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/ast.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/ast.py
import ast # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/ast.pyc
import _ast # builtin
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/inspect.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/inspect.py
import inspect # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/inspect.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/string.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/string.py
import string # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/string.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/strop.so", 2);
import strop # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/strop.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/dis.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/dis.py
import dis # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/dis.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/opcode.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/opcode.py
import opcode # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/opcode.pyc
import imp # builtin
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/tokenize.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/tokenize.py
import tokenize # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/tokenize.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/token.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/token.py
import token # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/token.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/operator.so", 2);
import operator # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/operator.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/collections.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/collections.py
import collections # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/collections.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_collections.so", 2);
import _collections # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_collections.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/keyword.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/keyword.py
import keyword # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/keyword.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/heapq.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/heapq.py
import heapq # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/heapq.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/itertools.so", 2);
import itertools # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/itertools.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/bisect.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/bisect.py
import bisect # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/bisect.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_bisect.so", 2);
import _bisect # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_bisect.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_heapq.so", 2);
import _heapq # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_heapq.so
import thread # builtin
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/functools.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/functools.py
import functools # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/functools.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_functools.so", 2);
import _functools # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_functools.so
import sage.misc.cachefunc # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/cachefunc.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/unique_representation.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/unique_representation.py
import sage.structure.unique_representation # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/unique_representation.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/classcall_metaclass.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/nested_class.so", 2);
import sage.misc.nested_class # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/nested_class.so
import sage.misc.classcall_metaclass # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/classcall_metaclass.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_attribute.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_attribute.py
import sage.misc.lazy_attribute # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_attribute.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/abstract_method.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/abstract_method.py
import sage.misc.abstract_method # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/abstract_method.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/parent_gens.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/parent_base.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/parent_old.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/parent.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/category_object.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/sage_object.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/cPickle.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/cStringIO.so", 2);
import cStringIO # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/cStringIO.so
import cPickle # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/cPickle.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_unittest.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_unittest.py
import sage.misc.sage_unittest # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_unittest.pyc
import unittest # directory /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/__init__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/__init__.py
import unittest # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/__init__.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/result.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/result.py
import unittest.result # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/result.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/StringIO.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/StringIO.py
import StringIO # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/StringIO.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/util.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/util.py
import unittest.util # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/util.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/case.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/case.py
import unittest.case # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/case.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/difflib.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/difflib.py
import difflib # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/difflib.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/pprint.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/pprint.py
import pprint # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/pprint.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/suite.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/suite.py
import unittest.suite # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/suite.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/loader.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/loader.py
import unittest.loader # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/loader.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/fnmatch.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/fnmatch.py
import fnmatch # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/fnmatch.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/main.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/main.py
import unittest.main # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/main.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/runner.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/runner.py
import unittest.runner # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/runner.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/time.so", 2);
import time # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/time.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/signals.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/signals.py
import unittest.signals # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/unittest/signals.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/weakref.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/weakref.py
import weakref # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/weakref.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/zlib.so", 2);
import zlib # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/zlib.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/bz2.so", 2);
import bz2 # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/bz2.so
import sage.structure.sage_object # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/sage_object.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/generators.so", 2);
import sage.structure.generators # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/generators.so
import sage.categories # directory /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/__init__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/__init__.py
import sage.categories # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/__init__.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/primer.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/primer.py
import sage.categories.primer # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/primer.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/category.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/category.py
import sage.categories.category # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/category.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/c3.so", 2);
import sage.misc.c3 # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/c3.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/unknown.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/unknown.py
import sage.misc.unknown # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/unknown.pyc
import sage.structure.category_object # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/category_object.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce_dict.so", 2);
import sage.structure.coerce_dict # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce_dict.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/element.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/misc.so", 2);
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_itertools.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_itertools.py
import sage.misc.sage_itertools # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_itertools.pyc
import sage.structure.misc # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/misc.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_format.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_format.py
import sage.misc.lazy_format # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_format.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/copy.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/copy.py
import copy # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/copy.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/map.so", 2);
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/homset.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/homset.py
import sage.categories.homset # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/homset.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/morphism.so", 2);
import sage.categories.morphism # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/morphism.so
import sage.categories.map # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/map.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/functor.so", 2);
import sage.categories.functor # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/functor.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/action.so", 2);
import sage.categories.action # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/action.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce_exceptions.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce_exceptions.py
import sage.structure.coerce_exceptions # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce_exceptions.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce_actions.so", 2);
import sage.structure.coerce_actions # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce_actions.so
import sage.structure.coerce # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/coerce.so
import sage.structure.element # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/element.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/sets_cat.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/sets_cat.py
import sage.categories.sets_cat # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/sets_cat.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_import.so", 2);
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/shutil.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/shutil.py
import shutil # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/shutil.pyc
import pwd # builtin
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/grp.so", 2);
import grp # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/grp.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/tempfile.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/tempfile.py
import tempfile # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/tempfile.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/random.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/random.py
import random # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/random.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/__future__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/__future__.py
import __future__ # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/__future__.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/math.so", 2);
import math # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/math.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/binascii.so", 2);
import binascii # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/binascii.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/hashlib.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/hashlib.py
import hashlib # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/hashlib.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_hashlib.so", 2);
import _hashlib # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_hashlib.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_random.so", 2);
import _random # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_random.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/fcntl.so", 2);
import fcntl # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/fcntl.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_import_cache.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_import_cache.py
import sage.misc.lazy_import_cache # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_import_cache.pyc
import sage.misc.lazy_import # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/lazy_import.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/category_singleton.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/constant_function.so", 2);
import sage.misc.constant_function # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/constant_function.so
import sage.categories.category_singleton # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/category_singleton.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/sets_with_partial_maps.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/sets_with_partial_maps.py
import sage.categories.sets_with_partial_maps # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/sets_with_partial_maps.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/objects.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/objects.py
import sage.categories.objects # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/objects.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/subquotients.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/subquotients.py
import sage.categories.subquotients # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/subquotients.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/covariant_functorial_construction.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/covariant_functorial_construction.py
import sage.categories.covariant_functorial_construction # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/covariant_functorial_construction.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/quotients.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/quotients.py
import sage.categories.quotients # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/quotients.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/subobjects.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/subobjects.py
import sage.categories.subobjects # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/subobjects.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/isomorphic_objects.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/isomorphic_objects.py
import sage.categories.isomorphic_objects # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/isomorphic_objects.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/algebra_functor.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/algebra_functor.py
import sage.categories.algebra_functor # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/algebra_functor.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/category_types.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/category_types.py
import sage.categories.category_types # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/category_types.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/latex.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/latex.py
import sage.misc.latex # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/latex.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/subprocess.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/subprocess.py
import subprocess # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/subprocess.pyc
import gc # builtin
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/select.so", 2);
import select # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/select.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/pickle.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/pickle.py
import pickle # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/pickle.pyc
import marshal # builtin
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/struct.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/struct.py
import struct # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/struct.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_struct.so", 2);
import _struct # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/lib-dynload/_struct.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/temporary_file.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/temporary_file.py
import sage.misc.temporary_file # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/temporary_file.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_eval.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_eval.py
import sage.misc.sage_eval # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_eval.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/preparser.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/preparser.py
import sage.misc.preparser # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/preparser.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/base64.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/base64.py
import base64 # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python/base64.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_ostools.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_ostools.py
import sage.misc.sage_ostools # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/sage_ostools.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/cartesian_product.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/cartesian_product.py
import sage.categories.cartesian_product # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/cartesian_product.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/realizations.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/realizations.py
import sage.categories.realizations # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/realizations.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/bindable_class.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/bindable_class.py
import sage.misc.bindable_class # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/bindable_class.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/with_realizations.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/with_realizations.py
import sage.categories.with_realizations # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/with_realizations.pyc
import sage.structure.parent # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/parent.so
import sage.structure.parent_old # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/parent_old.so
import sage.structure.parent_base # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/parent_base.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/defaults.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/defaults.py
import sage.misc.defaults # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/defaults.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/gens_py.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/gens_py.py
import sage.structure.gens_py # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/gens_py.pyc
import sage.structure.parent_gens # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/structure/parent_gens.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/prandom.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/prandom.py
import sage.misc.prandom # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/prandom.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/randstate.so", 2);
import sage.misc.randstate # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/randstate.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/rings.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/rings.py
import sage.categories.rings # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/rings.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/rngs.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/rngs.py
import sage.categories.rngs # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/rngs.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_groups.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_groups.py
import sage.categories.commutative_additive_groups # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_groups.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_monoids.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_monoids.py
import sage.categories.commutative_additive_monoids # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_monoids.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_semigroups.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_semigroups.py
import sage.categories.commutative_additive_semigroups # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_additive_semigroups.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/additive_magmas.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/additive_magmas.py
import sage.categories.additive_magmas # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/additive_magmas.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/semigroups.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/semigroups.py
import sage.categories.semigroups # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/semigroups.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/misc_c.so", 2);
import sage.misc.misc_c # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/misc_c.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/magmas.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/magmas.py
import sage.categories.magmas # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/magmas.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/semirings.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/semirings.py
import sage.categories.semirings # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/semirings.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/monoids.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/monoids.py
import sage.categories.monoids # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/monoids.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_rings.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_rings.py
import sage.categories.commutative_rings # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_rings.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/integral_domains.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/integral_domains.py
import sage.categories.integral_domains # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/integral_domains.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/domains.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/domains.py
import sage.categories.domains # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/domains.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/principal_ideal_domains.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/principal_ideal_domains.py
import sage.categories.principal_ideal_domains # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/principal_ideal_domains.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/unique_factorization_domains.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/unique_factorization_domains.py
import sage.categories.unique_factorization_domains # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/unique_factorization_domains.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/gcd_domains.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/gcd_domains.py
import sage.categories.gcd_domains # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/gcd_domains.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/euclidean_domains.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/euclidean_domains.py
import sage.categories.euclidean_domains # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/euclidean_domains.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/algebras.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/algebras.py
import sage.categories.algebras # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/algebras.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/dual.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/dual.py
import sage.categories.dual # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/dual.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/tensor.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/tensor.py
import sage.categories.tensor # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/tensor.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/modules.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/modules.py
import sage.categories.modules # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/modules.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/bimodules.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/bimodules.py
import sage.categories.bimodules # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/bimodules.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/left_modules.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/left_modules.py
import sage.categories.left_modules # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/left_modules.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/right_modules.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/right_modules.py
import sage.categories.right_modules # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/right_modules.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/fields.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/fields.py
import sage.categories.fields # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/fields.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/division_rings.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/division_rings.py
import sage.categories.division_rings # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/division_rings.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/field.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/field.py
import sage.rings.field # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/field.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/vector_spaces.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/vector_spaces.py
import sage.categories.vector_spaces # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/vector_spaces.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_algebras.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_algebras.py
import sage.categories.commutative_algebras # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/categories/commutative_algebras.pyc
import sage.rings.ring # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/ring.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/integer.so", 2);
import sage.libs # directory /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/__init__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/__init__.py
import sage.libs # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/__init__.pyc
import sage.libs.pari # directory /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/__init__.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/__init__.py
import sage.libs.pari # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/__init__.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/all.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/all.py
import sage.libs.pari.all # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/all.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/gen.so", 2);
import sage.libs.pari.gen # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/gen.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/allocator.so", 2);
import sage.misc.allocator # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/misc/allocator.so
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/infinity.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/infinity.py
import sage.rings.infinity # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/infinity.pyc
# /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/ring_element.pyc matches /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/ring_element.py
import sage.rings.ring_element # precompiled from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/ring_element.pyc
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/rational.so", 2);
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/fast_arith.so", 2);
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "integer.pxd", line 9, in init sage.rings.integer_ring (sage/rings/integer_ring.c:12178)
  File "integer.pyx", line 170, in init sage.rings.integer (sage/rings/integer.c:38390)
  File "/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/infinity.py", line 200, in <module>
    import sage.rings.rational
  File "fast_arith.pxd", line 5, in init sage.rings.rational (sage/rings/rational.c:27828)
  File "fast_arith.pyx", line 51, in init sage.rings.fast_arith (sage/rings/fast_arith.c:8060)
ImportError: cannot import name ZZ

Scary. All "usual suspects" are in place: Weak references, coercion, categories.

But what goes wrong? Is it a circular import? If so, what can be done to improve it?

[jpflori]

Replying to SimonKing:

Replying to jpflori:

Replying to vbraun:

Replying to SimonKing:

Interestingly, the error on my laptop looks different from what Volker reported in comment:15: ... dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/integer.so", 2); python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. ...

You die in a garbage collection cycle, presumably because you have < 32 GB RAM (= my desktop). So the garbage collector runs earlier.

Are you sure? I'd say this really looks like the weakref mess fixed up in 0.17.3.

After upgrading to Cython 0.17.3, I get an error that seems to confirm what Volker stated.

Do you get the same one as before? i.e. with a failing assert?

Or the one that Volker got (and I now get)? i.e. with a Fatal UNREF?

Just asking ot be really sure we (or I :) ) did not get confused.

And I just tested that adding -DPy_DEBUG does not help with the FATAL thing.

[jpflori]

Replying to SimonKing:

Question about the way to proceed:

This ticket provides the debug version of the python spkg, and it has a positive review. However, in the debug version, Sage does not even start.

So, shall we open a new ticket "Make Sage start when Python is build under SAGE_DEBUG", or shall we try to resolve the starting problem here?

I suggest to keep it here, thus changing the status into "needs work".

If we have to fix circular imports before merging this, it will just never get merged. So although I don't think either it is the right solution, I propose to merge this to let people who want play with it.

[jpflori]

Replying to SimonKing:

For the record: These imports occur when one tries to import the integer ring, as reported by python -v: {{{line 200, in <module>

import sage.rings.rational

File "fast_arith.pxd", line 5, in init sage.rings.rational (sage/rings/rational.c:27828) File "fast_arith.pyx", line 51, in init sage.rings.fast_arith (sage/rings/fast_arith.c:8060)

ImportError?: cannot import name ZZ }}}

Scary. All "usual suspects" are in place: Weak references, coercion, categories.

But what goes wrong? Is it a circular import? If so, what can be done to improve it?

I think it is indeed a circular import. I've been chasing those starting from ring_integer yesterday night, but it just gave be bad dreams. After three or four files where I had to remove various things imported from ring_integer and importing it, I got stuck in sage/libs/ntl/ntl_ZZ_p.pyx complaining about not being able to cast a ZZ_c* to a Py_Object and gave up.

[jpflori]

For info I also get the FATAL UNREF in the same place on Sage 5.2 so this is unrelated to memleaks patches.

[jpflori]

It seems that addition of Integer's is broken.

[jpflori]

As well as substraction and multiplication and powering, but not negation...

Note that powering an Integer by an int seems fine.

[vbraun]

Yes you can't do any arithmetic while the circular import is still on its way. For starters it invokes coercion so its a huge mess. The following lets me get past it

big = Integer('340282366920938463463374607431768211457')  # two**128 + one

Of course it dies in the next place where stupid computations are performed upon import.

[vbraun]

Actually now I'm getting

ImportError: /home/vbraun/opt/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/polynomial/multi_polynomial_libsingular.so: undefined symbol: _Z7_p_TestP8spolyrecP9sip_sringi

Or, unmangled:

[vbraun@volker-desktop sage-5.6.beta0]$ echo _Z7_p_TestP8spolyrecP9sip_sringi | c++filt 
_p_Test(spolyrec*, sip_sring*, int)

This seems to be a singular debug function that is not exported into the Singular libraries.

[vbraun]

Ah of course thats in a #ifdef PDEBUG block. I guess that means Polynomials DEBUG to Singular %-)

[jpflori]

Replying to vbraun:

Actually now I'm getting

ImportError: /home/vbraun/opt/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/polynomial/multi_polynomial_libsingular.so: undefined symbol: _Z7_p_TestP8spolyrecP9sip_sringi

Or, unmangled:

[vbraun@volker-desktop sage-5.6.beta0]$ echo _Z7_p_TestP8spolyrecP9sip_sringi | c++filt 
_p_Test(spolyrec*, sip_sring*, int)

I get the same indeed.

This seems to be a singular debug function that is not exported into the Singular libraries.

[jpflori]

Replying to vbraun:

Yes you can't do any arithmetic while the circular import is still on its way.

If I had statements at the end of integer_ring.pyx and integer.pyx they get executed before all this mess, so I thought that in this case it was not a circular import problem.

For starters it invokes coercion so its a huge mess.

I don't see any coercion involved, where did you spot it?

The following lets me get past it

big = Integer('340282366920938463463374607431768211457')  # two**128 + one

Of course it dies in the next place where stupid computations are performed upon import.

[jpflori]

What is strange is that I can compute one + one in integer.pyx and that succeeds and when the "same" computation is performed later in padic_ZZ_pX_element.pyx it fails when Python tries to dealloc the result...

In fact it depends on where I compute one + one in integer.pyx, I'm getting lost now.

[vbraun]

We (=distutils) usually compile Cython modules with -DNDEBUG, but apparently not in a debug build. This is what switches on Singular's debugging stuff.

Not sure EXACTLY why it blows up doing integer arithmetic, but according to Simon's import list it is also during import of the coercion system. Which will need to access sample elements of rings etc.

[jpflori]

My problem is that I'm not sure the problem Simon pointed out (and I can reproduce) and which definitely involves circular import is related to the one you reported (and i can reproduce as well, nut obviously issuing a different "import" command).

[jpflori]

Ok, I think I got the problem you reported (the one segfault doing basic arith with Integer's).

If I'm correct, the problem is that we hook up Integer creation and deletion and Python debug gets confused. Have a look at the end of integer.pyx. If you try to compute "one + one" before the call to hook_fast_tp_functions() it succeeds, but after that it will fail during deallocation.

[jpflori]

In particular the reference to the previous and next object in the doubly linked list of allocated object must be complete garbage, what makes the debug Python raise the FATAL invalid error.

[jpflori]

The question now is "can we -- or even should -- we fix this?".

The simplest way would be to detect at compile time that we're using a debug build and fill garbage, but satisfying for the debug checks, things in ob_prev and ob_next, let's say that they both point to the object itself?

[jpflori]

Or we could not call the hook install function, but I guess that would be cheating... although I'm going to try right now to see if further problems can be discovered.

[jpflori]

And got stuck as well by the Singular problem.

[vbraun]

For the Singular problem, replace #ifdef NDEBUG in mod2.h with #if 0 and rebuild (sage -ba)

[jpflori]

Oh, I tried to rebuild singular with SAGE_DEBUG=yes (which actually does something smart, contrary to what the doc currently says :) and just got a segfault when starting Sage.

[jpflori]

It fails in pynac (while trying to divide things).

In fact I cannot even rebuild pynac as it complains that it does not find distutils.

[jpflori]

Which had to be expected because the autoconf test will indeed not work with a debug build, it checks that the error (and standard) output are empty.

[jpflori]

After rebuilding pynac, it now segfaults in groupoid.py with an UNREF error.

[jpflori]

Nevermind, it seems to fail while initializing "I".

[jpflori]

It fails while doing some coercion stuff.

[vbraun]

Did you fix the pynac spkg? Ticket number?

[jdemeyer]

Could somebody please clarify: when I use this spkg as it currently is and build Sage with SAGE_DEBUG=yes, does it make it impossible to build or use Sage?

If the answer to the above question is "yes", I don't think we should merge this ticket. Building with SAGE_DEBUG=yes should result in a working Sage.

[jpflori]

Replying to vbraun:

Did you fix the pynac spkg? Ticket number?

Just a hack, I removed the test for distutils in configure. The problem is that the bad test is in the macro AX_PYTHON_DEVEL which also defines needed stuff.

Not sure if we can replace this macro by other macro defining what's needed.

Another solution is to report that "upstream" whereever upstrea autoconf-archive is.

[jpflori]

Replying to jdemeyer:

Could somebody please clarify: when I use this spkg as it currently is and build Sage with SAGE_DEBUG=yes, does it make it impossible to build or use Sage?

If the answer to the above question is "yes", I don't think we should merge this ticket. Building with SAGE_DEBUG=yes should result in a working Sage.

It does not currently starts (and wont even build because of pynac), so do not merge the ticket.

[jdemeyer]

OK, the tickets fixing building and/or running Sage with this spkg should be added as dependency then.

[jpflori]

Just found other hooked tp_* functions in real_double. Strange I did not stop them earlier.

And... now Sage starts!

[jpflori]

Ticket for Pynac at #13867, for fast_tp_* at #13868.

We should try to build from scratch with SAGE_DEBUG=yes to see what spkg fail.

Maybe we should check again for circular imports although I am currently able to issue "./sage" and to get to the prompt without error.

[jpflori]

Other good news: I don't get any problems while testing sage/modules/module.pyx now.

[jpflori]

Replying to jpflori:

Other good news: I don't get any problems while testing sage/modules/module.pyx now.

I think I've spoken too fast. I've rebuild a fresh 5.6.beta0 with cython 0.17.3 and still get the segfaults there.

[jpflori]

Replying to jpflori:

Replying to jpflori:

Other good news: I don't get any problems while testing sage/modules/module.pyx now.

I think I've spoken too fast. I've rebuild a fresh 5.6.beta0 with cython 0.17.3 and still get the segfaults there.

And now I've rebuilt on top of that install with debug enabled, the bug seems to be gone... So maybe the problem is with the fast tp_* functions (which I've disabled for now because the suggested fix in the comments lead to segfaults)?

[vbraun]

In debug mode the pymalloc allocator is still used by default (but extra debugging in the pymalloc allocator is active). We would have to disable pymalloc (--without-pymalloc) to use the system malloc/free. This means that it is still not useful for valgrind. Just wanted to throw this out so that everyone is aware of it. I don't mind the current state. The Python docs seem to suggest that this (i.e. --with-pymalloc, the default) is the more useful debug setting.

[vbraun]

I'm making this into a meta-ticket to collect information to build a Python debug version.

[SimonKing]

For the record (since this is a tracker ticket): The current problem when starting Sage in debug mode seems to be that tp_compare didn't return -1 or -2 for exception - tracked at #13868.

[vbraun]

It does work for me, for the record.

[SimonKing]

When I do as being advised, I get a starting sage-5.6.beta1 in debug mode.

Question: Do we need the optional gdb spkg from #13866 in order to properly use gdb with Sage's debug mode? Then we should perhaps add #13866 as a new dependency here.

[vbraun]

You don't need the optional gdb to see C stack traces. But its necessary if you want to use cygdb. I'll post a patch for that later. Its pretty cool ;-)

[SimonKing]

I am running "make testall".

So far, the tests don't look bad: I see some failures of optional tests - I don't know why they are executed? In the first run, I still had SAGE_DEBUG=yes, but repeating it without SAGE_DEBUG I still see that optional tests were executed.

Is that because of "make testall" versus "make test"?

Interestingly, it is in the optional tests that I see instances of negative refcounts.

    sage: magma('GF(5)!1 eq Rationals()!1')            # optional magma required
Exception raised:
    Traceback (most recent call last):
      File "/home/simon/SAGE/debug/sage-5.6.beta1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/simon/SAGE/debug/sage-5.6.beta1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/simon/SAGE/debug/sage-5.6.beta1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_28[2]>", line 1, in <module>
        magma('GF(5)!1 eq Rationals()!1')            # optional magma required###line 762:
    sage: magma('GF(5)!1 eq Rationals()!1')            # optional magma required
      File "/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python/site-packages/sage/interfaces/magma.py", line 752, in __call__
        A = Expect.__call__(self, x)
      File "/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python/site-packages/sage/interfaces/interface.py", line 195, in __call__
        return cls(self, x, name=name)
      File "/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python/site-packages/sage/interfaces/expect.py", line 1308, in __init__
        raise TypeError, x
    TypeError: Unable to start magma
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libcsage.so(print_backtrace+0x31)[0x7fa5e6666c87]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libcsage.so(sigdie+0x14)[0x7fa5e6666cb9]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libcsage.so(sage_signal_handler+0x1dd)[0x7fa5e666685d]
/lib64/libpthread.so.0(+0xfd00)[0x7fa5e8545d00]
/lib64/libc.so.6(gsignal+0x35)[0x7fa5e7b7cd95]
/lib64/libc.so.6(abort+0x17b)[0x7fa5e7b7e2ab]
/lib64/libc.so.6(+0x2d8fe)[0x7fa5e7b758fe]
/lib64/libc.so.6(+0x2d9a2)[0x7fa5e7b759a2]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x1a69c9)[0x7fa5e88f99c9]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0xb0bce)[0x7fa5e8803bce]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x1a6a3e)[0x7fa5e88f9a3e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x1a7a73)[0x7fa5e88faa73]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x1a7d96)[0x7fa5e88fad96]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(_PyObject_GC_Malloc+0xaa)[0x7fa5e88fba4a]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyType_GenericAlloc+0x5c)[0x7fa5e88331b7]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyType_GenericNew+0x2d)[0x7fa5e8833341]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0xed7cd)[0x7fa5e88407cd]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyCFunction_Call+0xc0)[0x7fa5e8808088]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ab75)[0x7fa5e88adb75]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x10e5)[0x7fa5e88ab0aa]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x913fb)[0x7fa5e87e43fb]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15c113)[0x7fa5e88af113]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7ea2)[0x7fa5e88a894b]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x10e5)[0x7fa5e88ab0aa]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x913fb)[0x7fa5e87e43fb]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x6e311)[0x7fa5e87c1311]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/lib-dynload/cPickle.so(+0x10b73)[0x7fa5e31abb73]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/lib-dynload/cPickle.so(+0x13a46)[0x7fa5e31aea46]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/lib-dynload/cPickle.so(+0x14bf9)[0x7fa5e31afbf9]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15a75c)[0x7fa5e88ad75c]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15b08a)[0x7fa5e88ae08a]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ad76)[0x7fa5e88add76]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15b08a)[0x7fa5e88ae08a]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ad76)[0x7fa5e88add76]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15b08a)[0x7fa5e88ae08a]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ad76)[0x7fa5e88add76]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x10e5)[0x7fa5e88ab0aa]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x913fb)[0x7fa5e87e43fb]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x6e311)[0x7fa5e87c1311]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x524a6)[0x7fa5e87a54a6]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_CallMethod+0x19c)[0x7fa5e87a58b8]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/site-packages/ZODB3-3.7.0-py2.7-linux-x86_64.egg/persistent/cPersistence.so(+0x2211)[0x7fa5b0931211]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/site-packages/ZODB3-3.7.0-py2.7-linux-x86_64.egg/persistent/cPersistence.so(+0x48c8)[0x7fa5b09338c8]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/site-packages/ZODB3-3.7.0-py2.7-linux-x86_64.egg/BTrees/_OOBTree.so(+0xb454)[0x7fa5b0b42454]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/site-packages/ZODB3-3.7.0-py2.7-linux-x86_64.egg/BTrees/_OOBTree.so(+0xf966)[0x7fa5b0b46966]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15a943)[0x7fa5e88ad943]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15b08a)[0x7fa5e88ae08a]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ad76)[0x7fa5e88add76]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15b08a)[0x7fa5e88ae08a]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ad76)[0x7fa5e88add76]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15b08a)[0x7fa5e88ae08a]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ad76)[0x7fa5e88add76]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x10e5)[0x7fa5e88ab0aa]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x913fb)[0x7fa5e87e43fb]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x6e311)[0x7fa5e87c1311]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python/site-packages/sage/structure/factory.so(+0x4081)[0x7fa5d2fe7081]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python/site-packages/sage/structure/factory.so(+0x3da7)[0x7fa5d2fe6da7]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ba58)[0x7fa5e88aea58]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ad92)[0x7fa5e88add92]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x7c11)[0x7fa5e88a86ba]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x10e5)[0x7fa5e88ab0aa]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x5a)[0x7fa5e88a0a88]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15d693)[0x7fa5e88b0693]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x437d)[0x7fa5e88a4e26]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x10e5)[0x7fa5e88ab0aa]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x913fb)[0x7fa5e87e43fb]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x6e311)[0x7fa5e87c1311]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(PyObject_Call+0x83)[0x7fa5e87a533e]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ba58)[0x7fa5e88aea58]
/home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpython2.7.so.1.0(+0x15ad92)[0x7fa5e88add92]
...
python(main+0x20)[0x4007b4]
/lib64/libc.so.6(__libc_start_main+0xed)[0x7fa5e7b6923d]
python[0x4006d9]

------------------------------------------------------------------------
Unhandled SIGABRT: An abort() occurred in Sage.
This probably occurred because a *compiled* component of Sage has a bug
in it and is not properly wrapped with sig_on(), sig_off(). You might
want to run Sage under gdb with 'sage -gdb' to debug this.
Sage will now terminate.
------------------------------------------------------------------------

Some tests actually opened a pdf viewer - at least with SAGE_DEBUG.

And some doctest processes were killed, such as

sage -t  --optional -force_lib "devel/sage/doc/en/tutorial/tour_rings.rst"
The doctested process was killed by signal 6
         [5.5 s]
sage -t  --optional -force_lib "devel/sage/doc/en/tutorial/interactive_shell.rst"
         [3.8 s]
sage -t  --optional -force_lib "devel/sage/doc/en/tutorial/tour_coercion.rst"
The doctested process was killed by signal 6
         [3.2 s]

In the latter cases, I see best chances to see an effect of #715 and friends.

I am retrying now after unsetting SAGE_DEBUG and only doing "make test", not "make testall".

[vbraun]

testall runs all tests, including optional.

I think the problem is in pickling, all failing doctests that I've seen revolve around that. The interfaces just trigger it because they pickle stuff.

[SimonKing]

Replying to vbraun:

I think the problem is in pickling, all failing doctests that I've seen revolve around that. The interfaces just trigger it because they pickle stuff.

Pickling? Interesting. How can this fail in a debug version?

[nbruin]

Replying to SimonKing:

And some doctest processes were killed, such as

sage -t  --optional -force_lib "devel/sage/doc/en/tutorial/tour_rings.rst"
The doctested process was killed by signal 6
         [5.5 s]
sage -t  --optional -force_lib "devel/sage/doc/en/tutorial/interactive_shell.rst"
         [3.8 s]
sage -t  --optional -force_lib "devel/sage/doc/en/tutorial/tour_coercion.rst"
The doctested process was killed by signal 6
         [3.2 s]

These tests succeeded for me with ZODB3-3.10.5. I don't know if these particular tests failed for me prior to upgrading ZODB, but I had lots of other tests fail. See ​sage-devel.

[SimonKing]

Replying to nbruin:

These tests succeeded for me with ZODB3-3.10.5. I don't know if these particular tests failed for me prior to upgrading ZODB, but I had lots of other tests fail.

Another Heisenbug?

Anyway. Here is a different test:

sage -t -gdb -force_lib "devel/sage/sage/tests/benchmark.py"
...
(gdb) r
Starting program: /home/simon/SAGE/debug/sage-5.6.beta1/local/bin/python /home/simon/.sage//tmp/benchmark_3070.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff6d8e8fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6d8e9a2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7b129c9 in visit_decref (op=0x3a78ed0, data=0x0) at Modules/gcmodule.c:326
#5  0x00007ffff7a1cbce in dict_traverse (op=0x30a04f0, visit=0x7ffff7b1290e <visit_decref>, arg=0x0) at Objects/dictobject.c:2088
#6  0x00007ffff7b12a3e in subtract_refs (containers=0x7ffff7dc14e0 <generations>) at Modules/gcmodule.c:345
#7  0x00007ffff7b13a73 in collect (generation=0) at Modules/gcmodule.c:874
#8  0x00007ffff7b13d96 in collect_generations () at Modules/gcmodule.c:996
#9  0x00007ffff7b14a4a in _PyObject_GC_Malloc (basicsize=56) at Modules/gcmodule.c:1457
#10 0x00007ffff7b14a82 in _PyObject_GC_New (tp=0x7ffff7d9a3c0 <PyList_Type>) at Modules/gcmodule.c:1467
#11 0x00007ffff7a02405 in PyList_New (size=242) at Objects/listobject.c:142
#12 0x00007ffff23b8395 in Pdata_popList (self=0x3a12f90, start=280) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:306
#13 0x00007ffff23c60ee in do_append (self=0x7fffbfaf26b8, x=280) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4401
#14 0x00007ffff23c6603 in load_appends (self=0x7fffbfaf26b8) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4450
#15 0x00007ffff23c7875 in load (self=0x7fffbfaf26b8) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4835
#16 0x00007ffff23c8bf9 in Unpickler_load (self=0x7fffbfaf26b8, unused=0x0) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:5393
#17 0x00007ffff7ac675c in call_function (pp_stack=0x7fffffff6cd0, oparg=0) at Python/ceval.c:4005
#18 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x362a940, throwflag=0) at Python/ceval.c:2666
#19 0x00007ffff7ac708a in fast_function (func=0x3a224f8, pp_stack=0x7fffffff7050, n=2, na=2, nk=0) at Python/ceval.c:4107
#20 0x00007ffff7ac6d76 in call_function (pp_stack=0x7fffffff7050, oparg=1) at Python/ceval.c:4042
#21 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x362a730, throwflag=0) at Python/ceval.c:2666
#22 0x00007ffff7ac708a in fast_function (func=0x3a225a0, pp_stack=0x7fffffff73d0, n=3, na=3, nk=0) at Python/ceval.c:4107
#23 0x00007ffff7ac6d76 in call_function (pp_stack=0x7fffffff73d0, oparg=2) at Python/ceval.c:4042
#24 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x362a500, throwflag=0) at Python/ceval.c:2666
#25 0x00007ffff7ac708a in fast_function (func=0x3a256f0, pp_stack=0x7fffffff7750, n=2, na=2, nk=0) at Python/ceval.c:4107
...
#52 0x00007ffff79fd3fb in function_call (func=0x7fffde9a53a8, arg=0x7fffbfa4db60, kw=0x36dbe00) at Objects/funcobject.c:526
#53 0x00007ffff79be33e in PyObject_Call (func=0x7fffde9a53a8, arg=0x7fffbfa4db60, kw=0x36dbe00) at Objects/abstract.c:2529
#54 0x00007ffff79da311 in instancemethod_call (func=0x7fffde9a53a8, arg=0x7fffbfa4db60, kw=0x36dbe00) at Objects/classobject.c:2578
#55 0x00007ffff79be33e in PyObject_Call (func=0x7fffbfd5a760, arg=0x7fffc2b4ad08, kw=0x36dbe00) at Objects/abstract.c:2529
#56 0x00007fffe21bd081 in __pyx_pf_4sage_9structure_7factory_13UniqueFactory_4__call__ (__pyx_v_self=0x7fffde99f418, __pyx_v_args=0x7fffc2b4ad08, __pyx_v_kwds=0x36dbe00)
    at sage/structure/factory.c:1028
#57 0x00007fffe21bcda7 in __pyx_pw_4sage_9structure_7factory_13UniqueFactory_5__call__ (__pyx_v_self=0x7fffde99f418, __pyx_args=0x7fffc2b4ad08, __pyx_kwds=0x0) at sage/structure/factory.c:979
#58 0x00007ffff79be33e in PyObject_Call (func=0x7fffde99f418, arg=0x7fffc2b4ad08, kw=0x0) at Objects/abstract.c:2529
#59 0x00007ffff7ac7a58 in do_call (func=0x7fffde99f418, pp_stack=0x7fffffff95a0, na=2, nk=0) at Python/ceval.c:4239
#60 0x00007ffff7ac6d92 in call_function (pp_stack=0x7fffffff95a0, oparg=2) at Python/ceval.c:4044
#61 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x38ca1b0, throwflag=0) at Python/ceval.c:2666
#62 0x00007ffff7ac40aa in PyEval_EvalCodeEx (co=0x7fffc02441a0, globals=0x34bc850, locals=0x34bc850, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#63 0x00007ffff7ab9a88 in PyEval_EvalCode (co=0x7fffc02441a0, globals=0x34bc850, locals=0x34bc850) at Python/ceval.c:667
#64 0x00007ffff7ac9693 in exec_statement (f=0x36c0720, prog=0x7fffc02441a0, globals=0x34bc850, locals=0x34bc850) at Python/ceval.c:4718
#65 0x00007ffff7abde26 in PyEval_EvalFrameEx (f=0x36c0720, throwflag=0) at Python/ceval.c:1880
#66 0x00007ffff7ac40aa in PyEval_EvalCodeEx (co=0x7fffc2bf9d50, globals=0x7140d0, locals=0x0, args=0x7fffbfd46238, argcount=5, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3253
#67 0x00007ffff79fd3fb in function_call (func=0x7fffc2c12840, arg=0x7fffbfd46210, kw=0x0) at Objects/funcobject.c:526
#68 0x00007ffff79be33e in PyObject_Call (func=0x7fffc2c12840, arg=0x7fffbfd46210, kw=0x0) at Objects/abstract.c:2529
#69 0x00007ffff79da311 in instancemethod_call (func=0x7fffc2c12840, arg=0x7fffbfd46210, kw=0x0) at Objects/classobject.c:2578
#70 0x00007ffff79be33e in PyObject_Call (func=0x7fffbfaf6060, arg=0x7fffbfd46210, kw=0x0) at Objects/abstract.c:2529
#71 0x00007ffff7ac7a58 in do_call (func=0x7fffbfaf6060, pp_stack=0x7fffffffa270, na=5, nk=0) at Python/ceval.c:4239
#72 0x00007ffff7ac6d92 in call_function (pp_stack=0x7fffffffa270, oparg=5) at Python/ceval.c:4044
#73 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x323cc40, throwflag=0) at Python/ceval.c:2666
#74 0x00007ffff7ac40aa in PyEval_EvalCodeEx (co=0x7fffc2eb00f0, globals=0x713f70, locals=0x0, args=0x37db250, argcount=5, kws=0x37db278, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3253

As much as I understand, the error occurs after finishing the tests. However, running in verbose mode, I get:

...
Trying:
    from sage.tests.benchmark import EllipticCurveMW###line 1612:_sage_    >>> from sage.tests.benchmark import EllipticCurveMW
Expecting nothing
ok
Trying:
    B = EllipticCurveMW([Integer(1),Integer(2),Integer(3),Integer(4),Integer(5)])###line 1613:_sage_    >>> B = EllipticCurveMW([1,2,3,4,5])
Expecting nothing
ok
Trying:
    sig_on_count()
Expecting:
    0
ok
Trying:
    set_random_seed(0L)
Expecting nothing
ok
Trying:
    change_warning_output(sys.stdout)
Expecting nothing
ok
Trying:
    from sage.tests.benchmark import FiniteExtFieldMult###line 1636:_sage_    >>> from sage.tests.benchmark import FiniteExtFieldMult
Expecting nothing
ok
Trying:
    B = FiniteExtFieldMult(GF(Integer(9), 'x'), Integer(2))###line 1637:_sage_    >>> B = FiniteExtFieldMult(GF(9, 'x'), 2)
Expecting nothing
python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6

So, is it in the middle of the tests, after all?

Anyway, the backtrace confirms Volker's statement that the problem is related with pickling.

[SimonKing]

Even better!! The above test yields a reproducible crash on the command line:

sage -gdb
...
sage: from sage.tests.benchmark import FiniteExtFieldMult
sage: B = FiniteExtFieldMult(GF(Integer(9), 'x'), Integer(2))
python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff6d8e8fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6d8e9a2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7b129c9 in visit_decref (op=0x7fffc1b83a38, data=0x0) at Modules/gcmodule.c:326
#5  0x00007ffff7a1cbce in dict_traverse (op=0x3017a60, visit=0x7ffff7b1290e <visit_decref>, arg=0x0) at Objects/dictobject.c:2088
#6  0x00007ffff7b12a3e in subtract_refs (containers=0x7ffff7dc14e0 <generations>) at Modules/gcmodule.c:345
#7  0x00007ffff7b13a73 in collect (generation=0) at Modules/gcmodule.c:874
#8  0x00007ffff7b13d96 in collect_generations () at Modules/gcmodule.c:996
#9  0x00007ffff7b14a4a in _PyObject_GC_Malloc (basicsize=56) at Modules/gcmodule.c:1457
#10 0x00007ffff7b14a82 in _PyObject_GC_New (tp=0x7ffff7d9a3c0 <PyList_Type>) at Modules/gcmodule.c:1467
#11 0x00007ffff7a02405 in PyList_New (size=168) at Objects/listobject.c:142
#12 0x00007ffff2a2c395 in Pdata_popList (self=0x7fffc1c1ff90, start=80) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:306
#13 0x00007ffff2a3a0ee in do_append (self=0x7fffc26c17a0, x=80) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4401
#14 0x00007ffff2a3a603 in load_appends (self=0x7fffc26c17a0) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4450
#15 0x00007ffff2a3b875 in load (self=0x7fffc26c17a0) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4835
#16 0x00007ffff2a3cbf9 in Unpickler_load (self=0x7fffc26c17a0, unused=0x0) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:5393
#17 0x00007ffff7ac675c in call_function (pp_stack=0x7fffffff81e0, oparg=0) at Python/ceval.c:4005
#18 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x301ca10, throwflag=0) at Python/ceval.c:2666
#19 0x00007ffff7ac708a in fast_function (func=0x7fffc1bac060, pp_stack=0x7fffffff8560, n=2, na=2, nk=0) at Python/ceval.c:4107
#20 0x00007ffff7ac6d76 in call_function (pp_stack=0x7fffffff8560, oparg=1) at Python/ceval.c:4042
#21 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x301c800, throwflag=0) at Python/ceval.c:2666
#22 0x00007ffff7ac708a in fast_function (func=0x7fffc1bac108, pp_stack=0x7fffffff88e0, n=3, na=3, nk=0) at Python/ceval.c:4107
#23 0x00007ffff7ac6d76 in call_function (pp_stack=0x7fffffff88e0, oparg=2) at Python/ceval.c:4042
#24 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x301c5d0, throwflag=0) at Python/ceval.c:2666
#25 0x00007ffff7ac708a in fast_function (func=0x7fffc1baf258, pp_stack=0x7fffffff8c60, n=2, na=2, nk=0) at Python/ceval.c:4107
#26 0x00007ffff7ac6d76 in call_function (pp_stack=0x7fffffff8c60, oparg=1) at Python/ceval.c:4042
#27 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x301c3a0, throwflag=0) at Python/ceval.c:2666
#28 0x00007ffff7ac40aa in PyEval_EvalCodeEx (co=0x7fffc1c57d50, globals=0x2f88d70, locals=0x0, args=0x7fffc2720358, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3253
#29 0x00007ffff79fd3fb in function_call (func=0x7fffc1baf1b0, arg=0x7fffc2720330, kw=0x0) at Objects/funcobject.c:526
#30 0x00007ffff79be33e in PyObject_Call (func=0x7fffc1baf1b0, arg=0x7fffc2720330, kw=0x0) at Objects/abstract.c:2529
#31 0x00007ffff79da311 in instancemethod_call (func=0x7fffc1baf1b0, arg=0x7fffc2720330, kw=0x0) at Objects/classobject.c:2578
...

I'd say this is something we now try to understand what is happening here. Could it be that there is some unpickle function that is not correctly raising the refcount?

[nbruin]

Replying to SimonKing:

sage -t -gdb -force_lib "devel/sage/sage/tests/benchmark.py"

Yes, that traceback is consistent with what I was observing prior to my ZODB upgrade. My guess is that if you step up to frame #4 and execute

(gdb) p PyString_AsString(PyObject_Repr(op)) 

you'll see it's stumbling on a BTree object. Upgrading ZODB should do the trick (i.e., download from ​http://pypi.python.org/pypi/ZODB3/3.10.5 and then run

./sage -sh
python setup.py install

The most immediate trigger is a garbage collection. ZODB had its own weakref implementation which played tricks with the reference counts, apparently confusing so checks activated by pydebug. It looks like they've solved that in 3.10.

[SimonKing]

Wow! You know what actually fails? The following one-liner!

sage: K = GF(Integer(9), 'x')
python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff6d8e8fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6d8e9a2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7b129c9 in visit_decref (op=0x6407cd8, data=0x0) at Modules/gcmodule.c:326
#5  0x00007ffff7a1cbce in dict_traverse (op=0x6465ac0, visit=0x7ffff7b1290e <visit_decref>, arg=0x0) at Objects/dictobject.c:2088
#6  0x00007ffff7b12a3e in subtract_refs (containers=0x7ffff7dc14e0 <generations>) at Modules/gcmodule.c:345
#7  0x00007ffff7b13a73 in collect (generation=0) at Modules/gcmodule.c:874
#8  0x00007ffff7b13d96 in collect_generations () at Modules/gcmodule.c:996
#9  0x00007ffff7b14a4a in _PyObject_GC_Malloc (basicsize=56) at Modules/gcmodule.c:1457
#10 0x00007ffff7b14a82 in _PyObject_GC_New (tp=0x7ffff7d9a3c0 <PyList_Type>) at Modules/gcmodule.c:1467
#11 0x00007ffff7a02405 in PyList_New (size=49) at Objects/listobject.c:142
#12 0x00007ffff2f86395 in Pdata_popList (self=0x6368f90, start=100) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:306
#13 0x00007ffff2f940ee in do_append (self=0x622f4e8, x=100) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4401
#14 0x00007ffff2f94603 in load_appends (self=0x622f4e8) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4450
#15 0x00007ffff2f95875 in load (self=0x622f4e8) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4835
#16 0x00007ffff2f96bf9 in Unpickler_load (self=0x622f4e8, unused=0x0) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:5393
...

However,

sage: K = GF(Integer(3), 'x')

works fine.

So, non-prime finite fields it is, then!

I don't know about ZODB. What is it doing, and how can I upgrade it?

[SimonKing]

Here is our problem:

sage: sage.databases.conway.ConwayPolynomials().has_polynomial(3,2)
<BOOM>

So, it is a look-up in a data base, which explains why (un)pickling is involved in the backtrace.

[SimonKing]

When one traces through the above command,

sage: trace("sage: sage.databases.conway.ConwayPolynomials().has_polynomial(3,2)")

one soon sees:

> /home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/site-packages/sage/databases/db.py(125)_load_()
    124 
--> 125     def _load_(self):
    126         import BTrees.OOBTree

ipdb> 
> /home/simon/SAGE/debug/sage-5.6.beta1/local/lib/python2.7/site-packages/sage/databases/db.py(126)_load_()
    125     def _load_(self):
--> 126         import BTrees.OOBTree
    127         from ZODB import FileStorage, DB

So, ZODB is involved! How can it be upgraded?

[SimonKing]

ZODB upgrade is at #10352.

[SimonKing]

Unfortunately, the new zodb spkg from #10352 does not solve the problem:

sage: sage.databases.conway.ConwayPolynomials().has_polynomial(3,2)
python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff6d8e8fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6d8e9a2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7b129c9 in visit_decref (op=0x688ee28, data=0x0) at Modules/gcmodule.c:326
#5  0x00007ffff7a1cbce in dict_traverse (op=0x637f690, visit=0x7ffff7b1290e <visit_decref>, arg=0x0) at Objects/dictobject.c:2088
#6  0x00007ffff7b12a3e in subtract_refs (containers=0x7ffff7dc14e0 <generations>) at Modules/gcmodule.c:345
#7  0x00007ffff7b13a73 in collect (generation=0) at Modules/gcmodule.c:874
#8  0x00007ffff7b13d96 in collect_generations () at Modules/gcmodule.c:996
#9  0x00007ffff7b14a4a in _PyObject_GC_Malloc (basicsize=120) at Modules/gcmodule.c:1457
#10 0x00007ffff7a4c1b7 in PyType_GenericAlloc (type=0x7fffc5af4660 <BucketType>, nitems=0) at Objects/typeobject.c:753
#11 0x00007ffff7a4c341 in PyType_GenericNew (type=0x7fffc5af4660 <BucketType>, args=0x7ffff7f90060, kwds=0x0) at Objects/typeobject.c:778
#12 0x00007ffff7a597cd in tp_new_wrapper (self=0x7fffc5af4660 <BucketType>, args=0x7ffff7f90060, kwds=0x0) at Objects/typeobject.c:4779
#13 0x00007ffff7a21088 in PyCFunction_Call (func=0x61a0948, arg=0xf1f3e0, kw=0x0) at Objects/methodobject.c:85
#14 0x00007ffff7ac6b75 in call_function (pp_stack=0x7fffffff8190, oparg=1) at Python/ceval.c:4021
#15 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x63727d0, throwflag=0) at Python/ceval.c:2666
#16 0x00007ffff7ac40aa in PyEval_EvalCodeEx (co=0x64f70f0, globals=0x6273ce0, locals=0x0, args=0xe20688, argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#17 0x00007ffff79fd3fb in function_call (func=0x6500060, arg=0xe20660, kw=0x0) at Objects/funcobject.c:526
#18 0x00007ffff79be33e in PyObject_Call (func=0x6500060, arg=0xe20660, kw=0x0) at Objects/abstract.c:2529
#19 0x00007ffff7ac8113 in ext_do_call (func=0x6500060, pp_stack=0x7fffffff8698, flags=1, na=1, nk=0) at Python/ceval.c:4334
#20 0x00007ffff7ac194b in PyEval_EvalFrameEx (f=0x63725b0, throwflag=0) at Python/ceval.c:2705
#21 0x00007ffff7ac40aa in PyEval_EvalCodeEx (co=0x64f7040, globals=0x6273ce0, locals=0x0, args=0x6516cb8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#22 0x00007ffff79fd3fb in function_call (func=0x64eff78, arg=0x6516c90, kw=0x0) at Objects/funcobject.c:526
#23 0x00007ffff79be33e in PyObject_Call (func=0x64eff78, arg=0x6516c90, kw=0x0) at Objects/abstract.c:2529
#24 0x00007ffff79da311 in instancemethod_call (func=0x64eff78, arg=0x6516c90, kw=0x0) at Objects/classobject.c:2578
#25 0x00007ffff79be33e in PyObject_Call (func=0x619e260, arg=0x81a840, kw=0x0) at Objects/abstract.c:2529
#26 0x00007ffff2f92b73 in load_binpersid (self=0x62c9a58) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4083
#27 0x00007ffff2f95a46 in load (self=0x62c9a58) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:4927
#28 0x00007ffff2f96bf9 in Unpickler_load (self=0x62c9a58, unused=0x0) at /home/simon/SAGE/debug/sage-5.6.beta1/spkg/build/python-2.7.3.p5/src/Modules/cPickle.c:5393
...

[SimonKing]

The spkg from #10352 does not solve the problem. However, when I manually install zodb as you described in comment:79, things start to work!

[nbruin]

Replying to SimonKing:

The spkg from #10352 does not solve the problem. However, when I manually install zodb as you described in comment:79, things start to work!

I have changed that ticket to link to an upgraded spkg, which should resolve the issue.

[SimonKing]

With a manually installed zodb3-3.10.5 and the other involved tickets, make ptest with MALLOC_CHECK_=3 results in the following crashes:

The following tests failed:

        sage -t  -force_lib devel/sage/sage/combinat/tiling.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_local_data.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_finite_field.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_curve_isogeny.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/BSD.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_number_field.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_torsion.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_field.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/heegner.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_point.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/gal_reps.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/modules/free_module.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/modules/matrix_morphism.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/algebras/quatalg/quaternion_algebra.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/quadratic_forms/quadratic_form__ternary_Tornaria.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/quadratic_forms/quadratic_form__genus.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/quadratic_forms/quadratic_form__equivalence_testing.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/quadratic_forms/genera/genus.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/homology/chain_complex.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/modular/quatalg/brandt.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/modular/abvar/abvar.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/modular/modform/find_generators.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/modular/modsym/ambient.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/rings/residue_field.pyx # Killed/crashed
        sage -t  -force_lib devel/sage/sage/rings/number_field/number_field_element.pyx # Killed/crashed
        sage -t  -force_lib devel/sage/sage/rings/number_field/number_field_ideal.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/rings/polynomial/pbori.pyx # Killed/crashed
        sage -t  -force_lib devel/sage/sage/rings/finite_rings/finite_field_base.pyx # Killed/crashed
        sage -t  -force_lib devel/sage/sage/matrix/benchmark.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/matrix/matrix_modn_dense_template.pxi # Killed/crashed
        sage -t  -force_lib devel/sage/sage/matrix/matrix_integer_dense_hnf.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/matrix/matrix_integer_dense_saturation.py # Killed/crashed
        sage -t  -force_lib devel/sage/sage/matrix/matrix_cyclo_dense.pyx # Killed/crashed
        sage -t  -force_lib devel/sage/sage/matrix/matrix2.pyx # Killed/crashed
----------------------------------------------------------------------
Total time for all tests: 7238.7 seconds

Nils has reported a failure in an assert in linbox/matric/permutation-matrix.h, that I can confirm. Again with MALLOC_CHECK_=3 and gdb:

sage: sage.homology.chain_complex.HomologyGroup(100)


ERROR (at getWritePointer in /home/simon/SAGE/debug/sage-5.6.beta1/local/include/linbox/matrix/permutation-matrix.h:175): 
Precondition not met:P_.size()
terminate called after throwing an instance of 'LinBox::PreconditionFailed'

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff2295f2d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x00007ffff22940d6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00007ffff2294103 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x00007ffff22941fe in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00007fffd718c62f in LinBox::BlasPermutation<unsigned long>::getWritePointer (this=0x7fffffff6760) at /home/simon/SAGE/debug/sage-5.6.beta1/local/include/linbox/matrix/permutation-matrix.h:175
#7  0x00007fffd717f0b1 in LinBox::LQUPMatrix<LinBox::Modular<double> >::LQUPMatrix (this=0x7fffffff6700, A=..., P=..., Q=...)
    at /home/simon/SAGE/debug/sage-5.6.beta1/local/include/linbox/matrix/factorized-matrix.inl:810
#8  0x00007fffd716b350 in LinBox::EchelonFormDomain<LinBox::Modular<double> >::columnReducedEchelon<LinBox::BlasMatrix<LinBox::Modular<double> > > (this=0x6a247c0, E=...)
    at /home/simon/SAGE/debug/sage-5.6.beta1/local/include/linbox/algorithms/echelon-form.h:205
#9  0x00007fffd716653e in LinBox::EchelonFormDomain<LinBox::Modular<double> >::rowReducedEchelon<LinBox::BlasMatrix<LinBox::Modular<double> > > (this=0x6a247c0, E=..., A=...)
    at /home/simon/SAGE/debug/sage-5.6.beta1/local/include/linbox/algorithms/echelon-form.h:107
#10 0x00007fffd71279f5 in __pyx_f_4sage_6matrix_24matrix_modn_dense_double_linbox_echelonize_efd (__pyx_v_modulus=21059, __pyx_v_entries=0x6a24730, __pyx_v_nrows=0, __pyx_v_ncols=100)
    at sage/matrix/matrix_modn_dense_double.cpp:4074
#11 0x00007fffd71417ca in __pyx_pf_4sage_6matrix_24matrix_modn_dense_double_26Matrix_modn_dense_template_34_echelonize_linbox (__pyx_v_self=0x66aa6d8, __pyx_v_efd=0x7ffff7d89ec0 <_Py_TrueStruct>)
    at sage/matrix/matrix_modn_dense_double.cpp:11280
#12 0x00007fffd71411c1 in __pyx_pw_4sage_6matrix_24matrix_modn_dense_double_26Matrix_modn_dense_template_35_echelonize_linbox (__pyx_v_self=0x66aa6d8, __pyx_args=0x7ffff7f90060, __pyx_kwds=0x6a39940)
    at sage/matrix/matrix_modn_dense_double.cpp:11185
#13 0x00007ffff7a21088 in PyCFunction_Call (func=0x66b40d8, arg=0x7ffff7f90060, kw=0x6a39940) at Objects/methodobject.c:85
#14 0x00007ffff79be33e in PyObject_Call (func=0x66b40d8, arg=0x7ffff7f90060, kw=0x6a39940) at Objects/abstract.c:2529
#15 0x00007fffd713f3ce in __pyx_pf_4sage_6matrix_24matrix_modn_dense_double_26Matrix_modn_dense_template_32echelonize (__pyx_v_self=0x66aa6d8, __pyx_v_algorithm=0x26ae2e0, __pyx_v_kwds=0x6a37920)
    at sage/matrix/matrix_modn_dense_double.cpp:10889
#16 0x00007fffd713e64a in __pyx_pw_4sage_6matrix_24matrix_modn_dense_double_26Matrix_modn_dense_template_33echelonize (__pyx_v_self=0x66aa6d8, __pyx_args=0x7ffff7f90060, __pyx_kwds=0x6a368f0)
    at sage/matrix/matrix_modn_dense_double.cpp:10738
#17 0x00007ffff7a21088 in PyCFunction_Call (func=0x65abc90, arg=0x7ffff7f90060, kw=0x6a368f0) at Objects/methodobject.c:85
#18 0x00007ffff79be33e in PyObject_Call (func=0x65abc90, arg=0x7ffff7f90060, kw=0x6a368f0) at Objects/abstract.c:2529
#19 0x00007fffd9b50c55 in __pyx_pf_4sage_6matrix_7matrix2_6Matrix_134echelon_form (__pyx_v_self=0x66aa568, __pyx_v_algorithm=0x653be0, __pyx_v_cutoff=0x61f5b8, __pyx_v_kwds=0x67154d0)
    at sage/matrix/matrix2.c:31710
#20 0x00007fffd9b4ffca in __pyx_pw_4sage_6matrix_7matrix2_6Matrix_135echelon_form (__pyx_v_self=0x66aa568, __pyx_args=0x7ffff7f90060, __pyx_kwds=0x0) at sage/matrix/matrix2.c:31508
#21 0x00007ffff7a21088 in PyCFunction_Call (func=0xfb0588, arg=0x7ffff7f90060, kw=0x0) at Objects/methodobject.c:85
#22 0x00007ffff79be33e in PyObject_Call (func=0xfb0588, arg=0x7ffff7f90060, kw=0x0) at Objects/abstract.c:2529
#23 0x00007fffd95f2e94 in __pyx_pf_4sage_6matrix_7matrix0_6Matrix_124pivots (__pyx_v_self=0x66aa568) at sage/matrix/matrix0.c:20883
...

Given the list of crashing tests above, a failure in echelonisation of matrices is likely to explain most of the crashes.

Question: Is that an upstream bug (in Linbox) or did we screw up by ourselves?

[SimonKing]

Since this is a tracker ticket, I have created #13878 to deal with the Linbox related problem.

[SimonKing]

Nils' preliminary patch from #13878 fixes the Linbox related problem. I am now running make ptest again.

[SimonKing]

#13878 turns out to be a considerable progress. With MALLOC_CHECK_=3, make ptest results in only two crashes and one file with three failures.

sage -t -force_lib "devel/sage/sage/misc/memory_info.py"    
**********************************************************************
File "/home/simon/SAGE/debug/sage-5.6.beta1/devel/sage/sage/misc/memory_info.py", line 342:
    sage: MemoryInfo().available_swap() > 0
Expected:
    True
Got:
    False
**********************************************************************
File "/home/simon/SAGE/debug/sage-5.6.beta1/devel/sage/sage/misc/memory_info.py", line 421:
    sage: MemoryInfo().available_swap() > 0
Expected:
    True
Got:
    False
**********************************************************************
File "/home/simon/SAGE/debug/sage-5.6.beta1/devel/sage/sage/misc/memory_info.py", line 203:
    sage: MemoryInfo().available_swap() > 0
Expected:
    True
Got:
    False
**********************************************************************
3 items had failures:
   1 of   5 in __main__.example_13
   1 of   5 in __main__.example_18
   1 of   5 in __main__.example_7
***Test Failed*** 3 failures.
For whitespace errors, see the file /home/simon/.sage//tmp/memory_info_26498.py
         [3.4 s]
 
----------------------------------------------------------------------
The following tests failed:


        sage -t -force_lib "devel/sage/sage/misc/memory_info.py"
Total time for all tests: 3.5 seconds

I don't know that these errors mean. Is it perhaps the case that a debug version of Sage does not use swap?

The crashes, with gdb backtraces, are

sage -t -gdb -force_lib devel/sage/sage/combinat/tiling.py
...
python: sage/combinat/matrices/dancing_links_c.h:217: void dancing_links::setup_columns(): Assertion `nr_columns > 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff6d8e8fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6d8e9a2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fffc4260dc2 in dancing_links::setup_columns (this=0x7fffc047e620) at sage/combinat/matrices/dancing_links_c.h:217
#5  0x00007fffc4261101 in dancing_links::add_rows (this=0x7fffc047e620, rows=...) at sage/combinat/matrices/dancing_links_c.h:268
#6  0x00007fffc425a51d in __pyx_pf_4sage_8combinat_8matrices_13dancing_links_20dancing_linksWrapper_14add_rows (__pyx_v_self=0x7fffc047e600, __pyx_v_rows=0x7fffbd3d21c8)
    at sage/combinat/matrices/dancing_links.cpp:2666
#7  0x00007fffc4259c18 in __pyx_pw_4sage_8combinat_8matrices_13dancing_links_20dancing_linksWrapper_15add_rows (__pyx_v_self=0x7fffc047e600, __pyx_v_rows=0x7fffbd3d21c8)
    at sage/combinat/matrices/dancing_links.cpp:2464
#8  0x00007ffff7a21151 in PyCFunction_Call (func=0x7fffbd2dcb28, arg=0x7fffbd386e60, kw=0x0) at Objects/methodobject.c:101
#9  0x00007ffff79be33e in PyObject_Call (func=0x7fffbd2dcb28, arg=0x7fffbd386e60, kw=0x0) at Objects/abstract.c:2529
#10 0x00007fffc4258130 in __pyx_pf_4sage_8combinat_8matrices_13dancing_links_20dancing_linksWrapper_2__cinit__ (__pyx_v_self=0x7fffc047e600, __pyx_v_rows=0x7fffbd3d21c8)
    at sage/combinat/matrices/dancing_links.cpp:2010
#11 0x00007fffc4257e79 in __pyx_pw_4sage_8combinat_8matrices_13dancing_links_20dancing_linksWrapper_3__cinit__ (__pyx_v_self=0x7fffc047e600, __pyx_args=0x7fffc2ebd920, __pyx_kwds=0x0)
    at sage/combinat/matrices/dancing_links.cpp:1946
#12 0x00007fffc425bb28 in __pyx_tp_new_4sage_8combinat_8matrices_13dancing_links_dancing_linksWrapper (t=0x7fffc446d000 <__pyx_type_4sage_8combinat_8matrices_13dancing_links_dancing_linksWrapper>, 
    a=0x7fffc2ebd920, k=0x0) at sage/combinat/matrices/dancing_links.cpp:3037
#13 0x00007ffff7a4bfe3 in type_call (type=0x7fffc446d000 <__pyx_type_4sage_8combinat_8matrices_13dancing_links_dancing_linksWrapper>, args=0x7fffc2ebd920, kwds=0x0) at Objects/typeobject.c:721
#14 0x00007ffff79be33e in PyObject_Call (func=0x7fffc446d000 <__pyx_type_4sage_8combinat_8matrices_13dancing_links_dancing_linksWrapper>, arg=0x7fffc2ebd920, kw=0x0) at Objects/abstract.c:2529
#15 0x00007fffc425af12 in __pyx_pf_4sage_8combinat_8matrices_13dancing_links_dlx_solver (__pyx_self=0x0, __pyx_v_rows=0x7fffbd3d21c8) at sage/combinat/matrices/dancing_links.cpp:2888
#16 0x00007fffc425ae33 in __pyx_pw_4sage_8combinat_8matrices_13dancing_links_1dlx_solver (__pyx_self=0x0, __pyx_v_rows=0x7fffbd3d21c8) at sage/combinat/matrices/dancing_links.cpp:2852
...

and

sage -t -gdb -force_lib "devel/sage/sage/rings/polynomial/pbori.pyx"
...
python: groebner/include/polybori/groebner/PolyEntryVector.h:74: virtual void polybori::groebner::PolyEntryVector::append(const polybori::groebner::PolyEntry&): Assertion `m_indices.checked(back().lead) == (size_type)-1' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff6d8e8fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6d8e9a2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fffd0dc0073 in polybori::groebner::PolyEntryVector::append(polybori::groebner::PolyEntry const&) [clone .part.506] ()
   from /home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpolybori_groebner-0.8.2.so.0
#5  0x00007fffd0ddc5b9 in polybori::groebner::FGLMStrategy::FGLMStrategy(polybori::BoolePolyRing const&, polybori::BoolePolyRing const&, std::vector<polybori::BoolePolynomial, std::allocator<polybori::BoolePolynomial> > const&) () from /home/simon/SAGE/debug/sage-5.6.beta1/local/lib/libpolybori_groebner-0.8.2.so.0
#6  0x00007fffd16fd8b7 in WrappedPtr<polybori::groebner::FGLMStrategy>::WrappedPtr<polybori::BoolePolyRing, polybori::BoolePolyRing, std::vector<polybori::BoolePolynomial> > (this=0x7fffffff9240, 
    arg1=..., arg2=..., arg3=...) at /home/simon/SAGE/debug/sage-5.6.beta1/local/include/csage/pb_wrap.h:197
#7  0x00007fffd16a1c5e in __pyx_pf_4sage_5rings_10polynomial_5pbori_12FGLMStrategy___init__ (__pyx_v_self=0x7fffc284f498, __pyx_v_from_ring=0x3e04420, __pyx_v_to_ring=0x3d5ca50, 
    __pyx_v_vec=0x7fffc2851060) at sage/rings/polynomial/pbori.cpp:39777
#8  0x00007fffd16a0e26 in __pyx_pw_4sage_5rings_10polynomial_5pbori_12FGLMStrategy_1__init__ (__pyx_v_self=0x7fffc284f498, __pyx_args=0x7fffc284b460, __pyx_kwds=0x0)
    at sage/rings/polynomial/pbori.cpp:39568
#9  0x00007ffff7a4c0dd in type_call (type=0x7fffd197ec20 <__pyx_type_4sage_5rings_10polynomial_5pbori_FGLMStrategy>, args=0x7fffc284b460, kwds=0x0) at Objects/typeobject.c:737
#10 0x00007ffff79be33e in PyObject_Call (func=0x7fffd197ec20 <__pyx_type_4sage_5rings_10polynomial_5pbori_FGLMStrategy>, arg=0x7fffc284b460, kw=0x0) at Objects/abstract.c:2529
#11 0x00007ffff7ac7a58 in do_call (func=0x7fffd197ec20 <__pyx_type_4sage_5rings_10polynomial_5pbori_FGLMStrategy>, pp_stack=0x7fffffff95a0, na=3, nk=0) at Python/ceval.c:4239
#12 0x00007ffff7ac6d92 in call_function (pp_stack=0x7fffffff95a0, oparg=3) at Python/ceval.c:4044
#13 0x00007ffff7ac16ba in PyEval_EvalFrameEx (f=0x3ec7250, throwflag=0) at Python/ceval.c:2666
#14 0x00007ffff7ac40aa in PyEval_EvalCodeEx (co=0x7fffa97d59e0, globals=0x30f8e00, locals=0x30f8e00, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253
#15 0x00007ffff7ab9a88 in PyEval_EvalCode (co=0x7fffa97d59e0, globals=0x30f8e00, locals=0x30f8e00) at Python/ceval.c:667
#16 0x00007ffff7ac9693 in exec_statement (f=0x3e03f20, prog=0x7fffa97d59e0, globals=0x30f8e00, locals=0x30f8e00) at Python/ceval.c:4718
#17 0x00007ffff7abde26 in PyEval_EvalFrameEx (f=0x3e03f20, throwflag=0) at Python/ceval.c:1880
...

I suggest someone (or I, a bit later) opens two new tickets for the two crashes, and a third for the lack of swap, if that is really an issue here.

[SimonKing]

I am not totally sure whether #13880 should be a dependency, since apparently only I am getting the error concerning available swap. Remove the new dependency, if you like.

[SimonKing]

I have opened #13882 for the combinat/matrix problem.

[SimonKing]

With the patch from #13882, sage -t -gdb -force_lib "devel/sage/sage/combinat/tiling.py" passes.

[SimonKing]

The ticket for the polybori issue is #13883

[jdemeyer]

#13880 has nothing to do with this.

[vbraun]

cygdb (gdb support for cython) is at #13881. If you are tired trying to figure out by hand what the cython stackframes are doing give it a spin ;-)

[vbraun]

Evil patch to trick Python into running a garbage collection in the TripleDictEraser? callback

[vbraun]

Evil patch to trick Python into running a garbage collection in the TripleDictEraser? callback

[vbraun]

crash log with gc_tester.patch

[SimonKing]

Should one add #13881 and/or #13889 as further dependencies? After all, this here is a task to enable debugging for Sage, and #13881/#13889 will help.

[vbraun]

I'm obviously in favor of adding #13881/#13889 ;-)

[jdemeyer]

This is a meta-ticket without anything to merge, right? Then adding dependencies doesn't hurt.

[jdemeyer]

However, on normal tickets, you should only add dependencies which are strictly required.

[SimonKing]

I lost track: Is there a ticket in which the question is discussed whether or not one should have self.D._refcache.__delitem__((k1,k2,k3)) or del self.D._refcache[k1,k2,k3] in the TripleDict code from #715?

Anyway, I see that the latter version of the code gives rise to a shorter C-code, and reportedly it fixes some problem. So, I think there should be a ticket.

[jpflori]

Replying to SimonKing:

I lost track: Is there a ticket in which the question is discussed whether or not one should have self.D._refcache.__delitem__((k1,k2,k3)) or del self.D._refcache[k1,k2,k3] in the TripleDict code from #715?

Anyway, I see that the latter version of the code gives rise to a shorter C-code, and reportedly it fixes some problem. So, I think there should be a ticket.

Indeed, it avoids the method name resolution and the creation of a tuple. And so potentially give less chances that garbage collections happen during the callback. So I'm for this change as well.

[SimonKing]

I have created #13904 for the TripleDict.__delitem__ problem.

[SimonKing]

I added #13883 as a dependency and ordered the list of dependencies.

[SimonKing]

As it turns out, the problem with segfaulting TripleDictEraser calls will be fixed with #13896 (Cython upgrade), so that I replace #13904 by #13896 in the list of dependencies.

[SimonKing]

I removed the phrase "At the end, Sage will die in a SIGABRT during "Testing startup" from the ticket description, since this is not the case anymore (at least not on bsd.math).

[SimonKing]

Apparently on also needs to remove the existing Singular spkg - when I tried to build on bsd.math, the wrong Singular version was built!!

[jdemeyer]

Replying to SimonKing:

Apparently on also needs to remove the existing Singular spkg - when I tried to build on bsd.math, the wrong Singular version was built!!

This depends on timestamps. The "latest" version of Singular will be built, where latest is defined using the timestamp on the .spkg files.

[vbraun]

I still get the crash in "testing startup". In fact without #13868 its pretty obvious that you must crash as soon as an integer from the free pool is being collected. You need to use integers for a bit to create->dealloc to pool->alloc from pool->dealloc but for me that happens during startup already.

[SimonKing]

Replying to vbraun:

I still get the crash in "testing startup". In fact without #13868 its pretty obvious that you must crash as soon as an integer from the free pool is being collected. You need to use integers for a bit to create->dealloc to pool->alloc from pool->dealloc but for me that happens during startup already.

I am now rebuilding on bsd.math from scratch, were I took care to remove "old" (version-wise, not timestamp-wise) spkgs. Let's see what happens.

[jpflori]

It seems very fishy to me that without the patches from #13868, everything is fine with pydebug.

[SimonKing]

Replying to SimonKing:

Replying to vbraun:

I still get the crash in "testing startup". In fact without #13868 its pretty obvious that you must crash as soon as an integer from the free pool is being collected. You need to use integers for a bit to create->dealloc to pool->alloc from pool->dealloc but for me that happens during startup already.

I am now rebuilding on bsd.math from scratch, were I took care to remove "old" (version-wise, not timestamp-wise) spkgs. Let's see what happens.

Yep, I get SIGABRT now. Good.

[SimonKing]

Too bad. I forgot to apply #13878 (which was missing in the ticket description). Without it, I got numerous crashes. So, trying again...

[SimonKing]

However, even with #13878, the following problem with pari/gp persists:

sage -t -force_lib "devel/sage/sage/schemes/plane_conics/con_rational_field.py"
**********************************************************************
File "/scratch/sking/sage-5.6.beta2/devel/sage/sage/schemes/plane_conics/con_rational_field.py", line 130:
    sage: C.has_rational_point(point = True)
Exception raised:
    Traceback (most recent call last):
      File "/scratch/sking/sage-5.6.beta2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/scratch/sking/sage-5.6.beta2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/scratch/sking/sage-5.6.beta2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_3[10]>", line 1, in <module>
        C.has_rational_point(point = True)###line 130:
    sage: C.has_rational_point(point = True)
      File "/scratch/sking/sage-5.6.beta2/local/lib/python/site-packages/sage/schemes/plane_conics/con_rational_field.py", line 174, in has_rational_point
        pt = qfsolve(M)
      File "/scratch/sking/sage-5.6.beta2/local/lib/python/site-packages/sage/quadratic_forms/qfsolve.py", line 100, in qfsolve
        ret = pari(gp('Qfsolve(%s)' % G._pari_()))
      File "/scratch/sking/sage-5.6.beta2/local/lib/python/site-packages/sage/interfaces/interface.py", line 195, in __call__
        return cls(self, x, name=name)
      File "/scratch/sking/sage-5.6.beta2/local/lib/python/site-packages/sage/interfaces/expect.py", line 1311, in __init__
        raise TypeError, x
    TypeError: Error executing code in GP:
    CODE:
        sage[4]=Qfsolve([1, 0, 0; 0, 113922743, 0; 0, 0, -310146482690273725409]);
    PARI/GP ERROR:
      ***   at top-level: sage[4]=Qfsolve([1,0,0;0,113
      ***                         ^--------------------
      ***   in function Qfsolve: ... du determinant"));factD=factor(-abs(2*d));if
      ***                                                    ^--------------------
      *** factor: bug in PARI/GP (Segmentation Fault), please report
**********************************************************************
1 items had failures:
   1 of  18 in __main__.example_3
***Test Failed*** 1 failures.
For whitespace errors, see the file /Users/SimonKing/.sage//tmp/con_rational_field_24352.py
         [7.4 s]
 
----------------------------------------------------------------------
The following tests failed:


        sage -t -force_lib "devel/sage/sage/schemes/plane_conics/con_rational_field.py"

Is that a known problem? I sense that we will soon have another dependency...

[SimonKing]

The total result of make ptest on bsd.math is:

        sage -t  -force_lib devel/sage/doc/en/bordeaux_2008/l_series.rst # 5 doctests failed
        sage -t  -force_lib devel/sage/doc/en/bordeaux_2008/elliptic_curves.rst # 7 doctests failed
        sage -t  -force_lib devel/sage/doc/en/reference/coercion.rst # 2 doctests failed
        sage -t  -force_lib devel/sage/sage/interfaces/genus2reduction.py # 14 doctests failed
        sage -t  -force_lib devel/sage/sage/interfaces/interface.py # 2 doctests failed
        sage -t  -force_lib devel/sage/sage/interfaces/gp.py # 8 doctests failed
        sage -t  -force_lib devel/sage/sage/lfunctions/lcalc.py # 2 doctests failed
        sage -t  -force_lib devel/sage/sage/lfunctions/dokchitser.py # 45 doctests failed
        sage -t  -force_lib devel/sage/sage/misc/misc.py # 3 doctests failed
        sage -t  -force_lib devel/sage/sage/misc/sage_eval.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/modular/buzzard.py # 2 doctests failed
        sage -t  -force_lib devel/sage/sage/modular/modform/eis_series.py # 6 doctests failed
        sage -t  -force_lib devel/sage/sage/modular/modform/element.py # 14 doctests failed
        sage -t  -force_lib devel/sage/sage/plot/line.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/quadratic_forms/qfsolve.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/rings/number_field/number_field.py # 5 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/BSD.py # 4 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_number_field.py # 13 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/gp_simon.py # 3 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/lseries_ell.py # 8 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # 17 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/period_lattice.py # 3 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/sha_tate.py # 11 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/plane_conics/con_number_field.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/plane_conics/con_rational_field.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/tests/benchmark.py # 4 doctests failed

So, no crash in Sage, but I guess many (or all?) of these errors could be explained by crashes in PARI/GP.

[SimonKing]

I have created a #13921 for the gp crash.

[jpflori]

Just wanted to point out that the original p4 spkg which actually sets pydebug when SAGE_DEBUG is set is not linked from anywhere anymore although the p5 spkg from #13867 is based on it, so let's just merge this second ticket quickly to actually be able to use Python in debug mode with a vanilla Sage.

[SimonKing]

With sage-5.7.beta2, another dependency is needed, namely #14059. Since several of the other dependencies were already merged, I changed the directions in the ticket description accordingly.

[SimonKing]

We are almost there. With sage-5.7.beta2 and the changes indicated in the ticket description and with MALLOC_CHECK_=3, make ptest yields:

        sage -t  -force_lib devel/sage/sage/graphs/generic_graph_pyx.pyx # Killed/crashed
        sage -t  -force_lib devel/sage/doc/en/bordeaux_2008/birds_other.rst # Time out

I don't know if the time-out is meaningful. Is there a ticket already for the generic graph problem?

[jpflori]

Not sure the crash is really related to the debug version. See ​https://groups.google.com/d/msg/sage-release/nvm8f82Vs-U/CdhZmprersUJ It might be a bug always present and triggered by MALLOC_CHECK_=3 even wihtout debug Python.

[SimonKing]

Even though in generic_graph is triggered my MALLOC_CHECK_=3 and not by the debug version, I think it won't hurt to add #14067 as a dependency, since running tests with MALLOC_CHECK_=3 is (at least for me) associated with debugging.

Moreover:

sage -t -force_lib "devel/sage/doc/en/bordeaux_2008/birds_other.rst"
         [318.2 s]
 
----------------------------------------------------------------------
All tests passed!

Hence, from my perspective, this ticket can be resolved as "fixed", as soon as all dependencies are merged. Since no patch will be merged from here, I suppose it has no author.