#5838 closed defect (fixed)
crash in Singular's polynomial gcd
Reported by:  cwitty  Owned by:  malb 

Priority:  major  Milestone:  sage6.4 
Component:  commutative algebra  Keywords:  sage_object unpickle_all() NTL::InvMod abort inverse undefined 
Cc:  malb  Merged in:  
Authors:  Ralf Stephan  Reviewers:  Peter Bruin 
Report Upstream:  N/A  Work issues:  
Branch:  87d608a (Commits, GitHub, GitLab)  Commit:  
Dependencies:  Stopgaps: 
Description
Here's what I type to provoke the crash:
std = os.environ['SAGE_DATA'] + '/extcode/pickle_jar/pickle_jar.tar.bz2' sage.structure.sage_object.unpickle_all(std) sage.structure.sage_object.unpickle_all(std)
(this is the doctest of unpickle_all, plus an extra repetition of the second line)
Here's the output:
sage: std = os.environ['SAGE_DATA'] + '/extcode/pickle_jar/pickle_jar.tar.bz2' sage: sage.structure.sage_object.unpickle_all(std) /home/cwitty/sage3.4.1.rc3/local/lib/python2.5/sitepackages/IPython/iplib.py:2073: DeprecationWarning: RQDF is deprecated; use RealField(212) instead. exec code_obj in self.user_global_ns, self.user_ns Successfully unpickled 483 objects. Failed to unpickle 0 objects. sage: sage.structure.sage_object.unpickle_all(std) /home/cwitty/sage3.4.1.rc3/local/bin/sagesage: line 198: 22270 Aborted sageipython "$@" i
The first section of the backtrace is:
#0 0x00007fc7c14c2105 in raise () from /lib/libc.so.6 #1 0x00007fc7c14c3623 in abort () from /lib/libc.so.6 #2 0x00007fc7bcedd6d8 in global_NTL_error_callback () from /home/cwitty/sage3.4.1.rc3/local/lib/libcsage.so #3 0x00007fc7bc9451ef in NTL::Error ( s=0x7fc7bc97def1 "InvMod: inverse undefined") at tools.c:38 #4 0x00007fc7bc88ebd9 in NTL::InvMod (a=<value optimized out>, n=244) at ZZ.c:351 #5 0x00007fc7bc90545f in NTL::PlainRem (r=@0x7fffca29e230, a=@0x7fffca29e230, b=@0x6) at ../include/NTL/lzz_p.h:278 #6 0x00007fc7bc9140bb in NTL::GCD (d=@0x7fffca29e360, u=<value optimized out>, v=@0x7fffca29e390) at lzz_pX1.c:558 #7 0x00007fc7a963206a in gcd_poly_p (f=@0x7fffca29e9e0, g=@0x7fffca29e9d0) at /home/cwitty/sage3.4.1.rc3/local/include/NTL/lzz_pX.h:696 #8 0x00007fc7a9633835 in gcd_poly (f=<value optimized out>, g=@0x7fffca29ee50) at cf_gcd.cc:538 #9 0x00007fc7a963401e in gcd (f=@0x7fffca29ee60, g=@0x7fffca29ee50) at cf_gcd.cc:776 #10 0x00007fc7a9634601 in gcd_test_one (f=@0x7fffca29f3b0, g=@0x7fffca29f3c0, swap=true) at cf_gcd.cc:76 #11 0x00007fc7a9631b70 in gcd_poly_p (f=@0x7fffca29f7b0, g=@0x7fffca29f7a0) at cf_gcd.cc:353 #12 0x00007fc7a963373d in gcd_poly (f=<value optimized out>, g=@0x7fffca29fbb0) at cf_gcd.cc:543 #13 0x00007fc7a9635d89 in chinrem_gcd (FF=<value optimized out>, GG=<value optimized out>) at cf_gcd.cc:1088 #14 0x00007fc7a96337e1 in gcd_poly (f=<value optimized out>, g=@0x7fffca2a0560) at cf_gcd.cc:601 #15 0x00007fc7a963401e in gcd (f=@0x7fffca2a0550, g=@0x7fffca2a0560) at cf_gcd.cc:776 #16 0x00007fc7a94a69f8 in singclap_gcd (f=0x7fc7a930b198, g=0x7fc7a930b058) at clapsing.cc:230 #17 0x00007fc7a992d2f9 in __pyx_pf_4sage_5rings_10polynomial_28multi_polynomial_libsingular_23MPolynomial_libsingular_gcd (__pyx_v_self=0x5e60830, __pyx_args=<value optimized out>, __pyx_kwds=<value optimized out>) at sage/rings/polynomial/multi_polynomial_libsingular.cpp:31063 #18 0x00000000004187e3 in PyObject_Call (func=0x58ad, arg=0x58ad, kw=0x6) at Objects/abstract.c:1861 #19 0x00007fc7b3aaf02f in __pyx_pf_4sage_5rings_22fraction_field_element_20FractionFieldElement_reduce (__pyx_v_self=0x5e5e2d0, unused=<value optimized out>) at sage/rings/fraction_field_element.c:2288 #20 0x00000000004187e3 in PyObject_Call (func=0x58ad, arg=0x58ad, kw=0x6) at Objects/abstract.c:1861 #21 0x00007fc7b3aa3b2b in __pyx_pf_4sage_5rings_22fraction_field_element_20FractionFieldElement___init__ (__pyx_v_self=0x5e5e2d0, __pyx_args=<value optimized out>, __pyx_kwds=<value optimized out>) at sage/rings/fraction_field_element.c:1905
Change History (31)
comment:1 Changed 9 years ago by
 Report Upstream set to N/A
comment:2 Changed 8 years ago by
 Milestone changed from sage5.11 to sage5.12
comment:3 Changed 8 years ago by
 Milestone changed from sage6.1 to sage6.2
comment:4 Changed 8 years ago by
As an update, because the system changed since, the commands to trigger this are:
sage: import os sage: std = os.environ['SAGE_SHARE'] + '/sage/ext/pickle_jar/pickle_jar.tar.bz2' sage: sage.structure.sage_object.unpickle_all(std) sage: sage.structure.sage_object.unpickle_all(std)
comment:5 Changed 7 years ago by
 Milestone changed from sage6.2 to sage6.3
comment:6 Changed 7 years ago by
 Milestone changed from sage6.3 to sage6.4
comment:7 Changed 7 years ago by
 Milestone changed from sage6.4 to sageduplicate/invalid/wontfix
 Status changed from new to needs_review
The output of the code in comment:4 is now
Successfully unpickled 586 objects. Failed to unpickle 0 objects.
which I guess is OK.
comment:8 followup: ↓ 10 Changed 7 years ago by
Wouldn't it make sense to add a doctest for this, to make sure that it stays fixed?
comment:9 Changed 7 years ago by
 Branch set to u/rws/crash_in_singular_s_polynomial_gcd
comment:10 in reply to: ↑ 8 Changed 7 years ago by
 Commit set to 1b91e8a85a560b02f9feebadea713a79a6b3ebfa
 Milestone changed from sageduplicate/invalid/wontfix to sage6.4
 Priority changed from critical to major
comment:11 followup: ↓ 12 Changed 7 years ago by
Question:
why there was a crash in Singular's polynomial gcd and how it got fixed?
It is not obvious for me...( I'm new to sage and do not yet understand everything)
comment:12 in reply to: ↑ 11 Changed 7 years ago by
Replying to jakobkroeker:
Question:
why there was a crash in Singular's polynomial gcd and how it got fixed?
It is not obvious for me...( I'm new to sage and do not yet understand everything)
The crash was quite old (5 years ago) and presumably happened because of an old Singular version. This triggered an unpickling crash. Pickling is the Python name for serialization, see https://en.wikipedia.org/wiki/Serialization
I admit the doctest tests the unpickling issue, i.e., the symptom not the cause. OTOH it has the potential to catch a lot of other bugs. The bug itself was probably fixed by a newer Singular version.
comment:13 Changed 7 years ago by
Q: should the tag 'long time' be added to the test?
comment:14 Changed 7 years ago by
You must certainly remove the # not tested
, otherwise you are just testing once, not twice...
comment:15 Changed 7 years ago by
 Status changed from needs_review to needs_work
comment:16 Changed 7 years ago by
 Commit changed from 1b91e8a85a560b02f9feebadea713a79a6b3ebfa to f102b9647c15597fafb081e3cff3f4c56822de6c
Branch pushed to git repo; I updated commit sha1. New commits:
f102b96  5838: fix previous doctest

comment:18 Changed 7 years ago by
 Branch changed from u/rws/crash_in_singular_s_polynomial_gcd to u/pbruin/5838double_unpickling
 Commit changed from f102b9647c15597fafb081e3cff3f4c56822de6c to 87d608a0d1cb1dc78186e89e46c3c651b6711090
 Reviewers set to Peter Bruin
The new commit moves the new doctest after the existing one testing the plain unpickle_all()
command, for clarity and to save one run of unpickle_all()
. Setting the location of the pickle jar (std
) probably had to be done by hand in earlier versions, but now one can just omit the dir
argument. I also made the explanation slightly more explicit. If you agree with the changes, you can set it to positive review; if not, you can just put back your branch.
comment:19 Changed 7 years ago by
 Status changed from needs_review to positive_review
Yes, much clearer, thanks.
comment:20 Changed 7 years ago by
 Branch changed from u/pbruin/5838double_unpickling to 87d608a0d1cb1dc78186e89e46c3c651b6711090
 Resolution set to fixed
 Status changed from positive_review to closed
comment:21 Changed 7 years ago by
 Commit 87d608a0d1cb1dc78186e89e46c3c651b6711090 deleted
Assuming, I understand everything correctly..
@pbruin
The commit http://git.sagemath.org/sage.git/commit/?id=87d608a0d1cb1dc78186e89e46c3c651b6711090
does not what Ralph Stephan intended, namely unpickle the 'pickle_jar.tar.bz2'
@rws did you not notice that?
comment:22 Changed 7 years ago by
Hm, I thought the double unpickling was the issue tested but now I'm not sure anymore...
comment:23 Changed 7 years ago by
I didn't understand everything correctly. Unpickling of pickle_jar.tar.bz2 _is_ checked by the doctest and everyhing is fine.
comment:24 Changed 7 years ago by
Hello,
this test fails on a dated gentoo(?) install with error
********************************************************************** File "src/sage/structure/sage_object.pyx", line 1411, in sage.structure.sage_object.unpickle_all Failed example: sage.structure.sage_object.unpickle_all() # (4s on sage.math, 2011) Expected: doctest:... DeprecationWarning: This class is replaced by Matrix_modn_dense_float/Matrix_modn_dense_double. See http://trac.sagemath.org/4260 for details. Successfully unpickled ... objects. Failed to unpickle 0 objects. Got: *** The old options style (`tar fx ...`) is highly discouraged, please use the modern form (leading dash before options). *** The standard way (with expanded options; you can still bundle options) for your command would be: tar f  x doctest:1: DeprecationWarning: OrderedAlphabet is deprecated; use Alphabet instead. See http://trac.sagemath.org/8920 for details. doctest:843: DeprecationWarning: This class is replaced by Matrix_modn_dense_float/Matrix_modn_dense_double. See http://trac.sagemath.org/4260 for details. Successfully unpickled 586 objects. Failed to unpickle 0 objects.
any idea how to fix it ?
comment:25 followups: ↓ 26 ↓ 29 Changed 5 years ago by
The second attempt to unpickle all crashes with both GMP 5.1.3, the current version in Sage, and GMP 6.1.1, in exactly the same manner as in the ticket description, with GCC 5.4.0, 6.1.0, and also Sage's GCC 4.9.3 (on Linux x86_64 at least).
Sage version 7.3.rc0 (NTL meanwhile at 9.8.1.p1, Singular still 3.1.7p1.p1).
(Presumably just incidentally) doesn't happen with Sage's MPIR, currently 2.7.2.
comment:26 in reply to: ↑ 25 ; followup: ↓ 28 Changed 5 years ago by
Replying to leif:
The second attempt to unpickle all crashes with both GMP 5.1.3, the current version in Sage, and GMP 6.1.1, in exactly the same manner as in the ticket description, with GCC 5.4.0, 6.1.0, and also Sage's GCC 4.9.3 (on Linux x86_64 at least).
Sage version 7.3.rc0 (NTL meanwhile at 9.8.1.p1, Singular still 3.1.7p1.p1).
(Presumably just incidentally) doesn't happen with Sage's MPIR, currently 2.7.2.
And the winners (apparently) are:
_class__sage_combinat_sf_macdonald_MacdonaldPolynomials_*.sobj
(When I skip these six, I can successfully unpickle_all()
as often as I want in a row.)
comment:27 Changed 5 years ago by
Don't know yet who's to blame (on Sage's side, perhaps Singular's), but I can pretty well imagine what's happening here; the only difference between MPIR and GMP here being that the latter reuses userfreed memory regions earlier than the former does.
Will investigate later, haven't looked at the code yet.
comment:28 in reply to: ↑ 26 Changed 5 years ago by
Replying to leif:
And the winners (apparently) are:
_class__sage_combinat_sf_macdonald_MacdonaldPolynomials_*.sobj
(When I skip these six, I can successfully
unpickle_all()
as often as I want in a row.)
To no surprise, these objects / classes have been added shortly before Carl Witty reported the issue (for Sage 3.4.1.rc3):
type(obj) = <class 'sage.combinat.sf.macdonald.MacdonaldPolynomials_ht'> version = 3.4.rc1 obj = 'Macdonald polynomials in the Ht basis over Fraction Field of Multivariate Polynomial Ring in q, t over Rational Field' type(obj) = <class 'sage.combinat.sf.macdonald.MacdonaldPolynomials_h'> version = 3.4.rc1 obj = 'Macdonald polynomials in the H basis over Fraction Field of Multivariate Polynomial Ring in q, t over Rational Field' type(obj) = <class 'sage.combinat.sf.macdonald.MacdonaldPolynomials_j'> version = 3.4.rc1 obj = 'Macdonald polynomials in the J basis over Fraction Field of Multivariate Polynomial Ring in q, t over Rational Field' type(obj) = <class 'sage.combinat.sf.macdonald.MacdonaldPolynomials_p'> version = 3.4.rc1 obj = 'Macdonald polynomials in the P basis over Fraction Field of Multivariate Polynomial Ring in q, t over Rational Field' type(obj) = <class 'sage.combinat.sf.macdonald.MacdonaldPolynomials_q'> version = 3.4.rc1 obj = 'Macdonald polynomials in the Q basis over Fraction Field of Multivariate Polynomial Ring in q, t over Rational Field' type(obj) = <class 'sage.combinat.sf.macdonald.MacdonaldPolynomials_s'> version = 3.4.rc1 obj = 'Macdonald polynomials in the S basis over Fraction Field of Multivariate Polynomial Ring in q, t over Rational Field'
comment:29 in reply to: ↑ 25 Changed 5 years ago by
Replying to leif:
The second attempt to unpickle all crashes with both GMP 5.1.3, the current version in Sage, and GMP 6.1.1, in exactly the same manner as in the ticket description, with GCC 5.4.0, 6.1.0, and also Sage's GCC 4.9.3 (on Linux x86_64 at least).
Sage version 7.3.rc0 (NTL meanwhile at 9.8.1.p1, Singular still 3.1.7p1.p1).
(Presumably just incidentally) doesn't happen with Sage's MPIR, currently 2.7.2.
Indeed, since the above failure incidentally vanishes with #21188. (So far only tried with GMP 5.1.3 and GCC 4.9.3 though.)
I.e., when building Sage / Singular with GMP, a configure
test for FLINT in Singular failed because it used lmpir
, such that Singular and libsingular "silently" got built without using FLINT.
comment:30 Changed 5 years ago by
 Keywords sage_object unpickle_all() NTL::InvMod abort inverse undefined added
comment:31 Changed 5 years ago by
I am seeing this doctest (2nd unpickling) failing on FreeBSD (see #22679).
This might be due to some toolchain issues I have there ATM (I see mysterious errors in sympow, depending upon unrecognised compiler options passed, or not...).
Looking a bit further into it, the actual pickles that trigger the problem are those in
sage.combinat.sf.macdonald
. Namely, if these ones are removed from the pickle jar, then the problem disappears...