Opened 11 years ago
Last modified 9 years ago
#9958 closed enhancement
Upgrade python to 2.7.x — at Version 269
Reported by:  mhampton  Owned by:  tbd 

Priority:  major  Milestone:  sage5.0 
Component:  packages: standard  Keywords:  
Cc:  jhpalmieri, leif, jason, kcrisman, kini  Merged in:  
Authors:  François Bissey, Steven Trogdon, Sébastien Labbé  Reviewers:  John Palmieri, KarlDieter Crisman, Jason Grout, Jeroen Demeyer, Steven Trogdon, Leif Leonhardy, Sébastien Labbé 
Report Upstream:  N/A  Work issues:  Fix upgrading 
Branch:  Commit:  
Dependencies:  #5852, #11986, #12085, #12096, #12124  Stopgaps: 
Description (last modified by )
From the release notes:
Python 2.7.2 was released on June 11th, 2011.
The Python 2.7 series is scheduled to be the last major version in the 2.x series before 2.x moves into an extended maintenance period. The 2.7 series contains many of the features that were first released in Python 3.1. Improvements in this release include:
 An ordered dictionary type
 New unittest features including test skipping, new assert methods, and test discovery
 A much faster io module
 Automatic numbering of fields in the str.format() method
 Float repr improvements backported from 3.x
 Tile support for Tkinter
 A backport of the memoryview object from 3.x
 Set literals
 Set and dictionary comprehensions
 Dictionary views
 New syntax for nested with statements
 The sysconfig module
Fixes #1159
New spkg: http://spkgupload.googlecode.com/files/python2.7.2.p0.spkg (diff for reviewing (without deleted files): python2.7.2.p0.diff)
New matplotlib spkg (required): http://sage.math.washington.edu/home/jason/matplotlib1.0.1.p1.spkg. (but see also #11915 to upgrade matplotlib).
Apply:
 trac_9958fixing_numericalnoisepart1_p4.patch
 trac_9958fixing_numericalnoisepart2.patch
 trac_9958fixing_numericalnoisepart3.patch
 trac_9958fixing_numericalnoisepart4.patch
 trac_9958fix_transcendental.patch
 trac_9958gsl_integration.patch
 trac_9958matrix_mod2e_dense.patch
 trac_9958fix_randstate.patch
 trac_9958_fix_rpy.patch
 trac_9958fixing_colorspy.patch
 trac_9958fixlist_index.patch
 trac_9958fixpureAssertError.patch
 trac_9958sage_unittest.patch
 trac_9958mixedfix_p1.patch
 trac_9958fixmisc_cythonpy.patch
 trac_9958e_one_star.patch
 trac_9958symbolic_callable.patch
 trac_9958finite_crystals.patch
 trac_9958fixreal_mpfr.patch
 trac_995832_64bit_messages.patch
 trac_9958_junk_valueerror.patch
 trac_9958nfactor_enumerable_wordrandomness.patch
 trac_9958suffix_treesvariations.patch
Change History (310)
comment:1 followup: ↓ 5 Changed 11 years ago by
comment:2 Changed 11 years ago by
 Cc jhpalmieri added
 Component changed from PLEASE CHANGE to packages
comment:3 Changed 11 years ago by
 Cc leif added
comment:4 Changed 11 years ago by
 Type changed from defect to enhancement
comment:5 in reply to: ↑ 1 Changed 11 years ago by
Replying to mhampton:
First attempt is at: http://sage.math.washington.edu/home/palmieri/SPKG/python2.7.p0.spkg
John (jhpalmieri) reports that this builds but Sage does not on sage.math, and "packages for twisted, zodb, pygments, and numpy don't build correctly."
Apparently the fix for http://bugs.python.org/issue7491 causes some of these problems.
This fix has been included in python2.6.5 onwards. We are using 2.6.5 which includes this fix and everything builds (haven't tried 2.7 yet). The only problem we have is with http://github.com/cschwan/sageongentoo/issues#issue/1 so I think it is unfair to single out this issue as the source of problems.
comment:6 Changed 10 years ago by
Is there still any interest in this upgrade? I wanted to use ordered dictionaries, which require 2.7. Unfortunately, active participation in this ticket is beyond my current capabilities...
comment:7 Changed 10 years ago by
It will happen at least in sageongentoo as we are following the system python. I have a couple of things in our tree to fix problems with python2.6.5 and 2.6.6 we are looking into 2.7 now. I should post any patch to have it working here.
We are not talking about using the new capabilities just porting, I imagine my counterpart in Mandriva does the same.
comment:8 Changed 10 years ago by
 Status changed from new to needs_work
I just attached a patch that is needed for python 2.6.5 and later.
comment:9 Changed 10 years ago by
 Cc jason added
comment:10 Changed 10 years ago by
Build sage4.6.2 (and dependency) against python2.7.1 on OS X. I get a lot of the following
Expected nothing Got: Failure in _test_pickling: Traceback (most recent call last): File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/sitepackages/sage/misc/sage_unittest.py", line 275, in run test_method(tester = tester) File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/sitepackages/sage/misc/sage_unittest.py", line 498, in _test_pickling tester.assertEqual(loads(dumps(self._instance)), self._instance) File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/unittest/case.py", line 493, in assertEqual assertion_func = self._getAssertEqualityFunc(first, second) File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/unittest/case.py", line 476, in _getAssertEqualityFunc asserter = self._type_equality_funcs.get(type(first)) AttributeError: 'InstanceTester' object has no attribute '_type_equality_funcs' 
I patched python with the cpickle patch. Any ideas?
comment:11 Changed 10 years ago by
A lot of "doctest... DeprecationWarning?: ..." lines that were expected are just gone. Which fails the test.
comment:12 Changed 10 years ago by
Lots of numerical and a little bit of formatting noise.
comment:13 Changed 10 years ago by
Ok so now I understand the differences between unittest and unittest2 which is shipped with python2.7. This will require a massive number of nonbackward compatible changes. I am starting to experiment with a few sage components but that promise to be long and boring.
comment:14 Changed 10 years ago by
 Cc kcrisman added
comment:15 Changed 10 years ago by
I have attached a log of sage testall. This is a sageongentoo install a few tests are expected to fail https://github.com/cschwan/sageongentoo/wiki/Knowntestfailures but it should give you an idea of the problems we face.
comment:16 Changed 10 years ago by
 Description modified (diff)
comment:17 Changed 10 years ago by
I attached a log of test failures with 4.7.alpha4 (+ #7377). It is mostly number of decimals and messages. I added PYTHONWARNINGS=default to sageenv so I collected extra messages. The most important one being the deprecations of "sets".
There are 3 tests killed reason currently unknown. And a few that may need special attention.
comment:18 Changed 10 years ago by
A little bit more details, these are curious:
sage t long force_lib devel/sagemain/sage/rings/padics/padic_capped_relative_element.pyx ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/rings/padics/padic_capped_relative_element.pyx", line 2297: sage: hash(R(1)) Expected: 95367431640624 Got: 1977800240
sage t long force_lib devel/sagemain/sage/combinat/words/suffix_trees.py ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/combinat/words/suffix_trees.py", line 1345: sage: t.trie_type_dict() == dict([[(0, W("a")), 4], [(0, W("b")), 3], [(3, W("a")), 2], [(4, W("b")), 5], [(5, W("a")), 1]]) Expected: True Got: False ********************************************************************** sage t long force_lib devel/sagemain/sage/combinat/words/suffix_trees.py ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/combinat/words/suffix_trees.py", line 1345: sage: t.trie_type_dict() == dict([[(0, W("a")), 4], [(0, W("b")), 3], [(3, W("a")), 2], [(4, W("b")), 5], [(5, W("a")), 1]]) Expected: True Got: False **********************************************************************
This one doesn't worry me as much but should be looked at
sage t long force_lib devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 22: sage: it.next() Expected: word: 5645 Got: word: 4564 ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 24: sage: it.next() Expected: word: 4564 Got: word: 5645 **********************************************************************
sage t long force_lib devel/sagemain/sage/structure/parent.pyx ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/structure/parent.pyx", line 634: sage: CCls()._test_eq() Expected: Traceback (most recent call last): ... AssertionError: <class '__main__.CCls'> == None Got nothing
The following were killed:
sage t long force_lib devel/sagemain/sage/rings/homset.py # Killed/crashed sage t long force_lib devel/sagemain/sage/schemes/generic/scheme.py # Killed/crashed
In my original run
sage t long force_lib "devel/sagemain/sage/rings/morphism.pyx"
also got killed but not in a subsequent run after I adopted a small change in sagedoctest to get rid of PYTHONWARNINGS=default in sageenv.
Example of killed test:
sage t long verbose force_lib "devel/sagemain/sage/rings/homset.py" ... Trying: phi = S.hom([b,a])###line 141:_sage_ >>> phi = S.hom([b,a]) Expecting nothing ok Trying: phi == loads(dumps(phi))###line 142:_sage_ >>> phi == loads(dumps(phi)) Expecting: True /usr/lib64/libcsage.so(print_backtrace+0x24)[0x7f83de940534] /usr/lib64/libcsage.so(sigdie+0x1d)[0x7f83de9405cd] /usr/lib64/libcsage.so(sage_signal_handler+0x131)[0x7f83de940741] /lib64/libpthread.so.0(+0xfae0)[0x7f83e2385ae0] /usr/lib64/libsingular.so.3(_Z9id_DeletePP10sip_sidealP9sip_sring+0x53)[0x7f83c5c48383] /usr/lib64/python2.7/sitepackages/sage/libs/singular/groebner_strategy.so(+0x388d)[0x7f83ad27488d] /usr/lib64/libpython2.7.so.1.0(PyDict_Clear+0xfc)[0x7f83e261597c] /usr/lib64/libpython2.7.so.1.0(+0x82a09)[0x7f83e2615a09] /usr/lib64/libpython2.7.so.1.0(+0x112e7e)[0x7f83e26a5e7e] /usr/lib64/libpython2.7.so.1.0(_PyObject_GC_Malloc+0x11c)[0x7f83e26a684c] /usr/lib64/libpython2.7.so.1.0(_PyObject_GC_New+0xd)[0x7f83e26a685d] /usr/lib64/libpython2.7.so.1.0(+0x7fe11)[0x7f83e2612e11] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/python2.7/sitepackages/sage/rings/polynomial/polydict.so(+0x168c3)[0x7f83c8f378c3] /usr/lib64/libpython2.7.so.1.0(+0xa0468)[0x7f83e2633468] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/python2.7/sitepackages/sage/rings/polynomial/polydict.so(+0xa451)[0x7f83c8f2b451] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f83e2670ca7] /usr/lib64/python2.7/libdynload/cPickle.so(+0x5f06)[0x7f83dcb2df06] /usr/lib64/python2.7/libdynload/cPickle.so(+0xaf47)[0x7f83dcb32f47] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/python2.7/sitepackages/sage/structure/sage_object.so(+0x152fc)[0x7f83dc7012fc] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x56bc)[0x7f83e267692c] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f83e26782f2] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a78)[0x7f83e2676ce8] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(+0x71282)[0x7f83e2604282] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/libpython2.7.so.1.0(+0x5a24f)[0x7f83e25ed24f] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x491d)[0x7f83e2675b8d] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x522e)[0x7f83e267649e] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(+0x71282)[0x7f83e2604282] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/libpython2.7.so.1.0(+0x5a24f)[0x7f83e25ed24f] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x491d)[0x7f83e2675b8d] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x522e)[0x7f83e267649e] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(+0x7138b)[0x7f83e260438b] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/libpython2.7.so.1.0(+0x5a24f)[0x7f83e25ed24f] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f83e25de453] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x491d)[0x7f83e2675b8d] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x522e)[0x7f83e267649e] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x522e)[0x7f83e267649e] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x522e)[0x7f83e267649e] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f83e26781dd] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f83e26782f2] /usr/lib64/libpython2.7.so.1.0(+0xff25c)[0x7f83e269225c] /usr/lib64/libpython2.7.so.1.0(PyRun_FileExFlags+0x90)[0x7f83e2693090] /usr/lib64/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0x1ff)[0x7f83e2693c6f] /usr/lib64/libpython2.7.so.1.0(Py_Main+0xb53)[0x7f83e26a4fc3] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f83e201cb6d] /usr/bin/python2.7[0x4008a9]  Unhandled SIGSEGV: A segmentation fault occurred in Sage. This probably occurred because a *compiled* component of Sage has a bug in it and is not properly wrapped with sig_on(), sig_off(). You might want to run Sage under gdb with 'sage gdb' to debug this. Sage will now terminate.  The doctested process was killed by signal 11 [2.5 s]
Changed 10 years ago by
small patch to sagedoctest in sage_script to reenable sage's deprecation warnings. Idea by my friend Steve Trogdon.
Changed 10 years ago by
this form of comparison is removed in python 2.6.5 and later, refreshed using git.
comment:19 Changed 10 years ago by
It looks like the crash problems originates somewhere in libsingular
sage: f = Zmod(6).cover() sage: f.kernel() Principal ideal (6) of Integer Ring sage: R.<x,y> = PolynomialRing(QQ, 2) sage: S.<a,b> = R.quo(x^2 + y^2) sage: phi = S.cover() sage: phi == loads(dumps(phi)) True sage: phi == R.quo(x^2 + y^3).cover() Program received signal SIGSEGV, Segmentation fault. id_Delete (h=0x50687d8, r=0x0) at ideals.cc:127 127 ideals.cc: No such file or directory. in ideals.cc (gdb) bt #0 id_Delete (h=0x50687d8, r=0x0) at ideals.cc:127 #1 0x00007fffb9670b49 in __pyx_pf_4sage_4libs_8singular_17groebner_strategy_16GroebnerStrategy_1__dealloc__ ( o=<value optimized out>) at sage/libs/singular/groebner_strategy.cpp:2570 #2 __pyx_tp_dealloc_4sage_4libs_8singular_17groebner_strategy_GroebnerStrategy (o=<value optimized out>) at sage/libs/singular/groebner_strategy.cpp:3190 #3 0x00007ffff7aa98c7 in PyDict_Clear (op=<value optimized out>) at Objects/dictobject.c:891 #4 0x00007ffff7aa990f in dict_tp_clear (op=<value optimized out>) at Objects/dictobject.c:2088 #5 0x00007ffff7b32f27 in delete_garbage (generation=0) at Modules/gcmodule.c:769 #6 collect (generation=0) at Modules/gcmodule.c:930 #7 0x00007ffff7b33616 in collect_generations (basicsize=<value optimized out>) at Modules/gcmodule.c:996 #8 _PyObject_GC_Malloc (basicsize=<value optimized out>) at Modules/gcmodule.c:1457 #9 0x00007ffff7ac4a96 in PyType_GenericAlloc (type=0x7ffff7d9ad40, nitems=0) at Objects/typeobject.c:744 #10 0x00007ffff7a8d256 in BaseException_new (type=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>) at Objects/exceptions.c:34 #11 0x00007ffff7ac57b5 in type_call (type=0x7ffff7d9ad40, args=0x4ae46d0, kwds=0x0) at Objects/typeobject.c:712 #12 0x00007ffff7a77539 in PyObject_Call (func=0x7ffff7d9ad40, arg=0x4ae46d0, kw=0x0) at Objects/abstract.c:2529 #13 0x00007ffff7b00b45 in PyEval_CallObjectWithKeywords (func=0x7ffff7d9ad40, arg=0x4ae46d0, kw=0x0) at Python/ceval.c:3881 #14 0x00007ffff7b10443 in PyErr_NormalizeException (exc=0x7fffffffa778, val=0x7fffffffa770, tb=0x7fffffffa768) at Python/errors.c:190 #15 0x00007fffe433021c in __Pyx_GetException (type=0x7fffffffa7c0, value=0x7fffffffa7b8, tb=0x7fffffffa7c8) at sage/structure/parent.c:20909 #16 0x00007fffe433e4d1 in __pyx_pf_4sage_9structure_6parent_6Parent_2element_class (__pyx_v_self=0x4aea500, unused=<value optimized out>) at sage/structure/parent.c:4251 #17 0x00007ffff7aac1e2 in PyCFunction_Call (func=0x504ebd8, arg=0x7ffff7f81050, kw=<value optimized out>) at Objects/methodobject.c:90 #18 0x00007ffff7a77539 in PyObject_Call (func=0x504ebd8, arg=0x7ffff7f81050, kw=0x0) at Objects/abstract.c:2529 #19 0x00007ffff7b00b45 in PyEval_CallObjectWithKeywords (func=0x504ebd8, arg=0x7ffff7f81050, kw=0x0) at Python/ceval.c:3881 #20 0x00007ffff7a8b51f in methoddescr_call (descr=<value optimized out>, args=0x7ffff7f81050, kwds=0x0) at Objects/descrobject.c:246 #21 0x00007ffff7a77539 in PyObject_Call (func=0x138e950, arg=0x7ffff7e9c050, kw=0x0) at Objects/abstract.c:2529 #22 0x00007ffff7b05efe in do_call (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4230 #23 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4035 #24 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2665 #25 0x00007ffff7b07b65 in PyEval_EvalCodeEx (co=0x1159e30, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252 #26 0x00007ffff7a99a97 in function_call (func=0x115d5f0, arg=0xfadeb0, kw=0x0) at Objects/funcobject.c:526 #27 0x00007ffff7a77539 in PyObject_Call (func=0x115d5f0, arg=0xfadeb0, kw=0x0) at Objects/abstract.c:2529 #28 0x00007ffff7a77cca in PyObject_CallFunctionObjArgs (callable=0x115d5f0) at Objects/abstract.c:2760 #29 0x00007ffff7ac8a8e in slot_tp_descr_get (self=0x13179d0, obj=0x4aea500, type=0x19e91f0) at Objects/typeobject.c:5621 #30 0x00007ffff7aae433 in _PyObject_GenericGetAttrWithDict (obj=0x4aea500, name=0x13c5a78, dict=0x5068fb0) at Objects/object.c:1432 #31 0x00007ffff7aae4d1 in PyObject_GenericGetAttr (obj=<value optimized out>, name=<value optimized out>) at Objects/object.c:1454 #32 0x00007fffe434ed62 in __pyx_tp_getattro_4sage_9structure_6parent_Parent (o=0x4aea500, n=0x13c5a78) at sage/structure/parent.c:18702 #33 0x00007ffff7aad8f4 in PyObject_GetAttr (v=0x4aea500, name=<value optimized out>) at Objects/object.c:1189 #34 0x00007ffff7afdbab in builtin_hasattr (self=<value optimized out>, args=<value optimized out>) at Python/bltinmodule.c:885 #35 0x00007ffff7aac1a0 in PyCFunction_Call (func=0x7ffff7fae3b0, arg=0x503d680, kw=<value optimized out>) at Objects/methodobject.c:81 #36 0x00007ffff7b05ba3 in call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4012 #37 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2665 #38 0x00007ffff7b05cfa in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4098 #39 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4033 #40 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2665 #41 0x00007ffff7b07b65 in PyEval_EvalCodeEx (co=0x1159e30, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252 #42 0x00007ffff7a99a97 in function_call (func=0x115d5f0, arg=0xfadbe0, kw=0x0) at Objects/funcobject.c:526
comment:20 Changed 10 years ago by
Same two doctests killed after upgrade to alpha5.
comment:21 Changed 10 years ago by
Overeager garbage collection in python2.7.1! Preceding the sequence by
sage: import gc sage: gc.disable()
Makes everything go smoothly.
comment:22 Changed 10 years ago by
 Description modified (diff)
comment:23 followup: ↓ 26 Changed 10 years ago by
trac_9958fix_cmp.patch is perfectly reasonable, since this is the idiom used everywhere else in similar situations. I don't have enough background to judge on the warning patch.
comment:24 Changed 10 years ago by
Put an updated list of tests failing with 4.7.alpha5 and the latest patches from Nicolas M. Thiery. I think the next area do need some work is the problem of garbage collection in polynomial rings that leads to "random" failures in libsingular. See: http://groups.google.com/group/sagedevel/browse_thread/thread/8c165c887d6b9e54
comment:25 Changed 10 years ago by
 Description modified (diff)
Split the deprecation issue in its own ticket in #11244  this is a different patch.
comment:26 in reply to: ↑ 23 Changed 10 years ago by
Replying to nthiery:
trac_9958fix_cmp.patch is perfectly reasonable, since this is the idiom used everywhere else in similar situations. I don't have enough background to judge on the warning patch.
I will put the cmp patch in its own ticket would you review it?
comment:27 Changed 10 years ago by
I have attached some patches to fix all the noise for numerics and formatting changes. What's left should be looked at before patching.
There are several failures because hashes have changed but I cannot say it is allright to just change the doctest:
sage t long force_lib "devel/sagemain/sage/rings/padics/padic_capped_relative_element.pyx" ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/rings/padics/padic_capped_relative_element.pyx", line 2339: sage: hash(R(1)) Expected: 95367431640624 Got: 1977800240 **********************************************************************
sage t long force_lib "devel/sagemain/sage/rings/integer.pyx" ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/rings/integer.pyx", line 2995: sage: n = 920390823904823094890238490238484; n.__hash__() Expected: 6874330978542788722 Got: 2623069716 ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/rings/integer.pyx", line 3010: sage: hash(n) Expected: 9223372036854767616 Got: 8192 ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/rings/integer.pyx", line 3013: sage: hash(n) == hash(int(n)) Expected: True Got: False **********************************************************************
That hash(n) == hash(int(n)) is now false should be looked into carefully.
sage t long force_lib "devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py" ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 20: sage: it.next() Expected: word: 6456 Got: word: 5645 ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 22: sage: it.next() Expected: word: 5645 Got: word: 6456 **********************************************************************
I think that one is OK, just a change of order but a second opinion would be nice.
Finally there are still three tests killed in libsingular's id_delete because a ring has been garbage collected even so there are some precautions in place for it: http://groups.google.com/group/sagedevel/browse_thread/thread/8c165c887d6b9e54
Next we need an updated python spkg so that more people can play and even try the patchbot on it.
comment:28 Changed 10 years ago by
 Description modified (diff)
The replacement of type.cmp by cmp in strata.py is now in its own ticket #11264 as it can be applied independently of a python upgrade.
comment:29 Changed 10 years ago by
I forgot because I build on sageongentoo but I'll post some patch to build against python2.7 a little bit later.
comment:30 Changed 10 years ago by
Unless someone steps in I'll make a new python2.7.1 spkg by the end of the week. I'll have to figure out if the latest change going in for sage4.7 will be needed for that one.
comment:31 Changed 10 years ago by
I won't be able to do that this week.
comment:32 Changed 10 years ago by
 Description modified (diff)
Our lead sageongentoo debugger Martin tracked down the segfault problem to cython, this is now #11339
comment:33 Changed 10 years ago by
I'd like to try this but don't have time to figure out exactly what patches are needed in what order  can someone update the description with this? Also, is it the same spkg that Marshall originally posted that one should try?
comment:34 followup: ↓ 35 Changed 10 years ago by
Also, is it possible to test this by ./sage f something, or would one have to take a fresh tarball and replace the Python spkg?
comment:35 in reply to: ↑ 34 Changed 10 years ago by
Replying to kcrisman:
Also, is it possible to test this by ./sage f something, or would one have to take a fresh tarball and replace the Python spkg?
Hi Karl,
I am mainly developing from sageongentoo at the moment. I need to make a current python2.7.1 spkg to match. I will update the description and give the patch order when there is a spkg (feel free to make one). The patch have initially been made against 4.7.1.alpha0 and the current version works against 4.7.1.alpha1. You would get failures against a 4.7 because #7377 finally landed.
I think the best course of action will be to start from scratch, it may even be useful to host an alternative tarball somewhere because the change in python version will be extremely messy. I don't know if we can do it without breaking sage upgrade or being overly complicated.
In the meantime you could have a look at #11244 (which I need to update for 4.7.1.alpha1 because of #10334).
Francois
comment:36 Changed 10 years ago by
OK if you want to look at #11244 it can be tested against either the latest 4.7 or 4.7.1.alpha1 now.
comment:37 followup: ↓ 38 Changed 10 years ago by
I see. Well, I just want to make sure that it all works fine on older Macs, of course. I definitely don't have time to make an spkg now, though I do have time to do ./sage f and slap a few patches on it :) Just let me know when you get around to it; I don't think there is a huge rush.
I should probably test 4.7.1.alpha1 on my box. I'm amazed at Jeroen's stamina on this, especially with several different versions simultaneously existing.
comment:38 in reply to: ↑ 37 Changed 10 years ago by
Replying to kcrisman:
I see. Well, I just want to make sure that it all works fine on older Macs, of course. I definitely don't have time to make an spkg now, though I do have time to do ./sage f and slap a few patches on it :) Just let me know when you get around to it; I don't think there is a huge rush.
I should probably test 4.7.1.alpha1 on my box. I'm amazed at Jeroen's stamina on this, especially with several different versions simultaneously existing.
That's why I asked about #11244 it is an issue I split from this ticket because it should apply on a sage running python2.6 without side effects. So it could be in place when we come around to the upgrade. Any ideas on #11339 which was recently split from this ticket is also welcome.
comment:39 Changed 10 years ago by
 Description modified (diff)
My friend Steven Trogdon made some quick and dirty spkgs. His early tests indicate that setuptools needs to be updated. This is now #11363 it also revealed that I missed something in misc/cython.py watch out for an updated patch sometimes this weekend. There were also a few things that needed to be done in two other patches which I refreshed yesterday. I have a python2.7.1 spkg ready to be uploaded probably sometimes today. If you want fun before that you can always use Steve's spkg (they are not in a reviewable state) at: http://www.d.umn.edu/~strogdon/sage/
comment:40 Changed 10 years ago by
 Description modified (diff)
Didn't take long. I have been naughty and committed two changes to patches that needed touching after my first commit of python2.7.1 so the log will show 3 entries for just the one SPKG.txt entry. They were minor changes. I dropped a number of patches because they were applied upstream. However in the case of two patches I dropped them because the upstream has changed in a way they don't apply anymore but I don't know if patching is still necessary for the underlying issues. If so new patches will need to be developed. The issues are
 readline on itanium
 multiprocessing on solaris
As promised I will give a patch order. The next to last one needs updating as stated earlier and the last one is really optional. The patches do not have to be taken n that order as they don't overlap.
comment:41 Changed 10 years ago by
I forgot apply the patches on 4.7.1.alpha1, there may be rejection on 4.7.rcX.
comment:42 Changed 10 years ago by
misc/cython.py patch updated for people who want to test it.
comment:43 Changed 10 years ago by
corrected a syntax error in the last patch.
Changed 10 years ago by
Not strictly necessary but this file needed a scrub. (refreshed for syntax errors)
comment:44 Changed 10 years ago by
Had to redo trac_9958build_module_listpy.patch again as I messed up the syntax of one bit.
comment:45 followup: ↓ 50 Changed 10 years ago by
My friend Steve Trogdon made a prepatched sage4.7.1.alpha1 spkg (4.7.1.alpha1.p0 we named it) available to make testing easier, find it with hos own test.log here http://www.d.umn.edu/~strogdon/sage/ It has all the necessary patches from #11244 and this this ticket. We are a bit puzzled by the cause of failure in sage/misc/cython.py, white spaces?
comment:46 followup: ↓ 47 Changed 10 years ago by
I am getting the twisted install fail that jhpalmieri refers to. So my guess is that the Gentoo system has some updated things along those lines that vanilla 4.7.1.alpha1 does not. Is that possible?
comment:47 in reply to: ↑ 46 Changed 10 years ago by
Replying to kcrisman:
I am getting the twisted install fail that jhpalmieri refers to. So my guess is that the Gentoo system has some updated things along those lines that vanilla 4.7.1.alpha1 does not. Is that possible?
Well sageongentoo uses a newer twisted. On the other I and Steve have build twisted in vanilla sage with python2.7.1. Did you get the new setuptools spkg from #11363? It is necessary to build twisted with python2.7.1.
comment:48 Changed 10 years ago by
No, I didn't realize there was an spkg involved with that. I installed that first, and it seems to be working now.
comment:49 Changed 10 years ago by
 Description modified (diff)
comment:50 in reply to: ↑ 45 Changed 10 years ago by
Replying to fbissey:
My friend Steve Trogdon made a prepatched sage4.7.1.alpha1 spkg (4.7.1.alpha1.p0 we named it) available to make testing easier, find it with hos own test.log here http://www.d.umn.edu/~strogdon/sage/ It has all the necessary patches from #11244 and this this ticket.
Is one of these patches needed before doing the Sage spkg itself? I get an immediate failure
gcc o src/convert.os c fPIC I/Users/student/Desktop/sage4.7.1.alpha1/local/include I/Users/student/Desktop/sage4.7.1.alpha1/local/include/python2.6 I/Users/student/Desktop/sage4.7.1.alpha1/local/include/NTL Iinclude src/convert.c In file included from include/memory.h:23, from /Users/student/Desktop/sage4.7.1.alpha1/local/include/pari/pari.h:42, from include/convert.h:12, from src/convert.c:14: include/interrupt.h:32:20: error: Python.h: No such file or directory scons: *** [src/convert.os] Error 1 *** TOUCHING ALL CYTHON (.pyx) FILES *** gcc o src/convert.os c fPIC I/Users/student/Desktop/sage4.7.1.alpha1/local/include I/Users/student/Desktop/sage4.7.1.alpha1/local/include/python2.6 I/Users/student/Desktop/sage4.7.1.alpha1/local/include/NTL Iinclude src/convert.c In file included from include/memory.h:23, from /Users/student/Desktop/sage4.7.1.alpha1/local/include/pari/pari.h:42, from include/convert.h:12, from src/convert.c:14: include/interrupt.h:32:20: error: Python.h: No such file or directory scons: *** [src/convert.os] Error 1  sage: Building and installing modified Sage library files. Installing c_lib gcc o src/convert.os c fPIC I/Users/student/Desktop/sage4.7.1.alpha1/local/include I/Users/student/Desktop/sage4.7.1.alpha1/local/include/python2.6 I/Users/student/Desktop/sage4.7.1.alpha1/local/include/NTL Iinclude src/convert.c In file included from include/memory.h:23, from /Users/student/Desktop/sage4.7.1.alpha1/local/include/pari/pari.h:42, from include/convert.h:12, from src/convert.c:14: include/interrupt.h:32:20: error: Python.h: No such file or directory scons: *** [src/convert.os] Error 1 ERROR: There was an error building c_lib.
which I assume is related to one of the patches involved here, though I couldn't say which one...
comment:51 Changed 10 years ago by
Did you rebuild cython after the python upgrade? In fact I have now added a list of package that needs to be rebuild after the python upgrade. Not all of them are necessary for building sage (scipy could be done after) but cython, numpy, matplotlib and pynac really need to be rebuilt.
Once you have done that apply all the patches with build in their names:
 trac_9958build_setuppy.patch
 trac_9958buildsage_clib.patch
 trac_9958build_misc_cythonpy.patch
 trac_9958build_module_listpy.patch (actually optional)
That should help sage build but will not correct many of the doctests. To fix most of the doctests apply the patches in #11244 as well as the one in the description of this bug. I have made a list now, have you seen it?
comment:52 followup: ↓ 53 Changed 10 years ago by
Well, I'm building from scratch, so that shouldn't affect it from that side.
Are you saying that before I build the Sage library, I have to apply these patches? That's a little tricky in a scratch build (for me, at least).
Yes, I saw the lists, but I thought they were only relevant for an upgrade/after actually building Sage. My apologies. I may not be able to finish testing this properly for a while now.
comment:53 in reply to: ↑ 52 Changed 10 years ago by
Replying to kcrisman:
Well, I'm building from scratch, so that shouldn't affect it from that side.
Are you saying that before I build the Sage library, I have to apply these patches? That's a little tricky in a scratch build (for me, at least).
Yes, I saw the lists, but I thought they were only relevant for an upgrade/after actually building Sage. My apologies. I may not be able to finish testing this properly for a while now.
That's all right there are still things to sort out. But you could use the sage spkg prepared by Steve with all patches included rather than patch yourself: http://www.d.umn.edu/~strogdon/sage/sage4.7.1.alpha1.p0.spkg
And yes it is tricky from scratch unless you have a premade spkg.
comment:54 followup: ↓ 55 Changed 10 years ago by
I may have made mistakes, but I tried to do this, starting with sage4.7.rc4:
 install the new python spkg
 install the new setuptools spkg (from #11363)
 apply the patches from #11244 to the main Sage library
 apply the patches from this ticket with "build" in their name  some of these didn't apply cleanly, so I had to work around that.
I used that to build a new version of Sage (using "sage sdist ..."). The tar file is here:
On sage.math, I've been able to untar this and build Sage, although tests haven't run yet. You can also try upgrading, although I haven't tested this at all: run
$ sage upgrade http://sage.math.washington.edu/home/palmieri/misc/python2.7/sage4.7.rc4python2.7/
comment:55 in reply to: ↑ 54 Changed 10 years ago by
Replying to jhpalmieri:
I may have made mistakes, but I tried to do this, starting with sage4.7.rc4:
 install the new python spkg
 install the new setuptools spkg (from #11363)
 apply the patches from #11244 to the main Sage library
 apply the patches from this ticket with "build" in their name  some of these didn't apply cleanly, so I had to work around that.
I used that to build a new version of Sage (using "sage sdist ..."). The tar file is here:
On sage.math, I've been able to untar this and build Sage, although tests haven't run yet. You can also try upgrading, although I haven't tested this at all: run
$ sage upgrade http://sage.math.washington.edu/home/palmieri/misc/python2.7/sage4.7.rc4python2.7/
I have tried to keep abreast of everything so the patches are based on 4.7.1.alpha1 there are enough differences that a lot of patches for 4.7 would need rebasing.
So I decided to go straight for the bleeding edge. I am doing some more work on the "build" patches, I am trying to do something smarter with the python version number so some change could be applied independently of this ticket. The module_list,py patch definitely won't apply cleanly on 4.7.rc mainly because of #10334. What's more #11264 and #11236 which are needed here are merged in 4.7.1.alpha0.
That's all a bit of hassle to start from 4.7.1.alpha, but I would have to start again if I was starting from 4.7.rc.
Changed 10 years ago by
let's not forget the hardcoded version of python in SConstruct for sage_clib
comment:56 Changed 10 years ago by
I have updated trac_9958buildsage_clib.patch it could now be splited in its own ticket. It enables sageclib to be smarter about the python version used. It would figure it on its own. I am looking at doing the same for setup.py, anyone would review that in its own ticket? Once in, we could at least build sage against python2.6 or 2.7 without changes.
Changed 10 years ago by
change needed in setup.py made it python version smart. I also scrubbed the debian stuff
comment:57 Changed 10 years ago by
OK I also made setup.py python version smart any volunteer to review a separate ticket?
comment:58 Changed 10 years ago by
g++ o libcsage.dylib single_module flat_namespace undefined dynamic_lookup dynamiclib src/convert.os src/interrupt.os src/memory.os src/mpn_pylong.os src/mpz_pylong.os src/mpz_longlong.os src/stdsage.os src/gmp_globals.os src/ZZ_pylong.os src/ntl_wrap.os L/Users/student/Desktop/sage4.7.1.alpha1/local/lib L/Users/student/Desktop/sage4.7.1.alpha1/local/lib/python/config lntl lpari lgmp lpython2.7 Traceback (most recent call last): File "setup.py", line 18, in <module> from module_list import ext_modules File "/Users/student/Desktop/sage4.7.1.alpha1/devel/sagemain/module_list.py", line 1529, in <module> flint_depends], TypeError: cannot concatenate 'str' and 'list' objects sage: There was an error installing modified sage library code. real 0m34.577s user 0m19.519s sys 0m6.901s Error building new version of SAGE. You might try typing 'sage ba' or write to sagesupport with as much information as possible. real 2m42.066s user 0m52.903s sys 0m25.133s sage: An error occurred while installing sage4.7.1.alpha1.p0
Note this is the p0 package referred to above.
comment:59 followup: ↓ 61 Changed 10 years ago by
Well, there was a problem with this but it was corrected. I've checked the present sage4.7.1.alpha.p0 and this problem should not be there. The md5sum (md5sum sage4.7.1.alpha1.p0.spkg) should be
b54da7bee966eaf15942964a05a8b6b1 sage4.7.1.alpha1.p0.spkg
If the spkg you have is not this could you download again and see if that fixed things.
Changed 10 years ago by
Some minor changes. The biggest change is to the doctest which explicitly refereed to python2.6 also we have extra include folders in a doctest apparently. [updated 20110525]
comment:60 Changed 10 years ago by
 Description modified (diff)
I am splitting the ticket again: in #11376 we make sage building system smart and work out the python version by itself. If we get this in the next alpha that will make testing easier as you won't need a patched sage spkg to build against python2.7.1. One will still need to patch the doctests afterwards and any other issues but it will build.
In #11377 I put the clean up of module_list.py this is not necessary for python2.7.1 per see but it would be a huge favour to me if it was going in (it makes work on sageongentoo and sageprefix much easier).
comment:61 in reply to: ↑ 59 ; followup: ↓ 62 Changed 10 years ago by
Replying to strogdon:
Well, there was a problem with this but it was corrected. I've checked the present sage4.7.1.alpha.p0 and this problem should not be there. The md5sum (md5sum sage4.7.1.alpha1.p0.spkg) should be
b54da7bee966eaf15942964a05a8b6b1 sage4.7.1.alpha1.p0.spkg
If the spkg you have is not this could you download again and see if that fixed things.
It didn't seem to fix things (and I did check the md5 sum). I'm now starting a build from scratch, having replaced the python and sage spkgs.
By the way, now that 4.7.1.alpha1 is "official", one may have to upgrade the dropin spkg.
comment:62 in reply to: ↑ 61 Changed 10 years ago by
Replying to kcrisman:
Replying to strogdon:
Well, there was a problem with this but it was corrected. I've checked the present sage4.7.1.alpha.p0 and this problem should not be there. The md5sum (md5sum sage4.7.1.alpha1.p0.spkg) should be b54da7bee966eaf15942964a05a8b6b1 sage4.7.1.alpha1.p0.spkg If the spkg you have is not this could you download again and see if that fixed things.
It didn't seem to fix things (and I did check the md5 sum). I'm now starting a build from scratch, having replaced the python and sage spkgs. By the way, now that 4.7.1.alpha1 is "official", one may have to upgrade the dropin spkg.
OK, for those interested I have a new dropin spkg based on the released 4.7.1.alpha1. This spkg includes the patches from this ticket as well as the patches from tickets #11244, #11373, #11376 and #11377 . I'm presently building Sage to make sure everything is as it should be. If successful, I'll post links to the dropin spkg later today.
comment:63 Changed 10 years ago by
comment:64 Changed 10 years ago by
The following have been uploaded:
http://www.d.umn.edu/~strogdon/sage/install.log4.7.1.alpha1
http://www.d.umn.edu/~strogdon/sage/test.log4.7.1.alpha1
http://www.d.umn.edu/~strogdon/sage/sage4.7.1.alpha1.p0.spkg
http://www.d.umn.edu/~strogdon/sage/sage4.7.1.alpha1.p0.spkg.md5
for comparisons, perhaps just under the wire for alpha2. Note, the dropin sage spkg has the same revision .p0 extension, so I've provided the md5 sum for the spkg.
comment:65 Changed 10 years ago by
 Description modified (diff)
Changed the cython.py patch in the list to accommodate new changes made in #11376.
comment:66 followup: ↓ 67 Changed 10 years ago by
End of a looong traceback while building from scratch on PPC 10.4 with the spkgs dropped in (though not the absolute most recent alpha1.p0 spkg, but that hasn't even been gotten to yet, so it shouldn't matter). Just FYI in case that helps.
File "/Users/student/Desktop/sage4.7.1.alpha1python/local/lib/python/os.py", line 284, in walk if isdir(join(top, name)): File "/Users/student/Desktop/sage4.7.1.alpha1python/local/lib/python/genericpath.py", line 44, in isdir return stat.S_ISDIR(st.st_mode) File "/Users/student/Desktop/sage4.7.1.alpha1python/local/lib/python/stat.py", line 41, in S_ISDIR return S_IFMT(mode) == S_IFDIR RuntimeError: maximum recursion depth exceeded Error installing Zope/Twisted interface. real 0m16.055s user 0m3.349s sys 0m4.671s sage: An error occurred while installing twisted9.0.p2
comment:67 in reply to: ↑ 66 Changed 10 years ago by
Replying to kcrisman:
End of a looong traceback while building from scratch on PPC 10.4 with the spkgs dropped in (though not the absolute most recent alpha1.p0 spkg, but that hasn't even been gotten to yet, so it shouldn't matter). Just FYI in case that helps.
File "/Users/student/Desktop/sage4.7.1.alpha1python/local/lib/python/os.py", line 284, in walk if isdir(join(top, name)): File "/Users/student/Desktop/sage4.7.1.alpha1python/local/lib/python/genericpath.py", line 44, in isdir return stat.S_ISDIR(st.st_mode) File "/Users/student/Desktop/sage4.7.1.alpha1python/local/lib/python/stat.py", line 41, in S_ISDIR return S_IFMT(mode) == S_IFDIR RuntimeError: maximum recursion depth exceeded Error installing Zope/Twisted interface. real 0m16.055s user 0m3.349s sys 0m4.671s sage: An error occurred while installing twisted9.0.p2
Were you using the new setuptools from #11363? If so it may be worth trying a newer twisted from #8741.
comment:68 Changed 10 years ago by
 Dependencies set to #11156 #11236 #11244 #11264 #11339 #11363 #11376
comment:69 Changed 10 years ago by
Updated trac_9958fixmisc_cythonpy.patch to take into account changes in #11376. For some reason the numpy include line appears after the application of the other patches over there. Why it didn't appear before is a mistery.
comment:70 Changed 10 years ago by
By the way, I don't know how feasible this is, but it would be very nice if the new python spkg passed selftests: set SAGE_CHECK to "yes" and install the spkg. Previous versions of Python have failed the selftests, but it's perhaps the only standard spkg for which this is true. With the spkg here, on an OS X box:
344 tests OK. 4 tests failed: test__locale test_ctypes test_distutils test_locale 38 tests skipped: test_aepack test_al test_applesingle test_bsddb test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_dl test_epoll test_gdb test_gdbm test_gl test_imageop test_imgfile test_largefile test_linuxaudiodev test_macos test_macostools test_ossaudiodev test_scriptpackages test_smtpnet test_socketserver test_startfile test_sunaudiodev test_timeout test_tk test_ttk_guionly test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 5 skips unexpected on darwin: test_aepack test_applesingle test_dl test_macos test_scriptpackages make: *** [test] Error 1 An error occurred while testing Python
On sage.math:
346 tests OK. 1 test failed: test_distutils 39 tests skipped: test_aepack test_al test_applesingle test_bsddb test_bsddb185 test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_dbm test_dl test_gdb test_gdbm test_gl test_imageop test_imgfile test_kqueue test_linuxaudiodev test_macos test_macostools test_ossaudiodev test_scriptpackages test_smtpnet test_socketserver test_startfile test_sunaudiodev test_timeout test_tk test_ttk_guionly test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 7 skips unexpected on linux2: test_bsddb test_bsddb3 test_dbm test_gdb test_gdbm test_tk test_ttk_guionly make: *** [test] Error 1 An error occurred while testing Python
(I didn't upgrade the distutils package or anything else, just ran "sage f python2.7.1.spkg", so some of these may go away.)
comment:71 Changed 10 years ago by
 Cc kini added
comment:72 Changed 10 years ago by
python testing is difficult. Even in gentoo the procedure is complicated. I'll have a look at it but I cannot make promise. There may be modules that we shouldn't build in the first place (nothing useful to sage, don't worry).
We have this for example in the ebuild:
if use berkdb; then ewarn "\"bsddb\" module is outofdate and no longer maintained inside devlang/python. It has" ewarn "been additionally removed in Python 3. You should use external, still maintained \"bsddb3\"" ewarn "module provided by devpython/bsddb3 which supports both Python 2 and Python 3." fi
gdb, gdm, dbm and tk require extra things on the host system so they may very well fail without. Actually according to the ebuild distutils and gdm are known to fail.
comment:73 followup: ↓ 76 Changed 10 years ago by
Just for the record:
http://trac.sagemath.org/sage_trac/ticket/8664#comment:33 gives an example on how to automatically rebuild packages that depend on a newly added spkg (here: Python).
(Artificial version bumps aren't necessary, and simply touch
ing spkg/installed/<spkgthatothersdependon>
alone doesn't work.)
In general:
 Copy the new spkg(s) to
$SAGE_ROOT/spkg/standard/
. export SAGE_UPGRADING=yes
 (Optionally set
MAKE
andSAGE_PARALLEL_SPKG_BUILD
, perhapsSAGE_CHECK
.)  Run
make
(or bettermake build
) rather than doing./sage f ...
 Apply patches to the Sage library.
 Run
./sage b
(or./sage baforce
in case not all dependencies are covered bymodule_list.py
, which in general I'd consider a bug, but this might be necessary upon a Python upgrade)  Run tests or whatever.
Note that Sage switches to the "main" branch whenever the Sage spkg gets reinstalled.
Of course patches to the build system are usually a bit trickier, but as far as I can see Francois already sorted these out.
comment:74 Changed 10 years ago by
Thanks for the pointer, it will make testing easier for everyone. The bad news is that all the bits that makes testing easier won't be in 4.7.1. The good news is that everything is merged in 4.7.2.alpha0 at the moment. So once the next release cycle begins things will get easier. No patch will be needed to the build system. All that will remain will be doctest fix and issues like #11339.
I will update trac_9958fixing_numericalnoisepart2.patch shortly as #11255 broke it. Which means an extra patch may appear later once I have put all the pieces back together.
comment:75 Changed 10 years ago by
OK. There are a few more patches needed for numerical and formatting noise in 4.7.2.alpha1 but now anyone should be able to upgrade python without a premade sage spkg. All fixes to be able to build sage against python2.7 out of the box are in!
comment:76 in reply to: ↑ 73 ; followup: ↓ 77 Changed 10 years ago by
(Artificial version bumps aren't necessary, and simply
touch
ingspkg/installed/<spkgthatothersdependon>
alone doesn't work.)In general:
 Copy the new spkg(s) to
$SAGE_ROOT/spkg/standard/
.export SAGE_UPGRADING=yes
Huh, I didn't know about that, and it would be quite useful. I can't find it in either the installation guide or the developer guide, though  is this documented yet? Just curious.
Francois, I'll try to give this a test run on PPC OS X and/or Cygwin if you like.
comment:77 in reply to: ↑ 76 Changed 10 years ago by
Replying to kcrisman:
Francois, I'll try to give this a test run on PPC OS X and/or Cygwin if you like.
Be my guest, I thought that the fact things just became easier warranted an announcement.
comment:78 Changed 10 years ago by
Wonderful, this is great news! :) Thanks!
comment:79 Changed 10 years ago by
So far Mac PPC is doing fine (I just swapped out the spkgs in a source tar). Numpy, mercurial, various other pythonrelated spkgs apparently built ok, though it isn't at Scipy or Sage yet. I did NOT check with SAGE_CHECK=yes
since I didn't want to have to watch the build and turn that off during the Python build... Anyway, looks good so far. If it finishes I'll apply all the patches :)
comment:80 Changed 10 years ago by
I tried this (new python spkg plus all of the patches, then built from scratch) and I'm seeing a bunch of doctest failures. Are some of these due to #11339?
The following tests failed: sage t long force_lib devel/sage/sage/categories/finite_crystals.py # 1 doctests failed sage t long force_lib devel/sage/sage/rings/morphism.pyx # 0 doctests failed sage t long force_lib devel/sage/sage/rings/integer.pyx # 3 doctests failed sage t long force_lib devel/sage/sage/rings/homset.py # 0 doctests failed sage t long force_lib devel/sage/sage/rings/padics/padic_capped_relative_element.pyx # 1 doctests failed sage t long force_lib devel/sage/sage/combinat/e_one_star.py # 1 doctests failed sage t long force_lib devel/sage/sage/combinat/words/nfactor_enumerable_word.py # 2 doctests failed sage t long force_lib devel/sage/sage/geometry/polyhedra.py # 1 doctests failed sage t long force_lib devel/sage/sage/schemes/generic/scheme.py # 0 doctests failed sage t long force_lib devel/sage/sage/symbolic/callable.py # 5 doctests failed
I think I did everything right. I put together a Sage distribution with the python spkg and the patches, based on 4.7.2.alpha1:
Upgrade path:
comment:81 Changed 10 years ago by
Any failures from #11339 gives you a killed doctest, so
sage t long force_lib devel/sage/sage/rings/morphism.pyx sage t long force_lib devel/sage/sage/rings/homset.py sage t long force_lib devel/sage/sage/schemes/generic/scheme.py
could all be, a verbose run would help figure it out. By nature #11339 is somewhat random, not everyone see the same failures but all these are familiar.
Just checking the others with my current list.
 sage/categories/finite_crystals.py is a new one introduced in 4.7.2 where the error message has changed slightly when using python2.7.
 sage/rings/integer.pyx is definitely python2.7 related but not patched yet because I am not sure what should be done there.
 sage/rings/padics/padic_capped_relative_element.pyx same thing
 sage/combinat/e_one_star.py I had a patch but it became to fuzzy haven't done a new one.
 sage/combinat/words/nfactor_enumerable_word.py python2.7 two successive results are inverted, I want someone else to look at that one to make sure we can just do the obvious thing.
 sage/symbolic/callable.py new in 4.7.2 change in the error message because we are running python2.7
 sage/geometry/polyhedra.py never seen that one before, can you rerun it and give more info on the failure?
As you can see a few more patches are needed and apart from #11339 which is a big show stopper because of the crashes at least two other tests point to some potential problems that should be looked at.
comment:82 followup: ↓ 88 Changed 10 years ago by
Here are the failures I get on x86:
sage t long force_lib devel/sagemain/sage/libs/cremona/newforms.pyx # 1 doctests failed sage t long force_lib devel/sagemain/sage/misc/sageinspect.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/misc/randstate.pyx # 1 doctests failed sage t long force_lib devel/sagemain/sage/rings/real_mpfr.pyx # 1 doctests failed sage t long force_lib devel/sagemain/sage/stats/intlist.pyx # 1 doctests failed sage t long force_lib devel/sagemain/sage/schemes/hyperelliptic_curves/hyperelliptic_finite_field.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/combinat/e_one_star.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/geometry/polyhedra.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/crypto/mq/mpolynomialsystem.py # Killed/crashed sage t long force_lib devel/sagemain/sage/symbolic/callable.py # 5 doctests failed sage t long force_lib devel/sagemain/sage/functions/transcendental.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/categories/finite_crystals.py # 1 doctests failed
The failures newforms.pyx, real_mpfr.pyx and intlist.pyx are all of the form:
File "/storage/sage/sage4.7.2.alpha1/devel/sagemain/sage/libs/cremona/newforms.pyx", line 53: sage: ECModularSymbol(E) Expected: Traceback (most recent call last): ... OverflowError: long int too large to convert to int Got: Traceback (most recent call last): File "/storage/sage/sage4.7.2.alpha1/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/storage/sage/sage4.7.2.alpha1/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/storage/sage/sage4.7.2.alpha1/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[11]>", line 1, in <module> ECModularSymbol(E)###line 53: sage: ECModularSymbol(E) File "newforms.pyx", line 69, in sage.libs.cremona.newforms.ECModularSymbol.__init__ (sage/libs/cremona/newforms.cpp:1820) a6 = new_bigint(int(E.a6())) OverflowError: Python int too large to convert to C long
These tests do not fail on amd64 because "long int too large to convert to int" is returned as the OverflowError? instead of "Python int too large to convert to C long".
comment:83 Changed 10 years ago by
I had forgotten about these. We need different test results for 32 and 64 bits for these.
comment:84 Changed 10 years ago by
Here's the failure for sage/geometry/polyhedra.py
:
sage t long force_lib "devel/sage/sage/geometry/polyhedra.py" ********************************************************************** File "/mnt/usb1/scratch/palmieri/9958/sage4.7.2.9958/devel/sage/sage/geometry/polyhedra.py", line 4948: sage: ppoints[0] Expected: (1.92296268638e15, 1.92296268638e15) Got: (0.0, 0.0) **********************************************************************
It looks like numerical noise, but is it an acceptable level of accuracy?
comment:85 Changed 10 years ago by
Now I remember that test, it is even in the log sage4.7.alpha5test.log.bz2 I produced a while ago. Strangely enough it doesn't happen anymore on my sageongentoo install. I also have pari2.5.0 and maxima5.25.0 on this install, I suspect one of them may be responsible for the disparition of that failure
comment:86 Changed 10 years ago by
I am wrong! Not maxima or pari, there is a patch for it in trac_9958fixing_numericalnoisepart1.patch. So it looks like that bit may not be applied properly.
comment:87 Changed 10 years ago by
 Description modified (diff)
Added three new patches. I will rework the patches that need adjusting between 32 and 64 bits next.
comment:88 in reply to: ↑ 82 ; followup: ↓ 89 Changed 10 years ago by
Replying to strogdon:
Here are the failures I get on x86:
sage t long force_lib devel/sagemain/sage/libs/cremona/newforms.pyx # 1 doctests failed sage t long force_lib devel/sagemain/sage/misc/sageinspect.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/misc/randstate.pyx # 1 doctests failed sage t long force_lib devel/sagemain/sage/rings/real_mpfr.pyx # 1 doctests failed sage t long force_lib devel/sagemain/sage/stats/intlist.pyx # 1 doctests failed sage t long force_lib devel/sagemain/sage/schemes/hyperelliptic_curves/hyperelliptic_finite_field.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/combinat/e_one_star.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/geometry/polyhedra.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/crypto/mq/mpolynomialsystem.py # Killed/crashed sage t long force_lib devel/sagemain/sage/symbolic/callable.py # 5 doctests failed sage t long force_lib devel/sagemain/sage/functions/transcendental.py # 1 doctests failed sage t long force_lib devel/sagemain/sage/categories/finite_crystals.py # 1 doctests failedThe failures newforms.pyx, real_mpfr.pyx and intlist.pyx are all of the form:
File "/storage/sage/sage4.7.2.alpha1/devel/sagemain/sage/libs/cremona/newforms.pyx", line 53: sage: ECModularSymbol(E) Expected: Traceback (most recent call last): ... OverflowError: long int too large to convert to int Got: Traceback (most recent call last): File "/storage/sage/sage4.7.2.alpha1/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/storage/sage/sage4.7.2.alpha1/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/storage/sage/sage4.7.2.alpha1/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[11]>", line 1, in <module> ECModularSymbol(E)###line 53: sage: ECModularSymbol(E) File "newforms.pyx", line 69, in sage.libs.cremona.newforms.ECModularSymbol.__init__ (sage/libs/cremona/newforms.cpp:1820) a6 = new_bigint(int(E.a6())) OverflowError: Python int too large to convert to C longThese tests do not fail on amd64 because "long int too large to convert to int" is returned as the OverflowError? instead of "Python int too large to convert to C long".
Steve, what about
sage t long force_lib devel/sagemain/sage/rings/polynomial/polynomial_rational_flint.pyx
comment:89 in reply to: ↑ 88 ; followup: ↓ 90 Changed 10 years ago by
comment:90 in reply to: ↑ 89 Changed 10 years ago by
Replying to strogdon:
Replying to fbissey:
Replying to strogdon:
Steve, what about
sage t long force_lib devel/sagemain/sage/rings/polynomial/polynomial_rational_flint.pyxHere on x86 and amd64 the test passes. So, there seems to be some sort of inconsistency, doesn't it?
Yes! I'll fix it but it should be investigated. Gut feeling is on variation in cython definitions.
comment:91 Changed 10 years ago by
 Description modified (diff)
comment:92 Changed 10 years ago by
I've tried to remove the failures known above here. Note that I didn't get the last few patches in time, but I removed those failures as well, assuming they were the same. The timeouts/killed are probably unrelated  my computer is < 1 GHz, so I get a lot of timeouts and I forgot to set SAGE_TIMEOUT_LONG
before I ran the tests  but at any rate I hope this will be useful. After the hurricane I'll start this up again and run the remaining tests.
sage t long "devel/sage/sage/crypto/mq/mpolynomialsystem.py" # Killed/crashed sage t long "devel/sage/sage/functions/transcendental.py" sage t long "devel/sage/sage/groups/generic.py" # Killed/crashed sage t long "devel/sage/sage/interfaces/maxima.py" # Time out sage t long "devel/sage/sage/interfaces/singular.py" # Killed/crashed sage t long "devel/sage/sage/libs/cremona/newforms.pyx" sage t long "devel/sage/sage/libs/ppl.pyx" # Time out sage t long "devel/sage/sage/libs/singular/ring.pyx" # Killed/crashed sage t long "devel/sage/sage/misc/randstate.pyx" sage t long "devel/sage/sage/misc/sageinspect.py" sage t long "devel/sage/sage/rings/multi_power_series_ring_element.py" # Killed/crashed sage t long "devel/sage/sage/rings/polynomial/multi_polynomial_ideal.py" # Killed/crashed sage t long "devel/sage/sage/rings/polynomial/multi_polynomial_libsingular.pyx" # Killed/crashed sage t long "devel/sage/sage/rings/polynomial/polynomial_singular_interface.py" # Killed/crashed sage t long "devel/sage/sage/rings/real_mpfr.pyx" sage t long "devel/sage/sage/rings/tests.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/BSD.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/ell_curve_isogeny.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/ell_field.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/ell_finite_field.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/ell_generic.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/ell_local_data.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/ell_number_field.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/ell_point.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py" # Time out sage t long "devel/sage/sage/schemes/elliptic_curves/gal_reps.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/heegner.py" # Killed/crashed sage t long "devel/sage/sage/schemes/elliptic_curves/period_lattice.py" # Killed/crashed sage t long "devel/sage/sage/schemes/generic/scheme.py" # Killed/crashed sage t long "devel/sage/sage/schemes/plane_conics/con_finite_field.py" # Killed/crashed sage t long "devel/sage/sage/schemes/plane_curves/projective_curve.py" # Killed/crashed sage t long "devel/sage/sage/stats/intlist.pyx" sage t long "devel/sage/sage/structure/sage_object.pyx" # Killed/crashed
comment:93 followup: ↓ 95 Changed 10 years ago by
Here is one example.
sage t long "devel/sage/sage/functions/transcendental.py" ********************************************************************** File "/Users/student/Desktop/sage4.7.2.alpha1/devel/sage/sage/functions/transcendental.py", line 83: sage: w = exponential_integral_1(2,4); w Expected: [0.04890051070806112, 0.0037793524098489067, 0.00036008245216265873, 3.7665622843924751e05] Got: [0.04890051070806112, 0.0037793524098489067, 0.00036008245216265873, 3.766562284392475e05] **********************************************************************
Here is one of the killed ones (not a timeout, I'm surprised):
sage t long "devel/sage/sage/crypto/mq/mpolynomialsystem.py" The doctested process was killed by signal 10 [352.0 s]
Could this be an unrelated 4.7.2.alpha1 issue on my platform?
comment:94 Changed 10 years ago by
I am particularly interested in the killed/crashed tests. And also those for which there is already a patch.
sage t long "devel/sage/sage/libs/cremona/newforms.pyx" sage t long "devel/sage/sage/rings/real_mpfr.pyx" sage t long "devel/sage/sage/stats/intlist.pyx"
are the 32/64 bits message difference already mentioned by Steve.
The timeouts are the regular suspect on ppc if I am not mistaken.
The transcendental.py doctest you are quoting is worrying me. It is patched in trac_9958fixing_numericalnoisepart1.patch but you have a slightly different result again (one less digit on the last number). We may have to consider a different patch.
All killed/crashed doctest are candidates for #11339 but you'll have to run sage gdb on them to be sure.
comment:95 in reply to: ↑ 93 Changed 10 years ago by
Replying to kcrisman:
Here is one of the killed ones (not a timeout, I'm surprised):
sage t long "devel/sage/sage/crypto/mq/mpolynomialsystem.py" The doctested process was killed by signal 10 [352.0 s]
Could this be an unrelated 4.7.2.alpha1 issue on my platform?
Does SIGUSR1
(#10) have a special meaning on Darwin?
comment:96 Changed 10 years ago by
Almost anything with "polynomial" in its name is likely to use singular (and crypto/mq/mpolynomialsystem.py is no exception), signal 10 is apparently a user defined signal so it could be anything.
We'll have to wait for the hurricane to move on before Karl can answer us on any of this points.
Looking back at you list Karl I am also very curious about
sage t long "devel/sage/sage/structure/sage_object.pyx sage t long "devel/sage/sage/misc/sageinspect.py" sage t long "devel/sage/sage/misc/randstate.pyx"
two of these have already been patched (sage_object and randstate) in this ticket so I am quite curious about what's wrong with them.
comment:97 Changed 10 years ago by
Yeah, I'm sorry I couldn't provide more information. The tests only just finished after classes, and I had one hour to copy/paste the above, restart just doing the failed tests, write a lecture, check email, and then turn off and unplug my PPC machine. I just couldn't do any more. I certainly wasn't taking an eMac home on the bike/train with me!
But we are just due for heavy rain and wind, so hopefully Monday I can get back on it and tell you more, save an extended power outage. Much better than an earthquake, I think fbissey would heartily agree. I wish I'd had a chance to run tests after #11339, but I just knew it would take eons. I'd do it on one of my home PPC machines (believe it or not, I have two more) but one I can no longer get !XCode for, and the other is also slow enough it wouldn't be ready by the time we have to unplug it... sigh.
Anyway, I'll keep this ticket on the priority list for the machine I did these tests on.
comment:98 Changed 10 years ago by
So, I did test this with Sage 4.7.2.alpha2 on my "last" 32bit machine, a Pentium4 Prescott (which has SSE3) running Ubuntu 9.04; my other Pentium4 (Northwood, without SSE3 / PNI) recently died, and I won't revive it in the near future.
The good news are: The patches all still apply to 4.7.2.alpha2, though many hunks with partially large offsets, but no fuzz.
The bad news: I expected some numerical noise because of 32bit architecture with (there rather rare) SSE3 (mfpmath=sse
), but also experienced at least one segfault (in sage/rings/homset.py
). I'll have to investigate the rest, for now just:
... sage t long force_lib "devel/sage/sage/geometry/polyhedra.py" ********************************************************************** File "/media/H1TBP6linux2/Sage/sage4.7.2.alpha2gcc4.5.1/devel/sage/sage/geometry/polyhedra.py", line 4948: sage: ppoints[0] Expected: (1.92296268638e15, 1.92296268638e15) Got: (0.0, 0.0) ********************************************************************** 1 items had failures: 1 of 8 in __main__.example_171 ***Test Failed*** 1 failures. For whitespace errors, see the file /home/leif/.sage//tmp/.doctest_polyhedra.py [166.0 s]  The following tests failed: sage t long force_lib "devel/sage/sage/libs/cremona/newforms.pyx" sage t long force_lib "devel/sage/sage/functions/transcendental.py" sage t long force_lib "devel/sage/sage/misc/randstate.pyx" sage t long force_lib "devel/sage/sage/combinat/words/suffix_trees.py" sage t long force_lib "devel/sage/sage/combinat/words/nfactor_enumerable_word.py" sage t long force_lib "devel/sage/sage/rings/morphism.pyx" sage t long force_lib "devel/sage/sage/rings/real_mpfr.pyx" sage t long force_lib "devel/sage/sage/rings/homset.py" sage t long force_lib "devel/sage/sage/stats/intlist.pyx" sage t long force_lib "devel/sage/sage/schemes/generic/scheme.py" sage t long force_lib "devel/sage/sage/geometry/polyhedra.py" Total time for all tests: 26263.8 seconds make: *** [testlong] Error 128 real 439m23.357s user 393m40.644s sys 31m7.485s leif@californication:~/Sage/sage4.7.2.alpha2gcc4.5.1$
(The tail of the log. More to come later.)
Note that I didn't test in parallel, although with some other CPUgreedy process running, but timeouts are unlikely. Vanilla 4.7.2.alpha2 passed all tests in exactly the same setting (and I did rebuild all dependent packages when installing the Python 2.7 spkg, modulo missing extension module dependencies in module_list.py
).
comment:99 followup: ↓ 100 Changed 10 years ago by
Ok, three segfaults (two of them without backtrace / Sage message, for whatever reason), some Python ints too large ("as usual" I guess), some numerical noise, the rest apparently related to dictionaries / hashing(?).
Attaching short logs of rerun tests (all reproducable)...
Changed 10 years ago by
Pentium4 Prescott (SSE3 / PNI, 32bit), GCC 4.5.1, native code, mfpmath=sse
, Ubuntu 9.04 x86
Changed 10 years ago by
Pentium4 Prescott (SSE3 / PNI, 32bit), GCC 4.5.1, native code, mfpmath=sse, Ubuntu 9.04 x86 (The three segfaulting files tested in verbose mode.)
comment:100 in reply to: ↑ 99 Changed 10 years ago by
Replying to leif:
Attaching short logs of rerun tests (all reproducable)...
Done.
Haha, the first segfault is an especially nice one:
... 10 tests in __main__.example_7 6 tests in __main__.example_8 7 tests in __main__.example_9 511 tests in 55 items. 511 passed and 0 failed. Test passed. Segmentation fault [9.8 s]  The following tests failed: sage t long verbose "devel/sage/sage/rings/morphism.pyx" Total time for all tests: 9.8 seconds real 0m10.013s user 0m6.980s sys 0m1.068s
comment:101 followups: ↓ 102 ↓ 106 Changed 10 years ago by
Thanks leif. Nothing worrying there. There is a patch for sage/geometry/polyhedra.py but obviously it doesn't apply properly anymore. transcendantal.py, randstate.pyx and intlist.pyx are all already patched but may need more tweaking, I'd be grateful for the output of these.
sage t long force_lib "devel/sage/sage/combinat/words/nfactor_enumerable_word.py"
is known to fail but is probably a minor adjustment.
sage t long force_lib "devel/sage/sage/combinat/words/suffix_trees.py"
is new. The rest look like #11339 material.
comment:102 in reply to: ↑ 101 Changed 10 years ago by
Replying to fbissey:
Thanks leif. Nothing worrying there. There is a patch for sage/geometry/polyhedra.py but obviously it doesn't apply properly anymore. transcendantal.py, randstate.pyx and intlist.pyx are all already patched but may need more tweaking, I'd be grateful for the output of these.
The failing doctests are all in Doctest_failures_Sage4.7.2.alpha2_Linux_x86_SSE3individual_tests_rerun.log.
[...] The rest look like #11339 material.
I'm just to apply the patches from there; the machine is dead slow, doing a ./sage baforce
atm to exclude missing extension module dependencies, as I've not built from scratch.
comment:103 Changed 10 years ago by
I was too fast. Some of the failures are already reported by Steve. I may try to update the patch set a bit tonight.
comment:104 Changed 10 years ago by
Just experienced some new(?) error when cloning the Sage library for #11339:
... bytecompiling /home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python2.7/sitepackages/sage/misc/copying.py to copying.pyc bytecompiling /home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python2.7/sitepackages/sage/version.py to version.pyc running install_egg_info Removing /home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python2.7/sitepackages/sage0.0.0py2.7.egginfo Writing /home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python2.7/sitepackages/sage0.0.0py2.7.egginfo Exception in thread Thread4 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python/threading.py", line 530, in __bootstrap_inner File "/home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python/threading.py", line 483, in run File "/home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python/multiprocessing/pool.py", line 272, in _handle_workers <type 'exceptions.TypeError'>: 'NoneType' object is not callable Exception in thread Thread1 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python/threading.py", line 530, in __bootstrap_inner File "/home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python/threading.py", line 483, in run File "/home/leif/Sage/sage4.7.2.alpha2gcc4.5.1/local/lib/python/multiprocessing/pool.py", line 272, in _handle_workers <type 'exceptions.TypeError'>: 'NoneType' object is not callable real 1m35.816s user 1m35.794s sys 0m5.648s After cloning, if you change any Sage library files and want to rebuild the html version of the reference manual, use the command sage docbuild reference html (after running 'sage b'). ...
comment:105 Changed 10 years ago by
Waaaaaah!
leif@californication:~/Sage/sage4.7.2.alpha2gcc4.5.1/devel/sage995811339$ ../../sage hg import v ~/Sage/patches/trac_11339_refcount_singular_polynomials.patch applying /home/leif/Sage/patches/trac_11339_refcount_singular_polynomials.patch patching file sage/rings/polynomial/multi_polynomial_libsingular.pxd patching file sage/rings/polynomial/multi_polynomial_libsingular.pyx Hunk #13 FAILED at 727 Hunk #14 FAILED at 741 Hunk #15 FAILED at 763 Hunk #16 FAILED at 773 Hunk #17 FAILED at 825 Hunk #18 succeeded at 910 with fuzz 2 (offset 72 lines). Hunk #19 FAILED at 875 Hunk #20 succeeded at 961 (offset 75 lines). Hunk #21 succeeded at 980 (offset 75 lines). Hunk #22 succeeded at 1006 (offset 75 lines). ...
(The first patch still applies, with offsets.)
I'm not going to fix that...
comment:106 in reply to: ↑ 101 ; followup: ↓ 107 Changed 10 years ago by
Replying to fbissey:
Thanks leif. Nothing worrying there. There is a patch for sage/geometry/polyhedra.py but obviously it doesn't apply properly anymore. transcendantal.py, randstate.pyx and intlist.pyx are all already patched but may need more tweaking, I'd be grateful for the output of these.
I'm sure the polyhedra.py patch was needed at one time but it now (?) appears that
@@ 4946,7 +4946,7 @@ sage: proj = ProjectionFuncStereographic([1.1,1.1,1.1]) sage: ppoints = [proj(vector(x)) for x in cube] sage: ppoints[0]  (0.0, 0.0) + (1.92296268638e15, 1.92296268638e15) """ def __init__(self, projection_point): """
from trac_9958fixing_numericalnoisepart1.patch is no longer needed.
comment:107 in reply to: ↑ 106 Changed 10 years ago by
Replying to strogdon:
Replying to fbissey:
Thanks leif. Nothing worrying there. There is a patch for sage/geometry/polyhedra.py but obviously it doesn't apply properly anymore. transcendantal.py, randstate.pyx and intlist.pyx are all already patched but may need more tweaking, I'd be grateful for the output of these.
I'm sure the polyhedra.py patch was needed at one time but it now (?) appears that
@@ 4946,7 +4946,7 @@ sage: proj = ProjectionFuncStereographic([1.1,1.1,1.1]) sage: ppoints = [proj(vector(x)) for x in cube] sage: ppoints[0]  (0.0, 0.0) + (1.92296268638e15, 1.92296268638e15) """ def __init__(self, projection_point): """from trac_9958fixing_numericalnoisepart1.patch is no longer needed.
You are right I am reading the test results backwards. It now seems to go to 0 properly. One less patch in the patch and one less worry.
comment:108 followup: ↓ 110 Changed 10 years ago by
Okay, you can get a full list of current failure and what they looked like at this link. Some are definitely known above, but others may be unknown. The timeouts are probably just due to this machine (and the Maxima one), but others I don't know whether they are here or #11339.
comment:109 followup: ↓ 115 Changed 10 years ago by
 Description modified (diff)
Updated the first patch for polyhedra.py, split transcendental.py in its own patch. I lowered the precision tested in the last numbers so that results from 32 and 64 bits are not different.
comment:110 in reply to: ↑ 108 Changed 10 years ago by
Replying to kcrisman:
Okay, you can get a full list of current failure and what they looked like at this link. Some are definitely known above, but others may be unknown. The timeouts are probably just due to this machine (and the Maxima one), but others I don't know whether they are here or #11339.
You have some nasty stuff in polyhedra.py
File "/Users/student/Desktop/sage4.7.2.alpha1/devel/sage/sage/geometry/polyhedra.py", line 5250: sage: _.show() Exception raised: Traceback (most recent call last): File "/Users/student/Desktop/sage4.7.2.alpha1/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/Users/student/Desktop/sage4.7.2.alpha1/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/Users/student/Desktop/sage4.7.2.alpha1/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_183[5]>", line 1, in <module> _.show()###line 5250: sage: _.show() File "/Users/student/Desktop/sage4.7.2.alpha1/local/lib/python/sitepackages/sage/misc/displayhook.py", line 174, in displayhook print_obj(sys.stdout, obj) File "/Users/student/Desktop/sage4.7.2.alpha1/local/lib/python/sitepackages/sage/misc/displayhook.py", line 142, in print_obj print >>out_stream, `obj` File "base.pyx", line 80, in sage.plot.plot3d.base.Graphics3d.__repr__ (sage/plot/plot3d/base.c:2526) File "base.pyx", line 1124, in sage.plot.plot3d.base.Graphics3d.show (sage/plot/plot3d/base.c:10344) File "base.pyx", line 663, in sage.plot.plot3d.base.Graphics3d.export_jmol (sage/plot/plot3d/base.c:5867) File "/Users/student/Desktop/sage4.7.2.alpha1/local/lib/python2.7/zipfile.py", line 1138, in writestr self.fp.write(zinfo.FileHeader()) File "/Users/student/Desktop/sage4.7.2.alpha1/local/lib/python2.7/zipfile.py", line 348, in FileHeader len(filename), len(extra)) error: integer out of range for 'H' format code
You get an extra digit in randstate compared to x86 :( [note to self it is in numerical_noise_part4].
sageinspect looks bad :( but not hopeless. The message is just overly long. I will have to postpone the rest for later.
comment:111 Changed 10 years ago by
comment:112 followup: ↓ 114 Changed 10 years ago by
In polyhedra.py it looks like you have a problem with jmol printing 3D images to file. It involves zip, looking at the pyx there is an instance of testing zip. Could you paste the cython code that produced sage/plot/plot3d/base.c:5867 ?
comment:113 Changed 10 years ago by
 Description modified (diff)
Updated the patch set again. split real_mpfr.pyx from the mixedfix patch. The 32/64 bits issues reported is separate from the message that was already patched. Added a patch to fix 32/64 bit messages.
comment:114 in reply to: ↑ 112 Changed 10 years ago by
Replying to fbissey:
In polyhedra.py it looks like you have a problem with jmol printing 3D images to file. It involves zip, looking at the pyx there is an instance of testing zip. Could you paste the cython code that produced sage/plot/plot3d/base.c:5867 ?
/* "sage/plot/plot3d/base.pyx":663 * f.write("isosurface fullylit; pmesh o* fullylit; set antialiasdisplay on;\n") * * render_params.output_archive.writestr('SCRIPT', f.getvalue()) # <<<<<<<<<<<<<< * render_params.output_archive.close() * */ __pyx_t_3 = PyObject_GetAttr(__pyx_v_render_params, __pyx_n_s__output_archive); if (unlikely(!__pyx_t_3)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 663; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_3); __pyx_t_2 = PyObject_GetAttr(__pyx_t_3, __pyx_n_s__writestr); if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 663; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_2); __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0; __pyx_t_3 = PyObject_GetAttr(__pyx_v_f, __pyx_n_s__getvalue); if (unlikely(!__pyx_t_3)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 663; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_3); __pyx_t_4 = PyObject_Call(__pyx_t_3, ((PyObject *)__pyx_empty_tuple), NULL); if (unlikely(!__pyx_t_4)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 663; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_4); __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0; __pyx_t_3 = PyTuple_New(2); if (unlikely(!__pyx_t_3)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 663; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(((PyObject *)__pyx_t_3)); __Pyx_INCREF(((PyObject *)__pyx_n_s__SCRIPT)); PyTuple_SET_ITEM(__pyx_t_3, 0, ((PyObject *)__pyx_n_s__SCRIPT)); __Pyx_GIVEREF(((PyObject *)__pyx_n_s__SCRIPT)); PyTuple_SET_ITEM(__pyx_t_3, 1, __pyx_t_4); __Pyx_GIVEREF(__pyx_t_4); __pyx_t_4 = 0; __pyx_t_4 = PyObject_Call(__pyx_t_2, ((PyObject *)__pyx_t_3), NULL); if (unlikely(!__pyx_t_4)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 663; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_4); __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0; __Pyx_DECREF(((PyObject *)__pyx_t_3)); __pyx_t_3 = 0; __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
5867 is
__pyx_t_4 = PyObject_Call(__pyx_t_2, ((PyObject *)__pyx_t_3), NULL); if (unlikely(!__pyx_t_4)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 663; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
I was also surprised by the polyhedra.py additional weirdness. Anything I can do to help identify it? Warning: I'm planning on building and testing alpha2 without Python 2.7 now, which will tie up most resources on this machine for a couple days.
comment:115 in reply to: ↑ 109 ; followup: ↓ 116 Changed 10 years ago by
Replying to fbissey:
Updated the first patch for polyhedra.py, split transcendental.py in its own patch. I lowered the precision tested in the last numbers so that results from 32 and 64 bits are not different.
Here the fix_transcendental patch fixed the associated test on x86 but now the test fails on amd64:
sage t long force_lib devel/sagemain/sage/functions/transcendental.py ********************************************************************** File "/storage/sage/sage4.7.2.alpha2/devel/sagemain/sage/functions/transcendental.p y", line 83: sage: w = exponential_integral_1(2,4); w Expected: [0.04890051070806112, 0.0037793524098489067, 0.00036008245216265873, 3.7665622843 924...e05] Got: [0.04890051070806112, 0.0037793524098489063, 0.00036008245216265873, 3.7665622843 924534e05] **********************************************************************
This is odd! I wonder what changed? Previously this test must have returned ...67 and not ...63 for it to pass.
comment:116 in reply to: ↑ 115 Changed 10 years ago by
Replying to strogdon:
Replying to fbissey:
Updated the first patch for polyhedra.py, split transcendental.py in its own patch. I lowered the precision tested in the last numbers so that results from 32 and 64 bits are not different.
Here the fix_transcendental patch fixed the associated test on x86 but now the test fails on amd64:
sage t long force_lib devel/sagemain/sage/functions/transcendental.py ********************************************************************** File "/storage/sage/sage4.7.2.alpha2/devel/sagemain/sage/functions/transcendental.p y", line 83: sage: w = exponential_integral_1(2,4); w Expected: [0.04890051070806112, 0.0037793524098489067, 0.00036008245216265873, 3.7665622843 924...e05] Got: [0.04890051070806112, 0.0037793524098489063, 0.00036008245216265873, 3.7665622843 924534e05] **********************************************************************This is odd! I wonder what changed? Previously this test must have returned ...67 and not ...63 for it to pass.
It's my fault for not noticing this difference between 32 and 64 bit results. It was there before but I didn't notice it. I thought the last number was the only thing different so when I truncated the last number to the common digits I dropped the difference between 32 and 64 bit. So it is entirely my fault.
Changed 10 years ago by
fixing numerical noise in transcendental.py split from numerical noise pt1  updated 2011092
comment:117 followup: ↓ 118 Changed 10 years ago by
transcendental patch updated.
comment:118 in reply to: ↑ 117 ; followup: ↓ 119 Changed 10 years ago by
comment:119 in reply to: ↑ 118 Changed 10 years ago by
comment:120 followup: ↓ 121 Changed 10 years ago by
Just for reference, on the PPC machine all tests pass with alpha2 except three timeouts which are longstanding (I didn't bother to set SAGE_TIMEOUT_LONG
but I've seen these many times).
So all the rest are due to this ticket or #11339.
Would it be helpful for me to do #11339 on this new build (without Python 2.7), or is that in limbo right now anyway? Alternately, is there any way to decouple this ticket from that one?
comment:121 in reply to: ↑ 120 Changed 10 years ago by
Replying to kcrisman:
Just for reference, on the PPC machine all tests pass with alpha2 except three timeouts which are longstanding (I didn't bother to set
SAGE_TIMEOUT_LONG
but I've seen these many times).So all the rest are due to this ticket or #11339.
That's a very useful point of reference. Thank you for providing it.
Would it be helpful for me to do #11339 on this new build (without Python 2.7), or is that in limbo right now anyway? Alternately, is there any way to decouple this ticket from that one?
Technically you can apply #11339 without python 2.7  the behavior we want to correct is only revealed when we use python 2.7 but I guess it would be useful to know if there are any side effects if you apply the patch on a python 2.6 install. Bear in mind that the current patch are not necessarily the best solution to the problem, what's more they are a whack a mole game. Fix something here and some new fault appears in a completely unexpected location.
comment:122 Changed 10 years ago by
I will update the python spkg to 2.7.3 when it will be released. 2.7.3 will include the patch to python that we have been carrying in sage for the longest time. See: http://bugs.python.org/issue7689
Does anyone know if the apocalypse is about to happen?
comment:123 Changed 10 years ago by
 Description modified (diff)
Fixes #1159, according to that ticket.
comment:124 Changed 10 years ago by
That's really weird. Some of the hashing problems I encounter with python 2.7 are almost the same as reported in http://trac.sagemath.org/sage_trac/ticket/4957#comment:3
sage t long force_lib "devel/sagemain/sage/rings/integer.pyx" ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/rings/integer.pyx", line 3046: sage: n = 920390823904823094890238490238484; n.__hash__() Expected: 6874330978542788722 Got: 2623069716 ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/rings/integer.pyx", line 3061: sage: hash(n) Expected: 9223372036854767616 Got: 8192 ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/rings/integer.pyx", line 3064: sage: hash(n) == hash(int(n)) Expected: True Got: False **********************************************************************
comment:125 Changed 10 years ago by
With python2.6:
sage: n=2^63+2^13 sage: n 9223372036854784000 sage: int(n) 9223372036854784000L sage: hash(n) 9223372036854767616 sage: hash(int(n)) 9223372036854767616
with python2.7
sage: n=2^63+2^13 sage: n 9223372036854784000 sage: hash(n) 8192 sage: int(n) 9223372036854784000L sage: hash(int(n)) 9223372036854767616
Furthermore in both case:
sage: type(n) <type 'sage.rings.integer.Integer'> sage: type(int(n)) <type 'long'>
So I would expect different hashes naively.
comment:126 Changed 10 years ago by
Regarding
sage t long force_lib "devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py" ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 20: sage: it.next() Expected: word: 6456 Got: word: 5645 ********************************************************************** File "/usr/share/sage/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 22: sage: it.next() Expected: word: 5645 Got: word: 6456 **********************************************************************
With python2.7
sage: w = Word([4,5,6])^7 sage: w word: 456456456456456456456 sage: it = w.factor_iterator(4) sage: it.next() word: 5645 sage: it.next() word: 6456 sage: it.next() word: 4564
with python2.6
sage: w = Word([4,5,6])^7 sage: w word: 456456456456456456456 sage: it = w.factor_iterator(4) sage: it.next() word: 6456 sage: it.next() word: 5645 sage: it.next() word: 4564
So it looks like with python 2.6 we where moving in the series of number with a stride 2 while python 2.7 works through the sequence with a stride 1.
comment:127 Changed 10 years ago by
Here on x86 I get only two failures:
sage t long force_lib devel/sagemain/sage/misc/randstate.pyx ********************************************************************** File "/storage/sage/sage4.7.2.rc0/devel/sagemain/sage/misc/randstate.pyx", line 61: sage: rtest() Expected: (978, 0.184109262667515, 3*x^2  1/12, (4,5), [ 0, 1, 1, 0, 0 ], 1161603091, 603 59, 0.83350776541997362) Got: (978, 0.184109262667515, 3*x^2  1/12, (4,5), [ 0, 1, 1, 0, 0 ], 1161603091, 603 59, 0.8335077654199736) **********************************************************************
and
sage t long force_lib devel/sagemain/sage/matrix/matrix_double_dense.pyx ********************************************************************** File "/storage/sage/sage4.7.2.rc0/devel/sagemain/sage/matrix/matrix_double_dense.py x", line 1046: sage: A.singular_values(eps='junk') Expected: Traceback (most recent call last): ... ValueError: invalid literal for float(): junk Got: Traceback (most recent call last): File "/storage/sage/sage4.7.2.rc0/local/bin/ncadoctest.py", line 1231, in run_ one_test self.run_one_example(test, example, filename, compileflags) File "/storage/sage/sage4.7.2.rc0/local/bin/sagedoctest.py", line 38, in run_o ne_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags ) File "/storage/sage/sage4.7.2.rc0/local/bin/ncadoctest.py", line 1172, in run_ one_example compileflags, 1) in test.globs File "<doctest __main__.example_19[37]>", line 1, in <module> A.singular_values(eps='junk')###line 1046: sage: A.singular_values(eps='junk') File "matrix_double_dense.pyx", line 1075, in sage.matrix.matrix_double_dense.M atrix_double_dense.singular_values (sage/matrix/matrix_double_dense.c:6338) eps = RDF(eps) File "parent.pyx", line 988, in sage.structure.parent.Parent.__call__ (sage/str ucture/parent.c:7326) File "coerce_maps.pyx", line 82, in sage.structure.coerce_maps.DefaultConvertMa p_unique._call_ (sage/structure/coerce_maps.c:3268) File "coerce_maps.pyx", line 77, in sage.structure.coerce_maps.DefaultConvertMa p_unique._call_ (sage/structure/coerce_maps.c:3171) File "real_double.pyx", line 548, in sage.rings.real_double.RealDoubleElement._ _init__ (sage/rings/real_double.c:5567) ValueError: could not convert string to float: junk **********************************************************************
The latter also fails here the same way on amd64 and is documented on sagedevel
http://groups.google.com/group/sagedevel/browse_thread/thread/8e1d7f665ac5be37
but with some slight differences.
comment:128 Changed 10 years ago by
OK Steve, I got the matrix_double_dense but you don't get the numerical noise. I believe the noise may come from python2.7.2. Interesting that the hashing and enumeration problem are 64bit only. I'll check on the mac.
comment:129 Changed 10 years ago by
 Description modified (diff)
New patch to fix the message about junk.
comment:130 Changed 10 years ago by
 Dependencies changed from #11156 #11236 #11244 #11264 #11339 #11363 #11376 to #11156 #11236 #11244 #11264 #11339 #11363 #11376 #11986
I have now split the hashing issue in #11986
comment:131 Changed 10 years ago by
 Description modified (diff)
One more patch added to fix combinat/words/nfactor_enumerable_word.py on 64 bit system. I believe is output is equivalent in both case so it is not important.
comment:132 Changed 10 years ago by
 Dependencies changed from #11156 #11236 #11244 #11264 #11339 #11363 #11376 #11986 to #11986
 Description modified (diff)
Removed old dependencies.
comment:133 followup: ↓ 135 Changed 10 years ago by
 Milestone set to sage4.8
The spkg needs to be rebased to #11447.
comment:134 Changed 10 years ago by
Permissions is the spkg should be 0644 (or 0755 for executables), currently many files are 0640 or 0750.
comment:135 in reply to: ↑ 133 ; followup: ↓ 136 Changed 10 years ago by
comment:136 in reply to: ↑ 135 Changed 10 years ago by
Replying to fbissey:
Replying to jdemeyer:
The spkg needs to be rebased to #11447.
I looked at that before making my spkg. I believe the strategy used in #11447 is in python2.7. Someone with a debian system proves me wrong but I think that works out of the box.
Well, we did take the patch from Python upstream, but IIRC there are further changes to spkginstall
, and the new spkg needs the previous one to be mentioned in SPKG.txt
anyway (to pass the merger's sanity check).
comment:137 Changed 10 years ago by
Indeed, there've been a lot of changes inbetween:

spkginstall
diff Nu python2.7.1/spkginstall python2.6.4.p11/spkginstall
old new 5 5 6 6 CUR=`pwd` 7 7 8 if [ "$SAGE_LOCAL" = "" ]; then9 echo "SAGE_LOCAL undefined ... exiting";10 echo"Maybe run 'sage sh'?"11 exit 18 if [ z "$SAGE_LOCAL" ]; then 9 echo >&2 "SAGE_LOCAL undefined ... exiting" 10 echo >&2 "Maybe run 'sage sh'?" 11 exit 1 12 12 fi 13 13 14 14 # PATCH 15 15 16 # Due to issues building Python on Ubuntu 11.04, a file is patched 17 # on these systems. 16 # When building Python 2.6 on Debian and derivatives with multiarch, 17 # system library directories are not correctly setup. This patch 18 # fixes this, with noop on other systems. 19 # The fix requires 'dpkgarchitecture' to be installed there though 20 # (see below). 21 echo "Patching src/setup.py for multiarch." 22 patch p0 < patches/setup.py.multiarch.patch 23 if [ $? ne 0 ]; then 24 echo >&2 "Error patching src/setup.py" 25 exit 1 26 fi 18 27 19 if [ f /etc/issue ] && grep "Ubuntu 11.04" /etc/issue ; then 20 echo "Patching src/Modules/Setup.dist as this is Ubuntu 11.04" 21 patch p0 < patches/Modules.Setup.dist.patch 28 # Due to a python bug with Solaris 29 # see http://bugs.python.org/issue1759169 30 # it is necessary to apply a patch to configure.in 31 # then run autoconf. This not only generates a 32 # new 'configure' script, but some subdirectories too 33 # so these will be copied. 34 35 if [ "x`uname`" = xSunOS ] ; then 36 echo "Applying a revised 'configure' script for Solaris" 37 echo "See http://bugs.python.org/issue1759169" 38 echo "http://trac.sagemath.org/sage_trac/ticket/7867" 39 cp r patches/autom4te.cache patches/configure patches/configure.in src 22 40 if [ $? ne 0 ]; then 23 echo "Error patching src/Modules/Setup.dist" 41 echo >&2 "Failed to apply the Solaris patches needed for" 42 echo >&2 "http://bugs.python.org/issue1759169" 43 echo >&2 "http://trac.sagemath.org/sage_trac/ticket/7867" 24 44 exit 1 25 45 fi 46 echo "Setting HAVE_FD_TRANSFER=0 for Solaris to allow" 47 echo "the python module '_multiprocessing' to build" 48 echo "See: http://trac.sagemath.org/sage_trac/ticket/8440" 49 cp patches/setup.py src 50 if [ $? ne 0 ]; then 51 echo >&2 "Failed to apply the Solaris patch needed for" 52 echo >&2 "http://trac.sagemath.org/sage_trac/ticket/8440" 53 exit 1 54 fi 55 fi 56 57 58 cp patches/locale.py src/Lib/locale.py 59 if [ $? ne 0 ]; then 60 echo >&2 "Error copying patched locale.py" 61 exit 1 26 62 fi 27 63 28 patch p0 < patches/Lib.distutils.command.sdist.patch 64 cp patches/Makefile.pre.in src/Makefile.pre.in 29 65 if [ $? ne 0 ]; then 30 echo "Error copying patched sdist.py"66 echo >&2 "Error copying patched Makefile.pre.in" 31 67 exit 1 32 68 fi 33 69 34 patch p0 < patches/Lib.socket.patch 70 cp patches/sdist.py src/Lib/distutils/command/sdist.py 35 71 if [ $? ne 0 ]; then 36 echo "Error copying patched socket.py"72 echo >&2 "Error copying patched sdist.py" 37 73 exit 1 38 74 fi 39 75 40 patch p0 < patches/dynamic_class_copyreg_py.patch 76 cp patches/socket.py src/Lib/socket.py 41 77 if [ $? ne 0 ]; then 42 echo "Error copying patched pickle.py"78 echo >&2 "Error copying patched socket.py" 43 79 exit 1 44 80 fi 45 81 46 patch p0 < patches/dynamic_class_copyreg_c.patch 82 cp patches/pickle.py src/Lib/pickle.py 47 83 if [ $? ne 0 ]; then 48 echo "Error copying patched cPickle.c"84 echo >&2 "Error copying patched pickle.py" 49 85 exit 1 50 86 fi 51 87 88 cp patches/cPickle.c src/Modules/cPickle.c 89 if [ $? ne 0 ]; then 90 echo >&2 "Error copying patched cPickle.c" 91 exit 1 92 fi 93 94 # Due to a problem with _socket not building on OpenSolaris on x64 95 # see http://bugs.python.org/issue8852 96 # http://trac.sagemath.org/sage_trac/ticket/9041 97 # http://trac.sagemath.org/sage_trac/ticket/9022 98 # Modules/socketmodule.c needs patching. The patch consists of 99 # only checking if things are defined before trying to build with them 100 # so it is safe (and desirable) on any platform. 101 102 cp patches/Modules.socketmodule.c src/Modules/socketmodule.c 103 if [ $? ne 0 ]; then 104 echo >&2 "Error copying patched socketmodule.c" 105 exit 1 106 fi 107 108 # The following patch for fixing broken readline behavior on Itanium 109 # Linux definitely does *not* work on anything else. 110 if [ "`uname m`" = "ia64" a "`uname`" = "Linux" ]; then 111 echo "Updating readline.c for Linux/Itanium" 112 cp patches/readline.cItaniumfix src/Modules/readline.c 113 else 114 # Readline patch: http://bugs.python.org/file14599/python2.6readline.patch 115 # Associated bug: http://bugs.python.org/issue5833 116 # 117 # Committed to Python as r75747 in the py26maint branch, but not 118 # in time for 2.6.4  so we can remove this patch the next time 119 # we update Python in Sage. 120 cp patches/readline.cspacebug src/Modules/readline.c 121 fi 122 123 if [ $? ne 0 ]; then 124 echo >&2 "Error copying patched readline.c" 125 exit 1 126 fi 127 128 52 129 # We are setting LDFLAGS and CPPFLAGS so that we pick up Sage's readline 53 130 LDFLAGS="L$SAGE_LOCAL/lib $LDFLAGS" 54 131 export LDFLAGS … … 62 139 EXTRAFLAGS="withoutpymalloc"; export EXTRAFLAGS 63 140 fi 64 141 142 # Program around weird bug in build process: 143 # Apparently if you have this: 144 # export DISTUTILS_DEBUG=1 145 # in your environment variables, the build craps out. No idea why this 146 # is. 147 #  Yi Qiang 148 # 149 # This bug was fixed in Python, but not yet in Python 2.6.4. So this fix 150 # can be removed the next time we upgrade our version of Python. See 151 # 152 # http://bugs.python.org/issue6954 153 # 154 # for the fix. 155 # 156 unset DISTUTILS_DEBUG 157 158 65 159 cd src 66 160 67 161 touch Include/* … … 93 187 enableunicode=ucs4 94 188 fi 95 189 else 96 ./configure $EXTRAFLAGS prefix="$SAGE_LOCAL" enableunicode=ucs4 CC="$CC $CFLAGS" CXX="$CXX $CXXFLAGS" 190 ./configure $EXTRAFLAGS prefix="$SAGE_LOCAL" enableunicode=ucs4 \ 191 CC="$CC $CFLAGS" CXX="$CXX $CXXFLAGS" 97 192 fi 98 193 99 194 100 195 if [ $? ne 0 ]; then 101 echo "Error configuring Python."196 echo >&2 "Error configuring Python." 102 197 exit 1 103 198 fi 104 199 105 200 $MAKE 106 201 if [ $? ne 0 ]; then 107 echo "Error building Python."202 echo >&2 "Error building Python." 108 203 exit 1 109 204 fi 110 205 111 206 # running 'make install' in parallel is a bad, bad idea 207 # FIXME: The proper way to disable parallel make is to use 208 # $MAKE j1 ... (We rely on GNU make anyway). 112 209 MAKE=make; export MAKE 113 210 # the "i" is crucial, esp. in the case of a major upgrade 114 211 make i install 115 212 if [ $? ne 0 ]; then 116 echo "Error installing Python."213 echo >&2 "Error installing Python." 117 214 exit 1 118 215 fi 119 216 } … … 122 219 # informative error message. This is helps in determining why the 123 220 # configuration, compilation or installation failed. So put this before the 124 221 # build() function. 125 set +e 222 set +e # This is redundant here, but doesn't hurt to keep it... ;) 126 223 127 224 build 128 225 129 cd $SAGE_LOCAL/lib226 cd "$SAGE_LOCAL/lib" 130 227 131 228 # move the python directory if we're upgrading from a version 132 # of sage with python 2. 6133 if [ d python2. 6/sitepackages ]; then134 mv python2. 6/sitepackages python2.7/sitepackagesold229 # of sage with python 2.5 230 if [ d python2.5/sitepackages ]; then 231 mv python2.5/sitepackages python2.6/sitepackagesold 135 232 fi 136 233 137 rm rf python/python2. 7 python/python2.6 python/python2.5 python/python python python2.4 python2.5138 ln s python2. 7python234 rm rf python/python2.6 python/python2.5 python/python python python2.4 python2.5 235 ln s python2.6 python 139 236 if [ $? ne 0 ]; then 140 echo "Error creating symbolic link"237 echo >&2 "Error creating symbolic link" 141 238 exit 1 142 239 fi 143 240 144 241 # Sleeping for three seconds so that parallel 'make install' catches up 145 242 # with the following test. 243 # XXX We have disabled parallel 'make install' above, but never mind. 146 244 echo "Sleeping for three seconds before testing python" 147 245 sleep 3 148 246 … … 150 248 # All these modules are important and if any one 151 249 # fails to build, Sage will not work. 152 250 251 import_errors=false 153 252 for module in math hashlib crypt ; do 154 253 "$SAGE_LOCAL/bin/python" c "import $module" 155 254 if [ $? eq 0 ] ; then 156 255 echo "$module module imported OK" 157 256 else 158 echo "$module module failed to import" 159 exit 1 257 echo >&2 "$module module failed to import" 258 import_errors=true 259 # exit 1 # not yet 160 260 fi 161 261 done 262 263 if $import_errors; then 264 echo >&2 "Error: One or more modules failed to import." 265 # Check if we are on Debian or one of its derivatives: 266 if command v dpkg && ! command v dpkgarchitecture >/dev/null; then 267 echo >&2 "You may have to install 'dpkgarchitecture'" 268 echo >&2 "which is part of the Debian package 'dpkgdev'." 269 echo >&2 "Try installing it by typing" 270 echo >&2 " sudo aptget install dpkgdev" 271 echo >&2 "and rerun 'make'." 272 fi 273 exit 1 274 fi 275
comment:138 followup: ↓ 139 Changed 10 years ago by
OK that's not a biggy while I had a look at #11447 I made my spkg before it was closed and possibly before it had reached its final shape. I wanted to wait for 2.7.3 but I will probably cut a 2.7.2 in which I will add this then.
I am keen to have this landing ASAP but I would understand if the release manager only wants to do it in a alpha0. That's both a big change and a big patch set. My concern is that my last patch may not be adequate. Steve tested a number of configurations and the order this comes out are all over the place. I may have to make it random which is not nice.
The other concern is #11986 which looks like a blast from the past and I have no clues on either of them.
comment:139 in reply to: ↑ 138 Changed 10 years ago by
Replying to fbissey:
OK that's not a biggy while I had a look at #11447 I made my spkg before it was closed and possibly before it had reached its final shape. I wanted to wait for 2.7.3 but I will probably cut a 2.7.2 in which I will add this then.
I am keen to have this landing ASAP but I would understand if the release manager only wants to do it in a alpha0. That's both a big change and a big patch set.
If you can finish it, I'd be happy to merge it. But to be realistic, I don't think this ticket is close to being finished. There is a huge amount of patches which needs to be reviewed, there is still #11986, there is the rebasing to #11447, maybe testing will reveal breakage on some systems...
comment:140 Changed 10 years ago by
There is a warning when building the reference manual:
docstring of sage.crypto.boolean_function:3: (WARNING/2) Block quote ends without a blank line; unexpected unindent.
comment:141 Changed 10 years ago by
Another minor comment: please use UTF8 in SPKG.txt
. Currently, "François" is encoded as latin1.
comment:142 Changed 10 years ago by
comment:143 Changed 10 years ago by
 Milestone changed from sage4.8 to sage5.0
comment:144 followup: ↓ 145 Changed 10 years ago by
 Description modified (diff)
 Summary changed from Upgrade python to 2.7 to Upgrade python to 2.7.x
By the way: Python 2.7.2 was released on June 11th, 2011 so you might as well go for that.
comment:145 in reply to: ↑ 144 ; followup: ↓ 146 Changed 10 years ago by
Replying to jdemeyer:
By the way: Python 2.7.2 was released on June 11th, 2011 so you might as well go for that.
sage5.0 is probably more realistic. Thanks for the useful comments, I hadn't seen the warnings in the documentation, this probably could be spun out of this ticket as it is probably a problem in the existing doc  just silent with the current python. I really would like to go to 2.7.3 when it comes out but I'll make a stop 2.7.2. I have a heavy workload this week but I'll try to put the spkg together. I know from sageongentto that there will be some more patching to be done unfortunately although in some case I think that would be just a band aid. So more dependencies may appear :(
I am surprised at the "latin1" encoding I am defaulting to UTF8 and even my Changelogs now come correct everywhere.
We need someone with the know how to look at #11986.
comment:146 in reply to: ↑ 145 Changed 10 years ago by
Replying to fbissey:
Replying to jdemeyer:
By the way: Python 2.7.2 was released on June 11th, 2011 so you might as well go for that.
sage5.0 is probably more realistic.
Especially since William essentially defined sage5.0 as the release which will contain this ticket.
I hadn't seen the warnings in the documentation, this probably could be spun out of this ticket as it is probably a problem in the existing doc  just silent with the current python.
True, but I have no idea how Python 2.7 has changed this particular doctest. It could also be that something changed inside Sphinx or Cython because of Python 2.7.
I am surprised at the "latin1" encoding I am defaulting to UTF8 and even my Changelogs now come correct everywhere.
And now you're punished for the rest of your live as a developer because your name is not ASCII (and please don't write "Francois", that just looks very wrong).
We need someone with the know how to look at #11986.
I might have a look, but I don't want to promise anything. In any case, once #11986 is really the only remaining issue, it will certainly get fixed.
comment:147 Changed 10 years ago by
 Description modified (diff)
All right. Very busy week so far. I am removing the last patch and will spin it in a separate ticket because Steve did more testing and it appears that the test is somewhat random and was only working everywhere the same with python 2.6 by sheer luck.
Steve had a few interesting suggestions for replacing the test but that's clearly another ticket on which we'll forward people who touched that file for comments.
comment:148 followup: ↓ 151 Changed 10 years ago by
 Description modified (diff)
I finished updating the spkg to 2.7.2 in a moment of freedom. I called it 2.7.2.p0 as it has some patches in it. Hope all the permissions are right this time. And SPKG.txt is UTF8 now.
I will need to update one patch and I think add a new one. At least relative to 4.7.2, I am so bogged down in work that I haven't been working at all on 4.8.0 which may require new patches.
comment:149 Changed 10 years ago by
Patch set updated for python2.7.2.
comment:150 Changed 10 years ago by
 Description modified (diff)
comment:151 in reply to: ↑ 148 ; followup: ↓ 153 Changed 10 years ago by
 Description modified (diff)
Replying to fbissey:
I finished updating the spkg to 2.7.2 in a moment of freedom. I called it 2.7.2.p0 as it has some patches in it. Hope all the permissions are right this time.
Unfortunately, they are not. I fixed this in http://boxen.math.washington.edu/home/jdemeyer/spkg/python2.7.2.p0.spkg
comment:152 Changed 10 years ago by
 Description modified (diff)
comment:153 in reply to: ↑ 151 ; followup: ↓ 154 Changed 10 years ago by
Replying to jdemeyer:
Replying to fbissey:
I finished updating the spkg to 2.7.2 in a moment of freedom. I called it 2.7.2.p0 as it has some patches in it. Hope all the permissions are right this time.
Unfortunately, they are not. I fixed this in http://boxen.math.washington.edu/home/jdemeyer/spkg/python2.7.2.p0.spkg
OK. Was it the src folder and its content? Or did I forget something else in the spkg root folder?
comment:154 in reply to: ↑ 153 ; followup: ↓ 158 Changed 10 years ago by
Replying to fbissey:
OK. Was it the src folder and its content?
Yes. I solved this by doing chmod ugo+rX R .
Or did I forget something else in the spkg root folder?
Also, yes. spkgcheck should be executable.
comment:155 Changed 10 years ago by
This fails to build from scratch with sage4.8.alpha2 on sage.math:
gcc pthread DNDEBUG g O3 Wall Wstrictprototypes L/scratch/jdemeyer/merger/sage5.0/local/lib Parser/acceler.o Parser/grammar1. o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar .o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o  lpthread ldl lutil o Parser/pgen Parser/pgen ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c touch Parser/pgen.stamp gcc pthread c fnostrictaliasing g O2 DNDEBUG g O3 Wall Wstrictprototypes I. IInclude I./Include I/scratch/jdemeyer/mer ger/sage5.0/local/include fPIC DPy_BUILD_CORE o Python/ast.o Python/ast.c gcc pthread c fnostrictaliasing g O2 DNDEBUG g O3 Wall Wstrictprototypes I. IInclude I./Include I/scratch/jdemeyer/mer ger/sage5.0/local/include fPIC DPy_BUILD_CORE o Python/compile.o Python/compile.c gcc pthread c fnostrictaliasing g O2 DNDEBUG g O3 Wall Wstrictprototypes I. IInclude I./Include I/scratch/jdemeyer/mer ger/sage5.0/local/include fPIC DPy_BUILD_CORE o Python/graminit.o Python/graminit.c gcc pthread c fnostrictaliasing g O2 DNDEBUG g O3 Wall Wstrictprototypes I. IInclude I./Include I/scratch/jdemeyer/mer ger/sage5.0/local/include fPIC DPy_BUILD_CORE o Python/symtable.o Python/symtable.c gcc pthread c fnostrictaliasing g O2 DNDEBUG g O3 Wall Wstrictprototypes I. IInclude I./Include I/scratch/jdemeyer/mer ger/sage5.0/local/include fPIC DPy_BUILD_CORE \ DSVNVERSION="\"`LC_ALL=C svnversion .`\"" \ DHGVERSION="\"`LC_ALL=C hg id i .`\"" \ DHGTAG="\"`LC_ALL=C hg id t .`\"" \ DHGBRANCH="\"`LC_ALL=C hg id b .`\"" \ o Modules/getbuildinfo.o ./Modules/getbuildinfo.c You must compile Sage first using 'make' in the Sage root directory. You must compile Sage first using 'make' in the Sage root directory. You must compile Sage first using 'make' in the Sage root directory. <commandline>: warning: missing terminating " character <commandline>:1:7: warning: missing terminating " character <commandline>:2:10: warning: missing terminating " character ./Modules/getbuildinfo.c: In function 'Py_GetBuildInfo': ./Modules/getbuildinfo.c:45: error: missing terminating " character ./Modules/getbuildinfo.c:45: error: expected expression before ')' token ./Modules/getbuildinfo.c:46: error: missing terminating " character ./Modules/getbuildinfo.c:46: error: missing terminating " character ./Modules/getbuildinfo.c:47: error: missing terminating " character ./Modules/getbuildinfo.c:47: error: missing terminating " character ./Modules/getbuildinfo.c:53: error: 'buildinfo' undeclared (first use in this function) ./Modules/getbuildinfo.c:53: error: (Each undeclared identifier is reported only once ./Modules/getbuildinfo.c:53: error: for each function it appears in.) ./Modules/getbuildinfo.c: In function '_Py_hgversion': ./Modules/getbuildinfo.c:72: error: missing terminating " character ./Modules/getbuildinfo.c:72: warning: 'return' with no value, in function returning nonvoid ./Modules/getbuildinfo.c: In function '_Py_hgidentifier': ./Modules/getbuildinfo.c:79: error: missing terminating " character ./Modules/getbuildinfo.c:79: error: expected expression before ';' token ./Modules/getbuildinfo.c:83: error: missing terminating " character ./Modules/getbuildinfo.c:83: error: expected expression before ';' token make[2]: *** [Modules/getbuildinfo.o] Error 1 make[2]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage5.0/spkg/build/python2.7.2.p0/src' Error building Python.
Mhy guess would be that spkginstall
is trying to run Mercurial or something, which obviously fails because Mercurial has not been compiled yet (and it can't since it depends on Python).
comment:156 Changed 10 years ago by
The error above might be fixed by #5852...
comment:157 Changed 10 years ago by
The current spkg installs fine on OS X 10.6 and 10.7, at least when I do "sage f ..." in an existing Sage installation. I haven't tried building from scratch, and I haven't tried doctests or the patches yet. Would it be possible to provide a single large patch in addition to the individual small ones? That would be easier to apply for testing, while the small ones are easier to review.
comment:158 in reply to: ↑ 154 Changed 10 years ago by
Replying to jdemeyer:
Replying to fbissey:
OK. Was it the src folder and its content?
Yes. I solved this by doing
chmod ugo+rX R .
Or did I forget something else in the spkg root folder?
Also, yes. spkgcheck should be executable.
Never used symbolic mode of chmod, always octal. But this looks like it is more effective to get the directory permissions right. Darn spkgcheck.
Thanks for the tips. Sorry for being a pest at times.
I haven't tested the spkg from scratch on OSX I did on linux x86_64 and 4.8.alpha2. I will see what happens on my 10.5 machine later. 2.7.1 worked from scratch there i am sure. That looks very strange, can you link the full python building log?
comment:159 Changed 10 years ago by
A build from scratch on OS X 10.6 worked just fine, by the way.
comment:160 Changed 10 years ago by
There is still the docbuild warning:
docstring of sage.crypto.boolean_function:3: WARNING: Block quote ends without a blank line; unexpected unindent.
I am investigating.
comment:161 Changed 10 years ago by
I absolutely do not understand the docbuild warning but created a ticket at #12085.
comment:162 Changed 10 years ago by
 Dependencies changed from #11986 to #5852, #11986, #12085
comment:163 followup: ↓ 164 Changed 10 years ago by
Just an idea! This is because it wants to execute some mercurial commands to find if there are tags in the source. If we were to set
DHGVERSION DHGTAG DHGBRANCH
To something or even nothing we may not have this problem. A normal run with a system mercurial installed looks like:
x86_64pclinuxgnugcc pthread c fnostrictaliasing O2 pipe march=native fwrapv DNDEBUG I. IInclude I./Include fPIC DPy_BUILD_CORE \ DSVNVERSION="\"`LC_ALL=C svnversion .`\"" \ DHGVERSION="\"`LC_ALL=C hg id i .`\"" \ DHGTAG="\"`LC_ALL=C hg id t .`\"" \ DHGBRANCH="\"`LC_ALL=C hg id b .`\"" \ o Modules/getbuildinfo.o ./Modules/getbuildinfo.c abort: repository . not found! abort: repository . not found! abort: repository . not found!
I will try to see if I can fit that in the spkg.
comment:164 in reply to: ↑ 163 Changed 10 years ago by
comment:165 followup: ↓ 166 Changed 10 years ago by
If you insist but it would be easy to path Makefile.pre.in to prevent this from happening. It is mainly used to get prerelease version info if you checked out python source from hg. It is supposed to fail if you are using a properly released tarball.
comment:166 in reply to: ↑ 165 Changed 10 years ago by
comment:167 Changed 10 years ago by
I may try something more subtle first. Makefile.pre.in has
SVNVERSION= @SVNVERSION@ HGVERSION= @HGVERSION@ HGTAG= @HGTAG@ HGBRANCH= @HGBRANCH@
and
$(CC) c $(PY_CFLAGS) \ DSVNVERSION="\"`LC_ALL=C $(SVNVERSION)`\"" \ DHGVERSION="\"`LC_ALL=C $(HGVERSION)`\"" \ DHGTAG="\"`LC_ALL=C $(HGTAG)`\"" \ DHGBRANCH="\"`LC_ALL=C $(HGBRANCH)`\"" \ o $@ $(srcdir)/Modules/getbuildinfo.c
later so exporting HG* to something like /bin/true on nonwindows OS would presumably solve the problem. But just patching to pass empty strings in the later would probably be safe everywhere.
comment:168 Changed 10 years ago by
The first patch doesn't apply in 4.8.alpha3 because of the changes in #12047. New patches will come some. There may be other surprises.
comment:169 Changed 10 years ago by
 Description modified (diff)
Updated the first patch. Haven't been able to test things to find any new problems yet.
comment:170 Changed 10 years ago by
There are a few test failures on a Pentium 4 32bit Linux system with sage4.8.alpha2 + #9958 + #11986:
sage t long force_lib devel/sage/sage/combinat/words/suffix_trees.py ********************************************************************** File "/home/jdemeyer/sage5.0/devel/sagemain/sage/combinat/words/suffix_trees.py", line 1323: sage: t.trie_type_dict() == dict([[(0, W("a")), 4], [(0, W("b")), 3], [(3, W("a")), 2], [(4, W("b")), 5], [(5, W("a")), 1]]) Expected: True Got: False ********************************************************************** sage t long force_lib devel/sage/sage/misc/randstate.pyx ********************************************************************** File "/home/jdemeyer/sage5.0/devel/sagemain/sage/misc/randstate.pyx", line 61: sage: rtest() Expected: (978, 0.184109262667515, 3*x^2  1/12, (4,5), [ 0, 1, 1, 0, 0 ], 1161603091, 60359, 0.83350776541997362) Got: (978, 0.184109262667515, 3*x^2  1/12, (4,5), [ 0, 1, 1, 0, 0 ], 1161603091, 60359, 0.8335077654199736) ********************************************************************** sage t long force_lib devel/sage/sage/combinat/words/nfactor_enumerable_word.py ********************************************************************** File "/home/jdemeyer/sage5.0/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 20: sage: it.next() Expected: word: 6456 Got: word: 4564 ********************************************************************** File "/home/jdemeyer/sage5.0/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 23: sage: it.next() Expected: word: 5645 Got: word: 6456 ********************************************************************** File "/home/jdemeyer/sage5.0/devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py", line 26: sage: it.next() Expected: word: 4564 Got: word: 5645 **********************************************************************
comment:171 Changed 10 years ago by
I have made a patch for the Integer hashing, see #11986.
comment:172 Changed 10 years ago by
I have seen your patch in #11986 that's a good rewrite of the hashing in question, the 1 bit is particularly tricky. I mentioned it before but we probably need a different approach for combinat/words/nfactor_enumerable_word.py, the sequence is pretty much random depending on arch and other factors it seems. The randstate.pyx must have escaped http://trac.sagemath.org/sage_trac/attachment/ticket/9958/trac_9958fixing_numericalnoisepart4.patch because the test results are identical to some performed later in the same file. I'll correct that shortly.
The suffix_trees.pyx rings a bell but I cannot remember where and when.
comment:173 Changed 10 years ago by
ha http://trac.sagemath.org/sage_trac/ticket/9958#comment:18 is where I had suffix_trees.pyx before. I don't have a 32 bit linux box anymore so it is more difficult. I note they were two you only see one now.
comment:174 Changed 10 years ago by
 Description modified (diff)
comment:175 Changed 10 years ago by
 Dependencies changed from #5852, #11986, #12085 to #5852, #11986, #12085, #12096
comment:176 Changed 10 years ago by
I need a kind soul with 32 bit linux system to explore the suffix tree issue, the doctest is as follow:
sage: from sage.combinat.words.suffix_trees import ImplicitSuffixTree, SuffixTrie sage: W = Words("ab") sage: t = ImplicitSuffixTree(W("aba")) sage: t.trie_type_dict() == dict([[(0, W("a")), 4], [(0, W("b")), 3], [(3, W("a")), 2], [(4, W("b")), 5], [(5, W("a")), 1]]) True
It would be helpful to get the values of "W", "t" and "t.trie_type_dict()" on python2.6 and 2.7. Then we can make a decision on whether the answer we get with python2.7 is equivalent to the current output or not.
comment:177 Changed 10 years ago by
Here (x86) with sage4.8.alpha3 + #9958 + #11986 the suffix_trees test passes:
./sage t long force_lib devel/sage/sage/combinat/words/suffix_trees.py sage t long force_lib "devel/sage/sage/combinat/words/suffix_trees.py" [5.1 s]  All tests passed!
The requested values are:
sage: W Words over Ordered Alphabet ['a', 'b'] sage: t Implicit Suffix Tree of the word: aba sage: t.trie_type_dict() {(4, word: b): 5, (0, word: a): 4, (0, word: b): 3, (5, word: a): 1, (3, word: a): 2}
FWIW, on amd64 the test fails with the same sage4.8.alpha3 + #9958 + #11986
comment:178 Changed 10 years ago by
On sage.math (x86_64 Linux), sage4.8.alpha3 + #12106 (unrelated) + #9958 + #11986 gives errors in two files:
sage t force_lib devel/sage/sage/matrix/matrix_mod2e_dense.pyx ********************************************************************** File "/mnt/usb1/scratch/jdemeyer/merger/sage5.0/devel/sagemain/sage/matrix/matrix_mod2e_dense.pyx", line 790: sage: float(A.density()) Expected: 0.099738... Got: 0.099739 ********************************************************************** File "/mnt/usb1/scratch/jdemeyer/merger/sage5.0/devel/sagemain/sage/matrix/matrix_mod2e_dense.pyx", line 798: sage: float(A.density()) Expected: 0.499759... Got: 0.49976 **********************************************************************
(related ticket: #9562) and
sage t force_lib devel/sage/sage/gsl/integration.pyx ********************************************************************** File "/mnt/usb1/scratch/jdemeyer/merger/sage5.0/devel/sagemain/sage/gsl/integration.pyx", line 110: sage: numerical_integral(x^2, 0, 1, max_points=100) Expected: (0.33333333333333331, 3.7007434154171879e15) Got: (0.3333333333333333, 3.700743415417188e15) ********************************************************************** File "/mnt/usb1/scratch/jdemeyer/merger/sage5.0/devel/sagemain/sage/gsl/integration.pyx", line 115: sage: numerical_integral(sin(x)^3 + sin(x), 0, pi) Expected: (3.333333333333333, 3.7007434154171883e14) Got: (3.333333333333333, 3.700743415417188e14) ********************************************************************** File "/mnt/usb1/scratch/jdemeyer/merger/sage5.0/devel/sagemain/sage/gsl/integration.pyx", line 120: sage: numerical_integral(lambda x: sin(x)^3 + sin(x), 0, pi) Expected: (3.333333333333333, 3.7007434154171883e14) Got: (3.333333333333333, 3.700743415417188e14) ********************************************************************** File "/mnt/usb1/scratch/jdemeyer/merger/sage5.0/devel/sagemain/sage/gsl/integration.pyx", line 131: sage: numerical_integral(f, 0, 1, max_points=200, eps_abs=1e7, eps_rel=1e7, rule=4) Expected: (0.33333333333333331, 3.7007434154171879e15) Got: (0.3333333333333333, 3.700743415417188e15) ********************************************************************** File "/mnt/usb1/scratch/jdemeyer/merger/sage5.0/devel/sagemain/sage/gsl/integration.pyx", line 172: sage: exp(1/x).nintegral(x, 1, 2) # via maxima Expected: (0.50479221787318396, 5.6043194293440752e15, 21, 0) Got: (0.504792217873184, 5.604319429344075e15, 21, 0) **********************************************************************
(related ticket: #12047)
Also, the spkg should be rebased to #12096.
comment:179 Changed 10 years ago by
I get on 32bit i386 Linux with sage4.8.alpha2 + various patches including #9958:
sage: from sage.combinat.words.suffix_trees import ImplicitSuffixTree, SuffixTrie sage: W = Words("ab") sage: t = ImplicitSuffixTree(W("aba")) sage: W Words over Ordered Alphabet ['a', 'b'] sage: t Implicit Suffix Tree of the word: aba sage: t.trie_type_dict() {(4, word: a): 1, (0, word: b): 5, (0, word: a): 3, (5, word: a): 2, (3, word: b): 4}
comment:180 followup: ↓ 181 Changed 10 years ago by
thanks for all that feedback. I knew there would be new stuff. I unfortunately (like all of us) not enough time to do things in a speedy fashion. I am bogged down as well with the sageongentoo version of alpha3 not wanting to play nice with #4260 https://github.com/cschwan/sageongentoo/issues/108 and it annoys me.
comment:181 in reply to: ↑ 180 Changed 10 years ago by
Replying to fbissey:
I am bogged down as well with the sageongentoo version of alpha3 not wanting to play nice with #4260 https://github.com/cschwan/sageongentoo/issues/108 and it annoys me.
Does vanilla Sage build and run on that same machine with the same compiler? Be sure to try a different gcc version, Linbox seems to be sensitive to that.
comment:182 Changed 10 years ago by
On x86 with sage4.8.alpha3 + #9958 + #11986 I have the following failures:
sage t long force_lib devel/sagemain/sage/gsl/integration.pyx ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/gsl/integration.pyx", line 110: sage: numerical_integral(x^2, 0, 1, max_points=100) Expected: (0.33333333333333331, 3.7007434154171879e15) Got: (0.3333333333333333, 3.700743415417188e15) ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/gsl/integration.pyx", line 115: sage: numerical_integral(sin(x)^3 + sin(x), 0, pi) Expected: (3.333333333333333, 3.7007434154171883e14) Got: (3.333333333333333, 3.700743415417188e14) ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/gsl/integration.pyx", line 120: sage: numerical_integral(lambda x: sin(x)^3 + sin(x), 0, pi) Expected: (3.333333333333333, 3.7007434154171883e14) Got: (3.333333333333333, 3.700743415417188e14) ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/gsl/integration.pyx", line 131: sage: numerical_integral(f, 0, 1, max_points=200, eps_abs=1e7, eps_rel=1e7, rule=4) Expected: (0.33333333333333331, 3.7007434154171879e15) Got: (0.3333333333333333, 3.700743415417188e15) ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/gsl/integration.pyx", line 172: sage: exp(1/x).nintegral(x, 1, 2) # via maxima Expected: (0.50479221787318396, 5.6043194293440752e15, 21, 0) Got: (0.504792217873184, 5.604319429344075e15, 21, 0) **********************************************************************
sage t long force_lib devel/sagemain/sage/misc/randstate.pyx ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/misc/randstate.pyx", line 61: sage: rtest() Expected: (978, 0.184109262667515, 3*x^2  1/12, (4,5), [ 0, 1, 1, 0, 0 ], 1161603091, 60359, 0.83350776541997362) Got: (978, 0.184109262667515, 3*x^2  1/12, (4,5), [ 0, 1, 1, 0, 0 ], 1161603091, 60359, 0.8335077654199736) **********************************************************************
sage t long force_lib devel/sagemain/sage/matrix/matrix_mod2e_dense.pyx ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/matrix/matrix_mod2e_dense.pyx", line 790: sage: float(A.density()) Expected: 0.099738... Got: 0.099739 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/matrix/matrix_mod2e_dense.pyx", line 798: sage: float(A.density()) Expected: 0.499759... Got: 0.49976 **********************************************************************
sage t long force_lib devel/sagemain/sage/functions/transcendental.py ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/functions/transcendental.py", line 80: sage: w = exponential_integral_1(2,4); w Expected: [0.04890051070806112, 0.0037793524098489067, 0.00036008245216265873, 3.7665622843924751e05] Got: [0.04890051070806112, 0.0037793524098489067, 0.00036008245216265873, 3.766562284392475e05] **********************************************************************
comment:183 Changed 10 years ago by
I now understand why you removed the transcendental patch Jereon some bits ended in my revised trac_9958fixing_numericalnoisepart1_p2.patch when they shouldn't have. That's what is happening to Steve. I will rectify that.
I removed the gsl bits because of #12047 so that has to be redone.
And I haven't had time to try vanilla yet but it is on my TODO list and I suspect g++ too.
comment:184 Changed 10 years ago by
 Description modified (diff)
updated the first patch and readded the transcendental patch to the list.
comment:185 Changed 10 years ago by
For suffix_tree it looks like your two values don't agree:
{(4, word: a): 1, (0, word: b): 5, (0, word: a): 3, (5, word: a): 2, (3, word: b): 4} {(4, word: b): 5, (0, word: a): 4, (0, word: b): 3, (5, word: a): 1, (3, word: a): 2}
If we add that I don't see it on OS X x86 that looks bad. I also have an x86_64 machine where the value is the same as Jeroen and the test fails. Steve get the right value on the other hand so there is something going there.
comment:186 Changed 10 years ago by
 Description modified (diff)
Added two new patches for gsl integration and m4rie tickets merged in 4.8.alpha3.
comment:188 Changed 10 years ago by
Everything passes against sage4.8.alpha3 on 64bit. Yay!
comment:189 followup: ↓ 190 Changed 10 years ago by
I forgot to add a new bit for randstate.pyx on 32 bit. I'll do that later but I know it is missing. Should we involve someone from combinatronics to get to the bottom of suffix_tree and nfactor_enumerate_word?
comment:190 in reply to: ↑ 189 Changed 10 years ago by
Replying to fbissey:
Should we involve someone from combinatronics to get to the bottom of suffix_tree and nfactor_enumerate_word?
Sounds good, try to send an email to sagedevel.
comment:191 followup: ↓ 193 Changed 10 years ago by
For the sake of completeness, I now have
32bit: one failure
sage t long force_lib devel/sagemain/sage/misc/randstate.pyx
which was mentioned above.
64bit: two failures
sage t long force_lib devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/combinat/words/nfactor_enumerable _word.py", line 20: sage: it.next() Expected: word: 5645 Got: word: 4564 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/combinat/words/nfactor_enumerable _word.py", line 23: sage: it.next() Expected: word: 6456 Got: word: 5645 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/combinat/words/nfactor_enumerable _word.py", line 26: sage: it.next() Expected: word: 4564 Got: word: 6456 ********************************************************************** sage t long force_lib devel/sagemain/sage/combinat/words/suffix_trees.py ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/combinat/words/suffix_trees.py", line 1323: sage: t.trie_type_dict() == dict([[(0, W("a")), 4], [(0, W("b")), 3], [(3, W("a")), 2] , [(4, W("b")), 5], [(5, W("a")), 1]]) Expected: True Got: False **********************************************************************
which have been mentioned previously.
comment:192 Changed 10 years ago by
This ticket made me notice a bug with the Lseries of Eisenstein series: #12124. With that ticket applied, the patch to sage/modular/modform/eis_series.py
here can be removed.
comment:193 in reply to: ↑ 191 Changed 10 years ago by
First of all, I'm not an expert of combinatorics on word. So I'd rather Sebastien Labbe to jump in the discussion. I just trying to help.
64bit: two failures
sage t long force_lib devel/sagemain/sage/combinat/words/nfactor_enumerable_word.py ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/combinat/words/nfactor_enumerable _word.py", line 20: sage: it.next() Expected: word: 5645 Got: word: 4564 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/combinat/words/nfactor_enumerable _word.py", line 23: sage: it.next() Expected: word: 6456 Got: word: 5645 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sagemain/sage/combinat/words/nfactor_enumerable _word.py", line 26: sage: it.next() Expected: word: 4564 Got: word: 6456 **********************************************************************
This first failure is not a problem. The iterator here is extracted from a dict of dict so it is not unexpected to have a random order. Fixing the test of marking it at random should be ok. This test is an explanation for the user, the real feature is tested further in the file.
which have been mentioned previously.
{(4, word: a): 1, (0, word: b): 5, (0, word: a): 3, (5, word: a): 2, (3, word: b): 4} {(4, word: b): 5, (0, word: a): 4, (0, word: b): 3, (5, word: a): 1, (3, word: a): 2}
This one is more tricky: As far as I understand, both result are perfectly legible: the second is the same as the first after applying the following cyclic permutation (3,4,5). The numbering is random. So (again as far as I understand), the problem in only created by the crossplatform nondeterminism of the set/dict data structure. I'm not sure here what is the best fix.
Cheers,
Florent
comment:194 Changed 10 years ago by
Thank you Florent, that is helpful. We kind of worked out that word was OK but we thought we could have a better test. If you think that's ok to mark it random I think we'll do that with some explanation.
The fact that both output in suffix_tree are legit is good to know. We can probably work around that. We may even have a complete patch set by Monday, cross fingers.
comment:195 Changed 10 years ago by
 Description modified (diff)
Upgraded succesfully from sage4.6.2, all tests pass except for the known problematic tests.
comment:196 followup: ↓ 197 Changed 10 years ago by
comment:197 in reply to: ↑ 196 Changed 10 years ago by
Replying to fbissey:
I'll work on the remaining tests tomorrow, I hope.
That would be great! I would love to see this finished in time, such that the next release can really be sage5.0.
Changed 10 years ago by
fixing numerical noise part 1  updated 20111211: separate randstate and eis_modn_series
comment:198 Changed 10 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
updating the patch set with a tentative solution for suffix_trees. It may not be exhaustive of all possible answers though. If we are going for it for the next release I can redo the python spkg to fix the lib/lib64 from #12131.
Putting to need review :)
comment:199 Changed 10 years ago by
Hi,
I just read randomly sagedevel (I was away from sage development for several weeks) and saw that the discussion about upgrading to 2.7 was having problem with some files I am related with (suffix_tree.py nfactor_enumerable_word.py). I still don't understand the dict problem for suffix tree (one minute was not enough). I will look into that to avoid similar problem in the future. If something else pops up in those file, I can do the fix next time.
Sébastien
comment:200 Changed 10 years ago by
Your input on the matter is welcome. According to Florent, the problem in suffix_trees is that there are several equivalent answers and some platforms now give a different one than the one you originally tested for. Note that we don't know what the trigger is for having a different answer it is not a 32/64 bit separation as we have seen both answers on the two kinds of systems.
comment:201 followup: ↓ 216 Changed 10 years ago by
Actually, you should have a look at what I did for both of these in trac_9958nfactor_enumerable_wordrandomness.patch and trac_9958suffix_treesvariations.patch. If you could give your opinion on the patches or if you want to submit patches of your own it would be appreciated.
comment:202 Changed 10 years ago by
 Description modified (diff)
With the current update I have the following failures:
32bit sage t long force_lib devel/sage9958/sage/combinat/words/nfactor_enumerable_word .py ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sage9958/sage/combinat/words/nfactor_enume rable_word.py", line 23: sage: it.next() Expected: word: 6456 # random Got: word: 6456 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sage9958/sage/combinat/words/nfactor_enume rable_word.py", line 25: sage: it.next() Expected: word: 5645 # random Got: word: 5645 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sage9958/sage/combinat/words/nfactor_enume rable_word.py", line 27: sage: it.next() Expected: word: 4564 # random Got: word: 4564 ********************************************************************** 64bit sage t long force_lib devel/sage9958/sage/combinat/words/nfactor_enumerable_word .py ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sage9958/sage/combinat/words/nfactor_enume rable_word.py", line 23: sage: it.next() Expected: word: 6456 # random Got: word: 4564 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sage9958/sage/combinat/words/nfactor_enume rable_word.py", line 25: sage: it.next() Expected: word: 5645 # random Got: word: 5645 ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sage9958/sage/combinat/words/nfactor_enume rable_word.py", line 27: sage: it.next() Expected: word: 4564 # random Got: word: 6456 **********************************************************************
and on 32bit and 64bit
sage t long force_lib devel/sage9958/sage/misc/randstate.pyx ********************************************************************** File "/storage/sage/sage4.8.alpha3/devel/sage9958/sage/misc/randstate.pyx", line 15 3: sage: random(), getrandbits(20), uniform(5.0, 10.0), normalvariate(0, 1) Expected: (0.82940228518742587, 624859L, 5.77894484361117, 0.42013668263087578) Got: (0.8294022851874259, 624859L, 5.77894484361117, 0.4201366826308758) **********************************************************************
I don't recall previously seeing this particular randstate.pyx failure. Perhaps I've missed something.
comment:203 Changed 10 years ago by
 Status changed from needs_review to needs_work
Yeah, the #random
should be on the line with the command to execute, not the result.
comment:204 Changed 10 years ago by
I had to make that mistake :(
The last randstate.pyx one is definitely new, I am sure it didn't happen before.
Changed 10 years ago by
take into account the randomness of it.next sequence of call  put random in the right place this time.
comment:205 Changed 10 years ago by
OK I replaced the trac_9958nfactor_enumerable_wordrandomness.patch in place. So random should be in the right place now. I am looking to see if I can reproduce this particular randstate.pyx problem.
comment:206 Changed 10 years ago by
I missed a line in randstate! And you cut the last digit of the line number Steve, now that caused me some confusion. Updating shortly.
comment:207 Changed 10 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
OK new randstate patch, this time it is even a .patch instead of .pyx. Thanks for pointing out those mistakes.
comment:208 Changed 10 years ago by
comment:209 followup: ↓ 212 Changed 10 years ago by
Unfortunately, I'd say it is too early. We need Karl to test it on his old ppc OS X 10.4 mac and a run on solaris is also in order. I removed a patch for ia64 that didn't obviously port to python2.7 so a look at that platform is also in order. In the mean time we can review #11986 and #12124 which I think can be done independently.
We could even split the combinat patches as they should work with python2.6. It just there is nothing obvious to fix there.
comment:210 Changed 10 years ago by
As I have said before, testing is easy. I am currently running tests for the upcoming sage4.8.alpha4, but after that, I could test the new Python on various systems.
But still, somebody needs to actually look at the patches and verify that they make sense.
comment:211 Changed 10 years ago by
OK Jeroen, that's a boring task for someone most of them are of the form:
diff git a/sage/misc/sage_unittest.py b/sage/misc/sage_unittest.py  a/sage/misc/sage_unittest.py +++ b/sage/misc/sage_unittest.py @@ 201,12 +201,12 @@ Failure in _test_b: Traceback (most recent call last): ...  AssertionError + AssertionError: None  Failure in _test_d: Traceback (most recent call last): ...  AssertionError + AssertionError: None  Failure in _test_pickling: Traceback (most recent call last): @@ 220,14 +220,14 @@ running ._test_b() . . . fail Traceback (most recent call last): ...  AssertionError + AssertionError: None  running ._test_c() . . . pass running ._test_category() . . . pass running ._test_d() . . . fail Traceback (most recent call last): ...  AssertionError + AssertionError: None  running ._test_not_implemented_methods() . . . pass running ._test_pickling() . . . fail @@ 249,7 +249,7 @@ File ..., in _test_b def _test_b(self, tester): tester.fail() ...  AssertionError + AssertionError: None In conjonction with ``%pdb on``, this allows for the debbuger to jump directly to the first failure location. @@ 311,7 +311,7 @@ sage: tester.assert_(1 == 0) Traceback (most recent call last): ...  AssertionError + AssertionError: False is not true sage: tester.assert_(1 == 0, "this is expected to fail") Traceback (most recent call last): ...
and other numerical noise niceties cutting one digit and sometimes the rounding makes a lot of digits disappear (and it looks better actually), for example in plot/colors.py
@@ 290,23 +290,23 @@ sage: from sage.plot.colors import rgbcolor sage: rgbcolor(Color(0.25, 0.4, 0.9))  (0.25, 0.40000000000000002, 0.90000000000000002) + (0.25, 0.4, 0.9) sage: rgbcolor('purple')
Only the two combinat patches are a bit more subtle.
comment:212 in reply to: ↑ 209 ; followup: ↓ 214 Changed 10 years ago by
Replying to fbissey:
Unfortunately, I'd say it is too early. We need Karl to test it on his old ppc OS X 10.4 mac and a run on solaris is also in order.
Is this all based on 4.8.alpha3? I can try to do this starting today. Jeroen might be done before me, though :) at least by the time I finish applying ALL of these patches.
comment:213 Changed 10 years ago by
comment:214 in reply to: ↑ 212 ; followup: ↓ 217 Changed 10 years ago by
Replying to kcrisman:
Replying to fbissey:
Unfortunately, I'd say it is too early. We need Karl to test it on his old ppc OS X 10.4 mac and a run on solaris is also in order.
Is this all based on 4.8.alpha3? I can try to do this starting today. Jeroen might be done before me, though :)
Go ahead, I'm currently tackling other beasts...
comment:215 Changed 10 years ago by
 Dependencies changed from #5852, #11986, #12085, #12096 to #5852, #11986, #12085, #12096, #12124
comment:216 in reply to: ↑ 201 ; followup: ↓ 218 Changed 10 years ago by
Replying to fbissey:
Actually, you should have a look at what I did for both of these in trac_9958nfactor_enumerable_wordrandomness.patch and trac_9958suffix_treesvariations.patch. If you could give your opinion on the patches or if you want to submit patches of your own it would be appreciated.
Hi,
I just look at the suffix tree code (written not by me but by Franco Saliola). The method trie_type_dict
is not used anywhere in the file neither in sage/combinat/words
neither in all Sage I would guess. So, it's kind of difficult to guess what the result should be (input and output blocks are absent in documentation). I think that method was used by Franco to compare the implementation of ImplicitSuffixTree
with that of SuffixTrie
but not seriously because the output is not practical (one can't create a DiGraph
out of it).
Anyway, there are two loops in that method that both go through .iteritems()
of a dictionary... Integer indexes present in the result depends on the ordering of those two loops. So, it is possible that a third different result is obtained in some other machine or system today or later. I just made a patch which should be more durable in the long term.
For the nfactor enumerable file, the fix is perfect.
comment:217 in reply to: ↑ 214 Changed 10 years ago by
Unfortunately, I'd say it is too early. We need Karl to test it on his old ppc OS X 10.4 mac and a run on solaris is also in order.
Is this all based on 4.8.alpha3? I can try to do this starting today. Jeroen might be done before me, though :)
Go ahead, I'm currently tackling other beasts...
Okay, at least everything applies and builds from this and the two other tickets. (Note that #11986 is listed as a dependency of and for #9958.)
So presumably sometime tomorrow I can report back. So far six files having tested well...
comment:218 in reply to: ↑ 216 Changed 10 years ago by
Replying to slabbe:
Anyway, there are two loops in that method that both go through
.iteritems()
of a dictionary... Integer indexes present in the result depends on the ordering of those two loops. So, it is possible that a third different result is obtained in some other machine or system today or later. I just made a patch which should be more durable in the long term.For the nfactor enumerable file, the fix is perfect.
Thanks. I thought there could be a third possible results from Florent's comments (permutation (3,4,5)) but couldn't figure what it would look like. Happy to have some serious review for that bit.
comment:219 Changed 10 years ago by
OK the upgrade of R in #12057 shipped with 4.8.alpha4 makes a small bit of patch for r.py obsolete. Updating it shortly.
comment:220 Changed 10 years ago by
 Description modified (diff)
First patch updated. I don't think there will be any more changes because of 4.8.alpha4 but I could be wrong.
comment:221 Changed 10 years ago by
It looks like a slight modification is still needed in interfaces/r.py. On both x86 and amd64 I get
sage t long force_lib devel/sagemain/sage/interfaces/r.py ********************************************************************** File "/storage/sage/sage4.8.alpha4/devel/sagemain/sage/interfaces/r.py", line 173: sage: sum(rr._sage_()) Expected: 9.97721251689810... Got: 9.97721251689811 **********************************************************************
attachment:trac_9958suffix_treesvariationssl.patch:ticket:9958 was used in the tests.
comment:222 Changed 10 years ago by
I have seen it on my sageongentoo test run but I hadn't tried yet on a vanilla to check if it was caused by other gentoo stuff. I will add a small patch ASAP.
comment:223 Changed 10 years ago by
 Description modified (diff)
New patch included for interface/r.py in 4.8.alpha4.
comment:224 followup: ↓ 225 Changed 10 years ago by
How do we test this ticket? Is it enough to untar a fresh tarball of the 4.8.alpha4 source, replace the existing python spkg with the one from this ticket, and do a normal make? Then apply the patches to the sage library and run doctests?
On OSX 10.6.8, I tried downloading a fresh copy of 4.8.alpha4, putting this python spkg in the spkg/standard directory and deleting the old python spkg. I ended up getting a build error, seemingly right after I finished building scipy (even though scipy said it was built succesfully, but I was also building spkgs in parallel). I typed "make" again to restart the process and the same thing happened after rpy builtit said rpy was built successfully, but then it stopped the build, saying "Error building Sage".
comment:225 in reply to: ↑ 224 Changed 10 years ago by
Replying to jason:
How do we test this ticket? Is it enough to untar a fresh tarball of the 4.8.alpha4 source, replace the existing python spkg with the one from this ticket, and do a normal make? Then apply the patches to the sage library and run doctests?
Yes that's how you should do it.
On OSX 10.6.8, I tried downloading a fresh copy of 4.8.alpha4, putting this python spkg in the spkg/standard directory and deleting the old python spkg. I ended up getting a build error, seemingly right after I finished building scipy (even though scipy said it was built succesfully, but I was also building spkgs in parallel). I typed "make" again to restart the process and the same thing happened after rpy builtit said rpy was built successfully, but then it stopped the build, saying "Error building Sage".
Could you post the log for the build of python somewhere? scipy and rpy are built after sage (or at least they can be since they are runtime and not buildtime dependencies) so it is possible you were building sage itself at the same time. Is there a log for the sage spkg (in spkg/log)?
comment:226 Changed 10 years ago by
I posted the install.log here: http://sage.math.washington.edu/home/jason/sage4.8.alpha4python2.7install.log.bz2
If you search for "Error building Sage", you'll see the multiple times I restarted the build (that message appeared after each time I restarted the build).
Here's the list of spkg logs I have:
sage4.8.alpha4newpython/spkg/logs% ls atlas3.8.4.log gdmodule0.56.p7.log moin1.9.1.p2.log ratpoints2.1.3.p1.log blas20070724.log genus2reduction0.3.p8.log mpfi1.3.4cvs20071125.p8.log readline6.2.p1.log boehm_gc7.2.alpha6.p1.log gfan0.4plus.p1.log mpfr2.4.2.log rpy22.0.8.log boostcropped1.34.1.log givaro3.2.13rc2.p2.log mpir2.1.3.p8.log rubiks20070912.p17.log bzip21.0.5.log glpk4.44.log mpmath0.17.log sage_root4.8.alpha4.log cddlib094f.p8.log gnutls2.2.1.p5.log networkx1.2.p1.log sage_scripts4.8.alpha4.log cephes2.8.log graphs20070722.p1.log ntl5.5.2.log sagenb0.8.25.log cliquer1.2.p10.log gsl1.15.log numpy1.5.1.log scipy0.9.p1.log conway_polynomials0.2.log iconv1.13.1.p3.log opencdk0.6.6.p5.log scons1.2.0.log cvxopt1.1.3.log iml1.0.1.p13.log palp1.1.p3.log setuptools0.6.16.log cython0.15.1.log ipython0.10.2.p0.log pari2.5.0.p2.log singular3133.p2.log dir0.1.log jinja22.5.5.log patch2.5.9.p2.log sphinx1.1.2.p0.log docutils0.7.p0.log lapack20071123.p2.log pexpect2.0.p4.log sqlalchemy0.5.8.log ecl11.1.2.cvs20111120.p0.log lcalc1.23.p9.log pil1.1.6.p4.log sqlite3.7.5.log eclib20100711.p0.log libfplll3.0.12.p1.log polybori0.7.1.p6.log symmetrica2.0.p7.log ecm6.3.p2.log libgcrypt1.4.4.p3.log polytopes_db20100210.log sympow1.018.1.p9.log elliptic_curves0.3.log libgpg_error1.6.p3.log ppl0.11.2.log sympy0.7.1.log extcode4.8.alpha4.log libm4ri20111004.log prereq0.9.log tachyon0.98.9.p5.log f2c20070816.p2.log libm4rie20111004.log pycrypto2.1.0.log termcap1.3.1.p1.log flint1.5.0.p10.log libpng1.2.35.p3.log pygments1.3.1.p0.log twisted9.0.p2.log flintqs20070817.p6.log linbox1.1.6.p5.log pynac0.2.3.log zlib1.2.5.p0.log fortran20100629.log matplotlib1.0.1.p0.log python2.7.2.p0.log zn_poly0.9.p5.log freetype2.3.5.p3.log maxima5.23.2.p2.log python_gnutls1.1.4.p7.log zodb33.7.0.p4.log gd2.0.35.p5.log mercurial1.8.4.log r2.14.0.p0.log
comment:227 Changed 10 years ago by
All is well on the PPC OS X front, even with the r.py, amazingly; it failed the first time but passed on a rerun.
comment:228 Changed 10 years ago by
Jason the log you sent is not very useful, at least I cannot easily find something interesting. Could you provide the following:
 python2.7.2.p0.log
 polybori0.7.1.p6.log
 ppl0.11.2.log
 pynac0.2.3.log
That's a shot in the dark so far. But the building of sage never started as there is no log for sage itself.
I digged a little bit more before posting it looks like R fails to build so I want to see r2.14.0.p0.log but that may be a separate issue from python. matplotlib seems to fail to build too so I want matplotlib1.0.1.p0.log and now we are talking real potential python trouble.
comment:229 Changed 10 years ago by
I'm going to try this with
export MAKE="make j2"
on the same platform and report back.
comment:230 Changed 10 years ago by
First, here are my standard build environment settings:
export SAGE_MATPLOTLIB_GUI=yes export SAGE_PARALLEL_SPKG_BUILD=yes export MAKE="make j6" export SAGE_SPKG_INSTALL_DOCS=yes
All the requested logs are tarred up at http://sage.math.washington.edu/home/jason/newpythonspkglogs.tar.gz
comment:231 Changed 10 years ago by
Tonight, I'll try building again without setting any of those environment variables to see if it succeeds with a "stock" build.
comment:232 Changed 10 years ago by
Replying to kcrisman:
I'm going to try this with
> export MAKE="make j2"
on the same platform and report back.
Interestingly, Scipy and R are definitely (right now) building before Sage. And they usually do not, for sure not Scipy, on my ancient older Mac.
Scipy has successfully installed, as has matplotlib. R takes forever but so far is fine. Maybe it was "just a race condition"?
comment:233 Changed 10 years ago by
I bet the matplotlib error shows up for me because I define SAGE_MATPLOTLIB_GUI (so that matplotlib tries to build the GUI backends).
comment:234 Changed 10 years ago by
It is quite possibly something broken with the GUI from what I can see of your original log. scipy can build whenever after numpy independently of sage so I don't see a problem there.
comment:235 Changed 10 years ago by
Yes I think matplotlib cannot find tk. We may want to look at this in depth on OS X.
comment:236 Changed 10 years ago by
The matplotlib error is fixed upstream: https://github.com/matplotlib/matplotlib/issues/139
comment:237 Changed 10 years ago by
Very good, we may want to bump the spkg to 1.1.0 then. From my sageongentoo experience there is just one minor doctest failure from matplotli1.1.0
sage t long force_lib "devel/sage/sage/plot/colors.py" ********************************************************************** File "/usr/share/sage/devel/sage/sage/plot/colors.py", line 1241: sage: len(maps.maps) Expected: 134 Got: 138 ********************************************************************** File "/usr/share/sage/devel/sage/sage/plot/colors.py", line 1285: sage: len(maps) Expected: 134 Got: 138 **********************************************************************
Fairly standard each time we bump matplotlib or almost.
You R package definitely broke the first time around. I am fairly sure this is a parallel make issue. I have seen at least one instance that I suspect is parallel make on Gentoo. Of course given that it stop because of
make[6]: gcc: Resource temporarily unavailable
I think, we may have something else altogether. It should go into a separate ticket.
comment:238 followup: ↓ 240 Changed 10 years ago by
Do you already have a 1.1.0 spkg? I'm working on one right now, but you already have one...
comment:239 Changed 10 years ago by
Scipy has successfully installed, as has matplotlib. R takes forever but so far is fine. Maybe it was "just a race condition"?
Everything built before the Sage library except Gap and SageTeX. No problems. I say this shouldn't keep the Python upgrade out, or?
comment:240 in reply to: ↑ 238 ; followup: ↓ 241 Changed 10 years ago by
Replying to jason:
Do you already have a 1.1.0 spkg? I'm working on one right now, but you already have one...
I don't have one. I don't need one for sageongentoo as I am using system packages. You go ahead and create a ticket to upgrade matplotlib. I say it is a show stopper because it would happen on other platforms. Hopefully we can review it quickly.
comment:241 in reply to: ↑ 240 Changed 10 years ago by
Replying to fbissey:
Replying to jason:
Do you already have a 1.1.0 spkg? I'm working on one right now, but you already have one...
I don't have one. I don't need one for sageongentoo as I am using system packages. You go ahead and create a ticket to upgrade matplotlib. I say it is a show stopper because it would happen on other platforms. Hopefully we can review it quickly.
What about just making a patch upgrade with the 1.5liner rather than making a whole new package which would require even MORE testing? Plus, then you don't have to worry about getting those four extra colormaps :) Anyway, I feel this is easier and more reasonable just to get this in 5.0.alpha0 (or whatever release is next for such a big spkg change as this ticket).
comment:242 Changed 10 years ago by
 Reviewers set to John Palmieri, KarlDieter Crisman, Jason Grout, Jeroen Demeyer, Steven Trogdon, Leif Leonhardy, Sébastien Labbé
comment:243 Changed 10 years ago by
I have no objections to that. Faster is better in that case because we really want to land this quickly because every other moves of the tree makes me rewrite stuff. It start to be hard to manage.
comment:244 Changed 10 years ago by
comment:245 Changed 10 years ago by
I added a link to a patched version of the current matplotlib spkg. I also added the diff as an informationonly patch above. Independent from this fix, I've also made a matplotlib 1.1.0 spkg, which also fixes the problem, but may need more testing than this very slight change to the existing matplotlib spkg.
comment:246 Changed 10 years ago by
 Description modified (diff)
comment:247 Changed 10 years ago by
 Description modified (diff)
comment:248 Changed 10 years ago by
By the way, the matplotlib 1.1.0 spkg is at #11915, but the spkg here should be sufficient.
comment:249 Changed 10 years ago by
#12131 has been merged, I'll rebase the python spkg by next Monday hopefully.
comment:250 Changed 10 years ago by
Using the new matplotlib1.0.1.p1.spkg above, and my standard build environment variables:
export SAGE_MATPLOTLIB_GUI=yes export SAGE_PARALLEL_SPKG_BUILD=yes export MAKE="make j6" export SAGE_SPKG_INSTALL_DOCS=yes
Everything except R builds nicely. R still has that message about resources not being available (something about fork). I then changed MAKE to "make j2" and restarted the make and everything finished just fine. Again, on OSX 10.6.8 (4G of memory).
That's a bummer that make j6 doesn't work. Regular old 4.8.alpha4 built fine, so I don't think it's just the new R spkg. I'll try building fresh one more time, but with MAKE=make j4 just to check things out.
comment:251 Changed 10 years ago by
 Description modified (diff)
comment:252 Changed 10 years ago by
Okay, I started with 4.8.alpha4, added the new python spkg from this ticket, the new matplotlib spkg from #11915, applied all of the patches from this ticket, #11976, #11986, and #12124. I have a tarball with all of these:
http://sage.math.washington.edu/home/palmieri/misc/9958/sage4.8.alpha49958.tar
along with an upgrade path:
http://sage.math.washington.edu/home/palmieri/misc/9958/sage4.8.alpha49958/
for testing.
comment:253 Changed 10 years ago by
Did you also apply the patch from #11915?
comment:254 Changed 10 years ago by
No, looks like I missed that one.
comment:255 Changed 10 years ago by
Okay, here are versions including the patch from #11915:
comment:256 Changed 10 years ago by
 Description modified (diff)
comment:257 Changed 10 years ago by
I took jhpalmieri's original tarball, applied the patch from #11915 myself, built it with these settings:
SAGE_MATPLOTLIB_GUI=yes SAGE_PARALLEL_SPKG_BUILD=yes MAKE=make j12 SAGE_SPKG_INSTALL_DOCS=yes
on Ubuntu 10.04. Everything built fine, and make ptestlong passed all tests. Yeah!
comment:258 followup: ↓ 259 Changed 10 years ago by
What remains to be done here for positive review?
comment:259 in reply to: ↑ 258 Changed 10 years ago by
Replying to kcrisman:
What remains to be done here for positive review?
First I have to rebase the python spkg to #12131. Second there is a new matplotlib spkg currently merged in 4.8.alpha5 (unreleased) that needs review in #11915. Then someone has to look at the patches to make sure they are not completely stupid (long tiedous job).
Then we can move this to a positive review. I was hoping to be able to to rebase the spkg on Sunday but it didn't happen. I am busy doing a bit of code for my paid work but once it is done. I'll go for it, that may be tomorrow in my time zone.
comment:260 Changed 10 years ago by
python2.7.2.p0 spkg rebased and available.
comment:261 Changed 10 years ago by
#11986 also needs review. It looks like it may need to go at the same time as python 2.7 as my own testing with python 2.6 on linux amd64 was not positive but everything is smooth with python 2.7.
comment:262 followup: ↓ 263 Changed 10 years ago by
For sage4.8.alpha4 with tickets #9958 + #11986 + #12124 + #11976 + #11915 all tests pass on x86 and on amd64 there is one TIME OUT
sage t long force_lib devel/sagematplotlib1.1.0/sage/interfaces/ecm.py *** *** Error: TIMED OUT! PROCESS KILLED! *** *** [1800.2 s]
The test did pass when run individually. I'm sure this is due to running the tests in parallel. I've seen it before, not with this ticket, but with sageongentoo builds.
comment:263 in reply to: ↑ 262 ; followup: ↓ 264 Changed 10 years ago by
Replying to strogdon:
For sage4.8.alpha4 with tickets #9958 + #11986 + #12124 + #11976 + #11915 all tests pass on x86 and on amd64 there is one TIME OUT
sage t long force_lib devel/sagematplotlib1.1.0/sage/interfaces/ecm.py *** *** Error: TIMED OUT! PROCESS KILLED! *** *** [1800.2 s]The test did pass when run individually. I'm sure this is due to running the tests in parallel. I've seen it before, not with this ticket, but with sageongentoo builds.
I have seen it before too  in parallel tests as well. It seems that ecm can be a bad boy, not sure what to do about it but that would be outside of the scope of this ticket in my opinion.
Steve, did you do any test of #11986 with python2.6?
comment:264 in reply to: ↑ 263 Changed 10 years ago by
comment:265 followup: ↓ 266 Changed 9 years ago by
Upgrading from sage4.5 fails with:
Testing that Sage starts... [20120103 13:47:34] Traceback (most recent call last): File "/mnt/usb1/scratch/jdemeyer/sage4.5python27/local/bin/sageeval", line 4, in <module> from sage.all import * File "/mnt/usb1/scratch/jdemeyer/sage4.5python27/local/lib/python2.7/sitepackages/sage/all.py", line 75, in <module> from sage.matrix.all import * File "/mnt/usb1/scratch/jdemeyer/sage4.5python27/local/lib/python2.7/sitepackages/sage/matrix/all.py", line 1, in <module> from matrix_space import MatrixSpace, is_MatrixSpace File "/mnt/usb1/scratch/jdemeyer/sage4.5python27/local/lib/python2.7/sitepackages/sage/matrix/matrix_space.py", line 37, in <module> import matrix_modn_dense File "matrix_modn_dense.pyx", line 1, in init sage.matrix.matrix_modn_dense (sage/matrix/matrix_modn_dense.c:15026) AttributeError: 'module' object has no attribute 'Matrix_dense' Sage failed to start up. Please email sagedevel (http://groups.google.com/group/sagedevel) explaining the problem and send the log file /mnt/usb1/scratch/jdemeyer/sage4.5python27/start.log Describe your computer, operating system, etc.
After sage baforce
, it works again.
comment:266 in reply to: ↑ 265 ; followup: ↓ 267 Changed 9 years ago by
Replying to jdemeyer:
Upgrading from sage4.5 fails with:
Testing that Sage starts... [20120103 13:47:34] Traceback (most recent call last): File "/mnt/usb1/scratch/jdemeyer/sage4.5python27/local/bin/sageeval", line 4, in <module> from sage.all import * File "/mnt/usb1/scratch/jdemeyer/sage4.5python27/local/lib/python2.7/sitepackages/sage/all.py", line 75, in <module> from sage.matrix.all import * File "/mnt/usb1/scratch/jdemeyer/sage4.5python27/local/lib/python2.7/sitepackages/sage/matrix/all.py", line 1, in <module> from matrix_space import MatrixSpace, is_MatrixSpace File "/mnt/usb1/scratch/jdemeyer/sage4.5python27/local/lib/python2.7/sitepackages/sage/matrix/matrix_space.py", line 37, in <module> import matrix_modn_dense File "matrix_modn_dense.pyx", line 1, in init sage.matrix.matrix_modn_dense (sage/matrix/matrix_modn_dense.c:15026) AttributeError: 'module' object has no attribute 'Matrix_dense' Sage failed to start up. Please email sagedevel (http://groups.google.com/group/sagedevel) explaining the problem and send the log file /mnt/usb1/scratch/jdemeyer/sage4.5python27/start.log Describe your computer, operating system, etc.After
sage baforce
, it works again.
Quite weird. You would think those particular libraries would have been rebuilt already after the linbox upgrade at least. Shouldn't a complete rebuild of sage occur after a python upgrade? In any case that seems mild for upgrading from 4.5 which was released on June 2010 many versions away.
comment:267 in reply to: ↑ 266 Changed 9 years ago by
 Status changed from needs_review to needs_work
 Work issues set to Fix upgrading
Replying to fbissey:
Shouldn't a complete rebuild of sage occur after a python upgrade?
I guess it should. Do you know how to make this happen?
comment:268 Changed 9 years ago by
 Description modified (diff)
comment:269 Changed 9 years ago by
 Description modified (diff)
First attempt is at: http://sage.math.washington.edu/home/palmieri/SPKG/python2.7.p0.spkg
John (jhpalmieri) reports that this builds but Sage does not on sage.math, and "packages for twisted, zodb, pygments, and numpy don't build correctly."
Apparently the fix for http://bugs.python.org/issue7491 causes some of these problems.
On the numpy/scipy lists Ralf Gommers says "Numpy 1.5 should work with Python 2.7 and 3.1 and not be too far off. In August hopefully".
So it looks like http://trac.sagemath.org/sage_trac/ticket/9808 should fix the numpy issues.