Opened 6 years ago
Last modified 3 months ago
#15530 needs_work enhancement
Metaticket: Add support for python 3.6+
Reported by:  ohanar  Owned by:  

Priority:  major  Milestone:  sagepending 
Component:  python3  Keywords:  python3 
Cc:  Merged in:  
Authors:  Reviewers:  
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437, #27024  Stopgaps: 
Description (last modified by )
How to build sage with python3:
make configure ./configure withpython=3 make
Beware that the second step above should be redone after every make distclean
.
For progress on making all doctests pass, see #26212
In order to support python 3.6, the following needs to be fixed:
 #15510  upgrade to a newer version of setuptools
 #15511  upgrade to a newer version of rpy2
 #15512  upgrade to a newer version of sympy
 #14854  upgrade to a newer version of pycrypto
 #15532  upgrade to a newer version of networkx
 #15537  fix csage to work with python3
 #15539  switch from using PIL to Pillow
 #15540  trivial python3 fixes to a few spkgs
 #15541  fix sagelocation and sagedownloadfile for python3
 #15593  remove sqlalchemy
 #17591  remove gdmodule
 #15620  Stop using
StandardError
 #15755  upgrade cython to version 0.20.1
 #15807  upgrade mpmath to version 0.18
 #15980  metaticket for python3 compatibility of the sage library (stage 1)
 #16052  metaticket for python3 compatibility of the sage library (stage 2)
 #17854  remove c_lib
 #18437  fix polybori and python 3
 #18537  fix Pynac at least building against python 3
 #17607  add python 3 package
 #27024  make sagetex python3 compatible
Possibly relevant timeline from PEP 373:
Being the last of the 2.x series, 2.7 will have an extended period of maintenance. The current plan is to support it for at least 10 years from the initial 2.7 release. This means there will be bugfix releases until 2020.
Change History (76)
comment:1 Changed 6 years ago by
 Dependencies changed from #15510, #15511, #15512 to #15510, #15511, #15512, #15531
 Description modified (diff)
comment:2 Changed 6 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531 to #15510, #15511, #15512, #15531, #15532
 Description modified (diff)
comment:3 Changed 6 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532 to #15510, #15511, #15512, #15531, #15532, #15537
 Description modified (diff)
comment:4 Changed 6 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537 to #15510, #15511, #15512, #15531, #15532, #15537, #15539
 Description modified (diff)
comment:5 Changed 6 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540
 Description modified (diff)
comment:6 Changed 6 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541
 Description modified (diff)
comment:7 Changed 6 years ago by
 Summary changed from Add support for python 3.3+ to Metaticket: Add support for python 3.3+
comment:8 Changed 6 years ago by
 Status changed from new to needs_review
comment:9 Changed 6 years ago by
 Description modified (diff)
 Status changed from needs_review to needs_work
Sorry, I should have put a note that I haven't listed all issues (since I haven't even encountered all the issues yet).
comment:10 Changed 6 years ago by
 Description modified (diff)
comment:11 Changed 6 years ago by
 Description modified (diff)
comment:12 Changed 6 years ago by
 Description modified (diff)
comment:13 Changed 6 years ago by
 Description modified (diff)
comment:14 Changed 6 years ago by
 Milestone changed from sage6.1 to sage6.2
comment:15 Changed 6 years ago by
 Description modified (diff)
comment:16 Changed 6 years ago by
 Description modified (diff)
comment:17 Changed 6 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807
 Description modified (diff)
comment:18 Changed 5 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980
 Description modified (diff)
comment:19 Changed 5 years ago by
 Description modified (diff)
comment:20 Changed 5 years ago by
Python 3.4 is out now...
(really just commenting to subscribe to updates)
comment:21 Changed 5 years ago by
 Milestone changed from sage6.2 to sage6.3
comment:22 Changed 5 years ago by
 Milestone changed from sage6.3 to sage6.4
comment:23 Changed 5 years ago by
 Keywords python3 added
comment:24 Changed 5 years ago by
 Description modified (diff)
comment:25 Changed 5 years ago by
 Description modified (diff)
comment:26 Changed 5 years ago by
 Description modified (diff)
comment:27 Changed 5 years ago by
 Description modified (diff)
comment:28 Changed 5 years ago by
 Description modified (diff)
comment:29 Changed 5 years ago by
 Description modified (diff)
comment:30 Changed 5 years ago by
 Description modified (diff)
comment:31 followup: ↓ 32 Changed 5 years ago by
We had to abandon the idea of using distutils for libcsage at least as is. We could certainly use getting rid of scons for it. Do we know if there is anyone currently active for polybori? My understanding is that Alexander Dreyer has gone to the industry and no one has really taken over polybori.
comment:32 in reply to: ↑ 31 Changed 5 years ago by
Replying to fbissey:
We had to abandon the idea of using distutils for libcsage at least as is.
There was also a autotools approach at #15594, which I think could be returned to (it was closed as won't fix in favor of distutils).
We could certainly use getting rid of scons for it. Do we know if there is anyone currently active for polybori? My understanding is that Alexander Dreyer has gone to the industry and no one has really taken over polybori.
I don't really know the situation, from an outsiders perspective, it certainly looks as if the project is dead. I don't really know how best to deal with it.
comment:33 followup: ↓ 34 Changed 5 years ago by
Yes we closed the autotools ticket when we decided to go distutils. And then we found that distutils couldn't be used that way. So we should try to revive that.
Someone should try to contact Alexander or Michael Brickenstein to see if they will make any new releases. As I see it the first problem will be to remove scons as the build system for polybori. Autotools+distutils feels like the right option for it. Then we could work out the python3 migration. Removing polybori would be interesting too but I think it is ingrained deep in sage.
comment:34 in reply to: ↑ 33 Changed 5 years ago by
Replying to fbissey:
Someone should try to contact Alexander or Michael Brickenstein to see if they will make any new releases. As I see it the first problem will be to remove scons as the build system for polybori. Autotools+distutils feels like the right option for it. Then we could work out the python3 migration. Removing polybori would be interesting too but I think it is ingrained deep in sage.
I don't think it is too deeply integrated into sage, it seems like it is mainly used for boolean polynomial rings (which themselves don't seem to be very widely used in the sage library). If it is easier to make a version of sage compatible with python 3 which doesn't have polybori, than it is to bring polybori up to speed, I would definitely vote for the former (especially if it is significantly easier, which I suspect might be the case ... but I could be completely wrong). In either case, I would keep polybori around for the python 2 version of sage.
comment:35 Changed 4 years ago by
 Description modified (diff)
comment:36 Changed 4 years ago by
 Description modified (diff)
comment:37 followup: ↓ 38 Changed 4 years ago by
Something for which there is no ticket yet but shouldn't be forgotten. pynac is python2.7 only last time I checked. ginac/numeric.cpp
needs a significant overhaul to move to python3, some of which I think I could do but some other like the rich cmp I wouldn't.
comment:38 in reply to: ↑ 37 Changed 4 years ago by
 Cc rws added
Something for which there is no ticket yet but shouldn't be forgotten. pynac is python2.7 only last time I checked.
ginac/numeric.cpp
needs a significant overhaul to move to python3, some of which I think I could do but some other like the rich cmp I wouldn't.
rws, you've been doing great work recently with a lot of the nittygritty stuff in Pynac  any ideas? We could at least open a ticket if this is not too impossible.
comment:39 Changed 4 years ago by
I was looking at porting pynac to python 3 a few weeks ago, and it doesn't look too bad (even the rich comp stuff, but I could always be wrong). I was either going to work on this or see if I can't figure something to do with polybori and python 3 during Sage Days 64.25.
comment:40 Changed 4 years ago by
I'm only starting now to get into Pynac's Python interface. I also have no idea which changes are necessary because of Python 3. So, if you cannot give me at least a lead by opening a detailed Pynac ticket then don't expect rapid progress there.
comment:41 Changed 4 years ago by
OK so I opened a pynac issue at https://github.com/pynac/pynac/issues/42 and I'll start making pull requests for what I can fix as I go along.
comment:42 Changed 4 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, pynac0.3.8
The pynac issue is now fixed in pynac master, pending a full Py3 doctest of Sage.
comment:43 Changed 4 years ago by
Pending insulation of polybori or making it python3.4 compatible and modest hacking (in pynac and libcsage) I should be able to make a version of sageongentoo that will install sage for both python 2.7 and 3.4 and test things further. I see polybori as the biggest hurdle, finishing the integration of libcsage would help too but polybori is really the biggest problem as far as I can see.
comment:44 followup: ↓ 45 Changed 4 years ago by
There is some progress on scons to move to python 3 (http://thread.gmane.org/gmane.comp.programming.tools.scons.devel/12929). Still, I think thats the least of the problems. If scons doesn't get support in time we can just build Python 2 + Python 3 and use python2 for scons at build time.
comment:45 in reply to: ↑ 44 Changed 4 years ago by
Replying to vbraun:
There is some progress on scons to move to python 3 (http://thread.gmane.org/gmane.comp.programming.tools.scons.devel/12929). Still, I think thats the least of the problems. If scons doesn't get support in time we can just build Python 2 + Python 3 and use python2 for scons at build time.
I agree in principle but scons is potentially not the only problem in polybori.
comment:46 Changed 4 years ago by
First attempt to get scons
to install polybori
as a python 3.4 thing failed. SConstruct
is relying on the version of python running scons
as being the version you should use I think. That leads to a number of odd behavior and misdetections very fast.
comment:47 Changed 4 years ago by
 Description modified (diff)
fyi, I started working on an autotools based build system for polybori at https://github.com/ohanar/PolyBoRi/tree/autotools. I'll probably work on it a bit more next weekend during SD 64.25, but the important bits (the c++ libraries) are currently working with only a couple of hacks. What remains (for sage at least) is just a little cleanup and throwing in the bit of the python bindings we use. I've created #18437 for future discussion of polybori and python 3.
comment:48 Changed 4 years ago by
 Cc rws removed
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, pynac0.3.8 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537
 Milestone changed from sage6.4 to sage6.8
comment:49 followups: ↓ 51 ↓ 53 Changed 4 years ago by
Does now every standard package except polybori support Python 3?
comment:50 Changed 4 years ago by
I think so. The current pexpect
would be the other problematic one but the current has been patched to use six
to support python 3 if we don't upgrade it to 3.3 + patch (#10295).
comment:51 in reply to: ↑ 49 ; followup: ↓ 52 Changed 4 years ago by
comment:52 in reply to: ↑ 51 Changed 4 years ago by
Replying to aapitzsch:
Replying to jdemeyer:
Does now every standard package except polybori support Python 3?
The Sphinx release notes for 1.3b1 mention
Add support for Python 3.4.
So I assume this version is not supported by the Sphinx 1.2.2 which is currently shipped with Sage. #18497
That's odd, Gentoo permits you to build and install sphinx 1.2.2 against python 3.4 without any patch. It is a stable version so I'll assume that it passes its test suite at least with python 3.4. But going to 1.3.1 would be good anyway.
comment:53 in reply to: ↑ 49 Changed 4 years ago by
Replying to jdemeyer:
Does now every standard package except polybori support Python 3?
Pynac doesn't really support Python 3 (it builds against Python 3, but it is almost certainly broken, since it can only really be tested when sage can be built against Python 3).
comment:54 Changed 4 years ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437
 Description modified (diff)
comment:55 Changed 3 years ago by
 Component changed from packages: standard to python3
 Description modified (diff)
 Milestone changed from sage6.8 to sagefeature
comment:56 Changed 3 years ago by
 Description modified (diff)
comment:57 Changed 3 years ago by
 Description modified (diff)
comment:58 Changed 2 years ago by
 Milestone changed from sagefeature to sage8.0
comment:59 Changed 18 months ago by
 Description modified (diff)
 Milestone changed from sage8.0 to sagepending
 Summary changed from Metaticket: Add support for python 3.3+ to Metaticket: Add support for python 3.6+
Changed ticket title since we're now targeting Python 3.6 at a minimum (though most of the Python 3 work could probably go back to older Python 3's since everything is still compatible with Python 2.7).
I guess we can leave this open as there are still 2 metatickets associated with it that are open...
comment:60 Changed 16 months ago by
 Description modified (diff)
comment:61 Changed 13 months ago by
I recently ran the doctests on my python3 branch (which contains many fixes not in develop yet, though I've yet to incorporate all the existing open tickets for python3 which might lead to more improvements). I got roughly 540 failing modules.
Out of curiosity I thought I'd try a little experiment and ran the test log through the following:
$ sort logs/ptest.log  uniq c  sort nr
This just gives the text that appears most frequently in the test output. Obviously the top examples are generic things like "6461 Traceback (most recent call last):"
But beyond that there are some interesting things that might help prioritize things to fix (and maybe some of these are already fixed in open tickets). The top 20 error messages this brings up (excluding NameError
s) are:
470 TypeError: unhashable type: 'NCPolynomialIdeal'
360 TypeError: unhashable type: 'HyperellipticCurve_g2_rational_field_with_category'
229 TypeError: expected bytes, str found
196 TypeError: object of type 'map' has no len()
176 TypeError: 'dict_values' object does not support indexing
176 AttributeError: attribute '__dict__' of 'type' objects is not writable
(this one I know is #24786, so I'm going to try merging that into my python3 branch and see what that helps)122 TypeError: '>' not supported between instances of 'float' and 'NoneType'
115 TypeError: unhashable type: 'HyperellipticCurve_padic_field_with_category'
104 AttributeError: 'list' object has no attribute 'items'
94 TypeError: sequence item 0: expected str instance, bytes found
90 TypeError: unhashable type: 'ChowGroup_class_with_category'
72 TypeError: <class 'sage.schemes.hyperelliptic_curves.hyperelliptic_g2_rational_field.HyperellipticCurve_g2_rational_field_with_category'> is not hashable and does not implement _cache_key()
58 ValueError: Polyomino must be non empty
58 TypeError: 'dict_keys' object is not subscriptable
46 KeyError: ((<class 'sage.algebras.lie_algebras.classical_lie_algebra.LieAlgebraChevalleyBasis'>, Rational Field, ['A', 2]), ())
45 TypeError: must be str, not bytes
44 AttributeError: 'function' object has no attribute '__func__'
44 AttributeError: 'dict' object has no attribute 'iteritems'
40 KeyError: ((0, False), ())
39 IndexError: tuple index out of range
Of course, from this experiment there's often no easy way to see what the context is to these errors, an some of them might just occur hundreds of times in one module and not have a widereaching effect. But it's worth looking into. Grepping the test log for individual error messages helps give a better idea of where they might come from.
comment:62 Changed 13 months ago by
Another potentially interesting heuristic, though I haven't tested how effective this is yet is something like:
$ grep 'doctest.*failed' logs/ptest.log  sort k 5 nr  head 20
showing me the modules with the most failures.
On my latest python3 branch (which has all the currently open python3 tickets merged in, as well as some other fixes) I have:
sage t src/sage/misc/explain_pickle.py # 71 doctests failed sage t src/sage/calculus/riemann.pyx # 67 doctests failed sage t src/sage/libs/lcalc/lcalc_Lfunction.pyx # 58 doctests failed sage t src/sage/combinat/free_dendriform_algebra.py # 50 doctests failed sage t src/sage/combinat/words/finite_word.py # 47 doctests failed sage t src/doc/en/thematic_tutorials/sandpile.rst # 45 doctests failed sage t src/sage/matrix/compute_J_ideal.py # 44 doctests failed sage t src/sage/graphs/generic_graph.py # 41 doctests failed sage t src/sage/schemes/riemann_surfaces/riemann_surface.py # 36 doctests failed sage t src/sage/sandpiles/sandpile.py # 34 doctests failed sage t src/sage/combinat/hillman_grassl.py # 34 doctests failed sage t src/sage/combinat/cluster_algebra_quiver/quiver.py # 33 doctests failed sage t src/sage/combinat/finite_state_machine.py # 32 doctests failed sage t src/sage/rings/polynomial/real_roots.pyx # 29 doctests failed sage t src/sage/coding/source_coding/huffman.py # 26 doctests failed sage t src/sage/combinat/diagram_algebras.py # 25 doctests failed sage t src/sage/matroids/matroid.pyx # 23 doctests failed sage t src/sage/combinat/rigged_configurations/kr_tableaux.py # 23 doctests failed sage t src/sage/combinat/cluster_algebra_quiver/cluster_seed.py # 22 doctests failed sage t src/sage/graphs/graph.py # 20 doctests failed
With the exception of "explain_pickle" which is kind of a special case (and one I'm not sure what to do with), my guess is that modules with many failing tests indicate something that is deeply broken enough which, if fixed, might have far reaching consequences both within that module itself, and other modules that depend on it.
Of course this isn't always going to be the casesometimes the failures might be very localized. But this gives some idea of where the most effort is needed.
comment:63 Changed 13 months ago by
Well, based on the example above, I found one oneline bug in sage.calculus.riemann
(#25974) which fixed all the tests in that module, as well as 7 other modules. Still very anecdotal, but at least one good result from that heuristic.
comment:64 Changed 10 months ago by
In sageongentoo I get in the following trouble a lot with python3
File "sage/libs/gap/util.pyx", line 199, in sage.libs.gap.util.initialize (/dev/shm/portage/scimathematics/sage9999/work/sage9999/srcpython3_6/build/cythonized/sage/libs/gap/util.c:4452) s = str_to_bytes(gap_root(), FS_ENCODING, "surrogateescape") File "sage/libs/gap/util.pyx", line 171, in sage.libs.gap.util.gap_root (/dev/shm/portage/scimathematics/sage9999/work/sage9999/srcpython3_6/build/cythonized/sage/libs/gap/util.c:4232) gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0] TypeError: 'filter' object is not subscriptable
But I am guessing that it is not seen in vanilla sage because I apply the following patch

sage/libs/gap/util.pyx
diff git a/sage/libs/gap/util.pyx b/sage/libs/gap/util.pyx index 97703ff..48dc617 100644
a b from .element cimport * 23 23 from sage.cpython.string import FS_ENCODING 24 24 from sage.cpython.string cimport str_to_bytes, char_to_str 25 25 from sage.interfaces.gap_workspace import prepare_workspace_dir 26 from sage.env import SAGE_LOCAL , GAP_ROOT_DIR26 from sage.env import SAGE_LOCAL 27 27 28 28 29 29 ############################################################################ … … def gap_root(): 167 167 '/home/vbraun/opt/sage5.3.rc0/local/gap/latest' 168 168 """ 169 169 import os.path 170 if os.path.exists(GAP_ROOT_DIR):171 return GAP_ROOT_DIR172 print('The gap4.5.5.spkg (or later) seems to be not installed!')173 170 gap_sh = open(os.path.join(SAGE_LOCAL, 'bin', 'gap')).read().splitlines() 174 171 gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0] 175 172 gapdir = gapdir.split('"')[1]
vanilla never shows the problem because it uses GAP_ROOT_DIR
, which I find an hindrance as it presuppose a way of installing gap
, the alternate code that comes after works well enough  at least with python2. But obviously not with python3.
Any suggestions on fixing this?
comment:65 followup: ↓ 69 Changed 10 months ago by
replace
gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]
by
gapdir = next(dir for dir in gap_sh if dir.strip().startswith('GAP_DIR'))
? Please make a ticket.
comment:66 followup: ↓ 67 Changed 10 months ago by
OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.
comment:67 in reply to: ↑ 66 ; followup: ↓ 68 Changed 10 months ago by
Replying to fbissey:
OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.
Perhaps it should be?
comment:68 in reply to: ↑ 67 Changed 10 months ago by
comment:69 in reply to: ↑ 65 Changed 10 months ago by
comment:70 Changed 9 months ago by
 Description modified (diff)
comment:71 Changed 7 months ago by
 Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437, #27024
 Description modified (diff)
comment:72 Changed 7 months ago by
Shouldn't this ticket maintain a branch of with all the positively reviewed py3related tickets?
comment:73 Changed 7 months ago by
That might be useful: I do something like that manually anyways.
comment:74 Changed 7 months ago by
 Description modified (diff)
comment:75 Changed 6 months ago by
 Description modified (diff)
comment:76 Changed 3 months ago by
 Description modified (diff)
It looks like everything is done. Now we just need to switch Sage to python 3...