Opened 9 years ago
Closed 2 years ago
#15530 closed enhancement (fixed)
Metaticket: Add support for python 3.6+
Reported by:  R. Andrew Ohana  Owned by:  

Priority:  major  Milestone:  sageduplicate/invalid/wontfix 
Component:  python3  Keywords:  python3 
Cc:  Merged in:  
Authors:  Reviewers:  Dima Pasechnik  
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437, #27024  Stopgaps: 
Description (last modified by )
How to build sage with python3:
make configure ./configure withpython=3 make
Beware that the second step above should be redone after every make distclean
.
For progress on making all doctests pass, see #26212 and #28298
In order to support python 3.7, the following needs to be fixed:
 #15510  upgrade to a newer version of setuptools
 #15511  upgrade to a newer version of rpy2
 #15512  upgrade to a newer version of sympy
 #14854  upgrade to a newer version of pycrypto
 #15532  upgrade to a newer version of networkx
 #15537  fix csage to work with python3
 #15539  switch from using PIL to Pillow
 #15540  trivial python3 fixes to a few spkgs
 #15541  fix sagelocation and sagedownloadfile for python3
 #15593  remove sqlalchemy
 #17591  remove gdmodule
 #15620  Stop using
StandardError
 #15755  upgrade cython to version 0.20.1
 #15807  upgrade mpmath to version 0.18
 #15980  metaticket for python3 compatibility of the sage library (stage 1)
 #16052  metaticket for python3 compatibility of the sage library (stage 2)
 #17854  remove c_lib
 #18437  fix polybori and python 3
 #18537  fix Pynac at least building against python 3
 #17607  add python 3 package
 #27024  make sagetex python3 compatible
Possibly relevant timeline from PEP 373:
Being the last of the 2.x series, 2.7 will have an extended period of maintenance. The current plan is to support it for at least 10 years from the initial 2.7 release. This means there will be bugfix releases until 2020.
Change History (88)
comment:1 Changed 9 years ago by
Dependencies:  #15510, #15511, #15512 → #15510, #15511, #15512, #15531 

Description:  modified (diff) 
comment:2 Changed 9 years ago by
Dependencies:  #15510, #15511, #15512, #15531 → #15510, #15511, #15512, #15531, #15532 

Description:  modified (diff) 
comment:3 Changed 9 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532 → #15510, #15511, #15512, #15531, #15532, #15537 

Description:  modified (diff) 
comment:4 Changed 9 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537 → #15510, #15511, #15512, #15531, #15532, #15537, #15539 

Description:  modified (diff) 
comment:5 Changed 9 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539 → #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540 

Description:  modified (diff) 
comment:6 Changed 9 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540 → #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541 

Description:  modified (diff) 
comment:7 Changed 9 years ago by
Summary:  Add support for python 3.3+ → Metaticket: Add support for python 3.3+ 

comment:8 Changed 9 years ago by
Status:  new → needs_review 

comment:9 Changed 9 years ago by
Description:  modified (diff) 

Status:  needs_review → needs_work 
Sorry, I should have put a note that I haven't listed all issues (since I haven't even encountered all the issues yet).
comment:10 Changed 9 years ago by
Description:  modified (diff) 

comment:11 Changed 9 years ago by
Description:  modified (diff) 

comment:12 Changed 9 years ago by
Description:  modified (diff) 

comment:13 Changed 9 years ago by
Description:  modified (diff) 

comment:14 Changed 9 years ago by
Milestone:  sage6.1 → sage6.2 

comment:15 Changed 9 years ago by
Description:  modified (diff) 

comment:16 Changed 9 years ago by
Description:  modified (diff) 

comment:17 Changed 9 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541 → #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807 

Description:  modified (diff) 
comment:18 Changed 9 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807 → #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980 

Description:  modified (diff) 
comment:19 Changed 9 years ago by
Description:  modified (diff) 

comment:20 Changed 9 years ago by
Python 3.4 is out now...
(really just commenting to subscribe to updates)
comment:21 Changed 9 years ago by
Milestone:  sage6.2 → sage6.3 

comment:22 Changed 8 years ago by
Milestone:  sage6.3 → sage6.4 

comment:23 Changed 8 years ago by
Keywords:  python3 added 

comment:24 Changed 8 years ago by
Description:  modified (diff) 

comment:25 Changed 8 years ago by
Description:  modified (diff) 

comment:26 Changed 8 years ago by
Description:  modified (diff) 

comment:27 Changed 8 years ago by
Description:  modified (diff) 

comment:28 Changed 8 years ago by
Description:  modified (diff) 

comment:29 Changed 8 years ago by
Description:  modified (diff) 

comment:30 Changed 8 years ago by
Description:  modified (diff) 

comment:31 followup: 32 Changed 8 years ago by
We had to abandon the idea of using distutils for libcsage at least as is. We could certainly use getting rid of scons for it. Do we know if there is anyone currently active for polybori? My understanding is that Alexander Dreyer has gone to the industry and no one has really taken over polybori.
comment:32 Changed 8 years ago by
Replying to fbissey:
We had to abandon the idea of using distutils for libcsage at least as is.
There was also a autotools approach at #15594, which I think could be returned to (it was closed as won't fix in favor of distutils).
We could certainly use getting rid of scons for it. Do we know if there is anyone currently active for polybori? My understanding is that Alexander Dreyer has gone to the industry and no one has really taken over polybori.
I don't really know the situation, from an outsiders perspective, it certainly looks as if the project is dead. I don't really know how best to deal with it.
comment:33 followup: 34 Changed 8 years ago by
Yes we closed the autotools ticket when we decided to go distutils. And then we found that distutils couldn't be used that way. So we should try to revive that.
Someone should try to contact Alexander or Michael Brickenstein to see if they will make any new releases. As I see it the first problem will be to remove scons as the build system for polybori. Autotools+distutils feels like the right option for it. Then we could work out the python3 migration. Removing polybori would be interesting too but I think it is ingrained deep in sage.
comment:34 Changed 8 years ago by
Replying to fbissey:
Someone should try to contact Alexander or Michael Brickenstein to see if they will make any new releases. As I see it the first problem will be to remove scons as the build system for polybori. Autotools+distutils feels like the right option for it. Then we could work out the python3 migration. Removing polybori would be interesting too but I think it is ingrained deep in sage.
I don't think it is too deeply integrated into sage, it seems like it is mainly used for boolean polynomial rings (which themselves don't seem to be very widely used in the sage library). If it is easier to make a version of sage compatible with python 3 which doesn't have polybori, than it is to bring polybori up to speed, I would definitely vote for the former (especially if it is significantly easier, which I suspect might be the case ... but I could be completely wrong). In either case, I would keep polybori around for the python 2 version of sage.
comment:35 Changed 8 years ago by
Description:  modified (diff) 

comment:36 Changed 8 years ago by
Description:  modified (diff) 

comment:37 followup: 38 Changed 8 years ago by
Something for which there is no ticket yet but shouldn't be forgotten. pynac is python2.7 only last time I checked. ginac/numeric.cpp
needs a significant overhaul to move to python3, some of which I think I could do but some other like the rich cmp I wouldn't.
comment:38 Changed 8 years ago by
Cc:  Ralf Stephan added 

Something for which there is no ticket yet but shouldn't be forgotten. pynac is python2.7 only last time I checked.
ginac/numeric.cpp
needs a significant overhaul to move to python3, some of which I think I could do but some other like the rich cmp I wouldn't.
rws, you've been doing great work recently with a lot of the nittygritty stuff in Pynac  any ideas? We could at least open a ticket if this is not too impossible.
comment:39 Changed 8 years ago by
I was looking at porting pynac to python 3 a few weeks ago, and it doesn't look too bad (even the rich comp stuff, but I could always be wrong). I was either going to work on this or see if I can't figure something to do with polybori and python 3 during Sage Days 64.25.
comment:40 Changed 8 years ago by
I'm only starting now to get into Pynac's Python interface. I also have no idea which changes are necessary because of Python 3. So, if you cannot give me at least a lead by opening a detailed Pynac ticket then don't expect rapid progress there.
comment:41 Changed 8 years ago by
OK so I opened a pynac issue at https://github.com/pynac/pynac/issues/42 and I'll start making pull requests for what I can fix as I go along.
comment:42 Changed 8 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980 → #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, pynac0.3.8 

The pynac issue is now fixed in pynac master, pending a full Py3 doctest of Sage.
comment:43 Changed 8 years ago by
Pending insulation of polybori or making it python3.4 compatible and modest hacking (in pynac and libcsage) I should be able to make a version of sageongentoo that will install sage for both python 2.7 and 3.4 and test things further. I see polybori as the biggest hurdle, finishing the integration of libcsage would help too but polybori is really the biggest problem as far as I can see.
comment:44 followup: 45 Changed 8 years ago by
There is some progress on scons to move to python 3 (http://thread.gmane.org/gmane.comp.programming.tools.scons.devel/12929). Still, I think thats the least of the problems. If scons doesn't get support in time we can just build Python 2 + Python 3 and use python2 for scons at build time.
comment:45 Changed 8 years ago by
Replying to vbraun:
There is some progress on scons to move to python 3 (http://thread.gmane.org/gmane.comp.programming.tools.scons.devel/12929). Still, I think thats the least of the problems. If scons doesn't get support in time we can just build Python 2 + Python 3 and use python2 for scons at build time.
I agree in principle but scons is potentially not the only problem in polybori.
comment:46 Changed 8 years ago by
First attempt to get scons
to install polybori
as a python 3.4 thing failed. SConstruct
is relying on the version of python running scons
as being the version you should use I think. That leads to a number of odd behavior and misdetections very fast.
comment:47 Changed 8 years ago by
Description:  modified (diff) 

fyi, I started working on an autotools based build system for polybori at https://github.com/ohanar/PolyBoRi/tree/autotools. I'll probably work on it a bit more next weekend during SD 64.25, but the important bits (the c++ libraries) are currently working with only a couple of hacks. What remains (for sage at least) is just a little cleanup and throwing in the bit of the python bindings we use. I've created #18437 for future discussion of polybori and python 3.
comment:48 Changed 8 years ago by
Cc:  Ralf Stephan removed 

Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, pynac0.3.8 → #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537 
Milestone:  sage6.4 → sage6.8 
comment:49 followups: 51 53 Changed 8 years ago by
Does now every standard package except polybori support Python 3?
comment:50 Changed 8 years ago by
I think so. The current pexpect
would be the other problematic one but the current has been patched to use six
to support python 3 if we don't upgrade it to 3.3 + patch (#10295).
comment:51 followup: 52 Changed 8 years ago by
comment:52 Changed 8 years ago by
Replying to aapitzsch:
Replying to jdemeyer:
Does now every standard package except polybori support Python 3?
The Sphinx release notes for 1.3b1 mention
Add support for Python 3.4.
So I assume this version is not supported by the Sphinx 1.2.2 which is currently shipped with Sage. #18497
That's odd, Gentoo permits you to build and install sphinx 1.2.2 against python 3.4 without any patch. It is a stable version so I'll assume that it passes its test suite at least with python 3.4. But going to 1.3.1 would be good anyway.
comment:53 Changed 8 years ago by
Replying to jdemeyer:
Does now every standard package except polybori support Python 3?
Pynac doesn't really support Python 3 (it builds against Python 3, but it is almost certainly broken, since it can only really be tested when sage can be built against Python 3).
comment:54 Changed 7 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537 → #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437 

Description:  modified (diff) 
comment:55 Changed 7 years ago by
Component:  packages: standard → python3 

Description:  modified (diff) 
Milestone:  sage6.8 → sagefeature 
comment:56 Changed 6 years ago by
Description:  modified (diff) 

comment:57 Changed 6 years ago by
Description:  modified (diff) 

comment:58 Changed 6 years ago by
Milestone:  sagefeature → sage8.0 

comment:59 Changed 5 years ago by
Description:  modified (diff) 

Milestone:  sage8.0 → sagepending 
Summary:  Metaticket: Add support for python 3.3+ → Metaticket: Add support for python 3.6+ 
Changed ticket title since we're now targeting Python 3.6 at a minimum (though most of the Python 3 work could probably go back to older Python 3's since everything is still compatible with Python 2.7).
I guess we can leave this open as there are still 2 metatickets associated with it that are open...
comment:60 Changed 5 years ago by
Description:  modified (diff) 

comment:61 Changed 4 years ago by
I recently ran the doctests on my python3 branch (which contains many fixes not in develop yet, though I've yet to incorporate all the existing open tickets for python3 which might lead to more improvements). I got roughly 540 failing modules.
Out of curiosity I thought I'd try a little experiment and ran the test log through the following:
$ sort logs/ptest.log  uniq c  sort nr
This just gives the text that appears most frequently in the test output. Obviously the top examples are generic things like "6461 Traceback (most recent call last):"
But beyond that there are some interesting things that might help prioritize things to fix (and maybe some of these are already fixed in open tickets). The top 20 error messages this brings up (excluding NameError
s) are:
470 TypeError: unhashable type: 'NCPolynomialIdeal'
360 TypeError: unhashable type: 'HyperellipticCurve_g2_rational_field_with_category'
229 TypeError: expected bytes, str found
196 TypeError: object of type 'map' has no len()
176 TypeError: 'dict_values' object does not support indexing
176 AttributeError: attribute '__dict__' of 'type' objects is not writable
(this one I know is #24786, so I'm going to try merging that into my python3 branch and see what that helps)122 TypeError: '>' not supported between instances of 'float' and 'NoneType'
115 TypeError: unhashable type: 'HyperellipticCurve_padic_field_with_category'
104 AttributeError: 'list' object has no attribute 'items'
94 TypeError: sequence item 0: expected str instance, bytes found
90 TypeError: unhashable type: 'ChowGroup_class_with_category'
72 TypeError: <class 'sage.schemes.hyperelliptic_curves.hyperelliptic_g2_rational_field.HyperellipticCurve_g2_rational_field_with_category'> is not hashable and does not implement _cache_key()
58 ValueError: Polyomino must be non empty
58 TypeError: 'dict_keys' object is not subscriptable
46 KeyError: ((<class 'sage.algebras.lie_algebras.classical_lie_algebra.LieAlgebraChevalleyBasis'>, Rational Field, ['A', 2]), ())
45 TypeError: must be str, not bytes
44 AttributeError: 'function' object has no attribute '__func__'
44 AttributeError: 'dict' object has no attribute 'iteritems'
40 KeyError: ((0, False), ())
39 IndexError: tuple index out of range
Of course, from this experiment there's often no easy way to see what the context is to these errors, an some of them might just occur hundreds of times in one module and not have a widereaching effect. But it's worth looking into. Grepping the test log for individual error messages helps give a better idea of where they might come from.
comment:62 Changed 4 years ago by
Another potentially interesting heuristic, though I haven't tested how effective this is yet is something like:
$ grep 'doctest.*failed' logs/ptest.log  sort k 5 nr  head 20
showing me the modules with the most failures.
On my latest python3 branch (which has all the currently open python3 tickets merged in, as well as some other fixes) I have:
sage t src/sage/misc/explain_pickle.py # 71 doctests failed sage t src/sage/calculus/riemann.pyx # 67 doctests failed sage t src/sage/libs/lcalc/lcalc_Lfunction.pyx # 58 doctests failed sage t src/sage/combinat/free_dendriform_algebra.py # 50 doctests failed sage t src/sage/combinat/words/finite_word.py # 47 doctests failed sage t src/doc/en/thematic_tutorials/sandpile.rst # 45 doctests failed sage t src/sage/matrix/compute_J_ideal.py # 44 doctests failed sage t src/sage/graphs/generic_graph.py # 41 doctests failed sage t src/sage/schemes/riemann_surfaces/riemann_surface.py # 36 doctests failed sage t src/sage/sandpiles/sandpile.py # 34 doctests failed sage t src/sage/combinat/hillman_grassl.py # 34 doctests failed sage t src/sage/combinat/cluster_algebra_quiver/quiver.py # 33 doctests failed sage t src/sage/combinat/finite_state_machine.py # 32 doctests failed sage t src/sage/rings/polynomial/real_roots.pyx # 29 doctests failed sage t src/sage/coding/source_coding/huffman.py # 26 doctests failed sage t src/sage/combinat/diagram_algebras.py # 25 doctests failed sage t src/sage/matroids/matroid.pyx # 23 doctests failed sage t src/sage/combinat/rigged_configurations/kr_tableaux.py # 23 doctests failed sage t src/sage/combinat/cluster_algebra_quiver/cluster_seed.py # 22 doctests failed sage t src/sage/graphs/graph.py # 20 doctests failed
With the exception of "explain_pickle" which is kind of a special case (and one I'm not sure what to do with), my guess is that modules with many failing tests indicate something that is deeply broken enough which, if fixed, might have far reaching consequences both within that module itself, and other modules that depend on it.
Of course this isn't always going to be the casesometimes the failures might be very localized. But this gives some idea of where the most effort is needed.
comment:63 Changed 4 years ago by
Well, based on the example above, I found one oneline bug in sage.calculus.riemann
(#25974) which fixed all the tests in that module, as well as 7 other modules. Still very anecdotal, but at least one good result from that heuristic.
comment:64 Changed 4 years ago by
In sageongentoo I get in the following trouble a lot with python3
File "sage/libs/gap/util.pyx", line 199, in sage.libs.gap.util.initialize (/dev/shm/portage/scimathematics/sage9999/work/sage9999/srcpython3_6/build/cythonized/sage/libs/gap/util.c:4452) s = str_to_bytes(gap_root(), FS_ENCODING, "surrogateescape") File "sage/libs/gap/util.pyx", line 171, in sage.libs.gap.util.gap_root (/dev/shm/portage/scimathematics/sage9999/work/sage9999/srcpython3_6/build/cythonized/sage/libs/gap/util.c:4232) gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0] TypeError: 'filter' object is not subscriptable
But I am guessing that it is not seen in vanilla sage because I apply the following patch

sage/libs/gap/util.pyx
diff git a/sage/libs/gap/util.pyx b/sage/libs/gap/util.pyx index 97703ff..48dc617 100644
a b from .element cimport * 23 23 from sage.cpython.string import FS_ENCODING 24 24 from sage.cpython.string cimport str_to_bytes, char_to_str 25 25 from sage.interfaces.gap_workspace import prepare_workspace_dir 26 from sage.env import SAGE_LOCAL , GAP_ROOT_DIR26 from sage.env import SAGE_LOCAL 27 27 28 28 29 29 ############################################################################ … … def gap_root(): 167 167 '/home/vbraun/opt/sage5.3.rc0/local/gap/latest' 168 168 """ 169 169 import os.path 170 if os.path.exists(GAP_ROOT_DIR):171 return GAP_ROOT_DIR172 print('The gap4.5.5.spkg (or later) seems to be not installed!')173 170 gap_sh = open(os.path.join(SAGE_LOCAL, 'bin', 'gap')).read().splitlines() 174 171 gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0] 175 172 gapdir = gapdir.split('"')[1]
vanilla never shows the problem because it uses GAP_ROOT_DIR
, which I find an hindrance as it presuppose a way of installing gap
, the alternate code that comes after works well enough  at least with python2. But obviously not with python3.
Any suggestions on fixing this?
comment:65 followup: 69 Changed 4 years ago by
replace
gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]
by
gapdir = next(dir for dir in gap_sh if dir.strip().startswith('GAP_DIR'))
? Please make a ticket.
comment:66 followup: 67 Changed 4 years ago by
OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.
comment:67 followup: 68 Changed 4 years ago by
Replying to fbissey:
OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.
Perhaps it should be?
comment:68 Changed 4 years ago by
comment:69 Changed 4 years ago by
comment:70 Changed 4 years ago by
Description:  modified (diff) 

comment:71 Changed 4 years ago by
Dependencies:  #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437 → #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437, #27024 

Description:  modified (diff) 
comment:72 Changed 4 years ago by
Shouldn't this ticket maintain a branch of with all the positively reviewed py3related tickets?
comment:74 Changed 4 years ago by
Description:  modified (diff) 

comment:75 Changed 4 years ago by
Description:  modified (diff) 

comment:76 Changed 4 years ago by
Description:  modified (diff) 

comment:77 Changed 3 years ago by
Description:  modified (diff) 

comment:78 Changed 3 years ago by
What are we going to do with animation support on Python 3? It seems it is broken  the only (nonconsole, not sure about console though) possibility seems to be via jupyter+j(s)mol, and this has not moved far: https://github.com/jupyter/notebook/issues/1872
See also the recent where people complain about broken animation on 8.9. https://groups.google.com/d/msg/sagedevel/OepzKdldklU/uf2eRVWuDAAJ
comment:79 Changed 3 years ago by
Dima, can you provide more details? For the second paragraph, the sagedevel thread gives an example of something which doesn't work in the legacy Sage notebook (their example starts with the command notebook()
). So that's not a Python 3 problem. It also doesn't seem to be discussing animation, but maybe I'm missing something. The problem in the first paragraph is about the Jupyter notebook, and I see the same problem with both Python 2 and Python 3.
comment:80 Changed 3 years ago by
yes, the sagedevel thread discusses mostly sagenb, but the question is how to do the job in jupyter instead, and this appears to be a blocker, as this is a serious regression.
Disclaimer: I do not know how to do animations in Sage. If someone who knows looked into this, it would be great.
comment:81 Changed 3 years ago by
Okay, it still feels like a Jupyter and/or JMol/threejs issue rather than a Python 3 issue.
comment:82 Changed 3 years ago by
it is a Python 3 issue, as one does not have sagenb there, while sagenb (at least before Sage 8.9) was able to do animations.
comment:83 Changed 3 years ago by
Regarding the animation issue, see my tests on the sagedevel thread just now. It does somehow seem to be an integration between sagenb and new j(s)mol thing. As for "how to do animations", all they meant was "make a 3d graphic, and then try to make it movable by clicking the little circle in jmol", not using an actual animation command. I'd encourage a few others to try using viewer='jmol'
in Jupyter in the 9.0 series to see what happens.
comment:86 Changed 2 years ago by
Milestone:  sagepending → sageduplicate/invalid/wontfix 

Status:  needs_work → needs_review 
Let's close this metaticket to signal the completion of the Python 3 transition.
comment:87 Changed 2 years ago by
Reviewers:  → Dima Pasechnik 

Status:  needs_review → positive_review 
ok, done
comment:88 Changed 2 years ago by
Resolution:  → fixed 

Status:  positive_review → closed 
It looks like everything is done. Now we just need to switch Sage to python 3...