Opened 5 years ago

Last modified 38 hours ago

#15530 needs_work enhancement

Metaticket: Add support for python 3.6+

Reported by: ohanar Owned by:
Priority: major Milestone: sage-pending
Component: python3 Keywords: python3
Cc: Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437 Stopgaps:

Description (last modified by chapoton)

For progress on making all doctests pass, see #26212

In order to support python 3.6, the following needs to be fixed:

  1. #15510 - upgrade to a newer version of setuptools
  2. #15511 - upgrade to a newer version of rpy2
  3. #15512 - upgrade to a newer version of sympy
  4. #14854 - upgrade to a newer version of pycrypto
  5. #15532 - upgrade to a newer version of networkx
  6. #15537 - fix csage to work with python3
  7. #15539 - switch from using PIL to Pillow
  8. #15540 - trivial python3 fixes to a few spkgs
  9. #15541 - fix sage-location and sage-download-file for python3
  10. #15593 - remove sqlalchemy
  11. #17591 - remove gdmodule
  12. #15620 - Stop using StandardError
  13. #15755 - upgrade cython to version 0.20.1
  14. #15807 - upgrade mpmath to version 0.18
  15. #15980 - meta-ticket for python3 compatibility of the sage library (stage 1)
  16. #16052 - meta-ticket for python3 compatibility of the sage library (stage 2)
  17. #17854 - remove c_lib
  18. #18437 - fix polybori and python 3
  19. #18537 - fix Pynac at least building against python 3
  20. #17607 - add python 3 package

How to build sage with python3:

make configure
./configure --with-python=3
make build

Possibly relevant timeline from PEP 373:

Being the last of the 2.x series, 2.7 will have an extended period of maintenance. The current plan is to support it for at least 10 years from the initial 2.7 release. This means there will be bugfix releases until 2020.

Change History (70)

comment:1 Changed 5 years ago by ohanar

  • Dependencies changed from #15510, #15511, #15512 to #15510, #15511, #15512, #15531
  • Description modified (diff)

comment:2 Changed 5 years ago by ohanar

  • Dependencies changed from #15510, #15511, #15512, #15531 to #15510, #15511, #15512, #15531, #15532
  • Description modified (diff)

comment:3 Changed 5 years ago by ohanar

  • Dependencies changed from #15510, #15511, #15512, #15531, #15532 to #15510, #15511, #15512, #15531, #15532, #15537
  • Description modified (diff)

comment:4 Changed 5 years ago by ohanar

  • Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537 to #15510, #15511, #15512, #15531, #15532, #15537, #15539
  • Description modified (diff)

comment:5 Changed 5 years ago by ohanar

  • Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540
  • Description modified (diff)

comment:6 Changed 5 years ago by ohanar

  • Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541
  • Description modified (diff)

comment:7 Changed 5 years ago by ohanar

  • Summary changed from Add support for python 3.3+ to Metaticket: Add support for python 3.3+

comment:8 Changed 5 years ago by tscrim

  • Status changed from new to needs_review

It looks like everything is done. Now we just need to switch Sage to python 3...

comment:9 Changed 5 years ago by ohanar

  • Description modified (diff)
  • Status changed from needs_review to needs_work

Sorry, I should have put a note that I haven't listed all issues (since I haven't even encountered all the issues yet).

comment:10 Changed 5 years ago by ohanar

  • Description modified (diff)

comment:11 Changed 5 years ago by ohanar

  • Description modified (diff)

comment:12 Changed 5 years ago by ohanar

  • Description modified (diff)

comment:13 Changed 5 years ago by ohanar

  • Description modified (diff)

comment:14 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:15 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:16 Changed 5 years ago by ohanar

  • Description modified (diff)

comment:17 Changed 5 years ago by ohanar

  • Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807
  • Description modified (diff)

comment:18 Changed 5 years ago by ohanar

  • Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980
  • Description modified (diff)

comment:19 Changed 5 years ago by wluebbe

  • Description modified (diff)

comment:20 Changed 5 years ago by jason

Python 3.4 is out now...

(really just commenting to subscribe to updates)

comment:21 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:22 Changed 4 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:23 Changed 4 years ago by aapitzsch

  • Keywords python3 added

comment:24 Changed 4 years ago by kcrisman

  • Description modified (diff)

comment:25 Changed 4 years ago by jdemeyer

  • Description modified (diff)

comment:26 Changed 4 years ago by jdemeyer

  • Description modified (diff)

comment:27 Changed 4 years ago by jdemeyer

  • Description modified (diff)

comment:28 Changed 4 years ago by jdemeyer

  • Description modified (diff)

comment:29 Changed 4 years ago by ohanar

  • Description modified (diff)

comment:30 Changed 4 years ago by ohanar

  • Description modified (diff)

comment:31 follow-up: Changed 4 years ago by fbissey

We had to abandon the idea of using distutils for libcsage at least as is. We could certainly use getting rid of scons for it. Do we know if there is anyone currently active for polybori? My understanding is that Alexander Dreyer has gone to the industry and no one has really taken over polybori.

comment:32 in reply to: ↑ 31 Changed 4 years ago by ohanar

Replying to fbissey:

We had to abandon the idea of using distutils for libcsage at least as is.

There was also a autotools approach at #15594, which I think could be returned to (it was closed as won't fix in favor of distutils).

We could certainly use getting rid of scons for it. Do we know if there is anyone currently active for polybori? My understanding is that Alexander Dreyer has gone to the industry and no one has really taken over polybori.

I don't really know the situation, from an outsiders perspective, it certainly looks as if the project is dead. I don't really know how best to deal with it.

comment:33 follow-up: Changed 4 years ago by fbissey

Yes we closed the autotools ticket when we decided to go distutils. And then we found that distutils couldn't be used that way. So we should try to revive that.

Someone should try to contact Alexander or Michael Brickenstein to see if they will make any new releases. As I see it the first problem will be to remove scons as the build system for polybori. Autotools+distutils feels like the right option for it. Then we could work out the python3 migration. Removing polybori would be interesting too but I think it is ingrained deep in sage.

comment:34 in reply to: ↑ 33 Changed 4 years ago by ohanar

Replying to fbissey:

Someone should try to contact Alexander or Michael Brickenstein to see if they will make any new releases. As I see it the first problem will be to remove scons as the build system for polybori. Autotools+distutils feels like the right option for it. Then we could work out the python3 migration. Removing polybori would be interesting too but I think it is ingrained deep in sage.

I don't think it is too deeply integrated into sage, it seems like it is mainly used for boolean polynomial rings (which themselves don't seem to be very widely used in the sage library). If it is easier to make a version of sage compatible with python 3 which doesn't have polybori, than it is to bring polybori up to speed, I would definitely vote for the former (especially if it is significantly easier, which I suspect might be the case ... but I could be completely wrong). In either case, I would keep polybori around for the python 2 version of sage.

comment:35 Changed 4 years ago by jdemeyer

  • Description modified (diff)

comment:36 Changed 4 years ago by jdemeyer

  • Description modified (diff)

comment:37 follow-up: Changed 4 years ago by fbissey

Something for which there is no ticket yet but shouldn't be forgotten. pynac is python2.7 only last time I checked. ginac/numeric.cpp needs a significant overhaul to move to python3, some of which I think I could do but some other like the rich cmp I wouldn't.

comment:38 in reply to: ↑ 37 Changed 4 years ago by kcrisman

  • Cc rws added

Something for which there is no ticket yet but shouldn't be forgotten. pynac is python2.7 only last time I checked. ginac/numeric.cpp needs a significant overhaul to move to python3, some of which I think I could do but some other like the rich cmp I wouldn't.

rws, you've been doing great work recently with a lot of the nitty-gritty stuff in Pynac - any ideas? We could at least open a ticket if this is not too impossible.

comment:39 Changed 4 years ago by ohanar

I was looking at porting pynac to python 3 a few weeks ago, and it doesn't look too bad (even the rich comp stuff, but I could always be wrong). I was either going to work on this or see if I can't figure something to do with polybori and python 3 during Sage Days 64.25.

comment:40 Changed 4 years ago by rws

I'm only starting now to get into Pynac's Python interface. I also have no idea which changes are necessary because of Python 3. So, if you cannot give me at least a lead by opening a detailed Pynac ticket then don't expect rapid progress there.

comment:41 Changed 4 years ago by fbissey

OK so I opened a pynac issue at https://github.com/pynac/pynac/issues/42 and I'll start making pull requests for what I can fix as I go along.

comment:42 Changed 4 years ago by rws

  • Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, pynac-0.3.8

The pynac issue is now fixed in pynac master, pending a full Py3 doctest of Sage.

comment:43 Changed 4 years ago by fbissey

Pending insulation of polybori or making it python3.4 compatible and modest hacking (in pynac and libcsage) I should be able to make a version of sage-on-gentoo that will install sage for both python 2.7 and 3.4 and test things further. I see polybori as the biggest hurdle, finishing the integration of libcsage would help too but polybori is really the biggest problem as far as I can see.

comment:44 follow-up: Changed 4 years ago by vbraun

There is some progress on scons to move to python 3 (http://thread.gmane.org/gmane.comp.programming.tools.scons.devel/12929). Still, I think thats the least of the problems. If scons doesn't get support in time we can just build Python 2 + Python 3 and use python2 for scons at build time.

comment:45 in reply to: ↑ 44 Changed 4 years ago by fbissey

Replying to vbraun:

There is some progress on scons to move to python 3 (http://thread.gmane.org/gmane.comp.programming.tools.scons.devel/12929). Still, I think thats the least of the problems. If scons doesn't get support in time we can just build Python 2 + Python 3 and use python2 for scons at build time.

I agree in principle but scons is potentially not the only problem in polybori.

comment:46 Changed 4 years ago by fbissey

First attempt to get scons to install polybori as a python 3.4 thing failed. SConstruct is relying on the version of python running scons as being the version you should use I think. That leads to a number of odd behavior and misdetections very fast.

comment:47 Changed 4 years ago by ohanar

  • Description modified (diff)

fyi, I started working on an autotools based build system for polybori at https://github.com/ohanar/PolyBoRi/tree/autotools. I'll probably work on it a bit more next weekend during SD 64.25, but the important bits (the c++ libraries) are currently working with only a couple of hacks. What remains (for sage at least) is just a little cleanup and throwing in the bit of the python bindings we use. I've created #18437 for future discussion of polybori and python 3.

comment:48 Changed 3 years ago by rws

  • Cc rws removed
  • Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, pynac-0.3.8 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537
  • Milestone changed from sage-6.4 to sage-6.8

comment:49 follow-ups: Changed 3 years ago by jdemeyer

Does now every standard package except polybori support Python 3?

comment:50 Changed 3 years ago by fbissey

I think so. The current pexpect would be the other problematic one but the current has been patched to use six to support python 3 if we don't upgrade it to 3.3 + patch (#10295).

comment:51 in reply to: ↑ 49 ; follow-up: Changed 3 years ago by aapitzsch

Replying to jdemeyer:

Does now every standard package except polybori support Python 3?

The Sphinx release notes for 1.3b1 mention Add support for Python 3.4. So I assume this version is not supported by the Sphinx 1.2.2 which is currently shipped with Sage. #18497

comment:52 in reply to: ↑ 51 Changed 3 years ago by fbissey

Replying to aapitzsch:

Replying to jdemeyer:

Does now every standard package except polybori support Python 3?

The Sphinx release notes for 1.3b1 mention Add support for Python 3.4. So I assume this version is not supported by the Sphinx 1.2.2 which is currently shipped with Sage. #18497

That's odd, Gentoo permits you to build and install sphinx 1.2.2 against python 3.4 without any patch. It is a stable version so I'll assume that it passes its test suite at least with python 3.4. But going to 1.3.1 would be good anyway.

comment:53 in reply to: ↑ 49 Changed 3 years ago by ohanar

Replying to jdemeyer:

Does now every standard package except polybori support Python 3?

Pynac doesn't really support Python 3 (it builds against Python 3, but it is almost certainly broken, since it can only really be tested when sage can be built against Python 3).

comment:54 Changed 3 years ago by kcrisman

  • Dependencies changed from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437
  • Description modified (diff)

comment:55 Changed 3 years ago by jdemeyer

  • Component changed from packages: standard to python3
  • Description modified (diff)
  • Milestone changed from sage-6.8 to sage-feature

comment:56 Changed 2 years ago by chapoton

  • Description modified (diff)

comment:57 Changed 2 years ago by chapoton

  • Description modified (diff)

comment:58 Changed 19 months ago by chapoton

  • Milestone changed from sage-feature to sage-8.0

comment:59 Changed 9 months ago by embray

  • Description modified (diff)
  • Milestone changed from sage-8.0 to sage-pending
  • Summary changed from Metaticket: Add support for python 3.3+ to Metaticket: Add support for python 3.6+

Changed ticket title since we're now targeting Python 3.6 at a minimum (though most of the Python 3 work could probably go back to older Python 3's since everything is still compatible with Python 2.7).

I guess we can leave this open as there are still 2 meta-tickets associated with it that are open...

comment:60 Changed 7 months ago by chapoton

  • Description modified (diff)

comment:61 Changed 4 months ago by embray

I recently ran the doctests on my python3 branch (which contains many fixes not in develop yet, though I've yet to incorporate all the existing open tickets for python3 which might lead to more improvements). I got roughly 540 failing modules.

Out of curiosity I thought I'd try a little experiment and ran the test log through the following:

$ sort logs/ptest.log | uniq -c | sort -nr

This just gives the text that appears most frequently in the test output. Obviously the top examples are generic things like "6461 Traceback (most recent call last):"

But beyond that there are some interesting things that might help prioritize things to fix (and maybe some of these are already fixed in open tickets). The top 20 error messages this brings up (excluding NameErrors) are:

  • 470 TypeError: unhashable type: 'NCPolynomialIdeal'
  • 360 TypeError: unhashable type: 'HyperellipticCurve_g2_rational_field_with_category'
  • 229 TypeError: expected bytes, str found
  • 196 TypeError: object of type 'map' has no len()
  • 176 TypeError: 'dict_values' object does not support indexing
  • 176 AttributeError: attribute '__dict__' of 'type' objects is not writable (this one I know is #24786, so I'm going to try merging that into my python3 branch and see what that helps)
  • 122 TypeError: '>' not supported between instances of 'float' and 'NoneType'
  • 115 TypeError: unhashable type: 'HyperellipticCurve_padic_field_with_category'
  • 104 AttributeError: 'list' object has no attribute 'items'
  • 94 TypeError: sequence item 0: expected str instance, bytes found
  • 90 TypeError: unhashable type: 'ChowGroup_class_with_category'
  • 72 TypeError: <class 'sage.schemes.hyperelliptic_curves.hyperelliptic_g2_rational_field.HyperellipticCurve_g2_rational_field_with_category'> is not hashable and does not implement _cache_key()
  • 58 ValueError: Polyomino must be non empty
  • 58 TypeError: 'dict_keys' object is not subscriptable
  • 46 KeyError: ((<class 'sage.algebras.lie_algebras.classical_lie_algebra.LieAlgebraChevalleyBasis'>, Rational Field, ['A', 2]), ())
  • 45 TypeError: must be str, not bytes
  • 44 AttributeError: 'function' object has no attribute '__func__'
  • 44 AttributeError: 'dict' object has no attribute 'iteritems'
  • 40 KeyError: ((0, False), ())
  • 39 IndexError: tuple index out of range

Of course, from this experiment there's often no easy way to see what the context is to these errors, an some of them might just occur hundreds of times in one module and not have a wide-reaching effect. But it's worth looking into. Grepping the test log for individual error messages helps give a better idea of where they might come from.

comment:62 Changed 4 months ago by embray

Another potentially interesting heuristic, though I haven't tested how effective this is yet is something like:

$ grep 'doctest.*failed' logs/ptest.log | sort -k 5 -nr | head -20

showing me the modules with the most failures.

On my latest python3 branch (which has all the currently open python3 tickets merged in, as well as some other fixes) I have:

sage -t src/sage/misc/explain_pickle.py  # 71 doctests failed
sage -t src/sage/calculus/riemann.pyx  # 67 doctests failed
sage -t src/sage/libs/lcalc/lcalc_Lfunction.pyx  # 58 doctests failed
sage -t src/sage/combinat/free_dendriform_algebra.py  # 50 doctests failed
sage -t src/sage/combinat/words/finite_word.py  # 47 doctests failed
sage -t src/doc/en/thematic_tutorials/sandpile.rst  # 45 doctests failed
sage -t src/sage/matrix/compute_J_ideal.py  # 44 doctests failed
sage -t src/sage/graphs/generic_graph.py  # 41 doctests failed
sage -t src/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed
sage -t src/sage/sandpiles/sandpile.py  # 34 doctests failed
sage -t src/sage/combinat/hillman_grassl.py  # 34 doctests failed
sage -t src/sage/combinat/cluster_algebra_quiver/quiver.py  # 33 doctests failed
sage -t src/sage/combinat/finite_state_machine.py  # 32 doctests failed
sage -t src/sage/rings/polynomial/real_roots.pyx  # 29 doctests failed
sage -t src/sage/coding/source_coding/huffman.py  # 26 doctests failed
sage -t src/sage/combinat/diagram_algebras.py  # 25 doctests failed
sage -t src/sage/matroids/matroid.pyx  # 23 doctests failed
sage -t src/sage/combinat/rigged_configurations/kr_tableaux.py  # 23 doctests failed
sage -t src/sage/combinat/cluster_algebra_quiver/cluster_seed.py  # 22 doctests failed
sage -t src/sage/graphs/graph.py  # 20 doctests failed

With the exception of "explain_pickle" which is kind of a special case (and one I'm not sure what to do with), my guess is that modules with many failing tests indicate something that is deeply broken enough which, if fixed, might have far reaching consequences both within that module itself, and other modules that depend on it.

Of course this isn't always going to be the case--sometimes the failures might be very localized. But this gives some idea of where the most effort is needed.

comment:63 Changed 4 months ago by embray

Well, based on the example above, I found one one-line bug in sage.calculus.riemann (#25974) which fixed all the tests in that module, as well as 7 other modules. Still very anecdotal, but at least one good result from that heuristic.

comment:64 Changed 2 weeks ago by fbissey

In sage-on-gentoo I get in the following trouble a lot with python3

      File "sage/libs/gap/util.pyx", line 199, in sage.libs.gap.util.initialize (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_6/build/cythonized/sage/libs/gap/util.c:4452)
        s = str_to_bytes(gap_root(), FS_ENCODING, "surrogateescape")
      File "sage/libs/gap/util.pyx", line 171, in sage.libs.gap.util.gap_root (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_6/build/cythonized/sage/libs/gap/util.c:4232)
        gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]
    TypeError: 'filter' object is not subscriptable

But I am guessing that it is not seen in vanilla sage because I apply the following patch

  • sage/libs/gap/util.pyx

    diff --git a/sage/libs/gap/util.pyx b/sage/libs/gap/util.pyx
    index 97703ff..48dc617 100644
    a b from .element cimport * 
    2323from sage.cpython.string import FS_ENCODING
    2424from sage.cpython.string cimport str_to_bytes, char_to_str
    2525from sage.interfaces.gap_workspace import prepare_workspace_dir
    26 from sage.env import SAGE_LOCAL, GAP_ROOT_DIR
     26from sage.env import SAGE_LOCAL
    2727
    2828
    2929############################################################################
    def gap_root(): 
    167167        '/home/vbraun/opt/sage-5.3.rc0/local/gap/latest'
    168168    """
    169169    import os.path
    170     if os.path.exists(GAP_ROOT_DIR):
    171         return GAP_ROOT_DIR
    172     print('The gap-4.5.5.spkg (or later) seems to be not installed!')
    173170    gap_sh = open(os.path.join(SAGE_LOCAL, 'bin', 'gap')).read().splitlines()
    174171    gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]
    175172    gapdir = gapdir.split('"')[1]

vanilla never shows the problem because it uses GAP_ROOT_DIR, which I find an hindrance as it presuppose a way of installing gap, the alternate code that comes after works well enough - at least with python2. But obviously not with python3.

Any suggestions on fixing this?

comment:65 follow-up: Changed 2 weeks ago by chapoton

replace

gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]

by

gapdir = next(dir for dir in gap_sh if dir.strip().startswith('GAP_DIR'))

? Please make a ticket.

comment:66 follow-up: Changed 2 weeks ago by fbissey

OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.

comment:67 in reply to: ↑ 66 ; follow-up: Changed 13 days ago by embray

Replying to fbissey:

OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.

Perhaps it should be?

comment:68 in reply to: ↑ 67 Changed 13 days ago by fbissey

Replying to embray:

Replying to fbissey:

OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.

Perhaps it should be?

Definitely.

comment:69 in reply to: ↑ 65 Changed 13 days ago by fbissey

Replying to chapoton:

replace

gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]

by

gapdir = next(dir for dir in gap_sh if dir.strip().startswith('GAP_DIR'))

? Please make a ticket.

That works, but to be complete, it will also need some doctest. It is now #26665.

comment:70 Changed 38 hours ago by chapoton

  • Description modified (diff)
Note: See TracTickets for help on using tickets.