Opened 3 years ago

Closed 2 years ago

#28190 closed enhancement (invalid)

Allow different versions of python modules depending on the python version used

Reported by: arojas Owned by:
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: packages: standard Keywords:
Cc: vdelecroix, gh-timokau, fbissey, embray, jdemeyer, chapoton, vbraun Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: public/ticket/28190 (Commits, GitHub, GitLab) Commit: ecb7133e0357c8693c63efad85998df2500b713d
Dependencies: Stopgaps:

Status badges

Description

See https://groups.google.com/forum/#!topic/sage-packaging/i9VosTO9-cM for details.

Until python2 is dropped, many packages which have dropped python 2 support upstream (ipython, sphinx, scipy, networkx, matplotlib...) are stuck to old, unmaintained versions in sage. This will pose a particular problem for distros once Sage defaults to python 3, since those distros will probably be carrying newer versions of these dependencies and sage will have to be heavily patched to work with them.

The aim of this ticket is to allow different versions for python2/python3 packages, so the python3 versions can be kept up to date in sage itself and the relevant patches can be upstreamed.

Change History (20)

comment:1 Changed 3 years ago by vdelecroix

  • Cc vdelecroix added

comment:2 Changed 3 years ago by gh-timokau

  • Cc gh-timokau added

comment:3 Changed 3 years ago by fbissey

  • Cc fbissey added

comment:4 Changed 3 years ago by jhpalmieri

Do you have ideas about how to implement this? Maybe a directory build/pkgs/python2-packages/ and if building with Python 2, first look there for the directory, and if that fails, look in build/pkgs? Or instead, within build/pkgs/sphinx/ (for example), have a python2 directory with py2 build information. In any case, I think that if there is a default setting, it should be Python 3.

comment:5 follow-up: Changed 3 years ago by arojas

Perhaps it would be simpler to keep the python2 and python3 versions as separate pkgs? (build/pkgs/sphinx vs build/pkgs/python2-sphinx)

comment:6 in reply to: ↑ 5 Changed 3 years ago by vdelecroix

Replying to arojas:

Perhaps it would be simpler to keep the python2 and python3 versions as separate pkgs? (build/pkgs/sphinx vs build/pkgs/python2-sphinx)

+1

It will also be simpler to remove the python2-* repos when the life of Python2 terminates.

comment:7 Changed 3 years ago by fbissey

That follows some model of binary distros, and I am OK with that. Implementation will have some kinks as it relies on the relationship between the folder name, the name of the package for dependencies and the name of the source tarball. I would say the latest is the hardest bit to implement - dependencies not so much, it is just tedious work.

comment:8 Changed 3 years ago by chapoton

  • Cc embray jdemeyer added

Who is going to take care of that ?

  • I would suggest not to change the name of the packages, not to duplicate.
  • Rather have (when required) inside build/pkg folders some new files checksums-legacy.ini and package-version-legacy.txt
  • Change the script in the appropriate place (where ?) so that it chooses the version to install according to the python version.

Erik, Jeroen : your opinions on the problem, please ?

comment:9 Changed 3 years ago by chapoton

  • Branch set to public/ticket/28190
  • Commit set to ecb7133e0357c8693c63efad85998df2500b713d

as a kind of provocation, here is a sketch of tentative..


New commits:

ecb7133first tentative of allowing a legacy version of packages

comment:10 in reply to: ↑ description ; follow-ups: Changed 3 years ago by jdemeyer

Let's start by analyzing the problem and possible solutions before discussing implementation details.

For a given package, let V2 be the latest version supporting Python 2 and let V3 be the latest version supporting Python 3. There are several possibilities (assuming obviously that V3 >= V2):

(A) V2 == V3: the package continues to support Python 2.

(B) V2 < V3 but the versions are compatible: the functionality that Sage uses works the same way with both versions.

(C) V2 and V3 are incompatible.

In case (A), nothing need to be done, so we don't need to discuss this further. In case (B), it doesn't really matter what we do: even if a distro uses a different version from Sage, there is no problem. So we really need to worry only about case (C). So first of all a serious question: are there packages where (C) currently applies? If not, we're solving a non-problem.

For (C), I see three possible solutions:

(i) continue using V2 in Sage and leave it up to the distros to deal with that.

(ii) use V2 when compiling with Python 2 and V3 when compiling with Python 3.

(iii) use V3 and drop support for Python 2 in Sage completely.

This ticket proposes (ii) but I'm not convinced that this is what we should do. It means that code in the Sage library needs to deal with supporting two incompatible versions of a package. That will complicate the code and increase chances of breakage.

If we can wait, maybe (iii) is actually a viable option: Python 2 is officially unsupported half a year from now. If Sage fully works under Python 3 by then, maybe we can drop Python 2 support at that time.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 3 years ago by arojas

Replying to jdemeyer:

So first of all a serious question: are there packages where (C) currently applies? If not, we're solving a non-problem.

Yes (see the linked sage-packaging post)

If we can wait, maybe (iii) is actually a viable option: Python 2 is officially unsupported half a year from now. If Sage fully works under Python 3 by then, maybe we can drop Python 2 support at that time.

That would be a solution, but would you really consider going from defaulting to python2 to completely dropping it without any deprecation period?

comment:12 in reply to: ↑ 11 ; follow-up: Changed 3 years ago by jdemeyer

Replying to arojas:

would you really consider going from defaulting to python2 to completely dropping it without any deprecation period?

What would that deprecation mean in practice?

From the user's point of view, Sage uses either Python 2 or Python 3 (whatever we default to at installation time), so there is going to be a sudden break anyway sooner or later. Once you're defaulting to Python 3, I don't know if we should still put effort in supporting Python 2.

comment:13 Changed 3 years ago by vdelecroix

As mentioned by Samuel on the mailing list, there is a distinct situation where we might want to upgrade packages to Python3 before the transition py2 -> py3: the jupyter notebook. It is not exactly covered by the ticket description.

More concretely, we could

  • python3 install the modules: jupyter_core, jupyter_client, traitlets, decorator, enum34, pyzmq, dateutil
  • remove python2 installation of jupyter_core and jupyter_client (and dead dependencies)
  • run the jupyter notebook under python3

Question: would that really be useful?

comment:14 in reply to: ↑ 12 Changed 3 years ago by gh-timokau

Replying to jdemeyer:

Replying to arojas:

would you really consider going from defaulting to python2 to completely dropping it without any deprecation period?

What would that deprecation mean in practice?

From the user's point of view, Sage uses either Python 2 or Python 3 (whatever we default to at installation time), so there is going to be a sudden break anyway sooner or later. Once you're defaulting to Python 3, I don't know if we should still put effort in supporting Python 2.

+1 on that point. If users don't want to migrate right away, they can continue using an older sage version. That may have security implications of course, but since some dependencies won't receive any updates anyway those implications are there either way.

There is little point to the "python2 deadline" if half of the projects decide "just a little longer".

comment:15 in reply to: ↑ 10 Changed 3 years ago by embray

Replying to jdemeyer:

This ticket proposes (ii) but I'm not convinced that this is what we should do. It means that code in the Sage library needs to deal with supporting two incompatible versions of a package. That will complicate the code and increase chances of breakage.

I don't think that's such a problem. Yes, it's annoying. But with care, it can be done fairly cleanly, and explicitly supporting multiple versions of a depdency can be more stable.

E.g. in Astropy we support something like 4 different Numpy versions, all of which occasionally need different version-specific worksarounds or features. This is usually done explicitly in the code with pre-defined constants like if NUMPY_LT_1_11: ....

comment:16 Changed 2 years ago by embray

  • Milestone changed from sage-8.9 to sage-9.1

Ticket retargeted after milestone closed

comment:17 Changed 2 years ago by arojas

Now that 9.0 is out, what is the plan for this? Is python2 going to be dropped for 9.1? My current ipython 7 patch is almost 5000 LOC and getting harder to maintain, it would be nice to upstream it. Also, updating to python 3.8 would be more complicated if python2 support is to be kept, see https://trac.sagemath.org/ticket/27754#comment:55

comment:18 Changed 2 years ago by embray

I don't think there should be a rush to upgrade to Python 3.8. With the first Python 3-based release just out, let's give it a little breathing room for things to shake out, as I'm sure it will still be a rocky transition for some users at first (hopefully not too much so though).

comment:19 Changed 2 years ago by mkoeppe

  • Cc chapoton vbraun added
  • Milestone changed from sage-9.1 to sage-duplicate/invalid/wontfix
  • Status changed from new to needs_review

It seems we are dropping Python2 support after 9.1. See #29141 for the reference to the discussion on sage-devel.

I propose to close this ticket, in favor of #28000: Remove python 2 support

comment:20 Changed 2 years ago by chapoton

  • Resolution set to invalid
  • Status changed from needs_review to closed

I agree, let us move forward.

Note: See TracTickets for help on using tickets.