Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#28657 closed enhancement (fixed)

Build Sage without "python"

Reported by: jhpalmieri Owned by:
Priority: blocker Milestone: sage-9.0
Component: build Keywords:
Cc: isuruf, fbissey, gh-timokau, arojas Merged in:
Authors: John Palmieri Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: 7a27daa (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description

Some new linux releases don't seem to include a python executable. They have python3 but no python or python2. Sage should build on such systems.

Change History (42)

comment:1 Changed 3 years ago by jhpalmieri

  • Branch set to u/jhpalmieri/no-python

comment:2 Changed 3 years ago by jhpalmieri

  • Commit set to 2d00060bd4f4b5a85df580ed4c08ebc11f282898

Here is an attempt. It probably needs work, but please test it.


New commits:

2d00060trac 28657: build Sage when there is no 'python' present.

comment:3 Changed 3 years ago by jhpalmieri

  • Status changed from new to needs_review

comment:4 Changed 3 years ago by jhpalmieri

See https://wiki.ubuntu.com/BionicBeaver/ReleaseNotes, for example: "Python 2 is no longer installed by default."

comment:5 Changed 3 years ago by vbraun

  • Priority changed from major to blocker

comment:6 Changed 3 years ago by dimpase

Are you sure there really are Linuxes without python (but with python3) ? E.g. in Gentoo python equals python3 since few years...

comment:7 Changed 3 years ago by jhpalmieri

I just installed an Ubuntu 19.10 virtual machine. I opened a terminal, typed "pyt" and hit the TAB key: it completed to "python3". There is no command "python".

comment:8 Changed 3 years ago by dimpase

does

apt install python 

provide Python(3) ?

comment:9 Changed 3 years ago by jhpalmieri

It provides python2. (Edit: and python, symlinked to python2.)

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:10 follow-up: Changed 3 years ago by dimpase

hmm, but that's insane that all the scripts needing just python need to be changed, no?

I would just install a symbolic link to python3 somewhere...

comment:11 Changed 3 years ago by jhpalmieri

I am not an Ubuntu expert, but I think that they were worried that old Python scripts might not be compatible with Python 3, so better not to make a symbolic link: just let people install Python 2 if they need it.

comment:12 follow-up: Changed 3 years ago by dimpase

Why not just create a link to python3 at build/bin/python ?

This patch is very invasive for no good reason I can see, and would make future use of Sage with system Python harder.

comment:13 Changed 3 years ago by git

  • Commit changed from 2d00060bd4f4b5a85df580ed4c08ebc11f282898 to 8905fe639777a452c9cfa4d53b14267a07b1f7bb

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8905fe6trac 28657: build Sage when there is no 'python' present.

comment:14 in reply to: ↑ 12 Changed 3 years ago by jhpalmieri

Removed changes to ipython/spkg-install which would have conflicted with #28673.

Replying to dimpase:

Why not just create a link to python3 at build/bin/python ?

That's basically what sage-system-python is already.

This patch is very invasive for no good reason I can see, and would make future use of Sage with system Python harder.

When installing a package, should we be using Sage's Python or the system Python? There are three established choices: during Sage's build use sage-system-python or sage-python23 (which means the Python that Sage builds: it uses either python2 or python3 from $SAGE_LOCAL/bin/). During runtime, There is also the script sage-python, which just runs python2 or python3. We should probably use that one more and sage-python23 less.

comment:15 Changed 3 years ago by jhpalmieri

I'm testing a branch which uses sage-python more and sage-python23 less. If it works, I'll push it.

comment:16 Changed 3 years ago by git

  • Commit changed from 8905fe639777a452c9cfa4d53b14267a07b1f7bb to 7a27daaaac2f1107cd2d7e80b669ad0c35e09d85

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d4835fctrac 28657: build Sage when there is no 'python' present.
f04cb28trac 28657: use sage-python instead of sage-python23 when possible.
7a27daatrac 28657: use sage-python or sage-system-python instead of sage-python23

comment:17 Changed 3 years ago by jhpalmieri

  • Cc isuruf fbissey gh-timokau arojas added

CC'ing some of the packaging people to see what they think.

comment:18 follow-ups: Changed 3 years ago by dimpase

IMHO there is no reason to use Sage's python during installation of a package, unless we are installing a Python/Cython? module for Sage's Python.

Now, think of a situation where Sage's Python and system Python are the same thing. It is actually not too far-fetched; since PEP 370 it's prefectly possible to install Python modules (for a system-wide Python) into ~/. And long-term it is probably the only viable option for Sage.

So, the fewer different Pythons we have around, the better...

comment:19 in reply to: ↑ 18 Changed 3 years ago by jhpalmieri

Replying to dimpase:

IMHO there is no reason to use Sage's python during installation of a package, unless we are installing a Python/Cython? module for Sage's Python.

I agree, and that's why my changes in build/pkgs are almost exclusively to use sage-system-python. The one exception is entrypoints, which installs a Python module.

Now, think of a situation where Sage's Python and system Python are the same thing. It is actually not too far-fetched; since PEP 370 it's prefectly possible to install Python modules (for a system-wide Python) into ~/. And long-term it is probably the only viable option for Sage.

So, the fewer different Pythons we have around, the better...

Right now we are using four different ones: python, sage-system-python, sage-python23, and sage-python. The main point here is to eliminate the use of python. I also wonder if we should avoid sage-python23, since it explicitly calls SAGE_LOCAL/bin/python[2 or 3], and instead use sage-python, which runs exec python2 or exec python3, letting the PATH determine where to find the program. I've done a little in that direction (changing sage-python23 to sage-python), but more could be done. It doesn't need to be done on this ticket.

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:20 in reply to: ↑ 10 Changed 3 years ago by isuruf

Replying to dimpase:

hmm, but that's insane that all the scripts needing just python need to be changed, no?

I would just install a symbolic link to python3 somewhere...

This was because python -> python3 link was explicitly recommended against in PEP-0394. In April 2018, this was changed to make it okay for virtual envs and in June 2019, it was okay for the distributor to decide.

Last edited 3 years ago by isuruf (previous) (diff)

comment:21 in reply to: ↑ 18 ; follow-up: Changed 3 years ago by gh-timokau

Replying to dimpase:

IMHO there is no reason to use Sage's python during installation of a package, unless we are installing a Python/Cython? module for Sage's Python.

Right, on distro all the python versions resolve to the system python anyway. So this change is close to a no-op from a distro perspective.

comment:22 in reply to: ↑ 21 ; follow-up: Changed 3 years ago by jhpalmieri

Replying to gh-timokau:

Replying to dimpase:

IMHO there is no reason to use Sage's python during installation of a package, unless we are installing a Python/Cython? module for Sage's Python.

Right, on distro all the python versions resolve to the system python anyway. So this change is close to a no-op from a distro perspective.

How are you handling systems which have no python executable, like the most recent Ubuntu distributions? Or has that issue not come up yet?

comment:23 in reply to: ↑ 22 ; follow-up: Changed 3 years ago by gh-timokau

Replying to jhpalmieri:

Replying to gh-timokau:

Replying to dimpase:

IMHO there is no reason to use Sage's python during installation of a package, unless we are installing a Python/Cython? module for Sage's Python.

Right, on distro all the python versions resolve to the system python anyway. So this change is close to a no-op from a distro perspective.

How are you handling systems which have no python executable, like the most recent Ubuntu distributions? Or has that issue not come up yet?

The distro *is* the "system", so I can just add python to the dependency list and the package manager will make sure it's available. python is not installed by default on ubuntu, but if the sage package depends on it it will still be installed for sage users.

comment:24 in reply to: ↑ 23 Changed 3 years ago by jhpalmieri

Replying to gh-timokau:

The distro *is* the "system", so I can just add python to the dependency list and the package manager will make sure it's available. python is not installed by default on ubuntu, but if the sage package depends on it it will still be installed for sage users.

Then you can interpret this ticket as being an effort to allow the use of either python or python3 as dependencies, rather than insisting on python. Or just change the dependency to python3.

comment:25 follow-up: Changed 3 years ago by vbraun

Are we good here? It seems like its a step in the right direction, even if it may not solve all problems for all time.

comment:26 Changed 3 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

comment:27 in reply to: ↑ 25 Changed 3 years ago by jhpalmieri

Replying to vbraun:

Are we good here? It seems like its a step in the right direction, even if it may not solve all problems for all time.

Well, I was trying to solve all problems for all time, but I'll take what I can get.

comment:28 Changed 3 years ago by fbissey

Should have looked at this more closely earlier. This is 0 improvement for sage-on-distro that don't want to install sage-system-python or sage-python or sage-python23. Possibly it requires more patching. I only see this as a transition period hopefully.

comment:29 Changed 3 years ago by jhpalmieri

It doesn't make anything worse for distros, I hope, and it should allow you to set the dependency to python3 instead of python, for systems where those are different.

comment:30 Changed 3 years ago by fbissey

I don't know about other distros but Gentoo has its own mechanism to switch between python versions. Without having to worry about python being python2 or python3, this is switched at a more fundamental level and there are ways to maintain "exceptions". So stuff like replacing python by sage-python in src/sage/doctest/control.py is actually harmful. I install sage for python2.7, python3.6 and python3.7 simultaneously and I don't need to patch anything for it know which python it is currently running.

comment:31 follow-up: Changed 3 years ago by jhpalmieri

What do you do with all of the scripts in src/bin or local/bin which call sage-python23?

comment:32 in reply to: ↑ 31 Changed 3 years ago by fbissey

Replying to jhpalmieri:

What do you do with all of the scripts in src/bin or local/bin which call sage-python23?

https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/sage/sage-9999.ebuild#L178

and thanks to your work I have now noticed there is one file in src/ext with the stuff.

The link above is only part of the solution anyway. Gentoo has an install helper called python_doscript that will wrap python scripts so multiple python versions are not a problem. So yes, I'll admit it, I don't have to have a sage-python thingy because I already have a better solution and you don't have one.

comment:33 follow-up: Changed 3 years ago by jhpalmieri

If you have a suggestion for what else to do with doctest/control.py, we can open a follow-up ticket.

comment:34 Changed 3 years ago by fbissey

To be honest I think we should wait for the dust about python2/3 to clear out. Once python2 is out of support the incentive to keep it will be reduced drastically.

comment:35 in reply to: ↑ 33 Changed 3 years ago by fbissey

Replying to jhpalmieri:

If you have a suggestion for what else to do with doctest/control.py, we can open a follow-up ticket.

Actually in the end I may have some :)

  • line 643: use sys.executable so it is always the python you are currently using.
  • line 1034 and associated doctests. Again sys.executable could be used and ... be put in doctest since we don't know what that particular result is. But there is a first simplification sage-runtests is executable and already starts with #!/usr/bin/env sage-python in your case why do you need to override the #! here? In fact why do you even need the full path? It is already in your $PATH at this stage.

comment:36 Changed 3 years ago by jhpalmieri

See #28709 for a followup incorporating these ideas.

comment:37 Changed 3 years ago by vbraun

  • Branch changed from u/jhpalmieri/no-python to 7a27daaaac2f1107cd2d7e80b669ad0c35e09d85
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:38 Changed 3 years ago by saraedum

  • Commit 7a27daaaac2f1107cd2d7e80b669ad0c35e09d85 deleted

It seems that this broke the docker images, see #28738.

Is it intentional that we now need a system Python to be installed to run Sage? Or is the situation in the docker image somehow special and other users won't be affected by this?

What confuses me is the following in sage:

# Check to see if the whole Sage install tree has moved.  If so,
# change various hardcoded paths.  Skip this if we don't have write
# access to $SAGE_LOCAL (e.g. when running as a different user) or
# if Python and sage-location haven't been installed yet.
maybe_sage_location()
{
    if [ -w "$SAGE_LOCAL" ]; then
        if [ -x "$SAGE_LOCAL/bin/python" ] && [ -x "$SAGE_LOCAL/bin/sage-location" ]; then
            sage-location || exit $?
        fi
    fi
}

But sage-location uses the system Python, so the check for $SAGE_LOCAL/bin/python seems to be wrong.

comment:39 Changed 3 years ago by jhpalmieri

It's explained in the comment, at least somewhat: I think the point is, don't bother to do this check if local/bin/python doesn't exist, because the Sage install tree is incomplete.

In any case, it would be better to check for the existence of local/bin/python or local/bin/python3.

comment:40 Changed 3 years ago by gh-timokau

Isn't moving the sage tree after build long deprecated?

comment:41 Changed 3 years ago by jhpalmieri

You can move the Sage tree at most once. This is how (for example) OS X binary distributions work: they are built with a placeholder for SAGE_ROOT, and then the first time it runs, the placeholder gets replaced with the permanent home. The script sage-location should print an error most of the time if you try to move the Sage tree.

comment:42 Changed 3 years ago by dimpase

I don't think you can move a "normal" Sage tree at all. It needs to be specially prepared to be movable, and such a prepared tree does not work without first being moved.

Note: See TracTickets for help on using tickets.