Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages)

[mkoeppe]

This is a followup on #27824: spkg-configure.m4 for python3.

In the next step, we make it possible for users to enable use of system Python packages (site packages).

It is crucial to distinguish two categories of Python packages.

A. Python-implemented applications not running in the same process as sagelib

For some specific packages that provide Python-implemented applications and do not run in the same process as sagelib, there are separate tickets that may use spkg-configure.m4

• #30124 enables use of system packages for the Python packages notebook (#30124), rst2ipynb, and their dependencies.

B. Packages that run in the same process as sagelib.

Use of these packages would be enabled by a new configure option --enable-system-site-packages.

This will set up the venv using build/bin/sage-venv --system-site-packages

Approach 1:

• #29665: configure --enable-system-site-packages via spkg-configure.m4

Approach 2:

• Separate the installation process of a Python package into several phases (makefile targets):

i) Downloading the spkg tarball (in most cases the tarball will already be an sdist, i.e., with embedded metadata)

ii) (When patches are needed or the upstream tarball is not a real sdist:) Unpack, patch, make a new sdist

iii) Copy the sdist to $SAGE_SPKG_WHEELS or a new directory next to it

iv) Build a wheel and copy to $SAGE_SPKG_WHEELS

v) Install the wheel

Questions to be resolved:

• ​https://trac.sagemath.org/ticket/32492#comment:17 -- installed packages are only considered by pip if they can also be found in an index. As we use --no-index, the workaround in #32492 provides the directory $SAGE_SPKG_WHEELS as the index.

To be checked if the specific installed version of the package needs to be found in the index. If presence of any version (= spkg version) will cause pip to consider the installed version as a candidate for resolution, then we can just leave the decision whether to use the installed package or the spkg to the resolver and the pip upgrade-strategy.

If a package needs any special preparation for building a wheel such as setting of environment variables, we need to:

a) either invoke phase iv) explicitly (so wheels of such packages may be built but remain unused);

b) or try to move the environment settings to the general build environment;

c) or patch the Python package's build script instead, treating it like phase ii).

• If we need a specific version (with patches that have not been upstreamed), should this be reflected in dependency information (install-requires.txt)?

• Investigate pip's and other tools behavior when installing an upgraded/downgraded version of a package in the venv when the package is installed in the system site packages

Related/follow-up tickets:

• #31543 Meta-ticket: Eliminate use of patches for Python packages
• #29041 at ./bootstrap time, generate requirements.txt, constraints.txt, setup.cfg [install_requires], Pipfile from build/pkgs
• #30578 Add src/requirements.txt and src/setup.cfg [install_requires] for installation of sagelib in a venv
• #30371 In-place (editable) installs of sagelib in a venv

Tickets/issues for individual packages (moved here from #27330, will be obsolete with this ticket):

• numpy
• scipy
• cypari (depends on pari)

---

References:

​https://packaging.python.org/discussions/install-requires-vs-requirements/

PEP 508 -- Dependency specification for Python Software Packages | Python.org​ ​https://www.python.org/dev/peps/pep-0508/#pep440

PEP 440 -- Version Identification and Dependency Specification | Python.org ​​https://www.python.org/dev/peps/pep-0440/#local-version-segments

​https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies

[gh-Volker-Weissmann]

I think there should be an option to disable the use of system packages. Sage's installation is already compilcated enough and I don't want a faulty package to mess up sage.

[jhpalmieri]

You can already do ./configure --with-system-python3=no to avoid using the system's Python. Are you asking for something which allows the use of the system Python but not the system Python packages? Is it likely that there will be a good system Python installation which has faulty Python packages?

[gh-Volker-Weissmann]

I just found out about the ./configure --help option. --with-system-python3 is actually exactly what I meant.

[mkoeppe]

Replying to gh-Volker-Weissmann:

I think there should be an option to disable the use of system packages. Sage's installation is already compilcated enough and I don't want a faulty package to mess up sage.

Thanks for the input. I've updated the ticket description.

[thansen]

I think once python system packages are detected, the spkg-configure mechanism becomes interesting for packagers. Looking forward to this change.

[mkoeppe]

I don't think python packages would be handled through spkg-configure. I think it's better to turn sagelib into a pip-installable package and provide standard prereq information -- see #29041 (at ./bootstrap time, generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs)

[gh-thierry-FreeBSD]

I tried to create an experimental spkg-configure.m4 for cvxopt in #29665, and it seems OK for me. Any comments?

[gh-Volker-Weissmann]

If you take look at sage-env you see the line

PYTHONUSERBASE="$DOT_SAGE/local"

but I think it should be

PYTHONUSERBASE="$SAGE_ROOT/local"

, because (at least on my machine) the folder $DOT_SAGE/local does not exist.

[mkoeppe]

The way to enable system site packages is to use --system-site-packages when creating the venv.

Using PYTHONUSERBASE is not a suitable way forward.

[mkoeppe]

(And neither is an ad-hoc modification of PYTHONPATH proposed in #29665.)

[mkoeppe]

Replying to gh-Volker-Weissmann:

the folder $DOT_SAGE/local does not exist.

Users create this folder if they want to install user packages, for example using sage -pip install --user ...

[gh-Volker-Weissmann]

Ok. Thank you. I thought it was a bug. But this means that if you run

sudo pip uninstall cython pip install cython

cou can't build sage.

[mkoeppe]

Well, isolating the Sage user installation scheme from the Python user installation scheme was done deliberately in the past to avoid possible breakage by incompatible installations.

This decision may have to be revisited at some point, but I don't think it's necessary for this ticket.

Sage's install scripts will continue to make sure that the Sage venv has all necessary packages - whether user of system site packages is enabled or not.

[mkoeppe]

Just a quick note that I plan to work on this for Sage 9.3, not earlier.

[gh-Volker-Weissmann]

Replying to mkoeppe:

Well, isolating the Sage user installation scheme from the Python user installation scheme was done deliberately in the past to avoid possible breakage by incompatible installations.

It's not always easy to know what the programmers thought. What is the best source to learn things like this? Reading the tickets?

[mkoeppe]

That, and asking questions on the sage-devel list, I suppose.

[mkoeppe]

In the case of PYTHONUSERBASE, comments in sage-env point to some relevant tickets.

[gh-Volker-Weissmann]

Thank you.

[mkoeppe]

Before it makes sense to work on this, we should update all Python packages (#29141); in particular, setuptools, pip, ... (#29803)

[mkoeppe]

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.