Opened 21 months ago

Last modified 2 weeks ago

#29023 new enhancement

Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages)

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.5
Component: packages: standard Keywords:
Cc: thansen, tmonteil, mjo, gh-timokau, fbissey, isuruf, mmezzarobba, embray, dimpase, gh-thierry-FreeBSD Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

This is a followup on #27824: spkg-configure.m4 for python3.

In the next step, we make it possible for users to enable use of system Python packages (site packages).

It is crucial to distinguish two categories of Python packages.

A. Python-implemented applications not running in the same process as sagelib

For some specific packages that provide Python-implemented applications and do not run in the same process as sagelib, there are separate tickets that may use spkg-configure.m4

  • #30124 enables use of system packages for the Python packages notebook (#30124), rst2ipynb, and their dependencies.

B. Packages that run in the same process as sagelib.

Use of these packages would be enabled by a new configure option --enable-system-site-packages.

  1. This will set up the venv using build/bin/sage-venv --system-site-packages
  1. Separate the installation process of a Python package into several phases (makefile targets):

i) Downloading the spkg tarball (in most cases the tarball will already be an sdist, i.e., with embedded metadata)

ii) (When patches are needed or the upstream tarball is not a real sdist:) Unpack, patch, make a new sdist

iii) Copy the sdist to $SAGE_SPKG_WHEELS or a new directory next to it

iv) Build a wheel and copy to $SAGE_SPKG_WHEELS

v) Install the wheel

Questions to be resolved:

To be checked if the specific installed version of the package needs to be found in the index. If presence of any version (= spkg version) will cause pip to consider the installed version as a candidate for resolution, then we can just leave the decision whether to use the installed package or the spkg to the resolver and the pip upgrade-strategy.

If a package needs any special preparation for building a wheel such as setting of environment variables, we need to:

a) either invoke phase iv) explicitly (so wheels of such packages may be built but remain unused);

b) or try to move the environment settings to the general build environment;

c) or patch the Python package's build script instead, treating it like phase ii).

  • If we need a specific version (with patches that have not been upstreamed), should this be reflected in dependency information (install-requires.txt)?
  • Investigate pip's and other tools behavior when installing an upgraded/downgraded version of a package in the venv when the package is installed in the system site packages

Related/follow-up tickets:

  • #31543 Meta-ticket: Eliminate use of patches for Python packages
  • #29041 at ./bootstrap time, generate requirements.txt, constraints.txt, setup.cfg [install_requires], Pipfile from build/pkgs
  • #30578 Add src/requirements.txt and src/setup.cfg [install_requires] for installation of sagelib in a venv
  • #30371 In-place (editable) installs of sagelib in a venv

Tickets/issues for individual packages (moved here from #27330, will be obsolete with this ticket):

  • numpy
  • scipy
  • cypari (depends on pari)
  • cvxopt (experimental spkg-configure.m4 for a Python library) (#29665)

References:

https://packaging.python.org/discussions/install-requires-vs-requirements/

PEP 508 -- Dependency specification for Python Software Packages | Python.org​ https://www.python.org/dev/peps/pep-0508/#pep440

PEP 440 -- Version Identification and Dependency Specification | Python.org ​https://www.python.org/dev/peps/pep-0440/#local-version-segments

https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies

Change History (37)

comment:1 follow-up: Changed 18 months ago by gh-Volker-Weissmann

I think there should be an option to disable the use of system packages. Sage's installation is already compilcated enough and I don't want a faulty package to mess up sage.

comment:2 Changed 18 months ago by jhpalmieri

You can already do ./configure --with-system-python3=no to avoid using the system's Python. Are you asking for something which allows the use of the system Python but not the system Python packages? Is it likely that there will be a good system Python installation which has faulty Python packages?

comment:3 Changed 18 months ago by gh-Volker-Weissmann

I just found out about the ./configure --help option. --with-system-python3 is actually exactly what I meant.

comment:4 Changed 18 months ago by mkoeppe

  • Description modified (diff)

comment:5 in reply to: ↑ 1 Changed 18 months ago by mkoeppe

Replying to gh-Volker-Weissmann:

I think there should be an option to disable the use of system packages. Sage's installation is already compilcated enough and I don't want a faulty package to mess up sage.

Thanks for the input. I've updated the ticket description.

comment:6 Changed 18 months ago by mkoeppe

  • Description modified (diff)

comment:7 Changed 18 months ago by thansen

  • Cc thansen added

I think once python system packages are detected, the spkg-configure mechanism becomes interesting for packagers. Looking forward to this change.

comment:8 Changed 18 months ago by mkoeppe

I don't think python packages would be handled through spkg-configure. I think it's better to turn sagelib into a pip-installable package and provide standard prereq information -- see #29041 (at ./bootstrap time, generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs)

comment:9 Changed 17 months ago by gh-thierry-FreeBSD

I tried to create an experimental spkg-configure.m4 for cvxopt in #29665, and it seems OK for me. Any comments?

comment:10 follow-up: Changed 17 months ago by gh-Volker-Weissmann

If you take look at sage-env you see the line

PYTHONUSERBASE="$DOT_SAGE/local"

but I think it should be

PYTHONUSERBASE="$SAGE_ROOT/local"

, because (at least on my machine) the folder $DOT_SAGE/local does not exist.

comment:11 Changed 17 months ago by mkoeppe

The way to enable system site packages is to use --system-site-packages when creating the venv.

Using PYTHONUSERBASE is not a suitable way forward.

comment:12 Changed 17 months ago by mkoeppe

(And neither is an ad-hoc modification of PYTHONPATH proposed in #29665.)

comment:13 in reply to: ↑ 10 Changed 17 months ago by mkoeppe

Replying to gh-Volker-Weissmann:

the folder $DOT_SAGE/local does not exist.

Users create this folder if they want to install user packages, for example using sage -pip install --user ...

comment:14 Changed 17 months ago by gh-Volker-Weissmann

Ok. Thank you. I thought it was a bug. But this means that if you run

sudo pip uninstall cython pip install cython

cou can't build sage.

comment:15 follow-up: Changed 17 months ago by mkoeppe

Well, isolating the Sage user installation scheme from the Python user installation scheme was done deliberately in the past to avoid possible breakage by incompatible installations.

This decision may have to be revisited at some point, but I don't think it's necessary for this ticket.

Sage's install scripts will continue to make sure that the Sage venv has all necessary packages - whether user of system site packages is enabled or not.

comment:16 Changed 17 months ago by mkoeppe

Just a quick note that I plan to work on this for Sage 9.3, not earlier.

comment:17 Changed 17 months ago by mkoeppe

  • Cc tmonteil mjo gh-timokau fbissey isuruf mmezzarobba embray dimpase gh-thierry-FreeBSD added
  • Description modified (diff)
  • Summary changed from In a python 3 build, use system Python packages to Meta-ticket: In a python 3 build, use system Python packages

comment:18 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:19 in reply to: ↑ 15 Changed 17 months ago by gh-Volker-Weissmann

Replying to mkoeppe:

Well, isolating the Sage user installation scheme from the Python user installation scheme was done deliberately in the past to avoid possible breakage by incompatible installations.

It's not always easy to know what the programmers thought. What is the best source to learn things like this? Reading the tickets?

comment:20 Changed 17 months ago by mkoeppe

That, and asking questions on the sage-devel list, I suppose.

comment:21 Changed 17 months ago by mkoeppe

In the case of PYTHONUSERBASE, comments in sage-env point to some relevant tickets.

comment:22 Changed 17 months ago by gh-Volker-Weissmann

Thank you.

comment:23 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:24 Changed 16 months ago by mkoeppe

  • Dependencies #27824 deleted
  • Milestone changed from sage-wishlist to sage-9.3

Before it makes sense to work on this, we should update all Python packages (#29141); in particular, setuptools, pip, ... (#29803)

comment:25 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:26 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:27 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:28 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:29 Changed 14 months ago by mkoeppe

  • Description modified (diff)

comment:30 Changed 11 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Meta-ticket: In a python 3 build, use system Python packages to Meta-ticket: In a python 3 build, optionally use system Python packages

comment:31 Changed 11 months ago by mkoeppe

  • Description modified (diff)

comment:32 Changed 11 months ago by mkoeppe

  • Description modified (diff)

comment:33 Changed 11 months ago by mkoeppe

  • Summary changed from Meta-ticket: In a python 3 build, optionally use system Python packages to Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages)

comment:34 Changed 7 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:35 Changed 2 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:36 Changed 2 weeks ago by mkoeppe

  • Description modified (diff)

comment:37 Changed 2 weeks ago by mkoeppe

  • Description modified (diff)
Note: See TracTickets for help on using tickets.