Opened 2 years ago
Last modified 2 months ago
#29023 new enhancement
Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages)
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.7 |
Component: | packages: standard | Keywords: | |
Cc: | thansen, tmonteil, mjo, gh-timokau, fbissey, isuruf, mmezzarobba, embray, dimpase, gh-thierry-FreeBSD, gh-dkwo | Merged in: | |
Authors: | Reviewers: | ||
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
This is a followup on #27824: spkg-configure.m4
for python3.
In the next step, we make it possible for users to enable use of system Python packages (site packages).
It is crucial to distinguish two categories of Python packages.
A. Python-implemented applications not running in the same process as sagelib
For some specific packages that provide Python-implemented applications and do not run in the same process as sagelib
, there are separate tickets that may use spkg-configure.m4
- #30124 enables use of system packages for the Python packages
notebook
(#30124),rst2ipynb
, and their dependencies.
B. Packages that run in the same process as sagelib.
Use of these packages would be enabled by a new configure option --enable-system-site-packages
.
This will set up the venv using build/bin/sage-venv --system-site-packages
Approach 1:
- #29665:
configure --enable-system-site-packages
viaspkg-configure.m4
Approach 2:
- Separate the installation process of a Python package into several phases (makefile targets):
i) Downloading the spkg tarball (in most cases the tarball will already be an sdist, i.e., with embedded metadata)
ii) (When patches are needed or the upstream tarball is not a real sdist:) Unpack, patch, make a new sdist
iii) Copy the sdist to
$SAGE_SPKG_WHEELS
or a new directory next to it
iv) Build a wheel and copy to
$SAGE_SPKG_WHEELS
v) Install the wheel
Questions to be resolved:
- https://trac.sagemath.org/ticket/32492#comment:17 -- installed packages are only considered by pip if they can also be found in an index. As we use
--no-index
, the workaround in #32492 provides the directory$SAGE_SPKG_WHEELS
as the index.
To be checked if the specific installed version of the package needs to be found in the index. If presence of any version (= spkg version) will cause pip to consider the installed version as a candidate for resolution, then we can just leave the decision whether to use the installed package or the spkg to the resolver and the pip upgrade-strategy.
If a package needs any special preparation for building a wheel such as setting of environment variables, we need to:
a) either invoke phase iv) explicitly (so wheels of such packages may be built but remain unused);
b) or try to move the environment settings to the general build environment;
c) or patch the Python package's build script instead, treating it like phase ii).
- If we need a specific version (with patches that have not been upstreamed), should this be reflected in dependency information (
install-requires.txt
)?
- Investigate pip's and other tools behavior when installing an upgraded/downgraded version of a package in the venv when the package is installed in the system site packages
Related/follow-up tickets:
- #31543 Meta-ticket: Eliminate use of patches for Python packages
- #29041 at
./bootstrap
time, generaterequirements.txt
,constraints.txt
,setup.cfg [install_requires]
,Pipfile
frombuild/pkgs
- #30578 Add
src/requirements.txt
andsrc/setup.cfg [install_requires]
for installation ofsagelib
in a venv - #30371 In-place (editable) installs of
sagelib
in a venv
Tickets/issues for individual packages (moved here from #27330, will be obsolete with this ticket):
- numpy
- scipy
- cypari (depends on pari)
References:
https://packaging.python.org/discussions/install-requires-vs-requirements/
PEP 508 -- Dependency specification for Python Software Packages | Python.org https://www.python.org/dev/peps/pep-0508/#pep440
PEP 440 -- Version Identification and Dependency Specification | Python.org https://www.python.org/dev/peps/pep-0440/#local-version-segments
https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies
Change History (41)
comment:1 follow-up: ↓ 5 Changed 2 years ago by
comment:2 Changed 2 years ago by
You can already do ./configure --with-system-python3=no
to avoid using the system's Python. Are you asking for something which allows the use of the system Python but not the system Python packages? Is it likely that there will be a good system Python installation which has faulty Python packages?
comment:3 Changed 2 years ago by
I just found out about the ./configure --help option. --with-system-python3 is actually exactly what I meant.
comment:4 Changed 2 years ago by
- Description modified (diff)
comment:5 in reply to: ↑ 1 Changed 2 years ago by
Replying to gh-Volker-Weissmann:
I think there should be an option to disable the use of system packages. Sage's installation is already compilcated enough and I don't want a faulty package to mess up sage.
Thanks for the input. I've updated the ticket description.
comment:6 Changed 2 years ago by
- Description modified (diff)
comment:7 Changed 2 years ago by
- Cc thansen added
I think once python system packages are detected, the spkg-configure mechanism becomes interesting for packagers. Looking forward to this change.
comment:8 Changed 2 years ago by
I don't think python packages would be handled through spkg-configure. I think it's better to turn sagelib into a pip-installable package and provide standard prereq information -- see #29041 (at ./bootstrap time, generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs)
comment:9 Changed 2 years ago by
I tried to create an experimental spkg-configure.m4
for cvxopt in #29665, and it seems OK for me. Any comments?
comment:10 follow-up: ↓ 13 Changed 2 years ago by
If you take look at sage-env you see the line
PYTHONUSERBASE="$DOT_SAGE/local"
but I think it should be
PYTHONUSERBASE="$SAGE_ROOT/local"
, because (at least on my machine) the folder $DOT_SAGE/local does not exist.
comment:11 Changed 2 years ago by
The way to enable system site packages is to use --system-site-packages
when creating the venv.
Using PYTHONUSERBASE is not a suitable way forward.
comment:12 Changed 2 years ago by
(And neither is an ad-hoc modification of PYTHONPATH proposed in #29665.)
comment:13 in reply to: ↑ 10 Changed 2 years ago by
Replying to gh-Volker-Weissmann:
the folder $DOT_SAGE/local does not exist.
Users create this folder if they want to install user packages, for example using sage -pip install --user ...
comment:14 Changed 2 years ago by
Ok. Thank you. I thought it was a bug. But this means that if you run
sudo pip uninstall cython pip install cython
cou can't build sage.
comment:15 follow-up: ↓ 19 Changed 2 years ago by
Well, isolating the Sage user installation scheme from the Python user installation scheme was done deliberately in the past to avoid possible breakage by incompatible installations.
This decision may have to be revisited at some point, but I don't think it's necessary for this ticket.
Sage's install scripts will continue to make sure that the Sage venv has all necessary packages - whether user of system site packages is enabled or not.
comment:16 Changed 2 years ago by
Just a quick note that I plan to work on this for Sage 9.3, not earlier.
comment:17 Changed 2 years ago by
- Cc tmonteil mjo gh-timokau fbissey isuruf mmezzarobba embray dimpase gh-thierry-FreeBSD added
- Description modified (diff)
- Summary changed from In a python 3 build, use system Python packages to Meta-ticket: In a python 3 build, use system Python packages
comment:18 Changed 2 years ago by
- Description modified (diff)
comment:19 in reply to: ↑ 15 Changed 2 years ago by
Replying to mkoeppe:
Well, isolating the Sage user installation scheme from the Python user installation scheme was done deliberately in the past to avoid possible breakage by incompatible installations.
It's not always easy to know what the programmers thought. What is the best source to learn things like this? Reading the tickets?
comment:20 Changed 2 years ago by
That, and asking questions on the sage-devel list, I suppose.
comment:21 Changed 2 years ago by
In the case of PYTHONUSERBASE, comments in sage-env point to some relevant tickets.
comment:22 Changed 2 years ago by
Thank you.
comment:23 Changed 2 years ago by
- Description modified (diff)
comment:24 Changed 2 years ago by
- Dependencies #27824 deleted
- Milestone changed from sage-wishlist to sage-9.3
comment:25 Changed 23 months ago by
- Description modified (diff)
comment:26 Changed 23 months ago by
- Description modified (diff)
comment:27 Changed 23 months ago by
- Description modified (diff)
comment:28 Changed 23 months ago by
- Description modified (diff)
comment:29 Changed 22 months ago by
- Description modified (diff)
comment:30 Changed 19 months ago by
- Description modified (diff)
- Summary changed from Meta-ticket: In a python 3 build, use system Python packages to Meta-ticket: In a python 3 build, optionally use system Python packages
comment:31 Changed 18 months ago by
- Description modified (diff)
comment:32 Changed 18 months ago by
- Description modified (diff)
comment:33 Changed 18 months ago by
- Summary changed from Meta-ticket: In a python 3 build, optionally use system Python packages to Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages)
comment:34 Changed 15 months ago by
- Milestone changed from sage-9.3 to sage-9.4
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
comment:35 Changed 10 months ago by
- Milestone changed from sage-9.4 to sage-9.5
comment:36 Changed 8 months ago by
- Description modified (diff)
comment:37 Changed 8 months ago by
- Description modified (diff)
comment:38 Changed 8 months ago by
- Cc gh-dkwo added
comment:39 Changed 5 months ago by
- Milestone changed from sage-9.5 to sage-9.6
comment:40 Changed 3 months ago by
- Description modified (diff)
comment:41 Changed 2 months ago by
- Milestone changed from sage-9.6 to sage-9.7
I think there should be an option to disable the use of system packages. Sage's installation is already compilcated enough and I don't want a faulty package to mess up sage.