#29500 closed enhancement (fixed)
Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.2 |
Component: | build | Keywords: | |
Cc: | slelievre, gh-tobiasdiez, embray, jhpalmieri, tscrim | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | Tobias Diez, John Palmieri |
Report Upstream: | N/A | Work issues: | |
Branch: | f2e7075 (Commits, GitHub, GitLab) | Commit: | |
Dependencies: | Stopgaps: |
Description (last modified by )
Our current version of pip already tries to install packages via bdist_wheel
.
In this ticket, we break the build process into these steps by modifying sdh_pip_install
:
- Build the wheel explicitly using
pip wheel --no-binary :all: --no-build-isolation -w "$SAGE_SPKG_WHEELS"
(orsetup.py bdist_wheel
), which stores the wheel inSAGE_SPKG_WHEELS=$SAGE_LOCAL/var/lib/sage/wheels
.
- Then install the wheel.
The wheels in $SAGE_SPKG_WHEELS
persist after the completed build. We manage them using the DESTDIR
staging mechanism -- there will be exactly 1 wheel for each installed package (and removing a package removes the wheel). Example (after rebuilding a few packages on this branch):
$ ls -l local/var/lib/sage/wheels/ ... -rw-r--r-- 1 mkoeppe staff 545293 Sep 8 12:44 Pillow-7.2.0-cp37-cp37m-macosx_10_9_x86_64.whl -rw-r--r-- 1 mkoeppe staff 4573510 Sep 8 12:06 numpy-1.19.1-cp37-cp37m-macosx_10_9_x86_64.whl -rw-r--r-- 1 mkoeppe staff 2237 Sep 8 12:44 sage_conf-9.2b12-py3-none-any.whl -rw-r--r-- 1 mkoeppe staff 17981103 Sep 8 12:59 scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl
Users can create virtual environments and install the wheels built by the Sage distribution into them, using standard tools such as pip install --find-links
.
- Example:
$ python3 -m venv some-venv $ cd some-venv/ $ bin/pip3 install -v --no-index --find-links=../local/var/lib/sage/wheels/ Pillow
- In follow-up ticket #30527, we create a PEP 503 simple repository for the wheels built during installation, which enables more convenient installation options for users.
In this ticket, we keep using the DESTDIR
staging mechanism also for the installation of the package from the wheel.
- In a follow-up ticket, we can change this -- to remove the problems described in #29585 -- by just holding the pip lock (if it is at all still necessary after the major pip upgrades in the 9.2 series) during the wheel installation (step 2).
References:
Change History (42)
comment:1 follow-up: ↓ 20 Changed 2 years ago by
- Cc slelievre added
comment:2 Changed 23 months ago by
- Milestone changed from sage-9.2 to sage-9.3
comment:3 Changed 22 months ago by
- Description modified (diff)
- Summary changed from Install all Python packages using pip to Install all Python packages via pip wheel
comment:4 Changed 22 months ago by
- Cc gh-tobiasdiez added
comment:5 Changed 22 months ago by
... Also consider pipenv + pipfile as suggested in #30317 ...
comment:6 Changed 22 months ago by
- Cc embray added
comment:7 Changed 22 months ago by
I would prefer not to have two separate package installation and management systems. If the only motivation of this is the race condition, like I said this can be solved with an additional file lock to prevent multiple Python packages from being installed simultaneously.
comment:8 Changed 22 months ago by
- Description modified (diff)
- Summary changed from Install all Python packages via pip wheel to Install all Python packages via pip wheel, create PEP 503 simple repository for wheels
comment:9 Changed 22 months ago by
- Description modified (diff)
comment:10 Changed 22 months ago by
- Description modified (diff)
comment:11 Changed 22 months ago by
- Description modified (diff)
comment:12 Changed 22 months ago by
- Dependencies set to #30024
comment:13 Changed 22 months ago by
- Description modified (diff)
comment:14 Changed 22 months ago by
- Description modified (diff)
comment:16 Changed 22 months ago by
- Branch set to u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels
comment:17 Changed 22 months ago by
- Commit set to 8aa6fd96607c207a0f9563c583978cd50ddc52a7
- Description modified (diff)
- Summary changed from Install all Python packages via pip wheel, create PEP 503 simple repository for wheels to Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels
New commits:
8aa6fd9 | build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it
|
comment:18 Changed 22 months ago by
- Cc jhpalmieri added
- Description modified (diff)
- Status changed from new to needs_review
comment:19 Changed 22 months ago by
- Dependencies #30024 deleted
comment:20 in reply to: ↑ 1 Changed 22 months ago by
- Status changed from needs_review to needs_work
Replying to slelievre:
Would that be
gambit
,numpy
andpillow
?And is this how to find them?
$ git grep "setup.py" build/pkgs
Yes, still need to take care of these.
comment:21 Changed 22 months ago by
- Commit changed from 8aa6fd96607c207a0f9563c583978cd50ddc52a7 to ca586938cbdd91114a10dd3364201c8076412485
Branch pushed to git repo; I updated commit sha1. New commits:
d369aab | build/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install
|
2d435ab | build/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel
|
55993b6 | build/bin/sage-dist-helpers: Fixup
|
0a64674 | build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
|
ca58693 | build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
|
comment:22 Changed 22 months ago by
- Description modified (diff)
comment:23 Changed 22 months ago by
- Status changed from needs_work to needs_review
comment:24 Changed 22 months ago by
- Description modified (diff)
comment:25 Changed 22 months ago by
- Reviewers set to Tobias Diez, ...
- Summary changed from Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels to Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels
comment:26 Changed 22 months ago by
- Commit changed from ca586938cbdd91114a10dd3364201c8076412485 to 5a747c48eca1be0104c202e4922e5386aa794d08
Branch pushed to git repo; I updated commit sha1. New commits:
5a747c4 | build/bin/sage-pip-{install,uninstall}: Remove pip2 support
|
comment:27 Changed 22 months ago by
- Cc tscrim added
comment:28 follow-up: ↓ 29 Changed 22 months ago by
- Why
sdh_store_and_pip_install_wheel requires . as only argument
? Why not just call it with no arguments?
- This function should be documented in
developer/packaging.rst
.
- Does
sage-pip-uninstall
leave the old wheels lying around?
- Typo in
sage-pip-uninstall
(copied fromsage-pip-install
): "this command hould use"
comment:29 in reply to: ↑ 28 ; follow-up: ↓ 30 Changed 22 months ago by
Replying to jhpalmieri:
- Why
sdh_store_and_pip_install_wheel requires . as only argument
? Why not just call it with no arguments?
I did this to emphasize the analogy to sage-pip-install
.
- This function should be documented in
developer/packaging.rst
.
Good idea, will do.
- Does
sage-pip-uninstall
leave the old wheels lying around?
sage-pip-uninstall
is not really used for regular uninstalls. Rather, it "force-uninstalls" previously installed versions - in case the site-packages directory was somehow messed up. (This code was previously part of sage-pip-install
.)
The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean
targets etc. automatically install the wheel.
- Typo in
sage-pip-uninstall
(copied fromsage-pip-install
): "this command hould use"
Thanks
comment:30 in reply to: ↑ 29 Changed 22 months ago by
Replying to mkoeppe:
The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the
-clean
targets etc. automatically install the wheel.
To clarify, both the wheels and the installation made from the wheels uses the staging mechanism.
comment:31 Changed 22 months ago by
- Commit changed from 5a747c48eca1be0104c202e4922e5386aa794d08 to 4135e8b236d2e5491b3ee96cbd00196dc5b6e036
Branch pushed to git repo; I updated commit sha1. New commits:
9ee2110 | build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
|
d7aac84 | src/doc/en/developer/packaging.rst: Update sdh_... documentation
|
9b7c7a0 | build/bin/sage-pip-{install,uninstall}: Fix typo in comment
|
4135e8b | build/bin/sage-pip-install: Remove an outdated comment
|
comment:32 Changed 22 months ago by
- Reviewers changed from Tobias Diez, ... to Tobias Diez, John Palmieri
- Status changed from needs_review to positive_review
Seems okay to me, and it built and passed tests before, so I think it's okay. Of course, I can no longer build Sage anymore, so perhaps take this review with a grain of salt.
comment:33 Changed 22 months ago by
Thanks!
comment:34 Changed 21 months ago by
- Status changed from positive_review to needs_work
Merge conflict
comment:35 Changed 21 months ago by
- Commit changed from 4135e8b236d2e5491b3ee96cbd00196dc5b6e036 to f2e707561ab3044221b57151ae4cc67d502fe9f8
Branch pushed to git repo; I updated commit sha1. New commits:
f2e7075 | Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels
|
comment:36 Changed 21 months ago by
- Status changed from needs_work to positive_review
comment:37 Changed 21 months ago by
- Branch changed from u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels to f2e707561ab3044221b57151ae4cc67d502fe9f8
- Resolution set to fixed
- Status changed from positive_review to closed
comment:38 Changed 21 months ago by
- Milestone changed from sage-9.3 to sage-9.2
comment:39 Changed 21 months ago by
- Commit f2e707561ab3044221b57151ae4cc67d502fe9f8 deleted
Follow-up: #30771
comment:40 follow-ups: ↓ 41 ↓ 42 Changed 20 months ago by
I'm not really sure what the advantage is of this complication. For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/
? I would very much prefer to not do that. I see no reason for Python packages to be arbitrarily treated differently in this regard. But I'm not sure if that is really the plan.
comment:41 in reply to: ↑ 40 Changed 20 months ago by
Replying to embray:
I'm not really sure what the advantage is of this complication.
https://wiki.sagemath.org/ReleaseTours/sage-9.2#Reusable_wheels_for_the_Python_packages_built_by_the_Sage_distribution explains an immediate use case.
comment:42 in reply to: ↑ 40 Changed 20 months ago by
Replying to embray:
For the follow-up do you propose eliminating the installation manifests installed to
/var/lib/sage/installed/
?
I don't have a ticket for this at the moment. But I think it's unavoidable to make some changes. Already now these installation manifests (which duplicate the .egg-info
record in site-packages
) can fall out of sync with the actual installation when the user uses sage -pip install
, which could lead to upgrades of some python package dependencies. It's best to use standard Python tools to manage Python packages.
Would that be
gambit
,numpy
andpillow
?And is this how to find them?