#29500 closed enhancement (fixed)
Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels
Reported by:  Matthias Köppe  Owned by:  

Priority:  major  Milestone:  sage9.2 
Component:  build  Keywords:  
Cc:  Samuel Lelièvre, Tobias Diez, Erik Bray, John Palmieri, Travis Scrimshaw  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  Tobias Diez, John Palmieri 
Report Upstream:  N/A  Work issues:  
Branch:  f2e7075 (Commits, GitHub, GitLab)  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
Our current version of pip already tries to install packages via bdist_wheel
.
In this ticket, we break the build process into these steps by modifying sdh_pip_install
:
 Build the wheel explicitly using
pip wheel nobinary :all: nobuildisolation w "$SAGE_SPKG_WHEELS"
(orsetup.py bdist_wheel
), which stores the wheel inSAGE_SPKG_WHEELS=$SAGE_LOCAL/var/lib/sage/wheels
.
 Then install the wheel.
The wheels in $SAGE_SPKG_WHEELS
persist after the completed build. We manage them using the DESTDIR
staging mechanism  there will be exactly 1 wheel for each installed package (and removing a package removes the wheel). Example (after rebuilding a few packages on this branch):
$ ls l local/var/lib/sage/wheels/ ... rwrr 1 mkoeppe staff 545293 Sep 8 12:44 Pillow7.2.0cp37cp37mmacosx_10_9_x86_64.whl rwrr 1 mkoeppe staff 4573510 Sep 8 12:06 numpy1.19.1cp37cp37mmacosx_10_9_x86_64.whl rwrr 1 mkoeppe staff 2237 Sep 8 12:44 sage_conf9.2b12py3noneany.whl rwrr 1 mkoeppe staff 17981103 Sep 8 12:59 scipy1.5.2cp37cp37mmacosx_10_9_x86_64.whl
Users can create virtual environments and install the wheels built by the Sage distribution into them, using standard tools such as pip install findlinks
.
 Example:
$ python3 m venv somevenv $ cd somevenv/ $ bin/pip3 install v noindex findlinks=../local/var/lib/sage/wheels/ Pillow
 In followup ticket #30527, we create a PEP 503 simple repository for the wheels built during installation, which enables more convenient installation options for users.
In this ticket, we keep using the DESTDIR
staging mechanism also for the installation of the package from the wheel.
 In a followup ticket, we can change this  to remove the problems described in #29585  by just holding the pip lock (if it is at all still necessary after the major pip upgrades in the 9.2 series) during the wheel installation (step 2).
References:
Change History (42)
comment:1 followup: 20 Changed 3 years ago by
Cc:  Samuel Lelièvre added 

comment:2 Changed 2 years ago by
Milestone:  sage9.2 → sage9.3 

comment:3 Changed 2 years ago by
Description:  modified (diff) 

Summary:  Install all Python packages using pip → Install all Python packages via pip wheel 
comment:4 Changed 2 years ago by
Cc:  Tobias Diez added 

comment:6 Changed 2 years ago by
Cc:  Erik Bray added 

comment:7 Changed 2 years ago by
I would prefer not to have two separate package installation and management systems. If the only motivation of this is the race condition, like I said this can be solved with an additional file lock to prevent multiple Python packages from being installed simultaneously.
comment:8 Changed 2 years ago by
Description:  modified (diff) 

Summary:  Install all Python packages via pip wheel → Install all Python packages via pip wheel, create PEP 503 simple repository for wheels 
comment:9 Changed 2 years ago by
Description:  modified (diff) 

comment:10 Changed 2 years ago by
Description:  modified (diff) 

comment:11 Changed 2 years ago by
Description:  modified (diff) 

comment:12 Changed 2 years ago by
Dependencies:  → #30024 

comment:13 Changed 2 years ago by
Description:  modified (diff) 

comment:14 Changed 2 years ago by
Description:  modified (diff) 

comment:16 Changed 2 years ago by
Branch:  → u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels 

comment:17 Changed 2 years ago by
Commit:  → 8aa6fd96607c207a0f9563c583978cd50ddc52a7 

Description:  modified (diff) 
Summary:  Install all Python packages via pip wheel, create PEP 503 simple repository for wheels → Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels 
New commits:
8aa6fd9  build/bin/sagedisthelpers (sdh_pip_install): Build a wheel, store it

comment:18 Changed 2 years ago by
Authors:  → Matthias Koeppe 

Cc:  John Palmieri added 
Description:  modified (diff) 
Status:  new → needs_review 
comment:19 Changed 2 years ago by
Dependencies:  #30024 

comment:20 Changed 2 years ago by
Status:  needs_review → needs_work 

Replying to slelievre:
Would that be
gambit
,numpy
andpillow
?And is this how to find them?
$ git grep "setup.py" build/pkgs
Yes, still need to take care of these.
comment:21 Changed 2 years ago by
Commit:  8aa6fd96607c207a0f9563c583978cd50ddc52a7 → ca586938cbdd91114a10dd3364201c8076412485 

Branch pushed to git repo; I updated commit sha1. New commits:
d369aab  build/bin/sagedisthelpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install

2d435ab  build/pkgs/numpy/spkginstall.in: Install via setup.py bdist_wheel

55993b6  build/bin/sagedisthelpers: Fixup

0a64674  build/pkgs/gambit/spkginstall.in: Install via bdist_wheel

ca58693  build/pkgs/pillow/spkginstall.in: Install via bdist_wheel

comment:22 Changed 2 years ago by
Description:  modified (diff) 

comment:23 Changed 2 years ago by
Status:  needs_work → needs_review 

comment:24 Changed 2 years ago by
Description:  modified (diff) 

comment:25 Changed 2 years ago by
Reviewers:  → Tobias Diez, ... 

Summary:  Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels → Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels 
comment:26 Changed 2 years ago by
Commit:  ca586938cbdd91114a10dd3364201c8076412485 → 5a747c48eca1be0104c202e4922e5386aa794d08 

Branch pushed to git repo; I updated commit sha1. New commits:
5a747c4  build/bin/sagepip{install,uninstall}: Remove pip2 support

comment:27 Changed 2 years ago by
Cc:  Travis Scrimshaw added 

comment:28 followup: 29 Changed 2 years ago by
 Why
sdh_store_and_pip_install_wheel requires . as only argument
? Why not just call it with no arguments?
 This function should be documented in
developer/packaging.rst
.
 Does
sagepipuninstall
leave the old wheels lying around?
 Typo in
sagepipuninstall
(copied fromsagepipinstall
): "this command hould use"
comment:29 followup: 30 Changed 2 years ago by
Replying to jhpalmieri:
 Why
sdh_store_and_pip_install_wheel requires . as only argument
? Why not just call it with no arguments?
I did this to emphasize the analogy to sagepipinstall
.
 This function should be documented in
developer/packaging.rst
.
Good idea, will do.
 Does
sagepipuninstall
leave the old wheels lying around?
sagepipuninstall
is not really used for regular uninstalls. Rather, it "forceuninstalls" previously installed versions  in case the sitepackages directory was somehow messed up. (This code was previously part of sagepipinstall
.)
The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the clean
targets etc. automatically install the wheel.
 Typo in
sagepipuninstall
(copied fromsagepipinstall
): "this command hould use"
Thanks
comment:30 Changed 2 years ago by
Replying to mkoeppe:
The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the
clean
targets etc. automatically install the wheel.
To clarify, both the wheels and the installation made from the wheels uses the staging mechanism.
comment:31 Changed 2 years ago by
Commit:  5a747c48eca1be0104c202e4922e5386aa794d08 → 4135e8b236d2e5491b3ee96cbd00196dc5b6e036 

Branch pushed to git repo; I updated commit sha1. New commits:
9ee2110  build/bin/sagedisthelpers: Also use $sudo for storing the wheel file

d7aac84  src/doc/en/developer/packaging.rst: Update sdh_... documentation

9b7c7a0  build/bin/sagepip{install,uninstall}: Fix typo in comment

4135e8b  build/bin/sagepipinstall: Remove an outdated comment

comment:32 Changed 2 years ago by
Reviewers:  Tobias Diez, ... → Tobias Diez, John Palmieri 

Status:  needs_review → positive_review 
Seems okay to me, and it built and passed tests before, so I think it's okay. Of course, I can no longer build Sage anymore, so perhaps take this review with a grain of salt.
comment:35 Changed 2 years ago by
Commit:  4135e8b236d2e5491b3ee96cbd00196dc5b6e036 → f2e707561ab3044221b57151ae4cc67d502fe9f8 

Branch pushed to git repo; I updated commit sha1. New commits:
f2e7075  Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels

comment:36 Changed 2 years ago by
Status:  needs_work → positive_review 

comment:37 Changed 2 years ago by
Branch:  u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels → f2e707561ab3044221b57151ae4cc67d502fe9f8 

Resolution:  → fixed 
Status:  positive_review → closed 
comment:38 Changed 2 years ago by
Milestone:  sage9.3 → sage9.2 

comment:39 Changed 2 years ago by
Commit:  f2e707561ab3044221b57151ae4cc67d502fe9f8 

Followup: #30771
comment:40 followups: 41 42 Changed 2 years ago by
I'm not really sure what the advantage is of this complication. For the followup do you propose eliminating the installation manifests installed to /var/lib/sage/installed/
? I would very much prefer to not do that. I see no reason for Python packages to be arbitrarily treated differently in this regard. But I'm not sure if that is really the plan.
comment:41 Changed 2 years ago by
Replying to embray:
I'm not really sure what the advantage is of this complication.
https://wiki.sagemath.org/ReleaseTours/sage9.2#Reusable_wheels_for_the_Python_packages_built_by_the_Sage_distribution explains an immediate use case.
comment:42 Changed 2 years ago by
Replying to embray:
For the followup do you propose eliminating the installation manifests installed to
/var/lib/sage/installed/
?
I don't have a ticket for this at the moment. But I think it's unavoidable to make some changes. Already now these installation manifests (which duplicate the .egginfo
record in sitepackages
) can fall out of sync with the actual installation when the user uses sage pip install
, which could lead to upgrades of some python package dependencies. It's best to use standard Python tools to manage Python packages.
Would that be
gambit
,numpy
andpillow
?And is this how to find them?