Opened 3 years ago

Closed 2 years ago

Last modified 2 years ago

#29500 closed enhancement (fixed)

Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.2
Component: build Keywords:
Cc: Samuel Lelièvre, Tobias Diez, Erik Bray, John Palmieri, Travis Scrimshaw Merged in:
Authors: Matthias Koeppe Reviewers: Tobias Diez, John Palmieri
Report Upstream: N/A Work issues:
Branch: f2e7075 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

Our current version of pip already tries to install packages via bdist_wheel.

In this ticket, we break the build process into these steps by modifying sdh_pip_install:

  1. Build the wheel explicitly using pip wheel --no-binary :all: --no-build-isolation -w "$SAGE_SPKG_WHEELS" (or setup.py bdist_wheel), which stores the wheel in SAGE_SPKG_WHEELS=$SAGE_LOCAL/var/lib/sage/wheels.
  1. Then install the wheel.

The wheels in $SAGE_SPKG_WHEELS persist after the completed build. We manage them using the DESTDIR staging mechanism -- there will be exactly 1 wheel for each installed package (and removing a package removes the wheel). Example (after rebuilding a few packages on this branch):

$ ls -l local/var/lib/sage/wheels/
...
-rw-r--r--  1 mkoeppe  staff    545293 Sep  8 12:44 Pillow-7.2.0-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff   4573510 Sep  8 12:06 numpy-1.19.1-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff      2237 Sep  8 12:44 sage_conf-9.2b12-py3-none-any.whl
-rw-r--r--  1 mkoeppe  staff  17981103 Sep  8 12:59 scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl

Users can create virtual environments and install the wheels built by the Sage distribution into them, using standard tools such as pip install --find-links.

  • Example:
$ python3 -m venv some-venv
$ cd some-venv/
$ bin/pip3 install -v --no-index --find-links=../local/var/lib/sage/wheels/ Pillow    
  • In follow-up ticket #30527, we create a PEP 503 simple repository for the wheels built during installation, which enables more convenient installation options for users.

In this ticket, we keep using the DESTDIR staging mechanism also for the installation of the package from the wheel.

  • In a follow-up ticket, we can change this -- to remove the problems described in #29585 -- by just holding the pip lock (if it is at all still necessary after the major pip upgrades in the 9.2 series) during the wheel installation (step 2).

References:

Change History (42)

comment:1 Changed 3 years ago by Samuel Lelièvre

Cc: Samuel Lelièvre added

Would that be gambit, numpy and pillow?

And is this how to find them?

$ git grep "setup.py" build/pkgs

comment:2 Changed 2 years ago by Matthias Köppe

Milestone: sage-9.2sage-9.3

comment:3 Changed 2 years ago by Matthias Köppe

Description: modified (diff)
Summary: Install all Python packages using pipInstall all Python packages via pip wheel

comment:4 Changed 2 years ago by Matthias Köppe

Cc: Tobias Diez added

comment:5 Changed 2 years ago by Matthias Köppe

... Also consider pipenv + pipfile as suggested in #30317 ...

comment:6 Changed 2 years ago by Matthias Köppe

Cc: Erik Bray added

comment:7 Changed 2 years ago by Erik Bray

I would prefer not to have two separate package installation and management systems. If the only motivation of this is the race condition, like I said this can be solved with an additional file lock to prevent multiple Python packages from being installed simultaneously.

comment:8 Changed 2 years ago by Matthias Köppe

Description: modified (diff)
Summary: Install all Python packages via pip wheelInstall all Python packages via pip wheel, create PEP 503 simple repository for wheels

comment:9 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:10 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:11 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:12 Changed 2 years ago by Matthias Köppe

Dependencies: #30024

comment:13 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:14 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:15 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

Narrowed the scope of the ticket.

comment:16 Changed 2 years ago by Matthias Köppe

Branch: u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels

comment:17 Changed 2 years ago by Matthias Köppe

Commit: 8aa6fd96607c207a0f9563c583978cd50ddc52a7
Description: modified (diff)
Summary: Install all Python packages via pip wheel, create PEP 503 simple repository for wheelsInstall all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels

New commits:

8aa6fd9build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it

comment:18 Changed 2 years ago by Matthias Köppe

Authors: Matthias Koeppe
Cc: John Palmieri added
Description: modified (diff)
Status: newneeds_review

comment:19 Changed 2 years ago by Matthias Köppe

Dependencies: #30024

comment:20 in reply to:  1 Changed 2 years ago by Matthias Köppe

Status: needs_reviewneeds_work

Replying to slelievre:

Would that be gambit, numpy and pillow?

And is this how to find them?

$ git grep "setup.py" build/pkgs

Yes, still need to take care of these.

comment:21 Changed 2 years ago by git

Commit: 8aa6fd96607c207a0f9563c583978cd50ddc52a7ca586938cbdd91114a10dd3364201c8076412485

Branch pushed to git repo; I updated commit sha1. New commits:

d369aabbuild/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install
2d435abbuild/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel
55993b6build/bin/sage-dist-helpers: Fixup
0a64674build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel

comment:22 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:23 Changed 2 years ago by Matthias Köppe

Status: needs_workneeds_review

comment:24 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:25 Changed 2 years ago by Matthias Köppe

Reviewers: Tobias Diez, ...
Summary: Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheelsInstall all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels

comment:26 Changed 2 years ago by git

Commit: ca586938cbdd91114a10dd3364201c80764124855a747c48eca1be0104c202e4922e5386aa794d08

Branch pushed to git repo; I updated commit sha1. New commits:

5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support

comment:27 Changed 2 years ago by Matthias Köppe

Cc: Travis Scrimshaw added

comment:28 Changed 2 years ago by John Palmieri

  • Why sdh_store_and_pip_install_wheel requires . as only argument? Why not just call it with no arguments?
  • This function should be documented in developer/packaging.rst.
  • Does sage-pip-uninstall leave the old wheels lying around?
  • Typo in sage-pip-uninstall (copied from sage-pip-install): "this command hould use"

comment:29 in reply to:  28 ; Changed 2 years ago by Matthias Köppe

Replying to jhpalmieri:

  • Why sdh_store_and_pip_install_wheel requires . as only argument? Why not just call it with no arguments?

I did this to emphasize the analogy to sage-pip-install.

  • This function should be documented in developer/packaging.rst.

Good idea, will do.

  • Does sage-pip-uninstall leave the old wheels lying around?

sage-pip-uninstall is not really used for regular uninstalls. Rather, it "force-uninstalls" previously installed versions - in case the site-packages directory was somehow messed up. (This code was previously part of sage-pip-install.)

The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean targets etc. automatically install the wheel.

  • Typo in sage-pip-uninstall (copied from sage-pip-install): "this command hould use"

Thanks

comment:30 in reply to:  29 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean targets etc. automatically install the wheel.

To clarify, both the wheels and the installation made from the wheels uses the staging mechanism.

comment:31 Changed 2 years ago by git

Commit: 5a747c48eca1be0104c202e4922e5386aa794d084135e8b236d2e5491b3ee96cbd00196dc5b6e036

Branch pushed to git repo; I updated commit sha1. New commits:

9ee2110build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
d7aac84src/doc/en/developer/packaging.rst: Update sdh_... documentation
9b7c7a0build/bin/sage-pip-{install,uninstall}: Fix typo in comment
4135e8bbuild/bin/sage-pip-install: Remove an outdated comment

comment:32 Changed 2 years ago by John Palmieri

Reviewers: Tobias Diez, ...Tobias Diez, John Palmieri
Status: needs_reviewpositive_review

Seems okay to me, and it built and passed tests before, so I think it's okay. Of course, I can no longer build Sage anymore, so perhaps take this review with a grain of salt.

comment:33 Changed 2 years ago by Matthias Köppe

Thanks!

comment:34 Changed 2 years ago by Volker Braun

Status: positive_reviewneeds_work

Merge conflict

comment:35 Changed 2 years ago by git

Commit: 4135e8b236d2e5491b3ee96cbd00196dc5b6e036f2e707561ab3044221b57151ae4cc67d502fe9f8

Branch pushed to git repo; I updated commit sha1. New commits:

f2e7075Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels

comment:36 Changed 2 years ago by Matthias Köppe

Status: needs_workpositive_review

comment:37 Changed 2 years ago by Volker Braun

Branch: u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheelsf2e707561ab3044221b57151ae4cc67d502fe9f8
Resolution: fixed
Status: positive_reviewclosed

comment:38 Changed 2 years ago by Matthias Köppe

Milestone: sage-9.3sage-9.2

comment:39 Changed 2 years ago by Matthias Köppe

Commit: f2e707561ab3044221b57151ae4cc67d502fe9f8

Follow-up: #30771

comment:40 Changed 2 years ago by Erik Bray

I'm not really sure what the advantage is of this complication. For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/? I would very much prefer to not do that. I see no reason for Python packages to be arbitrarily treated differently in this regard. But I'm not sure if that is really the plan.

comment:41 in reply to:  40 Changed 2 years ago by Matthias Köppe

Replying to embray:

I'm not really sure what the advantage is of this complication.

https://wiki.sagemath.org/ReleaseTours/sage-9.2#Reusable_wheels_for_the_Python_packages_built_by_the_Sage_distribution explains an immediate use case.

comment:42 in reply to:  40 Changed 2 years ago by Matthias Köppe

Replying to embray:

For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/?

I don't have a ticket for this at the moment. But I think it's unavoidable to make some changes. Already now these installation manifests (which duplicate the .egg-info record in site-packages) can fall out of sync with the actual installation when the user uses sage -pip install, which could lead to upgrades of some python package dependencies. It's best to use standard Python tools to manage Python packages.

Note: See TracTickets for help on using tickets.