Opened 2 years ago

Closed 21 months ago

Last modified 20 months ago

#29500 closed enhancement (fixed)

Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.2
Component: build Keywords:
Cc: slelievre, gh-tobiasdiez, embray, jhpalmieri, tscrim Merged in:
Authors: Matthias Koeppe Reviewers: Tobias Diez, John Palmieri
Report Upstream: N/A Work issues:
Branch: f2e7075 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

Our current version of pip already tries to install packages via bdist_wheel.

In this ticket, we break the build process into these steps by modifying sdh_pip_install:

  1. Build the wheel explicitly using pip wheel --no-binary :all: --no-build-isolation -w "$SAGE_SPKG_WHEELS" (or setup.py bdist_wheel), which stores the wheel in SAGE_SPKG_WHEELS=$SAGE_LOCAL/var/lib/sage/wheels.
  1. Then install the wheel.

The wheels in $SAGE_SPKG_WHEELS persist after the completed build. We manage them using the DESTDIR staging mechanism -- there will be exactly 1 wheel for each installed package (and removing a package removes the wheel). Example (after rebuilding a few packages on this branch):

$ ls -l local/var/lib/sage/wheels/
...
-rw-r--r--  1 mkoeppe  staff    545293 Sep  8 12:44 Pillow-7.2.0-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff   4573510 Sep  8 12:06 numpy-1.19.1-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff      2237 Sep  8 12:44 sage_conf-9.2b12-py3-none-any.whl
-rw-r--r--  1 mkoeppe  staff  17981103 Sep  8 12:59 scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl

Users can create virtual environments and install the wheels built by the Sage distribution into them, using standard tools such as pip install --find-links.

  • Example:
$ python3 -m venv some-venv
$ cd some-venv/
$ bin/pip3 install -v --no-index --find-links=../local/var/lib/sage/wheels/ Pillow    
  • In follow-up ticket #30527, we create a PEP 503 simple repository for the wheels built during installation, which enables more convenient installation options for users.

In this ticket, we keep using the DESTDIR staging mechanism also for the installation of the package from the wheel.

  • In a follow-up ticket, we can change this -- to remove the problems described in #29585 -- by just holding the pip lock (if it is at all still necessary after the major pip upgrades in the 9.2 series) during the wheel installation (step 2).

References:

Change History (42)

comment:1 follow-up: Changed 2 years ago by slelievre

  • Cc slelievre added

Would that be gambit, numpy and pillow?

And is this how to find them?

$ git grep "setup.py" build/pkgs

comment:2 Changed 23 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:3 Changed 22 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Install all Python packages using pip to Install all Python packages via pip wheel

comment:4 Changed 22 months ago by mkoeppe

  • Cc gh-tobiasdiez added

comment:5 Changed 22 months ago by mkoeppe

... Also consider pipenv + pipfile as suggested in #30317 ...

comment:6 Changed 22 months ago by mkoeppe

  • Cc embray added

comment:7 Changed 22 months ago by embray

I would prefer not to have two separate package installation and management systems. If the only motivation of this is the race condition, like I said this can be solved with an additional file lock to prevent multiple Python packages from being installed simultaneously.

comment:8 Changed 22 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Install all Python packages via pip wheel to Install all Python packages via pip wheel, create PEP 503 simple repository for wheels

comment:9 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:10 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:12 Changed 22 months ago by mkoeppe

  • Dependencies set to #30024

comment:13 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:14 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:15 Changed 22 months ago by mkoeppe

  • Description modified (diff)

Narrowed the scope of the ticket.

comment:16 Changed 22 months ago by mkoeppe

  • Branch set to u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels

comment:17 Changed 22 months ago by mkoeppe

  • Commit set to 8aa6fd96607c207a0f9563c583978cd50ddc52a7
  • Description modified (diff)
  • Summary changed from Install all Python packages via pip wheel, create PEP 503 simple repository for wheels to Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels

New commits:

8aa6fd9build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it

comment:18 Changed 22 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Cc jhpalmieri added
  • Description modified (diff)
  • Status changed from new to needs_review

comment:19 Changed 22 months ago by mkoeppe

  • Dependencies #30024 deleted

comment:20 in reply to: ↑ 1 Changed 22 months ago by mkoeppe

  • Status changed from needs_review to needs_work

Replying to slelievre:

Would that be gambit, numpy and pillow?

And is this how to find them?

$ git grep "setup.py" build/pkgs

Yes, still need to take care of these.

comment:21 Changed 22 months ago by git

  • Commit changed from 8aa6fd96607c207a0f9563c583978cd50ddc52a7 to ca586938cbdd91114a10dd3364201c8076412485

Branch pushed to git repo; I updated commit sha1. New commits:

d369aabbuild/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install
2d435abbuild/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel
55993b6build/bin/sage-dist-helpers: Fixup
0a64674build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel

comment:22 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:23 Changed 22 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:24 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:25 Changed 22 months ago by mkoeppe

  • Reviewers set to Tobias Diez, ...
  • Summary changed from Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels to Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels

comment:26 Changed 22 months ago by git

  • Commit changed from ca586938cbdd91114a10dd3364201c8076412485 to 5a747c48eca1be0104c202e4922e5386aa794d08

Branch pushed to git repo; I updated commit sha1. New commits:

5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support

comment:27 Changed 22 months ago by mkoeppe

  • Cc tscrim added

comment:28 follow-up: Changed 22 months ago by jhpalmieri

  • Why sdh_store_and_pip_install_wheel requires . as only argument? Why not just call it with no arguments?
  • This function should be documented in developer/packaging.rst.
  • Does sage-pip-uninstall leave the old wheels lying around?
  • Typo in sage-pip-uninstall (copied from sage-pip-install): "this command hould use"

comment:29 in reply to: ↑ 28 ; follow-up: Changed 22 months ago by mkoeppe

Replying to jhpalmieri:

  • Why sdh_store_and_pip_install_wheel requires . as only argument? Why not just call it with no arguments?

I did this to emphasize the analogy to sage-pip-install.

  • This function should be documented in developer/packaging.rst.

Good idea, will do.

  • Does sage-pip-uninstall leave the old wheels lying around?

sage-pip-uninstall is not really used for regular uninstalls. Rather, it "force-uninstalls" previously installed versions - in case the site-packages directory was somehow messed up. (This code was previously part of sage-pip-install.)

The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean targets etc. automatically install the wheel.

  • Typo in sage-pip-uninstall (copied from sage-pip-install): "this command hould use"

Thanks

comment:30 in reply to: ↑ 29 Changed 22 months ago by mkoeppe

Replying to mkoeppe:

The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean targets etc. automatically install the wheel.

To clarify, both the wheels and the installation made from the wheels uses the staging mechanism.

comment:31 Changed 22 months ago by git

  • Commit changed from 5a747c48eca1be0104c202e4922e5386aa794d08 to 4135e8b236d2e5491b3ee96cbd00196dc5b6e036

Branch pushed to git repo; I updated commit sha1. New commits:

9ee2110build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
d7aac84src/doc/en/developer/packaging.rst: Update sdh_... documentation
9b7c7a0build/bin/sage-pip-{install,uninstall}: Fix typo in comment
4135e8bbuild/bin/sage-pip-install: Remove an outdated comment

comment:32 Changed 22 months ago by jhpalmieri

  • Reviewers changed from Tobias Diez, ... to Tobias Diez, John Palmieri
  • Status changed from needs_review to positive_review

Seems okay to me, and it built and passed tests before, so I think it's okay. Of course, I can no longer build Sage anymore, so perhaps take this review with a grain of salt.

comment:33 Changed 22 months ago by mkoeppe

Thanks!

comment:34 Changed 21 months ago by vbraun

  • Status changed from positive_review to needs_work

Merge conflict

comment:35 Changed 21 months ago by git

  • Commit changed from 4135e8b236d2e5491b3ee96cbd00196dc5b6e036 to f2e707561ab3044221b57151ae4cc67d502fe9f8

Branch pushed to git repo; I updated commit sha1. New commits:

f2e7075Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels

comment:36 Changed 21 months ago by mkoeppe

  • Status changed from needs_work to positive_review

comment:37 Changed 21 months ago by vbraun

  • Branch changed from u/mkoeppe/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels to f2e707561ab3044221b57151ae4cc67d502fe9f8
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:38 Changed 21 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.2

comment:39 Changed 21 months ago by mkoeppe

  • Commit f2e707561ab3044221b57151ae4cc67d502fe9f8 deleted

Follow-up: #30771

comment:40 follow-ups: Changed 20 months ago by embray

I'm not really sure what the advantage is of this complication. For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/? I would very much prefer to not do that. I see no reason for Python packages to be arbitrarily treated differently in this regard. But I'm not sure if that is really the plan.

comment:41 in reply to: ↑ 40 Changed 20 months ago by mkoeppe

Replying to embray:

I'm not really sure what the advantage is of this complication.

https://wiki.sagemath.org/ReleaseTours/sage-9.2#Reusable_wheels_for_the_Python_packages_built_by_the_Sage_distribution explains an immediate use case.

comment:42 in reply to: ↑ 40 Changed 20 months ago by mkoeppe

Replying to embray:

For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/?

I don't have a ticket for this at the moment. But I think it's unavoidable to make some changes. Already now these installation manifests (which duplicate the .egg-info record in site-packages) can fall out of sync with the actual installation when the user uses sage -pip install, which could lead to upgrades of some python package dependencies. It's best to use standard Python tools to manage Python packages.

Note: See TracTickets for help on using tickets.