Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels

[Matthias Köppe]

Our current version of pip already tries to install packages via bdist_wheel.

In this ticket, we break the build process into these steps by modifying sdh_pip_install:

1. Build the wheel explicitly using pip wheel --no-binary :all: --no-build-isolation -w "$SAGE_SPKG_WHEELS" (or setup.py bdist_wheel), which stores the wheel in SAGE_SPKG_WHEELS=$SAGE_LOCAL/var/lib/sage/wheels.

2. Then install the wheel.

The wheels in $SAGE_SPKG_WHEELS persist after the completed build. We manage them using the DESTDIR staging mechanism -- there will be exactly 1 wheel for each installed package (and removing a package removes the wheel). Example (after rebuilding a few packages on this branch):

$ ls -l local/var/lib/sage/wheels/
...
-rw-r--r--  1 mkoeppe  staff    545293 Sep  8 12:44 Pillow-7.2.0-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff   4573510 Sep  8 12:06 numpy-1.19.1-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff      2237 Sep  8 12:44 sage_conf-9.2b12-py3-none-any.whl
-rw-r--r--  1 mkoeppe  staff  17981103 Sep  8 12:59 scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl

Users can create virtual environments and install the wheels built by the Sage distribution into them, using standard tools such as pip install --find-links.

• Example:

$ python3 -m venv some-venv
$ cd some-venv/
$ bin/pip3 install -v --no-index --find-links=../local/var/lib/sage/wheels/ Pillow    

• In follow-up ticket #30527, we create a PEP 503 simple repository for the wheels built during installation, which enables more convenient installation options for users.

In this ticket, we keep using the DESTDIR staging mechanism also for the installation of the package from the wheel.

• In a follow-up ticket, we can change this -- to remove the problems described in #29585 -- by just holding the pip lock (if it is at all still necessary after the major pip upgrades in the 9.2 series) during the wheel installation (step 2).

---

References:

• ​https://pip.pypa.io/en/stable/user_guide/#installing-from-local-packages

• ​https://pip.pypa.io/en/stable/reference/pip_cache/

[Samuel Lelièvre]

Would that be gambit, numpy and pillow?

And is this how to find them?

$ git grep "setup.py" build/pkgs

[Matthias Köppe]

... Also consider pipenv + pipfile as suggested in #30317 ...

[Erik Bray]

I would prefer not to have two separate package installation and management systems. If the only motivation of this is the race condition, like I said this can be solved with an additional file lock to prevent multiple Python packages from being installed simultaneously.

[Matthias Köppe]

Narrowed the scope of the ticket.

[Matthias Köppe]

New commits:

​8aa6fd9

build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it

[Matthias Köppe]

Replying to slelievre:

Would that be gambit, numpy and pillow?

And is this how to find them?

$ git grep "setup.py" build/pkgs

Yes, still need to take care of these.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d369aab	build/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install
​2d435ab	build/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel
​55993b6	build/bin/sage-dist-helpers: Fixup
​0a64674	build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
​ca58693	build/pkgs/pillow/spkg-install.in: Install via bdist_wheel

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5a747c4

build/bin/sage-pip-{install,uninstall}: Remove pip2 support

[John Palmieri]

• Why sdh_store_and_pip_install_wheel requires . as only argument? Why not just call it with no arguments?

• This function should be documented in developer/packaging.rst.

• Does sage-pip-uninstall leave the old wheels lying around?

• Typo in sage-pip-uninstall (copied from sage-pip-install): "this command hould use"

[Matthias Köppe]

Replying to jhpalmieri:

• Why sdh_store_and_pip_install_wheel requires . as only argument? Why not just call it with no arguments?

I did this to emphasize the analogy to sage-pip-install.

• This function should be documented in developer/packaging.rst.

Good idea, will do.

• Does sage-pip-uninstall leave the old wheels lying around?

sage-pip-uninstall is not really used for regular uninstalls. Rather, it "force-uninstalls" previously installed versions - in case the site-packages directory was somehow messed up. (This code was previously part of sage-pip-install.)

The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean targets etc. automatically install the wheel.

• Typo in sage-pip-uninstall (copied from sage-pip-install): "this command hould use"

Thanks

[Matthias Köppe]

Replying to mkoeppe:

The wheels are installed with the staging mechanism (DESTDIR), which records the installed files. In this way, the -clean targets etc. automatically install the wheel.

To clarify, both the wheels and the installation made from the wheels uses the staging mechanism.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9ee2110	build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
​d7aac84	src/doc/en/developer/packaging.rst: Update sdh_... documentation
​9b7c7a0	build/bin/sage-pip-{install,uninstall}: Fix typo in comment
​4135e8b	build/bin/sage-pip-install: Remove an outdated comment

[John Palmieri]

Seems okay to me, and it built and passed tests before, so I think it's okay. Of course, I can ​no longer build Sage anymore, so perhaps take this review with a grain of salt.

[Matthias Köppe]

Thanks!

[Volker Braun]

Merge conflict

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f2e7075

Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels

[Matthias Köppe]

Follow-up: #30771

[Erik Bray]

I'm not really sure what the advantage is of this complication. For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/? I would very much prefer to not do that. I see no reason for Python packages to be arbitrarily treated differently in this regard. But I'm not sure if that is really the plan.

[Matthias Köppe]

Replying to embray:

I'm not really sure what the advantage is of this complication.

​https://wiki.sagemath.org/ReleaseTours/sage-9.2#Reusable_wheels_for_the_Python_packages_built_by_the_Sage_distribution explains an immediate use case.

[Matthias Köppe]

Replying to embray:

For the follow-up do you propose eliminating the installation manifests installed to /var/lib/sage/installed/?

I don't have a ticket for this at the moment. But I think it's unavoidable to make some changes. Already now these installation manifests (which duplicate the .egg-info record in site-packages) can fall out of sync with the actual installation when the user uses sage -pip install, which could lead to upgrades of some python package dependencies. It's best to use standard Python tools to manage Python packages.