Opened 2 years ago

# Create PEP 503 simple repository for wheels built during installation

Reported by: Owned by: Matthias Köppe major sage-wishlist build sd111 Samuel Lelièvre, Tobias Diez, Erik Bray, John Palmieri Matthias Koeppe Tobias Diez N/A u/mkoeppe/pep-503-simple-repository-for-wheels 333e50c002787d5fc828109388d83f5f9b7bd78c

As a follow-up to #29500 (Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels): We create a PEP 503 compliant Simple Repository index in $SAGE_SPKG_WHEELS. Then users can create their virtual environment using tools such as pipenv [[source]] or pip install --index-url.

We do this using the command dir2pi from https://pypi.org/project/pip2pi/

Follow-up:

• #30578 - Add src/requirements.txt for installation of sagelib in a venv

References:

### comment:1 Changed 2 years ago by Matthias Köppe

piprepo can be used for this task, but it has excessive dependencies

### comment:2 Changed 2 years ago by Matthias Köppe

Dependencies: → #29500

### comment:3 Changed 2 years ago by Tobias Diez

There is also pypiserver, see eg https://stackoverflow.com/a/51528588/873661 for how to use it with pipenv

### comment:4 Changed 2 years ago by Matthias Köppe

Well, we don't want to run a server for this, but following this link I found https://github.com/wolever/pip2pi

### comment:5 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

### comment:6 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

### comment:7 Changed 2 years ago by Tobias Diez

Yeah I agree, a server is not required (although the overhead of creating a simple server on listing on localhost should be pretty minimal). If pip2pi works with pipenv and pip, then that's perfect!

### comment:8 Changed 2 years ago by Matthias Köppe

Branch: → u/mkoeppe/pep-503-simple-repository-for-wheels

### comment:9 Changed 2 years ago by Matthias Köppe

Authors: → Matthias Koeppe → 2555e19712ee6bc2f9b4edf982b5f86d1bb9a6fe new → needs_review

New commits:

 ​8aa6fd9 build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it ​d369aab build/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install ​2d435ab build/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel ​55993b6 build/bin/sage-dist-helpers: Fixup ​0a64674 build/pkgs/gambit/spkg-install.in: Install via bdist_wheel ​ca58693 build/pkgs/pillow/spkg-install.in: Install via bdist_wheel ​207d80f build/pkgs/pip2pi: New ​2555e19 build/make/install: At the end, update the repository index

### comment:10 follow-up:  11 Changed 2 years ago by Tobias Diez

What is the reason that sage needs to build wheels for standard packages, and not just downloads them from pypi?

Is it because some packages have patches applied? In this case, I would propose to migrate these patches to proper forks of the packages which are then published to pypi including their wheel. This has a few advantages, mainly that the build of sage is speedup and simplified, and that the wider python community can profit from these patches.

What do you think?

### comment:11 in reply to:  10 Changed 2 years ago by Matthias Köppe

What is the reason that sage needs to build wheels for standard packages, and not just downloads them from pypi?

Sage-the-distribution is a deployment mechanism. Compiling all packages from source gives us full control and eliminates possible ABI problems from mixing prebuilt wheels that were compiled on different systems. Building from source also has advantages such as generating architecture-specific code (see #27122).

Is it because some packages have patches applied?

In some cases, yes, in other cases, just specific configuration.

In this case, I would propose to migrate these patches to proper forks of the packages which are then published to pypi including their wheel. This has a few advantages, mainly that the build of sage is speedup and simplified, and that the wider python community can profit from these patches.

We don't have intentions to take over maintenance of these packages. Whenever possible, we send our patches upstream. In many cases, it is just a matter of mismatched release schedules, and we have to keep patches only for a limited time until upstream makes a new stable release including a particular patch. For example, consider the numpy patch added in #29766. It is actually from upstream, but not included in any stable release. But we need the patch because we want Sage-the-distribution to support a particular platform.

### comment:12 follow-ups:  13  14  15  16 Changed 2 years ago by Tobias Diez

Reviewers: → Tobias Diez,

Thanks for the detailed explanation. Makes things clearer for me now.

Compiling all packages from source gives us full control and eliminates possible ABI problems from mixing prebuilt wheels that were compiled on different systems. Building from source also has advantages such as generating architecture-specific code

But isn't the concept of compile-once-reuse-often the main point of wheels? By design, it's the responsibility of the the maintainer (= publisher of the wheel) that the wheel is useable and optimized for the particular platforms they are published. This also make sense since she is the export of the code of the library.

I feel none of the advantages of wheels outlined in https://www.python.org/dev/peps/pep-0427/ can really prove effective if wheels are rebuilt from code every time.

The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.

In my opinion, the build process of sage would immensely profit from all these advantages.

Anyway, this discussion is somewhat orthogonal to this ticket (sorry!). So concerning the changes, the code looks good to me. I'm only wondering if the pip2 support in the uninstallation script is really necessary (since only python3 is supported now). However, I don't feel familiar enough with the build script to change the status to positive review.

### comment:13 in reply to:  12 Changed 2 years ago by Matthias Köppe

But isn't the concept of compile-once-reuse-often the main point of wheels? By design, it's the responsibility of the the maintainer (= publisher of the wheel) that the wheel is useable and optimized for the particular platforms they are published. This also make sense since she is the export of the code of the library.

In practice, wheels are offered on PyPI with the broadest possible platform compatibility, typically using https://github.com/pypa/manylinux -- this is the opposite of "optimized".

### comment:14 in reply to:  12 ; follow-up:  18 Changed 2 years ago by Matthias Köppe

I'm only wondering if the pip2 support in the uninstallation script is really necessary (since only python3 is supported now).

Yes, this should be removed, but I prefer to keep tickets narrowly focused. A separate ticket can do this.

### comment:15 in reply to:  12 Changed 2 years ago by Matthias Köppe

But isn't the concept of compile-once-reuse-often the main point of wheels?

Wheels are a format for deployment. Public distribution of wheels on a package index is one use case. Here on this ticket we use it for local deployment -- users can installs these wheels into their venvs.

### comment:16 in reply to:  12 Changed 2 years ago by Matthias Köppe

I feel none of the advantages of wheels outlined in https://www.python.org/dev/peps/pep-0427/ can really prove effective if wheels are rebuilt from code every time. [...] In my opinion, the build process of sage would immensely profit from all these advantages.

I prefer to think about this not as a question of improving "the" build process of sage, but to offer more options for building sage. This is for example also the purpose of the modularization plan outlined in #29705 - make modularized parts of the Sage library deployable via PyPI.

### comment:17 Changed 2 years ago by Matthias Köppe

Reviewers: Tobias Diez, → Tobias Diez, ...

### comment:18 in reply to:  14 Changed 2 years ago by Tobias Diez

I'm only wondering if the pip2 support in the uninstallation script is really necessary (since only python3 is supported now).

Yes, this should be removed, but I prefer to keep tickets narrowly focused. A separate ticket can do this.

It's added in the new file sage-pip-uninstall.

### comment:19 Changed 2 years ago by Matthias Köppe

Ok, I'll clean it up - but in #29500 (dependency of this ticket)

### comment:20 Changed 2 years ago by git

Branch pushed to git repo; I updated commit sha1. New commits:

 ​5a747c4 build/bin/sage-pip-{install,uninstall}: Remove pip2 support ​e66243f Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep-503-simple-repository-for-wheels

### comment:21 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

### comment:22 Changed 2 years ago by git

Branch pushed to git repo; I updated commit sha1. New commits:

 ​9ee2110 build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file ​d7aac84 src/doc/en/developer/packaging.rst: Update sdh_... documentation ​9b7c7a0 build/bin/sage-pip-{install,uninstall}: Fix typo in comment ​4135e8b build/bin/sage-pip-install: Remove an outdated comment ​f2e7075 Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels ​5a2491a Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep-503-simple-repository-for-wheels ### comment:23 Changed 23 months ago by git Commit: 5a2491a5c4ef99ba8e63ebb5c81ae69e45605c60 → e79de9681ebd117165148769af5104dd125452e9 Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:  ​147ec0f build/pkgs/pip2pi: New ​e79de96 build/make/install: At the end, update the repository index ### comment:24 Changed 23 months ago by Matthias Köppe Cc: John Palmieri added #29500 Rebased, needs review ### comment:25 Changed 22 months ago by Tobias Diez Reviewers: Tobias Diez, ... → Tobias Diez needs_review → positive_review LGTM! ### comment:26 Changed 22 months ago by Matthias Köppe Thank you! ### comment:27 follow-ups: 29 34 Changed 22 months ago by Volker Braun Looks like some script is silently ignoring errors (and there is an error, too): $ ./sage -i pip2pi
[...]
[sagelib-9.3.beta2] Finished cleaning, time: 0.15 seconds.
[sagelib-9.3.beta2]
[sagelib-9.3.beta2] real	0m2.307s
[sagelib-9.3.beta2] user	0m1.765s
[sagelib-9.3.beta2] sys	0m0.539s
touch "/home/release/Sage/local/var/lib/sage/installed/sagelib-9.3.beta2"
make[1]: Leaving directory '/home/release/Sage/build/make'

real	0m4.280s
user	0m3.292s
sys	0m0.942s
Traceback (most recent call last):
File "/home/release/Sage/local/bin/dir2pi", line 8, in <module>
sys.exit(dir2pi())
File "/home/release/Sage/local/lib64/python3.8/site-packages/libpip2pi/commands.py", line 308, in dir2pi
return _dir2pi(option, argv)
File "/home/release/Sage/local/lib64/python3.8/site-packages/libpip2pi/commands.py", line 377, in _dir2pi
cgi.escape(pkg_dir_name),
AttributeError: module 'cgi' has no attribute 'escape'


Also docbuild fails on buildbots but not on my machine with

[dochtml] [spkg     ] building [inventory]: targets for 271 source files that are out of date
[dochtml] [spkg     ] updating environment: [new config] 271 added, 0 changed, 0 removed
[dochtml] [spkg     ] /Users/buildbot-sage/slave/sage_git/build/src/doc/en/reference/spkg/pip2pi.rst: WARNING: document isn't included in any toctree
[dochtml] [spkg     ] The inventory files are in local/share/doc/sage/inventory/en/reference/spkg.
[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
[dochtml]     return _run_code(code, main_globals, None,
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/runpy.py", line 87, in _run_code
[dochtml]     exec(code, run_globals)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 1730, in main
[dochtml]     builder()
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 343, in _wrapper
[dochtml]     getattr(get_builder(document), 'inventory')(*args, **kwds)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 569, in _wrapper
[dochtml]     self._build_everything_except_bibliography(lang, format, *args, **kwds)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 555, in _build_everything_except_bibliography
[dochtml]     build_many(build_ref_doc, non_references)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 295, in build_many
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/utils.py", line 289, in build_many
[dochtml]     raise worker_exc.original_exception
[dochtml] OSError: /Users/buildbot-sage/slave/sage_git/build/src/doc/en/reference/spkg/pip2pi.rst: WARNING: document isn't included in any toctree
make[1]: *** [doc-html-mathjax] Error 1


### comment:28 Changed 22 months ago by Volker Braun

Status: positive_review → needs_work

### comment:29 in reply to:  27 Changed 22 months ago by Matthias Köppe

Looks like some script is silently ignoring errors (and there is an error, too):

\$ ./sage -i pip2pi
[...]
[sagelib-9.3.beta2] Finished cleaning, time: 0.15 seconds.
[sagelib-9.3.beta2]
[sagelib-9.3.beta2] real	0m2.307s
[sagelib-9.3.beta2] user	0m1.765s
[sagelib-9.3.beta2] sys	0m0.539s
touch "/home/release/Sage/local/var/lib/sage/installed/sagelib-9.3.beta2"
make[1]: Leaving directory '/home/release/Sage/build/make'

real	0m4.280s
user	0m3.292s
sys	0m0.942s
Traceback (most recent call last):
File "/home/release/Sage/local/bin/dir2pi", line 8, in <module>
sys.exit(dir2pi())
File "/home/release/Sage/local/lib64/python3.8/site-packages/libpip2pi/commands.py", line 308, in dir2pi
return _dir2pi(option, argv)
File "/home/release/Sage/local/lib64/python3.8/site-packages/libpip2pi/commands.py", line 377, in _dir2pi
cgi.escape(pkg_dir_name),
AttributeError: module 'cgi' has no attribute 'escape'


Looks like this package is not ready for Python 3.8, https://github.com/wolever/pip2pi/issues/111

### comment:31 Changed 22 months ago by git

Commit: e79de9681ebd117165148769af5104dd125452e9 → 4e7f9681d286bf74b8b589eb3387a41f02460117

Branch pushed to git repo; I updated commit sha1. New commits:

 ​ee4a743 Merge tag '9.3.beta2' into t/30527/pep-503-simple-repository-for-wheels ​9bae6a1 build/pkgs/pip2pi: Make it a normal package ​4e7f968 build/pkgs/pip2pi/patches: Add patch for python >=3.8 support

### comment:32 Changed 22 months ago by Matthias Köppe

Status: needs_work → needs_review

### comment:33 Changed 22 months ago by git

Commit: 4e7f9681d286bf74b8b589eb3387a41f02460117 → 333e50c002787d5fc828109388d83f5f9b7bd78c

Branch pushed to git repo; I updated commit sha1. New commits:

 ​333e50c build/make/install: Do not ignore errors from dir2pi

### comment:34 in reply to:  27 Changed 22 months ago by Matthias Köppe

Also docbuild fails on buildbots but not on my machine with

[dochtml] [spkg     ] /Users/buildbot-sage/slave/sage_git/build/src/doc/en/reference/spkg/pip2pi.rst: WARNING: document isn't included in any toctree


To debug this on the affected buildbots, it should be checked whether the file index.rst in the same directory is generated correctly by bootstrap.

### comment:36 Changed 22 months ago by Matthias Köppe

Waiting for review

### comment:37 Changed 19 months ago by Matthias Köppe

Milestone: sage-9.3 → sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

### comment:38 Changed 15 months ago by Matthias Köppe

Milestone: sage-9.4 → sage-9.5

Setting a new milestone for this ticket based on a cursory review.

### comment:39 Changed 12 months ago by John Palmieri

I tried make pip2pi (after rebasing to the latest beta), and although the package built, I got an error message afterwards:

[pip2pi-0.8.1] Finished installing pip2pi-0.8.1

real	0m33.466s
user	0m3.670s
sys	0m1.412s
Traceback (most recent call last):
File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/bin/dir2pi", line 8, in <module>
sys.exit(dir2pi())
File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/lib/python3.9/site-packages/libpip2pi/commands.py", line 312, in dir2pi
return _dir2pi(option, argv)
File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/lib/python3.9/site-packages/libpip2pi/commands.py", line 340, in _dir2pi
raise ValueError("no such directory: %r" %(pkgdir, ))
ValueError: no such directory: ''
Error updating PEP 503 simple repository index
make: *** [pip2pi] Error 1


### comment:40 Changed 12 months ago by Matthias Köppe

Milestone: sage-9.5 → sage-wishlist needs_review → needs_work

Thanks for testing. Looks like this now needs updating to the recent build system changes (probably the SAGE_LOCAL/SAGE_VENV split). There's no urgency to this ticket because I learned that one can use --find-links instead of --index-url. So I've changed the milestone

Note: See TracTickets for help on using tickets.