Opened 2 years ago

Last modified 12 months ago

#30527 needs_work enhancement

Create PEP 503 simple repository for wheels built during installation

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-wishlist
Component: build Keywords: sd111
Cc: Samuel Lelièvre, Tobias Diez, Erik Bray, John Palmieri Merged in:
Authors: Matthias Koeppe Reviewers: Tobias Diez
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/pep-503-simple-repository-for-wheels (Commits, GitHub, GitLab) Commit: 333e50c002787d5fc828109388d83f5f9b7bd78c
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

As a follow-up to #29500 (Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels):

We create a PEP 503 compliant Simple Repository index in $SAGE_SPKG_WHEELS. Then users can create their virtual environment using tools such as pipenv [[source]] or pip install --index-url.

We do this using the command dir2pi from https://pypi.org/project/pip2pi/

Follow-up:

  • #30578 - Add src/requirements.txt for installation of sagelib in a venv

References:

Change History (40)

comment:1 Changed 2 years ago by Matthias Köppe

piprepo can be used for this task, but it has excessive dependencies

comment:2 Changed 2 years ago by Matthias Köppe

Dependencies: #29500

comment:3 Changed 2 years ago by Tobias Diez

There is also pypiserver, see eg https://stackoverflow.com/a/51528588/873661 for how to use it with pipenv

comment:4 Changed 2 years ago by Matthias Köppe

Well, we don't want to run a server for this, but following this link I found https://github.com/wolever/pip2pi

comment:5 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:6 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:7 Changed 2 years ago by Tobias Diez

Yeah I agree, a server is not required (although the overhead of creating a simple server on listing on localhost should be pretty minimal). If pip2pi works with pipenv and pip, then that's perfect!

comment:8 Changed 2 years ago by Matthias Köppe

Branch: u/mkoeppe/pep-503-simple-repository-for-wheels

comment:9 Changed 2 years ago by Matthias Köppe

Authors: Matthias Koeppe
Commit: 2555e19712ee6bc2f9b4edf982b5f86d1bb9a6fe
Status: newneeds_review

New commits:

8aa6fd9build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it
d369aabbuild/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install
2d435abbuild/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel
55993b6build/bin/sage-dist-helpers: Fixup
0a64674build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
207d80fbuild/pkgs/pip2pi: New
2555e19build/make/install: At the end, update the repository index

comment:10 Changed 2 years ago by Tobias Diez

What is the reason that sage needs to build wheels for standard packages, and not just downloads them from pypi?

Is it because some packages have patches applied? In this case, I would propose to migrate these patches to proper forks of the packages which are then published to pypi including their wheel. This has a few advantages, mainly that the build of sage is speedup and simplified, and that the wider python community can profit from these patches.

What do you think?

comment:11 in reply to:  10 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

What is the reason that sage needs to build wheels for standard packages, and not just downloads them from pypi?

Sage-the-distribution is a deployment mechanism. Compiling all packages from source gives us full control and eliminates possible ABI problems from mixing prebuilt wheels that were compiled on different systems. Building from source also has advantages such as generating architecture-specific code (see #27122).

Is it because some packages have patches applied?

In some cases, yes, in other cases, just specific configuration.

In this case, I would propose to migrate these patches to proper forks of the packages which are then published to pypi including their wheel. This has a few advantages, mainly that the build of sage is speedup and simplified, and that the wider python community can profit from these patches.

We don't have intentions to take over maintenance of these packages. Whenever possible, we send our patches upstream. In many cases, it is just a matter of mismatched release schedules, and we have to keep patches only for a limited time until upstream makes a new stable release including a particular patch. For example, consider the numpy patch added in #29766. It is actually from upstream, but not included in any stable release. But we need the patch because we want Sage-the-distribution to support a particular platform.

comment:12 Changed 2 years ago by Tobias Diez

Reviewers: Tobias Diez,

Thanks for the detailed explanation. Makes things clearer for me now.

Compiling all packages from source gives us full control and eliminates possible ABI problems from mixing prebuilt wheels that were compiled on different systems. Building from source also has advantages such as generating architecture-specific code

But isn't the concept of compile-once-reuse-often the main point of wheels? By design, it's the responsibility of the the maintainer (= publisher of the wheel) that the wheel is useable and optimized for the particular platforms they are published. This also make sense since she is the export of the code of the library.

I feel none of the advantages of wheels outlined in https://www.python.org/dev/peps/pep-0427/ can really prove effective if wheels are rebuilt from code every time.

The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.

In my opinion, the build process of sage would immensely profit from all these advantages.

Anyway, this discussion is somewhat orthogonal to this ticket (sorry!). So concerning the changes, the code looks good to me. I'm only wondering if the pip2 support in the uninstallation script is really necessary (since only python3 is supported now). However, I don't feel familiar enough with the build script to change the status to positive review.

comment:13 in reply to:  12 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

But isn't the concept of compile-once-reuse-often the main point of wheels? By design, it's the responsibility of the the maintainer (= publisher of the wheel) that the wheel is useable and optimized for the particular platforms they are published. This also make sense since she is the export of the code of the library.

In practice, wheels are offered on PyPI with the broadest possible platform compatibility, typically using https://github.com/pypa/manylinux -- this is the opposite of "optimized".

comment:14 in reply to:  12 ; Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

I'm only wondering if the pip2 support in the uninstallation script is really necessary (since only python3 is supported now).

Yes, this should be removed, but I prefer to keep tickets narrowly focused. A separate ticket can do this.

comment:15 in reply to:  12 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

But isn't the concept of compile-once-reuse-often the main point of wheels?

Wheels are a format for deployment. Public distribution of wheels on a package index is one use case. Here on this ticket we use it for local deployment -- users can installs these wheels into their venvs.

comment:16 in reply to:  12 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

I feel none of the advantages of wheels outlined in https://www.python.org/dev/peps/pep-0427/ can really prove effective if wheels are rebuilt from code every time. [...] In my opinion, the build process of sage would immensely profit from all these advantages.

I prefer to think about this not as a question of improving "the" build process of sage, but to offer more options for building sage. This is for example also the purpose of the modularization plan outlined in #29705 - make modularized parts of the Sage library deployable via PyPI.

comment:17 Changed 2 years ago by Matthias Köppe

Reviewers: Tobias Diez,Tobias Diez, ...

comment:18 in reply to:  14 Changed 2 years ago by Tobias Diez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

I'm only wondering if the pip2 support in the uninstallation script is really necessary (since only python3 is supported now).

Yes, this should be removed, but I prefer to keep tickets narrowly focused. A separate ticket can do this.

It's added in the new file sage-pip-uninstall.

comment:19 Changed 2 years ago by Matthias Köppe

Ok, I'll clean it up - but in #29500 (dependency of this ticket)

comment:20 Changed 2 years ago by git

Commit: 2555e19712ee6bc2f9b4edf982b5f86d1bb9a6fee66243f23a20a6499b0ad33c5538ae4f36b31764

Branch pushed to git repo; I updated commit sha1. New commits:

5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support
e66243fMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep-503-simple-repository-for-wheels

comment:21 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:22 Changed 2 years ago by git

Commit: e66243f23a20a6499b0ad33c5538ae4f36b317645a2491a5c4ef99ba8e63ebb5c81ae69e45605c60

Branch pushed to git repo; I updated commit sha1. New commits:

9ee2110build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
d7aac84src/doc/en/developer/packaging.rst: Update sdh_... documentation
9b7c7a0build/bin/sage-pip-{install,uninstall}: Fix typo in comment
4135e8bbuild/bin/sage-pip-install: Remove an outdated comment
f2e7075Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels
5a2491aMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep-503-simple-repository-for-wheels

comment:23 Changed 23 months ago by git

Commit: 5a2491a5c4ef99ba8e63ebb5c81ae69e45605c60e79de9681ebd117165148769af5104dd125452e9

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

147ec0fbuild/pkgs/pip2pi: New
e79de96build/make/install: At the end, update the repository index

comment:24 Changed 23 months ago by Matthias Köppe

Cc: John Palmieri added
Dependencies: #29500

Rebased, needs review

comment:25 Changed 22 months ago by Tobias Diez

Reviewers: Tobias Diez, ...Tobias Diez
Status: needs_reviewpositive_review

LGTM!

comment:26 Changed 22 months ago by Matthias Köppe

Thank you!

comment:27 Changed 22 months ago by Volker Braun

Looks like some script is silently ignoring errors (and there is an error, too):

$ ./sage -i pip2pi
[...]
[sagelib-9.3.beta2] Finished cleaning, time: 0.15 seconds.
[sagelib-9.3.beta2] 
[sagelib-9.3.beta2] real	0m2.307s
[sagelib-9.3.beta2] user	0m1.765s
[sagelib-9.3.beta2] sys	0m0.539s
touch "/home/release/Sage/local/var/lib/sage/installed/sagelib-9.3.beta2"
make[1]: Leaving directory '/home/release/Sage/build/make'

real	0m4.280s
user	0m3.292s
sys	0m0.942s
Traceback (most recent call last):
  File "/home/release/Sage/local/bin/dir2pi", line 8, in <module>
    sys.exit(dir2pi())
  File "/home/release/Sage/local/lib64/python3.8/site-packages/libpip2pi/commands.py", line 308, in dir2pi
    return _dir2pi(option, argv)
  File "/home/release/Sage/local/lib64/python3.8/site-packages/libpip2pi/commands.py", line 377, in _dir2pi
    cgi.escape(pkg_dir_name),
AttributeError: module 'cgi' has no attribute 'escape'
Sage build/upgrade complete!

Also docbuild fails on buildbots but not on my machine with

[dochtml] [spkg     ] building [inventory]: targets for 271 source files that are out of date
[dochtml] [spkg     ] updating environment: [new config] 271 added, 0 changed, 0 removed
[dochtml] [spkg     ] /Users/buildbot-sage/slave/sage_git/build/src/doc/en/reference/spkg/pip2pi.rst: WARNING: document isn't included in any toctree
[dochtml] [spkg     ] The inventory files are in local/share/doc/sage/inventory/en/reference/spkg.
[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
[dochtml]     return _run_code(code, main_globals, None,
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/runpy.py", line 87, in _run_code
[dochtml]     exec(code, run_globals)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 1730, in main
[dochtml]     builder()
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 343, in _wrapper
[dochtml]     getattr(get_builder(document), 'inventory')(*args, **kwds)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 569, in _wrapper
[dochtml]     self._build_everything_except_bibliography(lang, format, *args, **kwds)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 555, in _build_everything_except_bibliography
[dochtml]     build_many(build_ref_doc, non_references)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 295, in build_many
[dochtml]     _build_many(target, args, processes=NUM_THREADS)
[dochtml]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.8/site-packages/sage_setup/docbuild/utils.py", line 289, in build_many
[dochtml]     raise worker_exc.original_exception
[dochtml] OSError: /Users/buildbot-sage/slave/sage_git/build/src/doc/en/reference/spkg/pip2pi.rst: WARNING: document isn't included in any toctree
make[1]: *** [doc-html-mathjax] Error 1

comment:28 Changed 22 months ago by Volker Braun

Status: positive_reviewneeds_work

comment:29 in reply to:  27 Changed 22 months ago by Matthias Köppe

Replying to vbraun:

Looks like some script is silently ignoring errors (and there is an error, too):

$ ./sage -i pip2pi
[...]
[sagelib-9.3.beta2] Finished cleaning, time: 0.15 seconds.
[sagelib-9.3.beta2] 
[sagelib-9.3.beta2] real	0m2.307s
[sagelib-9.3.beta2] user	0m1.765s
[sagelib-9.3.beta2] sys	0m0.539s
touch "/home/release/Sage/local/var/lib/sage/installed/sagelib-9.3.beta2"
make[1]: Leaving directory '/home/release/Sage/build/make'

real	0m4.280s
user	0m3.292s
sys	0m0.942s
Traceback (most recent call last):
  File "/home/release/Sage/local/bin/dir2pi", line 8, in <module>
    sys.exit(dir2pi())
  File "/home/release/Sage/local/lib64/python3.8/site-packages/libpip2pi/commands.py", line 308, in dir2pi
    return _dir2pi(option, argv)
  File "/home/release/Sage/local/lib64/python3.8/site-packages/libpip2pi/commands.py", line 377, in _dir2pi
    cgi.escape(pkg_dir_name),
AttributeError: module 'cgi' has no attribute 'escape'
Sage build/upgrade complete!

Looks like this package is not ready for Python 3.8, https://github.com/wolever/pip2pi/issues/111

comment:31 Changed 22 months ago by git

Commit: e79de9681ebd117165148769af5104dd125452e94e7f9681d286bf74b8b589eb3387a41f02460117

Branch pushed to git repo; I updated commit sha1. New commits:

ee4a743Merge tag '9.3.beta2' into t/30527/pep-503-simple-repository-for-wheels
9bae6a1build/pkgs/pip2pi: Make it a normal package
4e7f968build/pkgs/pip2pi/patches: Add patch for python >=3.8 support

comment:32 Changed 22 months ago by Matthias Köppe

Status: needs_workneeds_review

comment:33 Changed 22 months ago by git

Commit: 4e7f9681d286bf74b8b589eb3387a41f02460117333e50c002787d5fc828109388d83f5f9b7bd78c

Branch pushed to git repo; I updated commit sha1. New commits:

333e50cbuild/make/install: Do not ignore errors from dir2pi

comment:34 in reply to:  27 Changed 22 months ago by Matthias Köppe

Replying to vbraun:

Also docbuild fails on buildbots but not on my machine with

[dochtml] [spkg     ] /Users/buildbot-sage/slave/sage_git/build/src/doc/en/reference/spkg/pip2pi.rst: WARNING: document isn't included in any toctree

To debug this on the affected buildbots, it should be checked whether the file index.rst in the same directory is generated correctly by bootstrap.

comment:35 Changed 22 months ago by Matthias Köppe

Keywords: sd111 added

comment:36 Changed 22 months ago by Matthias Köppe

Waiting for review

comment:37 Changed 19 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:38 Changed 15 months ago by Matthias Köppe

Milestone: sage-9.4sage-9.5

Setting a new milestone for this ticket based on a cursory review.

comment:39 Changed 12 months ago by John Palmieri

I tried make pip2pi (after rebasing to the latest beta), and although the package built, I got an error message afterwards:

[pip2pi-0.8.1] Finished installing pip2pi-0.8.1

real	0m33.466s
user	0m3.670s
sys	0m1.412s
Traceback (most recent call last):
  File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/bin/dir2pi", line 8, in <module>
    sys.exit(dir2pi())
  File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/lib/python3.9/site-packages/libpip2pi/commands.py", line 312, in dir2pi
    return _dir2pi(option, argv)
  File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/lib/python3.9/site-packages/libpip2pi/commands.py", line 340, in _dir2pi
    raise ValueError("no such directory: %r" %(pkgdir, ))
ValueError: no such directory: ''
Error updating PEP 503 simple repository index
make: *** [pip2pi] Error 1

comment:40 Changed 12 months ago by Matthias Köppe

Milestone: sage-9.5sage-wishlist
Status: needs_reviewneeds_work

Thanks for testing. Looks like this now needs updating to the recent build system changes (probably the SAGE_LOCAL/SAGE_VENV split). There's no urgency to this ticket because I learned that one can use --find-links instead of --index-url. So I've changed the milestone

Note: See TracTickets for help on using tickets.