Opened 8 months ago

Last modified 4 days ago

#31396 needs_review enhancement

relocatable wheel version of package sage_conf — at Version 56

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.5
Component: relocation Keywords:
Cc: culler, gh-kliem, dimpase, jhpalmieri, slelievre Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf (Commits, GitHub, GitLab) Commit: c5a823cb524914ab985eaea09f52a87aa93c03cb
Dependencies: #29039, #31409, #31521, #31552, #31562, #31567 Stopgaps:

Status badges

Description (last modified by mkoeppe)

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse local/var/lib/sage/wheels), making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's /var/tmp surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

To test:

./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v)

Specifically for building a wheel on macOS that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)

This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

In the meantime, you can use a prebuilt wheel (so far, macOS python3.8 x86_64 only) made available at https://github.com/mkoeppe/sage/releases; install the wheel using

/usr/bin/python3 -m pip install --user https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl

Then proceed with the same instructions as https://pypi.org/project/sage-conf/

Follow-up steps: Do not package up the wheel-building venv (247MB in sage_root/local/lib/python3.8/site-packages/); do not ship any any wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible

Change History (56)

comment:1 Changed 8 months ago by mkoeppe

  • Branch set to u/mkoeppe/relocatable_wheel_version_of_package_sage_conf

comment:2 Changed 8 months ago by git

  • Commit set to 9573b7c12afc5de1de743e53ff182d20490b340c

Branch pushed to git repo; I updated commit sha1. New commits:

952fb9aFix up package_data
9573b7cCloser to working

comment:3 Changed 8 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Cc gh-kliem added
  • Description modified (diff)

comment:4 follow-up: Changed 8 months ago by mkoeppe

  • Description modified (diff)

On macOS, some shared libraries from homebrew still leak into the build:

find src/pkgs/sage_conf-relocatable/.tox/python/lib/python3.9/site-packages/sage_root -name "*.so" -o -name "*.dylib" | xargs otool -L
libgiac.dylib:
/usr/local/opt/gettext/lib/libintl.8.dylib
R/library/cluster/libs/*.dylib:
	/usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0)
	/usr/local/opt/pcre/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.12.0)
	/usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)

Definitely should disable the R build (#30383).

Also check whether setting CC=gcc -sysroot.... helps getting rid of /usr/local leaking in.

Also need to add --enable-fat-binary.

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink. Also copytree from sage_root_source should be changed so it overwrites (updates) the existing contents of SAGE_ROOT.

the tox test should include the test with otool for self-containedness.

comment:5 Changed 8 months ago by git

  • Commit changed from 9573b7c12afc5de1de743e53ff182d20490b340c to f785d3cde259485597664306e191a21ad0ab9d58

Branch pushed to git repo; I updated commit sha1. New commits:

f785d3csrc/pkgs/sage_conf-relocatable/tox.ini: Add environment macos-10.15

comment:6 Changed 8 months ago by mkoeppe

Using -isysroot helps but giac configuration needs more work - #31403

comment:7 in reply to: ↑ 4 Changed 8 months ago by mkoeppe

Replying to mkoeppe:

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.

Does not work because sage resolves symlinks when it determines SAGE_ROOT.

comment:8 Changed 8 months ago by mkoeppe

  • Dependencies changed from #29039 to #29039, #31409

comment:9 Changed 8 months ago by git

  • Commit changed from f785d3cde259485597664306e191a21ad0ab9d58 to e56a2a956a96bd70dbb443869697e89b7fb2b58a

Branch pushed to git repo; I updated commit sha1. New commits:

6da7ba2Add variable TARGETS
b2d4ab8build/pkgs/{r,rpy2}: Downgrade to optional
8732076src/sage/repl/ipython_tests.py: Mark R interface tests # optional - rpy2
dbdf084src/sage/structure/sage_object.pyx: Mark R interface test # optional - rpy2
737b21csrc/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2
4403924src/sage/interfaces/r.py: Mark all tests # optional - rpy2
e56a2a9Merge branch 't/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional' into t/31396/relocatable_wheel_version_of_package_sage_conf

comment:10 Changed 8 months ago by git

  • Commit changed from e56a2a956a96bd70dbb443869697e89b7fb2b58a to a2289744baf90e3dcb288f96ef52d471ab8251bb

Branch pushed to git repo; I updated commit sha1. New commits:

a228974src/pkgs/sage_conf-relocatable/setup.py: Move the SAGE_ROOT directory away after building

comment:11 Changed 8 months ago by git

  • Commit changed from a2289744baf90e3dcb288f96ef52d471ab8251bb to c260942d3085fa369117746c635b48da5a572489

Branch pushed to git repo; I updated commit sha1. New commits:

0cac965build/pkgs/ecl: Update to 21.2.1, remove patches included in new version
6a62484delete all merged upstream patches: MR 210, MR 231
62fac00Merge branch 't/31336/upgrade_ecl_to_21_2_1' into t/31396/relocatable_wheel_version_of_package_sage_conf
c260942src/pkgs/sage_conf-relocatable/setup.py: Use --enable-download-from-upstream-url

comment:12 Changed 8 months ago by git

  • Commit changed from c260942d3085fa369117746c635b48da5a572489 to 4f17a047ec84130462cb7f91f28ed284816cfa74

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

dc5e225build/pkgs/python3/spkg-configure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS
df17ad4tox.ini: Add configuration factors for specific homebrew python3.x
914ea58build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST
986739bm4/sage_check_python_for_venv.m4: Fix reason
0b3e70dbuild/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it
372adcb.github/workflows/tox.yml: Update xcode versions
fc8b676SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting
5ae8044Merge branch 'u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' of git://trac.sagemath.org/sage into t/31396/relocatable_wheel_version_of_package_sage_conf
c5c73a9src/pkgs/sage_conf-relocatable/tox.ini: Define environments macos-python3.8, macos-python3.7, macos-python3_xcode
4f17a04src/pkgs/sage_conf-relocatable/setup.py: Use --enable-fat-binary

comment:13 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:14 Changed 8 months ago by git

  • Commit changed from 4f17a047ec84130462cb7f91f28ed284816cfa74 to 851435289634de431fc8708e0c4c59a37b578b71

Branch pushed to git repo; I updated commit sha1. New commits:

8514352src/pkgs/sage_conf-relocatable/tox.ini: Run delocate-listdeps

comment:15 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 7 months ago by git

  • Commit changed from 851435289634de431fc8708e0c4c59a37b578b71 to 1baf390bc2bc5c29a7bfa6d7c73d31f39a1b5e65

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

c3e4093Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES
4916415Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default
a35428eMerge #30383
e098133src/pkgs/sage_conf/sage_conf.py.in: Add SAGE_SPKG_WHEELS
5dbd712src/pkgs/sage_conf-pypi/setup.py: Use 'configure --disable-notebook', run 'make build' so that wheels are built
839c534src/pkgs/sage_conf-pypi/setup.py: Fix typo
100ff51src/pkgs/sage_conf-pypi/MANIFEST.in: Graft sage_root/build/pkgs/sage_docbuild/src
afa2ffdAdd configure option --disable-sagelib
a4f75b8src/pkgs/sage_conf-pypi: Use configure --disable-sagelib, update README
1baf390Merge branch 't/29039/pip_installable_package_sage_bootstrap' into t/31396/relocatable_wheel_version_of_package_sage_conf

comment:17 Changed 7 months ago by git

  • Commit changed from 1baf390bc2bc5c29a7bfa6d7c73d31f39a1b5e65 to 31bba97d13f86c531eeef0a26383b98fa3671c45

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

f04c134Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional
27c4bb9Merge #31409
66c2987build/pkgs/{sage_sws2rst,sage_docbuild}/install-requires.txt: New
7f8ec6cbuild/pkgs/sagelib/src/tox.ini: Add factor sitepackages
0283da5build/make/Makefile.in: Add wheel, setuptools_wheel to PYTHON_TOOLCHAIN to make sure that PEP 517 packages have a complete build system
f720722build/pkgs/sagelib/src/tox.ini: Add factor nobuildisolation
c451b31src/setup.cfg.m4 (install_requires): Add sage_conf
6700223Merge tag '9.3.rc0' into t/30913/sagelib__add_setup_cfg__install_requires_
ac0f563Merge branch 't/30913/sagelib__add_setup_cfg__install_requires_' into t/29039/pip_installable_package_sage_bootstrap
31bba97Merge #29039

comment:18 Changed 7 months ago by mkoeppe

  • Cc dimpase added

comment:19 Changed 7 months ago by git

  • Commit changed from 31bba97d13f86c531eeef0a26383b98fa3671c45 to f5f4a157e3a1630fc789f8f61655e123b6573eb6

Branch pushed to git repo; I updated commit sha1. New commits:

f5f4a15src/pkgs/sage_conf-relocatable/setup.py: Build the wheelhouse, but --disable-notebook --disable-r --disable-sagelib

comment:20 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:21 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:22 Changed 7 months ago by git

  • Commit changed from f5f4a157e3a1630fc789f8f61655e123b6573eb6 to 87db1d0a371b70040b4d7448f0fadc9b50f18b54

Branch pushed to git repo; I updated commit sha1. New commits:

87db1d0src/pkgs/sage_conf-relocatable/setup.py: Use --with-mp=gmp instead of broken --without-system-gmp option

comment:23 Changed 7 months ago by mkoeppe

  • Dependencies changed from #29039, #31409 to #29039, #31409, #31521

comment:24 Changed 7 months ago by git

  • Commit changed from 87db1d0a371b70040b4d7448f0fadc9b50f18b54 to b871e09db5d847470fcf33eab2960554e0787bc1

Branch pushed to git repo; I updated commit sha1. New commits:

128e8efRevert "do not allow numpy intrinsics when building fat binary"
b871e09Merge #31521

comment:25 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:26 Changed 7 months ago by mkoeppe

delocate-listdeps reports the following library dependencies:

/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/usr/local/Cellar/gmp/6.2.1/lib/libgmp.10.dylib
/usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib
/usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib
/usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib
/usr/local/Cellar/little-cms2/2.12/lib/liblcms2.2.dylib
/usr/local/Cellar/mpfr/4.1.0/lib/libmpfr.6.dylib
/usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/usr/local/Cellar/readline/8.1/lib/libreadline.8.1.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib
/usr/local/Cellar/zeromq/4.3.3/lib/libzmq.5.dylib
/usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib
@executable_path/../../../../Python3
liba.dylib
libb.dylib
libc.dylib

comment:27 Changed 7 months ago by git

  • Commit changed from b871e09db5d847470fcf33eab2960554e0787bc1 to 74a0ae02a05213aa5ec201245aaf3f55e8c5b037

Branch pushed to git repo; I updated commit sha1. New commits:

8b5b273build/pkgs/singular/checksums.ini: Use 4.2.0p1+2021-03-24+sage-2
f71b013Merge #31552
74a0ae0src/pkgs/sage_conf-relocatable/setup.py: Do not package up $SAGE_LOCAL/var/tmp

comment:28 Changed 7 months ago by git

  • Commit changed from 74a0ae02a05213aa5ec201245aaf3f55e8c5b037 to 5eaaa8788437022cb2f070ef12e461e8e9619532

Branch pushed to git repo; I updated commit sha1. New commits:

aa8ff50src/pkgs/sage_conf-relocatable/setup.py: Use --without-system-readline
5574aa3build/pkgs/giac/spkg-install.in [macOS]: Pass --with-included-gettext so that it does not link to Homebrew libintl
f71ba56configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.
5eaaa87src/pkgs/sage_conf-relocatable/setup.py: Configure libgd without freetype

comment:29 Changed 7 months ago by mkoeppe

/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib
/usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib
/usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib
/usr/local/Cellar/little-cms2/2.12/lib/liblcms2.2.dylib
/usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib
/usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib
@executable_path/../../../../Python3
liba.dylib
libb.dylib
libc.dylib

comment:30 follow-up: Changed 7 months ago by mkoeppe

libintl is still coming in through giac

comment:31 Changed 7 months ago by git

  • Commit changed from 5eaaa8788437022cb2f070ef12e461e8e9619532 to b7051a99473e4ffa3f2d17c980b619bb45472980

Branch pushed to git repo; I updated commit sha1. New commits:

0ccba0cconfigure.ac (--disable-notebook): Disable argon2_cffi
b7051a9configure.ac (--disable-notebook --disable-sagelib): Disable ipython

comment:32 follow-up: Changed 7 months ago by mkoeppe

The remaining shared libraries from /usr/local come in through our freetype (which uses libharfbuzz from /usr/local), libgd (which uses libtiff), and pillow (libtiff and everything else).

We could try not to ship matplotlib at all in the wheelhouse - it is responsible for pulling in freetype and pillow - but it still shares one shared library with some of our packages - libpng.

comment:33 Changed 7 months ago by git

  • Commit changed from b7051a99473e4ffa3f2d17c980b619bb45472980 to 41ac551c9b89fb9615e5b52b6164fa0ea67e3a30

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

760b6f0configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.
0df51e5configure.ac (--disable-notebook): Disable argon2_cffi
42db661configure.ac (--disable-notebook --disable-sagelib): Disable ipython
c63ccb0src/pkgs/sage_conf-relocatable/tox.ini (macos): Also use -isysroot with g++
1d12382src/pkgs/sage_conf-relocatable/setup.py: Use LIBGD_CONFIGURE='--without-...almost everything...'
1ef46d0build/pkgs/giac: Add another patch in patches/autotools
1b0a9eebuild/pkgs/giac: Update to 1.6.0.47p3
41ac551Merge #31562

comment:34 Changed 7 months ago by mkoeppe

  • Dependencies changed from #29039, #31409, #31521 to #29039, #31409, #31521, #31552, #31562

comment:35 Changed 7 months ago by git

  • Commit changed from 41ac551c9b89fb9615e5b52b6164fa0ea67e3a30 to 00e39560744ad3832aa0614a34b8059a47db5a14

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

89f61d2build/pkgs/giac: Update to 1.6.0.47p3
00e3956Merge #31562

comment:36 in reply to: ↑ 30 Changed 7 months ago by mkoeppe

Replying to mkoeppe:

libintl is still coming in through giac

Solved now via #31562.

comment:37 Changed 7 months ago by git

  • Commit changed from 00e39560744ad3832aa0614a34b8059a47db5a14 to e41932acfaf19f3d19c640d91868696f9b8c5b94

Branch pushed to git repo; I updated commit sha1. New commits:

e41932asrc/pkgs/sage_conf-relocatable/tox.ini (macos): Add more SDK paths

comment:38 Changed 7 months ago by mkoeppe

  • Work issues set to Refactor through #31567

comment:39 in reply to: ↑ 32 Changed 7 months ago by mkoeppe

Replying to mkoeppe:

The remaining shared libraries from /usr/local come in through our freetype (which uses libharfbuzz from /usr/local)

freetype checks for harfbuzz using pkg-config, accepts configure --without-harfbuzz. https://github.com/freetype/freetype/blob/f9f6adb625c48ef15b5d61a3ac1709a068ea95a3/builds/unix/configure.raw#L504

comment:40 Changed 7 months ago by mkoeppe

  • Dependencies changed from #29039, #31409, #31521, #31552, #31562 to #29039, #31409, #31521, #31552, #31562, #31567

comment:41 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:42 Changed 7 months ago by git

  • Commit changed from e41932acfaf19f3d19c640d91868696f9b8c5b94 to 4ed1bf421731e1cf7d4ad7d1179fa660e5d4364d

Branch pushed to git repo; I updated commit sha1. New commits:

87e2dbatox.ini: Add local-macos-nohomebrew and configuration facors macos-{10.14,10.15,11.1}
0458f23tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8}
39629ddMerge #31552
9d9c562Merge #31562
a5f1fc7Merge #31567
3dd4885build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no
68c447ftox.ini (local -- bash): Set PS1
4ed1bf4src/pkgs/sage_conf-relocatable: Remove settings that are now set in SAGE_ROOT/tox.ini

comment:43 Changed 7 months ago by mkoeppe

  • Work issues Refactor through #31567 deleted

comment:44 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:45 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:46 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:47 Changed 7 months ago by git

  • Commit changed from 4ed1bf421731e1cf7d4ad7d1179fa660e5d4364d to e4270dd90169c7ba7eb0d9a2c4356dc9c2514941

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

300d054build/pkgs/pillow/spkg-install.in: Rename environment variable extra_build_ext to PILLOW_BUILD_EXT
8c8d83dbuild/pkgs/freetype/spkg-install.in: Pass environment variable FREETYPE_CONFIGURE to configure
12c84a3tox.ini (macos-nohomebrew): Prevent /usr/local from leaking in through boost, freetype, pillow
7966b66Merge #31409
fadd2b9Merge #31567
59f41b9src/pkgs/sage_conf-relocatable/setup.py: Resolve SAGE_ROOT
b5edde2tox.ini (macos-nohomebrew): Fix typo in PILLOW_...
e15b152Merge #31567
dffcdbctox.ini (macos-nohomebrew): Fix up pillow build by setting ZLIB_ROOT
e4270ddMerge #31567

comment:48 Changed 7 months ago by git

  • Commit changed from e4270dd90169c7ba7eb0d9a2c4356dc9c2514941 to 587142f996c7c1457344e960fea682522f00d8e4

Branch pushed to git repo; I updated commit sha1. New commits:

4069297src/pkgs/sage_conf-relocatable/setup.py: Unlink a previous symlink 'sage_root' in the package source tree before setting the new one
3779a25tox.ini (macos-nohomebrew): Disable more libraries to remove dependencies on homebrew
587142fMerge #31567

comment:49 Changed 7 months ago by git

  • Commit changed from 587142f996c7c1457344e960fea682522f00d8e4 to c5a823cb524914ab985eaea09f52a87aa93c03cb

Branch pushed to git repo; I updated commit sha1. New commits:

c5a823csrc/pkgs/sage_conf-relocatable/setup.py: Make sure that we do not package a copy of the sagelib source tree

comment:50 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:51 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:52 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:53 Changed 7 months ago by mkoeppe

  • Cc jhpalmieri slelievre added

comment:54 Changed 7 months ago by mkoeppe

  • Status changed from new to needs_review

comment:55 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:56 Changed 7 months ago by mkoeppe

  • Description modified (diff)
Note: See TracTickets for help on using tickets.