relocatable wheel version of package sage_conf

[mkoeppe]

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

So far, a wheel has been built for XCode python 3.8 on macOS 10.15. The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in ​https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.

The wheel declares dependencies (install-requires) to all Python packages in the Sage distribution that have extension modules. The dependencies are specific using @ to URLs on ​https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1, where I have uploaded the binary wheels. Unfortunately, by PyPI policy, such @ references are not allowed for packages on PyPI. Hence, the pip invocation needs to use a URL.

(See also ​https://twitter.com/mkoeppe_math/status/1378860285537054723)

Instructions for testing:

$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin
$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
$ sage

Procedure for building the sage_conf wheel and compatible wheels:

For building wheels on macOS that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

This creates:

• src/pkgs/sage_conf-relocatable/dist/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
• src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

Follow-up steps:

• build wheels for manylinux using tox -e docker-manylinux....
• disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible;
• #31602: split out sage_root from sage_conf and use install-requires in the sage_conf sdist/wheel with @ links to GH releases to the sage_root wheel and the built non-any wheels.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​952fb9a	Fix up package_data
​9573b7c	Closer to working

[mkoeppe]

On macOS, some shared libraries from homebrew still leak into the build:

find src/pkgs/sage_conf-relocatable/.tox/python/lib/python3.9/site-packages/sage_root -name "*.so" -o -name "*.dylib" | xargs otool -L
libgiac.dylib:
/usr/local/opt/gettext/lib/libintl.8.dylib
R/library/cluster/libs/*.dylib:
	/usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0)
	/usr/local/opt/pcre/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.12.0)
	/usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)

Definitely should disable the R build (#30383).

Also check whether setting CC=gcc -sysroot.... helps getting rid of /usr/local leaking in.

Also need to add --enable-fat-binary.

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink. Also copytree from sage_root_source should be changed so it overwrites (updates) the existing contents of SAGE_ROOT.

the tox test should include the test with otool for self-containedness.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f785d3c

src/pkgs/sage_conf-relocatable/tox.ini: Add environment macos-10.15

[mkoeppe]

Using -isysroot helps but giac configuration needs more work - #31403

[mkoeppe]

Replying to mkoeppe:

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.

Does not work because sage resolves symlinks when it determines SAGE_ROOT.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6da7ba2	Add variable TARGETS
​b2d4ab8	build/pkgs/{r,rpy2}: Downgrade to optional
​8732076	src/sage/repl/ipython_tests.py: Mark R interface tests # optional - rpy2
​dbdf084	src/sage/structure/sage_object.pyx: Mark R interface test # optional - rpy2
​737b21c	src/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2
​4403924	src/sage/interfaces/r.py: Mark all tests # optional - rpy2
​e56a2a9	Merge branch 't/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional' into t/31396/relocatable_wheel_version_of_package_sage_conf

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a228974

src/pkgs/sage_conf-relocatable/setup.py: Move the SAGE_ROOT directory away after building

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0cac965	build/pkgs/ecl: Update to 21.2.1, remove patches included in new version
​6a62484	delete all merged upstream patches: MR 210, MR 231
​62fac00	Merge branch 't/31336/upgrade_ecl_to_21_2_1' into t/31396/relocatable_wheel_version_of_package_sage_conf
​c260942	src/pkgs/sage_conf-relocatable/setup.py: Use --enable-download-from-upstream-url

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​dc5e225	build/pkgs/python3/spkg-configure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS
​df17ad4	tox.ini: Add configuration factors for specific homebrew python3.x
​914ea58	build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST
​986739b	m4/sage_check_python_for_venv.m4: Fix reason
​0b3e70d	build/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it
​372adcb	.github/workflows/tox.yml: Update xcode versions
​fc8b676	SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting
​5ae8044	Merge branch 'u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' of git://trac.sagemath.org/sage into t/31396/relocatable_wheel_version_of_package_sage_conf
​c5c73a9	src/pkgs/sage_conf-relocatable/tox.ini: Define environments macos-python3.8, macos-python3.7, macos-python3_xcode
​4f17a04	src/pkgs/sage_conf-relocatable/setup.py: Use --enable-fat-binary

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8514352

src/pkgs/sage_conf-relocatable/tox.ini: Run delocate-listdeps

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​c3e4093	Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES
​4916415	Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default
​a35428e	Merge #30383
​e098133	src/pkgs/sage_conf/sage_conf.py.in: Add SAGE_SPKG_WHEELS
​5dbd712	src/pkgs/sage_conf-pypi/setup.py: Use 'configure --disable-notebook', run 'make build' so that wheels are built
​839c534	src/pkgs/sage_conf-pypi/setup.py: Fix typo
​100ff51	src/pkgs/sage_conf-pypi/MANIFEST.in: Graft sage_root/build/pkgs/sage_docbuild/src
​afa2ffd	Add configure option --disable-sagelib
​a4f75b8	src/pkgs/sage_conf-pypi: Use configure --disable-sagelib, update README
​1baf390	Merge branch 't/29039/pip_installable_package_sage_bootstrap' into t/31396/relocatable_wheel_version_of_package_sage_conf

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​f04c134	Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional
​27c4bb9	Merge #31409
​66c2987	build/pkgs/{sage_sws2rst,sage_docbuild}/install-requires.txt: New
​7f8ec6c	build/pkgs/sagelib/src/tox.ini: Add factor sitepackages
​0283da5	build/make/Makefile.in: Add wheel, setuptools_wheel to PYTHON_TOOLCHAIN to make sure that PEP 517 packages have a complete build system
​f720722	build/pkgs/sagelib/src/tox.ini: Add factor nobuildisolation
​c451b31	src/setup.cfg.m4 (install_requires): Add sage_conf
​6700223	Merge tag '9.3.rc0' into t/30913/sagelib__add_setup_cfg__install_requires_
​ac0f563	Merge branch 't/30913/sagelib__add_setup_cfg__install_requires_' into t/29039/pip_installable_package_sage_bootstrap
​31bba97	Merge #29039

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f5f4a15

src/pkgs/sage_conf-relocatable/setup.py: Build the wheelhouse, but --disable-notebook --disable-r --disable-sagelib

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​87db1d0

src/pkgs/sage_conf-relocatable/setup.py: Use --with-mp=gmp instead of broken --without-system-gmp option

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​128e8ef	Revert "do not allow numpy intrinsics when building fat binary"
​b871e09	Merge #31521

[mkoeppe]

delocate-listdeps reports the following library dependencies:

/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/usr/local/Cellar/gmp/6.2.1/lib/libgmp.10.dylib
/usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib
/usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib
/usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib
/usr/local/Cellar/little-cms2/2.12/lib/liblcms2.2.dylib
/usr/local/Cellar/mpfr/4.1.0/lib/libmpfr.6.dylib
/usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/usr/local/Cellar/readline/8.1/lib/libreadline.8.1.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib
/usr/local/Cellar/zeromq/4.3.3/lib/libzmq.5.dylib
/usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib
@executable_path/../../../../Python3
liba.dylib
libb.dylib
libc.dylib

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8b5b273	build/pkgs/singular/checksums.ini: Use 4.2.0p1+2021-03-24+sage-2
​f71b013	Merge #31552
​74a0ae0	src/pkgs/sage_conf-relocatable/setup.py: Do not package up $SAGE_LOCAL/var/tmp

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​aa8ff50	src/pkgs/sage_conf-relocatable/setup.py: Use --without-system-readline
​5574aa3	build/pkgs/giac/spkg-install.in [macOS]: Pass --with-included-gettext so that it does not link to Homebrew libintl
​f71ba56	configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.
​5eaaa87	src/pkgs/sage_conf-relocatable/setup.py: Configure libgd without freetype

[mkoeppe]

/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib
/usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib
/usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib
/usr/local/Cellar/little-cms2/2.12/lib/liblcms2.2.dylib
/usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib
/usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib
@executable_path/../../../../Python3
liba.dylib
libb.dylib
libc.dylib

[mkoeppe]

libintl is still coming in through giac

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0ccba0c	configure.ac (--disable-notebook): Disable argon2_cffi
​b7051a9	configure.ac (--disable-notebook --disable-sagelib): Disable ipython

[mkoeppe]

The remaining shared libraries from /usr/local come in through our freetype (which uses libharfbuzz from /usr/local), libgd (which uses libtiff), and pillow (libtiff and everything else).

We could try not to ship matplotlib at all in the wheelhouse - it is responsible for pulling in freetype and pillow - but it still shares one shared library with some of our packages - libpng.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​760b6f0	configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.
​0df51e5	configure.ac (--disable-notebook): Disable argon2_cffi
​42db661	configure.ac (--disable-notebook --disable-sagelib): Disable ipython
​c63ccb0	src/pkgs/sage_conf-relocatable/tox.ini (macos): Also use -isysroot with g++
​1d12382	src/pkgs/sage_conf-relocatable/setup.py: Use LIBGD_CONFIGURE='--without-...almost everything...'
​1ef46d0	build/pkgs/giac: Add another patch in patches/autotools
​1b0a9ee	build/pkgs/giac: Update to 1.6.0.47p3
​41ac551	Merge #31562

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​89f61d2	build/pkgs/giac: Update to 1.6.0.47p3
​00e3956	Merge #31562

[mkoeppe]

Replying to mkoeppe:

libintl is still coming in through giac

Solved now via #31562.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e41932a

src/pkgs/sage_conf-relocatable/tox.ini (macos): Add more SDK paths

[mkoeppe]

Replying to mkoeppe:

The remaining shared libraries from /usr/local come in through our freetype (which uses libharfbuzz from /usr/local)

freetype checks for harfbuzz using pkg-config, accepts configure --without-harfbuzz. ​https://github.com/freetype/freetype/blob/f9f6adb625c48ef15b5d61a3ac1709a068ea95a3/builds/unix/configure.raw#L504

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​87e2dba	tox.ini: Add local-macos-nohomebrew and configuration facors macos-{10.14,10.15,11.1}
​0458f23	tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8}
​39629dd	Merge #31552
​9d9c562	Merge #31562
​a5f1fc7	Merge #31567
​3dd4885	build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no
​68c447f	tox.ini (local -- bash): Set PS1
​4ed1bf4	src/pkgs/sage_conf-relocatable: Remove settings that are now set in SAGE_ROOT/tox.ini

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​300d054	build/pkgs/pillow/spkg-install.in: Rename environment variable extra_build_ext to PILLOW_BUILD_EXT
​8c8d83d	build/pkgs/freetype/spkg-install.in: Pass environment variable FREETYPE_CONFIGURE to configure
​12c84a3	tox.ini (macos-nohomebrew): Prevent /usr/local from leaking in through boost, freetype, pillow
​7966b66	Merge #31409
​fadd2b9	Merge #31567
​59f41b9	src/pkgs/sage_conf-relocatable/setup.py: Resolve SAGE_ROOT
​b5edde2	tox.ini (macos-nohomebrew): Fix typo in PILLOW_...
​e15b152	Merge #31567
​dffcdbc	tox.ini (macos-nohomebrew): Fix up pillow build by setting ZLIB_ROOT
​e4270dd	Merge #31567

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4069297	src/pkgs/sage_conf-relocatable/setup.py: Unlink a previous symlink 'sage_root' in the package source tree before setting the new one
​3779a25	tox.ini (macos-nohomebrew): Disable more libraries to remove dependencies on homebrew
​587142f	Merge #31567

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c5a823c

src/pkgs/sage_conf-relocatable/setup.py: Make sure that we do not package a copy of the sagelib source tree

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​30926d8	.github/workflows/tox*.yml: Replace ubuntu-focal-i386 by manylinux-2_24-i686
​a06ccf7	Merge #31541
​bf6e5f8	build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV
​832f96a	build/make/Makefile.in: Rework using tree_... variables
​3436ad6	build/make/Makefile.in: Define SAGE_VENV earlier
​8cc0b3c	WIP configure --with-sage-venv
​1cfbe2b	build/bin/sage-spkg: Change more SAGE_LOCAL to SAGE_INST_LOCAL
​2202f84	configure.ac: Finish --with-sage-venv handling
​fa897c6	Merge branch 't/29013/separate_venvs' into t/31396/relocatable_wheel_version_of_package_sage_conf
​a3ccb60	src/pkgs/sage_conf-relocatable/setup.py: Build the wheels in a venv outside of SAGE_LOCAL

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b604a68	build/make/Makefile.in: Also source sage-src-env-config for script packages
​92f1a1b	build/pkgs/pynac/spkg-install.in: Build with PYTHON_FOR_VENV
​c94af4b	Merge branch 't/29013/separate_venvs' into t/31396/relocatable_wheel_version_of_package_sage_conf
​9f1f794	src/pkgs/sage_conf-relocatable/setup.py: Include plat_name in the directory name for SAGE_ROOT
​d19c5a4	Fixup
​5382ec7	src/pkgs/sage_conf-relocatable/setup.py: Working version
​a295b81	.gitignore: Add more

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e4e6a1e	src/pkgs/sage_conf-relocatable/setup.py: Fixup
​323dcf1	src/pkgs/sage_conf-relocatable/setup.py: Add first install_requires @ GH
​4049273	src/pkgs/sage_conf-relocatable/setup.py: Add install_requires for all built binary wheels

[mkoeppe]

Unfortunately it turns out that PyPI does not allow @ references in install requires (see ​https://github.com/pypa/pip/issues/6301)

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Invalid value for requires_dist. Error: Can't have direct dependency: 'Cython @ https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/Cython-0.29.21-cp38-cp38-macosx_10_15_x86_64.whl'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​916bda8	configure.ac: Add configure option --disable-doc
​5871e7c	Have sagemath_standard wheel depend on sage_conf wheel @ GH

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7ff26df	git grep -l SAGE_SUITESPARSE | xargs sed -i .bak s/SAGE_SUITESPARSE_LOCALINSTALL/SAGE_SUITESPARSE_PREFIX/g
​52537d1	build/pkgs/cvxopt/spkg-install.in: Actually use the value of SAGE_SUITESPARSE_PREFIX
​ed1a089	Merge #31584
​be5b7bc	Merge #31567
​1fb3a9b	build/pkgs/pynac/spkg-install.in: Back to PYTHON=python3
​24ebfb5	src/bin/sage: Only do the check for an incomplete installation when run from the source tree
​f331ee7	src/pkgs/sage_conf-relocatable/setup.py: Pass CONFIGURED_{CC,CXX} to configure if set
​6a3d380	src/pkgs/sage_conf-relocatable/setup.py: Package libexec too