New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ...

[mkoeppe]

For all script packages SPKG that have an embedded source tree -- currently sagelib, sage_docbuild, sage_sws2rst -- we add a new target:

• SPKG-tox-% (which calls tox -e % in the package's source tree)

These targets are particularly useful for invoking them within the portability tests of the top-level tox:

• $ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox invokes the tox testing of the sage_docbuild distribution
• $ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
• $ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
• $ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nobuildisolation-nopypi (needs #30913, #31384) uses the src/environment.yml file to set up a complete environment from conda and then tests sagelib in a venv using tox

[mkoeppe]

New commits:

​1eef6d6

build/make/Makefile.in: Add targets SPKG-sdist, SPKG-tox, SPKG-tox-% for script packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6964933

build/make/Makefile.in: fixup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9993348	build/make/Makefile.in: Run targets SPKG-tox, SPKG-tox-% outside of sage-env, using SAGE_ORIG_PATH - so that tox from conda works
​4533018	tox.ini: Add local-conda-environment-src
​188bf2e	tox.ini, src/doc/bootstrap: Get tox into standard builds

[mkoeppe]

Moving this ticket to 9.4, as it seems unlikely that it will be merged in 9.3, which is in the release candidate stage

[mkoeppe]

Setting a new milestone for this ticket based on a cursory review.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​8af6041	Merge tag '9.5.beta2' into t/32442/__configure__make___with_sage_venv_the_default
​b3f7d34	.gitignore: Add /venv
​fc4b641	configure.ac: Remove conveniene symlinks before (re)creating them
​8bcbf1d	Makefile (distclean): Remove convenience symlinks prefix, venv
​dcb4a08	configure.ac: Indicate default in help string for --with-sage-venv
​e33bc89	Merge tag '9.5.beta3' into t/32442/__configure__make___with_sage_venv_the_default
​2101b8c	build/pkgs/python3/spkg-build.in: Make sure that python finds sqlite3 when determining which extension modules to build
​3bbc5d8	build/pkgs/python3/spkg-build.in: Set rpath
​124b605	Merge #32698
​bca2141	pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bdbab3a

build/make/Makefile.in: Remove SPKG-tox, SPKG-sdist targets

[mkoeppe]

Reduced the scope of the ticket

[mjo]

I've tried to test these locally but I always get some complaint about sage_setup.py. For example,

$ EXTRA_CONFIGURE_ARGS="--without-system-python3" tox -e local-direct -- build sagelib-tox-python-sagewheels

This proceeded to build everything inside of .tox/local-direct, but then...

cd '/home/mjo/src/sage.git/build/pkgs/sagelib/src' && export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/12/bin:/home/mjo/bin:/home/mjo/bin" && SAGE_SPKG_WHEELS=/home/mjo/src/sage.git/.tox/local-direct/local/var/lib/sage/wheels tox -v -v -v -e python-sagewheels
using tox.ini: /home/mjo/src/sage.git/pkgs/sagemath-standard/tox.ini (pid 12133)
  removing /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/log
using tox-3.24.4 from /usr/lib/python3.9/site-packages/tox/__init__.py (pid 12133)
GLOB start: packaging 
GLOB sdist-make: /home/mjo/src/sage.git/pkgs/sagemath-standard/setup.py
[12140] /home/mjo/src/sage.git/pkgs/sagemath-standard$ /usr/bin/python3.9 setup.py sdist --formats=zip --dist-dir /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/dist >.tox/log/GLOB-0.log
ERROR: invocation failed (exit code 1), logfile: /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/log/GLOB-0.log
================================== log start ===================================
Traceback (most recent call last):
  File "/home/mjo/src/sage.git/pkgs/sagemath-standard/setup.py", line 31, in <module>
    from sage_setup.excepthook import excepthook
ModuleNotFoundError: No module named 'sage_setup'

=================================== log end ====================================
ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python3.9 setup.py sdist --formats=zip --dist-dir /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/dist', 1)
make[1]: *** [Makefile:2798: sagelib-tox-python-sagewheels] Error 2
make[1]: Leaving directory '/home/mjo/src/sage.git/build/make'

real	0m4.988s
user	0m1.223s
sys	0m0.145s
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make sagelib-tox-python-sagewheels'):

It is safe to delete any log files and build directories, but they
contain information that is helpful for debugging build problems.
WARNING: If you now run 'make' again, the build directory of the
same version of the package will, by default, be deleted. Set the
environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

make: *** [Makefile:39: sagelib-tox-python-sagewheels] Error 1
ERROR: InvocationError for command /bin/bash -c 'export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/12/bin:/home/mjo/bin:/home/mjo/bin && : &&  case "" in 1|y*|Y*);; *) ./bootstrap ;; esac &&  : &&  case "" in 1|y*|Y*);; *) ./configure --prefix=/home/mjo/src/sage.git/.tox/local-direct/local --enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes  --without-system-python3 ;; esac &&  case "build sagelib-tox-python-sagewheels" in  bash)    bash -i; exit ;;  config*) ;;  *)       make -k V=0 base-toolchain ;;  esac &&  make -k V=0 SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!rpy2,!sage_sws2rst"  build sagelib-tox-python-sagewheels && ( [ -z "" ] || make -k V=0 SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!rpy2,!sage_sws2rst"  || echo "(error ignored)" ) ' (exited with code 2)
__________________________________________________________________________________ summary __________________________________________________________________________________
ERROR:   local-direct: commands failed

Is that expected (did I construct my example wrong?)

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​e5ad518	pkgs/sagemath-standard/setup.py: Do not depend on sage_setup for sdist or egg_info
​821d16e	src/doc/en/developer/packaging_sage_library.rst: Add bootstrap to testing instructions to avoid the trap of #32868
​77c957f	src/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher
​9375ef8	pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
​c6fc111	Prettier diagram
​7d6a44c	Use :mod: as markup for packages/modules
​f83d424	Improve ABC example
​817a8d0	src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup
​b555735	src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata
​ad12d25	Merge #32899

[mkoeppe]

Should be better with this branch

[mkoeppe]

(needs ./bootstrap)

[mjo]

I just noticed these as well, probably from an empty $CC --version in sage_setup. No idea if they're new, I probably just happened to be looking in the right place at exactly the right time:

  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Running command /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/python-s\
agewheels-nopypi/bin/python /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/\
python-sagewheels-nopypi/lib/python3.9/site-packages/pip/_vendor/pep517/in_proc\
ess/_in_process.py prepare_metadata_for_build_wheel /tmp/tmphl6_9kin
    /bin/sh: 1: --version: not found

  Building wheel for sagemath-standard (PEP 517): started
  Running command /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/python-sag\
ewheels-nopypi/bin/python /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/py\
thon-sagewheels-nopypi/lib/python3.9/site-packages/pip/_vendor/pep517/in_proces\
s/_in_process.py build_wheel /tmp/tmpqovldekw
  /bin/sh: 1: --version: not found

[mkoeppe]

Yes, this is #30876

[mjo]

Looking better now; regardless, the Makefile targets themselves are working as intended.

[mkoeppe]

Thanks!