#31535 closed enhancement (fixed)

New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ...

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.5
Component: porting Keywords:
Cc: jhpalmieri, dimpase, gh-kliem, mjo, gh-tobiasdiez Merged in:
Authors: Matthias Koeppe Reviewers: Michael Orlitzky
Report Upstream: N/A Work issues:
Branch: ad12d25 (Commits, GitHub, GitLab) Commit: ad12d251cc7a6a5a7a5a369310bc1e7ad89b8f66
Dependencies: #32899 Stopgaps:

Status badges

Description (last modified by mkoeppe)

For all script packages SPKG that have an embedded source tree -- currently sagelib, sage_docbuild, sage_sws2rst -- we add a new target:

  • SPKG-tox-% (which calls tox -e % in the package's source tree)

These targets are particularly useful for invoking them within the portability tests of the top-level tox:

  • $ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox invokes the tox testing of the sage_docbuild distribution
  • $ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
  • $ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
  • $ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nobuildisolation-nopypi (needs #30913, #31384) uses the src/environment.yml file to set up a complete environment from conda and then tests sagelib in a venv using tox

Change History (27)

comment:1 Changed 23 months ago by mkoeppe

Branch: u/mkoeppe/new_make_targets__sagelib_sdist____sage_docbuild_sdist_______and__sagelib_tox__________

comment:2 Changed 23 months ago by mkoeppe

Authors: Matthias Koeppe
Cc: jhpalmieri dimpase gh-kliem added
Commit: 1eef6d6a8a2eeff9508b936ac8ececbf9f60f362
Status: newneeds_review

New commits:

1eef6d6build/make/Makefile.in: Add targets SPKG-sdist, SPKG-tox, SPKG-tox-% for script packages

comment:3 Changed 23 months ago by git

Commit: 1eef6d6a8a2eeff9508b936ac8ececbf9f60f36269649337a75524486c51aff2547f0b8ead9e36cf

Branch pushed to git repo; I updated commit sha1. New commits:

6964933build/make/Makefile.in: fixup

comment:4 Changed 23 months ago by mkoeppe

Description: modified (diff)

comment:5 Changed 23 months ago by mkoeppe

Description: modified (diff)

comment:6 Changed 23 months ago by mkoeppe

Description: modified (diff)

comment:7 Changed 23 months ago by git

Commit: 69649337a75524486c51aff2547f0b8ead9e36cf188bf2e3bd076e5b1966422c10e7b9dae5affd36

Branch pushed to git repo; I updated commit sha1. New commits:

9993348build/make/Makefile.in: Run targets SPKG-tox, SPKG-tox-% outside of sage-env, using SAGE_ORIG_PATH - so that tox from conda works
4533018tox.ini: Add local-conda-environment-src
188bf2etox.ini, src/doc/bootstrap: Get tox into standard builds

comment:8 Changed 23 months ago by mkoeppe

Description: modified (diff)

comment:9 Changed 23 months ago by mkoeppe

Description: modified (diff)

comment:10 Changed 23 months ago by mkoeppe

Status: needs_reviewneeds_work

comment:11 Changed 22 months ago by mkoeppe

Milestone: sage-9.3sage-9.4

Moving this ticket to 9.4, as it seems unlikely that it will be merged in 9.3, which is in the release candidate stage

comment:12 Changed 19 months ago by mkoeppe

Milestone: sage-9.4sage-9.5

Setting a new milestone for this ticket based on a cursory review.

comment:13 Changed 16 months ago by mkoeppe

Dependencies: #32442, #32698

comment:14 Changed 16 months ago by git

Commit: 188bf2e3bd076e5b1966422c10e7b9dae5affd36bca214134dc225603801470f7bb2999e9e6bdbca

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

8af6041Merge tag '9.5.beta2' into t/32442/__configure__make___with_sage_venv_the_default
b3f7d34.gitignore: Add /venv
fc4b641configure.ac: Remove conveniene symlinks before (re)creating them
8bcbf1dMakefile (distclean): Remove convenience symlinks prefix, venv
dcb4a08configure.ac: Indicate default in help string for --with-sage-venv
e33bc89Merge tag '9.5.beta3' into t/32442/__configure__make___with_sage_venv_the_default
2101b8cbuild/pkgs/python3/spkg-build.in: Make sure that python finds sqlite3 when determining which extension modules to build
3bbc5d8build/pkgs/python3/spkg-build.in: Set rpath
124b605Merge #32698
bca2141pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels

comment:15 Changed 16 months ago by mkoeppe

Cc: mjo gh-tobiasdiez added
Status: needs_workneeds_review

comment:16 Changed 16 months ago by git

Commit: bca214134dc225603801470f7bb2999e9e6bdbcabdbab3aca65138c1a80009c546df53af04d740f6

Branch pushed to git repo; I updated commit sha1. New commits:

bdbab3abuild/make/Makefile.in: Remove SPKG-tox, SPKG-sdist targets

comment:17 Changed 16 months ago by mkoeppe

Description: modified (diff)
Summary: New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ...

Reduced the scope of the ticket

comment:18 Changed 14 months ago by mjo

I've tried to test these locally but I always get some complaint about sage_setup.py. For example,

$ EXTRA_CONFIGURE_ARGS="--without-system-python3" tox -e local-direct -- build sagelib-tox-python-sagewheels

This proceeded to build everything inside of .tox/local-direct, but then...

cd '/home/mjo/src/sage.git/build/pkgs/sagelib/src' && export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/12/bin:/home/mjo/bin:/home/mjo/bin" && SAGE_SPKG_WHEELS=/home/mjo/src/sage.git/.tox/local-direct/local/var/lib/sage/wheels tox -v -v -v -e python-sagewheels
using tox.ini: /home/mjo/src/sage.git/pkgs/sagemath-standard/tox.ini (pid 12133)
  removing /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/log
using tox-3.24.4 from /usr/lib/python3.9/site-packages/tox/__init__.py (pid 12133)
GLOB start: packaging 
GLOB sdist-make: /home/mjo/src/sage.git/pkgs/sagemath-standard/setup.py
[12140] /home/mjo/src/sage.git/pkgs/sagemath-standard$ /usr/bin/python3.9 setup.py sdist --formats=zip --dist-dir /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/dist >.tox/log/GLOB-0.log
ERROR: invocation failed (exit code 1), logfile: /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/log/GLOB-0.log
================================== log start ===================================
Traceback (most recent call last):
  File "/home/mjo/src/sage.git/pkgs/sagemath-standard/setup.py", line 31, in <module>
    from sage_setup.excepthook import excepthook
ModuleNotFoundError: No module named 'sage_setup'

=================================== log end ====================================
ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python3.9 setup.py sdist --formats=zip --dist-dir /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/dist', 1)
make[1]: *** [Makefile:2798: sagelib-tox-python-sagewheels] Error 2
make[1]: Leaving directory '/home/mjo/src/sage.git/build/make'

real	0m4.988s
user	0m1.223s
sys	0m0.145s
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make sagelib-tox-python-sagewheels'):

It is safe to delete any log files and build directories, but they
contain information that is helpful for debugging build problems.
WARNING: If you now run 'make' again, the build directory of the
same version of the package will, by default, be deleted. Set the
environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

make: *** [Makefile:39: sagelib-tox-python-sagewheels] Error 1
ERROR: InvocationError for command /bin/bash -c 'export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/12/bin:/home/mjo/bin:/home/mjo/bin && : &&  case "" in 1|y*|Y*);; *) ./bootstrap ;; esac &&  : &&  case "" in 1|y*|Y*);; *) ./configure --prefix=/home/mjo/src/sage.git/.tox/local-direct/local --enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes  --without-system-python3 ;; esac &&  case "build sagelib-tox-python-sagewheels" in  bash)    bash -i; exit ;;  config*) ;;  *)       make -k V=0 base-toolchain ;;  esac &&  make -k V=0 SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!rpy2,!sage_sws2rst"  build sagelib-tox-python-sagewheels && ( [ -z "" ] || make -k V=0 SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!rpy2,!sage_sws2rst"  || echo "(error ignored)" ) ' (exited with code 2)
__________________________________________________________________________________ summary __________________________________________________________________________________
ERROR:   local-direct: commands failed

Is that expected (did I construct my example wrong?)

comment:19 Changed 14 months ago by mkoeppe

Dependencies: #32442, #32698#32899

comment:20 Changed 14 months ago by git

Commit: bdbab3aca65138c1a80009c546df53af04d740f6ad12d251cc7a6a5a7a5a369310bc1e7ad89b8f66

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

e5ad518pkgs/sagemath-standard/setup.py: Do not depend on sage_setup for sdist or egg_info
821d16esrc/doc/en/developer/packaging_sage_library.rst: Add bootstrap to testing instructions to avoid the trap of #32868
77c957fsrc/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher
9375ef8pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
c6fc111Prettier diagram
7d6a44cUse :mod: as markup for packages/modules
f83d424Improve ABC example
817a8d0src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup
b555735src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata
ad12d25Merge #32899

comment:21 Changed 14 months ago by mkoeppe

Should be better with this branch

comment:22 Changed 14 months ago by mkoeppe

(needs ./bootstrap)

comment:23 Changed 14 months ago by mjo

I just noticed these as well, probably from an empty $CC --version in sage_setup. No idea if they're new, I probably just happened to be looking in the right place at exactly the right time:

  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Running command /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/python-s\
agewheels-nopypi/bin/python /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/\
ess/_in_process.py prepare_metadata_for_build_wheel /tmp/tmphl6_9kin
    /bin/sh: 1: --version: not found
  Building wheel for sagemath-standard (PEP 517): started
  Running command /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/python-sag\
ewheels-nopypi/bin/python /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/py\
s/_in_process.py build_wheel /tmp/tmpqovldekw
  /bin/sh: 1: --version: not found

comment:24 Changed 14 months ago by mkoeppe

Yes, this is #30876

comment:25 Changed 14 months ago by mjo

Reviewers: Michael Orlitzky
Status: needs_reviewpositive_review

Looking better now; regardless, the Makefile targets themselves are working as intended.

comment:26 Changed 14 months ago by mkoeppe


comment:27 Changed 14 months ago by vbraun

Branch: u/mkoeppe/new_make_targets__sagelib_sdist____sage_docbuild_sdist_______and__sagelib_tox__________ad12d251cc7a6a5a7a5a369310bc1e7ad89b8f66
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.