Opened 2 years ago

Last modified 5 weeks ago

#29146 new enhancement

Meta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh

Reported by: mkoeppe Owned by:
Priority: minor Milestone: sage-9.6
Component: build Keywords:
Cc: embray, vbraun, dimpase, jhpalmieri, gh-sheerluck, slelievre, gh-tobiasdiez, slabbe Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/meta_ticket__refactor_and_improve_system_package_related_scripts__tox_ini__build_bin_write_dockerfile_sh (Commits, GitHub, GitLab) Commit: fd9b928d573b6740a9aadb1dca04a066d268cc26
Dependencies: #29417 Stopgaps:

Status badges

Description (last modified by mkoeppe)

Improvements to low-level system packages scripts:

  • #30861, #30865, #30947, #30951, #30968: Refactor system package code in tox.ini, build/bin/write_dockerfile.sh by extending and using sage_bootstrap
  • Extend build/bin/sage-print-system-package-command so that it can provide all distribution-specific commands such as apt-get install for build/bin/write_dockerfile.sh

Improvements to the system package database:

  • #29124 Add script packages build/pkgs/_prereq, build/pkgs/_toolchain, build/pkgs/_bootstrap
  • define shell quoting in files like debian.txt, handle options correctly when installing packages one by one, fix up passing installation options when IGNORE_MISSING_PACKAGES=yes

Improvements to tox docker:

  • #29536 Make docker images from GitHub CI workflow and regular Sage Docker images interoperable
  • optionally create a script for docker run instead of a Dockerfile (this gives more flexibility for multiarch - see #29143)
  • cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
  • move log extraction code from tox.yml to tox.ini (#29530 did some preparation already)
  • system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
  • Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use docker ps --filter label=.... to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
  • setenv/passenv SAGE_SERVER
  • Remove TARGETS_PRE/TARGETS distinction: Instead leave a stamp file and exit from make with an error to trigger the ADD of src/; for example, add sagelib dependency on $(SAGE_SRC)/sage, and add the rule to Makefile:
    $(SAGE_SRC)/sage:
    	touch .need_src; exit 1
    
  • use environment variable MAKE for docker builds as well, instead of asking users to pass the USE_MAKEFLAGS variable

Improvements to tox local:

  • #31216: tox.ini (local): Add environment variables to skip system package installs, mechanism for a local interactive shell
  • #31064: ci-cygwin*.yml: delegate to tox, add more stages, use more specific SAGE_LOCAL
  • #21469: Enable VPATH builds (several independent build trees connected to one source tree)
  • #29416: tox.ini: Add local-copy
  • the complicated rule for the commands of local consists entirely of workarounds for things that should really be fixed/improved in our Makefiles:
    • #30721: build/make/Makefile.in: base-toolchain should be a dependency of every non-base, non-toolchain package
    • #32785: tox.ini and build/bin/write-dockerfile.sh use better defaults for SAGE_CHECK_PACKAGES; this should be moved to build/make/Makefile.in (SAGE_CHECK_PACKAGES_DEFAULT_yes).
    • #31535 New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...

Other improvements to tox.ini:

  • #32764: Make tox -p auto and tox --listenvs useful again (it does not understand the nested braces)
  • #31574: tox.ini, build/bin/write-dockerfile.sh: Do not use make -k by default

Improvements to auto-generated parts of the manuals:

  • #29558 Autogenerated parts of installation guide
  • #29557 Add script package _recommended and generate "recommended system packages" info

Improvements for messages at the end of a ./configure run:

  • #29560 sage-guess-package-system: Ignore conda if no environment is activated
  • Perhaps sage-guess-package-system should report several package systems such as conda ubuntu or homebrew cpan when that makes sense
  • #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
  • #30624: Improve wording and formatting of configure's recommendation message
  • #29372 - At the end of configure, show installation hints for disabled optional packages separately
  • Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
  • #32060 configure: Change defaults to --with-system-gcc=force, --with-system-python3=force
  • #29586: Improve configure's recommendations
  • #30863: refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4

Tickets regarding sage_bootstrap:

  • #20023 sage_bootstrap: Remove Python 2.6 support
  • #20104 sage --package: Add commands list :standard:, update-latest (for packages from PyPI), upload
  • #29919 Restore "huge" package type

See also:

  • #29060 Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Change History (48)

comment:1 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:2 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:3 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:4 Changed 23 months ago by mkoeppe

  • Description modified (diff)

comment:5 Changed 23 months ago by mkoeppe

  • Description modified (diff)

comment:6 Changed 23 months ago by mkoeppe

  • Cc jhpalmieri added
  • Description modified (diff)
  • Summary changed from Refactor build/bin/write_dockerfile.sh to Refactor tox.ini and build/bin/write_dockerfile.sh

comment:7 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:8 Changed 22 months ago by mkoeppe

  • Cc slelievre added
  • Summary changed from Refactor tox.ini and build/bin/write_dockerfile.sh to Meta-ticket: Refactor tox.ini and build/bin/write_dockerfile.sh

comment:9 Changed 22 months ago by mkoeppe

  • Summary changed from Meta-ticket: Refactor tox.ini and build/bin/write_dockerfile.sh to Meta-ticket: Refactor system package related scripts, tox.ini, build/bin/write_dockerfile.sh

comment:10 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:12 Changed 22 months ago by mkoeppe

  • Dependencies set to #29417
  • Description modified (diff)
  • Milestone changed from sage-9.1 to sage-9.2

comment:13 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:14 Changed 21 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Meta-ticket: Refactor system package related scripts, tox.ini, build/bin/write_dockerfile.sh to Meta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh

comment:15 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:17 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:18 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:19 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:20 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:21 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:22 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:23 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:24 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:25 Changed 21 months ago by mkoeppe

  • Branch set to u/mkoeppe/meta_ticket__refactor_and_improve_system_package_related_scripts__tox_ini__build_bin_write_dockerfile_sh

comment:26 Changed 21 months ago by git

  • Commit set to fd9b928d573b6740a9aadb1dca04a066d268cc26

Branch pushed to git repo; I updated commit sha1. New commits:

fd9b928build/bin/sage-logger [V=0]: Show build times of successful builds

comment:27 Changed 19 months ago by mkoeppe

  • Cc gh-sheerluckslelievre added; slelievre removed

comment:28 Changed 19 months ago by mkoeppe

  • Cc gh-sheerluck slelievre added; gh-sheerluckslelievre removed

comment:29 Changed 19 months ago by mkoeppe

  • Description modified (diff)

comment:30 Changed 18 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:31 Changed 17 months ago by mkoeppe

  • Cc gh-tobiasdiez added

comment:32 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:33 Changed 15 months ago by mkoeppe

  • Cc slabbe added
  • Description modified (diff)

comment:34 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:35 Changed 14 months ago by mkoeppe

  • Description modified (diff)

comment:36 Changed 14 months ago by mkoeppe

  • Description modified (diff)

comment:37 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:38 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:39 Changed 10 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

comment:40 Changed 10 months ago by mkoeppe

  • Description modified (diff)

comment:41 Changed 10 months ago by mkoeppe

  • Description modified (diff)

comment:42 Changed 10 months ago by mkoeppe

  • Description modified (diff)

comment:43 Changed 10 months ago by mkoeppe

  • Description modified (diff)

comment:44 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:45 Changed 6 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:46 Changed 3 months ago by mkoeppe

  • Description modified (diff)

comment:47 Changed 3 months ago by mkoeppe

  • Description modified (diff)

comment:48 Changed 5 weeks ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6
Note: See TracTickets for help on using tickets.