Opened 3 years ago

Last modified 3 months ago

#29146 new enhancement

Meta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh

Reported by: Matthias Köppe Owned by:
Priority: minor Milestone: sage-9.8
Component: build Keywords:
Cc: Erik Bray, Volker Braun, Dima Pasechnik, John Palmieri, Andrew, Samuel Lelièvre, Tobias Diez, Sébastien Labbé Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/meta_ticket__refactor_and_improve_system_package_related_scripts__tox_ini__build_bin_write_dockerfile_sh (Commits, GitHub, GitLab) Commit: fd9b928d573b6740a9aadb1dca04a066d268cc26
Dependencies: #29417 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

Improvements to low-level system packages scripts:

  • #30861, #30865, #30947, #30951, #30968: Refactor system package code in tox.ini, build/bin/write_dockerfile.sh by extending and using sage_bootstrap
  • #33873: Extend build/bin/sage-print-system-package-command so that it can provide all distribution-specific commands such as apt-get install for build/bin/write_dockerfile.sh
  • #29041 sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"
  • #33860 sage-package: Add commands "dependencies", "trees"

Improvements to the system package database:

  • #29124 Add script packages build/pkgs/_prereq, build/pkgs/_toolchain, build/pkgs/_bootstrap
  • define shell quoting in files like debian.txt, handle options correctly when installing packages one by one, fix up passing installation options when IGNORE_MISSING_PACKAGES=yes

Improvements to tox docker:

  • #29536 Make docker images from GitHub CI workflow and regular Sage Docker images interoperable
  • optionally create a script for docker run instead of a Dockerfile (this gives more flexibility for multiarch - see #29143)
  • cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
  • move log extraction code from tox.yml to tox.ini (#29530 did some preparation already)
  • system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
  • Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use docker ps --filter label=.... to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
  • setenv/passenv SAGE_SERVER
  • Remove TARGETS_PRE/TARGETS distinction: Instead leave a stamp file and exit from make with an error to trigger the ADD of src/; for example, add sagelib dependency on $(SAGE_SRC)/sage, and add the rule to Makefile:
    $(SAGE_SRC)/sage:
    	touch .need_src; exit 1
    
  • use environment variable MAKE for docker builds as well, instead of asking users to pass the USE_MAKEFLAGS variable

Improvements to tox local:

  • #31216: tox.ini (local): Add environment variables to skip system package installs, mechanism for a local interactive shell
  • #31064: ci-cygwin*.yml: delegate to tox, add more stages, use more specific SAGE_LOCAL
  • #21469: Enable VPATH builds (several independent build trees connected to one source tree)
  • #29416: tox.ini: Add local-copy
  • the complicated rule for the commands of local consists entirely of workarounds for things that should really be fixed/improved in our Makefiles:
    • #30721: build/make/Makefile.in: base-toolchain should be a dependency of every non-base, non-toolchain package
    • #32785: tox.ini and build/bin/write-dockerfile.sh use better defaults for SAGE_CHECK_PACKAGES; this should be moved to build/make/Makefile.in (SAGE_CHECK_PACKAGES_DEFAULT_yes).
    • #31535 New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...

Other improvements to tox.ini:

  • #32764: Make tox -p auto and tox --listenvs useful again (it does not understand the nested braces)
  • #31574: tox.ini, build/bin/write-dockerfile.sh: Do not use make -k by default

Improvements to auto-generated parts of the manuals:

  • #29558 Autogenerated parts of installation guide
  • #29557 Add script package _recommended and generate "recommended system packages" info

Improvements for messages at the end of a ./configure run:

  • #29560 sage-guess-package-system: Ignore conda if no environment is activated
  • Perhaps sage-guess-package-system should report several package systems such as conda ubuntu or homebrew cpan when that makes sense
  • #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
  • #30624: Improve wording and formatting of configure's recommendation message
  • #29372 - At the end of configure, show installation hints for disabled optional packages separately
  • Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
  • #32060 configure: Change defaults to --with-system-gcc=force, --with-system-python3=force
  • #29586: Improve configure's recommendations
  • #30863: refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4

Tickets regarding sage_bootstrap:

  • #20023 sage_bootstrap: Remove Python 2.6 support
  • #20104 sage --package: Add commands list :standard:, update-latest (for packages from PyPI), upload
  • #29919 Restore "huge" package type

See also:

  • #29060 Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Change History (52)

comment:1 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:2 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:3 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:4 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:5 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:6 Changed 3 years ago by Matthias Köppe

Cc: John Palmieri added
Description: modified (diff)
Summary: Refactor build/bin/write_dockerfile.shRefactor tox.ini and build/bin/write_dockerfile.sh

comment:7 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:8 Changed 3 years ago by Matthias Köppe

Cc: Samuel Lelièvre added
Summary: Refactor tox.ini and build/bin/write_dockerfile.shMeta-ticket: Refactor tox.ini and build/bin/write_dockerfile.sh

comment:9 Changed 3 years ago by Matthias Köppe

Summary: Meta-ticket: Refactor tox.ini and build/bin/write_dockerfile.shMeta-ticket: Refactor system package related scripts, tox.ini, build/bin/write_dockerfile.sh

comment:10 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:11 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:12 Changed 3 years ago by Matthias Köppe

Dependencies: #29417
Description: modified (diff)
Milestone: sage-9.1sage-9.2

comment:13 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:14 Changed 3 years ago by Matthias Köppe

Description: modified (diff)
Summary: Meta-ticket: Refactor system package related scripts, tox.ini, build/bin/write_dockerfile.shMeta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh

comment:15 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:16 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:17 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:18 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:19 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:20 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:21 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:22 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:23 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:24 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:25 Changed 3 years ago by Matthias Köppe

Branch: u/mkoeppe/meta_ticket__refactor_and_improve_system_package_related_scripts__tox_ini__build_bin_write_dockerfile_sh

comment:26 Changed 3 years ago by git

Commit: fd9b928d573b6740a9aadb1dca04a066d268cc26

Branch pushed to git repo; I updated commit sha1. New commits:

fd9b928build/bin/sage-logger [V=0]: Show build times of successful builds

comment:27 Changed 2 years ago by Matthias Köppe

Cc: gh-sheerluckslelievre added; Samuel Lelièvre removed

comment:28 Changed 2 years ago by Matthias Köppe

Cc: Andrew Samuel Lelièvre added; gh-sheerluckslelievre removed

comment:29 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:30 Changed 2 years ago by Matthias Köppe

Milestone: sage-9.2sage-9.3

comment:31 Changed 2 years ago by Matthias Köppe

Cc: Tobias Diez added

comment:32 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:33 Changed 2 years ago by Matthias Köppe

Cc: Sébastien Labbé added
Description: modified (diff)

comment:34 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:35 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:36 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:37 Changed 23 months ago by Matthias Köppe

Description: modified (diff)

comment:38 Changed 23 months ago by Matthias Köppe

Description: modified (diff)

comment:39 Changed 21 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

comment:40 Changed 20 months ago by Matthias Köppe

Description: modified (diff)

comment:41 Changed 20 months ago by Matthias Köppe

Description: modified (diff)

comment:42 Changed 20 months ago by Matthias Köppe

Description: modified (diff)

comment:43 Changed 20 months ago by Matthias Köppe

Description: modified (diff)

comment:44 Changed 17 months ago by Matthias Köppe

Description: modified (diff)

comment:45 Changed 17 months ago by Matthias Köppe

Milestone: sage-9.4sage-9.5

comment:46 Changed 13 months ago by Matthias Köppe

Description: modified (diff)

comment:47 Changed 13 months ago by Matthias Köppe

Description: modified (diff)

comment:48 Changed 12 months ago by Matthias Köppe

Milestone: sage-9.5sage-9.6

comment:49 Changed 8 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:50 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:51 Changed 6 months ago by Matthias Köppe

Description: modified (diff)

comment:52 Changed 3 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8
Note: See TracTickets for help on using tickets.