Opened 2 years ago

Last modified 3 weeks ago

#29060 new task

Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.7
Component: build Keywords: docker, ContinuousIntegration, sd111
Cc: vbraun, fbissey, isuruf, dimpase, embray, saraedum, arojas, slelievre, gh-sheerluck, gh-tobiasdiez Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

Testing of source and binary distributions relies too much on manual testing by people.

We propose to set up Dockerfiles and CI scripts that document and test the expected capabilities of source and binary Sage distributions; and of downstream distribution packaging.

Testing of source trees and source distributions: (in progress)

With the important changes brought by #27330 (Meta-ticket: spkg-configure: Try to use as many system packages as possible), sage-the-distribution has become less monolithic; it now interacts in a more complex way with distribution packaging. We propose to add infrastructure for testing correct installation of sage source distributions on a variety of platforms.

  1. We collect information about distribution packages systematically and store it in on a per-SPKG basis in build/pkgs/SPKG/distros/. (Right now this information is scattered - in downstream sage distribution packagers' build scripts, trac tickets, replies to bug reports in google groups, the sage installation manual, personal knowledge...) This is part of #29053, for debian/fedora/arch/conda (with follow-up tickets #29105, #29106 for other platforms).
  1. We generate well-defined test environments in the form of Dockerfiles. This makes it possible to test, on one's development computer, the correct installation of sage-the-distribution on a variety of platforms and configurations. #29053 provides build/bin/write-dockerfile.sh that generates the Dockerfile using the information in 1. The minimal configuration has just the packages that are needed for a build to succeed. The standard configuration installs all distribution packages that sage knows how to use.
  1. Running tests on many test environments is automated using tox in #29053. The top-level tox.ini file defines the test environments. Using tox's factor conditions (https://tox.readthedocs.io/en/latest/config.html#complex-factor-conditions), there is a succinct description of many test environments. For example, we can run tox -e docker-debian-stretch-minimal,docker-arch-latest-standard.
  1. #29087 automates running the tox tests for a selection of 34 platforms/configurations using a GitHub Actions workflow on every git push to a GitHub repository. An example run: https://github.com/mkoeppe/sage/actions/runs/32812271

Tickets:

  • #29053: Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add tox.ini --- this helps test that all spkg-configure.m4 scripts work as expected
  • #29066: Add Dockerfile that sets up sage prerequisites on Arch Linux
  • #29104: Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local)
  • #29143: Add to tox.ini 32-bit linux environments and other architectures supported by docker
  • #29129: Add more system packages to build/pkgs/SPKG/distros/
  • #29273: add system packages info for R and others
  • #29106: Add cygwin package information
  • #29295: Add GitHub CI workflow for cygwin
  • #29087: Add GitHub Actions workflow for testing spkg-configure / build on various Linux distributions
  • #29367: tox.ini: Add tests for python2
  • #29415: tox.ini: Add environment local-conda
  • #29417: tox.ini, GitHub CI workflow - minor fixes
  • #29401: Add documentation of tox and GitHub actions workflow to developer's manual
  • #29366: archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages
  • #29341: GitHub CI workflow: Add testing of optional packages
  • #29403: ci-cygwin.yml: Multi-stage build
  • #29530: Make Docker images generated by GitHub CI workflows available; minor CI fixes
  • #29669: For Sage 9.2: Remove Python 2 support from the build system and CI scripts
  • #29757: .github/workflows/tox-gcc_spkg.yml: Remove python2
  • #29901: .github/workflows/: Test all optional packages, add workflow for testing all experimental packages
  • #29815: .github/workflows/tox-gcc_spkg.yml: Disable tests on some platforms that fail --without-system-gcc
  • #29105: Add gentoo package information, extend dockerfile generator to gentoo
  • #29851: tox.ini: debian-bullseye, -sid have Python3.8 instead of 3.7
  • #30064: Fix tox-docker builds broken by #29884
  • #29091: tox.ini: Add raspbian-buster
  • #29929: tox.ini: Various CI fixes
  • #30195: tox.ini: Add centos-6, docker-nobootstrap, manylinux-*
  • #30044: tox.ini: Test on voidlinux
  • #30144: tox.ini / tox.yml: Move code for tagging/pushing docker images into tox.ini
  • #29130: Add nix package information, extend dockerfile generator to nix
  • #30216 GH Actions: Test Windows with WSL
  • #30923: tox.ini: Add environments local-sudo-ubuntu-standard, etc.
  • #30619: update opensuse.txt across build/pkgs/
  • #30154: cygwin: CI fixes
  • #28745: Add SAGE_ROOT/environment.yml for "conda env create -f".
  • #30726: GH Actions: Test optional/experimental packages with Xcode 12
  • #29527: tox.ini: Add docker-opensuse
  • #30213: tox.ini: Add scientificlinux
  • #32113: GH Actions: conda-forge tests broken
  • #32947: tox.ini, GH Actions: Update Linux platforms
  • #33196: tox.ini: Add centos-stream-8, centos-stream-9, remove defunct centos-6
  • #33296: tox.ini: Add -recommended package factor
  • #33140: GH Actions (macOS): Fix filtering of experimental packages
  • #31415: GH Actions workflow that builds documentation on each push
  • #33306: tox.ini: Linux platform updates
  • #32965: tox / GH Actions: Add centos-7-devtoolset-gcc_{9,10,11}
  • #33263: Add github action running on each push
  • #30845: GH Actions: Add test for conda without SPKG
  • #32703: GH Actions: Run a job for make build-local first, cache image for job make build
  • #31211: ci-cygwin*.yml: Fix packaging up of SAGE_LOCAL/var/tmp/sage/build/*
  • #30933: GH Actions: Repair upload of docker images
  • #32784: Filter singular system package out of tox.ini for homebrew
  • #31088: GH Actions: WSL workflow: No space left on device
  • #31413: GH Actions: Add workflow with configure --enable-editable
  • #32570: tox.ini, GH Actions: Update macOS environments
  • #32966: tox / GH Actions: Add ubuntu variants using ppa:ubuntu-toolchain-r
  • #32531: SAGE_ROOT/tox.ini: Add variants that provision a system python using pyenv
  • #30845: tox.ini, GH Actions: Add test for conda without SPKG
  • #30505: tox.ini: Add wsl as a technology, to complement local and docker
  • #30924: tox.ini: Add cygwin as a technology, to complement local and docker
  • #29536: Make docker images from GitHub CI workflow and regular Sage Docker images interoperable
  • #30217: tox.ini: Add tests with Ubuntu + deadsnakes
  • test gentoo "prefix" builds on top of some Linux distribution
  • #29416: tox.ini: Add local-copy
  • #30506: GH Actions: Use cancel-all-previous-workflow-runs
  • #32280: Manage disk space constraints in continuous integration
  • #21469: Enable VPATH builds (several independent build trees connected to one source tree)
  • #29283: tox.ini: Add test environments using LXC
  • #29159: Add to tox.ini a toxenv docker-linuxbrew
  • #29535: Customize CI workflows on GitHub Actions using specially structured tag names
  • #33338 Refactor GH Actions workflows using https://docs.github.com/en/actions/learn-github-actions/reusing-workflows, also use reusable workflows for the upstream deployments of ci-sage.yml (suggested in https://github.com/pypa/setuptools/pull/2923#discussion_r795977169)

Related meta-tickets:

  • #29356: Meta-ticket: more system pkgs info for various linuxes
  • #29146: Meta-ticket: Refactor/improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh

Ideas without tickets so far:

  • Testing that it is possible to build without error when upgrading with git from some list of previous releases.
  • Test that packages can be cleanly uninstalled

Testing of the sage binary distributions: moved to #31133


Testing of downstream packaging of sage: (planning stage)


Symptoms:

  • #29055: Installing any optional package on the binary distribution triggers full recompile of sagelib
  • #29051: Paths configured in installed numpy site.cfg [DEFAULT/ALL] do not affect scipy

Change History (53)

comment:1 follow-up: Changed 2 years ago by saraedum

  • Cc slelievre added
  • Keywords docker ContinuousIntegration added

I like the general idea of doing more CI a lot. Probably we should split this ticket up though or turn it into a meta ticket that tracks similar CI improvements such as #28457, #24854, #25262.

For this to be actually noticed by people we probably would need some integration with Trac. If I understood embray correctly, adding something like that, similar to the patchbot status, would not be difficult. (Since #28457 has been ready for a while, it might be a good first candidate for such an integration.)

I would propose to use GitLab CI for probably all the things you describe. We already have some CI setup there that has been mostly stable recently (though most people don't know about it https://gitlab.com/sagemath/dev/trac/pipelines?page=1&scope=all since it's not visible in trac yet.) I had made some experiments with macOS & Windows there that were not completely disappointing. For macOS you cannot use docker but still GitLab CI works, see e.g. #25980. Windows can be run inside docker (on Windows) but two years ago, the performance was not sufficient, see #25805. Without docker it worked fine though.

For our Linux CI needs, we can mostly use the free GitLab runners but last time we tried we could also easily get plenty of free credits on Google Cloud. For macOS & Windows, I had at some point built a PoC runner that slelievre volunteered to host; unfortunately, I never finished that project.

comment:2 in reply to: ↑ 1 Changed 2 years ago by mkoeppe

  • Summary changed from Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages to Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Replying to saraedum:

I like the general idea of doing more CI a lot. Probably we should split this ticket up though or turn it into a meta ticket that tracks similar CI improvements such as #28457, #24854, #25262.

For this to be actually noticed by people we probably would need some integration with Trac.

Yes, that would be nice. But it would already help if this were run at the time that betas and releases are prepared. I don't think it needs to be run on every ticket.

comment:3 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:4 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:5 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:6 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:7 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:8 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:9 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:10 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:12 follow-up: Changed 2 years ago by dimpase

This needs a writeup to explain how all these cogs: tox, docker, github actions, etc. fit together and move.

comment:13 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:14 in reply to: ↑ 12 Changed 2 years ago by mkoeppe

Replying to dimpase:

This needs a writeup to explain how all these cogs: tox, docker, github actions, etc. fit together and move.

Good idea, please take a look at the revised description and let me know what needs more detail.

comment:15 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:17 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:18 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:19 Changed 2 years ago by dimpase

On some branches there are also lists of <distro> packages in build/pkgs/<distro>.txt which appear to be used for some kind of bootstrapping. What are these?

comment:20 Changed 2 years ago by mkoeppe

build/pkgs/DISTRO.txt contains the minimal requirements for building sage from a source distribution and build/pkgs/DISTRO-bootstrap.txt are the additional requirements so that ./bootstrap works. In #29124 I propose to create pseudo-spkgs to capture these, instead, to make this a bit more uniform.

comment:21 Changed 2 years ago by mkoeppe

Comments on the top of build/pkgs/debian.txt explain this.

comment:22 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:23 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:24 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:25 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:26 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:27 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:28 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:29 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:30 Changed 2 years ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

comment:31 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:32 Changed 2 years ago by mkoeppe

  • Cc gh-sheerluck added
  • Description modified (diff)

comment:33 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:34 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:35 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:36 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:37 Changed 20 months ago by mkoeppe

  • Description modified (diff)

comment:38 Changed 19 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:39 Changed 18 months ago by mkoeppe

  • Description modified (diff)

comment:40 Changed 18 months ago by mkoeppe

  • Description modified (diff)

comment:41 Changed 18 months ago by mkoeppe

  • Keywords sd111 added

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

comment:42 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:43 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:44 Changed 13 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Moving to 9.4, as 9.3 has been released.

comment:45 Changed 10 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:46 Changed 5 months ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6

comment:47 Changed 5 months ago by mkoeppe

  • Cc gh-tobiasdiez added
  • Description modified (diff)

comment:48 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:49 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:50 Changed 3 months ago by mkoeppe

  • Description modified (diff)

comment:51 Changed 3 months ago by mkoeppe

  • Description modified (diff)

comment:52 Changed 3 months ago by mkoeppe

  • Description modified (diff)

comment:53 Changed 3 weeks ago by mkoeppe

  • Milestone changed from sage-9.6 to sage-9.7
Note: See TracTickets for help on using tickets.