ci-cygwin*.yml: delegate to tox, add more stages, use more specific SAGE_LOCAL

[mkoeppe]

We revise the CI workflows for Cygwin, now going through SAGE_ROOT/tox.ini for the package installation and invoking the new system package scripts from there.

We also add more stages to make the workflow more robust; this was originally done in #29152.

We also make another improvement that was easy to do: The build now uses SAGE_LOCAL=/opt/sage-$COMMIT instead of /sage/local. This will make it easier to download and install several versions of Sage.

[mkoeppe]

Last 10 new commits:

​898758d	tox.ini: Add local-root
​559dd8e	tox.ini (local-root): Pass --enable-build-as-root to configure
​a85f41c	tox.ini (local): Do not build the toolchain when posargs = config.status or posargs = configure
​1de912a	tox.ini (local-sudo): Run apt-get update with sudo
​3c7e5c4	tox.ini (local-root, local-sudo): Run output of sage-print-system-package-command through eval
​f7ff30c	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard
​e9ca2c1	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard
​be4177b	Merge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
​f74fe3d	tox.ini (local-cygwin-choco): New
​b2ae68a	.github/workflows/ci-cygwin*.yml: Install cygwin packages via tox

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​11cf8db

.github/workflows/ci-cygwin*.yml: Use SAGE_LOCAL as set by tox

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​05d57c0

.github/workflows/ci-cygwin-*.yml: Pass PREFIX to tox

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d87e160

.github/workflows/extract-sage-local.sh: Use PREFIX

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0827c85	Quoting fixes
​00a9182	Just use github.sha
​b327031	.github/workflows/extract-sage-local.sh: Fix up path

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9ad611d	tox.ini, build/bin/write-dockerfile.sh: Add !sage_sws2rst, !rpy2 to SAGE_CHECK_PACKAGES
​48657d2	build/pkgs/sage_sws2rst/dependencies: Fix typo - use SAGE_CHECK_sage_sws2rst
​380c718	Merge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31062/tox___gh_actions__disable_testsuites_of_packages_depending_on_pip_packages__pytest_______if_there_is_no_ssl
​b25ea3b	build/pkgs/rpy2/dependencies: If SAGE_CHECK=yes, depend on ipython
​a0d99d7	Merge branch 't/31062/tox___gh_actions__disable_testsuites_of_packages_depending_on_pip_packages__pytest_______if_there_is_no_ssl' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq

[mkoeppe]

Merged #31062 to fix a merge conflict

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cea4cd5	ci-cygwin-standard.yml: More stages, continue-on-error: true
​cf31b79	Fixup after cherry-pick

[mkoeppe]

Integrated the workflow changes from #29152

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​09d1737	Makefile: Add targets ptest-nodoc etc.
​0dca776	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/31084/makefile__add__ptest__targets_that_do_not_depend_on_the_docbuild
​073124c	Merge branch 't/31084/makefile__add__ptest__targets_that_do_not_depend_on_the_docbuild' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq
​8769bd6	.github/workflows/ci-cygwin-*.yml: Separate docbuild and ptest

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d2e4b83

Fix up stage iv

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​3826222

Fix up stage iv

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​598b0c2

local-cygwin-choco: Do not pass through PATH, use full pathname of choco instead

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​3cf5ee4	Merge commit 'a50ddf88975086b14a49895e371477df00fd57b5' of git://trac.sagemath.org/sage into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​b4927e1	sage.misc.package: Remove/adjust non-robust doctests
​e5fe752	Merge tag '9.3.beta4' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​78ff9d5	src/sage/misc/package.py: Add one more # optional - build
​a44042f	Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap
​64bde5f	Merge tag '9.3.beta5' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​e9a7572	src/sage/misc/package.py: Improve source formatting
​c7bcda9	Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap
​9988c5f	ci-cygwin*.yml: Adjust to new script packages _bootstrap, _prereq
​d65299c	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ab19133	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard
​a5e4051	Merge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq

[mkoeppe]

Follow-up: #31211

[gh-tobiasdiez]

A few remarks after skimming the changes:

+ PACKAGES="python38 python38-pip"

I think it's better to install python using the python action, which brings a few goodies along: ​https://github.com/marketplace/actions/setup-python

+ C:\tools\cygwin\bin\bash -l -x -c

Can you try to put this as shell: C:\\tools\\cygwin\\bin\\bash -l -x -c {0}? Would make the workflow easier to read.

And then my usual rant about using tox for this ;-). I know you are a big fan of using tox, and this is fine. But I still think it's not the best tool for github workflows. You are cramping a lot of things into one command, which makes it hard to see at which stake the workflow fails. I would propose to add helper scripts so that the workflow looks like

• Install dependencies: choco install (build/bin/get-dependencies cygwin) --source cygwin
• Build: build/bin/build-local (with the script that is currently in tox: local, running bootstrap, configure and make)
• Test: tox (running all python tests)

The scripts can of course be reused in tox.ini still giving you the chance to locally use tox for this (although I find it counterintuitive that tox might change my system installation).

[mkoeppe]

Replying to gh-tobiasdiez:

+ C:\tools\cygwin\bin\bash -l -x -c

Can you try to put this as shell: C:\\tools\\cygwin\\bin\\bash -l -x -c {0}? Would make the workflow easier to read.

I'm pretty sure I went through several iterations, including use of the shell directive, before settling on something that actually works

[mkoeppe]

Replying to gh-tobiasdiez:

You are cramping a lot of things into one command, which makes it hard to see at which stake the workflow fails.

This has not been a problem in my experience.

[mkoeppe]

Let me explain again why I like to use tox to drive as much as possible in our CI. tox is a stable tool that has been under long term maintenance. For context, before we (well, I) started to use GH Actions, we (well, individual developers many of whom no longer have sufficient time) used buildbot, Circle CI, Travis CI, GitLab pipelines, ... Because our project is surfing on "free compute", the available technologies are changing faster than our project can sustain. Therefore it is important not to spend too much effort on development specific to the CI platform.

The present ticket consolidates GH Actions-specific code for installing Cygwin to just become part of the tox workflow. It is not enough to just factor through scripts that "could" also be run by developers on a local Windows machine: We currently do not even have any developers who do that. By using tox inside the GH Actions workflow, we actually make sure that these scripts are usable (and remain usable when changes are made) outside of a GH Actions environment.

[mkoeppe]

Replying to gh-tobiasdiez:

A few remarks after skimming the changes:

+ PACKAGES="python38 python38-pip"

I think it's better to install python using the python action, which brings a few goodies along: ​https://github.com/marketplace/actions/setup-python

This is used in choco install $PACKAGES --source cygwin to install Cygwin and then these package in Cygwin.

I don't think the setup-python action can do that.

[gh-tobiasdiez]

Thanks for the explanation. That's indeed something I understand and appreciate. My only problem is that managing the user system locally using tox is not very handy in my experience.

Here is an example I was experiencing the last few hours: The tox run encountered problems in the docbuild stage. The reason was/is a that ecl under wsl 1 has some problems and I needed to play around installing/deinstalling it combined with other changes. However, I couldn't use the tox command that the GH workflow uses since this installs ecl again. What I ended up doing was to copy & paste the commands printed by the tox run on github. That worked to some extend, but was a rather awful experience.

I think it is good to have an easy way for developers to investigate problems in the GH workflow on their local system. And for this, it is in my opinion crucial that they can quickly recreate the state that is used. In my opinion a clearer separation in system-setup, build and test is helpful in this regard. If this can be done using tox, sure why not.

[mkoeppe]

Replying to gh-tobiasdiez:

managing the user system locally using tox is not very handy in my experience.

Right. Managing the global installation of Cygwin is not the final form of this development. What I would actually prefer (but have not had time to figure out -- any help is welcome) is to have the local-cygwin tox environment make an isolated installation of Cygwin. (I do this for the local-conda and local-homebrew environments.)

[mkoeppe]

Replying to gh-tobiasdiez:

Here is an example I was experiencing the last few hours: The tox run encountered problems in the docbuild stage. The reason was/is a that ecl under wsl 1 has some problems and I needed to play around installing/deinstalling it combined with other changes. However, I couldn't use the tox command that the GH workflow uses since this installs ecl again. What I ended up doing was to copy & paste the commands printed by the tox run on github. That worked to some extend, but was a rather awful experience.

OK, I have run into this situation too. I think what is needed for the local-... environments is:

• an environment variable that can be passed to tox to disable system package installs
• a target or environment variable to give an interactive shell.

I will open a ticket for this.

[gh-tobiasdiez]

That would also work, thanks! Please also give a thought to my suggestion to extract some of the scripts in tox.ini to scripts that can be called independently of tox.

[mkoeppe]

This is now #31216

[mkoeppe]

Replying to gh-tobiasdiez:

Please also give a thought to my suggestion to extract some of the scripts in tox.ini to scripts that can be called independently of tox.

I see two directions of improvement here:

• Moving stuff from tox.ini (for tox -e local-...) and build/bin/write-dockerfile.sh (for tox -e docker-...) into our top-level Makefile. For example, the complicated rule for the commands of local consists entirely of workarounds for things that should really be fixed/improved there.
• Improving the system package information scripts such as sage-print-system-package-command further.

Meta-ticket #29146 keeps track of such improvements, let's continue there.

On the other hand, I would be reluctant to add more scripts that developers can use to set up their system automatically. As I have written elsewhere, I think the current approach of printing command lines to educate developers about how to do things on their system is better for the Sage community than trying to provide scripts that do things automagically.

[gh-tobiasdiez]

Replying to mkoeppe:

I see two directions of improvement here: Meta-ticket #29146 keeps track of such improvements, let's continue there.

Sounds good to me! Sorry for starting this somewhat off-topic discussion, but I very much appreciate your detailed explanations. It's always good to learn more about sage's inner workings ;-). I'll review this ticket as soon as the dependencies are merged (as it's hard to see right now which changes are coming from this ticket).

[mkoeppe]

Great, and thanks for the discussion.

[dimpase]

lgtm

[mkoeppe]

Thank you!

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​42f4458

Merge tag '9.3.beta7' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq