Opened 3 years ago

Closed 3 years ago

#29417 closed enhancement (fixed)

tox.ini, GitHub CI workflow - minor fixes, more homebrew configuration factors, improve docker interface

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.1
Component: porting Keywords:
Cc: John Palmieri, Dima Pasechnik, Michael Orlitzky, gh-kliem, Volker Braun Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: b7bb418 (Commits, GitHub, GitLab) Commit: b7bb418343ab34c77a10af6d0f725111ba780662
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

This makes the following changes to tox.ini:

  • tox -e local-homebrew-macos-standard now uses python3 from homebrew.

This attempts to reproduce the environment of https://groups.google.com/d/msg/sage-release/kU5M1QVuQQY/Zu10Q-CKAwAJ

  • tox -e local-homebrew-macos-standard-python3_xcode uses /usr/bin/python3 (was the old default)
  • tox -e local-homebrew-macos-standard-python3_pythonorg uses the Python 3.7 installation from the binary package from python.org

New toolchain factors: gcc_spkg, gcc_9.

Improved docker interface.

Change History (74)

comment:1 Changed 3 years ago by Matthias Köppe

Dependencies: #29087, #29354

comment:2 Changed 3 years ago by Matthias Köppe

Branch: u/mkoeppe/tox-ci-minor-fixes

comment:3 Changed 3 years ago by Matthias Köppe

Authors: Matthias Koeppe
Commit: b2596714c1ea73f69f65d6e081a8156c0d9e7f75
Dependencies: #29087, #29354#29087, #29354, #29418

Last 10 new commits:

fbb7393Fixup packages
cfcd8f2build/bin/sage-guess-package-system, build/bin/sage-print-system-package-command: Handle slackware too
136feebbuild/bin/sage-spkg: Show equivalent system packages for ALL distros
9939051Merge branch 't/29354/tox_ini__add_slackware_environments' into t/29087/CI-gh-actions-tox-configure-build
4ac9e24Reduce SAGE_NUM_THREADS to 3
1cb8f98build/pkgs/curl/distros/slackware.txt: Add openldap-client
eee2877build/pkgs/slackware.txt: Add ca-certificates
c1931f5build/pkgs/curl/distros/slackware.txt: Add libssh2
3145467Merge branch 't/29354/tox_ini__add_slackware_environments' into t/29087/CI-gh-actions-tox-configure-build
b259671[tox-local-homebrew-macos] Repair make parallelization

comment:4 Changed 3 years ago by git

Commit: b2596714c1ea73f69f65d6e081a8156c0d9e7f75b1489e791332bacd4760036430d0c5e886a09b31

Branch pushed to git repo; I updated commit sha1. New commits:

bd24641build/bin/sage-download-file: Add option --no-check-certificate
e143939build/bin/sage-spkg: Append to SAGE_DOWNLOAD_FILE_OPTIONS instead of overwriting it
710b52dMerge branch 't/29418/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3' into t/29417/tox-ci-minor-fixes
b1489e7tox.ini: Set SAGE_DOWNLOAD_FILE_OPTIONS=--no-check-certificate

comment:5 Changed 3 years ago by Matthias Köppe

Dependencies: #29087, #29354, #29418#29087, #29354, #29418, #29415

comment:6 Changed 3 years ago by git

Commit: b1489e791332bacd4760036430d0c5e886a09b31846f14ee93f40daa75443fa17a70fc852e641d1c

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

354cf90Use conda-install --yes --quiet
df609e0Merge branch 't/29415/tox_ini__add_environment_local_conda' into t/29417/tox-ci-minor-fixes
cf27321build/bin/sage-download-file: Add option --no-check-certificate
90ea00bbuild/bin/sage-spkg: Append to SAGE_DOWNLOAD_FILE_OPTIONS instead of overwriting it
bccc2c2Merge branch 't/29418/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3' into t/29417/tox-ci-minor-fixes
87b7351tox.ini: Add configuration factors python3, macos-python3_xcode, macos-python3_pythonorg
329ea99tox.yml: Test more homebrew configs
de1bce2tox.yml: Add local-ubuntu-conda test
86d5c84tox.ini: Add toolchain factors
846f14etox.yml: Test more Fedora, no conda

comment:7 Changed 3 years ago by Matthias Köppe

Dependencies: #29087, #29354, #29418, #29415#29087, #29418, #29415

comment:8 Changed 3 years ago by git

Commit: 846f14ee93f40daa75443fa17a70fc852e641d1c9838fd77534b2675010a64cdfa03b029c37ee4ed

Branch pushed to git repo; I updated commit sha1. New commits:

9838fd7fixup

comment:9 Changed 3 years ago by git

Commit: 9838fd77534b2675010a64cdfa03b029c37ee4eda3ccd4644e87ef1cc869d3c183463556156ec2b7

Branch pushed to git repo; I updated commit sha1. New commits:

a3ccd46build/pkgs/python3/distros/homebrew.txt: New

comment:10 Changed 3 years ago by Matthias Köppe

Cc: John Palmieri Dima Pasechnik added
Description: modified (diff)
Summary: tox.ini, GitHub CI workflow - minor fixestox.ini, GitHub CI workflow - minor fixes, more homebrew configuration factors

comment:11 Changed 3 years ago by Matthias Köppe

Dependencies: #29087, #29418, #29415#29087, #29418, #29415, #29401

comment:12 Changed 3 years ago by git

Commit: a3ccd4644e87ef1cc869d3c183463556156ec2b72bc124328a8b7abdcea32c074dc0091d5f1c12d0

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

6153945merge the sections on tox factors
ab11ebdelide some details from terminal output
c7fcf91change 'ls -l' to 'ls -la' and elide some details
e73bc8dElide more details, shorten directory names more
a31b2a0Reviewer patch
b7b45feReviewer patch
f7d2dc9Remove old section head
99aca39Reformat list, remove unfinished commented out section
2c53ae2Merge branch 't/29401/add_documentation_of_tox_and_github_actions_workflow_to_developer_s_manual' into t/29417/tox-ci-minor-fixes
2bc1243src/doc/en/developer/portability_testing.rst: Add documentation on new local-conda and local-homebrew environments

comment:13 Changed 3 years ago by git

Commit: 2bc124328a8b7abdcea32c074dc0091d5f1c12d0a54ab32b76caaaaa6e46f4315aaab23d3a6e5e30

Branch pushed to git repo; I updated commit sha1. New commits:

5dd7652tox.ini: Add more linuxmint versions
c6cc486tox.ini: IGNORE_MISSING_SYSTEM_PACKAGES=yes for old linuxmint versions
cd72da9tox.ini: Add linuxmint-i386
7986ab3tox.yml: Add more linuxmint
a54ab32tox.ini: Remove linuxmint-20 (no docker image)

comment:14 Changed 3 years ago by git

Commit: a54ab32b76caaaaa6e46f4315aaab23d3a6e5e304281c921878ab82c946b58e4b6e05464798d55c6

Branch pushed to git repo; I updated commit sha1. New commits:

3757212tox.ini: Switch local-conda-forge to miniforge
a61c48etox.ini [local-conda]: Activate conda before calling conda install
8f58411Merge branch 't/29415/tox_ini__add_environment_local_conda' into t/29417/tox-ci-minor-fixes
4281c92Replace commands_pre by commands, to support tox < 3.4

comment:15 Changed 3 years ago by git

Commit: 4281c921878ab82c946b58e4b6e05464798d55c649aeac3892aa8750e0317788f152548b7ee3cb9b

Branch pushed to git repo; I updated commit sha1. New commits:

49aeac3tox.ini: New factor nokegonly for homebrew

comment:16 Changed 3 years ago by git

Commit: 49aeac3892aa8750e0317788f152548b7ee3cb9bc7a7061079e3391600e55d7db6bde6fbfe003c82

Branch pushed to git repo; I updated commit sha1. New commits:

c7a7061tox.ini: Fix up 'nokegonly'

comment:17 Changed 3 years ago by git

Commit: c7a7061079e3391600e55d7db6bde6fbfe003c825477536d9b6ad3c5d1158db2f6e9229b0e999371

Branch pushed to git repo; I updated commit sha1. New commits:

5477536tox.ini: Pass through DOCKER_BUILDKIT

comment:18 Changed 3 years ago by git

Commit: 5477536d9b6ad3c5d1158db2f6e9229b0e9993710b5a73ab955ca75f5326001c4ea86d472f7152f6

Branch pushed to git repo; I updated commit sha1. New commits:

0b5a73atox.ini: Add factor python3_spkg

comment:19 Changed 3 years ago by git

Commit: 0b5a73ab955ca75f5326001c4ea86d472f7152f685b3c516c5fd1de4674d6c91158022a8c8999f1c

Branch pushed to git repo; I updated commit sha1. New commits:

85b3c51tox.ini: Set fedora <= 29 to IGNORE_MISSING_SYSTEM_PACKAGES=yes

comment:20 Changed 3 years ago by git

Commit: 85b3c516c5fd1de4674d6c91158022a8c8999f1c1414208509fdcb965a61762f7cf7c7bf3cba1c56

Branch pushed to git repo; I updated commit sha1. New commits:

1414208Remove linuxmint-20, IGNORE_MISSING_SYSTEM_PACKAGES=yes for all linuxmint because of libbraiding

comment:21 Changed 3 years ago by git

Commit: 1414208509fdcb965a61762f7cf7c7bf3cba1c56ba57c1c8dfa36b3313a528fe6b04f269aecd78a9

Branch pushed to git repo; I updated commit sha1. New commits:

ba57c1ctox.yml: Add debian-stretch

comment:22 Changed 3 years ago by git

Commit: ba57c1c8dfa36b3313a528fe6b04f269aecd78a98f7e774c15fd7af477d3ca3205ff9e97eae90ca6

Branch pushed to git repo; I updated commit sha1. New commits:

544d886build/bin/write-dockerfile.sh: Add 'FROM ... AS ...' so that docker build --target works
2845728build/bin/write-dockerfile.sh: Add V=0 to default USE_MAKEFLAGS, to match 'tox local'
e655a03tox.ini [docker]: Loop through DOCKER_TARGETS, tag intermediate images
09bdad0tox.ini [docker]: Override default TARGETS_PRE and TARGETS using posargs, to match 'tox local' interface
4c8c72fFixup: Need to repeat ARG after FROM
8f7e774src/doc/en/developer/portability_testing.rst: Document docker targets

comment:23 Changed 3 years ago by Matthias Köppe

Cc: Michael Orlitzky gh-kliem added
Description: modified (diff)
Status: newneeds_review
Summary: tox.ini, GitHub CI workflow - minor fixes, more homebrew configuration factorstox.ini, GitHub CI workflow - minor fixes, more homebrew configuration factors, improve docker interface

comment:24 Changed 3 years ago by git

Commit: 8f7e774c15fd7af477d3ca3205ff9e97eae90ca603faba11ec9d1e6cad5e60466faa8ef696605d76

Branch pushed to git repo; I updated commit sha1. New commits:

03faba1tox.ini: Fixup: Pass --always to git describe so it does not fail

comment:25 Changed 3 years ago by git

Commit: 03faba11ec9d1e6cad5e60466faa8ef696605d7667dc11c42392bd1e033372f4dbb783254549c85a

Branch pushed to git repo; I updated commit sha1. New commits:

67dc11cbuild/bin/write-dockerfile.sh: Fixup for arch

comment:26 Changed 3 years ago by git

Commit: 67dc11c42392bd1e033372f4dbb783254549c85ab564e063bb762ae240c314ac9104bfb9a421fea8

Branch pushed to git repo; I updated commit sha1. New commits:

b564e06tox.ini [debian-stretch]: Set IGNORE_MISSING_SYSTEM_PACKAGES=yes

comment:27 Changed 3 years ago by git

Commit: b564e063bb762ae240c314ac9104bfb9a421fea80c648dc25c0c40ee37daf18c4fa36addfbad9bbc

Branch pushed to git repo; I updated commit sha1. New commits:

0c648dctox.ini [ubuntu-bionic]: Set IGNORE_MISSING_SYSTEM_PACKAGES=yes

comment:28 Changed 3 years ago by git

Commit: 0c648dc25c0c40ee37daf18c4fa36addfbad9bbca2b12066d0284dbcca66c212ba890ad22f5ed47c

Branch pushed to git repo; I updated commit sha1. New commits:

a2b1206.github/workflows/tox.yml: Rename the jobs to improve display in GitHub Actions summary

comment:29 Changed 3 years ago by gh-kliem

Do you know how to resolve this? Is this resolved already?

 Error processing tar file(exit status 1): write /sage/local/share/doc/sage/html/en/reference/modsym/sage/modular/modsym/relation_matrix_pyx.html: no space left on device
ERROR: InvocationError: '/usr/bin/docker build . -f /home/runner/work/sage-test-27122/sage-test-27122/.tox/docker-ubuntu-trusty-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS=   --build-arg BASE_IMAGE=ubuntu:trusty --build-arg USE_MAKEFLAGS=-k V=0 SAGE_NUM_THREADS=3 --build-arg TARGETS=build ptest'
___________________________________ summary ____________________________________
ERROR:   docker-ubuntu-trusty-standard: commands failed
##[error]Process completed with exit code 1.

This appears to be the reason that every single test is marked as failed, which is kind of annoying.

https://github.com/kliem/sage-test-27122/actions/runs/72102779

comment:30 in reply to:  29 Changed 3 years ago by Matthias Köppe

Replying to gh-kliem:

Do you know how to resolve this? Is this resolved already?

 Error processing tar file(exit status 1): write /sage/local/share/doc/sage/html/en/reference/modsym/sage/modular/modsym/relation_matrix_pyx.html: no space left on device
ERROR: InvocationError: '/usr/bin/docker build . -f /home/runner/work/sage-test-27122/sage-test-27122/.tox/docker-ubuntu-trusty-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS=   --build-arg BASE_IMAGE=ubuntu:trusty --build-arg USE_MAKEFLAGS=-k V=0 SAGE_NUM_THREADS=3 --build-arg TARGETS=build ptest'
___________________________________ summary ____________________________________
ERROR:   docker-ubuntu-trusty-standard: commands failed
##[error]Process completed with exit code 1.

Help with this would be most welcome!

I have the following browser tabs open, which should help with finding a solution. Essentially, one needs to reconfigure the docker daemon to use larger dm.basesize.

Docker - Using systemd to control the Docker daemon https://success.docker.com/article/using-systemd-to-control-the-docker-daemon

"systemctl set-property docker" - Google Search https://www.google.com/search?safe=off&biw=834&bih=836&sxsrf=ALeKk02AnzFOj0bspN6PGW6rKhDhhp3f4w%3A1586407757228&ei=TamOXtqxDYKWtQW5zr3YBA&q=%22systemctl+set-property+docker%22&oq=%22systemctl+set-property+docker%22&gs_lcp=CgZwc3ktYWIQAzIECAAQHjoECAAQRzoGCAAQFhAeSgkIFxIFMTItNjJKCAgYEgQxMi00UNOnAVi1rQFg2a8BaABwAngAgAF4iAHlAZIBAzAuMpgBAKABAaoBB2d3cy13aXo&sclient=psy-ab&ved=0ahUKEwiajNq2xdroAhUCS60KHTlnD0s4ChDh1QMIDA&uact=5

Docker - How to resolve Resource Temporarily Unavailable errors due to TaskMax? setting https://success.docker.com/article/how-to-reserve-resource-temporarily-unavailable-errors-due-to-tasksmax-setting

"systemctl status docker.service" dm.basesize - Google Search https://www.google.com/search?safe=off&sxsrf=ALeKk01dBCgUutnZt6DU3HHzqaJyqDfA_w%3A1586408161041&ei=4aqOXpKKAtHKswXerrtY&q=%22systemctl+status+docker.service%22+dm.basesize&oq=%22systemctl+status+docker.service%22+dm.basesize&gs_lcp=CgZwc3ktYWIQAzIFCCEQoAE6BAgAEEc6BggAEBYQHjoFCAAQzQI6CAghEBYQHRAeSgoIFxIGMTItMTE0SgkIGBIFMTItMThQpp4DWN7FA2DvyANoBHADeACAAaIBiAHcD5IBBDAuMTWYAQCgAQGqAQdnd3Mtd2l6&sclient=psy-ab&ved=0ahUKEwjS_qD3xtroAhVR5awKHV7XDgsQ4dUDCAw&uact=5

Cannot Change Docker Size on Ubuntu 16.04 - DevOps? Stack Exchange https://devops.stackexchange.com/questions/4062/cannot-change-docker-size-on-ubuntu-16-04

uggds/vagrant-docker https://github.com/uggds/vagrant-docker

Can't install Docker - Cloudera Community https://community.cloudera.com/t5/Support-Questions/Can-t-install-Docker/td-p/205969

docker disk size - Google Search https://www.google.com/search?safe=off&sxsrf=ALeKk02gDtblI6DspE7pP0chAR0f7lCuJQ%3A1586404959534&ei=X56OXs-YIMHisAWJ-rWwBg&q=docker+disk+size&oq=docker+disk+size&gs_lcp=CgZwc3ktYWIQAzICCAAyBggAEAcQHjIGCAAQBxAeMgYIABAHEB4yAggAMgYIABAHEB4yBggAEAcQHjIGCAAQBxAeMgYIABAHEB4yAggAOgQIABBHOgYIABAFEB46CAgAEAcQBRAeOgQIABANOggIABAHEAoQHjoICAAQCBAHEB5KEAgXEgwxMC0yMTJnMTg0ZzdKDAgYEggxMC0xZzJnMlDhZVjfdGDjeGgDcAJ4AIAB4gGIAfwNkgEFMC40LjWYAQCgAQGqAQdnd3Mtd2l6&sclient=psy-ab&ved=0ahUKEwiPxNSAu9roAhVBMawKHQl9DWYQ4dUDCAw&uact=5

elasticsearch - Disk size of a docker image - Stack Overflow https://stackoverflow.com/questions/48751032/disk-size-of-a-docker-image

dockerd | Docker Documentation https://docs.docker.com/engine/reference/commandline/dockerd/

virtual-environments/Ubuntu1804-README.md at master · actions/virtual-environments https://github.com/actions/virtual-environments/blob/master/images/linux/Ubuntu1804-README.md

Docker Engine 18.09 release notes | Docker Documentation https://docs.docker.com/engine/release-notes/18.09/

systemctl(1) — systemd — Debian jessie — Debian Manpages https://manpages.debian.org/jessie/systemd/systemctl.1.en.html

comment:31 Changed 3 years ago by gh-kliem

This looks really complicated. I'm afraid this is way above what I know.

Does this help?

https://github.community/t5/GitHub-Actions/BUG-Strange-quot-No-space-left-on-device-quot-IOExceptions-on/m-p/47691#M6920

comment:32 Changed 3 years ago by Matthias Köppe

No worries - at least I was able to close this browser window now!

comment:33 Changed 3 years ago by gh-kliem

It appears that the solution in the link I provided, is fixing at least part of the problem.

I have put this

       - uses: actions/checkout@v2
+      - name: free disk space
+        run: |
+          sudo swapoff -a
+          sudo rm -f /swapfile
+          sudo apt clean
+          docker rmi $(docker image ls -aq)
+          df -h
       - name: Install test prerequisites

in the debian/ubuntu in .github/workflows/tox.yml and it seems to help. There are green checkmarks again. E.g. debian-jessie minimal passes https://github.com/kliem/sage-test-27122/actions/runs/75035611, which previously had that no space left on disk error.

Now many of minimal runs pass, but the standards show error messages as

The command '/bin/sh -c make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS}' returned a non-zero code: 2
ERROR: InvocationError: '/bin/bash -c for docker_target in with-targets; do  docker build . -f /home/runner/work/sage-test-27122/sage-test-27122/.tox/docker-debian-jessie-standard/Dockerfile  --target $docker_target  --tag sage-docker-debian-jessie-standard-$docker_target:$(git describe --dirty --always)  --build-arg EXTRA_CONFIGURE_ARGS="  "  --build-arg BASE_IMAGE=debian:jessie  --build-arg TARGETS_PRE=sagelib-build-deps  --build-arg TARGETS=build  --build-arg USE_MAKEFLAGS="-k V=0 SAGE_NUM_THREADS=3" --build-arg TARGETS="build ptest"; done'

comment:34 Changed 3 years ago by gh-kliem

In the link they also cleanup old artifacts

    - name: cleanup old action artifacts
      run: .github/workflows/purge_artifacts.sh ${{ secrets.REPO_ACCESS_TOKEN }}

I don't know what this is for. But maybe that is also helpful.

comment:35 in reply to:  33 ; Changed 3 years ago by gh-kliem

That is probably the expected behavior when not all tests pass.

Replying to gh-kliem:

The command '/bin/sh -c make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS}' returned a non-zero code: 2
ERROR: InvocationError: '/bin/bash -c for docker_target in with-targets; do  docker build . -f /home/runner/work/sage-test-27122/sage-test-27122/.tox/docker-debian-jessie-standard/Dockerfile  --target $docker_target  --tag sage-docker-debian-jessie-standard-$docker_target:$(git describe --dirty --always)  --build-arg EXTRA_CONFIGURE_ARGS="  "  --build-arg BASE_IMAGE=debian:jessie  --build-arg TARGETS_PRE=sagelib-build-deps  --build-arg TARGETS=build  --build-arg USE_MAKEFLAGS="-k V=0 SAGE_NUM_THREADS=3" --build-arg TARGETS="build ptest"; done'

comment:36 in reply to:  35 ; Changed 3 years ago by Matthias Köppe

Replying to gh-kliem:

That is probably the expected behavior when not all tests pass.

Replying to gh-kliem:

The command '/bin/sh -c make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS}' returned a non-zero code: 2
ERROR: InvocationError: '/bin/bash -c for docker_target in with-targets; do  docker build . -f /home/runner/work/sage-test-27122/sage-test-27122/.tox/docker-debian-jessie-standard/Dockerfile  --target $docker_target  --tag sage-docker-debian-jessie-standard-$docker_target:$(git describe --dirty --always)  --build-arg EXTRA_CONFIGURE_ARGS="  "  --build-arg BASE_IMAGE=debian:jessie  --build-arg TARGETS_PRE=sagelib-build-deps  --build-arg TARGETS=build  --build-arg USE_MAKEFLAGS="-k V=0 SAGE_NUM_THREADS=3" --build-arg TARGETS="build ptest"; done'

Yes, this is expected behavior.

But it would perhaps be better to change it so that the docker build succeeds if there are only testsuite failures

comment:37 in reply to:  36 Changed 3 years ago by Matthias Köppe

Replying to mkoeppe:

But it would perhaps be better to change it so that the docker build succeeds if there are only testsuite failures

Working on a solution for this.

comment:38 Changed 3 years ago by gh-kliem

I think the above thing with the disk space is more important.

I like the green check marks, because then I can just trust that everything worked fine. However, it would still be nice, if one can quickly access which tests failed. But maybe that is non-trivial and then that's just the way it is.

Also I still hope that very soon we will have most of the runs clean either way.

comment:39 in reply to:  38 Changed 3 years ago by Matthias Köppe

Replying to gh-kliem:

I think the above thing with the disk space is more important.

Yes, I'm trying out your patch

comment:40 Changed 3 years ago by gh-kliem

Maybe one could have failing doctests communicated as neutral runs (according to github this is also a possible state). I don't know how much work it would be to accomplish this, but it would be really nice. Than one knows at the first glance what's going on.

comment:41 Changed 3 years ago by gh-kliem

I don't think it is supported anymore https://help.github.com/en/actions/building-actions/setting-exit-codes-for-actions

I rather have failed tests as failure so that one has to check the details. Otherwise we might overlook critical failing tests.

comment:42 in reply to:  41 Changed 3 years ago by Matthias Köppe

Replying to gh-kliem:

I rather have failed tests as failure so that one has to check the details. Otherwise we might overlook critical failing tests.

In the branch that I'm testing right now this is conveniently configurable

comment:44 Changed 3 years ago by git

Commit: a2b12066d0284dbcca66c212ba890ad22f5ed47ce45bc0e2904e646fe49653c03240dc972db7d2d2

Branch pushed to git repo; I updated commit sha1. New commits:

856e35ctox.ini, tox.yml: Do not fail if errors occur while making $TARGETS_OPTIONAL (default: ptest)
484e01dnaive guess to clean disk space
94f64a8tox fixup
e389763tox fixup
ad359c4.github/workflows/scan-logs.sh: Fixup regex
e45bc0e.github/workflows/scan-logs.sh: Rewording for better display

comment:45 Changed 3 years ago by git

Commit: e45bc0e2904e646fe49653c03240dc972db7d2d23a020319251682375bcc30f862ceb5a92c6d106f

Branch pushed to git repo; I updated commit sha1. New commits:

19edf1dtox.ini: Store successfully built docker tags in {toxenv}/Dockertags
3a02031tox.yml: Retrieve logs from successful builds via {toxenv}/Dockertags

comment:47 Changed 3 years ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:48 Changed 3 years ago by git

Commit: 3a020319251682375bcc30f862ceb5a92c6d106f53e2a83bee28cc0d5eb0d49a429b99a152300e03

Branch pushed to git repo; I updated commit sha1. New commits:

06c28fe.github/workflows/tox.yml: Pass TARGETS to tox so that errors in doc-html count as failures
5ce717b.github/workflows/scan-logs.sh: Fixup regexp for scanning ptest.log
53e2a83.github/workflows/scan-logs.sh: Also catch errors of the form ImportError

comment:49 Changed 3 years ago by Matthias Köppe

Status: needs_workneeds_review

comment:50 Changed 3 years ago by Matthias Köppe

This now seems to work very well. https://github.com/mkoeppe/sage/actions/runs/76629492

Failing TARGETS_OPTIONAL (default = ptest) do not abort the build. These show as green checkmarks in the left pane, but for all failing doctest logs (both packages and sagelib ptest) also a yellow "Check warning" is displayed as annotation in the right pane.

By setting TARGETS_OPTIONAL to include names of optional packages, we can also test these in the same way.

comment:51 Changed 3 years ago by Matthias Köppe

Dependencies: #29087, #29418, #29415, #29401

comment:52 Changed 3 years ago by git

Commit: 53e2a83bee28cc0d5eb0d49a429b99a152300e0370d472ed7608131e677f99c29320960092bdb252

Branch pushed to git repo; I updated commit sha1. New commits:

70d472etox.ini: Quoting fix

comment:53 Changed 3 years ago by git

Commit: 70d472ed7608131e677f99c29320960092bdb252d94969c9e1c62ddca2ce76b596f0889d9d2cb310

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

6fc9a18tox fixup
4d2366btox fixup
828394b.github/workflows/scan-logs.sh: Fixup regex
a39ca99.github/workflows/scan-logs.sh: Rewording for better display
0a02589tox.ini: Store successfully built docker tags in {toxenv}/Dockertags
d209e8etox.yml: Retrieve logs from successful builds via {toxenv}/Dockertags
f1858e7.github/workflows/tox.yml: Pass TARGETS to tox so that errors in doc-html count as failures
f9aca59.github/workflows/scan-logs.sh: Fixup regexp for scanning ptest.log
b32e1bf.github/workflows/scan-logs.sh: Also catch errors of the form ImportError
d94969ctox.ini: Quoting fix

comment:54 Changed 3 years ago by Matthias Köppe

Rebased on 9.1.rc0

comment:55 Changed 3 years ago by Matthias Köppe

comment:56 Changed 3 years ago by git

Commit: d94969c9e1c62ddca2ce76b596f0889d9d2cb310faf0ed6844c26f8f2c10f48ef28d10e1d6ad7a31

Branch pushed to git repo; I updated commit sha1. New commits:

faf0ed6tox.ini: Another quoting fix

comment:57 Changed 3 years ago by git

Commit: faf0ed6844c26f8f2c10f48ef28d10e1d6ad7a31aed3fb8ac844d499696da2b164251d1b8afa4787

Branch pushed to git repo; I updated commit sha1. New commits:

aed3fb8src/doc/en/developer/portability_testing.rst: Explain how to read GitHub Actions page

comment:58 Changed 3 years ago by Matthias Köppe

Can we finalize the review of this one please to get it into 9.1?

comment:59 Changed 3 years ago by Dima Pasechnik

running GH Actions tests on this here: https://github.com/dimpase/sage/pull/3

comment:60 in reply to:  59 Changed 3 years ago by Dima Pasechnik

Replying to dimpase:

running GH Actions tests on this here: https://github.com/dimpase/sage/pull/3

I see

/Users/runner/runners/2.168.0/work/sage/sage/build/bin/sage-venv "/Users/runner/runners/2.168.0/work/sage/sage/.tox/local-homebrew-macos-python3_pythonorg-standard/local"
2020-04-15T16:31:11.9205990Z /bin/sh: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3: No such file or directory
2020-04-15T16:31:11.9206560Z make[1]: *** [/Users/runner/runners/2.168.0/work/sage/sage/.tox/local-homebrew-macos-python3_pythonorg-standard/local/pyvenv.cfg] Error 127
2020-04-15T16:31:11.9207060Z make[1]: Target `doc-html' not remade because of errors.

there. Isn't this ticked supposed to be fixing this?

comment:61 Changed 3 years ago by Matthias Köppe

Thanks for catching this. Turns out I forgot to install the python.org python3 in tox.yml. (This cannot be done in tox.ini because it is not relocatable, see comment there)

comment:62 Changed 3 years ago by git

Commit: aed3fb8ac844d499696da2b164251d1b8afa4787d39d627cdf483437a2f7ee22bb5cd4603347f9d7

Branch pushed to git repo; I updated commit sha1. New commits:

d39d627tox.yml [python3_xcode]: Install the package

comment:63 Changed 3 years ago by git

Commit: d39d627cdf483437a2f7ee22bb5cd4603347f9d7ba0c26646db41f60981f29d94797d0fbe9327e81

Branch pushed to git repo; I updated commit sha1. New commits:

ba0c266fixup

comment:64 Changed 3 years ago by git

Commit: ba0c26646db41f60981f29d94797d0fbe9327e819060d237204d7a4ffadb7caf12baf0980d7c1e06

Branch pushed to git repo; I updated commit sha1. New commits:

9060d23fixup

comment:65 Changed 3 years ago by git

Commit: 9060d237204d7a4ffadb7caf12baf0980d7c1e06b7bb418343ab34c77a10af6d0f725111ba780662

Branch pushed to git repo; I updated commit sha1. New commits:

b7bb418tox.yml [macos-python3_pythonorg]: Use sudo with installer

comment:66 Changed 3 years ago by Dima Pasechnik

comment:67 Changed 3 years ago by Dima Pasechnik

homebrew-macos-python3_pythonorg, standard (https://github.com/dimpase/sage/runs/591414002) crashes at [dochtml] stage - not sure whether one can get complete logs/dochtml.log file from GitHub run above

comment:68 Changed 3 years ago by Dima Pasechnik

Reviewers: Dima Pasechnik
Status: needs_reviewpositive_review

I presume that various errors on local-macos standard runs are to be debugged individually.

e.g. on https://github.com/dimpase/sage/runs/591413919 (homebrew-macos-python3_xcode, minimal) I see ##[error] ==== ERROR IN LOG FILE artifacts/logs-commit-8bb28ec8ffd6869c6b86e3f5224333decb049d09-tox-local-homebrew-macos-python3_xcode-minimal/log/pkgs/gc-7.6.4.p0.log ==== and Warning: Error testing package gc-7.6.4.p0 (ignored)

comment:69 Changed 3 years ago by Matthias Köppe

Thanks!

comment:70 Changed 3 years ago by Matthias Köppe

Yes, errors are to be fixed on their own tickets.

comment:71 Changed 3 years ago by Matthias Köppe

Some of the errors are actually "only" testsuite failures. Tickets such as #29501 clean up some of the messages.

comment:72 in reply to:  67 Changed 3 years ago by Matthias Köppe

Replying to dimpase:

homebrew-macos-python3_pythonorg, standard (https://github.com/dimpase/sage/runs/591414002) crashes at [dochtml] stage - not sure whether one can get complete logs/dochtml.log file from GitHub run above

Yes, download the log artifact. But it only says

------------------------------------------------------------------------
(no backtrace available)
------------------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred.
This probably occurred because a *compiled* module has a bug
in it and is not properly wrapped with sig_on(), sig_off().
Python will now terminate.
------------------------------------------------------------------------
/Users/runner/runners/2.169.0/work/sage/sage/src/bin/sage-python: line 2: 14003 Segmentation fault: 11  sage -python "$@"

comment:73 Changed 3 years ago by Matthias Köppe

Cc: Volker Braun added

comment:74 Changed 3 years ago by Volker Braun

Branch: u/mkoeppe/tox-ci-minor-fixesb7bb418343ab34c77a10af6d0f725111ba780662
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.