#29622 closed defect (fixed)

cygwin, conda, homebrew: Fixes for CI

Reported by: mkoeppe Owned by:
Priority: blocker Milestone: sage-9.1
Component: porting: Cygwin Keywords:
Cc: dimpase, slelievre, embray Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 630576b (Commits, GitHub, GitLab) Commit: 630576b0ced872081c792a3b339b3b505d65b404
Dependencies: #29530 Stopgaps:

Status badges

Description

Hopefully the last change for the CI for 9.1

Change History (26)

comment:1 Changed 18 months ago by mkoeppe

  • Summary changed from cygwin: Fixes for CI to cygwin, conda: Fixes for CI

comment:2 Changed 18 months ago by mkoeppe

  • Branch set to u/mkoeppe/cygwin__fixes_for_ci

comment:3 Changed 18 months ago by mkoeppe

  • Commit set to f9700fd16d6faa0701088fd8f9743e0ee2d08207
  • Dependencies set to #29530
  • Summary changed from cygwin, conda: Fixes for CI to cygwin, conda, homebrew: Fixes for CI

Last 10 new commits:

652787b.github/workflows/tox*.yml: Make copying logs from containers more robust
f8c6e7atox.ini: Commit a failed build container so that logs are copied from it
5c8e685tox-optional.yml: Fix likewise
4f4fe60ci-cygwin.yml: Split the file so that unrelated stages do not wait
27c63f1Improve docker login instructions
dbf461dMerge tag '9.1.rc2' into t/29530/make_docker_images_generated_by_github_ci_workflows_available
163562dRename cygwin workflows
97a96c6ci-cygwin*.yml: Fix typos
9187128tox.yml: Add docker-conda-forge
f9700fdtox.yml: Add homebrew-macos-python3_xcode-gcc_spkg

comment:4 Changed 18 months ago by mkoeppe

This fixes some syntax errors in the cygwin build. Now it times out in stage 3. This would need to be rebalanced or split into several parallel stages

Last edited 18 months ago by mkoeppe (previous) (diff)

comment:5 Changed 18 months ago by mkoeppe

  • Status changed from new to needs_review

comment:6 Changed 18 months ago by git

  • Commit changed from f9700fd16d6faa0701088fd8f9743e0ee2d08207 to 8df6b206ee3bd5a7b4bba676a8b6ef0c036bbe89

Branch pushed to git repo; I updated commit sha1. New commits:

8406e25Rebalance stages
99342f1Update cygwin-minimal, cygwin-standard-python2
8df6b20Fixup, rebalance

comment:8 Changed 18 months ago by mkoeppe

Fatal error: can't write 24 bytes to section .text of build/temp.cygwin-3.1.4-x86_64-3.7/build/cythonized/sage/rings/laurent_series_ring_element.o: 'No space left on device'

comment:9 Changed 18 months ago by git

  • Commit changed from 8df6b206ee3bd5a7b4bba676a8b6ef0c036bbe89 to ea92c284b5204560d65478e84c60806cc3a8c2ea

Branch pushed to git repo; I updated commit sha1. New commits:

ea92c28Factor out .github/workflows/extra-sage-local.sh from ci-cygwin*.yml, remove archives after extracting

comment:10 Changed 18 months ago by git

  • Commit changed from ea92c284b5204560d65478e84c60806cc3a8c2ea to 0d4655355f32bb312dcda24292c9a5676428ad2b

Branch pushed to git repo; I updated commit sha1. New commits:

0d46553Fixup

comment:11 Changed 18 months ago by git

  • Commit changed from 0d4655355f32bb312dcda24292c9a5676428ad2b to 9b1214fb51a1d9c5d5790d834ce406bbf51e6490

Branch pushed to git repo; I updated commit sha1. New commits:

2eb8351.github/workflows/*.yml: Use max-parallel so that the multi-stage workflows get a chance to run
9b1214ffiuxp

comment:12 Changed 18 months ago by git

  • Commit changed from 9b1214fb51a1d9c5d5790d834ce406bbf51e6490 to d3b5f8cb275fc7fd9b5aed8a86003c527d140ac2

Branch pushed to git repo; I updated commit sha1. New commits:

801a0d0Rebalance
bf15a04Use tar --remove-files
016df8aPrepare sage-local artifacts in /tmp
c400c37.github/workflows/extract-sage-local.sh: Show df
d3b5f8cFix up listing tar files

comment:13 Changed 18 months ago by mkoeppe

This now manages to build Sage on Cygwin (see cygwin-stage-iii in https://github.com/mkoeppe/sage/runs/640373882)

Last edited 18 months ago by mkoeppe (previous) (diff)

comment:14 Changed 18 months ago by git

  • Commit changed from d3b5f8cb275fc7fd9b5aed8a86003c527d140ac2 to 630576b0ced872081c792a3b339b3b505d65b404

Branch pushed to git repo; I updated commit sha1. New commits:

630576bUpdate cygwin-minimal, cygwin-standard-python2

comment:15 Changed 18 months ago by mkoeppe

Ready for review

comment:16 Changed 18 months ago by mkoeppe

  • Cc slelievre embray added

cygwin-minimal, stage-ii-a and stage-ii-c still time out (https://github.com/mkoeppe/sage/runs/641053384); more rebalancing should be done in the next release cycle. I stop for this ticket and 9.1.

Uploading and downloading of our large sage-local artifacts is somewhat brittle, a game of chance: For example, in https://github.com/mkoeppe/sage/runs/641209977 (cygwin-standard), stage-iii :

Exponential backoff for retry #5. Waiting for 33036.92082171844 milliseconds before continuing the upload at offset 1379926016
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #5, continuing with upload
Total file count: 1 ---- Processed file #0 (0.0%)
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Retry limit has been reached for chunk at offset 1379926016 to https://pipelines.actions.githubusercontent.com/4D1poTSDSSApU42GXYeGgQX6WjHwGPK9ZzdaE2SeHdlUs0ZBhW/_apis/resources/Containers/3201477?itemPath=sage-local-commit-b2c012bfed0fb9414e0db531acc43a3abf1ae1ba-cygwin-standard%5Csage-local-iii.tar
##[warning]Aborting upload for C:\tools\cygwin\tmp\sage-local-iii.tar due to failure
Total size of all the files uploaded is 1379926016 bytes

In the same run, for https://github.com/mkoeppe/sage/runs/641394549 (cygwin-standard-python2), the upload of stage-iii succeeded, and stage-iv-a ... stage-iv-d downloaded it successfully.

  • stage-iv-a (make ptest, https://github.com/mkoeppe/sage/runs/641394549) terminates, with a number of doctest errors and timeouts:
    sage -t src/doc/en/constructions/plotting.rst  # Timed out
    sage -t src/doc/fr/tutorial/interfaces.rst  # 1 doctest failed
    sage -t src/doc/ja/tutorial/interfaces.rst  # 2 doctests failed
    sage -t src/doc/pt/tutorial/interfaces.rst  # Timed out
    sage -t src/doc/ru/tutorial/interfaces.rst  # 6 doctests failed
    sage -t src/sage/coding/binary_code.pyx  # Timed out
    sage -t src/sage/doctest/control.py  # 1 doctest failed
    sage -t src/sage/functions/log.py  # 1 doctest failed
    sage -t src/sage/functions/orthogonal_polys.py  # 1 doctest failed
    sage -t src/sage/functions/other.py  # 2 doctests failed
    sage -t src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # Timed out
    sage -t src/sage/interfaces/gap.py  # 1 doctest failed
    sage -t src/sage/groups/class_function.py  # Timed out
    sage -t src/sage/interfaces/giac.py  # 41 doctests failed
    sage -t src/sage/libs/gap/element.pyx  # 1 doctest failed
    sage -t src/sage/libs/glpk/error.pyx  # 2 doctests failed
    sage -t src/sage/matrix/matrix1.pyx  # 1 doctest failed
    sage -t src/sage/numerical/backends/glpk_backend.pyx  # 2 doctests failed
    sage -t src/sage/parallel/map_reduce.py  # Timed out after testing finished
    sage -t src/sage/repl/interpreter.py  # 3 doctests failed
    sage -t src/sage/rings/number_field/number_field_element.pyx  # Timed out
    sage -t src/sage/sandpiles/sandpile.py  # Timed out
    sage -t src/sage/symbolic/constants.py  # 3 doctests failed
    sage -t src/sage/symbolic/expression_conversions.py  # 1 doctest failed
    sage -t src/sage/symbolic/expression.pyx  # 1 doctest failed
    sage -t src/sage/tests/benchmark.py  # 2 doctests failed
    sage -t src/sage/tests/books/computational-mathematics-with-sagemath/sol/graphique_doctest.py  # 1 doctest failed
    sage -t src/sage/tests/cmdline.py  # 3 doctests failed
    
    Looks like the giac interface is broken.
  • stage-iv-b builds a few optional packages; I have noted build failures in some separate tickets.
  • stage-iv-c builds cbc, sage_numerical_backends_coin successfully.
Last edited 18 months ago by mkoeppe (previous) (diff)

comment:17 Changed 18 months ago by mkoeppe

Needs review. Let's get this into 9.1 please

comment:18 Changed 18 months ago by mkoeppe

  • Priority changed from major to critical

comment:19 follow-up: Changed 18 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

this looks good; one minor question is how to get logs for particular failed spkg builds. Is the latter possible/documented, and I'm just not looking at the right place?

comment:20 in reply to: ↑ 19 Changed 18 months ago by mkoeppe

Replying to dimpase:

one minor question is how to get logs for particular failed spkg builds. Is the latter possible/documented, and I'm just not looking at the right place?

Do you see the artifact named "logs-...." in https://github.com/mkoeppe/sage/actions/runs/94565308?

comment:21 follow-up: Changed 18 months ago by mkoeppe

It contains both the logs of all packages, and the contents of $SAGE_LOCAL/var/tmp/sage/build/ (all failed packages)

comment:22 Changed 18 months ago by mkoeppe

Alternatively, if one does not want to download these huge log archives, one can open the section "Print out logs for immediate inspection" (for example in https://github.com/mkoeppe/sage/runs/641394520?check_suite_focus=true) to see logs of packages with errors. (This already works before artifacts are made available.)

comment:23 in reply to: ↑ 21 Changed 18 months ago by dimpase

Replying to mkoeppe:

It contains both the logs of all packages, and the contents of $SAGE_LOCAL/var/tmp/sage/build/ (all failed packages)

ok - I only looked at "view raw logs" - and these are, despite the name, less complete than artefacts, right?

comment:24 Changed 18 months ago by mkoeppe

"View raw logs" just gives you what is displayed in the right pane. This is a GitHub Actions feature. The log artefacts have a lot more, from my scripts.

comment:25 Changed 18 months ago by mkoeppe

  • Priority changed from critical to blocker

comment:26 Changed 18 months ago by vbraun

  • Branch changed from u/mkoeppe/cygwin__fixes_for_ci to 630576b0ced872081c792a3b339b3b505d65b404
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.