Opened 4 months ago

Last modified 8 weeks ago

#33222 needs_work enhancement

GH Actions: Create daily integration branch

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.7
Component: distribution Keywords:
Cc: vbraun, dimpase Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/gh_actions__create_daily_integration_branch (Commits, GitHub, GitLab) Commit: 37c76a6bacdfd1c43e5bc8541e1af0c2055bb5a0
Dependencies: #33233, #31529, #33103, #33277 Stopgaps:

Status badges

Description (last modified by mkoeppe)

We create a GH Actions workflow, run daily (intended for running on sagetrac-mirror) as a cron job ( https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule).

It can also be triggered manually via workflow_dispatch.

Example run at https://github.com/mkoeppe/sage/runs/5060070979

  1. Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using git releasemgr merge-all --milestone=current
  1. Run using tox/docker on a stable platform (ubuntu-focal-recommended). (The new package factor recommended is like standard, but with texlive etc. added, as needed for doc-pdf). We set SAGE_DOCTEST_RANDOM_SEED=0 to reduce random failures.

Only testsuite failures that do not already fail in the baseline are considered failures.

  1. Docker images are pushed to ghcr.io, see https://github.com/mkoeppe?tab=packages&q=recommended

Related issues and PRs:

Refinements:

--> Trac account requested

  • Run git log --first-parent on files with merge conflict
  • Provide it with GitHub credentials so it can push a branch $RELEASE_TAG+integration-$DATE, if it passes build + doctests + doc-pdf.
  • Perhaps git bisect --first-parent for passing; set first failing ticket back to needs_work
  • Another workflow for integration testing of all tickets in the current milestone with --patchbot-status=Spkg and --status=needs_review positive_review with the full portability CI.

Change History (87)

comment:1 Changed 4 months ago by mkoeppe

  • Cc vbraun added

Volker, would you be willing to share your existing scripts that probably already implement something like step 2 above? It does not have to be pretty.

comment:2 Changed 4 months ago by vbraun

Scripts are part of https://github.com/sagemath/git-trac-command:

$ git releasemgr merge-all -h
usage: git-releasemgr merge-all [-h] [--limit LIMIT]

optional arguments:
  -h, --help     show this help message and exit
  --limit LIMIT  Merge this many tickets

comment:3 Changed 4 months ago by mkoeppe

Thanks for the pointer

comment:4 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:5 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:6 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:7 Changed 4 months ago by mkoeppe

  • Dependencies set to #30933

comment:8 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:9 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:10 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:12 Changed 4 months ago by mkoeppe

  • Branch set to u/mkoeppe/gh_actions__create_daily_integration_branch

comment:13 Changed 4 months ago by git

  • Commit set to e803539cc5fa34f6ab26b3d922ba8acb4d8bef56

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e803539.github/workflows/integration.yml: New

comment:14 Changed 4 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Description modified (diff)

comment:15 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:17 Changed 4 months ago by mkoeppe

  • Dependencies changed from #30933 to #30933, #33233

comment:18 Changed 4 months ago by git

  • Commit changed from e803539cc5fa34f6ab26b3d922ba8acb4d8bef56 to c452317e872dfe88c407d88132b16b034cc2f398

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7db353e.github/workflows/integration.yml: New
6d30ef9src/sage/tests/cmdline.py: Add diagnostics to a test that often fails
73450a8src/bin/sage-runtests: New option --baseline-stats-path
3551e19src/sage/doctest/control.py: Load base line stats
09ce9e0src/sage/doctest/reporting.py: No error status for failures already seen in baseline
3eecc40Merge #33233
edecb85.github/workflows/integration.yml: Check out from current repo, use sagetrac-mirror remote for develop and tags
c452317.github/workflows/integration.yml: Check out from current repo, use sagetrac-mirror remote for develop and tags

comment:19 Changed 4 months ago by git

  • Commit changed from c452317e872dfe88c407d88132b16b034cc2f398 to 7513533e63ce7995d851d22284398ecdfebd1c8f

Branch pushed to git repo; I updated commit sha1. New commits:

1903e03.github/workflows/integration.yml: First build baseline, then test integration head
45af154.github/workflows/integration.yml: Make git available in the container
e4c3890.github/workflows/integration.yml: Add .git
7513533.github/workflows/integration.yml: Actually build baseline

comment:20 Changed 4 months ago by mkoeppe

  • Dependencies changed from #30933, #33233 to #30933, #33233, #31529

comment:21 Changed 4 months ago by git

  • Commit changed from 7513533e63ce7995d851d22284398ecdfebd1c8f to ec1f7c191ed2a2022cff8ada974a36e28a4502ed

Branch pushed to git repo; I updated commit sha1. New commits:

96771be.github/workflows/integration.yml: Run 'git reset --hard' in the container so that non-ADDed files are not shown as deleted
9ddbcd1.github/workflows/integration.yml: Do not fail when baseline ptest has failures
5d35b97WIP
9dceb34fix M4 bugs
65d67d1build/pkgs/texlive/distros: Add *.txt
b09d0c7Merge #31529
ec1f7c1.github/workflows/integration.yml: Install texlive

comment:22 Changed 4 months ago by git

  • Commit changed from ec1f7c191ed2a2022cff8ada974a36e28a4502ed to 51e47b3944983503078b82f05841fd96ec4993ae

Branch pushed to git repo; I updated commit sha1. New commits:

5733f4ebuild/pkgs/texlive/distros/debian.txt: Move texlive system packages here from build/pkgs/_recommended/distros/debian.txt
79852cesrc/doc/bootstrap: Add texlive to RECOMMENDED_SYSTEM_PACKAGES
51e47b3Merge #31529

comment:23 Changed 4 months ago by mkoeppe

  • Dependencies changed from #30933, #33233, #31529 to #30933, #33233, #31529, #30933

comment:24 Changed 4 months ago by git

  • Commit changed from 51e47b3944983503078b82f05841fd96ec4993ae to 1c4a7ea2a915a41ab716125b42a882a0f1beae98

Branch pushed to git repo; I updated commit sha1. New commits:

b1c705bbuild/pkgs/texlive/distros/debian.txt: Add more
a2a014cMerge #31529
21afe51.github/workflows/tox*.yml, tox.ini: Pass ddocker.pkg.github.com credentials via DOCKER_CONFIG_FILE
0f2a49aMerge #30933
87b93f7.github/workflows/integration.yml: Publish Docker images
d17b995.github/workflows/integration.yml: Put git-trac-command outside of Sage tree
f7c4f2b.github/workflows/integration.yml, tox.ini: Use new packages factor 'recommended'
bd0a101.github/workflows/integration.yml: Also push images for stages 'configured' and 'with-targets-optional'
f4fca0e.github/workflows/integration.yml: Fixup package name
1c4a7ea.github/workflows/integration.yml: Include action URL in Trac comments via git releasemgr --trac-context

comment:25 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:26 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:27 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:28 Changed 4 months ago by git

  • Commit changed from 1c4a7ea2a915a41ab716125b42a882a0f1beae98 to 744790913ba224c8f395716eb526ebc79a3ca376

Branch pushed to git repo; I updated commit sha1. New commits:

7e317b2build/pkgs/texlive/distros/debian.txt: Add texlive-fonts-recommended
37d569dbuild/pkgs/texlive/distros/debian.txt: Add texlive-lang-*
7447909Merge #31529

comment:29 Changed 4 months ago by mkoeppe

  • Dependencies changed from #30933, #33233, #31529, #30933 to #30933, #33233, #31529, #30933, #33243

comment:30 Changed 4 months ago by git

  • Commit changed from 744790913ba224c8f395716eb526ebc79a3ca376 to d2bda362f977f16353b24e5b8d944066f4744750

Branch pushed to git repo; I updated commit sha1. New commits:

22feeb5build/make/Makefile.in (PLUS): Make it work in GNU make 4.x
d2bda36Merge #33243

comment:31 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:32 Changed 4 months ago by git

  • Commit changed from d2bda362f977f16353b24e5b8d944066f4744750 to 3ae35d65caa0ab2b85f46b487e0ffe0bd4462aea

Branch pushed to git repo; I updated commit sha1. New commits:

3ae35d6.github/workflows/integration.yml: Create integration branch

comment:33 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:34 Changed 4 months ago by git

  • Commit changed from 3ae35d65caa0ab2b85f46b487e0ffe0bd4462aea to ca95c24b874181a17d030978c6ad828765f63101

Branch pushed to git repo; I updated commit sha1. New commits:

ca95c24.github/workflows/integration.yml: First run a job 'baseline_build', use cache

comment:35 Changed 4 months ago by mkoeppe

https://github.com/mkoeppe/sage/runs/4993397402?check_suite_focus=true

#5 importing cache manifest from docker.pkg.github.com/mkoeppe/sage/sage-docker-ubuntu-focal-recommended-with-targets
#5 sha256:c3f8749b6884f89e52d82a7beb83dd3938671c5e8c91d13592abcfa40146f0ad
#5 ERROR: unexpected status code [manifests latest]: 401 Unauthorized

This may be the same as https://github.com/docker/buildx/issues/780

comment:36 Changed 4 months ago by git

  • Commit changed from ca95c24b874181a17d030978c6ad828765f63101 to 26d9b8d301b53cd0c06cee0d07f5618b1c64cec1

Branch pushed to git repo; I updated commit sha1. New commits:

26d9b8d.github/workflows/integration.yml: Switch from docker.pkg.github.com to ghcr.io

comment:37 Changed 4 months ago by mkoeppe

  • Dependencies changed from #30933, #33233, #31529, #30933, #33243 to #30933, #33233, #31529, #30933, #33243, #33103

comment:38 Changed 4 months ago by git

  • Commit changed from 26d9b8d301b53cd0c06cee0d07f5618b1c64cec1 to 898acf5489deed82f1a819453448b051ba27f9fb

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

6285cd9.gitpod.Dockerfile: Save space by removing apt lists
7e74607.gitpod.Dockerfile: Remove built Sage documentation
f71c8b6docker/.gitpod.Dockerfile: Clean more
f953228Merge tag '9.5.rc0' into t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflow
2d0902asrc/doc/en/developer/workspace.rst: Fix markup
078f247.github/workflows/tox*.yml: Transform GITHUB_REF_NAME to a valid docker tag
b095b47Merge #33103
036df6dtox.ini: Use --cache-from if DOCKER_PUSH_REPOSITORY is given
de2a455.github/workflows/integration.yml: Pass BUILDKIT_INLINE_CACHE=1 as a build-arg
898acf5.github/workflows/integration.yml: Reactivate full baseline build

comment:39 Changed 4 months ago by mkoeppe

  • Cc dimpase added
  • Description modified (diff)

comment:40 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:41 Changed 4 months ago by git

  • Commit changed from 898acf5489deed82f1a819453448b051ba27f9fb to a545c4e822be54d2508db34b73f2b257fa64d619

Branch pushed to git repo; I updated commit sha1. New commits:

a545c4e.github/workflows/integration.yml: Fixup

comment:42 Changed 4 months ago by git

  • Commit changed from a545c4e822be54d2508db34b73f2b257fa64d619 to 627c65c142ac64fc0bee9261059e2171edbeb2ee

Branch pushed to git repo; I updated commit sha1. New commits:

521b12f.github/workflows/integration.yml: Actually run ptest
627c65c.github/workflows/integration.yml: Use ptestlong

comment:43 Changed 4 months ago by git

  • Commit changed from 627c65c142ac64fc0bee9261059e2171edbeb2ee to 72eff23db5c24a09e2aac4496e8cdfa6e6382a0e

Branch pushed to git repo; I updated commit sha1. New commits:

72eff23Merge tag '9.5' into t/33222/gh_actions__create_daily_integration_branch

comment:44 Changed 4 months ago by mkoeppe

  • Dependencies changed from #30933, #33233, #31529, #30933, #33243, #33103 to #33233, #31529, #33103

comment:45 Changed 4 months ago by git

  • Commit changed from 72eff23db5c24a09e2aac4496e8cdfa6e6382a0e to b143b95b8341bf6c1fd70f4efe712784391cbc6f

Branch pushed to git repo; I updated commit sha1. New commits:

10259f2.github/workflows, src/doc/en/developer/portability_testing.rst: Switch from docker.pkg.github.com to ghcr.io
47c4723.github/workflows/integration.yml: Set EXTRA_DOCKER_TAGS
b143b95tox.ini: Also --cache-from ...:dev

comment:46 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:47 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:48 Changed 4 months ago by mkoeppe

  • Description modified (diff)
  • Status changed from new to needs_review

comment:49 Changed 4 months ago by git

  • Commit changed from b143b95b8341bf6c1fd70f4efe712784391cbc6f to 6af8abb9ac0aab7b7cef9963e3a81ea915ad0a28

Branch pushed to git repo; I updated commit sha1. New commits:

1dc511cAdd Arch package info
a505487build/pkgs/texlive/distros/fedora.txt: Add texlive-collection-latexextra
7daedd9build/pkgs/texlive/distros/fedora.txt: Add texlive-collection-lang*
6af8abbMerge #31529

comment:50 Changed 4 months ago by git

  • Commit changed from 6af8abb9ac0aab7b7cef9963e3a81ea915ad0a28 to ff8a8d6092c010e9177765b631e7b3d13f607bee

Branch pushed to git repo; I updated commit sha1. New commits:

ff8a8d6build/pkgs/ffmpeg/distros/fedora.txt: Remove ffmpeg - it is not in the standard repo

comment:51 Changed 4 months ago by mkoeppe

  • Status changed from needs_review to needs_work

With ptestlong it exceeds the 6 hours, so I'll have to split out another job

comment:52 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:53 Changed 4 months ago by git

  • Commit changed from ff8a8d6092c010e9177765b631e7b3d13f607bee to 5117e9221ebb81011766ebf03bf26dacb0c06d2d

Branch pushed to git repo; I updated commit sha1. New commits:

5117e92.github/workflows/integration.yml: Separate job baseline_test

comment:54 Changed 4 months ago by git

  • Commit changed from 5117e9221ebb81011766ebf03bf26dacb0c06d2d to 921ea0f4823c25983e67fd9a088161c1b3a81eed

Branch pushed to git repo; I updated commit sha1. New commits:

9647521.github/workflows/integration.yml: Fixup
921ea0f.github/workflows/integration.yml: Fixup

comment:55 Changed 4 months ago by mkoeppe

  • Dependencies changed from #33233, #31529, #33103 to #33233, #31529, #33103, #33277

comment:56 Changed 4 months ago by git

  • Commit changed from 921ea0f4823c25983e67fd9a088161c1b3a81eed to 5de17242ccf46985c56ad7da50c1a2c9b4800aa9

Branch pushed to git repo; I updated commit sha1. New commits:

6952de4build/pkgs/texlive/spkg-configure.m4: Check for pdflatex, latexmk, dvipng
b85356fbuild/pkgs/texlive/spkg-configure.m4: Check for tgtermes.sty
62fbc45build/pkgs/sagemath_doc_pdf/dependencies: Add texlive
e63b9f2Merge tag '9.5' into t/31529/packages/texlive/add_spkg_configure_m4_and_system_package_information__remove_install_script
8b2490ebuild/pkgs/texlive/spkg-configure.m4: ALso check for fncychap.sty
7b97dd4update list of gentoo packages
016ff89Merge #31529
855314dsrc/doc/bootstrap: Get recommended spkg from build/pkgs/_recommended/dependencies
0097d0eMerge #33277
5de1724tox.ini (recommended): Use build/pkgs/_recommended/dependencies

comment:57 Changed 4 months ago by git

  • Commit changed from 5de17242ccf46985c56ad7da50c1a2c9b4800aa9 to 37c76a6bacdfd1c43e5bc8541e1af0c2055bb5a0

Branch pushed to git repo; I updated commit sha1. New commits:

79a3905.github/workflows/integration.yml (baseline_test): Also fetch tags so that we refer to the right image tag
37c76a6tox.ini (docker): --cache-from all docker_targets, EXTRA_CACHE_FROM_REPOSITORIES

comment:58 Changed 4 months ago by mkoeppe

  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:59 follow-ups: Changed 4 months ago by gh-tobiasdiez

Uff that's quite complicated. Why do you first need to create a new baseline image for the integration tests and not reuse the docker image build during the latest push to the develop branch?

Instead of manually handling the docker pull (using cli commands) I would propose to use the container tag for the workflow (or job). Makes things easier and I assume github also takes care of the permission handling.

Also the integration job is currently timing out, and if I read the logs correctly then it is actually not merging the tickets for the new milestone (but tries to merge tickets for the already finished milestone 9.5): No tickets for milestone sage-9.5 are ready to be merged.

Finally a general remark: please use the new reusable actions instead of copy&pasting stuff around, like the "free space" or "login to ghcr" parts (which shouldn't be necessary in the final integration job because that's only reading, right?).

comment:60 in reply to: ↑ 59 ; follow-up: Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

Finally a general remark: please use the new reusable actions instead of copy&pasting stuff around

Refactoring will not happen in this ticket

comment:61 in reply to: ↑ 59 ; follow-ups: Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

Why do you first need to create a new baseline image for the integration tests and not reuse the docker image build during the latest push to the develop branch?

I need an image with more packages, which is not built by the CI. This is to run make doc-pdf, an essential step that currently no CI is testing.

Instead of manually handling the docker pull (using cli commands) I would propose to use the container tag for the workflow (or job).

I am deliberately setting this up as an image build, not just a container run, because I want to create a Docker image that can be pulled for local inspection.

comment:62 in reply to: ↑ 59 Changed 4 months ago by mkoeppe

  • Status changed from needs_review to needs_work

Replying to gh-tobiasdiez:

Also the integration job is currently timing out

Yes, there's still something going wrong with the caching, so it's rebuilding something that it should have taken from the previous job

comment:63 in reply to: ↑ 59 Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

"login to ghcr" parts [...] shouldn't be necessary in the final integration job because that's only reading, right?

Actually one does need to login even just for reading packages.

comment:64 Changed 4 months ago by mkoeppe

(And the last step does push something - the image with the included repository.)

comment:65 in reply to: ↑ 60 Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Finally a general remark: please use the new reusable actions instead of copy&pasting stuff around

Refactoring will not happen in this ticket

I didn't meant you should refactor the existing workflows. But if you want to reuse some steps in the newly created workflow than they should be extracted to a new action (instead of copy&paste them). Refactoring the other workflows to use this new action can be done later. (You wouldn't also let python code pass that copy&pastes some methods around, right?)

comment:66 in reply to: ↑ 61 ; follow-up: Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Why do you first need to create a new baseline image for the integration tests and not reuse the docker image build during the latest push to the develop branch?

I need an image with more packages, which is not built by the CI. This is to run make doc-pdf, an essential step that currently no CI is testing.

If it's so important, why not add it as a new target to the existing CI?

comment:67 in reply to: ↑ 61 ; follow-ups: Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

I am deliberately setting this up as an image build, not just a container run, because I want to create a Docker image that can be pulled for local inspection.

Can you expand on why this should be helpful, i.e. where are the advantages over just checking out the branch in your local git (and maybe running it in the baseline-integration docker image if you like). Also you can just execute the github workflow locally using act if you like.

Also, are you aware that there are special actions for the login and publish, which make the whole process way easier (and we don't need to maintain them)? https://docs.github.com/en/actions/publishing-packages/publishing-docker-images#publishing-images-to-github-packages

comment:68 in reply to: ↑ 66 ; follow-ups: Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

This is to run make doc-pdf, an essential step that currently no CI is testing.

If it's so important, why not add it as a new target to the existing CI?

Yes, changing the existing CI to build the -recommended instead of -standard workflow may be something that we want to do at some point in the future. But this will need #33062 (split the workflow, with caching of Docker layers). That's exactly what the changes to tox.ini here on the ticket are implementing!

Can't change everything at the same time.

comment:69 in reply to: ↑ 67 ; follow-ups: Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

I am deliberately setting this up as an image build, not just a container run, because I want to create a Docker image that can be pulled for local inspection.

Can you expand on why this should be helpful, i.e. where are the advantages over just checking out the branch in your local git (and maybe running it in the baseline-integration docker image if you like).

The build & test can take long. If it is prebuilt on GH Actions and I can download it for inspection, that's a benefit.

Also you can just execute the github workflow locally using act if you like.

I don't like

comment:70 in reply to: ↑ 67 ; follow-up: Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

Also, are you aware that there are special actions for the login and publish

Yes, but they don't help much. The login action could replace these 5 lines:

            TOKEN="${{ secrets.DOCKER_PKG_GITHUB_TOKEN }}"
            if [ -z "$TOKEN" ]; then
              TOKEN="${{ secrets.GITHUB_TOKEN }}"
            fi
            if echo "$TOKEN" | docker login ghcr.io -u ${{ github.actor }} --password-stdin; then

And the other one is not useful

comment:71 in reply to: ↑ 68 Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Replying to mkoeppe:

This is to run make doc-pdf, an essential step that currently no CI is testing.

If it's so important, why not add it as a new target to the existing CI?

Yes, changing the existing CI to build the -recommended instead of -standard workflow may be something that we want to do at some point in the future. But this will need #33062 (split the workflow, with caching of Docker layers). That's exactly what the changes to tox.ini here on the ticket are implementing!

Can't change everything at the same time.

Okay, makes sense. Thanks for the explanation.

comment:72 in reply to: ↑ 69 Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Replying to mkoeppe:

I am deliberately setting this up as an image build, not just a container run, because I want to create a Docker image that can be pulled for local inspection.

Can you expand on why this should be helpful, i.e. where are the advantages over just checking out the branch in your local git (and maybe running it in the baseline-integration docker image if you like).

The build & test can take long. If it is prebuilt on GH Actions and I can download it for inspection, that's a benefit.

But the push happens before the final "Build & test integration head", so it neither contains the test results nor the build, or I'm missing something?

comment:73 Changed 4 months ago by mkoeppe

Yes, that's true as it is in the current branch - but that final step can be extended to also build; just not done yet

comment:74 in reply to: ↑ 70 Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Also, are you aware that there are special actions for the login and publish

Yes, but they don't help much. The login action could replace these 5 lines:

            TOKEN="${{ secrets.DOCKER_PKG_GITHUB_TOKEN }}"
            if [ -z "$TOKEN" ]; then
              TOKEN="${{ secrets.GITHUB_TOKEN }}"
            fi
            if echo "$TOKEN" | docker login ghcr.io -u ${{ github.actor }} --password-stdin; then

Yes, the docker/login-action is not doing much more: https://github.com/docker/login-action/blob/17f28ab24d0d2832d5ff23a1409bbfc373ebcb96/src/docker.ts#L30-L50 Except a bit of proper error handling and logging you out again. Anyway, why not reuse something official so that you don't have to worry about it?

comment:75 Changed 4 months ago by mkoeppe

Because of the lines that follow if.

comment:76 Changed 4 months ago by gh-tobiasdiez

I guess it's getting late here for me, but your messages get more and more cryptic. There are a lot of ifs here...

comment:77 follow-up: Changed 4 months ago by mkoeppe

Sorry, I meant the last if but it also applies to the first if ;)

comment:78 in reply to: ↑ 69 ; follow-up: Changed 4 months ago by dimpase

Replying to mkoeppe:

Also you can just execute the github workflow locally using act if you like.

wow, I never knew about act (guess it's https://github.com/nektos/act)

It should be in our developer manual.

comment:79 in reply to: ↑ 78 ; follow-up: Changed 4 months ago by gh-tobiasdiez

Replying to dimpase:

Replying to mkoeppe:

Also you can just execute the github workflow locally using act if you like.

wow, I never knew about act (guess it's https://github.com/nektos/act)

Yes, that's the one (should have directly provided a link, sry). It's quite powerful and usually works really well. Don't know how Matthias can work on the github workflows without quickly checking locally if everything goes well.

comment:80 in reply to: ↑ 77 Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Sorry, I meant the last if but it also applies to the first if ;)

Even after sleep I don't understand your comment. The second if doesn't do anything in my opinion since the Github token is always there and we need to login to ghcr as you said earlier. For the first if, this should be password = ${{ secrets.DOCKER_PKG_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} with the docker/login-action

comment:81 in reply to: ↑ 68 ; follow-up: Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Replying to mkoeppe:

This is to run make doc-pdf, an essential step that currently no CI is testing.

If it's so important, why not add it as a new target to the existing CI?

Yes, changing the existing CI to build the -recommended instead of -standard workflow may be something that we want to do at some point in the future.

Thinking about this again: would it be an idea to run the integration workflow as follows:

  • use the docker image build during the ci for the develop branch
  • merge all new tickets
  • re-build sage, including the packages that are needed for doc-pdf
  • run tests (using the ci artifact as baseline)
  • build docs

Not sure how long the last step takes, but step 4 should finish in 30-60 mins (depending on how much packages are touched in the to-be-merged tickets) and step 5 about 1h. So this should all fit very nicely in one workflow. So the main advantage would be that you don't have to rebuild all packages from scratch during the integration workflow.

This is under the assumption that the packages needed for the pdf build of the documentation don't influence the tests, because otherwise the test-baseline could be wrong (but is probably still good enough).

comment:82 Changed 4 months ago by vbraun

Just FYI, I'm planning on adding a CI step that runs for each ticket at #33294. Sufficiently different from this ticket that both have merits, just a heads up.

Last edited 4 months ago by vbraun (previous) (diff)

comment:83 in reply to: ↑ 79 Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

Don't know how Matthias can work on the github workflows without quickly checking locally if everything goes well.

With a lot of patience!

comment:84 in reply to: ↑ 81 Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

  • re-build sage, including the packages that are needed for doc-pdf

these are system packages, not sage packages

comment:85 in reply to: ↑ 68 Changed 4 months ago by mkoeppe

Replying to mkoeppe:

changing the existing CI to build the -recommended instead of -standard workflow may be something that we want to do at some point in the future.

Preparation for that is now in #33296

comment:86 Changed 3 months ago by gh-tobiasdiez

Mainly to better understand the process behind the scenes, why is an additional "integration" branch necessary? Can one not simply push to "develop" (in case there are no errors)? I guess this comes down to the question "what does the release manager has to do in the background that demands a manual merge into the develop branch".

comment:87 Changed 8 weeks ago by mkoeppe

  • Milestone changed from sage-9.6 to sage-9.7
Note: See TracTickets for help on using tickets.