Opened 6 months ago

Closed 5 months ago

#33103 closed enhancement (fixed)

gitpod integration using Docker images from portability testing workflow

Reported by: mkoeppe Owned by:
Priority: critical Milestone: sage-9.6
Component: docker Keywords:
Cc: gh-tobiasdiez, dimpase, saraedum Merged in:
Authors: Tobias Diez, Matthias Koeppe Reviewers: Matthias Koeppe, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 078f247 (Commits, GitHub, GitLab) Commit: 078f247c9a5ea383178499e9b0e990777f497a6e
Dependencies: #30933, #29285 Stopgaps:

Status badges

Description (last modified by gh-tobiasdiez)

Gitpod allows one to setup a complete dev environment in the cloud. It is free to use for up to 50 hours per month. In this ticket the config necessary for making this work with sagemath is added.

You can try this out by going to https://gitpod.io/#https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates.

It works by building docker images on github, push them to GH packages (https://github.com/mkoeppe?tab=packages&q=gitpod) and then use these ones as the base for the gitpod workspace.

After this ticket is merged, we should activate the automatic prebuild by installing the github app (https://www.gitpod.io/docs/prebuilds#on-github). With this enabled, every push would trigger a prebuild of the complete environment (including the build of all dependencies and cythonizion) so that one has a up-to-date code env very quickly without the need to build anything.

Follow-up: #33113 gitpod usability improvements

Alternative: #32749, which only builds the docker image on gitpod (which is currently not really feasible as they have a time limit of 1h)

Change History (157)

comment:1 Changed 6 months ago by mkoeppe

  • Branch set to u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow

comment:2 Changed 6 months ago by mkoeppe

  • Authors set to Tobias Diez, Matthias Koeppe
  • Commit set to bde9f74bc7be3ca657f3d6ca8ab420709abfbeda

Last 10 new commits:

e99a09esed -i.bak 's/ubunty/ubuntu/g' .github/workflows/*.yml
edb4364.github/workflows/tox.yml: Replace homebrew-macos-python3_xcode-standard by homebrew-macos-usrlocal-python3_xcode-standard
e27e161Merge #32703
96a45d1tox.ini, .github/workflows: Remove ubuntu-groovy
59ec3d0Merge #33068
cbf03d1.github/workflows/tox.yml: Add docker-gitpod
4a9edfcbuild/bin/write-dockerfile.sh: Handle __SUDO
d697d94build/bin/write-dockerfile.sh: Use relative workdir 'sage'
c7cc5cd.gitpod.yml, docker/.gitpod.Dockerfile: Reverse SAGE_LOCAL symlink
bde9f74build/bin/write-dockerfile.sh: Handle $__CHOWN

comment:3 Changed 6 months ago by git

  • Commit changed from bde9f74bc7be3ca657f3d6ca8ab420709abfbeda to 76b5e76ab2c28a3ee4a5c1571737bc2f8571f5ff

Branch pushed to git repo; I updated commit sha1. New commits:

21afe51.github/workflows/tox*.yml, tox.ini: Pass ddocker.pkg.github.com credentials via DOCKER_CONFIG_FILE
76b5e76Merge #30933

comment:4 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:5 Changed 6 months ago by git

  • Commit changed from 76b5e76ab2c28a3ee4a5c1571737bc2f8571f5ff to 7ef6a6e3333a43153b92e54dceb3b06ab0713387

Branch pushed to git repo; I updated commit sha1. New commits:

7ef6a6edocker/.gitpod.Dockerfile: Remove, use custom image from GH Packages instead

comment:6 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:7 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:8 Changed 6 months ago by git

  • Commit changed from 7ef6a6e3333a43153b92e54dceb3b06ab0713387 to 6b41c4af670b9e8a653e8c2e4049848e4e87d8a1

Branch pushed to git repo; I updated commit sha1. New commits:

6b41c4a.github/workflows/tox.yml: Also push ...-with-targets-pre

comment:9 Changed 6 months ago by mkoeppe

Running into pyenv issues (#29285) here too:

[pytest] installing. Log file: /home/gitpod/sage/logs/pkgs/pytest.log
  [pytest] error installing, exit status 1. End of log file:
  [pytest]   /home/gitpod/sage-local/var/lib/sage/venv-python3.8/bin/python3: No module named pip
  [pytest] Full log file: /home/gitpod/sage/logs/pkgs/pytest.log
make[4]: *** [Makefile:2756: pytest-no-deps] Error 1

comment:10 Changed 6 months ago by mkoeppe

  • Dependencies changed from #33068, #30933 to #33068, #30933, #29285

comment:11 Changed 6 months ago by git

  • Commit changed from 6b41c4af670b9e8a653e8c2e4049848e4e87d8a1 to 5f0c36cc73b59e64eba006ed5a5dfb51efd920bf

Branch pushed to git repo; I updated commit sha1. New commits:

376932fbuild/bin/sage-bootstrap-python: Reject pyenv shims
5f0c36cMerge #29285

comment:12 Changed 6 months ago by mkoeppe

  • Dependencies changed from #33068, #30933, #29285 to #33068, #30933, #29285, #32868

comment:13 Changed 6 months ago by mkoeppe

  • Dependencies changed from #33068, #30933, #29285, #32868 to #33068, #30933, #29285, #32868, #33067

comment:14 Changed 6 months ago by git

  • Commit changed from 5f0c36cc73b59e64eba006ed5a5dfb51efd920bf to af7c8d3c2237980a12b89ffad4c0cc0531662ad3

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

aeca1a0Makefile (pypi-sdists): Add sage_setup
cd3dce9pkgs/sage-conf_pypi/setup.py: Update directory of configured sage_conf.py
c695921pkgs/sage-conf/sage_conf.py.in (SAGE_SPKG_WHEELS): Update so it works if SAGE_VENV != SAGE_LOCAL
cebcda3Merge tag '9.5.beta4' into t/29039/pip_installable_package_sage_bootstrap
0026892Merge tag '9.5.beta5' into t/29039/pip_installable_package_sage_bootstrap
2a4a323Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap
02234ceMerge #29039
b1413d4build/bin/write-dockerfile.sh: Move 'ADD pkgs' earlier
859be3ebuild/bin/write-dockerfile.sh: ADD src/*.m4 files for bootstrapping
af7c8d3Merge #33067

comment:15 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 6 months ago by git

  • Commit changed from af7c8d3c2237980a12b89ffad4c0cc0531662ad3 to fad8942c93702e6c098fe6b9cd2d68704b256334

Branch pushed to git repo; I updated commit sha1. New commits:

fad8942build/bin/write-dockerfile.sh: Use relative workdir 'sage' (again)

comment:17 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:18 Changed 6 months ago by git

  • Commit changed from fad8942c93702e6c098fe6b9cd2d68704b256334 to 1258c702dac22a428cf0d1ede65ccea35cf50d0f

Branch pushed to git repo; I updated commit sha1. New commits:

601a3b5.gitpod.yml: Remove pyenv workarounds
1258c70.gitpod.yml: configure --enable-download-from-upstream-url

comment:19 Changed 6 months ago by mkoeppe

This works. To be done:

Last edited 6 months ago by mkoeppe (previous) (diff)

comment:20 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:21 Changed 6 months ago by git

  • Commit changed from 1258c702dac22a428cf0d1ede65ccea35cf50d0f to fdb49a1d7865f8105c50470715368af9a8ede02d

Branch pushed to git repo; I updated commit sha1. New commits:

fdb49a1tox.ini, .github/workflows/tox*.yml: Also tag the Docker images with GITHUB_REF_NAME

comment:22 Changed 6 months ago by mkoeppe

  • Reviewers set to https://github.com/mkoeppe/sage/runs/4686167146

comment:23 Changed 6 months ago by git

  • Commit changed from fdb49a1d7865f8105c50470715368af9a8ede02d to ab757ba44a661a0c358764d3aa40c49709f4c6cd

Branch pushed to git repo; I updated commit sha1. New commits:

ab757ba.github/workflows/tox.yml: Fix passing of EXTRA_DOCKER_TAGS

comment:24 Changed 6 months ago by git

  • Commit changed from ab757ba44a661a0c358764d3aa40c49709f4c6cd to 96528ebee43305123a813b340ad3e027ba2f8712

Branch pushed to git repo; I updated commit sha1. New commits:

96528eb.github/workflows/tox*.yml: Tag docker images 'latest' or 'dev' when building a release tag

comment:25 Changed 6 months ago by mkoeppe

  • Reviewers changed from https://github.com/mkoeppe/sage/runs/4686167146 to https://github.com/mkoeppe/sage/runs/4686610379

comment:26 Changed 6 months ago by mkoeppe

  • Reviewers changed from https://github.com/mkoeppe/sage/runs/4686610379 to https://github.com/mkoeppe/sage/runs/4687073955

comment:27 Changed 6 months ago by mkoeppe

  • Reviewers changed from https://github.com/mkoeppe/sage/runs/4687073955 to https://github.com/mkoeppe/sage/runs/4687310789

comment:28 Changed 6 months ago by mkoeppe

  • Reviewers changed from https://github.com/mkoeppe/sage/runs/4687310789 to https://github.com/mkoeppe/sage/runs/4687310728

comment:29 Changed 6 months ago by git

  • Commit changed from 96528ebee43305123a813b340ad3e027ba2f8712 to 5d1dc0fa92a7192bee0bb56e9ea13944da97007f

Branch pushed to git repo; I updated commit sha1. New commits:

5d1dc0f.gitpod.yml: Use tag :dev

comment:30 Changed 6 months ago by mkoeppe

  • Status changed from new to needs_review
  • Work issues set to Change .gitpod.yml to use sagemath instead of mkoeppe

comment:31 Changed 6 months ago by mkoeppe

Also the configuration of VS Code should be improved. It seems to start up using the wrong interpreter (not venv/bin/python3), and also one needs to manually select the right Jupyter kernel.

The scipy folks have some good documentation and configuration (https://scipy.github.io/devdocs/dev/contributor/quickstart_gitpod.html), perhaps we can lift something from there.

comment:32 follow-ups: Changed 6 months ago by gh-tobiasdiez

Is the prebuild working for this branch?

How can one work with branches that update sage packages? With #32749 one could trigger a rebuild of the docker image (and thus of the system packages) using the #imagebuild context. Can something like this also be implemented here? For example, take the github action build docker as base, and then re-install system or python packages on top of this if they have newer versions?

For the VS integration, #30677 is closed but not yet merged into develop.

comment:33 in reply to: ↑ 32 ; follow-up: Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

Is the prebuild working for this branch?

I've just merged #33071 into the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates to try this out.

How can one work with branches that update sage packages?

Nothing special, make just rebuilds what needs rebuilding.

comment:34 in reply to: ↑ 32 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

For the VS integration, #30677 is closed but not yet merged into develop.

I forgot to say that the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates already contains #30677 and other updates

comment:35 in reply to: ↑ 33 Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Is the prebuild working for this branch?

I've just merged #33071 into the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates to try this out.

How can one work with branches that update sage packages?

Nothing special, make just rebuilds what needs rebuilding.

So this would then be done as part of the prebuild?

comment:36 Changed 6 months ago by mkoeppe

Yes, I think so

comment:37 Changed 6 months ago by mkoeppe

Looks like I'll have to add some configuration so that it does prebuilds on all branches

comment:38 Changed 6 months ago by mkoeppe

I created a "project" and now it looks like the prebuild triggered by a push is running - https://gitpod.io/projects/sage

comment:39 Changed 6 months ago by git

  • Commit changed from 5d1dc0fa92a7192bee0bb56e9ea13944da97007f to 3e28e13afa9d3a61aceb4d6b3c3d66451f47133f

Branch pushed to git repo; I updated commit sha1. New commits:

3e28e13.gitpod.yml: Set github prebuild options to true

comment:40 follow-up: Changed 6 months ago by mkoeppe

Before the push I had merged #32968 with the Sphinx update; and the prebuild has just installed the upgraded package

comment:41 in reply to: ↑ 40 Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

Before the push I had merged #32968 with the Sphinx update; and the prebuild has just installed the upgraded package

That's very nice!

comment:42 Changed 6 months ago by git

  • Commit changed from 3e28e13afa9d3a61aceb4d6b3c3d66451f47133f to 49575f56f74343fb81b401a6077a3f05bb037c61

Branch pushed to git repo; I updated commit sha1. New commits:

49575f5.gitpod.yml: Switch to image ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev

comment:43 Changed 6 months ago by mkoeppe

  • Description modified (diff)
  • Work issues Change .gitpod.yml to use sagemath instead of mkoeppe deleted

comment:44 Changed 6 months ago by mkoeppe

The prebuild was just killed because of the 35m timeout. What's the best choice of timeout according to your experiments?

comment:45 Changed 6 months ago by gh-tobiasdiez

If its really the timeout that killed the prebuild (in which the prebuild would be green and be used for the next workspace, but maybe doesn't contain all packages) then you can go to 50min. Maybe a bit less since the docker image is quite big so this part may take some time. But essentially you have to calculate 60min - bootstrap and config (2-3min) - docker init and copying (?? min).

However, for me the prebuild often timed out after 1h (then its shown red in the project view). The only solution I've found for this is not using symlinks into /home/.

comment:46 Changed 6 months ago by mkoeppe

https://gitpod.io/projects/sage/e8780c03-b8b8-4810-a772-d4cc2994b6bf shows

[sagelib-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sagelib-9.5.beta9.log
  [pytest] successfully installed.
  [sphinx-4.3.1] successfully installed.
make[1]: *** [Makefile:39: all-build] Terminated
make: *** [Makefile:16: build] Terminated
exit

but it is still claims "RUNNING: Prebuild in progress"

comment:47 Changed 6 months ago by mkoeppe

comment:48 follow-up: Changed 6 months ago by mkoeppe

🤙 This task ran as a workspace prebuild
🎉 Well done on saving 37 minutes

comment:49 follow-up: Changed 6 months ago by gh-tobiasdiez

A few more points:

comment:50 in reply to: ↑ 48 Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

🤙 This task ran as a workspace prebuild
🎉 Well done on saving 37 minutes

Nice, then you can go to 50min indeed.

comment:51 Changed 6 months ago by git

  • Commit changed from 49575f56f74343fb81b401a6077a3f05bb037c61 to e7d4e966d8d25281befe14c5b66bae991c4736e3

Branch pushed to git repo; I updated commit sha1. New commits:

e7d4e96.gitpod.yml: Increase timeout for make in prebuild, increase parallelism

comment:52 Changed 6 months ago by mkoeppe

Presumably the before script should try to resume the build in case the prebuild timed out.

comment:53 Changed 6 months ago by gh-tobiasdiez

I agree. But init is called after before during the prebuild, and I couldn't find a way to detect if one is in a prebuild or not...

comment:54 follow-up: Changed 6 months ago by git

  • Commit changed from e7d4e966d8d25281befe14c5b66bae991c4736e3 to fdb332d2a52db9f63fc54960d029a5ad9c529a61

Branch pushed to git repo; I updated commit sha1. New commits:

fdb332d.gitpod.yml: Run 'make' again in 'before' task if prebuild 'make' timed out

comment:55 Changed 6 months ago by git

  • Commit changed from fdb332d2a52db9f63fc54960d029a5ad9c529a61 to aaa97778417b2142d9f3cf83c29dce9d38d1853e

Branch pushed to git repo; I updated commit sha1. New commits:

aaa9777.gitpod.yml: Use SAGE_NUM_THREADS=8

comment:56 in reply to: ↑ 49 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

I've updated it - the ssh stuff was already same

comment:57 Changed 6 months ago by git

  • Commit changed from aaa97778417b2142d9f3cf83c29dce9d38d1853e to 87900a7d0188247be6d0c6db7b87349759621ca3

Branch pushed to git repo; I updated commit sha1. New commits:

87900a7.gitpod.yml: Refer to /home/gitpod as /Users/mkoeppe

comment:58 Changed 6 months ago by git

  • Commit changed from 87900a7d0188247be6d0c6db7b87349759621ca3 to 0864c6ec3e00d24566b586472ffb88e1471a2e30

Branch pushed to git repo; I updated commit sha1. New commits:

0864c6e.github/workflows/tox*.yml: Push stable releases also to Docker tag 'dev'

comment:59 Changed 6 months ago by mkoeppe

https://gitpod.io/projects/sage/134676a9-2a0e-429a-816d-68ad30441f58

[sphinx-4.3.1] installing. Log file: /workspace/sage/logs/pkgs/sphinx-4.3.1.log
[sagelib-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sagelib-9.5.beta9.log
  [pytest] successfully installed.
  [sphinx-4.3.1] successfully installed.
  [sagelib-9.5.beta9] successfully installed.
make --no-print-directory sage_docbuild-SAGE_VENV-no-deps
[sage_docbuild-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sage_docbuild-9.5.beta9.log
  [sage_docbuild-9.5.beta9] successfully installed.
make[2]: Leaving directory '/workspace/sage/build/make'

real    31m8.684s
user    101m44.392s
sys     3m47.093s
SAGE_CHECK=warn, so scanning the log files. This may take a few seconds.
Sage build/upgrade complete!
make[1]: Leaving directory '/workspace/sage'
exit

🤙 This task ran as a workspace prebuild
🎉 Well done on saving 34 minutes

comment:60 in reply to: ↑ 54 Changed 6 months ago by mkoeppe

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

fdb332d.gitpod.yml: Run 'make' again in 'before' task if prebuild 'make' timed out

This worked but for some reason the Sphinx package installation was repeated in this step. It's fast but I'll try if refreshing the timestamps of the install records will avoid it.

comment:61 Changed 6 months ago by git

  • Commit changed from 0864c6ec3e00d24566b586472ffb88e1471a2e30 to c9fa0152e86c3631ae69cb6d7ad75ed133e93bd6

Branch pushed to git repo; I updated commit sha1. New commits:

36e0c54.gitpod.yml (before): Set timestamps of installation records to same mtime
c9fa015.gitpod.yml: Use /workspace/sage/local as physical location of SAGE_LOCAL

comment:62 Changed 6 months ago by mkoeppe

There are some weird synchronization issues on the server side - it seems that if one creates a workspace right after a prebuild has completed, it can happen that it does not use the prebuild.

comment:63 Changed 6 months ago by git

  • Commit changed from c9fa0152e86c3631ae69cb6d7ad75ed133e93bd6 to 38dafc60837e3de5455e24e6913150791b05e509

Branch pushed to git repo; I updated commit sha1. New commits:

38dafc6.gitpod.yml (before): Refer to /workspace/sage/local by full path

comment:64 Changed 6 months ago by git

  • Commit changed from 38dafc60837e3de5455e24e6913150791b05e509 to e6be067e2c7210f5f04e0903a5e45a101461da63

Branch pushed to git repo; I updated commit sha1. New commits:

e6be067.gitpod.yml (init): Make sure to remove the directory /home/gitpod/sage-local

comment:65 Changed 6 months ago by git

  • Commit changed from e6be067e2c7210f5f04e0903a5e45a101461da63 to cdcf2acb4e53644963484644e2c0e244f28d36b1

Branch pushed to git repo; I updated commit sha1. New commits:

cdcf2ac.gitpod.yml (before): establish SAGE_LOCAL here, not in 'init'

comment:66 Changed 6 months ago by git

  • Commit changed from cdcf2acb4e53644963484644e2c0e244f28d36b1 to f58453d1aeeb84d5508498d6178751405798657a

Branch pushed to git repo; I updated commit sha1. New commits:

5b44b6b.gitpod.yml: Remove install-record timestamping
f58453d.gitpod.yml: Also remove the source tree used by the Docker build

comment:67 Changed 6 months ago by git

  • Commit changed from f58453d1aeeb84d5508498d6178751405798657a to aef75ebe0986f6c499585fcdad36231808e5d21b

Branch pushed to git repo; I updated commit sha1. New commits:

aef75eb.gitpod.yml (before): Always remove the source tree used by the Docker build

comment:68 follow-ups: Changed 6 months ago by mkoeppe

OK, now it's in good shape.

Could you check that the SSH stuff works as intended and matches the added documentation? I see

$ ls -l ~/.ssh
total 24
-rw-r--r-- 1 gitpod gitpod   381 Jan  4 08:53 authorized_keys
-rw------- 1 gitpod gitpod     0 Jan  4 08:53 id_rsa
-rw-r--r-- 1 gitpod gitpod   664 Jan  4 08:53 known_hosts
-rw-r--r-- 1 gitpod gitpod 15544 Jan  4 08:53 supervisor_env

and am not sure what the empty id_rsa is intended for

comment:70 follow-up: Changed 6 months ago by gh-tobiasdiez

https://github.com/sagemath/sagetrac-mirror/compare/develop...u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow#diff-370a022e48cb18faf98122794ffc5ce775b2606b09a9d1f80b71333425ec078eR17

I'm not sure if I understand the logic behind this and related commands (maybe add a bit more documentation of what's going on?). But please be aware that the code may be in a different location such as /workspace/sagetrac-mirror if someone starts gitpod from https://github.com/sagemath/sagetrac-mirror (which is probably what's going to happen if you want to check someone elses branch)

comment:71 in reply to: ↑ 68 ; follow-up: Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

Could you check that the SSH stuff works as intended and matches the added documentation? and am not sure what the empty id_rsa is intended for

Did you added a PRIVATE_SSH_KEY env variable in gitpod with your trac ssh as described in the docs? It's working for me.

Last edited 6 months ago by gh-tobiasdiez (previous) (diff)

comment:72 in reply to: ↑ 68 ; follow-up: Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

OK, now it's in good shape.

For me it takes over 5min to start a workspace due to the large workspace image (it displays "fetching build image" or something like this in the loading screen). I think this should be optimized as part of this ticket since otherwise I fear people will not adopt gitpod for e.g. reviewing of tickets.

Also it still loads pyenv: pyenv shell 3.8.12 so https://github.com/tobiasdiez/sagetrac-mirror/blob/20ca179bed4ff605de4111362b02c8e2acca617e/.gitpod.yml#L18-L19 should probably be readded.

comment:74 in reply to: ↑ 72 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

Also it still loads pyenv: pyenv shell 3.8.12 so https://github.com/tobiasdiez/sagetrac-mirror/blob/20ca179bed4ff605de4111362b02c8e2acca617e/.gitpod.yml#L18-L19 should probably be readded.

It no longer breaks the Sage build since #29285, but I'll add it anyway

comment:75 Changed 6 months ago by git

  • Commit changed from aef75ebe0986f6c499585fcdad36231808e5d21b to fa237c7780e17f5e41babab91bf088908ce721a7

Branch pushed to git repo; I updated commit sha1. New commits:

6215a9fRevert ".gitpod.yml (before): Refer to /workspace/sage/local by full path"
cec1f44.gitpod.yml (before): No timeout for 'before' task that runs after the prebuild
fa237c7.gitpod.yml (before): Disable pyenv

comment:77 in reply to: ↑ 70 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

please be aware that the code may be in a different location such as /workspace/sagetrac-mirror if someone starts gitpod from https://github.com/sagemath/sagetrac-mirror

Thanks, done.

comment:78 in reply to: ↑ 71 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

Could you check that the SSH stuff works as intended and matches the added documentation? and am not sure what the empty id_rsa is intended for

Did you added a PRIVATE_SSH_KEY env variable in gitpod with your trac ssh as described in the docs? It's working for me.

Thanks for checking - I had missed a step.

comment:79 Changed 6 months ago by git

  • Commit changed from fa237c7780e17f5e41babab91bf088908ce721a7 to 8d32070e4d5389244c563db39897a3e61347aeb3

Branch pushed to git repo; I updated commit sha1. New commits:

8d32070.gitpod.yml: Add comments

comment:80 Changed 6 months ago by git

  • Commit changed from 8d32070e4d5389244c563db39897a3e61347aeb3 to 081de0e1219da5718d6c25a93f7e1b12089288fb

Branch pushed to git repo; I updated commit sha1. New commits:

7b9eccf.gitpod.yml: Fix inconsistent indentation
081de0e.gitpod.yml: Log to logs/install.log

comment:81 in reply to: ↑ 76 ; follow-up: Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

https://github.com/sagemath/sagetrac-mirror/compare/develop...u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow#diff-ef2cef9f88b4fe09ca3082140e67f5ad34fb65fb6e228f119d3812261ae51449R438-R442

Is the somewhat strange indention by design?

The link does not show a particular line -- what are you referring to?

Strange, normally that works. It's the gitpod related changes in tox.ini (l. 438)

comment:82 in reply to: ↑ 81 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Is the somewhat strange indention by design?

the gitpod related changes in tox.ini (l. 438)

It is indented as intended. This file aspires to be a spreadsheet.

comment:83 Changed 6 months ago by mkoeppe

There are still some minor details wrong. In logs/sphinx-4.3.1.log:

Uninstalling existing 'sphinx'
Warning: File '/home/gitpod/sage-local/var/lib/sage/wheels/Sphinx-4.2.0-py3-none-any.whl' not found
Warning: Directory '/home/gitpod/sage-local/var/lib/sage/wheels' not found
Running pip-uninstall script for 'sphinx'
/home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/scripts/sphinx/spkg-piprm: line 14: /home/gitpod/sage/build/bin/sage-dist-helpers: No such file or directory
Error: failed to source /home/gitpod/sage/build/bin/sage-dist-helpers
Is /home/gitpod/sage the correct SAGE_ROOT?
Warning: Error running the pip-uninstall script for 'sphinx'; uninstallation may have left behind some files
Removing stamp file '/home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/installed/sphinx-4.2.0'

comment:84 Changed 6 months ago by mkoeppe

Package-removal scripts ({prefix,venv}/var/lib/sage/scripts/*/spkg-piprm) hardcode the SAGE_ROOT and SAGE_SRC from the time when the package was installed (the Docker-build phase); so I will also put a symlink in place for that.

comment:85 Changed 6 months ago by git

  • Commit changed from 081de0e1219da5718d6c25a93f7e1b12089288fb to 9f89af7a68f9a36a59135f6f541bda204f05896f

Branch pushed to git repo; I updated commit sha1. New commits:

952bb4e.gitpod.yml (before): Create symlink from /home/gitpod/sage to the workspace
9f89af7build/bin/sage-spkg: Run sage-spkg-uninstall on the correct tree

comment:86 Changed 6 months ago by mkoeppe

Fixed now, logs/sphinx-4.3.1.log now shows:

Found existing installation: Sphinx 4.2.0
Uninstalling Sphinx-4.2.0:
  Successfully uninstalled Sphinx-4.2.0
Removing stamp file '/home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/installed/sphinx-4.2.0'
Installing sphinx-4.3.1

and also venv/var/lib/sage/wheels/ now only contains one sphinx wheel.

comment:87 Changed 6 months ago by mkoeppe

  • Cc dimpase added
  • Reviewers changed from https://github.com/mkoeppe/sage/runs/4687310728 to Matthias Koeppe, ...

comment:88 follow-up: Changed 6 months ago by mkoeppe

Nice, also docker-in-docker works in the gitpod, so I can run commands such as tox -e docker-ubuntu-bionic-minimal -- config.status there

comment:89 Changed 6 months ago by git

  • Commit changed from 9f89af7a68f9a36a59135f6f541bda204f05896f to 15e112a8b86cdde1a21224edc414e4779488b85d

Branch pushed to git repo; I updated commit sha1. New commits:

0eadcd1Add VS Code config for SAGE_ROOT/venv
075b22cMerge #30677
15e112a.vscode/settings.json: Update interpreter settings

comment:90 Changed 6 months ago by mkoeppe

  • Dependencies changed from #33068, #30933, #29285, #32868, #33067 to #33068, #30933, #29285, #32868, #33067, #30677

comment:91 Changed 6 months ago by mkoeppe

Now it starts with the right interpreter, and also "Jupyter: Create New Jupyter Notebook" works - "Select Kernel" offers the Sage kernel.

comment:92 Changed 6 months ago by gh-tobiasdiez

  • Status changed from needs_review to needs_work
  • Work issues set to Optimize docker image

It still takes over 8min until the workspace is loaded for me due to "Pulling container image". This should really optimized as part of this ticket.

Also pyenv shell still seems to be activated, I think it needs to be disabled in before and init.

comment:93 Changed 6 months ago by git

  • Commit changed from 15e112a8b86cdde1a21224edc414e4779488b85d to 2bf20ed651e0914847d82105891f04a04cc23464

Branch pushed to git repo; I updated commit sha1. New commits:

2bf20ed.github/workflows/tox.yml (docker): Also push -with-system-packages

comment:95 Changed 6 months ago by git

  • Commit changed from 2bf20ed651e0914847d82105891f04a04cc23464 to cb678a8df1b4dafe0281b6f2def371d955ff7ef7

Branch pushed to git repo; I updated commit sha1. New commits:

c6edf67.gitpod.yml: Use a Dockerfile again
cb678a8docker/.gitpod.Dockerfile: Go through ARGs BASE_GITHUB_REPOSITORY, BASE_TAG

comment:96 Changed 6 months ago by git

  • Commit changed from cb678a8df1b4dafe0281b6f2def371d955ff7ef7 to e01dd838a52064f37079c7ed62b7a7aa513ccbe6

Branch pushed to git repo; I updated commit sha1. New commits:

e01dd83.gitpod.yml (before): Run 'pyenv global system'

comment:97 Changed 6 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:98 Changed 6 months ago by mkoeppe

  • Work issues Optimize docker image deleted

comment:99 follow-ups: Changed 6 months ago by gh-tobiasdiez

That sadly doesn't solve the issue for me. Starting a workspace still takes 8+ mins, while #32749 takes about 2mins.

It also seems that the workspace image is quite large at around 16GB. In particular, the sage folder contains

4.9G    local
6.2G    src

I'm not sure if the following folders are really necessary or if they can be deleted (latest at the end of the prebuild):

1.8G    ./local/share/doc
4.3G    ./src/build

Moreover,

3.0G    /usr
3.3G    /home

is also bigger than the one from the other image of #32749

2.2G    /usr
2.3G    /home

So I suggest to use gitpod/workspace-base as a base.

comment:100 in reply to: ↑ 99 ; follow-up: Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

Starting a workspace still takes 8+ mins

Probably depends on the data center region. I just timed it here, it took < 3.5 minutes; but there is considerable variance

comment:101 in reply to: ↑ 99 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

I'm not sure if the following folders are really necessary or if they can be deleted (latest at the end of the prebuild):

1.8G    ./local/share/doc
4.3G    ./src/build

I'll check if we can delete them without sacrificing fast rebuilds of documentation (make doc-html) and sagelib (./sage -b).

For documentation, I see

[sagemath_doc_html-none] make doc-inventory--reference-references
[sagemath_doc_html-none] cd /workspace/sage && ./sage --docbuild --no-pdf-links reference/references inventory 
[sagemath_doc_html-none] [reference] loading pickled environment... failed
[sagemath_doc_html-none] [reference] failed: source directory has changed

... so something is not ideal there anyway

comment:102 in reply to: ↑ 100 Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Starting a workspace still takes 8+ mins

Probably depends on the data center region. I just timed it here, it took < 3.5 minutes; but there is considerable variance

Apparently, images are built in the US and then downloaded every time from there without caching. This explains why it takes so much longer for me. Gitpod is working on a solution https://github.com/gitpod-io/gitpod/issues/7002 and https://github.com/gitpod-io/gitpod/issues/6145. But having a 15GB+ image is not ideal anyway.

comment:103 Changed 6 months ago by mkoeppe

top-level Makefile has targets micro_release, fast-rebuild-clean that do related stuff.

comment:104 Changed 6 months ago by git

  • Commit changed from e01dd838a52064f37079c7ed62b7a7aa513ccbe6 to 874dddd4ef5445ace3d8855106f36e37d7d56472

Branch pushed to git repo; I updated commit sha1. New commits:

4814334docker/.gitpod.Dockerfile: Remove doctrees/inventory
874ddddtox.ini (gitpod): Use gitpod/workspace-base by default

comment:105 in reply to: ↑ 99 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

3.0G    /usr
3.3G    /home

is also bigger than the one from the other image of #32749

2.2G    /usr
2.3G    /home

So I suggest to use gitpod/workspace-base as a base.

Building a new image at https://github.com/mkoeppe/sage/runs/4731195747

comment:106 Changed 6 months ago by mkoeppe

fast-rebuild-clean: misc-clean
	rm -rf upstream/
	rm -rf build/pkgs/sagelib/src/build/temp.*
	# The .py files in src/build are restored from src/sage without their
	# mtimes changed.
	-find build/pkgs/sagelib/src/build -name '*.py' -exec rm \{\} \;
	# Remove leftovers from ancient branches
	rm -rf src/build

This is not compatible with configure --enable-editable, which uses src/build/ instead of build/pkgs/sagelib/src/build/.

comment:107 Changed 6 months ago by gh-tobiasdiez

Why does the editable install actually needs src/build or even copies something there? The compiled cython code (.so files) are placed next to the cython file.

comment:108 Changed 6 months ago by mkoeppe

I think that's just standard setuptools behavior

comment:109 Changed 6 months ago by mkoeppe

The setuptools option --build-temp could be used to put the temp directories outside of /workspace.

Or we could just unconditionally remove the temp directory at the end of build/pkgs/sagelib/spkg-install.

comment:110 Changed 6 months ago by git

  • Commit changed from 874dddd4ef5445ace3d8855106f36e37d7d56472 to c97dc015a437f7746635e7de3577dc25c422213e

Branch pushed to git repo; I updated commit sha1. New commits:

c97dc01build/pkgs/sagelib/spkg-install: Remove build/temp.* directories after build

comment:111 Changed 6 months ago by mkoeppe

It's much smaller now

gitpod /workspace/sage (mkoeppe-9.5.beta9+updates) $ du -sh src local /usr /home
3.1G    src
3.9G    local
2.4G    /usr
76K     /home

and rebuilds are still fast

comment:112 Changed 6 months ago by git

  • Commit changed from c97dc015a437f7746635e7de3577dc25c422213e to 36050766eb9d84679210ead65650869ba0e46412

Branch pushed to git repo; I updated commit sha1. New commits:

3605076build/pkgs/sagelib/spkg-install: Fix up deleting temp.*

comment:113 Changed 6 months ago by gh-tobiasdiez

Thanks, its way better now (around 4-5min). I would say the doc folder can still be removed. If someone is working on the docs, then one can easily only create the relevant file without the need to build the whole docs.

comment:114 Changed 6 months ago by gh-tobiasdiez

The following folders should be deleted as well

212M    /tmp
32M     /var/lib/apt/lists (remove as part of the apt-get install command)

A few packages that I think are not used by the built script or needed in general:

3722    python2.7-minimal
9370    libflint-dev
10026   pari-gp (and other pari-related stuff)
13215   ecl
14279   libsingular4-dev (and other singular-related stuff)
19531   libgiac0 (and other giac-related stuff)
30765   vim-runtime
41291   emacs-nox
66816   emacs-common
330712  libgl1-mesa-dri (?)

I would suggest to exclude them from the installation / uninstall them, similar to what is done in #32749.

Together that should give another 1GB.

comment:115 Changed 6 months ago by git

  • Commit changed from 36050766eb9d84679210ead65650869ba0e46412 to c02c03c8465a2515676b25f0686c1485083bb757

Branch pushed to git repo; I updated commit sha1. New commits:

d87027csage --package list: Add option --exclude
c02c03ctox.ini (gitpod-standard): Exclude some packages

comment:116 follow-up: Changed 6 months ago by mkoeppe

comment:117 Changed 6 months ago by git

  • Commit changed from c02c03c8465a2515676b25f0686c1485083bb757 to 7e74607979224e0b8f7d2803fc8b68f3802a64f4

Branch pushed to git repo; I updated commit sha1. New commits:

6285cd9.gitpod.Dockerfile: Save space by removing apt lists
7e74607.gitpod.Dockerfile: Remove built Sage documentation

comment:118 Changed 6 months ago by gh-tobiasdiez

For the record, that's how gitpod is cleaning up the package installation cache/log: https://github.com/gitpod-io/workspace-images/blob/master/base/install-packages#L32-L36

comment:119 in reply to: ↑ 116 Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

New image building at https://github.com/mkoeppe/sage/runs/4742213151

Thanks! Will try it once its finished.

comment:120 Changed 6 months ago by git

  • Commit changed from 7e74607979224e0b8f7d2803fc8b68f3802a64f4 to f71c8b6c625017b0d37e956b0386111630ad6fe1

Branch pushed to git repo; I updated commit sha1. New commits:

f71c8b6docker/.gitpod.Dockerfile: Clean more

comment:121 Changed 6 months ago by mkoeppe

$ du -sh src local /usr /home /tmp /var/tmp
3.1G    src
3.1G    local
2.0G    /usr
36K     /home
31M     /tmp
0       /var/tmp

comment:122 Changed 6 months ago by mkoeppe

I don't think it's necessary to remove packages such as emacs that come with gitpod/workspace-base.

comment:123 in reply to: ↑ 88 Changed 6 months ago by mkoeppe

Replying to mkoeppe:

Nice, also docker-in-docker works in the gitpod, so I can run commands such as tox -e docker-ubuntu-bionic-minimal -- config.status there

Too bad, this no longer works with the smaller base image because it does not have docker.

And sudo apt-get install docker.io is not sufficient to fix it:

$ docker run -it ubuntu:bionic bash
[...]
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.
ERRO[0008] error waiting for container: context canceled 

$ sudo docker run -it ubuntu:bionic bash
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.
ERRO[0001] error waiting for container: context canceled 
Last edited 6 months ago by mkoeppe (previous) (diff)

comment:124 Changed 6 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:125 Changed 6 months ago by gh-tobiasdiez

Thanks, this seems to work reasonable well for me (still 4 mins, but well...).

One could still shave off 250mb by uninstalling emacs-nox, vim and python2.7 but that is not very significant in view of an 8GB image in total.

Last edited 6 months ago by gh-tobiasdiez (previous) (diff)

comment:126 Changed 6 months ago by slelievre

Ticket description misses verb in this sentence:

After this ticket is merged, we should the automatic prebuild using a github app

comment:127 Changed 6 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:128 Changed 6 months ago by git

  • Commit changed from f71c8b6c625017b0d37e956b0386111630ad6fe1 to f95322851c6494818bc08cd6fb3ce94e9c27debb

Branch pushed to git repo; I updated commit sha1. New commits:

f953228Merge tag '9.5.rc0' into t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflow

comment:129 Changed 6 months ago by mkoeppe

  • Dependencies changed from #33068, #30933, #29285, #32868, #33067, #30677 to #30933, #29285

comment:130 follow-up: Changed 6 months ago by dimpase

  • Reviewers changed from Matthias Koeppe, ... to Matthias Koeppe, Dima Pasechnik
  • Status changed from needs_review to positive_review

let's get this in, it seems to work, I can get ./sage in console there, etc. ./sage -n leads to problems, but probably that's not the way it's intendend.

comment:131 Changed 6 months ago by gh-tobiasdiez

Thanks for the review!

comment:132 in reply to: ↑ 130 Changed 6 months ago by mkoeppe

Replying to dimpase:

./sage -n leads to problems, but probably that's not the way it's intendend.

There's no web browser in the container, so you can't use sage -n. To use Jupyter notebooks, use the Command Palette to run "Jupyter: Create New Jupyter Notebook", see https://code.visualstudio.com/docs/datascience/jupyter-notebooks

comment:133 Changed 6 months ago by git

  • Commit changed from f95322851c6494818bc08cd6fb3ce94e9c27debb to 2d0902a1fee81d0749cc274fbec1d965cc522466
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

2d0902asrc/doc/en/developer/workspace.rst: Fix markup

comment:134 Changed 6 months ago by mkoeppe

  • Status changed from needs_review to positive_review

Fixes the PDF docbuild

comment:135 Changed 6 months ago by git

  • Commit changed from 2d0902a1fee81d0749cc274fbec1d965cc522466 to 08048c3e5d5d995cb5ebfd952d84eb204cb8c41c
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

08048c3.github/workflows/tox.yml: Transform GITHUB_REF_NAME to a valid docker tag

comment:136 Changed 6 months ago by git

  • Commit changed from 08048c3e5d5d995cb5ebfd952d84eb204cb8c41c to 078f247c9a5ea383178499e9b0e990777f497a6e

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

078f247.github/workflows/tox*.yml: Transform GITHUB_REF_NAME to a valid docker tag

comment:137 Changed 6 months ago by mkoeppe

  • Status changed from needs_review to positive_review

comment:138 Changed 6 months ago by dimpase

Is it easy to configure launching this (with prebuids) on branches in https://github.com/sagemath/sagetrac-mirror ?

comment:139 follow-up: Changed 6 months ago by gh-tobiasdiez

Yes, it's only a few clicks see: https://www.gitpod.io/docs/prebuilds#projects-and-prebuilds However, it needs to be done by someone with rights to install apps in the sagemath org.

comment:140 in reply to: ↑ 139 Changed 6 months ago by dimpase

Replying to gh-tobiasdiez:

Yes, it's only a few clicks see: https://www.gitpod.io/docs/prebuilds#projects-and-prebuilds However, it needs to be done by someone with rights to install apps in the sagemath org.

I've sent you an invite. I guess I installed their "app", but I don't see how to proceed. (I think it should be enabled on the branch of this ticket only, before it's merged.)

Did you see my email message about trac access?

comment:141 follow-ups: Changed 6 months ago by gh-tobiasdiez

Thanks! I've now created a "team" sagemath and added the sage and sagetrac-mirror as projects. If you have a gitpod account and want to be invited to this team (in order to access the config), just post your email address and I send an invite.

Currently, the creation of the image fails however due to a permission error:

 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

Matthias, can you have a look please?

https://gitpod.io/#prebuild/https://github.com/sagemath/sagetrac-mirror/tree/u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow

comment:142 in reply to: ↑ 141 Changed 6 months ago by dimpase

Replying to gh-tobiasdiez:

Thanks! I've now created a "team" sagemath and added the sage and sagetrac-mirror as projects. If you have a gitpod account and want to be invited to this team (in order to access the config), just post your email address and I send an invite.

my email is shown on my GitHub account https://github.com/dimpase

Thanks!

comment:143 in reply to: ↑ 141 ; follow-up: Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

Currently, the creation of the image fails however due to a permission error:

 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

I think you need to deposit a secret providing read access similar to what is explained in https://www.gitpod.io/docs/self-hosted/latest/configuration/docker-registry

comment:144 Changed 6 months ago by mkoeppe

... alternatively I suppose we could push the required Docker images to DockerHub? -- but I don't have write access there; @saraedum, could you deposit an access token for that as a secret here? https://github.com/organizations/sagemath/settings/secrets/actions

comment:145 Changed 6 months ago by mkoeppe

  • Cc saraedum added

comment:146 in reply to: ↑ 143 ; follow-up: Changed 6 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Currently, the creation of the image fails however due to a permission error:

 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

I think you need to deposit a secret providing read access similar to what is explained in https://www.gitpod.io/docs/self-hosted/latest/configuration/docker-registry

That link is for a self-hosted gitpod and thus does not apply to our situation (we don't have a gitpod config other than what is committed in this branch).

I think the issue is that the image sage-docker-gitpod-standard-with-targets is not yet published under sagemath/sage. But how is this then working with your account Matthias?

comment:147 in reply to: ↑ 146 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

the image sage-docker-gitpod-standard-with-targets is not yet published under sagemath/sage. But how is this then working with your account Matthias?

Because on my testing branch used for the prebuild, I am setting the correct repo: https://github.com/mkoeppe/sage/blob/mkoeppe-9.5.beta9%2Bupdates/docker/.gitpod.Dockerfile

comment:148 Changed 6 months ago by mkoeppe

(And because every GH workflow automatically has the credentials to access its own GH packages (gchr.io)!)

comment:149 follow-up: Changed 6 months ago by gh-tobiasdiez

Ahh okay, I thought this is an environment variable that is picked up somewhere.

So in this case we just have to wait until this ticket is merged, which would then build and publish the docker image to sagemath/sage, right?

comment:150 in reply to: ↑ 149 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

wait until this ticket is merged, which would then build and publish the docker image to sagemath/sage, right?

Yes, on the next release that includes this branch, the Docker image will be pushed (by separate instances of the workflow) both to gchr.io/sagemath/sage and to gchr.io/sagemath/sagetrac-mirror.

We will then see whether the 403 Forbidden will go away. I think it will not, for workflows run on github.com/sagemath/sagetrac-mirror, because it may not have the read:packages credential for gchr.io/sagemath/sage.

comment:154 Changed 6 months ago by mkoeppe

No, this makes no sense.

comment:155 Changed 5 months ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6

comment:156 Changed 5 months ago by mkoeppe

  • Priority changed from major to critical

comment:157 Changed 5 months ago by vbraun

  • Branch changed from u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow to 078f247c9a5ea383178499e9b0e990777f497a6e
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.