Opened 6 months ago
Closed 5 months ago
#33103 closed enhancement (fixed)
gitpod integration using Docker images from portability testing workflow
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | critical | Milestone: | sage-9.6 |
Component: | docker | Keywords: | |
Cc: | gh-tobiasdiez, dimpase, saraedum | Merged in: | |
Authors: | Tobias Diez, Matthias Koeppe | Reviewers: | Matthias Koeppe, Dima Pasechnik |
Report Upstream: | N/A | Work issues: | |
Branch: | 078f247 (Commits, GitHub, GitLab) | Commit: | 078f247c9a5ea383178499e9b0e990777f497a6e |
Dependencies: | #30933, #29285 | Stopgaps: |
Description (last modified by )
Gitpod allows one to setup a complete dev environment in the cloud. It is free to use for up to 50 hours per month. In this ticket the config necessary for making this work with sagemath is added.
You can try this out by going to https://gitpod.io/#https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates.
It works by building docker images on github, push them to GH packages (https://github.com/mkoeppe?tab=packages&q=gitpod) and then use these ones as the base for the gitpod workspace.
After this ticket is merged, we should activate the automatic prebuild by installing the github app (https://www.gitpod.io/docs/prebuilds#on-github). With this enabled, every push would trigger a prebuild of the complete environment (including the build of all dependencies and cythonizion) so that one has a up-to-date code env very quickly without the need to build anything.
Follow-up: #33113 gitpod usability improvements
Alternative: #32749, which only builds the docker image on gitpod (which is currently not really feasible as they have a time limit of 1h)
Change History (157)
comment:1 Changed 6 months ago by
- Branch set to u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow
comment:2 Changed 6 months ago by
- Commit set to bde9f74bc7be3ca657f3d6ca8ab420709abfbeda
comment:3 Changed 6 months ago by
- Commit changed from bde9f74bc7be3ca657f3d6ca8ab420709abfbeda to 76b5e76ab2c28a3ee4a5c1571737bc2f8571f5ff
comment:4 Changed 6 months ago by
- Description modified (diff)
comment:5 Changed 6 months ago by
- Commit changed from 76b5e76ab2c28a3ee4a5c1571737bc2f8571f5ff to 7ef6a6e3333a43153b92e54dceb3b06ab0713387
Branch pushed to git repo; I updated commit sha1. New commits:
7ef6a6e | docker/.gitpod.Dockerfile: Remove, use custom image from GH Packages instead
|
comment:6 Changed 6 months ago by
- Description modified (diff)
comment:7 Changed 6 months ago by
- Description modified (diff)
comment:8 Changed 6 months ago by
- Commit changed from 7ef6a6e3333a43153b92e54dceb3b06ab0713387 to 6b41c4af670b9e8a653e8c2e4049848e4e87d8a1
Branch pushed to git repo; I updated commit sha1. New commits:
6b41c4a | .github/workflows/tox.yml: Also push ...-with-targets-pre
|
comment:9 Changed 6 months ago by
Running into pyenv
issues (#29285) here too:
[pytest] installing. Log file: /home/gitpod/sage/logs/pkgs/pytest.log [pytest] error installing, exit status 1. End of log file: [pytest] /home/gitpod/sage-local/var/lib/sage/venv-python3.8/bin/python3: No module named pip [pytest] Full log file: /home/gitpod/sage/logs/pkgs/pytest.log make[4]: *** [Makefile:2756: pytest-no-deps] Error 1
comment:10 Changed 6 months ago by
- Dependencies changed from #33068, #30933 to #33068, #30933, #29285
comment:11 Changed 6 months ago by
- Commit changed from 6b41c4af670b9e8a653e8c2e4049848e4e87d8a1 to 5f0c36cc73b59e64eba006ed5a5dfb51efd920bf
comment:12 Changed 6 months ago by
- Dependencies changed from #33068, #30933, #29285 to #33068, #30933, #29285, #32868
comment:13 Changed 6 months ago by
- Dependencies changed from #33068, #30933, #29285, #32868 to #33068, #30933, #29285, #32868, #33067
comment:14 Changed 6 months ago by
- Commit changed from 5f0c36cc73b59e64eba006ed5a5dfb51efd920bf to af7c8d3c2237980a12b89ffad4c0cc0531662ad3
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
aeca1a0 | Makefile (pypi-sdists): Add sage_setup
|
cd3dce9 | pkgs/sage-conf_pypi/setup.py: Update directory of configured sage_conf.py
|
c695921 | pkgs/sage-conf/sage_conf.py.in (SAGE_SPKG_WHEELS): Update so it works if SAGE_VENV != SAGE_LOCAL
|
cebcda3 | Merge tag '9.5.beta4' into t/29039/pip_installable_package_sage_bootstrap
|
0026892 | Merge tag '9.5.beta5' into t/29039/pip_installable_package_sage_bootstrap
|
2a4a323 | Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap
|
02234ce | Merge #29039
|
b1413d4 | build/bin/write-dockerfile.sh: Move 'ADD pkgs' earlier
|
859be3e | build/bin/write-dockerfile.sh: ADD src/*.m4 files for bootstrapping
|
af7c8d3 | Merge #33067
|
comment:15 Changed 6 months ago by
- Description modified (diff)
comment:16 Changed 6 months ago by
- Commit changed from af7c8d3c2237980a12b89ffad4c0cc0531662ad3 to fad8942c93702e6c098fe6b9cd2d68704b256334
Branch pushed to git repo; I updated commit sha1. New commits:
fad8942 | build/bin/write-dockerfile.sh: Use relative workdir 'sage' (again)
|
comment:17 Changed 6 months ago by
- Description modified (diff)
comment:18 Changed 6 months ago by
- Commit changed from fad8942c93702e6c098fe6b9cd2d68704b256334 to 1258c702dac22a428cf0d1ede65ccea35cf50d0f
comment:19 Changed 6 months ago by
This works. To be done:
- Additional tags for the images, for example
latest
for releases, or branch/tag name (${GITHUB_REF_NAME}
- https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables) - suitable for.gitpod.yml
- Could use
docker/.gitpod.Dockerfile
again withCOPY --from
to make the final image smaller (currently /home/gitpod/sage has the old source tree)
comment:20 Changed 6 months ago by
- Description modified (diff)
comment:21 Changed 6 months ago by
- Commit changed from 1258c702dac22a428cf0d1ede65ccea35cf50d0f to fdb49a1d7865f8105c50470715368af9a8ede02d
Branch pushed to git repo; I updated commit sha1. New commits:
fdb49a1 | tox.ini, .github/workflows/tox*.yml: Also tag the Docker images with GITHUB_REF_NAME
|
comment:22 Changed 6 months ago by
- Reviewers set to https://github.com/mkoeppe/sage/runs/4686167146
comment:23 Changed 6 months ago by
- Commit changed from fdb49a1d7865f8105c50470715368af9a8ede02d to ab757ba44a661a0c358764d3aa40c49709f4c6cd
Branch pushed to git repo; I updated commit sha1. New commits:
ab757ba | .github/workflows/tox.yml: Fix passing of EXTRA_DOCKER_TAGS
|
comment:24 Changed 6 months ago by
- Commit changed from ab757ba44a661a0c358764d3aa40c49709f4c6cd to 96528ebee43305123a813b340ad3e027ba2f8712
Branch pushed to git repo; I updated commit sha1. New commits:
96528eb | .github/workflows/tox*.yml: Tag docker images 'latest' or 'dev' when building a release tag
|
comment:25 Changed 6 months ago by
- Reviewers changed from https://github.com/mkoeppe/sage/runs/4686167146 to https://github.com/mkoeppe/sage/runs/4686610379
comment:26 Changed 6 months ago by
- Reviewers changed from https://github.com/mkoeppe/sage/runs/4686610379 to https://github.com/mkoeppe/sage/runs/4687073955
comment:27 Changed 6 months ago by
- Reviewers changed from https://github.com/mkoeppe/sage/runs/4687073955 to https://github.com/mkoeppe/sage/runs/4687310789
comment:28 Changed 6 months ago by
- Reviewers changed from https://github.com/mkoeppe/sage/runs/4687310789 to https://github.com/mkoeppe/sage/runs/4687310728
comment:29 Changed 6 months ago by
- Commit changed from 96528ebee43305123a813b340ad3e027ba2f8712 to 5d1dc0fa92a7192bee0bb56e9ea13944da97007f
Branch pushed to git repo; I updated commit sha1. New commits:
5d1dc0f | .gitpod.yml: Use tag :dev
|
comment:30 Changed 6 months ago by
- Status changed from new to needs_review
- Work issues set to Change .gitpod.yml to use sagemath instead of mkoeppe
comment:31 Changed 6 months ago by
Also the configuration of VS Code should be improved. It seems to start up using the wrong interpreter (not venv/bin/python3
), and also one needs to manually select the right Jupyter kernel.
The scipy folks have some good documentation and configuration (https://scipy.github.io/devdocs/dev/contributor/quickstart_gitpod.html), perhaps we can lift something from there.
comment:32 follow-ups: ↓ 33 ↓ 34 Changed 6 months ago by
Is the prebuild working for this branch?
How can one work with branches that update sage packages? With #32749 one could trigger a rebuild of the docker image (and thus of the system packages) using the #imagebuild
context. Can something like this also be implemented here? For example, take the github action build docker as base, and then re-install system or python packages on top of this if they have newer versions?
For the VS integration, #30677 is closed but not yet merged into develop.
comment:33 in reply to: ↑ 32 ; follow-up: ↓ 35 Changed 6 months ago by
Replying to gh-tobiasdiez:
Is the prebuild working for this branch?
I've just merged #33071 into the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates to try this out.
How can one work with branches that update sage packages?
Nothing special, make
just rebuilds what needs rebuilding.
comment:34 in reply to: ↑ 32 Changed 6 months ago by
Replying to gh-tobiasdiez:
For the VS integration, #30677 is closed but not yet merged into develop.
I forgot to say that the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates already contains #30677 and other updates
comment:35 in reply to: ↑ 33 Changed 6 months ago by
Replying to mkoeppe:
Replying to gh-tobiasdiez:
Is the prebuild working for this branch?
I've just merged #33071 into the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates to try this out.
How can one work with branches that update sage packages?
Nothing special,
make
just rebuilds what needs rebuilding.
So this would then be done as part of the prebuild?
comment:36 Changed 6 months ago by
Yes, I think so
comment:37 Changed 6 months ago by
Looks like I'll have to add some configuration so that it does prebuilds on all branches
comment:38 Changed 6 months ago by
I created a "project" and now it looks like the prebuild triggered by a push is running - https://gitpod.io/projects/sage
comment:39 Changed 6 months ago by
- Commit changed from 5d1dc0fa92a7192bee0bb56e9ea13944da97007f to 3e28e13afa9d3a61aceb4d6b3c3d66451f47133f
Branch pushed to git repo; I updated commit sha1. New commits:
3e28e13 | .gitpod.yml: Set github prebuild options to true
|
comment:40 follow-up: ↓ 41 Changed 6 months ago by
Before the push I had merged #32968 with the Sphinx update; and the prebuild has just installed the upgraded package
comment:41 in reply to: ↑ 40 Changed 6 months ago by
comment:42 Changed 6 months ago by
- Commit changed from 3e28e13afa9d3a61aceb4d6b3c3d66451f47133f to 49575f56f74343fb81b401a6077a3f05bb037c61
Branch pushed to git repo; I updated commit sha1. New commits:
49575f5 | .gitpod.yml: Switch to image ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev
|
comment:43 Changed 6 months ago by
- Description modified (diff)
- Work issues Change .gitpod.yml to use sagemath instead of mkoeppe deleted
comment:44 Changed 6 months ago by
The prebuild was just killed because of the 35m timeout. What's the best choice of timeout according to your experiments?
comment:45 Changed 6 months ago by
If its really the timeout that killed the prebuild (in which the prebuild would be green and be used for the next workspace, but maybe doesn't contain all packages) then you can go to 50min. Maybe a bit less since the docker image is quite big so this part may take some time. But essentially you have to calculate 60min - bootstrap and config (2-3min) - docker init and copying (?? min).
However, for me the prebuild often timed out after 1h (then its shown red in the project view). The only solution I've found for this is not using symlinks into /home/.
comment:46 Changed 6 months ago by
https://gitpod.io/projects/sage/e8780c03-b8b8-4810-a772-d4cc2994b6bf shows
[sagelib-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sagelib-9.5.beta9.log [pytest] successfully installed. [sphinx-4.3.1] successfully installed. make[1]: *** [Makefile:39: all-build] Terminated make: *** [Makefile:16: build] Terminated exit
but it is still claims "RUNNING: Prebuild in progress"
comment:47 Changed 6 months ago by
However, https://gitpod.io/projects/sage/prebuilds shows READY.
comment:48 follow-up: ↓ 50 Changed 6 months ago by
🤙 This task ran as a workspace prebuild 🎉 Well done on saving 37 minutes
comment:49 follow-up: ↓ 56 Changed 6 months ago by
A few more points:
- please use the recent version of gitpod.yml from https://github.com/tobiasdiez/sagetrac-mirror/blob/develop/.gitpod.yml (the ssh stuff and num threads = 8)
--prefix=$HOME/sage-local
in gitpod.yml should probably be/workspace/sage-local
or a few lines above$HOME
should be used?- probably 8 parallel tasks are fine. gitpod gives 6 (or 7?) vCPUs to users.
comment:50 in reply to: ↑ 48 Changed 6 months ago by
Replying to mkoeppe:
🤙 This task ran as a workspace prebuild 🎉 Well done on saving 37 minutes
Nice, then you can go to 50min indeed.
comment:51 Changed 6 months ago by
- Commit changed from 49575f56f74343fb81b401a6077a3f05bb037c61 to e7d4e966d8d25281befe14c5b66bae991c4736e3
Branch pushed to git repo; I updated commit sha1. New commits:
e7d4e96 | .gitpod.yml: Increase timeout for make in prebuild, increase parallelism
|
comment:52 Changed 6 months ago by
Presumably the before
script should try to resume the build in case the prebuild timed out.
comment:53 Changed 6 months ago by
I agree. But init
is called after before
during the prebuild, and I couldn't find a way to detect if one is in a prebuild or not...
comment:54 follow-up: ↓ 60 Changed 6 months ago by
- Commit changed from e7d4e966d8d25281befe14c5b66bae991c4736e3 to fdb332d2a52db9f63fc54960d029a5ad9c529a61
Branch pushed to git repo; I updated commit sha1. New commits:
fdb332d | .gitpod.yml: Run 'make' again in 'before' task if prebuild 'make' timed out
|
comment:55 Changed 6 months ago by
- Commit changed from fdb332d2a52db9f63fc54960d029a5ad9c529a61 to aaa97778417b2142d9f3cf83c29dce9d38d1853e
Branch pushed to git repo; I updated commit sha1. New commits:
aaa9777 | .gitpod.yml: Use SAGE_NUM_THREADS=8
|
comment:56 in reply to: ↑ 49 Changed 6 months ago by
Replying to gh-tobiasdiez:
- please use the recent version of gitpod.yml from https://github.com/tobiasdiez/sagetrac-mirror/blob/develop/.gitpod.yml (the ssh stuff and num threads = 8)
I've updated it - the ssh stuff was already same
comment:57 Changed 6 months ago by
- Commit changed from aaa97778417b2142d9f3cf83c29dce9d38d1853e to 87900a7d0188247be6d0c6db7b87349759621ca3
Branch pushed to git repo; I updated commit sha1. New commits:
87900a7 | .gitpod.yml: Refer to /home/gitpod as /Users/mkoeppe
|
comment:58 Changed 6 months ago by
- Commit changed from 87900a7d0188247be6d0c6db7b87349759621ca3 to 0864c6ec3e00d24566b586472ffb88e1471a2e30
Branch pushed to git repo; I updated commit sha1. New commits:
0864c6e | .github/workflows/tox*.yml: Push stable releases also to Docker tag 'dev'
|
comment:59 Changed 6 months ago by
https://gitpod.io/projects/sage/134676a9-2a0e-429a-816d-68ad30441f58
[sphinx-4.3.1] installing. Log file: /workspace/sage/logs/pkgs/sphinx-4.3.1.log [sagelib-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sagelib-9.5.beta9.log [pytest] successfully installed. [sphinx-4.3.1] successfully installed. [sagelib-9.5.beta9] successfully installed. make --no-print-directory sage_docbuild-SAGE_VENV-no-deps [sage_docbuild-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sage_docbuild-9.5.beta9.log [sage_docbuild-9.5.beta9] successfully installed. make[2]: Leaving directory '/workspace/sage/build/make' real 31m8.684s user 101m44.392s sys 3m47.093s SAGE_CHECK=warn, so scanning the log files. This may take a few seconds. Sage build/upgrade complete! make[1]: Leaving directory '/workspace/sage' exit 🤙 This task ran as a workspace prebuild 🎉 Well done on saving 34 minutes
comment:60 in reply to: ↑ 54 Changed 6 months ago by
Replying to git:
Branch pushed to git repo; I updated commit sha1. New commits:
fdb332d .gitpod.yml: Run 'make' again in 'before' task if prebuild 'make' timed out
This worked but for some reason the Sphinx package installation was repeated in this step. It's fast but I'll try if refreshing the timestamps of the install records will avoid it.
comment:61 Changed 6 months ago by
- Commit changed from 0864c6ec3e00d24566b586472ffb88e1471a2e30 to c9fa0152e86c3631ae69cb6d7ad75ed133e93bd6
comment:62 Changed 6 months ago by
There are some weird synchronization issues on the server side - it seems that if one creates a workspace right after a prebuild has completed, it can happen that it does not use the prebuild.
comment:63 Changed 6 months ago by
- Commit changed from c9fa0152e86c3631ae69cb6d7ad75ed133e93bd6 to 38dafc60837e3de5455e24e6913150791b05e509
Branch pushed to git repo; I updated commit sha1. New commits:
38dafc6 | .gitpod.yml (before): Refer to /workspace/sage/local by full path
|
comment:64 Changed 6 months ago by
- Commit changed from 38dafc60837e3de5455e24e6913150791b05e509 to e6be067e2c7210f5f04e0903a5e45a101461da63
Branch pushed to git repo; I updated commit sha1. New commits:
e6be067 | .gitpod.yml (init): Make sure to remove the directory /home/gitpod/sage-local
|
comment:65 Changed 6 months ago by
- Commit changed from e6be067e2c7210f5f04e0903a5e45a101461da63 to cdcf2acb4e53644963484644e2c0e244f28d36b1
Branch pushed to git repo; I updated commit sha1. New commits:
cdcf2ac | .gitpod.yml (before): establish SAGE_LOCAL here, not in 'init'
|
comment:66 Changed 6 months ago by
- Commit changed from cdcf2acb4e53644963484644e2c0e244f28d36b1 to f58453d1aeeb84d5508498d6178751405798657a
comment:67 Changed 6 months ago by
- Commit changed from f58453d1aeeb84d5508498d6178751405798657a to aef75ebe0986f6c499585fcdad36231808e5d21b
Branch pushed to git repo; I updated commit sha1. New commits:
aef75eb | .gitpod.yml (before): Always remove the source tree used by the Docker build
|
comment:68 follow-ups: ↓ 71 ↓ 72 Changed 6 months ago by
OK, now it's in good shape.
Could you check that the SSH stuff works as intended and matches the added documentation? I see
$ ls -l ~/.ssh total 24 -rw-r--r-- 1 gitpod gitpod 381 Jan 4 08:53 authorized_keys -rw------- 1 gitpod gitpod 0 Jan 4 08:53 id_rsa -rw-r--r-- 1 gitpod gitpod 664 Jan 4 08:53 known_hosts -rw-r--r-- 1 gitpod gitpod 15544 Jan 4 08:53 supervisor_env
and am not sure what the empty id_rsa
is intended for
comment:69 Changed 6 months ago by
comment:70 follow-up: ↓ 77 Changed 6 months ago by
I'm not sure if I understand the logic behind this and related commands (maybe add a bit more documentation of what's going on?). But please be aware that the code may be in a different location such as /workspace/sagetrac-mirror
if someone starts gitpod from https://github.com/sagemath/sagetrac-mirror (which is probably what's going to happen if you want to check someone elses branch)
comment:71 in reply to: ↑ 68 ; follow-up: ↓ 78 Changed 6 months ago by
Replying to mkoeppe:
Could you check that the SSH stuff works as intended and matches the added documentation? and am not sure what the empty
id_rsa
is intended for
Did you added a PRIVATE_SSH_KEY
env variable in gitpod with your trac ssh as described in the docs? It's working for me.
comment:72 in reply to: ↑ 68 ; follow-up: ↓ 74 Changed 6 months ago by
Replying to mkoeppe:
OK, now it's in good shape.
For me it takes over 5min to start a workspace due to the large workspace image (it displays "fetching build image" or something like this in the loading screen). I think this should be optimized as part of this ticket since otherwise I fear people will not adopt gitpod for e.g. reviewing of tickets.
Also it still loads pyenv
: pyenv shell 3.8.12
so https://github.com/tobiasdiez/sagetrac-mirror/blob/20ca179bed4ff605de4111362b02c8e2acca617e/.gitpod.yml#L18-L19 should probably be readded.
comment:73 follow-up: ↓ 76 Changed 6 months ago by
comment:74 in reply to: ↑ 72 Changed 6 months ago by
Replying to gh-tobiasdiez:
Also it still loads
pyenv
:pyenv shell 3.8.12
so https://github.com/tobiasdiez/sagetrac-mirror/blob/20ca179bed4ff605de4111362b02c8e2acca617e/.gitpod.yml#L18-L19 should probably be readded.
It no longer breaks the Sage build since #29285, but I'll add it anyway
comment:75 Changed 6 months ago by
- Commit changed from aef75ebe0986f6c499585fcdad36231808e5d21b to fa237c7780e17f5e41babab91bf088908ce721a7
comment:76 in reply to: ↑ 73 ; follow-up: ↓ 81 Changed 6 months ago by
Replying to gh-tobiasdiez:
Is the somewhat strange indention by design?
The link does not show a particular line -- what are you referring to?
comment:77 in reply to: ↑ 70 Changed 6 months ago by
Replying to gh-tobiasdiez:
please be aware that the code may be in a different location such as
/workspace/sagetrac-mirror
if someone starts gitpod from https://github.com/sagemath/sagetrac-mirror
Thanks, done.
comment:78 in reply to: ↑ 71 Changed 6 months ago by
Replying to gh-tobiasdiez:
Replying to mkoeppe:
Could you check that the SSH stuff works as intended and matches the added documentation? and am not sure what the empty
id_rsa
is intended forDid you added a
PRIVATE_SSH_KEY
env variable in gitpod with your trac ssh as described in the docs? It's working for me.
Thanks for checking - I had missed a step.
comment:79 Changed 6 months ago by
- Commit changed from fa237c7780e17f5e41babab91bf088908ce721a7 to 8d32070e4d5389244c563db39897a3e61347aeb3
Branch pushed to git repo; I updated commit sha1. New commits:
8d32070 | .gitpod.yml: Add comments
|
comment:80 Changed 6 months ago by
- Commit changed from 8d32070e4d5389244c563db39897a3e61347aeb3 to 081de0e1219da5718d6c25a93f7e1b12089288fb
comment:81 in reply to: ↑ 76 ; follow-up: ↓ 82 Changed 6 months ago by
Replying to mkoeppe:
Replying to gh-tobiasdiez:
Is the somewhat strange indention by design?
The link does not show a particular line -- what are you referring to?
Strange, normally that works. It's the gitpod related changes in tox.ini
(l. 438)
comment:82 in reply to: ↑ 81 Changed 6 months ago by
Replying to gh-tobiasdiez:
Replying to mkoeppe:
Replying to gh-tobiasdiez:
Is the somewhat strange indention by design?
the gitpod related changes in
tox.ini
(l. 438)
It is indented as intended. This file aspires to be a spreadsheet.
comment:83 Changed 6 months ago by
There are still some minor details wrong. In logs/sphinx-4.3.1.log
:
Uninstalling existing 'sphinx' Warning: File '/home/gitpod/sage-local/var/lib/sage/wheels/Sphinx-4.2.0-py3-none-any.whl' not found Warning: Directory '/home/gitpod/sage-local/var/lib/sage/wheels' not found Running pip-uninstall script for 'sphinx' /home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/scripts/sphinx/spkg-piprm: line 14: /home/gitpod/sage/build/bin/sage-dist-helpers: No such file or directory Error: failed to source /home/gitpod/sage/build/bin/sage-dist-helpers Is /home/gitpod/sage the correct SAGE_ROOT? Warning: Error running the pip-uninstall script for 'sphinx'; uninstallation may have left behind some files Removing stamp file '/home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/installed/sphinx-4.2.0'
comment:84 Changed 6 months ago by
Package-removal scripts ({prefix,venv}/var/lib/sage/scripts/*/spkg-piprm
) hardcode the SAGE_ROOT
and SAGE_SRC
from the time when the package was installed (the Docker-build phase); so I will also put a symlink in place for that.
comment:85 Changed 6 months ago by
- Commit changed from 081de0e1219da5718d6c25a93f7e1b12089288fb to 9f89af7a68f9a36a59135f6f541bda204f05896f
comment:86 Changed 6 months ago by
Fixed now, logs/sphinx-4.3.1.log
now shows:
Found existing installation: Sphinx 4.2.0 Uninstalling Sphinx-4.2.0: Successfully uninstalled Sphinx-4.2.0 Removing stamp file '/home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/installed/sphinx-4.2.0' Installing sphinx-4.3.1
and also venv/var/lib/sage/wheels/
now only contains one sphinx wheel.
comment:87 Changed 6 months ago by
- Cc dimpase added
- Reviewers changed from https://github.com/mkoeppe/sage/runs/4687310728 to Matthias Koeppe, ...
comment:88 follow-up: ↓ 123 Changed 6 months ago by
Nice, also docker-in-docker works in the gitpod, so I can run commands such as tox -e docker-ubuntu-bionic-minimal -- config.status
there
comment:89 Changed 6 months ago by
- Commit changed from 9f89af7a68f9a36a59135f6f541bda204f05896f to 15e112a8b86cdde1a21224edc414e4779488b85d
comment:90 Changed 6 months ago by
- Dependencies changed from #33068, #30933, #29285, #32868, #33067 to #33068, #30933, #29285, #32868, #33067, #30677
comment:91 Changed 6 months ago by
Now it starts with the right interpreter, and also "Jupyter: Create New Jupyter Notebook" works - "Select Kernel" offers the Sage kernel.
comment:92 Changed 6 months ago by
- Status changed from needs_review to needs_work
- Work issues set to Optimize docker image
It still takes over 8min until the workspace is loaded for me due to "Pulling container image". This should really optimized as part of this ticket.
Also pyenv shell still seems to be activated, I think it needs to be disabled in before and init.
comment:93 Changed 6 months ago by
- Commit changed from 15e112a8b86cdde1a21224edc414e4779488b85d to 2bf20ed651e0914847d82105891f04a04cc23464
Branch pushed to git repo; I updated commit sha1. New commits:
2bf20ed | .github/workflows/tox.yml (docker): Also push -with-system-packages
|
comment:94 Changed 6 months ago by
Ready now in https://github.com/mkoeppe?tab=packages&q=gitpod
comment:95 Changed 6 months ago by
- Commit changed from 2bf20ed651e0914847d82105891f04a04cc23464 to cb678a8df1b4dafe0281b6f2def371d955ff7ef7
comment:96 Changed 6 months ago by
- Commit changed from cb678a8df1b4dafe0281b6f2def371d955ff7ef7 to e01dd838a52064f37079c7ed62b7a7aa513ccbe6
Branch pushed to git repo; I updated commit sha1. New commits:
e01dd83 | .gitpod.yml (before): Run 'pyenv global system'
|
comment:97 Changed 6 months ago by
- Status changed from needs_work to needs_review
comment:98 Changed 6 months ago by
- Work issues Optimize docker image deleted
comment:99 follow-ups: ↓ 100 ↓ 101 ↓ 105 Changed 6 months ago by
That sadly doesn't solve the issue for me. Starting a workspace still takes 8+ mins, while #32749 takes about 2mins.
It also seems that the workspace image is quite large at around 16GB. In particular, the sage folder contains
4.9G local 6.2G src
I'm not sure if the following folders are really necessary or if they can be deleted (latest at the end of the prebuild):
1.8G ./local/share/doc 4.3G ./src/build
Moreover,
3.0G /usr 3.3G /home
is also bigger than the one from the other image of #32749
2.2G /usr 2.3G /home
So I suggest to use gitpod/workspace-base as a base.
comment:100 in reply to: ↑ 99 ; follow-up: ↓ 102 Changed 6 months ago by
Replying to gh-tobiasdiez:
Starting a workspace still takes 8+ mins
Probably depends on the data center region. I just timed it here, it took < 3.5 minutes; but there is considerable variance
comment:101 in reply to: ↑ 99 Changed 6 months ago by
Replying to gh-tobiasdiez:
I'm not sure if the following folders are really necessary or if they can be deleted (latest at the end of the prebuild):
1.8G ./local/share/doc 4.3G ./src/build
I'll check if we can delete them without sacrificing fast rebuilds of documentation (make doc-html
) and sagelib (./sage -b
).
For documentation, I see
[sagemath_doc_html-none] make doc-inventory--reference-references [sagemath_doc_html-none] cd /workspace/sage && ./sage --docbuild --no-pdf-links reference/references inventory [sagemath_doc_html-none] [reference] loading pickled environment... failed [sagemath_doc_html-none] [reference] failed: source directory has changed
... so something is not ideal there anyway
comment:102 in reply to: ↑ 100 Changed 6 months ago by
Replying to mkoeppe:
Replying to gh-tobiasdiez:
Starting a workspace still takes 8+ mins
Probably depends on the data center region. I just timed it here, it took < 3.5 minutes; but there is considerable variance
Apparently, images are built in the US and then downloaded every time from there without caching. This explains why it takes so much longer for me. Gitpod is working on a solution https://github.com/gitpod-io/gitpod/issues/7002 and https://github.com/gitpod-io/gitpod/issues/6145. But having a 15GB+ image is not ideal anyway.
comment:103 Changed 6 months ago by
top-level Makefile
has targets micro_release
, fast-rebuild-clean
that do related stuff.
comment:104 Changed 6 months ago by
- Commit changed from e01dd838a52064f37079c7ed62b7a7aa513ccbe6 to 874dddd4ef5445ace3d8855106f36e37d7d56472
comment:105 in reply to: ↑ 99 Changed 6 months ago by
Replying to gh-tobiasdiez:
3.0G /usr 3.3G /homeis also bigger than the one from the other image of #32749
2.2G /usr 2.3G /homeSo I suggest to use gitpod/workspace-base as a base.
Building a new image at https://github.com/mkoeppe/sage/runs/4731195747
comment:106 Changed 6 months ago by
fast-rebuild-clean: misc-clean rm -rf upstream/ rm -rf build/pkgs/sagelib/src/build/temp.* # The .py files in src/build are restored from src/sage without their # mtimes changed. -find build/pkgs/sagelib/src/build -name '*.py' -exec rm \{\} \; # Remove leftovers from ancient branches rm -rf src/build
This is not compatible with configure --enable-editable
, which uses src/build/
instead of build/pkgs/sagelib/src/build/
.
comment:107 Changed 6 months ago by
Why does the editable install actually needs src/build
or even copies something there? The compiled cython code (.so files) are placed next to the cython file.
comment:108 Changed 6 months ago by
I think that's just standard setuptools
behavior
comment:109 Changed 6 months ago by
The setuptools
option --build-temp
could be used to put the temp directories outside of /workspace
.
Or we could just unconditionally remove the temp directory at the end of build/pkgs/sagelib/spkg-install
.
comment:110 Changed 6 months ago by
- Commit changed from 874dddd4ef5445ace3d8855106f36e37d7d56472 to c97dc015a437f7746635e7de3577dc25c422213e
Branch pushed to git repo; I updated commit sha1. New commits:
c97dc01 | build/pkgs/sagelib/spkg-install: Remove build/temp.* directories after build
|
comment:111 Changed 6 months ago by
It's much smaller now
gitpod /workspace/sage (mkoeppe-9.5.beta9+updates) $ du -sh src local /usr /home 3.1G src 3.9G local 2.4G /usr 76K /home
and rebuilds are still fast
comment:112 Changed 6 months ago by
- Commit changed from c97dc015a437f7746635e7de3577dc25c422213e to 36050766eb9d84679210ead65650869ba0e46412
Branch pushed to git repo; I updated commit sha1. New commits:
3605076 | build/pkgs/sagelib/spkg-install: Fix up deleting temp.*
|
comment:113 Changed 6 months ago by
Thanks, its way better now (around 4-5min). I would say the doc folder can still be removed. If someone is working on the docs, then one can easily only create the relevant file without the need to build the whole docs.
comment:114 Changed 6 months ago by
The following folders should be deleted as well
212M /tmp 32M /var/lib/apt/lists (remove as part of the apt-get install command)
A few packages that I think are not used by the built script or needed in general:
3722 python2.7-minimal 9370 libflint-dev 10026 pari-gp (and other pari-related stuff) 13215 ecl 14279 libsingular4-dev (and other singular-related stuff) 19531 libgiac0 (and other giac-related stuff) 30765 vim-runtime 41291 emacs-nox 66816 emacs-common 330712 libgl1-mesa-dri (?)
I would suggest to exclude them from the installation / uninstall them, similar to what is done in #32749.
Together that should give another 1GB.
comment:115 Changed 6 months ago by
- Commit changed from 36050766eb9d84679210ead65650869ba0e46412 to c02c03c8465a2515676b25f0686c1485083bb757
comment:116 follow-up: ↓ 119 Changed 6 months ago by
New image building at https://github.com/mkoeppe/sage/runs/4742213151
comment:117 Changed 6 months ago by
- Commit changed from c02c03c8465a2515676b25f0686c1485083bb757 to 7e74607979224e0b8f7d2803fc8b68f3802a64f4
comment:118 Changed 6 months ago by
For the record, that's how gitpod is cleaning up the package installation cache/log: https://github.com/gitpod-io/workspace-images/blob/master/base/install-packages#L32-L36
comment:119 in reply to: ↑ 116 Changed 6 months ago by
Replying to mkoeppe:
New image building at https://github.com/mkoeppe/sage/runs/4742213151
Thanks! Will try it once its finished.
comment:120 Changed 6 months ago by
- Commit changed from 7e74607979224e0b8f7d2803fc8b68f3802a64f4 to f71c8b6c625017b0d37e956b0386111630ad6fe1
Branch pushed to git repo; I updated commit sha1. New commits:
f71c8b6 | docker/.gitpod.Dockerfile: Clean more
|
comment:121 Changed 6 months ago by
$ du -sh src local /usr /home /tmp /var/tmp 3.1G src 3.1G local 2.0G /usr 36K /home 31M /tmp 0 /var/tmp
comment:122 Changed 6 months ago by
I don't think it's necessary to remove packages such as emacs
that come with gitpod/workspace-base
.
comment:123 in reply to: ↑ 88 Changed 6 months ago by
Replying to mkoeppe:
Nice, also docker-in-docker works in the gitpod, so I can run commands such as
tox -e docker-ubuntu-bionic-minimal -- config.status
there
Too bad, this no longer works with the smaller base image because it does not have docker.
And sudo apt-get install docker.io
is not sufficient to fix it:
$ docker run -it ubuntu:bionic bash [...] docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown. ERRO[0008] error waiting for container: context canceled $ sudo docker run -it ubuntu:bionic bash docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown. ERRO[0001] error waiting for container: context canceled
comment:124 Changed 6 months ago by
- Description modified (diff)
comment:125 Changed 6 months ago by
Thanks, this seems to work reasonable well for me (still 4 mins, but well...).
One could still shave off 250mb by uninstalling emacs-nox, vim and python2.7 but that is not very significant in view of an 8GB image in total.
comment:126 Changed 6 months ago by
Ticket description misses verb in this sentence:
After this ticket is merged, we should the automatic prebuild using a github app
comment:127 Changed 6 months ago by
- Description modified (diff)
comment:128 Changed 6 months ago by
- Commit changed from f71c8b6c625017b0d37e956b0386111630ad6fe1 to f95322851c6494818bc08cd6fb3ce94e9c27debb
Branch pushed to git repo; I updated commit sha1. New commits:
f953228 | Merge tag '9.5.rc0' into t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflow
|
comment:129 Changed 6 months ago by
- Dependencies changed from #33068, #30933, #29285, #32868, #33067, #30677 to #30933, #29285
comment:130 follow-up: ↓ 132 Changed 6 months ago by
- Reviewers changed from Matthias Koeppe, ... to Matthias Koeppe, Dima Pasechnik
- Status changed from needs_review to positive_review
let's get this in, it seems to work, I can get ./sage
in console there, etc.
./sage -n
leads to problems, but probably that's not the way it's intendend.
comment:131 Changed 6 months ago by
Thanks for the review!
comment:132 in reply to: ↑ 130 Changed 6 months ago by
Replying to dimpase:
./sage -n
leads to problems, but probably that's not the way it's intendend.
There's no web browser in the container, so you can't use sage -n
.
To use Jupyter notebooks, use the Command Palette to run "Jupyter: Create New Jupyter Notebook", see
https://code.visualstudio.com/docs/datascience/jupyter-notebooks
comment:133 Changed 6 months ago by
- Commit changed from f95322851c6494818bc08cd6fb3ce94e9c27debb to 2d0902a1fee81d0749cc274fbec1d965cc522466
- Status changed from positive_review to needs_review
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:
2d0902a | src/doc/en/developer/workspace.rst: Fix markup
|
comment:134 Changed 6 months ago by
- Status changed from needs_review to positive_review
Fixes the PDF docbuild
comment:135 Changed 6 months ago by
- Commit changed from 2d0902a1fee81d0749cc274fbec1d965cc522466 to 08048c3e5d5d995cb5ebfd952d84eb204cb8c41c
- Status changed from positive_review to needs_review
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:
08048c3 | .github/workflows/tox.yml: Transform GITHUB_REF_NAME to a valid docker tag
|
comment:136 Changed 6 months ago by
- Commit changed from 08048c3e5d5d995cb5ebfd952d84eb204cb8c41c to 078f247c9a5ea383178499e9b0e990777f497a6e
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
078f247 | .github/workflows/tox*.yml: Transform GITHUB_REF_NAME to a valid docker tag
|
comment:137 Changed 6 months ago by
- Status changed from needs_review to positive_review
comment:138 Changed 6 months ago by
Is it easy to configure launching this (with prebuids) on branches in https://github.com/sagemath/sagetrac-mirror ?
comment:139 follow-up: ↓ 140 Changed 6 months ago by
Yes, it's only a few clicks see: https://www.gitpod.io/docs/prebuilds#projects-and-prebuilds However, it needs to be done by someone with rights to install apps in the sagemath org.
comment:140 in reply to: ↑ 139 Changed 6 months ago by
Replying to gh-tobiasdiez:
Yes, it's only a few clicks see: https://www.gitpod.io/docs/prebuilds#projects-and-prebuilds However, it needs to be done by someone with rights to install apps in the sagemath org.
I've sent you an invite. I guess I installed their "app", but I don't see how to proceed. (I think it should be enabled on the branch of this ticket only, before it's merged.)
Did you see my email message about trac access?
comment:141 follow-ups: ↓ 142 ↓ 143 Changed 6 months ago by
Thanks! I've now created a "team" sagemath and added the sage
and sagetrac-mirror
as projects. If you have a gitpod account and want to be invited to this team (in order to access the config), just post your email address and I send an invite.
Currently, the creation of the image fails however due to a permission error:
> [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev: ------ .gitpod.Dockerfile:3 -------------------- 1 | ARG BASE_GITHUB_REPOSITORY=sagemath/sage 2 | ARG BASE_TAG=dev 3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets 4 | RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/* 5 | # Fast doc rebuilds do not work because -------------------- error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden
Matthias, can you have a look please?
comment:142 in reply to: ↑ 141 Changed 6 months ago by
Replying to gh-tobiasdiez:
Thanks! I've now created a "team" sagemath and added the
sage
andsagetrac-mirror
as projects. If you have a gitpod account and want to be invited to this team (in order to access the config), just post your email address and I send an invite.
my email is shown on my GitHub account https://github.com/dimpase
Thanks!
comment:143 in reply to: ↑ 141 ; follow-up: ↓ 146 Changed 6 months ago by
Replying to gh-tobiasdiez:
Currently, the creation of the image fails however due to a permission error:
> [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev: ------ .gitpod.Dockerfile:3 -------------------- 1 | ARG BASE_GITHUB_REPOSITORY=sagemath/sage 2 | ARG BASE_TAG=dev 3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets 4 | RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/* 5 | # Fast doc rebuilds do not work because -------------------- error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden
I think you need to deposit a secret providing read access similar to what is explained in https://www.gitpod.io/docs/self-hosted/latest/configuration/docker-registry
comment:144 Changed 6 months ago by
... alternatively I suppose we could push the required Docker images to DockerHub? -- but I don't have write access there; @saraedum, could you deposit an access token for that as a secret here? https://github.com/organizations/sagemath/settings/secrets/actions
comment:145 Changed 6 months ago by
- Cc saraedum added
comment:146 in reply to: ↑ 143 ; follow-up: ↓ 147 Changed 6 months ago by
Replying to mkoeppe:
Replying to gh-tobiasdiez:
Currently, the creation of the image fails however due to a permission error:
> [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev: ------ .gitpod.Dockerfile:3 -------------------- 1 | ARG BASE_GITHUB_REPOSITORY=sagemath/sage 2 | ARG BASE_TAG=dev 3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets 4 | RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/* 5 | # Fast doc rebuilds do not work because -------------------- error: failed to solve: failed to fetch anonymous token: unexpected status: 403 ForbiddenI think you need to deposit a secret providing read access similar to what is explained in https://www.gitpod.io/docs/self-hosted/latest/configuration/docker-registry
That link is for a self-hosted gitpod and thus does not apply to our situation (we don't have a gitpod config other than what is committed in this branch).
I think the issue is that the image sage-docker-gitpod-standard-with-targets
is not yet published under sagemath/sage
. But how is this then working with your account Matthias?
comment:147 in reply to: ↑ 146 Changed 6 months ago by
Replying to gh-tobiasdiez:
the image
sage-docker-gitpod-standard-with-targets
is not yet published undersagemath/sage
. But how is this then working with your account Matthias?
Because on my testing branch used for the prebuild, I am setting the correct repo: https://github.com/mkoeppe/sage/blob/mkoeppe-9.5.beta9%2Bupdates/docker/.gitpod.Dockerfile
comment:148 Changed 6 months ago by
(And because every GH workflow automatically has the credentials to access its own GH packages (gchr.io)!)
comment:149 follow-up: ↓ 150 Changed 6 months ago by
Ahh okay, I thought this is an environment variable that is picked up somewhere.
So in this case we just have to wait until this ticket is merged, which would then build and publish the docker image to sagemath/sage
, right?
comment:150 in reply to: ↑ 149 Changed 6 months ago by
Replying to gh-tobiasdiez:
wait until this ticket is merged, which would then build and publish the docker image to
sagemath/sage
, right?
Yes, on the next release that includes this branch, the Docker image will be pushed (by separate instances of the workflow) both to gchr.io/sagemath/sage
and to gchr.io/sagemath/sagetrac-mirror
.
We will then see whether the 403 Forbidden will go away. I think it will not, for workflows run on github.com/sagemath/sagetrac-mirror
, because it may not have the read:packages
credential for gchr.io/sagemath/sage
.
comment:151 Changed 6 months ago by
comment:152 Changed 6 months ago by
Well, no, all packages are already set to "public": For example https://github.com/orgs/sagemath/packages/container/sage%2Fsage-docker-ubuntu-focal-standard-configured/settings
comment:153 Changed 6 months ago by
comment:154 Changed 6 months ago by
No, this makes no sense.
comment:155 Changed 5 months ago by
- Milestone changed from sage-9.5 to sage-9.6
comment:156 Changed 5 months ago by
- Priority changed from major to critical
comment:157 Changed 5 months ago by
- Branch changed from u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow to 078f247c9a5ea383178499e9b0e990777f497a6e
- Resolution set to fixed
- Status changed from positive_review to closed
Last 10 new commits:
sed -i.bak 's/ubunty/ubuntu/g' .github/workflows/*.yml
.github/workflows/tox.yml: Replace homebrew-macos-python3_xcode-standard by homebrew-macos-usrlocal-python3_xcode-standard
Merge #32703
tox.ini, .github/workflows: Remove ubuntu-groovy
Merge #33068
.github/workflows/tox.yml: Add docker-gitpod
build/bin/write-dockerfile.sh: Handle __SUDO
build/bin/write-dockerfile.sh: Use relative workdir 'sage'
.gitpod.yml, docker/.gitpod.Dockerfile: Reverse SAGE_LOCAL symlink
build/bin/write-dockerfile.sh: Handle $__CHOWN