gitpod integration using Docker images from portability testing workflow

[mkoeppe]

​Gitpod allows one to setup a complete dev environment in the cloud. It is free to use for up to 50 hours per month. In this ticket the config necessary for making this work with sagemath is added.

You can try this out by going to ​https://gitpod.io/#https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates.

It works by building docker images on github, push them to GH packages (​https://github.com/mkoeppe?tab=packages&q=gitpod) and then use these ones as the base for the gitpod workspace.

After this ticket is merged, we should activate the automatic prebuild by installing the github app (​https://www.gitpod.io/docs/prebuilds#on-github). With this enabled, every push would trigger a prebuild of the complete environment (including the build of all dependencies and cythonizion) so that one has a up-to-date code env very quickly without the need to build anything.

Follow-up: #33113 gitpod usability improvements

Alternative: #32749, which only builds the docker image on gitpod (which is currently not really feasible as they have a time limit of 1h)

[mkoeppe]

Last 10 new commits:

​e99a09e	sed -i.bak 's/ubunty/ubuntu/g' .github/workflows/*.yml
​edb4364	.github/workflows/tox.yml: Replace homebrew-macos-python3_xcode-standard by homebrew-macos-usrlocal-python3_xcode-standard
​e27e161	Merge #32703
​96a45d1	tox.ini, .github/workflows: Remove ubuntu-groovy
​59ec3d0	Merge #33068
​cbf03d1	.github/workflows/tox.yml: Add docker-gitpod
​4a9edfc	build/bin/write-dockerfile.sh: Handle __SUDO
​d697d94	build/bin/write-dockerfile.sh: Use relative workdir 'sage'
​c7cc5cd	.gitpod.yml, docker/.gitpod.Dockerfile: Reverse SAGE_LOCAL symlink
​bde9f74	build/bin/write-dockerfile.sh: Handle $__CHOWN

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​21afe51	.github/workflows/tox*.yml, tox.ini: Pass ddocker.pkg.github.com credentials via DOCKER_CONFIG_FILE
​76b5e76	Merge #30933

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7ef6a6e

docker/.gitpod.Dockerfile: Remove, use custom image from GH Packages instead

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6b41c4a

.github/workflows/tox.yml: Also push ...-with-targets-pre

[mkoeppe]

Running into pyenv issues (#29285) here too:

[pytest] installing. Log file: /home/gitpod/sage/logs/pkgs/pytest.log
  [pytest] error installing, exit status 1. End of log file:
  [pytest]   /home/gitpod/sage-local/var/lib/sage/venv-python3.8/bin/python3: No module named pip
  [pytest] Full log file: /home/gitpod/sage/logs/pkgs/pytest.log
make[4]: *** [Makefile:2756: pytest-no-deps] Error 1

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​376932f	build/bin/sage-bootstrap-python: Reject pyenv shims
​5f0c36c	Merge #29285

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​aeca1a0	Makefile (pypi-sdists): Add sage_setup
​cd3dce9	pkgs/sage-conf_pypi/setup.py: Update directory of configured sage_conf.py
​c695921	pkgs/sage-conf/sage_conf.py.in (SAGE_SPKG_WHEELS): Update so it works if SAGE_VENV != SAGE_LOCAL
​cebcda3	Merge tag '9.5.beta4' into t/29039/pip_installable_package_sage_bootstrap
​0026892	Merge tag '9.5.beta5' into t/29039/pip_installable_package_sage_bootstrap
​2a4a323	Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap
​02234ce	Merge #29039
​b1413d4	build/bin/write-dockerfile.sh: Move 'ADD pkgs' earlier
​859be3e	build/bin/write-dockerfile.sh: ADD src/*.m4 files for bootstrapping
​af7c8d3	Merge #33067

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fad8942

build/bin/write-dockerfile.sh: Use relative workdir 'sage' (again)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​601a3b5	.gitpod.yml: Remove pyenv workarounds
​1258c70	.gitpod.yml: configure --enable-download-from-upstream-url

[mkoeppe]

This works. To be done:

• Additional tags for the images, for example latest for releases, or branch/tag name (${GITHUB_REF_NAME} - ​https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables) - suitable for .gitpod.yml
• Could use docker/.gitpod.Dockerfile again with COPY --from to make the final image smaller (currently /home/gitpod/sage has the old source tree)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fdb49a1

tox.ini, .github/workflows/tox*.yml: Also tag the Docker images with GITHUB_REF_NAME

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ab757ba

.github/workflows/tox.yml: Fix passing of EXTRA_DOCKER_TAGS

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​96528eb

.github/workflows/tox*.yml: Tag docker images 'latest' or 'dev' when building a release tag

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5d1dc0f

.gitpod.yml: Use tag :dev

[mkoeppe]

Also the configuration of VS Code should be improved. It seems to start up using the wrong interpreter (not venv/bin/python3), and also one needs to manually select the right Jupyter kernel.

The scipy folks have some good documentation and configuration (​https://scipy.github.io/devdocs/dev/contributor/quickstart_gitpod.html), perhaps we can lift something from there.

[gh-tobiasdiez]

Is the prebuild working for this branch?

How can one work with branches that update sage packages? With #32749 one could trigger a rebuild of the docker image (and thus of the system packages) using the #imagebuild context. Can something like this also be implemented here? For example, take the github action build docker as base, and then re-install system or python packages on top of this if they have newer versions?

For the VS integration, #30677 is closed but not yet merged into develop.

[mkoeppe]

Replying to gh-tobiasdiez:

Is the prebuild working for this branch?

I've just merged #33071 into the branch ​https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates to try this out.

How can one work with branches that update sage packages?

Nothing special, make just rebuilds what needs rebuilding.

[mkoeppe]

Replying to gh-tobiasdiez:

For the VS integration, #30677 is closed but not yet merged into develop.

I forgot to say that the branch ​https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates already contains #30677 and other updates

[gh-tobiasdiez]

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Is the prebuild working for this branch?

I've just merged #33071 into the branch ​https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates to try this out.

How can one work with branches that update sage packages?

Nothing special, make just rebuilds what needs rebuilding.

So this would then be done as part of the prebuild?

[mkoeppe]

Yes, I think so

[mkoeppe]

Looks like I'll have to add some configuration so that it does prebuilds on all branches

[mkoeppe]

I created a "project" and now it looks like the prebuild triggered by a push is running - ​https://gitpod.io/projects/sage

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3e28e13

.gitpod.yml: Set github prebuild options to true

[mkoeppe]

Before the push I had merged #32968 with the Sphinx update; and the prebuild has just installed the upgraded package

[gh-tobiasdiez]

Replying to mkoeppe:

Before the push I had merged #32968 with the Sphinx update; and the prebuild has just installed the upgraded package

That's very nice!

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​49575f5

.gitpod.yml: Switch to image ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev

[mkoeppe]

The prebuild was just killed because of the 35m timeout. What's the best choice of timeout according to your experiments?

[gh-tobiasdiez]

If its really the timeout that killed the prebuild (in which the prebuild would be green and be used for the next workspace, but maybe doesn't contain all packages) then you can go to 50min. Maybe a bit less since the docker image is quite big so this part may take some time. But essentially you have to calculate 60min - bootstrap and config (2-3min) - docker init and copying (?? min).

However, for me the prebuild often timed out after 1h (then its shown red in the project view). The only solution I've found for this is not using symlinks into /home/.

[mkoeppe]

​https://gitpod.io/projects/sage/e8780c03-b8b8-4810-a772-d4cc2994b6bf shows

[sagelib-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sagelib-9.5.beta9.log
  [pytest] successfully installed.
  [sphinx-4.3.1] successfully installed.
make[1]: *** [Makefile:39: all-build] Terminated
make: *** [Makefile:16: build] Terminated
exit

but it is still claims "RUNNING: Prebuild in progress"

[mkoeppe]

However, ​https://gitpod.io/projects/sage/prebuilds shows READY.

[mkoeppe]

🤙 This task ran as a workspace prebuild
🎉 Well done on saving 37 minutes

[gh-tobiasdiez]

A few more points:

• please use the recent version of gitpod.yml from ​https://github.com/tobiasdiez/sagetrac-mirror/blob/develop/.gitpod.yml (the ssh stuff and num threads = 8)
• --prefix=$HOME/sage-local in gitpod.yml should probably be /workspace/sage-local or a few lines above $HOME should be used?
• probably 8 parallel tasks are fine. gitpod gives 6 (or 7?) vCPUs to users.

[gh-tobiasdiez]

Replying to mkoeppe:

🤙 This task ran as a workspace prebuild
🎉 Well done on saving 37 minutes

Nice, then you can go to 50min indeed.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e7d4e96

.gitpod.yml: Increase timeout for make in prebuild, increase parallelism

[mkoeppe]

Presumably the before script should try to resume the build in case the prebuild timed out.

[gh-tobiasdiez]

I agree. But init is called after before during the prebuild, and I couldn't find a way to detect if one is in a prebuild or not...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fdb332d

.gitpod.yml: Run 'make' again in 'before' task if prebuild 'make' timed out

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​aaa9777

.gitpod.yml: Use SAGE_NUM_THREADS=8

[mkoeppe]

Replying to gh-tobiasdiez:

• please use the recent version of gitpod.yml from ​https://github.com/tobiasdiez/sagetrac-mirror/blob/develop/.gitpod.yml (the ssh stuff and num threads = 8)

I've updated it - the ssh stuff was already same

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​87900a7

.gitpod.yml: Refer to /home/gitpod as /Users/mkoeppe

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0864c6e

.github/workflows/tox*.yml: Push stable releases also to Docker tag 'dev'

[mkoeppe]

​https://gitpod.io/projects/sage/134676a9-2a0e-429a-816d-68ad30441f58

[sphinx-4.3.1] installing. Log file: /workspace/sage/logs/pkgs/sphinx-4.3.1.log
[sagelib-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sagelib-9.5.beta9.log
  [pytest] successfully installed.
  [sphinx-4.3.1] successfully installed.
  [sagelib-9.5.beta9] successfully installed.
make --no-print-directory sage_docbuild-SAGE_VENV-no-deps
[sage_docbuild-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sage_docbuild-9.5.beta9.log
  [sage_docbuild-9.5.beta9] successfully installed.
make[2]: Leaving directory '/workspace/sage/build/make'

real    31m8.684s
user    101m44.392s
sys     3m47.093s
SAGE_CHECK=warn, so scanning the log files. This may take a few seconds.
Sage build/upgrade complete!
make[1]: Leaving directory '/workspace/sage'
exit

🤙 This task ran as a workspace prebuild
🎉 Well done on saving 34 minutes

[mkoeppe]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​fdb332d

.gitpod.yml: Run 'make' again in 'before' task if prebuild 'make' timed out

This worked but for some reason the Sphinx package installation was repeated in this step. It's fast but I'll try if refreshing the timestamps of the install records will avoid it.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​36e0c54	.gitpod.yml (before): Set timestamps of installation records to same mtime
​c9fa015	.gitpod.yml: Use /workspace/sage/local as physical location of SAGE_LOCAL

[mkoeppe]

There are some weird synchronization issues on the server side - it seems that if one creates a workspace right after a prebuild has completed, it can happen that it does not use the prebuild.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​38dafc6

.gitpod.yml (before): Refer to /workspace/sage/local by full path

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e6be067

.gitpod.yml (init): Make sure to remove the directory /home/gitpod/sage-local

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cdcf2ac

.gitpod.yml (before): establish SAGE_LOCAL here, not in 'init'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5b44b6b	.gitpod.yml: Remove install-record timestamping
​f58453d	.gitpod.yml: Also remove the source tree used by the Docker build

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​aef75eb

.gitpod.yml (before): Always remove the source tree used by the Docker build

[mkoeppe]

OK, now it's in good shape.

Could you check that the SSH stuff works as intended and matches the added documentation? I see

$ ls -l ~/.ssh
total 24
-rw-r--r-- 1 gitpod gitpod   381 Jan  4 08:53 authorized_keys
-rw------- 1 gitpod gitpod     0 Jan  4 08:53 id_rsa
-rw-r--r-- 1 gitpod gitpod   664 Jan  4 08:53 known_hosts
-rw-r--r-- 1 gitpod gitpod 15544 Jan  4 08:53 supervisor_env

and am not sure what the empty id_rsa is intended for

[gh-tobiasdiez]

​https://github.com/sagemath/sagetrac-mirror/compare/develop...u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow#diff-370a022e48cb18faf98122794ffc5ce775b2606b09a9d1f80b71333425ec078eR27

There you don't need/want the timeout.

[gh-tobiasdiez]

​https://github.com/sagemath/sagetrac-mirror/compare/develop...u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow#diff-370a022e48cb18faf98122794ffc5ce775b2606b09a9d1f80b71333425ec078eR17

I'm not sure if I understand the logic behind this and related commands (maybe add a bit more documentation of what's going on?). But please be aware that the code may be in a different location such as /workspace/sagetrac-mirror if someone starts gitpod from ​https://github.com/sagemath/sagetrac-mirror (which is probably what's going to happen if you want to check someone elses branch)

[gh-tobiasdiez]

Replying to mkoeppe:

Could you check that the SSH stuff works as intended and matches the added documentation? and am not sure what the empty id_rsa is intended for

Did you added a PRIVATE_SSH_KEY env variable in gitpod with your trac ssh as described in the docs? It's working for me.

[gh-tobiasdiez]

Replying to mkoeppe:

OK, now it's in good shape.

For me it takes over 5min to start a workspace due to the large workspace image (it displays "fetching build image" or something like this in the loading screen). I think this should be optimized as part of this ticket since otherwise I fear people will not adopt gitpod for e.g. reviewing of tickets.

Also it still loads pyenv: pyenv shell 3.8.12 so ​https://github.com/tobiasdiez/sagetrac-mirror/blob/20ca179bed4ff605de4111362b02c8e2acca617e/.gitpod.yml#L18-L19 should probably be readded.

[gh-tobiasdiez]

​https://github.com/sagemath/sagetrac-mirror/compare/develop...u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow#diff-ef2cef9f88b4fe09ca3082140e67f5ad34fb65fb6e228f119d3812261ae51449R438-R442

Is the somewhat strange indention by design?

[mkoeppe]

Replying to gh-tobiasdiez:

Also it still loads pyenv: pyenv shell 3.8.12 so ​https://github.com/tobiasdiez/sagetrac-mirror/blob/20ca179bed4ff605de4111362b02c8e2acca617e/.gitpod.yml#L18-L19 should probably be readded.

It no longer breaks the Sage build since #29285, but I'll add it anyway

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6215a9f	Revert ".gitpod.yml (before): Refer to /workspace/sage/local by full path"
​cec1f44	.gitpod.yml (before): No timeout for 'before' task that runs after the prebuild
​fa237c7	.gitpod.yml (before): Disable pyenv

[mkoeppe]

Replying to gh-tobiasdiez:

​https://github.com/sagemath/sagetrac-mirror/compare/develop...u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow#diff-ef2cef9f88b4fe09ca3082140e67f5ad34fb65fb6e228f119d3812261ae51449R438-R442

Is the somewhat strange indention by design?

The link does not show a particular line -- what are you referring to?

[mkoeppe]

Replying to gh-tobiasdiez:

please be aware that the code may be in a different location such as /workspace/sagetrac-mirror if someone starts gitpod from ​https://github.com/sagemath/sagetrac-mirror

Thanks, done.

[mkoeppe]

Replying to gh-tobiasdiez:

Replying to mkoeppe:

Could you check that the SSH stuff works as intended and matches the added documentation? and am not sure what the empty id_rsa is intended for

Did you added a PRIVATE_SSH_KEY env variable in gitpod with your trac ssh as described in the docs? It's working for me.

Thanks for checking - I had missed a step.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8d32070

.gitpod.yml: Add comments

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7b9eccf	.gitpod.yml: Fix inconsistent indentation
​081de0e	.gitpod.yml: Log to logs/install.log

[gh-tobiasdiez]

Replying to mkoeppe:

Replying to gh-tobiasdiez:

​https://github.com/sagemath/sagetrac-mirror/compare/develop...u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow#diff-ef2cef9f88b4fe09ca3082140e67f5ad34fb65fb6e228f119d3812261ae51449R438-R442

Is the somewhat strange indention by design?

The link does not show a particular line -- what are you referring to?

Strange, normally that works. It's the gitpod related changes in tox.ini (l. 438)

[mkoeppe]

Replying to gh-tobiasdiez:

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Is the somewhat strange indention by design?

the gitpod related changes in tox.ini (l. 438)

It is indented as intended. This file aspires to be a spreadsheet.

[mkoeppe]

There are still some minor details wrong. In logs/sphinx-4.3.1.log:

Uninstalling existing 'sphinx'
Warning: File '/home/gitpod/sage-local/var/lib/sage/wheels/Sphinx-4.2.0-py3-none-any.whl' not found
Warning: Directory '/home/gitpod/sage-local/var/lib/sage/wheels' not found
Running pip-uninstall script for 'sphinx'
/home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/scripts/sphinx/spkg-piprm: line 14: /home/gitpod/sage/build/bin/sage-dist-helpers: No such file or directory
Error: failed to source /home/gitpod/sage/build/bin/sage-dist-helpers
Is /home/gitpod/sage the correct SAGE_ROOT?
Warning: Error running the pip-uninstall script for 'sphinx'; uninstallation may have left behind some files
Removing stamp file '/home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/installed/sphinx-4.2.0'

[mkoeppe]

Package-removal scripts ({prefix,venv}/var/lib/sage/scripts/*/spkg-piprm) hardcode the SAGE_ROOT and SAGE_SRC from the time when the package was installed (the Docker-build phase); so I will also put a symlink in place for that.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​952bb4e	.gitpod.yml (before): Create symlink from /home/gitpod/sage to the workspace
​9f89af7	build/bin/sage-spkg: Run sage-spkg-uninstall on the correct tree

[mkoeppe]

Fixed now, logs/sphinx-4.3.1.log now shows:

Found existing installation: Sphinx 4.2.0
Uninstalling Sphinx-4.2.0:
  Successfully uninstalled Sphinx-4.2.0
Removing stamp file '/home/gitpod/sage-local/var/lib/sage/venv-python3.8/var/lib/sage/installed/sphinx-4.2.0'
Installing sphinx-4.3.1

and also venv/var/lib/sage/wheels/ now only contains one sphinx wheel.

[mkoeppe]

Nice, also docker-in-docker works in the gitpod, so I can run commands such as tox -e docker-ubuntu-bionic-minimal -- config.status there

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0eadcd1	Add VS Code config for SAGE_ROOT/venv
​075b22c	Merge #30677
​15e112a	.vscode/settings.json: Update interpreter settings

[mkoeppe]

Now it starts with the right interpreter, and also "Jupyter: Create New Jupyter Notebook" works - "Select Kernel" offers the Sage kernel.

[gh-tobiasdiez]

It still takes over 8min until the workspace is loaded for me due to "Pulling container image". This should really optimized as part of this ticket.

Also pyenv shell still seems to be activated, I think it needs to be disabled in before and init.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2bf20ed

.github/workflows/tox.yml (docker): Also push -with-system-packages

[mkoeppe]

Ready now in ​https://github.com/mkoeppe?tab=packages&q=gitpod

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c6edf67	.gitpod.yml: Use a Dockerfile again
​cb678a8	docker/.gitpod.Dockerfile: Go through ARGs BASE_GITHUB_REPOSITORY, BASE_TAG

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e01dd83

.gitpod.yml (before): Run 'pyenv global system'

[gh-tobiasdiez]

That sadly doesn't solve the issue for me. Starting a workspace still takes 8+ mins, while #32749 takes about 2mins.

It also seems that the workspace image is quite large at around 16GB. In particular, the sage folder contains

4.9G    local
6.2G    src

I'm not sure if the following folders are really necessary or if they can be deleted (latest at the end of the prebuild):

1.8G    ./local/share/doc
4.3G    ./src/build

Moreover,

3.0G    /usr
3.3G    /home

is also bigger than the one from the other image of #32749

2.2G    /usr
2.3G    /home

So I suggest to use gitpod/workspace-base as a base.

[mkoeppe]

Replying to gh-tobiasdiez:

Starting a workspace still takes 8+ mins

Probably depends on the data center region. I just timed it here, it took < 3.5 minutes; but there is considerable variance

[mkoeppe]

Replying to gh-tobiasdiez:

I'm not sure if the following folders are really necessary or if they can be deleted (latest at the end of the prebuild):

1.8G    ./local/share/doc
4.3G    ./src/build

I'll check if we can delete them without sacrificing fast rebuilds of documentation (make doc-html) and sagelib (./sage -b).

For documentation, I see

[sagemath_doc_html-none] make doc-inventory--reference-references
[sagemath_doc_html-none] cd /workspace/sage && ./sage --docbuild --no-pdf-links reference/references inventory 
[sagemath_doc_html-none] [reference] loading pickled environment... failed
[sagemath_doc_html-none] [reference] failed: source directory has changed

... so something is not ideal there anyway

[gh-tobiasdiez]

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Starting a workspace still takes 8+ mins

Probably depends on the data center region. I just timed it here, it took < 3.5 minutes; but there is considerable variance

Apparently, images are built in the US and then downloaded every time from there without caching. This explains why it takes so much longer for me. Gitpod is working on a solution ​https://github.com/gitpod-io/gitpod/issues/7002 and ​https://github.com/gitpod-io/gitpod/issues/6145. But having a 15GB+ image is not ideal anyway.

[mkoeppe]

top-level Makefile has targets micro_release, fast-rebuild-clean that do related stuff.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4814334	docker/.gitpod.Dockerfile: Remove doctrees/inventory
​874dddd	tox.ini (gitpod): Use gitpod/workspace-base by default

[mkoeppe]

Replying to gh-tobiasdiez:

3.0G    /usr
3.3G    /home

is also bigger than the one from the other image of #32749

2.2G    /usr
2.3G    /home

So I suggest to use gitpod/workspace-base as a base.

Building a new image at ​https://github.com/mkoeppe/sage/runs/4731195747

[mkoeppe]

fast-rebuild-clean: misc-clean
	rm -rf upstream/
	rm -rf build/pkgs/sagelib/src/build/temp.*
	# The .py files in src/build are restored from src/sage without their
	# mtimes changed.
	-find build/pkgs/sagelib/src/build -name '*.py' -exec rm \{\} \;
	# Remove leftovers from ancient branches
	rm -rf src/build

This is not compatible with configure --enable-editable, which uses src/build/ instead of build/pkgs/sagelib/src/build/.

[gh-tobiasdiez]

Why does the editable install actually needs src/build or even copies something there? The compiled cython code (.so files) are placed next to the cython file.

[mkoeppe]

I think that's just standard setuptools behavior

[mkoeppe]

The setuptools option --build-temp could be used to put the temp directories outside of /workspace.

Or we could just unconditionally remove the temp directory at the end of build/pkgs/sagelib/spkg-install.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c97dc01

build/pkgs/sagelib/spkg-install: Remove build/temp.* directories after build

[mkoeppe]

It's much smaller now

gitpod /workspace/sage (mkoeppe-9.5.beta9+updates) $ du -sh src local /usr /home
3.1G    src
3.9G    local
2.4G    /usr
76K     /home

and rebuilds are still fast

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3605076

build/pkgs/sagelib/spkg-install: Fix up deleting temp.*

[gh-tobiasdiez]

Thanks, its way better now (around 4-5min). I would say the doc folder can still be removed. If someone is working on the docs, then one can easily only create the relevant file without the need to build the whole docs.

[gh-tobiasdiez]

The following folders should be deleted as well

212M    /tmp
32M     /var/lib/apt/lists (remove as part of the apt-get install command)

A few packages that I think are not used by the built script or needed in general:

3722    python2.7-minimal
9370    libflint-dev
10026   pari-gp (and other pari-related stuff)
13215   ecl
14279   libsingular4-dev (and other singular-related stuff)
19531   libgiac0 (and other giac-related stuff)
30765   vim-runtime
41291   emacs-nox
66816   emacs-common
330712  libgl1-mesa-dri (?)

I would suggest to exclude them from the installation / uninstall them, similar to what is done in #32749.

Together that should give another 1GB.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d87027c	sage --package list: Add option --exclude
​c02c03c	tox.ini (gitpod-standard): Exclude some packages

[mkoeppe]

New image building at ​https://github.com/mkoeppe/sage/runs/4742213151

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6285cd9	.gitpod.Dockerfile: Save space by removing apt lists
​7e74607	.gitpod.Dockerfile: Remove built Sage documentation

[gh-tobiasdiez]

For the record, that's how gitpod is cleaning up the package installation cache/log: ​https://github.com/gitpod-io/workspace-images/blob/master/base/install-packages#L32-L36

[gh-tobiasdiez]

Replying to mkoeppe:

New image building at ​https://github.com/mkoeppe/sage/runs/4742213151

Thanks! Will try it once its finished.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f71c8b6

docker/.gitpod.Dockerfile: Clean more

[mkoeppe]

$ du -sh src local /usr /home /tmp /var/tmp
3.1G    src
3.1G    local
2.0G    /usr
36K     /home
31M     /tmp
0       /var/tmp

[mkoeppe]

I don't think it's necessary to remove packages such as emacs that come with gitpod/workspace-base.

[mkoeppe]

Replying to mkoeppe:

Nice, also docker-in-docker works in the gitpod, so I can run commands such as tox -e docker-ubuntu-bionic-minimal -- config.status there

Too bad, this no longer works with the smaller base image because it does not have docker.

And sudo apt-get install docker.io is not sufficient to fix it:

$ docker run -it ubuntu:bionic bash
[...]
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.
ERRO[0008] error waiting for container: context canceled 

$ sudo docker run -it ubuntu:bionic bash
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.
ERRO[0001] error waiting for container: context canceled 

[gh-tobiasdiez]

Thanks, this seems to work reasonable well for me (still 4 mins, but well...).

One could still shave off 250mb by uninstalling emacs-nox, vim and python2.7 but that is not very significant in view of an 8GB image in total.

[slelievre]

Ticket description misses verb in this sentence:

After this ticket is merged, we should the automatic prebuild using a github app

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f953228

Merge tag '9.5.rc0' into t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflow

[dimpase]

let's get this in, it seems to work, I can get ./sage in console there, etc. ./sage -n leads to problems, but probably that's not the way it's intendend.

[gh-tobiasdiez]

Thanks for the review!

[mkoeppe]

Replying to dimpase:

./sage -n leads to problems, but probably that's not the way it's intendend.

There's no web browser in the container, so you can't use sage -n. To use Jupyter notebooks, use the Command Palette to run "Jupyter: Create New Jupyter Notebook", see ​https://code.visualstudio.com/docs/datascience/jupyter-notebooks

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​2d0902a

src/doc/en/developer/workspace.rst: Fix markup

[mkoeppe]

Fixes the PDF docbuild

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​08048c3

.github/workflows/tox.yml: Transform GITHUB_REF_NAME to a valid docker tag

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​078f247

.github/workflows/tox*.yml: Transform GITHUB_REF_NAME to a valid docker tag

[dimpase]

Is it easy to configure launching this (with prebuids) on branches in ​https://github.com/sagemath/sagetrac-mirror ?

[gh-tobiasdiez]

Yes, it's only a few clicks see: ​https://www.gitpod.io/docs/prebuilds#projects-and-prebuilds However, it needs to be done by someone with rights to install apps in the sagemath org.

[dimpase]

Replying to gh-tobiasdiez:

Yes, it's only a few clicks see: ​https://www.gitpod.io/docs/prebuilds#projects-and-prebuilds However, it needs to be done by someone with rights to install apps in the sagemath org.

I've sent you an invite. I guess I installed their "app", but I don't see how to proceed. (I think it should be enabled on the branch of this ticket only, before it's merged.)

Did you see my email message about trac access?

[gh-tobiasdiez]

Thanks! I've now created a "team" sagemath and added the sage and sagetrac-mirror as projects. If you have a gitpod account and want to be invited to this team (in order to access the config), just post your email address and I send an invite.

Currently, the creation of the image fails however due to a permission error:

 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

Matthias, can you have a look please?

​https://gitpod.io/#prebuild/https://github.com/sagemath/sagetrac-mirror/tree/u/mkoeppe/gitpod_integration_using_docker_images_from_portability_testing_workflow

[dimpase]

Replying to gh-tobiasdiez:

Thanks! I've now created a "team" sagemath and added the sage and sagetrac-mirror as projects. If you have a gitpod account and want to be invited to this team (in order to access the config), just post your email address and I send an invite.

my email is shown on my GitHub account ​https://github.com/dimpase

Thanks!

[mkoeppe]

Replying to gh-tobiasdiez:

Currently, the creation of the image fails however due to a permission error:

 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

I think you need to deposit a secret providing read access similar to what is explained in ​https://www.gitpod.io/docs/self-hosted/latest/configuration/docker-registry

[mkoeppe]

... alternatively I suppose we could push the required Docker images to DockerHub? -- but I don't have write access there; @saraedum, could you deposit an access token for that as a secret here? ​https://github.com/organizations/sagemath/settings/secrets/actions

[gh-tobiasdiez]

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Currently, the creation of the image fails however due to a permission error:

 > [internal] load metadata for ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev:
------
.gitpod.Dockerfile:3
--------------------
   1 |     ARG BASE_GITHUB_REPOSITORY=sagemath/sage
   2 |     ARG BASE_TAG=dev
   3 | >>> FROM ghcr.io/${BASE_GITHUB_REPOSITORY}/sage-docker-gitpod-standard-with-targets:${BASE_TAG} as with-targets
   4 |     RUN sudo rm -rf /var/cache/debconf/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
   5 |     # Fast doc rebuilds do not work because
--------------------
error: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

I think you need to deposit a secret providing read access similar to what is explained in ​https://www.gitpod.io/docs/self-hosted/latest/configuration/docker-registry

That link is for a self-hosted gitpod and thus does not apply to our situation (we don't have a gitpod config other than what is committed in this branch).

I think the issue is that the image sage-docker-gitpod-standard-with-targets is not yet published under sagemath/sage. But how is this then working with your account Matthias?

[mkoeppe]

Replying to gh-tobiasdiez:

the image sage-docker-gitpod-standard-with-targets is not yet published under sagemath/sage. But how is this then working with your account Matthias?

Because on my testing branch used for the prebuild, I am setting the correct repo: ​https://github.com/mkoeppe/sage/blob/mkoeppe-9.5.beta9%2Bupdates/docker/.gitpod.Dockerfile

[mkoeppe]

(And because every GH workflow automatically has the credentials to access its own GH packages (gchr.io)!)

[gh-tobiasdiez]

Ahh okay, I thought this is an environment variable that is picked up somewhere.

So in this case we just have to wait until this ticket is merged, which would then build and publish the docker image to sagemath/sage, right?

[mkoeppe]

Replying to gh-tobiasdiez:

wait until this ticket is merged, which would then build and publish the docker image to sagemath/sage, right?

Yes, on the next release that includes this branch, the Docker image will be pushed (by separate instances of the workflow) both to gchr.io/sagemath/sage and to gchr.io/sagemath/sagetrac-mirror.

We will then see whether the 403 Forbidden will go away. I think it will not, for workflows run on github.com/sagemath/sagetrac-mirror, because it may not have the read:packages credential for gchr.io/sagemath/sage.

[mkoeppe]

I think we may need to do this: ​https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#configuring-visibility-of-container-images-for-an-organization

[mkoeppe]

Well, no, all packages are already set to "public": For example ​https://github.com/orgs/sagemath/packages/container/sage%2Fsage-docker-ubuntu-focal-standard-configured/settings

[mkoeppe]

But perhaps this is needed: ​https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#github-actions-access-for-organization-owned-container-images

[mkoeppe]

No, this makes no sense.