py3: Docker images for python3-based Sage

[galois]

Docker images for python3-based Sage are useful in general, and in particular for continuous integration or other testing workflows for packages that depend on Sage.

The infrastructure introduced by #24655 automatically produces Docker images for python2-based Sage. In order to extend it to also produce python3-based ones,

• we update the micro_release target for make in Makefile to not remove the config directory

• we adapt .gitlab-ci.yml and docker/Dockerfile, duplicating all the docker targets for py3, creating a WITH_PYTHON and a DOCKER_TAG_SUFFIX environment variables, and using them in the shell scripts docker/*.sh.

To play with the generated Python 2/3 docker images, you can go to ​https://gitlab-hooks-flau3jeaza-ew.a.run.app/status/trac/branch/u%2Fsaraedum%2F28041

[slelievre]

Still need to make the changes to the docker/*.sh shell scripts.

[embray]

I'm trying to get this working, but the build is stalling out at:

Step 61/80 : RUN sudo $SAGE_ROOT/sage --nodotsage -c "install_scripts('/usr/bin')"
 ---> Running in d4ce02ca02d6
Forcing sage-location, probably because a new package was installed.
Cleaning up, do not interrupt this.
Done cleaning.
Traceback (most recent call last):
  File "/home/sage/sage/src/bin/sage-eval", line 4, in <module>
    from sage.all import *
ImportError: No module named sage.all
The command '/bin/sh -c sudo $SAGE_ROOT/sage --nodotsage -c "install_scripts('/usr/bin')"' returned a non-zero code: 1

The problem is that SAGE_PYTHON_VERSION is not being set properly when running $SAGE_ROOT/sage.

This is because there are two copies of sage-env-config, where SAGE_PYTHON_VERSION is defined: One under $SAGE_ROOT/src/bin and one other $SAGE_ROOT/local/bin. This is normal when doing Sage development, and I believe, if it exists, it's the one under src/bin that will be used by default when developing/building.

The problem in this case is that although the one in local/bin correctly has SAGE_PYTHON_VERSION=3, the one in src/bin still has SAGE_PYTHON_VERSION=2 for some reason.

I wonder if this is a remnant from an earlier build stage that was cached, but not properly updated for some reason (I thought that different --build-arg flags would avoid use incompatible cached images built with different --build-args, but it's possible that that arg isn't being included in the build early enough.

[embray]

The problem seems to start here:

Step 56/80 : RUN make micro_release
 ---> Using cache
 ---> 2fc1a353e2ac

When running make micro_release, ./configure gets re-run, but without any flags we passed it previously. So it doesn't preserve our --with-python=3 from earlier.

[embray]

(CC Odile who was interested in having a working Python 3 Docker image)

[saraedum]

Sorry for this. We had already figured the broken ./configure when micro_release is run. I thought that Samuel had pushed a fix for this here already.

[saraedum]

Isn't the preservation of config good enough for this? (See the changeset here.)

[embray]

I was just about to say: I think the preservation of config.status is good enough in fact.

[embray]

Specifically, one of the prereqs of micro_release is the bdist-clean target, which in turn calls $(MAKE) misc-clean which deletes config.status, forcing configure to be re-run.

[saraedum]

What the bdist-clean? Is this a automake target? Otherwise is it reasonable that the build system's config is visible after running it?

[embray]

Not sure, but I think just swapping sagelib-clean and bdist-clean in the prereqs for micro_release should suffice. It's fragile, but I think it will work. Testing now.

[embray]

Replying to saraedum:

What the bdist-clean? Is this a automake target? Otherwise is it reasonable that the build system's config is visible after running it?

No, this is just a pseudo-target in Sage's top-level Makefile. I have no idea why specifically it's named that. It's also used by the fast-rebuild-clean target. It just calls clean (which is in build/make/Makefile for some reason), and then misc-clean.

All clean does apparently is:

clean:
    @echo "Deleting package build directories..."
    rm -rf "$(SAGE_LOCAL)/var/tmp/sage/build"

which is certainly worth doing. But maybe this needs to be detangled a bit...

[slelievre]

The following files in docker/ probably still need changes:

• build-docker.sh
• pull-gitlab.sh
• push-dockerhub.sh
• push-gitlab.sh
• setup-make-parallelity.sh
• test-dev.sh
• update-env.sh

[embray]

Yes, most likely. One I just get the docker build working locally I will try adding all the other relevant updates.

It would be nice for starters to get a Python 3 image on Docker Hub, even if I have to push it manually, then get future ones building automatically.

[saraedum]

I think that once your Dockerfile goes through, you just need to add your branch to build-from-clean, pull your image from the GitLab registry and push it to Docker Hub.

[embray]

For Odile's use I went ahead and pushed a prototype of this up to Dockerhub: sagemath/sagemath-dev-py3:alpha

Unfortunately it still has a bug. The sage-entrypoint still tries to run make build (understandably) which results in re-configuring (needlessly) and breaking things.

A workaround for now is to set the PREREQ_OPTIONS variable which is used by Sage's top-level Makefile when it runs configure. So you can set this when running the container like:

docker run -ti --rm -e PREREQ_OPTIONS='--with-python=3' sagemath/sagemath-dev-py3:alpha

Next up, I think we should modify the Docker build to at least keep config.status so that this doesn't keep happening.

[embray]

Apparently I was not able to successfully upload the Docker image to hub.docker.com because (AFAICT) of ​issue.

But I was able to upload it to our image registry on gitlab.com. So try:

docker run -ti --rm -e PREREQ_OPTIONS='--with-python=3' registry.gitlab.com/sagemath/dev/sage/sagemath-dev-py3:alpha

You can also use this image as a base for another Docker image by doing something like:

FROM registry.gitlab.com/sagemath/dev/sage/sagemath-dev-py3:alpha
ENV PREREQ_OPTIONS='--with-python=3'
...

That is, you would ensure PREREQ_OPTIONS is set as an environment variable in the new image, by default.

This is not ideal and should be fixed, but it's a start anyways...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2f108ac

Build py3 from clean

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f5ba6d3

Fix VARIABLES of Python3 builds

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a21467a

Limit docker build output

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6619e31

Limit docker build output

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​497077d

Merge latest beta into docker-py3

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​155e3ac

Build docker in parallel

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7c9ea55

Build docker in parallel

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d36ffe1

Push Python 3 docker images to *-py3

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a229ffc

Undo docker changes

[saraedum]

See #28367 for the related README update.

[saraedum]

We pushed initial docker -py3 images to the Docker Hub.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​22da087

do not remove config.status from dev image

[saraedum]

New commits:

​22da087

do not remove config.status from dev image

[embray]

When this is for sure ready can we squash some of these commits?

[embray]

Just out of curiosity, is there a specific meaning to having a section in .gitlab-ci.yml that begins with a ., as in .py3? Does that just emphasize that it's not a real build stage (rather a "mix-in" for multiple build stages)? Or is that a syntax specifically recognized by the file format?

If I understand correctly, this is now keeping config.status in the image. But what if someone runs sage -i <some_optional_package> in one of these py3 images? In other words, if a user (or dockerfile author) does something that still results in ./configure being re-run, will this keep the --with-python=3 setting?

I fear not, since we still haven't fully resolved #27373 (which related to system package settings specifically, but more broadly to retaining the values of flags passed to ./configure). The larger problem of course being that we have "runtime" features like sage -i that depend on running ./configure at all...

[saraedum]

Replying to embray:

When this is for sure ready can we squash some of these commits?

Feel free to squash.

[saraedum]

Replying to embray:

Just out of curiosity, is there a specific meaning to having a section in .gitlab-ci.yml that begins with a ., as in .py3? Does that just emphasize that it's not a real build stage (rather a "mix-in" for multiple build stages)? Or is that a syntax specifically recognized by the file format?

Without out the . it would be a build job. As a default as port of the test stage I believe.

If I understand correctly, this is now keeping config.status in the image. But what if someone runs sage -i <some_optional_package> in one of these py3 images? In other words, if a user (or dockerfile author) does something that still results in ./configure being re-run, will this keep the --with-python=3 setting?

Does sage -i run configure when config.status is present?

I fear not, since we still haven't fully resolved #27373 (which related to system package settings specifically, but more broadly to retaining the values of flags passed to ./configure). The larger problem of course being that we have "runtime" features like sage -i that depend on running ./configure at all...

I don't think making --with-python=3 persistent is within the scope of this ticket. But it's certainly a problem that we should be aware of.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c3d9c14

Keep misc-clean around longer in Makefile

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5431987

Merge remote-tracking branch 'trac/develop' into 28041

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​45268e7	fix grammar
​6e28954	Cleanup py-3 script
​cb61f30	Merge remote-tracking branch 'trac/develop' into 28041

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​9a9dd58

Merge remote-tracking branch 'trac/develop' into 28041

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​89c78f8

Cleanup py-3 script

[saraedum]

the docker images should be available on binder shortly: ​https://gitlab.com/sagemath/dev/trac/pipelines/81996545

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1d02785

Add a test-hook for build-from-clean

[saraedum]

I started a full build-from-clean here: ​https://gitlab.com/sagemath/dev/trac/pipelines/82005383

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ca24332

Do not run build-from-latest for explicit builds

[saraedum]

The Python 2 and Python 3 images seem to be functional. (At least they work in Binder.)

[roed]

Looks good to me. Both Python 2 and Python 3 built fine.

[saraedum]

Not sure if this comes from the changes here, but there's a MAKEFLAGS=w -j -l7.8 --jobserver-fds=3,4 V=1 in the output. Let me try to see who put the w there.

[saraedum]

Ok. That's how it's supposed to work ​https://lists.gnu.org/archive/html/help-make/2016-03/msg00009.html.

[chapoton]

moving milestone to 9.0 (after release of 8.9)