Changes between Version 209 and Version 279 of Ticket #24655


Ignore:
Timestamp:
07/12/18 07:57:51 (3 years ago)
Author:
saraedum
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Ticket #24655

    • Property Commit changed from 83f71e7948c8eb8a72e63d4d1451f9f20779bb8b to 624101115b9af149f8bf0d7cfe2f63d9ccf5201d
    • Property Work issues changed from test all variations of branches and tags on CircleCI & GitLab CI to check that everything works ⇒ (provisional) positive review
  • Ticket #24655 – Description

    v209 v279  
    1 It would be nice to update our docker images automatically through continuous integration services. Of course it's nice to have these images up-to-date without manual intervention but this is also convenient as a starting point for people who want to use CI for their own branches of Sage (besides the patchbot.)¹
     1It would be nice to update our docker images automatically through continuous integration services. Of course it's good to have these images up-to-date without manual intervention but this is also convenient as a starting point for people who want to use CI for their own branches of Sage (besides the patchbot.)¹
    22
    33This ticket proposes recipes for GitLab CI and CircleCI to build our docker images automatically. On the respective websites, the CI can be configured to push automatically to the Docker Hub. A webhook (on github) updates the README on Docker Hub automatically.
    44
    5 I implemented this for both GitLab CI and CircleCI. I think GitLab CI is more relevant in the long  run, also it's open source and people can provision their own machines as test runners. CircleCI at the same time works out of the box for Sage without private test runners and it also allows for easier debugging as you can logon to the machine running your tests with SSH. I tried to share most code between the two implementations.
     5I implemented this for both GitLab CI and CircleCI. I think GitLab CI is more relevant in the long  run, also it's open source and people can provision their own machines as runners. CircleCI at the same time works out of the box for Sage without private test runners and it also allows for easier debugging as you can logon to the machine running your tests with SSH. I tried to share most code between the two implementations.
    66
    77See also https://github.com/sagemath/docker-images/issues/13 and https://github.com/sagemath/sage-binder-env/issues/3 for a followup (automatically provide jupyter notebooks for easier review.)
     
    1313=== GitLab CI
    1414
    15 If I provision my own runner from Google Cloud with four threads, it takes about 3 hours to build Sage from scratch, run rudimentary tests on the docker images, and upload them to Docker Hub and GitLab's registry.
     15If I provision my own runner from Google Cloud with two threads, it takes about 5 hours to build Sage from scratch, run rudimentary tests on the docker images, and upload them to Docker Hub and GitLab's registry.
    1616
    17 [[Image(gitlab.png, 640, center, link=https://gitlab.com/saraedum/sage/pipelines/18022699)]]
     17[[Image(gitlab.png, 640, center, link=https://gitlab.com/saraedum/sage/pipelines)]]
    1818
    19 Recycling the build artifacts from the last run on the develop branch brings this down to about 35 minutes (on GitLab's free shared runners with two threads.) This roughly breaks down as:
    20 * 20 minutes for `build-from-latest:
    21   * 6 minutes for the actual build (most of which is spent in the docbuild; caused by a known Sphinx bug to some extent)
    22   * 5 minutes are spent pulling the sagemath-dev image from Docker Hub (this usually goes away if you provision your own runners and expose the host's docker daemon by setting `DOCKER_HOST`.)
     19Recycling the build artifacts from the last run on the develop branch brings this down to about **??** minutes (on GitLab's free shared runners with two threads.) This roughly breaks down as:
     20* **??** minutes for `build-from-latest:
     21  * **??** minutes for the actual build (most of which is spent in the docbuild; caused by a known Sphinx bug to some extent)
     22  * **??** minutes are spent pulling the sagemath-dev image from Docker Hub (this usually goes away if you provision your own runners and expose the host's docker daemon by setting `DOCKER_HOST`.)
    2323  * a few minutes running through all the fast stages of the Dockerfile.
    2424  * a few minutes to push the resulting images to GitLab's registry. (using GitLab's `cache`, this could probably be improved, at least for runners that we provision ourselves.)
    25 * 2 - 12 minutes for each test (run in parallel,); the relevant test is `test-dev.sh` which spents 6 minutes in the actual docbuild (just as in `build-from-latest`) and some 5 minutes to pull the sagemath-dev image from the GitLab registry. (That part should go away with a provisioned runner that exposes the host's docker daemon.)
    26 * 5 minutes for the publishing to Docker Hub, most of which is spent pulling the images from the GitLab registry, and the other half pushing them to Docker Hub roughly. (Again, exposing the host's docker daemon would probably cut that time in half.)
     25* **??** - **??** minutes for each test (run in parallel,); the relevant test is `test-dev.sh` which spents 6 minutes in the actual docbuild (just as in `build-from-latest`) and some 5 minutes to pull the sagemath-dev image from the GitLab registry. (That part should go away with a provisioned runner that exposes the host's docker daemon.)
     26* **??** minutes for the publishing to Docker Hub, most of which is spent pulling the images from the GitLab registry, and the other half pushing them to Docker Hub roughly. (Again, exposing the host's docker daemon would probably cut that time in half.)
    2727
    2828With some tricks we could probably bring this down to 25 minutes (see CircleCI below) but we won't get this down to this without giving up on the CI being split up into different stages (as is for technical reasons necessary for CircleCI.) To go well below that, we would need to pull binary packages from somewhere…I don't see a sustainable way of doing this with the current SPKG system.
     
    3232=== CircleCI
    3333
    34 It typically takes about 4 hours to build Sage from scratch on CircleCI, run rudimentary tests on the docker images, and upload them to Docker Hub.
     34It typically takes about **??** hours to build Sage from scratch on CircleCI, run rudimentary tests on the docker images, and upload them to Docker Hub.
    3535
    36 [[Image(circleci.png, 640, center, link=https://circleci.com/gh/saraedum/sage/195)]]
     36[[Image(circleci.png, 640, center, link=https://circleci.com/gh/saraedum/workflows/sage)]]
    3737
    38 Recycling the build artifacts from the last run on the develop branch brings this down to about 25 minutes usually. 5 minutes could be saved by not testing the `sagemath-dev` and probably another minute or two if we do not build it at all. To go significantly below 15 minutes is probably hard with the huge sage-the-distribution (7GB uncompressed/2GB compressed) that we have to pull every time at the moment.
     38Recycling the build artifacts from the last run on the develop branch brings this down to about minutes usually. 5 minutes could be saved by not testing the `sagemath-dev` and probably another minute or two if we do not build it at all. To go significantly below 15 minutes is probably hard with the huge sage-the-distribution (7GB uncompressed/2GB compressed) that we have to pull every time at the moment.
    3939
    40 [[Image(circleci-rebuild.png, 640, center, link=https://circleci.com/gh/saraedum/sage/203)]]
     40[[Image(circleci-rebuild.png, 640, center, link=https://circleci.com/gh/saraedum/workflows/sage)]]
    4141
    4242=== Docker Hub
    4343
    44 A push to github updates the README on the Docker Hub page. The current sizes are [[Image(https://img.shields.io/microbadger/image-size/sagemath/sagemath/latest.svg)]] and [[Image(https://img.shields.io/microbadger/image-size/sagemath/sagemath-dev/latest.svg)]]
     44A push to github updates the README on the Docker Hub page. The current sizes are [[Image(https://img.shields.io/microbadger/image-size/sagemath/sagemath/latest.svg)]] and [[Image(https://img.shields.io/microbadger/image-size/sagemath/sagemath-dev/latest.svg)]]; unfortunately MicroBadger is somewhat unstable so these numbers are incorrectly reported as 0 sometimes.
    4545
    4646[[Image(dockerhub.png, 640, center, link=https://hub.docker.com/r/sagemath/sagemath)]]
     
    5151
    5252* Setup an account for sagemath on Circle CI.
    53 * Add Docker Hub credentials on  Circle CI or GitLab.
     53* Add Docker Hub credentials on Circle CI or GitLab.
    5454
    5555To see a demo of what the result looks like, go to https://hub.docker.com/r/sagemath/sagemath/. The CircleCI runs can be seen here https://circleci.com/gh/saraedum/sage, and the GitLab CI runs are here https://gitlab.com/saraedum/sage/pipelines.