Opened 8 months ago

Closed 6 months ago

#33739 closed enhancement (fixed)

Migrate gitpod to conda

Reported by: Tobias Diez Owned by:
Priority: major Milestone: sage-9.7
Component: build Keywords:
Cc: Matthias Köppe, Dima Pasechnik, Isuru Fernando Merged in:
Authors: Tobias Diez Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: 5748c21 (Commits, GitHub, GitLab) Commit: 5748c2157b28ef12311062743d2e830ec7f79a85
Dependencies: #33740, #33855 Stopgaps:

Status badges

Description

The current solution to prebuild the gitpod docker image using github actions does not work nicely, since prebuilds triggered shortly after a new release still use the old gitpod image (and it may take up to a few days until the new image is available). This is especially notable for the develop branch, which is the default entry point if you want to start a new branch on gitpod.

In this ticket, we migrate the gitpod config to use the conda environment, which is now generally stable enough for serious development (e.g. almost all tests pass). As a positive side effect, the gitpod config simplifies and startup times are considerably reduced.

Change History (42)

comment:1 Changed 8 months ago by git

Commit: a1602c9b06c66281c9221a79d1e4857e48db8ed99361a337924203c14a4fdffc1194d50f12e2c15f

Branch pushed to git repo; I updated commit sha1. New commits:

9361a33Cleanup

comment:2 Changed 8 months ago by git

Commit: 9361a337924203c14a4fdffc1194d50f12e2c15fbd2f3d3bc7829f1ccc30f1aa274259d8ab138603

Branch pushed to git repo; I updated commit sha1. New commits:

bd2f3d3Remove gitpod docker image build

comment:3 Changed 8 months ago by Tobias Diez

Dependencies: #33740

comment:4 Changed 8 months ago by Tobias Diez

Status: newneeds_review

comment:5 Changed 7 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:6 Changed 7 months ago by git

Commit: bd2f3d3bc7829f1ccc30f1aa274259d8ab13860367c74f2bc1ed80e81ceb599e7d5cb64570157378

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

10338a7Change ci as well
0cf7334Add sage-setup as install-requires dep
6d2b149Use relative paths to not install src package
1dba3baMaybe thats the correct shebang?
c1b2fbfRevert install_requires change
ad56ba3Go back to two pip installs for now
b42f5f5Readd how to use specific python version
87f360cReadd name arg
82fe6a6Merge remote-tracking branch 'origin/public/build/conda-dev-env' into public/build/conda_gitpod
67c74f2Update based on recent changes in deps

comment:7 Changed 7 months ago by Tobias Diez

This is now using the latest version of #33740, so ready for review again.

comment:8 Changed 7 months ago by Matthias Köppe

I don't know if it's a good idea to make this change because it means that gitpod can no longer be used for trying out package updates in the Sage distribution.

Too bad that Gitpod has decided that a project can only have one Gitpod configuration - an obvious design limitation

comment:9 Changed 7 months ago by Tobias Diez

Can you not install the updated package manually using make xyz? But yes, there are a few things that might be harder to test in a conda environment. On the other hand, you also got a few new tools at your hand. For example, you can easily install specific versions of dependencies using conda.

I think its also not a good idea to optimize gitpod for package update tickets, they should only be a small subset of all tickets. And for those the conda-based gitpod works more reliable and has shorter startup times.

comment:10 in reply to:  9 ; Changed 7 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Can you not install the updated package manually using make xyz?

As the documentation says -

"Note that make is not used at all. All dependencies (including all Python packages) are provided by conda. [...] this will invalidate the use of any Sage-the-distribution commands such as sage -i. Do not use them."

Last edited 7 months ago by Matthias Köppe (previous) (diff)

comment:11 Changed 7 months ago by Matthias Köppe

I think the way forward could be to work on the devcontainer ticket (#33671). There can be multiple devcontainers (we could have them for all the platforms that we generate Docker images for). This would be a good replacement for what gitpod right now offers in terms of testing package upgrades.

When done, I agree that switching gitpod to conda is a good idea.

Also consider that both the conda stuff and the gitpod stuff are new -- for onboarding current developers, right now I think it's better if gitpod shows them something that is more familiar.

comment:12 in reply to:  11 ; Changed 7 months ago by Tobias Diez

Replying to mkoeppe:

I think the way forward could be to work on the devcontainer ticket (#33671). There can be multiple devcontainers (we could have them for all the platforms that we generate Docker images for). This would be a good replacement for what gitpod right now offers in terms of testing package upgrades.

How would devcontainer help here? Gitpod doesn't support them yet and its not clear yet when they will and in which form; the linked github issue reads more like that the are considering this as a alternative syntax for their gitpod.yml file and not that they would offer prebuilds for all defined devcontainers.

comment:13 in reply to:  10 Changed 7 months ago by Tobias Diez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Can you not install the updated package manually using make xyz?

As the documentation says -

"Note that make is not used at all. All dependencies (including all Python packages) are provided by conda. [...] this will invalidate the use of any Sage-the-distribution commands such as sage -i. Do not use them."

I just run make pari in the conda-based gitpod and it worked just fine.

comment:14 in reply to:  12 ; Changed 7 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

I think the way forward could be to work on the devcontainer ticket (#33671). There can be multiple devcontainers (we could have them for all the platforms that we generate Docker images for). This would be a good replacement for what gitpod right now offers in terms of testing package upgrades.

How would devcontainer help here?

If done right, it would make working on package upgrades almost as convenient as using gitpod. Sure, using local compilation (unless we orchestrate devcontainer prebuild), but convenient

comment:15 in reply to:  14 ; Changed 7 months ago by Tobias Diez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Replying to mkoeppe:

I think the way forward could be to work on the devcontainer ticket (#33671). There can be multiple devcontainers (we could have them for all the platforms that we generate Docker images for). This would be a good replacement for what gitpod right now offers in terms of testing package upgrades.

How would devcontainer help here?

If done right, it would make working on package upgrades almost as convenient as using gitpod. Sure, using local compilation (unless we orchestrate devcontainer prebuild), but convenient

Even with a conda-based gitpod, you can simply deactivate the conda environment and run configure && make. This is pretty much a disposable devcontainer without prebuild.

comment:16 in reply to:  15 Changed 7 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Even with a conda-based gitpod, you can simply deactivate the conda environment and run configure && make.

That's true, of course

This is pretty much a disposable devcontainer without prebuild.

Well no, because it wouldn't even have a SAGE_LOCAL. So it offers nothing over just building the whole thing locally

Last edited 7 months ago by Matthias Köppe (previous) (diff)

comment:17 Changed 7 months ago by Tobias Diez

The advantage would be that you can run whatever commands you want without worrying how it affects the rest of your system / the main installation of sage.

Anyway, I still think that package updates are only a small subset of all tickets and making them temporarily slightly less convenient to test with gitpod is a price worth paying if gitpod gets more useful for the rest of the tickets. Moreover, as you mention, gitpod support is new so developers can easily go back to whatever they were using before to test package updates until further improvements like #33671 are implemented.

It would be nice if this ticket could be merged before the sagedays so that I don't have to wait 5mins during my talk until the workspace started.

comment:18 Changed 7 months ago by git

Commit: 67c74f2bc1ed80e81ceb599e7d5cb64570157378f56a535d712eb598b207c5d88d3c2f32fbbcafaa

Branch pushed to git repo; I updated commit sha1. New commits:

f56a535Install esbonio using conda

comment:19 Changed 7 months ago by Matthias Köppe

Is it really starting up faster?

comment:20 Changed 7 months ago by Matthias Köppe

Regression: The SageMath jupyter kernel cannot be selected (Command Palette - Create: New Jupyter Notebook, then "Select Kernel" in the upper right corner)

Last edited 7 months ago by Matthias Köppe (previous) (diff)

comment:21 Changed 7 months ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:22 in reply to:  19 Changed 7 months ago by Tobias Diez

Replying to mkoeppe:

Is it really starting up faster?

yes for me its <1 min until the workspace is started.

comment:23 Changed 7 months ago by Matthias Köppe

I haven't timed it but here it took much longer than that. I'll use the timer to get more data later.

comment:24 in reply to:  20 Changed 7 months ago by Tobias Diez

Replying to mkoeppe:

Regression: The SageMath jupyter kernel cannot be selected (Command Palette - Create: New Jupyter Notebook, then "Select Kernel" in the upper right corner)

I can select the correct "venv" there, what are you missing?

comment:25 Changed 7 months ago by Matthias Köppe

Missing the "SageMath" kernel (not the IPython kernel).

comment:26 Changed 7 months ago by Tobias Diez

How is this supposed to work? I found src/sage_setup/command/sage_install.py which installs the jupyter kernel but is only referenced in the install hook in setup.py and thus not invoked for editable installs. So this doesn't seem specific to the conda or github setup. What do I miss?

comment:27 Changed 7 months ago by Matthias Köppe

I think currently it is working because sagelib is first installed without --enable-editable in the image.

comment:28 Changed 7 months ago by Matthias Köppe

Dependencies: #33740#33740, #33855

comment:29 Changed 7 months ago by Matthias Köppe

Let's add it in #33855.

comment:30 Changed 7 months ago by git

Commit: f56a535d712eb598b207c5d88d3c2f32fbbcafaa88d769b8c396a61f6e5dff91434cf724766f06ad

Branch pushed to git repo; I updated commit sha1. New commits:

6c5903dsrc/sage_setup/command/sage_install.py: Split sage_install_and_clean into several classes
40b44a1build/pkgs/sagelib/spkg-install: Do installation of sagemath kernelspec here, not in sage_setup.command.sage_install
1c1dbc5src/setup.py: No need to use custom sage_install
6862287build/pkgs/sagelib/spkg-install: Un-poison SAGE_DOC for installing the kernelspec
88d769bMerge #33855

comment:31 Changed 7 months ago by Matthias Köppe

Status: needs_workneeds_review

comment:32 Changed 7 months ago by Matthias Köppe

Status: needs_reviewneeds_work

Of course by itself #33855 doesn't fix it -- the same line in spkg-install needs to be run manually...

comment:33 Changed 7 months ago by git

Commit: 88d769b8c396a61f6e5dff91434cf724766f06ad2b9ef5aa985213832f07fe299fa1c2f608fd0d4e

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

6c5903dsrc/sage_setup/command/sage_install.py: Split sage_install_and_clean into several classes
4473375src/sage_setup/command/sage_install.py: Add sage_develop, use it in src/setup.py, pkgs/sagemath-standard/setup.py
2b9ef5aMerge #33855

comment:34 Changed 7 months ago by Matthias Köppe

Status: needs_workneeds_review

Rolled back the merge and merged the new version of #33855.

comment:35 Changed 7 months ago by Matthias Köppe

Now the kernel is available

comment:36 Changed 7 months ago by Matthias Köppe

Cc: Dima Pasechnik added
Reviewers: Matthias Koeppe

comment:37 Changed 7 months ago by Matthias Köppe

Cc: Isuru Fernando added
Status: needs_reviewpositive_review

I haven't tested it thoroughly, but it looks like a nice improvement. It does start faster than the previous solution using our huge prebuilt Docker images.

comment:38 Changed 7 months ago by Tobias Diez

Thanks!

comment:39 Changed 7 months ago by Volker Braun

Status: positive_reviewneeds_work

Merge failure on top of:

054e8ed8d9 Trac #33316: Drop support for GCC < 6.3 in Sage 9.7

fa3b529c41 Trac #25833: Upgrade to MathJax? 3 and configure Sage

e2ebf445d0 Trac #33825: Use pytest-xdist to run pytest in parallel

b4009625f3 Trac #33824: make gens and orbits of PermutationGroup? immutable

af23627f18 Trac #33809: some pathlib in combinat and groups

955b5d994c Trac #33803: Fixes for the distributions sagemath-objects, sagemath-categories

3e6b41f7d6 Trac #33799: Replace SAGE_TMP in doctests of sage.misc.{persist,ostools}, sage.doctest, sage.repl

a3fd7185e3 Trac #33797: sage.misc.temporary_file: Remove use of SAGE_TMP

2ca053010a Trac #33794: modules/fp_graded/morphism.py test failure

7037fba482 Trac #33793: sage.misc.cython: Replace use of SPYX_TMP by a new cached function in sage.misc.temporary_file

d1152701c4 Trac #33787: Installation manual: Update section "system-wide install"

0ae55652cf Trac #33782: ci-cygwin: Update python version

833f53d1cc Trac #33779: Remove use of SAGE_TMP in sage.interfaces.latte

b376a8d71c Trac #33771: SSLContext needs an argument

df168c8112 Trac #33763: Refactor src/sage/docs

9597eafded Trac #33748: make accessing coefficients of finite-field elements easier

f02236f5b6 Trac #33744: Compute bases/circuits in MatroidUnion?

8943dc07fc Trac #33743: Faster random tree generator

773ec37bec Trac #33740: Add conda dev environment

5e65c16108 Trac #33734: variety() for polynomial systems over ℚ using msolve

8e7dcca8d3 Trac #33733: allow to use flint for Stirling numbers

6f4efb0bf3 Updated SageMath version to 9.7.beta0

merge was not clean: conflicts in .github/workflows/tox.yml

comment:40 Changed 6 months ago by git

Commit: 2b9ef5aa985213832f07fe299fa1c2f608fd0d4e5748c2157b28ef12311062743d2e830ec7f79a85

Branch pushed to git repo; I updated commit sha1. New commits:

5748c21Merge tag '9.7.beta1' into t/33739/public/build/conda_gitpod

comment:41 Changed 6 months ago by Matthias Köppe

Status: needs_workpositive_review

comment:42 Changed 6 months ago by Volker Braun

Branch: public/build/conda_gitpod5748c2157b28ef12311062743d2e830ec7f79a85
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.