tox.ini: Add environments local-sudo-ubuntu-standard, etc.

[Matthias Köppe]

These environments would just run sudo apt-get ... etc. to install (but not remove!) packages, for use at the user's own risk.

To test (on an Ubuntu system):

    tox -e local-sudo-ubuntu-bionic-standard

or

    tox -e local-sudo-standard

[Tobias Diez]

Is it possible to put the sudo outside? I.e. sudo tox -r local-ubuntu-standard?

[Matthias Köppe]

But we would only want to do the package installation as root, not the Sage build

[Matthias Köppe]

Here's a first approximation.

---

New commits:

​30e624a

tox.ini: Add local-sudo

[Tobias Diez]

I think you forgot the confirmation -y after install: ​https://github.com/tobiasdiez/sage/commit/942149ee76a97628046042b7255a2d31630e7721

Moreover, I was wondering if one should also read the debian-bootstrap.txt files.

Finally, there are problems that some of the packages are not found. First, one should probably add sudo add-apt-repository ppa:deadsnakes/ppa since otherwise libpython3.7-dev is not found on ubuntu 20.04. Second, texlive-generic-extra doesn't exist on ubuntu 20.04. ​https://github.com/tobiasdiez/sage/runs/1404001507?check_suite_focus=true

Other than that, looks good to me! Thanks a lot.

[Matthias Köppe]

Yes, I'll add bootstrap

I don't think we have libpython3.7-dev listed in the debian.txt files any more.

But yes, handling IGNORE_MISSING_SYSTEM_PACKAGES needs to be done.

For deadsnakes, there's #30217. This would not be the default, but a mix-in flavor.

[Tobias Diez]

On a second thought, do you really thing it's the job of tox to install system packages? I would feel more comfortable to put the code into a shell script that you can run as build/install_system_packages ubuntu-2004, say. (Having such a script might be a good idea anyway).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​75ecd11

tox.ini (local-sudo): Also use ...-bootstrap.txt

[Tobias Diez]

Replying to mkoeppe:

But yes, handling IGNORE_MISSING_SYSTEM_PACKAGES needs to be done.

and/or change texlive-generic-extra that exist on modern ubuntu systems as well.

[Tobias Diez]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​75ecd11

tox.ini (local-sudo): Also use ...-bootstrap.txt

Don't forget to change local-sudo-standard/maximal as well.

[Matthias Köppe]

Replying to gh-tobiasdiez:

On a second thought, do you really thing it's the job of tox to install system packages? I would feel more comfortable to put the code into a shell script that you can run as build/install_system_packages ubuntu-2004, say. (Having such a script might be a good idea anyway).

It's working well with tox, in particular the environment factors give a nice compact way of expressing different configurations.

And I definitely do not want to add user-facing scripts to Sage that promises to install all necessary system packages. We deliberately only offer system package *advice* (at the end of ./configure). Already now users complain if that advice is not entirely accurate -- so we definitely do not want to make stronger promises! We simply do not have enough developers who could keep it up to date.

But there is a lot of room for refactoring what is in tox.ini into various helper scripts. I have described some possible directions in #30865 -- help is very welcome there.

[Matthias Köppe]

Replying to gh-tobiasdiez:

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​75ecd11

tox.ini (local-sudo): Also use ...-bootstrap.txt

Don't forget to change local-sudo-standard/maximal as well.

No, that line is also executed for these environments - by the magic of tox's factor conditions.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0dc79eb	build/bin/sage-print-system-package-command: Handle --no-install-recommends, --yes for systems for which write-dockerfile.sh knows these flags
​050dcb8	tox.ini (local-sudo): Use --yes --no-install-recommends

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c8fbe0b

tox.ini (local-sudo): Ignore errors when IGNORE_MISSING_SYSTEM_PACKAGES=yes

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ff34897

tox.ini (local): Guess the package system if it is not provided as a factor

[Tobias Diez]

Replying to mkoeppe:

Replying to gh-tobiasdiez:

On a second thought, do you really thing it's the job of tox to install system packages? I would feel more comfortable to put the code into a shell script that you can run as build/install_system_packages ubuntu-2004, say. (Having such a script might be a good idea anyway).

It's working well with tox, in particular the environment factors give a nice compact way of expressing different configurations.

Sure, but it feels a bit like you are using tox for something it wasn't designed. Normally, in a tox setting, "environment" refers to a virtual python env, but for sage it is the complete system env including the os and system packages. I have the feeling a more general-purpose framework like invoke would be a better solution to manage these things.

And I definitely do not want to add user-facing scripts to Sage that promises to install all necessary system packages.

I didn't meant it as a user-facing script, but only for developers (i.e don't ship it).

[Tobias Diez]

I've tested the current code in the wsl github workflow and it seems to work well: ​https://github.com/tobiasdiez/sage/actions/runs/365823571

[Matthias Köppe]

Sounds like a positive review?

[Tobias Diez]

Yes, in principle. But I cannot really judge the changes in the bash script.

[Dima Pasechnik]

ok

[Matthias Köppe]

Thanks!

[Tobias Diez]

Is it possible to call only the system package install code via tox (i.e. don't run rest of build and tests)?

That would be handy for #30371.

[Matthias Köppe]

Replying to gh-tobiasdiez:

Is it possible to call only the system package install code via tox (i.e. don't run rest of build and tests)?

Yes, you can pass make targets to tox, e.g., tox -e local-sudo-standard -- configure will only run bootstrap, or tox -e local-sudo-standard -- config.status will stop after running ./configure

[Tobias Diez]

Thanks! That kind of worked. I had to install sudo manually, since this is not installed by default on github actions and also not necessary. It would be nice if you could try to make it work without sudo.

local-sudo-standard run-test: commands[2] | bash -c 'PACKAGES=`sed "s/#.*//;" build/pkgs/$(build/bin/sage-guess-package-system)*.txt`; $(build/bin/sage-print-system-package-command $(build/bin/sage-guess-package-system) --sudo --yes --no-install-recommends install $PACKAGES) || [ "$IGNORE_MISSING_SYSTEM_PACKAGES" = yes ] && echo "(ignoring errors)" '
/usr/bin/bash: sudo: command not found

ERROR: InvocationError for command /usr/bin/bash -c 'PACKAGES=`sed "s/#.*//;" build/pkgs/$(build/bin/sage-guess-package-system)*.txt`; $(build/bin/sage-print-system-package-command $(build/bin/sage-guess-package-system) --sudo --yes --no-install-recommends install $PACKAGES) || [ "$IGNORE_MISSING_SYSTEM_PACKAGES" = yes ] && echo "(ignoring errors)" ' (exited with code 1)

Edit: Moreover, DEBIAN_FRONTEND: noninteractive was/is missing. And it fails with

configure: error: You cannot build Sage as root, switch to an unprivileged user.  (If building in a container, use --enable-build-as-root.)

Since using this tox command for ci was one of the motivations but doesn't really work at the moment without extra modifications, I'll set the ticket back to needs work.

[Volker Braun]

Can you make a separate ticket for that? Once a ticket is in positive review I'll only unmerge it for critical bugs, not things that would also be good to have.

[Tobias Diez]

Sure! It's now #30944. Sorry if I broke any (unwritten) laws; I wasn't sure about resetting it to "needs work".