tox: Improve local-sudo-ubuntu-standard

[gh-tobiasdiez]

As a follow up to #30923, we improve the local-sudo-... environments as follows:

• Provide alternative environments local-root..., which assume that we are already root, removing the sudo requirement (which is not installed on github actions and not needed there)
• Use configure --enable-build-as-root for local-root
• Add DEBIAN_FRONTEND: noninteractive to the installation of packages via apt get (otherwise installation of tzdata blocks CI)
• Make tox -e local-sudo-standard -- config.status only build config.status instead of also invoking make base-toolchain (which is run as the required first step of the Sage build system before something like make numpy can be run; see #30721).

[mkoeppe]

The variant could be called local-root-ubuntu-standard, I guess

[mkoeppe]

Replying to gh-tobiasdiez:

• Make tox -e local-sudo-standard -- config.status stop after configure (and don't run make)

Right, I forgot about the mess with TARGETS_PRE and TARGETS_OPTIONAL in tox.ini, which do not get the same defaults for local as they do for docker - I'll fix this

---

New commits:

​30e624a	tox.ini: Add local-sudo
​75ecd11	tox.ini (local-sudo): Also use ...-bootstrap.txt
​0dc79eb	build/bin/sage-print-system-package-command: Handle --no-install-recommends, --yes for systems for which write-dockerfile.sh knows these flags
​050dcb8	tox.ini (local-sudo): Use --yes --no-install-recommends
​c8fbe0b	tox.ini (local-sudo): Ignore errors when IGNORE_MISSING_SYSTEM_PACKAGES=yes
​ff34897	tox.ini (local): Guess the package system if it is not provided as a factor
​e9e5d47	build/bin/sage-print-system-package-command (debian --yes): Use DEBIAN_FRONTEND=noninteractive

[mkoeppe]

Replying to mkoeppe:

The variant could be called local-root-ubuntu-standard, I guess

Adding this in #29124

[mkoeppe]

Replying to ​gh-tobiasdiez in #29124:

I'm not sure about the naming of the local- environments. Why should I expect as a developer that local-sudo installs system packages? What about local-system-packages-ubuntu (for root) and local-sudo-system-packages-ubuntu (for sudo)?

All of these environments install system packages... The only one that does not is local-direct

[gh-tobiasdiez]

Ah ok...than it makes sense.

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​d318067	build/pkgs/_prereq/spkg-configure.m4: New
​85f8ce5	src/doc/bootstrap: Remove special casing of build/pkgs/$SYSTEM.txt - _prereq is now just a standard package with spkg-configure.m4
​0e97683	tox.ini: Simplify, use sage-get-system-packages, sage-package list
​da3f657	tox.ini: Fixup - always include _bootstrap
​8fde1dc	build/bin/write-dockerfile.sh: Remove special casing of build/pkgs/$SYSTEM.txt
​8d725b8	src/doc/en/developer/portability_testing.rst: Update paths in documentation
​bd7f66c	Makefile (configure): Update dependency on moved files build/pkgs/*.txt
​c18e935	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard
​f18f849	tox.ini: Add local-root
​fd47bc3	tox.ini (local-root): Pass --enable-build-as-root to configure

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d700ab2

Merge tag '9.3.beta2' into t/30944/tox__improve_local_sudo_ubuntu_standard

[mkoeppe]

To simplify the build rule for local, the next step is to do #30721 so we can eliminate the separate make base-toolchain step.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5e9366d	Merge tag '9.3.beta2' into t/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_
​6b9f3a3	Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap
​9414be1	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e959873

tox.ini (local): Do not build the toolchain when posargs = config.status or posargs = configure

[mkoeppe]

Replying to mkoeppe:

To simplify the build rule for local, the next step is to do #30721 so we can eliminate the separate make base-toolchain step.

Well, I'll work on that another time. For now I have implemented it in a different way, special casing on "config*" targets.

[gh-tobiasdiez]

Thanks! Looks good to me, as far as I understand the changes.

Just to make sure:

Make tox -e local-sudo-standard -- config.status stop after configure

Is fixed as well?

[mkoeppe]

Replying to gh-tobiasdiez:

Thanks! Looks good to me, as far as I understand the changes.

Just to make sure:

Make tox -e local-sudo-standard -- config.status stop after configure

Is fixed as well?

Yes. This is what I meant in the previous comment - "special casing on config* targets"

[gh-tobiasdiez]

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Thanks! Looks good to me, as far as I understand the changes.

Just to make sure:

Make tox -e local-sudo-standard -- config.status stop after configure

Is fixed as well?

Yes. This is what I meant in the previous comment - "special casing on config* targets"

Ok, then is good! Me with my limited bash knowledge thought that the final make that builds the sage packages is still invoked even though -- config.status is specified.

[mkoeppe]

It's only invoked on nonempty $TARGETS_OPTIONAL

[gh-tobiasdiez]

Running tox -e local-sudo-ubuntu-standard -- SAGE_NUM_THREADS=16 build (in WLS) yields

Reading package lists... Done
E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
E: Unable to lock directory /var/lib/apt/lists/
W: Problem unlinking the file /var/cache/apt/pkgcache.bin - RemoveCaches (13: Permission denied)
W: Problem unlinking the file /var/cache/apt/srcpkgcache.bin - RemoveCaches (13: Permission denied)
ERROR: InvocationError for command /usr/bin/bash -c '$(build/bin/sage-print-system-package-command debian update) #' (exited with code 100)

Running the same command with sudo is successful, so I guess a --sudo is missing somewhere.

Moreover, ./bootstrap didn't run with following error:

/mnt/d/Programming/sage/src/bin/sage-env: line 133: cd: /mnt/d/Programming/Projects/sage: No such file or directory
Warning: overwriting SAGE_ROOT environment variable:
Old SAGE_ROOT=/mnt/d/Programming/Projects/sage
New SAGE_ROOT=
Error: You must set either the SAGE_LOCAL or SAGE_SCRIPTS_DIR environment variable to run this
Error setting environment variables by sourcing '/mnt/d/Programming/sage/src/bin/sage-env';
possibly contact sage-devel (see http://groups.google.com/group/sage-devel).

until I deleted src/bin/sage-env manually. And the root version sudo tox -e local-root-ubuntu-standard -- SAGE_NUM_THREADS=10 build yields

local-root-ubuntu-standard run-test: commands[2] | bash -c 'PACKAGES=$(build/bin/sage-get-system-packages debian $(PATH=build/bin:$PATH build/bin/sage-package list --has-file=spkg-configure.m4 :standard:) _bootstrap); $(build/bin/sage-print-system-package-command debian  --yes --no-install-recommends install $PACKAGES) || [ "$IGNORE_MISSING_SYSTEM_PACKAGES" = yes ] && echo "(ignoring errors)" '
/usr/bin/bash: DEBIAN_FRONTEND=noninteractive: command not found
ERROR: InvocationError for command /usr/bin/bash -c 'PACKAGES=$(build/bin/sage-get-system-packages debian $(PATH=build/bin:$PATH build/bin/sage-package list --has-file=spkg-configure.m4 :standard:) _bootstrap); $(build/bin/sage-print-system-package-command debian  --yes --no-install-recommends install $PACKAGES) || [ "$IGNORE_MISSING_SYSTEM_PACKAGES" = yes ] && echo "(ignoring errors)" ' (exited with code 1)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​27368b1

tox.ini (local-sudo): Run apt-get update with sudo

[mkoeppe]

Replying to gh-tobiasdiez:

Moreover, ./bootstrap didn't run with following error:

/mnt/d/Programming/sage/src/bin/sage-env: line 133: cd: /mnt/d/Programming/Projects/sage: No such file or directory
Warning: overwriting SAGE_ROOT environment variable:
Old SAGE_ROOT=/mnt/d/Programming/Projects/sage
New SAGE_ROOT=
Error: You must set either the SAGE_LOCAL or SAGE_SCRIPTS_DIR environment variable to run this
Error setting environment variables by sourcing '/mnt/d/Programming/sage/src/bin/sage-env';
possibly contact sage-devel (see http://groups.google.com/group/sage-devel).

until I deleted src/bin/sage-env manually.

This is fixed by #22731, I think. I'll merge it

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8f88709	src/bin/sage: Only source sage-env-config if it exists
​ab2655e	src/bin/sage: Use python3 etc. from PATH instead of using SAGE_LOCAL
​4577f37	src/bin/sage-env: Make sage-env-config optional
​c35c170	sage.env: Add SAGE_VENV
​c585d94	src/sage: Replace some SAGE_LOCAL by SAGE_VENV
​2fd195f	src/bin/sage: Fix reference to trac ticket
​f3b7a9c	src/sage/env.py: Add documentation
​1fe77a8	src/bin/sage-env: Do not set SAGE_SCRIPTS_DIR
​38eebc3	Merge tag '9.3.beta2' into t/22731/22731
​7216c67	Merge branch 'u/mkoeppe/22731' of git://trac.sagemath.org/sage into t/30944/tox__improve_local_sudo_ubuntu_standard

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1c2dfc6

tox.ini (local-root, local-sudo): Run output of sage-print-system-package-command through eval

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​35241e7	src/doc/en/developer/portability_testing.rst: Update paths in documentation
​68baff1	Makefile (configure): Update dependency on moved files build/pkgs/*.txt
​fae4bd0	bootstrap: Remove --enable-_recommended etc.
​a9bd145	m4/sage_spkg_collect.m4: Remove _recommended etc. from the configure package summary
​93f5e32	build/bin/sage-print-system-package-command (debian --yes): Use DEBIAN_FRONTEND=noninteractive
​898758d	tox.ini: Add local-root
​559dd8e	tox.ini (local-root): Pass --enable-build-as-root to configure
​a85f41c	tox.ini (local): Do not build the toolchain when posargs = config.status or posargs = configure
​1de912a	tox.ini (local-sudo): Run apt-get update with sudo
​3c7e5c4	tox.ini (local-root, local-sudo): Run output of sage-print-system-package-command through eval

[mkoeppe]

Rebased on top of 9.3.beta3 and rebased #29124

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​184ae6d	Merge tag '9.3.beta3' into t/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_
​084fbf6	src/doc/bootstrap: Use ./sage
​571cc49	Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap
​f7ff30c	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c529729	Merge commit '3bb309944b7e8542b2ac88ed3c9d9a60e68644d7' of git://trac.sagemath.org/sage into t/29124/script-packages-prereq-toolchain-bootstrap
​b73d6f0	Merge tag '9.3.beta4' into t/29124/script-packages-prereq-toolchain-bootstrap
​e9ca2c1	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard

[mkoeppe]

Merged newest version of #29124. Still needs review

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​3cf5ee4	Merge commit 'a50ddf88975086b14a49895e371477df00fd57b5' of git://trac.sagemath.org/sage into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​b4927e1	sage.misc.package: Remove/adjust non-robust doctests
​e5fe752	Merge tag '9.3.beta4' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​78ff9d5	src/sage/misc/package.py: Add one more # optional - build
​a44042f	Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap
​64bde5f	Merge tag '9.3.beta5' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​e9a7572	src/sage/misc/package.py: Improve source formatting
​c7bcda9	Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap
​9988c5f	ci-cygwin*.yml: Adjust to new script packages _bootstrap, _prereq
​ab19133	Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard

[mkoeppe]

Merged newest #29124, needs review

[mkoeppe]

Let's get this in please (dependency of #31064 and #31099)

[gh-kliem]

I think #29124 isn't merged completely.

[gh-kliem]

What does

+    local:             case "{posargs:}" in \
+    local:                 config*) ;; \
+    local:                 *)       make -k V=0 base-toolchain ;; \
+    local:             esac && \
-    local:                     make -k V=0 base-toolchain && \

do? (That is not the diff, but I think morally it is.)

[mkoeppe]

It implements the last bullet point of the ticket description: When something like config.status is given as the target, make base-toolchain should not be run.

[gh-kliem]

Thanks. But does it stop then or are the other things still run?:

+    local:             make -k V=0 SAGE_SPKG="sage-spkg -y -o" SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!nose,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!networkx,!symengine_py" {env:TARGETS_PRE:} {posargs:build} && \
+    local:             ([ -z "{env:TARGETS_OPTIONAL:}" ] || make -k V=0 SAGE_SPKG="sage-spkg -y -o" SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!nose,!gap,!cysignals,!linbox,!git,!ppl,!cmake" {env:TARGETS_OPTIONAL:} || echo "(error ignored)" ) '

[mkoeppe]

I have reworded the ticket description to clarify it.

[gh-kliem]

LGTM.

[mkoeppe]

Thank you!