#32966 closed enhancement (fixed)

tox / GH Actions: Add ubuntu variants using ppa:ubuntu-toolchain-r

Reported by: Matthias Köppe Owned by:
Priority: critical Milestone: sage-9.6
Component: porting Keywords:
Cc: Dima Pasechnik, Tobias Diez Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: d42bc03 (Commits, GitHub, GitLab) Commit: d42bc03b1a39736de8534dcf3a05c2b02ec3ce89
Dependencies: #33187, #33103 Stopgaps:

Status badges

Description (last modified by Dima Pasechnik)

(from #30876)

see https://askubuntu.com/a/1149383/309919

To test:

NB: Your Docker should have enough RAM allowed (I'd advice 8GB). In Docker Desktop this setting is in Preferences->Resources -> Advanced.

Change History (69)

comment:1 Changed 13 months ago by Matthias Köppe

Milestone: sage-9.5sage-9.6

comment:2 Changed 13 months ago by Matthias Köppe

Description: modified (diff)

comment:3 Changed 13 months ago by Matthias Köppe

Branch: u/mkoeppe/tox___gh_actions__add_ubuntu_variants_using_ppa_ubuntu_toolchain_r

comment:4 Changed 13 months ago by git

Commit: de42909be8a8e9bf7261a2d528b0fafd5a382bb0

Branch pushed to git repo; I updated commit sha1. New commits:

de42909build/pkgs/_gcc*/distros: Add homebrew.txt

comment:5 Changed 13 months ago by Matthias Köppe

Authors: Matthias Koeppe
Status: newneeds_review

comment:6 Changed 12 months ago by Matthias Köppe

Description: modified (diff)

comment:7 Changed 12 months ago by Matthias Köppe

Cc: Tobias Diez added

comment:8 Changed 12 months ago by Tobias Diez

Do we really need to add tests for each possible combination of system packages? This would also mean that we voucher to support these systems (otherwise, what's the point of these tests?). I think maintaining the "standard" system config is already enough overhead. How much people on ubuntu add custom package repos?

Maybe it would be worthwhile to have a general discussion about which systems sage wants to support (how old, which system configs, when to drop support etc) and then based on this outcome add tests. (Please excuse my ignorance if there have been already such a discussion.)

Also this ticket seems to be the exact opposite of https://trac.sagemath.org/ticket/32532 and as far as I followed the discussion on the dev list no final conclusion has been reached there yet.

comment:9 Changed 12 months ago by Dima Pasechnik

The standard ones on these museum-grade systems are too old to be supported. IMHO we must be much more aggressive in dropping old systems than we currently are...

comment:10 in reply to:  8 ; Changed 12 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

This would also mean that we vouch to support these systems (otherwise, what's the point of these tests?).

No, this ticket (and all of tox.ini) does not come with a promise that we support a particular system configuration.

It only makes it easy to test it.

comment:11 in reply to:  9 Changed 12 months ago by Matthias Köppe

Replying to dimpase:

The standard ones on these museum-grade systems are too old to be supported.

Exactly, in particular because of upcoming updates of several of our standard packages, see #32074.

comment:12 in reply to:  10 Changed 12 months ago by Tobias Diez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

This would also mean that we vouch to support these systems (otherwise, what's the point of these tests?).

No, this ticket (and all of tox.ini) does not come with a promise that we support a particular system configuration.

It only makes it easy to test it.

But why do you want to test something (and spend resources on implementing and maintaining such tests) if there is no intention to support this particular config?

comment:13 Changed 12 months ago by Matthias Köppe

Once we can easily test if something works, a decision can be made whether to support it.

comment:14 Changed 12 months ago by Matthias Köppe

Also, this infrastructure is now used by several upstream projects for portability testing; most recent add: https://github.com/pypa/setuptools/pull/2923

comment:15 in reply to:  13 ; Changed 12 months ago by Tobias Diez

Replying to mkoeppe:

Once we can easily test if something works, a decision can be made whether to support it.

I think you have proven very successfully that the existing infrastructure can easily be extended to incorporate new system configs. Thus, the important question is if we want to support a certain config, and not if its technically possible to test it. I don't have enough experience with the linux environment to judge if custom gcc installations are common enough on recent systems. However, a common theme in the discussion about removing the gcc package was that's a pain in the * * to maintain different configs. So I think there should be good reasons to add another config to the already pretty long list of variations.

comment:16 Changed 12 months ago by Matthias Köppe

Let me rephrase for clarity:

Once we can easily test if something works on a platform, a decision can be made whether to support SageMath on this platform.

Last edited 12 months ago by Matthias Köppe (previous) (diff)

comment:17 Changed 12 months ago by Dima Pasechnik

As we saw recently a few times, such obsolete systems, where one needs such non-standard toolchains to get anywhere, are well-supported by conda.

Direct support of such configurations is not buying us much.

comment:18 in reply to:  15 Changed 12 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe: a common theme in the discussion about removing the gcc package was that's a pain in the * * to maintain different configs. So I think there should be good reasons to add another config to the already pretty long list of variations.

No, it sounds like you have misunderstood the discussion. We want people to be able to use GCC (and other toolchain components) from their distribution -- to reduce our maintenance burden for our project regarding the gcc spkg.

comment:19 Changed 12 months ago by Tobias Diez

I think a long-term idea of what systems to support and in which way would be preferably. For example, numpy has a clear strategy where they recommend people to install it via conda and are relatively restrictive on which compilers they support for from-source installations. Not saying that sage should go the same path, but I think narrowing down the range of supported distribution channels would open up more resources for actual features and bug fixes.

Anyway, should we transfer this discussion to the dev mailing list?

comment:20 in reply to:  17 Changed 12 months ago by Matthias Köppe

I have made a plan for platform support in Sage 9.6, 9.7 in #32074 because major changes are coming from upstream.

comment:21 Changed 12 months ago by Matthias Köppe

The present ticket does not need more discussion. It's an improvement because it expands the platforms that we can easily test.

Being able to test is better than not being able to test.

comment:22 Changed 12 months ago by Dima Pasechnik

Sending build context to Docker daemon  519.8MB
Step 1/46 : ARG BASE_IMAGE=ubuntu:latest
Step 2/46 : FROM ${BASE_IMAGE} as with-system-packages
bionic: Pulling from library/ubuntu
68e7bb398b9f: Pull complete 
Digest: sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad
Status: Downloaded newer image for ubuntu:bionic
 ---> dcf4d4bef137
Step 3/46 : RUN DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes software-properties-common
 ---> Running in 600f78310b95
E: Unable to locate package software-properties-common
The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes software-properties-common' returned a non-zero code: 100
sha256:18cedbf3aa838b99f9bfd9684434a6f4ed1de5d8be02a579bcb51212082c839f
ERROR: InvocationError for command /bin/bash -c 'for docker_target in with-targets; do  BUILD_TAG=sage-docker-ubuntu-bionic-standard-toolchain-gcc_9-$docker_target:$(git describe --dirty --always);  DOCKER_BUILDKIT=0  docker build . -f /Users/dima/sage/.tox/docker-ubuntu-bionic-standard-toolchain-gcc_9/Dockerfile  --target $docker_target  --tag $BUILD_TAG  --build-arg EXTRA_CONFIGURE_ARGS="--enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes --with-system-gcc=force CC=gcc-9 CXX=g++-9 FC=gfortran-9 "  --build-arg BASE_IMAGE=ubuntu:bionic  --build-arg BOOTSTRAP="./bootstrap"  --build-arg TARGETS_PRE="$(if test -n "$TARGETS_PRE"; then echo $TARGETS_PRE; else echo all-sage-local; fi)"  --build-arg TARGETS="build"  --build-arg TARGETS_OPTIONAL="ptest"  ; status=$?;  if [ $status != 0 ]; then BUILD_TAG="$BUILD_TAG-failed"; docker commit $(docker ps -l -q) $BUILD_TAG; fi;  echo $BUILD_TAG >> /Users/dima/sage/.tox/docker-ubuntu-bionic-standard-toolchain-gcc_9/Dockertags;  if [ x"" != x ]; then echo Pushing $BUILD_TAG; docker push $BUILD_TAG || echo "(ignoring errors)"; fi;  if [ $status != 0 ]; then exit $status; fi; done' (exited with code 100)
_______________________________________________________ summary ________________________________________________________
ERROR:   docker-ubuntu-bionic-standard-toolchain-gcc_9: commands failed

comment:23 Changed 12 months ago by Dima Pasechnik

Step 1/46 : ARG BASE_IMAGE=ubuntu:latest
Step 2/46 : FROM ${BASE_IMAGE} as with-system-packages
 ---> dcf4d4bef137
Step 3/46 : RUN DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes software-properties-common
 ---> Using cache
 ---> 18cedbf3aa83
Step 4/46 : RUN add-apt-repository ppa:ubuntu-toolchain-r/test
 ---> Running in 8f18951793c2
/bin/sh: 1: add-apt-repository: not found
The command '/bin/sh -c add-apt-repository ppa:ubuntu-toolchain-r/test' returned a non-zero code: 127
sha256:96636bafcff5cb8b60716791c2613a2d9a01ad33e8bf5b825d0f0fd32caaaeb8
ERROR: InvocationError for command /bin/bash -c 'for docker_target in with-targets; do  BUILD_TAG=sage-docker-ubuntu-bionic-standard-toolchain-gcc_10-$docker_target:$(git describe --dirty --always);  DOCKER_BUILDKIT=0  docker build . -f /Users/dima/sage/.tox/docker-ubuntu-bionic-standard-toolchain-gcc_10/Dockerfile  --target $docker_target  --tag $BUILD_TAG  --build-arg EXTRA_CONFIGURE_ARGS="--enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes --with-system-gcc=force CC=gcc-10 CXX=g++-10 FC=gfortran-10 "  --build-arg BASE_IMAGE=ubuntu:bionic  --build-arg BOOTSTRAP="./bootstrap"  --build-arg TARGETS_PRE="$(if test -n "$TARGETS_PRE"; then echo $TARGETS_PRE; else echo all-sage-local; fi)"  --build-arg TARGETS="build"  --build-arg TARGETS_OPTIONAL="ptest"  ; status=$?;  if [ $status != 0 ]; then BUILD_TAG="$BUILD_TAG-failed"; docker commit $(docker ps -l -q) $BUILD_TAG; fi;  echo $BUILD_TAG >> /Users/dima/sage/.tox/docker-ubuntu-bionic-standard-toolchain-gcc_10/Dockertags;  if [ x"" != x ]; then echo Pushing $BUILD_TAG; docker push $BUILD_TAG || echo "(ignoring errors)"; fi;  if [ $status != 0 ]; then exit $status; fi; done' (exited with code 127)
_______________________________________________________ summary ________________________________________________________
ERROR:   docker-ubuntu-bionic-standard-toolchain-gcc_10: commands failed

comment:24 Changed 12 months ago by Dima Pasechnik

although docker-ubuntu-bionic-standard-toolchain-gcc_11 fares better so far.

comment:25 Changed 12 months ago by Matthias Köppe

Looks like this needs an apt-get update first sometimes.

comment:26 Changed 12 months ago by git

Commit: de42909be8a8e9bf7261a2d528b0fafd5a382bb04e2d53621d0a2ab5ae8b8b30056b6d7644daf491

Branch pushed to git repo; I updated commit sha1. New commits:

4e2d536build/bin/write-dockerfile.sh (ubuntu): Run apt-get update before installing software-properties-common

comment:27 Changed 12 months ago by Dima Pasechnik

Can you rebase over the latest beta?


New commits:

4e2d536build/bin/write-dockerfile.sh (ubuntu): Run apt-get update before installing software-properties-common

comment:28 Changed 12 months ago by git

Commit: 4e2d53621d0a2ab5ae8b8b30056b6d7644daf49180057ba50ef98ff097b930efded426e5f51b6e1b

Branch pushed to git repo; I updated commit sha1. New commits:

80057baMerge tag '9.6.beta0' into t/32966/tox___gh_actions__add_ubuntu_variants_using_ppa_ubuntu_toolchain_r

comment:29 Changed 12 months ago by Dima Pasechnik

with docker-ubuntu-bionic-standard-toolchain-gcc_9, g++ dies building fplll (not such a surprise...) Here docker is running on a macOS 12.2 MacBook Pro.

  [fplll-5.4.1] error installing, exit status 1. End of log file:
  [fplll-5.4.1]   Makefile:1697: recipe for target 'enum-parallel/libfplll_la-enumlib_dim.40.lo' failed
  [fplll-5.4.1]   make[6]: *** [enum-parallel/libfplll_la-enumlib_dim.40.lo] Error 1
  [fplll-5.4.1]   /bin/bash ../libtool  --tag=CXX   --mode=compile g++-9 -std=gnu++11 -std=c++11 -DHAVE_CONFIG_H -I. -I..  -DFPLLL_DEFAULT_STRATEGY_PATH=\"/sage/local/share/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/sage/local/share/fplll/strategies/default.json\" -I./.. -I/sage/local/include/ -pthread -g -O2 -fPIC -I/sage/local/include/ -L/sage/local/lib -c -o enum-parallel/libfplll_la-enumlib_dim.20.lo `test -f 'enum-parallel/enumlib_dim.20.cpp' || echo './'`enum-parallel/enumlib_dim.20.cpp
  [fplll-5.4.1]   libtool: compile:  g++-9 -std=gnu++11 -std=c++11 -DHAVE_CONFIG_H -I. -I.. -DFPLLL_DEFAULT_STRATEGY_PATH=\"/sage/local/share/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/sage/local/share/fplll/strategies/default.json\" -I./.. -I/sage/local/include/ -pthread -g -O2 -fPIC -I/sage/local/include/ -L/sage/local/lib -c enum-parallel/enumlib_dim.20.cpp  -fPIC -DPIC -o enum-parallel/.libs/libfplll_la-enumlib_dim.20.o
  [fplll-5.4.1]   g++-9: fatal error: Killed signal terminated program cc1plus
  [fplll-5.4.1]   compilation terminated.

that's not something to fix here, it's just an artefact...

Last edited 12 months ago by Dima Pasechnik (previous) (diff)

comment:30 Changed 12 months ago by Dima Pasechnik

with docker-ubuntu-bionic-standard-toolchain-gcc_10, fflas_ffpack and fplll don't build.

with docker-ubuntu-bionic-standard-toolchain-gcc_11, fplll doesn't build.

Last edited 12 months ago by Dima Pasechnik (previous) (diff)

comment:31 Changed 12 months ago by Dima Pasechnik

It's also basically the same error

libtool: compile:  g++-11 -std=gnu++11 -std=c++11 -DHAVE_CONFIG_H -I. -I.. -DFPLLL_DEFAULT_STRATEGY_PATH=\"/sage/local/s
hare/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/sage/local/share/fplll/strategies/default.json\" -I./.. -I/sage/loca
l/include/ -pthread -g -O2 -fPIC -I/sage/local/include/ -L/sage/local/lib -c enum-parallel/enumlib_dim.70.cpp  -fPIC -DP
IC -o enum-parallel/.libs/libfplll_la-enumlib_dim.70.o
/bin/bash ../libtool  --tag=CXX   --mode=compile g++-11 -std=gnu++11 -std=c++11 -DHAVE_CONFIG_H -I. -I..  -DFPLLL_DEFAUL
T_STRATEGY_PATH=\"/sage/local/share/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/sage/local/share/fplll/strategies/def
ault.json\" -I./.. -I/sage/local/include/ -pthread -g -O2 -fPIC -I/sage/local/include/ -L/sage/local/lib -c -o enum-para
llel/libfplll_la-enumlib_dim.60.lo `test -f 'enum-parallel/enumlib_dim.60.cpp' || echo './'`enum-parallel/enumlib_dim.60
.cpp
libtool: compile:  g++-11 -std=gnu++11 -std=c++11 -DHAVE_CONFIG_H -I. -I.. -DFPLLL_DEFAULT_STRATEGY_PATH=\"/sage/local/s
hare/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/sage/local/share/fplll/strategies/default.json\" -I./.. -I/sage/loca
l/include/ -pthread -g -O2 -fPIC -I/sage/local/include/ -L/sage/local/lib -c enum-parallel/enumlib_dim.60.cpp  -fPIC -DP
IC -o enum-parallel/.libs/libfplll_la-enumlib_dim.60.o
g++-11: fatal error: Killed signal terminated program cc1plus
compilation terminated.

That's most probably due to incorrect installation of the toolchain, as I get (in the container):

root@aa964bbe7c63:/sage# g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04) 

EDIT - the latter is OK. Probably it runs out of RAM - indeed, running top inside container shows at some point

58565 root      20   0  117276  83472  23364 R  53.8  4.1   0:01.63 cc1                                                
58437 root      20   0  122516  89136  24412 R  51.5  4.4   0:02.01 cc1plus                                            
51528 root      20   0 1104248 991.5m  24356 R  49.2 49.9   0:34.28 cc1plus                                            
58678 root      20   0  180576 140236  17928 R  39.9  6.9   0:01.20 cc1plus                                            
58670 root      20   0  103460  70476  22816 R  35.2  3.5   0:01.06 cc1                                                
58833 root      20   0   92080  49640  17428 R   6.3  2.4   0:00.19 cc1plus    
  • no wonder on heavier C++ code it might get killed due to inability to get more RAM...

In my Docker settings I see it uses 2GB of RAM, that's way too little. let me try with more RAM, 8GB...

Last edited 12 months ago by Dima Pasechnik (previous) (diff)

comment:32 Changed 12 months ago by Dima Pasechnik

Description: modified (diff)

comment:33 Changed 12 months ago by Dima Pasechnik

OK, with bigger RAM, docker-ubuntu-bionic-standard-toolchain-gcc_11 builds fine. Sorry for noise.

Now, checking the rest.

comment:34 Changed 12 months ago by Dima Pasechnik

I looked at tags assigned, and saw e.g.

sage-docker-ubuntu-bionic-standard-toolchain-gcc_9-with-targets    9.2.rc2-12022-g9c4903c921

Why is it 9.2 there?

comment:35 Changed 12 months ago by Dima Pasechnik

gcc-12 obviously needs #33187, to have a chance of working. As well, tox -e docker-ubuntu-jammy-standard-toolchain-gcc_12 fails early on without the following patch:

comment:36 Changed 12 months ago by Dima Pasechnik

Also, on jammy recognition of libsingular fails, as it's /usr/lib/x86_64-linux-gnu/libsingular-Singular-4.2.1.so there, and so dlopen($LIBSINGULAR_PATH) test fails, as it expects libSingular.so. Potentially, the correct library name can be found by looking at the output of of pkg-config

# pkg-config --libs singular
-lsingular-Singular -lsingular-polys -lflint -lmpfr -lgmp -lsingular-factory -lflint -lmpfr -lgmp -lntl -lgmp -lsingular-omalloc -lsingular-resources

PS. - this is tracked in #32789

Last edited 12 months ago by Dima Pasechnik (previous) (diff)

comment:37 Changed 12 months ago by Dima Pasechnik

OK, subject to comment:35, all good.

comment:38 in reply to:  34 Changed 12 months ago by Matthias Köppe

Replying to dimpase:

I looked at tags assigned, and saw e.g.

sage-docker-ubuntu-bionic-standard-toolchain-gcc_9-with-targets    9.2.rc2-12022-g9c4903c921

Why is it 9.2 there?

That's just the output of git describe --dirty --always. It seems you haven't run git fetch --tags for a long time, so it expresses version as being 12022 commits ahead of 9.2.rc2

comment:39 in reply to:  35 ; Changed 12 months ago by Matthias Köppe

Replying to dimpase:

tox -e docker-ubuntu-jammy-standard-toolchain-gcc_12

fails early on without the following patch:

I'd guess this is just temporary breakage in their packaging. jammy is not released yet

comment:40 in reply to:  39 Changed 12 months ago by Dima Pasechnik

Replying to mkoeppe:

I'd guess this is just temporary breakage in their packaging. jammy is not released yet

it doesn't hurt to list these explicitly, anyway.

comment:41 Changed 12 months ago by Matthias Köppe

What are they needed for?

comment:42 in reply to:  41 ; Changed 12 months ago by Dima Pasechnik

Replying to mkoeppe:

What are they needed for?

without them, one gets an error about needing gpg-agent very early on - it seems that apt is using it for the gcc_12 repo (i.e. for ppa:ubuntu-toolchain-r/test).

Last edited 12 months ago by Dima Pasechnik (previous) (diff)

comment:43 in reply to:  42 Changed 12 months ago by Dima Pasechnik

Replying to dimpase:

Replying to mkoeppe:

What are they needed for?

without them, one gets an error about needing gpg-agent very early on - it seems that apt is using it for the gcc_12 repo (i.e. for ppa:ubuntu-toolchain-r/test).

...
Running hooks in /etc/ca-certificates/update.d...
done.
Processing triggers for dbus (1.12.20-2ubuntu2) ...
Removing intermediate container 97b378424ff4
 ---> a7905d898dc3
Step 4/46 : RUN add-apt-repository ppa:ubuntu-toolchain-r/test
 ---> Running in 089462f9b5bb
gpg: error running '/usr/bin/gpg-agent': probably not installed
gpg: failed to start agent '/usr/bin/gpg-agent': Configuration error
gpg: can't connect to the agent: Configuration error
PPA publishes dbgsym, you may need to include 'main/debug' component
Repository: 'deb https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu/ jammy main'
Description:
Toolchain test builds; see https://wiki.ubuntu.com/ToolChain

More info: https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
Adding repository.
Adding deb entry to /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-jammy.list
Adding disabled deb-src entry to /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-jammy.list
Adding key to /etc/apt/trusted.gpg.d/ubuntu-toolchain-r-ubuntu-test.gpg with fingerprint 60C317803A41BA51845E371A1E9377A2BA9EF27F
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/softwareproperties/shortcuthandler.py", line 423, in add_key
    subprocess.run(cmd.split(), check=True, input=keys)
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['gpg', '-q', '--no-options', '--no-default-keyring', '--batch', '--keyring', '/etc/apt/trusted.gpg.d/ubuntu-toolchain-r-ubuntu-test.gpg', '--homedir', '/tmp/tmpxpjb8_ko', '--import']' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/add-apt-repository", line 364, in <module>
    sys.exit(0 if addaptrepo.main() else 1)
  File "/usr/bin/add-apt-repository", line 357, in main
    shortcut.add()
  File "/usr/lib/python3/dist-packages/softwareproperties/shortcuthandler.py", line 222, in add
    self.add_key()
  File "/usr/lib/python3/dist-packages/softwareproperties/shortcuthandler.py", line 425, in add_key
    raise ShortcutException(e)
softwareproperties.shortcuthandler.ShortcutException: Command '['gpg', '-q', '--no-options', '--no-default-keyring', '--batch', '--keyring', '/etc/apt/trusted.gpg.d/ubuntu-toolchain-r-ubuntu-test.gpg', '--homedir', '/tmp/tmpxpjb8_ko', '--import']' returned non-zero exit status 2.
The command '/bin/sh -c add-apt-repository ppa:ubuntu-toolchain-r/test' returned a non-zero code: 1
sha256:33b87097694e74588596c1c4562ad74acf4538864dc306022e8a29465c38099e
ERROR: InvocationError for command /bin/bash -c 'for docker_target in with-targets; do  BUILD_TAG=sage-docker-ubuntu-jammy-standard-toolchain-gcc_12-$docker_target:$(git describe --dirty --always);  DOCKER_BUILDKIT=0  docker build . -f /Users/dima/sage/.tox/docker-ubuntu-jammy-standard-toolchain-gcc_12/Dockerfile  --target $docker_target  --tag $BUILD_TAG  --build-arg EXTRA_CONFIGURE_ARGS="--enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes --with-system-gcc=force CC=gcc-12 CXX=g++-12 FC=gfortran-12 "  --build-arg BASE_IMAGE=ubuntu:jammy  --build-arg BOOTSTRAP="./bootstrap"  --build-arg TARGETS_PRE="$(if test -n "$TARGETS_PRE"; then echo $TARGETS_PRE; else echo all-sage-local; fi)"  --build-arg TARGETS="build"  --build-arg TARGETS_OPTIONAL="ptest"  ; status=$?;  if [ $status != 0 ]; then BUILD_TAG="$BUILD_TAG-failed"; docker commit $(docker ps -l -q) $BUILD_TAG; fi;  echo $BUILD_TAG >> /Users/dima/sage/.tox/docker-ubuntu-jammy-standard-toolchain-gcc_12/Dockertags;  if [ x"" != x ]; then echo Pushing $BUILD_TAG; docker push $BUILD_TAG || echo "(ignoring errors)"; fi;  if [ $status != 0 ]; then exit $status; fi; done' (exited with code 1)
_______________________________________________________ summary ________________________________________________________
ERROR:   docker-ubuntu-jammy-standard-toolchain-gcc_12: commands failed

I guess gpg/gpg-agent is used for signature verification of that ppa.

comment:44 Changed 12 months ago by git

Commit: 80057ba50ef98ff097b930efded426e5f51b6e1b6bc45ce6cae778073e5e205f11a5e2834bc5fa35

Branch pushed to git repo; I updated commit sha1. New commits:

6bc45cebuild/bin/write-dockerfile.sh (debian) [EXTRA_REPOSITORY]: Also install gpg, gpg-agent

comment:45 Changed 12 months ago by Matthias Köppe

Thanks for catching this!

comment:46 Changed 12 months ago by Matthias Köppe

Dependencies: #33187

comment:47 Changed 12 months ago by git

Commit: 6bc45ce6cae778073e5e205f11a5e2834bc5fa35459749bac30503f16b84abe2ddb00410a164a85b

Branch pushed to git repo; I updated commit sha1. New commits:

07bbd64build/pkgs/{gcc,gfortran}/spkg-configure.m4: Accept 12.x
734ba23build/pkgs/gcc/spkg-configure.m4: Reject GCC 4.9.x
35857d8build/pkgs/singular: Add upstream PR 1128 for GCC 12 support
f27ff17build/pkgs/giac/spkg-install.in: Use -std=gnu++11
459749bMerge #33187

comment:48 in reply to:  35 Changed 12 months ago by Matthias Köppe

Replying to dimpase:

gcc-12 obviously needs #33187, to have a chance of working.

OK, I have made it a dependency

comment:49 Changed 12 months ago by git

Commit: 459749bac30503f16b84abe2ddb00410a164a85b983b4a85964362e7545012d331323c39a9e2f55e

Branch pushed to git repo; I updated commit sha1. New commits:

983b4a8src/doc/en/developer/portability_testing.rst: Document RAM needs for Docker containers

comment:50 Changed 12 months ago by Dima Pasechnik

Reviewers: Dima Pasechnik
Status: needs_reviewpositive_review

OK, it's good to go.

comment:51 Changed 12 months ago by Matthias Köppe

Thank you!

comment:52 Changed 12 months ago by Matthias Köppe

Priority: majorcritical

comment:53 Changed 12 months ago by Volker Braun

Status: positive_reviewneeds_work

Merge failure on top of:

b34c0d5ded Trac #31529: texlive: Add spkg-configure.m4 and system package information; remove install script

c4e4a6387f Trac #31415: GH Actions workflow that builds documentation on each push

9cc853b81b Trac #33107: Generic cholesky() fails on the trivial matrix

88a98d50c3 Trac #33103: gitpod integration using Docker images from portability testing workflow

d4466d825b Trac #33102: some details in shuffle and multizeta

09de266470 Trac #33098: Add more void packages to distros/void.txt

73dd8d1cf5 Trac #33096: src/tox.ini (relint): Exclude editor temp files etc.

a0d0888f54 Trac #33095: again some details in Dyck words

4fa191a754 Trac #33071: Update pillow to 8.4

733ed250a3 Trac #33070: Package upgrades for Python 3.10 support

e4f3f1dc9b Trac #33069: Remove spkg speaklater

18ba1d8cdd Trac #33064: sage_docbuild: fails when cache cannot be saved

467c32cce3 Trac #33049: Update scipy to 1.7.3

2cf334f97d Trac #33048: Update setuptools to 59.8.0

3a4cc169b4 Trac #33045: ffmpeg tests time out

8fb89a37bb Trac #33039: Random doctest failure in continued_fraction_gosper

236856d0da Trac #33026: make BinaryQF.solve_integer() work for square discriminants

2ad7305f87 Trac #33015: Random failure in number_field_element.pyx

c45f12e9e6 Trac #33009: Move is_prism and is_bipyramid to combinatorial polyhedron

87011e965e Trac #33008: Polyhedron_base.facet_adjacency_matrix: Do not use face_lattice

46cc022732 Trac #33007: Remove imports from sage.interfaces.all

57fe9e661a Trac #33005: Add feature for pdftocairo

69d9945eb0 Trac #32993: add pictures to line.py documentation

3834fded18 Trac #32788: Permanently get rid of bare except: statements

0207af8674 Trac #32209: Allow evaluation of morphisms of schemes on points over different rings

206ff0b058 Trac #33269: fix for build/pkgs/tox/distros/gentoo.txt

6d154b634f Trac #33226: Support giac-1.7.0.45 in doctests

d5d2e23df5 Trac #33187: Allow gcc/gfortran 12

59c0620436 Trac #33140: GH Actions (macOS): Fix filtering of experimental packages

0669c95412 Trac #32211: Upgrade FLINT to 2.8.4, arb to 2.22.1

508ea2d518 Trac #33284: Fix macro name clash introduced in #25633

b511685469 Trac #32989: Remove imports from sage.misc.all

9c6afb97c8 Trac #32505: Finitely presented graded modules over graded connected algebras

226937dfa9 Updated SageMath version to 9.6.beta0

merge was not clean: conflicts in build/bin/write-dockerfile.sh

comment:54 Changed 12 months ago by Matthias Köppe

Dependencies: #33187#33187, #33103

comment:55 Changed 12 months ago by git

Commit: 983b4a85964362e7545012d331323c39a9e2f55ea78c10907c3950c934a07765a51be5f50e02ab03

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

3605076build/pkgs/sagelib/spkg-install: Fix up deleting temp.*
d87027csage --package list: Add option --exclude
c02c03ctox.ini (gitpod-standard): Exclude some packages
6285cd9.gitpod.Dockerfile: Save space by removing apt lists
7e74607.gitpod.Dockerfile: Remove built Sage documentation
f71c8b6docker/.gitpod.Dockerfile: Clean more
f953228Merge tag '9.5.rc0' into t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflow
2d0902asrc/doc/en/developer/workspace.rst: Fix markup
078f247.github/workflows/tox*.yml: Transform GITHUB_REF_NAME to a valid docker tag
a78c109Merge #33103

comment:56 Changed 12 months ago by Matthias Köppe

Status: needs_workneeds_review

comment:57 Changed 12 months ago by Matthias Köppe

Status: needs_reviewneeds_work

Just noticed that now tox -e docker-ubuntu-xenial-standard-toolchain-gcc_9 -- config.status gives

E: Package 'gpg' has no installation candidate
E: Unable to locate package gpg-agent
The command '/bin/sh -c apt-get update &&  DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes software-properties-common gpg gpg-agent' returned a non-zero code: 100

comment:58 Changed 12 months ago by git

Commit: a78c10907c3950c934a07765a51be5f50e02ab03f98e0dc0c94687344589387bf093445fef9bb6c7

Branch pushed to git repo; I updated commit sha1. New commits:

f98e0dcbuild/bin/write-dockerfile.sh (debian) [EXTRA_REPOSITORY]: Do not fail if gpg, gpg-agent cannot be installed

comment:59 Changed 12 months ago by Matthias Köppe

Status: needs_workneeds_review

comment:60 in reply to:  26 ; Changed 12 months ago by Dima Pasechnik

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

4e2d536build/bin/write-dockerfile.sh (ubuntu): Run apt-get update before installing software-properties-common

you forgot this one, and without it one gets an error early on.

comment:61 in reply to:  60 Changed 12 months ago by Dima Pasechnik

Replying to dimpase:

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

4e2d536build/bin/write-dockerfile.sh (ubuntu): Run apt-get update before installing software-properties-common

you forgot this one, and without it one gets an error early on.

oops, sorry, I messed up my branches. It's good here. Still testing.

comment:62 Changed 12 months ago by Dima Pasechnik

Please add wiping up /etc/dpkg/dpkg.cfg.d/excludes in handling of Ubuntu images, otherwise apt will drop stuff there, see https://trac.sagemath.org/ticket/32789#comment:61 (and https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1960749 mentioned there)

comment:63 Changed 12 months ago by Matthias Köppe

I've pushed this change to the branch of #32789

comment:64 Changed 12 months ago by Dima Pasechnik

Status: needs_reviewpositive_review

OK, lgtm

comment:65 Changed 12 months ago by Matthias Köppe

Thanks!

comment:66 Changed 11 months ago by Dima Pasechnik

needs a rebase

comment:67 Changed 11 months ago by git

Commit: f98e0dc0c94687344589387bf093445fef9bb6c7d42bc03b1a39736de8534dcf3a05c2b02ec3ce89
Status: positive_reviewneeds_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

d42bc03Merge tag '9.6.beta1' into t/32966/tox___gh_actions__add_ubuntu_variants_using_ppa_ubuntu_toolchain_r

comment:68 Changed 11 months ago by Matthias Köppe

Status: needs_reviewpositive_review

comment:69 Changed 11 months ago by Volker Braun

Branch: u/mkoeppe/tox___gh_actions__add_ubuntu_variants_using_ppa_ubuntu_toolchain_rd42bc03b1a39736de8534dcf3a05c2b02ec3ce89
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.