archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages

[mkoeppe]

​https://groups.google.com/d/msg/sage-devel/JgArPVx7dwg/gMxeB4b7AgAJ

To test:

  tox -e docker-archlinux-latest-standard

[dimpase]

not sure there is much on Sage's side that can be done - apart from keeping correct lists of packages to install.

[vdelecroix]

Indeed, I would like to be sure that it is not on Sage side and have an up to date developer manual for the relevant packages.

[mkoeppe]

In case you haven't seen it yet: the system package lists in the developer's manual are now autogenerated. But adding some Arch specific instructions would be a useful addition

[mkoeppe]

Installation manual, that is

[dimpase]

Well, config.log is clear enough on e.g. what's missing w.r.t. Pari: namely, install elldata, galpol, seadata, as seen in

configure:14982: checking is pari_elldata installed?
1735	  ***   at top-level: r=ellinit("11a1");r[11]
1736	  ***                   ^---------------------
1737	  *** ellinit: error opening elldata file: `/usr/share/pari/elldata/ell0'.
1738	configure:14989: result: no; cannot use system pari/GP without elldata package
1739	configure:14991: Install elldata package and reconfigure.
1740	configure:14993: Otherwise Sage will build its own pari/GP.
1741	configure:14997: checking is pari_galdata installed?
1742	configure:15001: result: yes
1743	configure:15012: checking is pari_galpol installed?
1744	  ***   at top-level: galoisgetname(12,1)=="C3 : C4"
1745	  ***                 ^------------------------------
1746	  *** galoisgetname: error opening galpol file: `/usr/share/pari/galpol/12/nb'.
1747	configure:15019: result: no; cannot use system pari/GP without galpol package
1748	configure:15021: Install galpol package and reconfigure.
1749	configure:15023: Otherwise Sage will build its own pari/GP.
1750	configure:15027: checking is pari_seadata installed?
1751	  ***   at top-level: poldegree(ellmodulareqn(211)[1])
1752	  ***                           ^----------------------
1753	  *** ellmodulareqn: error opening seadata file: `/usr/share/pari/seadata/sea211'.
1754	configure:15034: result: no; cannot use system pari/GP without seadata package
1755	configure:15036: Install seadata package and reconfigure.
1756	configure:15038: Otherwise Sage will build its own pari/GP.
1757	configure:15266: no suitable system package found for SPKG pari
1758	configure:15284: result: using Sage's pari SPKG

[arojas]

Nothing to fix on Arch side either - givaro is not used because you have an older version on /usr/local which is detected first, and pari is not used because of missing optional packages

[mkoeppe]

Could you also take a look at the build logs at ​https://github.com/mkoeppe/sage/runs/518106289 please?

[arojas]

Replying to mkoeppe:

Could you also take a look at the build logs at ​https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled. Other packages (fplll, ecm) are simply not installed on the build machine.

[mkoeppe]

Replying to arojas:

Replying to mkoeppe:

Could you also take a look at the build logs at ​https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

Other packages (fplll, ecm) are simply not installed on the build machine.

OK, this means we need to add arch.txt files for these packages

[mkoeppe]

and for more of the following listed here:

configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll git givaro gp2c isl lcalc libatomic_ops libsemigroups mpfi ninja_build ntl pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_term_readline_gnu yasm zeromq

[dimpase]

Replying to mkoeppe:

Replying to arojas:

Replying to mkoeppe:

Could you also take a look at the build logs at ​https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

I believe that I was able to use Arch's NTL in builds few betas ago. This again points at #29339 as being too drastic in its approach.

[arojas]

Replying to dimpase:

Replying to mkoeppe:

Replying to arojas:

Replying to mkoeppe:

Could you also take a look at the build logs at ​https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

I believe that I was able to use Arch's NTL in builds few betas ago. This again points at #29339 as being too drastic in its approach.

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

[mkoeppe]

Replying to arojas:

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

Well, it is currently disabled as of last NTL upgrade #20590.

[arojas]

Replying to mkoeppe:

Replying to arojas:

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

Well, it is currently disabled as of last NTL upgrade #20590.

I mean the Arch sagemath package, not sage-the-distro

[mkoeppe]

Ah, thanks for the clarification

[vdelecroix]

Replying to mkoeppe:

and for more of the following listed here:

configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll git givaro gp2c isl lcalc libatomic_ops libsemigroups mpfi ninja_build ntl pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_term_readline_gnu yasm zeromq

Note that the packages pari-seadata and pari-seadata-small conflict.

After removing the pkgconfig file of givaro in /usr/local (my bad) and installing the three pari packages (pari-seadata, pari-galpol, pari-elldata), everything is now taken from the system! Thanks.

---

New commits:

​9a21aa7

build/pkgs/readline/distros/arch.txt: New

[mkoeppe]

So what are we missing in our arch.txt files?

[vdelecroix]

I don't understand what these arch.txt are supposed to say. It looks like a kind of translation sage package -> arch package. However, for pari to be recognized by the sagemath configure, it needs to be installed with the optional packages pari-seadata, pari-galpol, pari-elldata. Where is this specified?

[dimpase]

Here is an example from build/pkgs/pari/distros/debian.txt:

pari-gp2c libpari-dev
# #29319: cypari2 needs gphelp at installation time
pari-doc
# We add these data packages because they are checked by spkg-configure.m4
pari-elldata pari-galdata pari-galpol pari-seadata

Put the equivalent of these Debian package names into build/pkgs/pari/distros/arch.txt

[vdelecroix]

Where is the format documented?

[dimpase]

As many things in our build system, nowhere, AFAIK - unless I miss a ticket that didn't make it into Sage 9.1.beta8 (the renaming spkg-install -> spkg-install.in did make it there, though). Please feel free to add things to Developer Guide, in src/doc/en/developer/packaging.rst

[dimpase]

these files are processed by src/doc/bootstrap called from ./bootstrap.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2acd21d

src/doc/bootstrap: Also process arch.txt

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9a21aa7	build/pkgs/readline/distros/arch.txt: New
​2acd21d	src/doc/bootstrap: Also process arch.txt

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​187761e

packages for pari

[vdelecroix]

Also, both pari-seadata and pari-elldata are from ​AUR.

[mkoeppe]

$ tox -e docker-archlinux-latest-standard
docker-archlinux-latest-standard run-test-pre: PYTHONHASHSEED='2835960636'
docker-archlinux-latest-standard run-test: commands[0] | bash -c 'build/bin/write-dockerfile.sh arch "standard" yes no > /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/docker-archlinux-latest-standard/Dockerfile'
docker-archlinux-latest-standard run-test: commands[1] | docker build . -f /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/docker-archlinux-latest-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS= --build-arg BASE_IMAGE=archlinux:latest --build-arg http_proxy=http://host.docker.internal:3128 --build-arg https_proxy=http://host.docker.internal:3128
Sending build context to Docker daemon  179.1MB
Step 1/23 : ARG BASE_IMAGE=archlinux:latest
Step 2/23 : FROM ${BASE_IMAGE}
 ---> 0152bf6f0800
Step 3/23 : ENV PACKAGES="binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon"
 ---> Using cache
 ---> 61ceb4b4c200
Step 4/23 : RUN  pacman -Syu --noconfirm binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon
 ---> Running in a4547b23d1cc
:: Synchronizing package databases...
downloading core.db...
downloading extra.db...
downloading community.db...
warning: perl-5.30.1-1 is up to date -- reinstalling
warning: tar-1.32-3 is up to date -- reinstalling
error: target not found: pari-seadata
error: target not found: pari-elldata
The command '/bin/sh -c pacman -Syu --noconfirm binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon' returned a non-zero code: 1
ERROR: InvocationError for command /usr/local/bin/docker build . -f .tox/docker-archlinux-latest-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS= --build-arg BASE_IMAGE=archlinux:latest --build-arg http_proxy=http://host.docker.internal:3128 --build-arg https_proxy=http://host.docker.internal:3128 (exited with code 1)

[dimpase]

pacman cannot install packages from AUR, they need to be built from source: ​https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages

(in case of these packages "building" is instant though, it just needs different commands to be run)

Yes, it'd be possible to script this, so these packages need to at least be tagged somehow.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8fdaa9e	build/pkgs/pari/distros/arch.txt: Comment out these packages (uninstallable)
​4cd1b55	src/doc/en/installation/source.rst: Include arch*.txt

[mkoeppe]

This is not complete but let's get it into 9.1

[dimpase]

there are replacements for pacman which allow automatic install of packages from AUR and the standard repo, such as yay.

so the script can first install yay, and then use it for the rest.

[arojas]

Replying to dimpase:

there are replacements for pacman which allow automatic install of packages from AUR and the standard repo, such as yay.

so the script can first install yay, and then use it for the rest.

Please don't. AUR is unsupported, user generated content. There's no guarantee that it couldn't be replaced with broken or even malicious content at any time.

If necessary, I could move these packages to the binary repos. But I don't really see the need to check for those at build time, at least on Arch, given that they can be installed by the user at any time after Sage install.

[dimpase]

well, if we recommend AUR packages, it's better if we test with them too, no? But, indeed, if they can be moved to the regular package repo, why not?

[arojas]

Alright, packages are now in the official repos

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​5a62319	build/pkgs/readline/distros/arch.txt: New
​1236a4e	src/doc/bootstrap: Also process arch.txt
​5e203e9	packages for pari
​ba6f6f9	build/pkgs/pari/distros/arch.txt: Comment out these packages (uninstallable)
​cefaf09	src/doc/en/installation/source.rst: Include arch*.txt
​338468f	build/pkgs/pari*/distros/arch.txt: Update/add

[mkoeppe]

Replying to arojas:

Alright, packages are now in the official repos

Thanks very much, that's great.

configure now finds the packages!

tox -e docker-archlinux-latest-standard -- base-toolchain
...
Checking whether SageMath should install SPKG pari...
checking installing gmp/mpir or readline? ... no
checking installing PARI/GP packages? ... checking for gp... /usr/sbin/gp
checking for gphelp... /usr/sbin/gphelp
checking whether gphelp has access to the documentation... yes
checking is pari_elldata installed? ... yes
checking is pari_galdata installed? ... yes
checking is pari_galpol installed? ... yes
checking is pari_seadata installed? ... yes
configure: will use system package and not install SPKG pari
using pari/gp from the system

Needs review.

[dimpase]

lgtm

[mkoeppe]

Thank you!