#29366 closed enhancement (fixed)

archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.1
Component: porting Keywords:
Cc: vdelecroix, arojas, dimpase Merged in:
Authors: Matthias Koeppe, Vincent Delecroix Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 338468f (Commits, GitHub, GitLab) Commit: 338468facda0ba5aef8ab8c505bdcaa4e47efb7b
Dependencies: Stopgaps:

Status badges

Attachments (2)

config.log (213.6 KB) - added by vdelecroix 18 months ago.
configure-output.log (61.2 KB) - added by vdelecroix 18 months ago.

Download all attachments as: .zip

Change History (48)

comment:1 Changed 18 months ago by mkoeppe

  • Cc dimpase added

comment:2 Changed 18 months ago by mkoeppe

  • Summary changed from archlinux: Fix detection detection of system packages to archlinux: Fix detection of system packages

Changed 18 months ago by vdelecroix

comment:3 Changed 18 months ago by dimpase

not sure there is much on Sage's side that can be done - apart from keeping correct lists of packages to install.

comment:4 Changed 18 months ago by vdelecroix

Indeed, I would like to be sure that it is not on Sage side and have an up to date developer manual for the relevant packages.

Changed 18 months ago by vdelecroix

comment:5 Changed 18 months ago by mkoeppe

In case you haven't seen it yet: the system package lists in the developer's manual are now autogenerated. But adding some Arch specific instructions would be a useful addition

comment:6 Changed 18 months ago by mkoeppe

Installation manual, that is

comment:7 Changed 18 months ago by dimpase

Well, config.log is clear enough on e.g. what's missing w.r.t. Pari: namely, install elldata, galpol, seadata, as seen in

configure:14982: checking is pari_elldata installed?
1735	  ***   at top-level: r=ellinit("11a1");r[11]
1736	  ***                   ^---------------------
1737	  *** ellinit: error opening elldata file: `/usr/share/pari/elldata/ell0'.
1738	configure:14989: result: no; cannot use system pari/GP without elldata package
1739	configure:14991: Install elldata package and reconfigure.
1740	configure:14993: Otherwise Sage will build its own pari/GP.
1741	configure:14997: checking is pari_galdata installed?
1742	configure:15001: result: yes
1743	configure:15012: checking is pari_galpol installed?
1744	  ***   at top-level: galoisgetname(12,1)=="C3 : C4"
1745	  ***                 ^------------------------------
1746	  *** galoisgetname: error opening galpol file: `/usr/share/pari/galpol/12/nb'.
1747	configure:15019: result: no; cannot use system pari/GP without galpol package
1748	configure:15021: Install galpol package and reconfigure.
1749	configure:15023: Otherwise Sage will build its own pari/GP.
1750	configure:15027: checking is pari_seadata installed?
1751	  ***   at top-level: poldegree(ellmodulareqn(211)[1])
1752	  ***                           ^----------------------
1753	  *** ellmodulareqn: error opening seadata file: `/usr/share/pari/seadata/sea211'.
1754	configure:15034: result: no; cannot use system pari/GP without seadata package
1755	configure:15036: Install seadata package and reconfigure.
1756	configure:15038: Otherwise Sage will build its own pari/GP.
1757	configure:15266: no suitable system package found for SPKG pari
1758	configure:15284: result: using Sage's pari SPKG
Last edited 18 months ago by dimpase (previous) (diff)

comment:8 Changed 18 months ago by arojas

Nothing to fix on Arch side either - givaro is not used because you have an older version on /usr/local which is detected first, and pari is not used because of missing optional packages

comment:9 follow-up: Changed 18 months ago by mkoeppe

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

comment:10 in reply to: ↑ 9 ; follow-up: Changed 18 months ago by arojas

Replying to mkoeppe:

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled. Other packages (fplll, ecm) are simply not installed on the build machine.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 18 months ago by mkoeppe

Replying to arojas:

Replying to mkoeppe:

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

Other packages (fplll, ecm) are simply not installed on the build machine.

OK, this means we need to add arch.txt files for these packages

comment:12 follow-up: Changed 18 months ago by mkoeppe

and for more of the following listed here:

configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll git givaro gp2c isl lcalc libatomic_ops libsemigroups mpfi ninja_build ntl pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_term_readline_gnu yasm zeromq

comment:13 in reply to: ↑ 11 ; follow-up: Changed 18 months ago by dimpase

Replying to mkoeppe:

Replying to arojas:

Replying to mkoeppe:

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

I believe that I was able to use Arch's NTL in builds few betas ago. This again points at #29339 as being too drastic in its approach.

comment:14 in reply to: ↑ 13 ; follow-up: Changed 18 months ago by arojas

Replying to dimpase:

Replying to mkoeppe:

Replying to arojas:

Replying to mkoeppe:

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

I believe that I was able to use Arch's NTL in builds few betas ago. This again points at #29339 as being too drastic in its approach.

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

comment:15 in reply to: ↑ 14 ; follow-up: Changed 18 months ago by mkoeppe

Replying to arojas:

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

Well, it is currently disabled as of last NTL upgrade #20590.

comment:16 in reply to: ↑ 15 Changed 18 months ago by arojas

Replying to mkoeppe:

Replying to arojas:

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

Well, it is currently disabled as of last NTL upgrade #20590.

I mean the Arch sagemath package, not sage-the-distro

comment:17 Changed 18 months ago by mkoeppe

Ah, thanks for the clarification

comment:18 Changed 18 months ago by mkoeppe

  • Summary changed from archlinux: Fix detection of system packages to archlinux: Add missing system packages (arch.txt), fix detection of system packages

comment:19 Changed 18 months ago by mkoeppe

  • Summary changed from archlinux: Add missing system packages (arch.txt), fix detection of system packages to archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages

comment:20 Changed 18 months ago by mkoeppe

  • Branch set to u/mkoeppe/archlinux__add_missing_system_packages__arch_txt___add_arch_section_to_installation_manual__fix_detection_of_system_packages

comment:21 in reply to: ↑ 12 Changed 18 months ago by vdelecroix

  • Commit set to 9a21aa77889fb4cababf0edde195bec499164b71

Replying to mkoeppe:

and for more of the following listed here:

configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll git givaro gp2c isl lcalc libatomic_ops libsemigroups mpfi ninja_build ntl pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_term_readline_gnu yasm zeromq

Note that the packages pari-seadata and pari-seadata-small conflict.

After removing the pkgconfig file of givaro in /usr/local (my bad) and installing the three pari packages (pari-seadata, pari-galpol, pari-elldata), everything is now taken from the system! Thanks.


New commits:

9a21aa7build/pkgs/readline/distros/arch.txt: New

comment:22 Changed 18 months ago by mkoeppe

So what are we missing in our arch.txt files?

comment:23 Changed 18 months ago by vdelecroix

I don't understand what these arch.txt are supposed to say. It looks like a kind of translation sage package -> arch package. However, for pari to be recognized by the sagemath configure, it needs to be installed with the optional packages pari-seadata, pari-galpol, pari-elldata. Where is this specified?

comment:24 Changed 18 months ago by dimpase

Here is an example from build/pkgs/pari/distros/debian.txt:

pari-gp2c libpari-dev
# #29319: cypari2 needs gphelp at installation time
pari-doc
# We add these data packages because they are checked by spkg-configure.m4
pari-elldata pari-galdata pari-galpol pari-seadata

Put the equivalent of these Debian package names into build/pkgs/pari/distros/arch.txt

comment:25 Changed 18 months ago by vdelecroix

Where is the format documented?

comment:26 Changed 18 months ago by dimpase

As many things in our build system, nowhere, AFAIK - unless I miss a ticket that didn't make it into Sage 9.1.beta8 (the renaming spkg-install -> spkg-install.in did make it there, though). Please feel free to add things to Developer Guide, in src/doc/en/developer/packaging.rst

comment:27 Changed 18 months ago by dimpase

these files are processed by src/doc/bootstrap called from ./bootstrap.

comment:28 Changed 18 months ago by git

  • Commit changed from 9a21aa77889fb4cababf0edde195bec499164b71 to 2acd21d61ee2a2f9bfb5c9e0e3abbabec8d9f7a1

Branch pushed to git repo; I updated commit sha1. New commits:

2acd21dsrc/doc/bootstrap: Also process arch.txt

comment:29 Changed 18 months ago by vdelecroix

  • Branch changed from u/mkoeppe/archlinux__add_missing_system_packages__arch_txt___add_arch_section_to_installation_manual__fix_detection_of_system_packages to public/29366
  • Commit 2acd21d61ee2a2f9bfb5c9e0e3abbabec8d9f7a1 deleted

comment:30 Changed 18 months ago by git

  • Commit set to 2acd21d61ee2a2f9bfb5c9e0e3abbabec8d9f7a1

Branch pushed to git repo; I updated commit sha1. New commits:

9a21aa7build/pkgs/readline/distros/arch.txt: New
2acd21dsrc/doc/bootstrap: Also process arch.txt

comment:31 Changed 18 months ago by git

  • Commit changed from 2acd21d61ee2a2f9bfb5c9e0e3abbabec8d9f7a1 to 187761e1d2a00eb75bb50eef1cac97f8aa807201

Branch pushed to git repo; I updated commit sha1. New commits:

187761epackages for pari

comment:32 Changed 18 months ago by vdelecroix

Also, both pari-seadata and pari-elldata are from AUR.

comment:33 Changed 18 months ago by mkoeppe

$ tox -e docker-archlinux-latest-standard
docker-archlinux-latest-standard run-test-pre: PYTHONHASHSEED='2835960636'
docker-archlinux-latest-standard run-test: commands[0] | bash -c 'build/bin/write-dockerfile.sh arch "standard" yes no > /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/docker-archlinux-latest-standard/Dockerfile'
docker-archlinux-latest-standard run-test: commands[1] | docker build . -f /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/docker-archlinux-latest-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS= --build-arg BASE_IMAGE=archlinux:latest --build-arg http_proxy=http://host.docker.internal:3128 --build-arg https_proxy=http://host.docker.internal:3128
Sending build context to Docker daemon  179.1MB
Step 1/23 : ARG BASE_IMAGE=archlinux:latest
Step 2/23 : FROM ${BASE_IMAGE}
 ---> 0152bf6f0800
Step 3/23 : ENV PACKAGES="binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon"
 ---> Using cache
 ---> 61ceb4b4c200
Step 4/23 : RUN  pacman -Syu --noconfirm binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon
 ---> Running in a4547b23d1cc
:: Synchronizing package databases...
downloading core.db...
downloading extra.db...
downloading community.db...
warning: perl-5.30.1-1 is up to date -- reinstalling
warning: tar-1.32-3 is up to date -- reinstalling
error: target not found: pari-seadata
error: target not found: pari-elldata
The command '/bin/sh -c pacman -Syu --noconfirm binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon' returned a non-zero code: 1
ERROR: InvocationError for command /usr/local/bin/docker build . -f .tox/docker-archlinux-latest-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS= --build-arg BASE_IMAGE=archlinux:latest --build-arg http_proxy=http://host.docker.internal:3128 --build-arg https_proxy=http://host.docker.internal:3128 (exited with code 1)

comment:34 Changed 18 months ago by dimpase

pacman cannot install packages from AUR, they need to be built from source: https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages

(in case of these packages "building" is instant though, it just needs different commands to be run)

Yes, it'd be possible to script this, so these packages need to at least be tagged somehow.

comment:35 Changed 18 months ago by git

  • Commit changed from 187761e1d2a00eb75bb50eef1cac97f8aa807201 to 4cd1b55817f88191b3759ddfd2cc3863010e8223

Branch pushed to git repo; I updated commit sha1. New commits:

8fdaa9ebuild/pkgs/pari/distros/arch.txt: Comment out these packages (uninstallable)
4cd1b55src/doc/en/installation/source.rst: Include arch*.txt

comment:36 Changed 18 months ago by mkoeppe

  • Authors set to Matthias Koeppe, Vincent Delecroix
  • Status changed from new to needs_review

This is not complete but let's get it into 9.1

comment:37 follow-up: Changed 18 months ago by dimpase

there are replacements for pacman which allow automatic install of packages from AUR and the standard repo, such as yay.

so the script can first install yay, and then use it for the rest.

comment:38 in reply to: ↑ 37 Changed 18 months ago by arojas

Replying to dimpase:

there are replacements for pacman which allow automatic install of packages from AUR and the standard repo, such as yay.

so the script can first install yay, and then use it for the rest.

Please don't. AUR is unsupported, user generated content. There's no guarantee that it couldn't be replaced with broken or even malicious content at any time.

If necessary, I could move these packages to the binary repos. But I don't really see the need to check for those at build time, at least on Arch, given that they can be installed by the user at any time after Sage install.

comment:39 Changed 18 months ago by dimpase

well, if we recommend AUR packages, it's better if we test with them too, no? But, indeed, if they can be moved to the regular package repo, why not?

comment:40 follow-up: Changed 18 months ago by arojas

Alright, packages are now in the official repos

Last edited 18 months ago by arojas (previous) (diff)

comment:41 Changed 18 months ago by git

  • Commit changed from 4cd1b55817f88191b3759ddfd2cc3863010e8223 to 338468facda0ba5aef8ab8c505bdcaa4e47efb7b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

5a62319build/pkgs/readline/distros/arch.txt: New
1236a4esrc/doc/bootstrap: Also process arch.txt
5e203e9packages for pari
ba6f6f9build/pkgs/pari/distros/arch.txt: Comment out these packages (uninstallable)
cefaf09src/doc/en/installation/source.rst: Include arch*.txt
338468fbuild/pkgs/pari*/distros/arch.txt: Update/add

comment:42 Changed 18 months ago by mkoeppe

  • Description modified (diff)

comment:43 in reply to: ↑ 40 Changed 18 months ago by mkoeppe

Replying to arojas:

Alright, packages are now in the official repos

Thanks very much, that's great.

configure now finds the packages!

tox -e docker-archlinux-latest-standard -- base-toolchain
...
Checking whether SageMath should install SPKG pari...
checking installing gmp/mpir or readline? ... no
checking installing PARI/GP packages? ... checking for gp... /usr/sbin/gp
checking for gphelp... /usr/sbin/gphelp
checking whether gphelp has access to the documentation... yes
checking is pari_elldata installed? ... yes
checking is pari_galdata installed? ... yes
checking is pari_galpol installed? ... yes
checking is pari_seadata installed? ... yes
configure: will use system package and not install SPKG pari
using pari/gp from the system

Needs review.

comment:44 Changed 18 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

lgtm

comment:45 Changed 18 months ago by mkoeppe

Thank you!

comment:46 Changed 18 months ago by vbraun

  • Branch changed from public/29366 to 338468facda0ba5aef8ab8c505bdcaa4e47efb7b
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.