Opened 2 years ago
Closed 2 years ago
#29366 closed enhancement (fixed)
archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.1 |
Component: | porting | Keywords: | |
Cc: | vdelecroix, arojas, dimpase | Merged in: | |
Authors: | Matthias Koeppe, Vincent Delecroix | Reviewers: | Dima Pasechnik |
Report Upstream: | N/A | Work issues: | |
Branch: | 338468f (Commits, GitHub, GitLab) | Commit: | 338468facda0ba5aef8ab8c505bdcaa4e47efb7b |
Dependencies: | Stopgaps: |
Description (last modified by )
https://groups.google.com/d/msg/sage-devel/JgArPVx7dwg/gMxeB4b7AgAJ
To test:
tox -e docker-archlinux-latest-standard
Attachments (2)
Change History (48)
comment:1 Changed 2 years ago by
- Cc dimpase added
comment:2 Changed 2 years ago by
- Summary changed from archlinux: Fix detection detection of system packages to archlinux: Fix detection of system packages
Changed 2 years ago by
comment:3 Changed 2 years ago by
comment:4 Changed 2 years ago by
Indeed, I would like to be sure that it is not on Sage side and have an up to date developer manual for the relevant packages.
Changed 2 years ago by
comment:5 Changed 2 years ago by
In case you haven't seen it yet: the system package lists in the developer's manual are now autogenerated. But adding some Arch specific instructions would be a useful addition
comment:6 Changed 2 years ago by
Installation manual, that is
comment:7 Changed 2 years ago by
Well, config.log is clear enough on e.g. what's missing w.r.t. Pari: namely, install elldata, galpol, seadata, as seen in
configure:14982: checking is pari_elldata installed? 1735 *** at top-level: r=ellinit("11a1");r[11] 1736 *** ^--------------------- 1737 *** ellinit: error opening elldata file: `/usr/share/pari/elldata/ell0'. 1738 configure:14989: result: no; cannot use system pari/GP without elldata package 1739 configure:14991: Install elldata package and reconfigure. 1740 configure:14993: Otherwise Sage will build its own pari/GP. 1741 configure:14997: checking is pari_galdata installed? 1742 configure:15001: result: yes 1743 configure:15012: checking is pari_galpol installed? 1744 *** at top-level: galoisgetname(12,1)=="C3 : C4" 1745 *** ^------------------------------ 1746 *** galoisgetname: error opening galpol file: `/usr/share/pari/galpol/12/nb'. 1747 configure:15019: result: no; cannot use system pari/GP without galpol package 1748 configure:15021: Install galpol package and reconfigure. 1749 configure:15023: Otherwise Sage will build its own pari/GP. 1750 configure:15027: checking is pari_seadata installed? 1751 *** at top-level: poldegree(ellmodulareqn(211)[1]) 1752 *** ^---------------------- 1753 *** ellmodulareqn: error opening seadata file: `/usr/share/pari/seadata/sea211'. 1754 configure:15034: result: no; cannot use system pari/GP without seadata package 1755 configure:15036: Install seadata package and reconfigure. 1756 configure:15038: Otherwise Sage will build its own pari/GP. 1757 configure:15266: no suitable system package found for SPKG pari 1758 configure:15284: result: using Sage's pari SPKG
comment:8 Changed 2 years ago by
Nothing to fix on Arch side either - givaro is not used because you have an older version on /usr/local which is detected first, and pari is not used because of missing optional packages
comment:9 follow-up: ↓ 10 Changed 2 years ago by
Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?
comment:10 in reply to: ↑ 9 ; follow-up: ↓ 11 Changed 2 years ago by
Replying to mkoeppe:
Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?
Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled. Other packages (fplll, ecm) are simply not installed on the build machine.
comment:11 in reply to: ↑ 10 ; follow-up: ↓ 13 Changed 2 years ago by
Replying to arojas:
Replying to mkoeppe:
Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?
Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.
Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS
Other packages (fplll, ecm) are simply not installed on the build machine.
OK, this means we need to add arch.txt
files for these packages
comment:12 follow-up: ↓ 21 Changed 2 years ago by
and for more of the following listed here:
configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll git givaro gp2c isl lcalc libatomic_ops libsemigroups mpfi ninja_build ntl pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_term_readline_gnu yasm zeromq
comment:13 in reply to: ↑ 11 ; follow-up: ↓ 14 Changed 2 years ago by
Replying to mkoeppe:
Replying to arojas:
Replying to mkoeppe:
Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?
Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.
Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS
I believe that I was able to use Arch's NTL in builds few betas ago. This again points at #29339 as being too drastic in its approach.
comment:14 in reply to: ↑ 13 ; follow-up: ↓ 15 Changed 2 years ago by
Replying to dimpase:
Replying to mkoeppe:
Replying to arojas:
Replying to mkoeppe:
Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?
Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.
Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS
I believe that I was able to use Arch's NTL in builds few betas ago. This again points at #29339 as being too drastic in its approach.
And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever
comment:15 in reply to: ↑ 14 ; follow-up: ↓ 16 Changed 2 years ago by
comment:16 in reply to: ↑ 15 Changed 2 years ago by
comment:17 Changed 2 years ago by
Ah, thanks for the clarification
comment:18 Changed 2 years ago by
- Summary changed from archlinux: Fix detection of system packages to archlinux: Add missing system packages (arch.txt), fix detection of system packages
comment:19 Changed 2 years ago by
- Summary changed from archlinux: Add missing system packages (arch.txt), fix detection of system packages to archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages
comment:20 Changed 2 years ago by
- Branch set to u/mkoeppe/archlinux__add_missing_system_packages__arch_txt___add_arch_section_to_installation_manual__fix_detection_of_system_packages
comment:21 in reply to: ↑ 12 Changed 2 years ago by
- Commit set to 9a21aa77889fb4cababf0edde195bec499164b71
Replying to mkoeppe:
and for more of the following listed here:
configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll git givaro gp2c isl lcalc libatomic_ops libsemigroups mpfi ninja_build ntl pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_term_readline_gnu yasm zeromq
Note that the packages pari-seadata
and pari-seadata-small
conflict.
After removing the pkgconfig file of givaro in /usr/local
(my bad) and installing the three pari packages (pari-seadata
, pari-galpol
, pari-elldata
), everything is now taken from the system! Thanks.
New commits:
9a21aa7 | build/pkgs/readline/distros/arch.txt: New
|
comment:22 Changed 2 years ago by
So what are we missing in our arch.txt files?
comment:23 Changed 2 years ago by
I don't understand what these arch.txt
are supposed to say. It looks like a kind of translation sage package
-> arch package
. However, for pari to be recognized by the sagemath configure, it needs to be installed with the optional packages pari-seadata
, pari-galpol
, pari-elldata
. Where is this specified?
comment:24 Changed 2 years ago by
Here is an example from build/pkgs/pari/distros/debian.txt
:
pari-gp2c libpari-dev # #29319: cypari2 needs gphelp at installation time pari-doc # We add these data packages because they are checked by spkg-configure.m4 pari-elldata pari-galdata pari-galpol pari-seadata
Put the equivalent of these Debian package names
into build/pkgs/pari/distros/arch.txt
comment:25 Changed 2 years ago by
Where is the format documented?
comment:26 Changed 2 years ago by
As many things in our build system, nowhere, AFAIK - unless I miss a ticket that didn't make it into Sage 9.1.beta8 (the renaming spkg-install
-> spkg-install.in
did make it there, though). Please feel free to add things to Developer Guide, in src/doc/en/developer/packaging.rst
comment:27 Changed 2 years ago by
these files are processed by src/doc/bootstrap
called from ./bootstrap
.
comment:28 Changed 2 years ago by
- Commit changed from 9a21aa77889fb4cababf0edde195bec499164b71 to 2acd21d61ee2a2f9bfb5c9e0e3abbabec8d9f7a1
Branch pushed to git repo; I updated commit sha1. New commits:
2acd21d | src/doc/bootstrap: Also process arch.txt
|
comment:29 Changed 2 years ago by
- Branch changed from u/mkoeppe/archlinux__add_missing_system_packages__arch_txt___add_arch_section_to_installation_manual__fix_detection_of_system_packages to public/29366
- Commit 2acd21d61ee2a2f9bfb5c9e0e3abbabec8d9f7a1 deleted
comment:30 Changed 2 years ago by
- Commit set to 2acd21d61ee2a2f9bfb5c9e0e3abbabec8d9f7a1
comment:31 Changed 2 years ago by
- Commit changed from 2acd21d61ee2a2f9bfb5c9e0e3abbabec8d9f7a1 to 187761e1d2a00eb75bb50eef1cac97f8aa807201
Branch pushed to git repo; I updated commit sha1. New commits:
187761e | packages for pari
|
comment:32 Changed 2 years ago by
Also, both pari-seadata and pari-elldata are from AUR.
comment:33 Changed 2 years ago by
$ tox -e docker-archlinux-latest-standard docker-archlinux-latest-standard run-test-pre: PYTHONHASHSEED='2835960636' docker-archlinux-latest-standard run-test: commands[0] | bash -c 'build/bin/write-dockerfile.sh arch "standard" yes no > /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/docker-archlinux-latest-standard/Dockerfile' docker-archlinux-latest-standard run-test: commands[1] | docker build . -f /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/docker-archlinux-latest-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS= --build-arg BASE_IMAGE=archlinux:latest --build-arg http_proxy=http://host.docker.internal:3128 --build-arg https_proxy=http://host.docker.internal:3128 Sending build context to Docker daemon 179.1MB Step 1/23 : ARG BASE_IMAGE=archlinux:latest Step 2/23 : FROM ${BASE_IMAGE} ---> 0152bf6f0800 Step 3/23 : ENV PACKAGES="binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon" ---> Using cache ---> 61ceb4b4c200 Step 4/23 : RUN pacman -Syu --noconfirm binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon ---> Running in a4547b23d1cc :: Synchronizing package databases... downloading core.db... downloading extra.db... downloading community.db... warning: perl-5.30.1-1 is up to date -- reinstalling warning: tar-1.32-3 is up to date -- reinstalling error: target not found: pari-seadata error: target not found: pari-elldata The command '/bin/sh -c pacman -Syu --noconfirm binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon' returned a non-zero code: 1 ERROR: InvocationError for command /usr/local/bin/docker build . -f .tox/docker-archlinux-latest-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS= --build-arg BASE_IMAGE=archlinux:latest --build-arg http_proxy=http://host.docker.internal:3128 --build-arg https_proxy=http://host.docker.internal:3128 (exited with code 1)
comment:34 Changed 2 years ago by
pacman cannot install packages from AUR, they need to be built from source: https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages
(in case of these packages "building" is instant though, it just needs different commands to be run)
Yes, it'd be possible to script this, so these packages need to at least be tagged somehow.
comment:35 Changed 2 years ago by
- Commit changed from 187761e1d2a00eb75bb50eef1cac97f8aa807201 to 4cd1b55817f88191b3759ddfd2cc3863010e8223
comment:36 Changed 2 years ago by
- Status changed from new to needs_review
This is not complete but let's get it into 9.1
comment:37 follow-up: ↓ 38 Changed 2 years ago by
there are replacements for pacman
which allow automatic install of packages from AUR and the standard repo, such as yay
.
so the script can first install yay, and then use it for the rest.
comment:38 in reply to: ↑ 37 Changed 2 years ago by
Replying to dimpase:
there are replacements for
pacman
which allow automatic install of packages from AUR and the standard repo, such asyay
.so the script can first install yay, and then use it for the rest.
Please don't. AUR is unsupported, user generated content. There's no guarantee that it couldn't be replaced with broken or even malicious content at any time.
If necessary, I could move these packages to the binary repos. But I don't really see the need to check for those at build time, at least on Arch, given that they can be installed by the user at any time after Sage install.
comment:39 Changed 2 years ago by
well, if we recommend AUR packages, it's better if we test with them too, no? But, indeed, if they can be moved to the regular package repo, why not?
comment:40 follow-up: ↓ 43 Changed 2 years ago by
Alright, packages are now in the official repos
comment:41 Changed 2 years ago by
- Commit changed from 4cd1b55817f88191b3759ddfd2cc3863010e8223 to 338468facda0ba5aef8ab8c505bdcaa4e47efb7b
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
5a62319 | build/pkgs/readline/distros/arch.txt: New
|
1236a4e | src/doc/bootstrap: Also process arch.txt
|
5e203e9 | packages for pari
|
ba6f6f9 | build/pkgs/pari/distros/arch.txt: Comment out these packages (uninstallable)
|
cefaf09 | src/doc/en/installation/source.rst: Include arch*.txt
|
338468f | build/pkgs/pari*/distros/arch.txt: Update/add
|
comment:42 Changed 2 years ago by
- Description modified (diff)
comment:43 in reply to: ↑ 40 Changed 2 years ago by
Replying to arojas:
Alright, packages are now in the official repos
Thanks very much, that's great.
configure
now finds the packages!
tox -e docker-archlinux-latest-standard -- base-toolchain ... Checking whether SageMath should install SPKG pari... checking installing gmp/mpir or readline? ... no checking installing PARI/GP packages? ... checking for gp... /usr/sbin/gp checking for gphelp... /usr/sbin/gphelp checking whether gphelp has access to the documentation... yes checking is pari_elldata installed? ... yes checking is pari_galdata installed? ... yes checking is pari_galpol installed? ... yes checking is pari_seadata installed? ... yes configure: will use system package and not install SPKG pari using pari/gp from the system
Needs review.
comment:44 Changed 2 years ago by
- Reviewers set to Dima Pasechnik
- Status changed from needs_review to positive_review
lgtm
comment:45 Changed 2 years ago by
Thank you!
comment:46 Changed 2 years ago by
- Branch changed from public/29366 to 338468facda0ba5aef8ab8c505bdcaa4e47efb7b
- Resolution set to fixed
- Status changed from positive_review to closed
not sure there is much on Sage's side that can be done - apart from keeping correct lists of packages to install.