Opened 2 years ago

Last modified 2 years ago

#29053 closed enhancement

Add debian/fedora/arch package information to build/pkgs, generate Dockerfiles and installation help — at Version 48

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.1
Component: build Keywords:
Cc: dimpase, embray, arojas, isuruf, gh-timokau Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time (Commits, GitHub, GitLab) Commit: 8c30f07c1a4bd4231989d8b71de48e2d94713bb4
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation gives apt-get install and yum install command lines to install system packages that will be recognized by build/pkgs/SPKG/spkg-configure.m4.

Several packages are missing, see comments in https://trac.sagemath.org/attachment/ticket/27824/Dockerfile-ubuntu-minimal

This ticket puts this information on a per-package basis into build/pkgs/SPKG/distros/debian.txt, .../fedora.txt, .../arch.txt for the following purposes:

  • sage -info SPKG prints out this information.
  • Provide a tox.ini that generates and runs Dockerfiles that test the installation on various systems. Example (running 3 builds in parallel):
       $ tox -p auto -e archlinux-latest,ubuntu-focal-minimal,debian-buster-maximal
    ⠸
    

Incidental change on this ticket:

  • Mark gfortran as a type=standard package. It is a prereq of standard package numpy. Its status as a type=optional package predates the clarifications brought by the development of the spkg-configure.m4 mechanism (#27330).

Possible follow-up and related tickets:

  • #29087 - Add a GitHub workflow that runs this automatically, with many parallel jobs.
  • Also the apt-cyg command lines at https://trac.sagemath.org/wiki/Cygwin64Port could be generated in the same way.
  • Aggregate and format it (during ./bootstrap) to produce the command lines shown in the manual. This ticket already prepares it by moving these command lines into separate .txt files.
  • #29041: at ./bootstrap time, generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs
  • #28745: Add SAGE_ROOT/environment.yml for conda env create -f.

Resources:

Change History (48)

comment:1 Changed 2 years ago by mkoeppe

  • Branch set to u/mkoeppe/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time

comment:2 Changed 2 years ago by git

  • Commit set to 3a5cfb7703396f606cea2403070db012e0c8e294

Branch pushed to git repo; I updated commit sha1. New commits:

3a5cfb7build/pkgs/*/debian.txt: Add optional packages

comment:3 Changed 2 years ago by git

  • Commit changed from 3a5cfb7703396f606cea2403070db012e0c8e294 to 79d912e78343da940734450643695d6b860d8683

Branch pushed to git repo; I updated commit sha1. New commits:

79d912ebuild/pkgs/*/fedora.txt: New, from installation manual 9.0

comment:4 Changed 2 years ago by git

  • Commit changed from 79d912e78343da940734450643695d6b860d8683 to 8a739b101261939d7b2d5eae5086aa465b5b1c53

Branch pushed to git repo; I updated commit sha1. New commits:

8a739b1sage -info: Print info on equivalent system packages

comment:5 Changed 2 years ago by git

  • Commit changed from 8a739b101261939d7b2d5eae5086aa465b5b1c53 to 221c3449d65d48bec25c2d10f96c9654d3289db0

Branch pushed to git repo; I updated commit sha1. New commits:

3ef64c4Document the debian.txt, fedora.txt files.
ebee631Add build/pkgs/gcc/fedora.txt
221c344Strip comments from debian.txt, fedora.txt files

comment:6 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:7 follow-up: Changed 2 years ago by dimpase

how do you generate these per package?

by the way, Arch has namewise a naming scheme close to Gentoo and Homebrew.

comment:8 in reply to: ↑ 7 Changed 2 years ago by mkoeppe

Replying to dimpase:

how do you generate these per package?

I created these files by manual copy-paste from the installation manual.

comment:9 Changed 2 years ago by git

  • Commit changed from 221c3449d65d48bec25c2d10f96c9654d3289db0 to ff4a5516324039e37a585763def9ebc3d68e503e

Branch pushed to git repo; I updated commit sha1. New commits:

46957b4tox.ini: New - test spkg-configure on ubuntu docker images
ce99fadMerge tag '9.1.beta1' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time
0dd2c03build/bin/sage-spkg: Reformat the install command to 1 line
b5abc8bbuild/bin/sage-spkg: fixup
97b09deMake gfortran a standard package - it is a prereq of standard package numpy
ff4a551Fix typo

comment:10 Changed 2 years ago by mkoeppe

  • Component changed from documentation to build
  • Description modified (diff)
  • Summary changed from Installation manual: Generate 'apt-get install' and 'yum install' lines from build/pkgs at ./bootstrap time to Add debian/fedora package information to build/pkgs, generate Dockerfiles and installation help

comment:11 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:12 Changed 2 years ago by mkoeppe

  • Cc embray added
  • Description modified (diff)

comment:13 Changed 2 years ago by git

  • Commit changed from ff4a5516324039e37a585763def9ebc3d68e503e to 0c2148ee60f045564f3d90739ae674e89a7c3d85

Branch pushed to git repo; I updated commit sha1. New commits:

a472a9aAdd fedora and debian Docker tests
1a31b40src/doc/en/installation/{debian|fedora}[-optional].txt
0c2148emore envs

comment:14 Changed 2 years ago by git

  • Commit changed from 0c2148ee60f045564f3d90739ae674e89a7c3d85 to e38dd8ea72ec87473bb51ec818012725bf3d6110

Branch pushed to git repo; I updated commit sha1. New commits:

e38dd8esrc/doc/en/installation/source.rst: Use literalinclude

comment:15 Changed 2 years ago by mkoeppe

  • Status changed from new to needs_review

comment:16 Changed 2 years ago by mkoeppe

  • Authors set to Matthias Koeppe

comment:17 Changed 2 years ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Add debian/fedora package information to build/pkgs, generate Dockerfiles and installation help to Add debian/fedora/arch package information to build/pkgs, generate Dockerfiles and installation help

comment:18 Changed 2 years ago by git

  • Commit changed from e38dd8ea72ec87473bb51ec818012725bf3d6110 to 53273a295852da1203cb2c9999c8481d3740c209

Branch pushed to git repo; I updated commit sha1. New commits:

3f46309tox.ini: Add arch build
53273a2build/pkgs/xz/debian.txt: fixup

comment:19 Changed 2 years ago by git

  • Commit changed from 53273a295852da1203cb2c9999c8481d3740c209 to 15fa95deb818f6bb4a735433eb0e759d40ffdbaf

Branch pushed to git repo; I updated commit sha1. New commits:

15fa95dbuild/bin/sage-spkg: Add arch to 'sage -info SPKG'

comment:20 Changed 2 years ago by git

  • Commit changed from 15fa95deb818f6bb4a735433eb0e759d40ffdbaf to 7a2d4114cf22bfc433a75fb002f1a87a1df32109

Branch pushed to git repo; I updated commit sha1. New commits:

7a2d411Add more arch, fedora packages; by default, do not configure with force

comment:21 Changed 2 years ago by git

  • Commit changed from 7a2d4114cf22bfc433a75fb002f1a87a1df32109 to 3c382c91651a18d05cc24cc00e6080ccf507306e

Branch pushed to git repo; I updated commit sha1. New commits:

3c382c9fixup

comment:22 Changed 2 years ago by dimpase

  • Cc arojas added

gf2x 1.3.0 is also on Arch, so you can add it (package name gf2x). It'll work after #29081 is merged.

comment:23 follow-up: Changed 2 years ago by dimpase

You've changed build/pkgs/gfortran/type to standard - is this intentional?

comment:24 in reply to: ↑ 23 ; follow-up: Changed 2 years ago by mkoeppe

Replying to dimpase:

You've changed build/pkgs/gfortran/type to standard - is this intentional?

Yes - it's an intended correction.

comment:25 in reply to: ↑ 24 ; follow-up: Changed 2 years ago by dimpase

Replying to mkoeppe:

Replying to dimpase:

You've changed build/pkgs/gfortran/type to standard - is this intentional?

Yes - it's an intended correction.

can this be documented - ideally both here and in the commit message?

I guess it was optional just as it predated the spkg-configure.m4 machinery, and it was not fixed then, right?

comment:26 in reply to: ↑ 25 Changed 2 years ago by mkoeppe

Replying to dimpase:

Replying to mkoeppe:

Replying to dimpase:

You've changed build/pkgs/gfortran/type to standard - is this intentional?

Yes - it's an intended correction.

can this be documented - ideally both here and in the commit message?

The commit message already says:

​97b09de	Make gfortran a standard package - it is a prereq of standard package numpy

I guess it was optional just as it predated the spkg-configure.m4 machinery, and it was not fixed then, right?

Yes, I agree with this theory.

comment:27 Changed 2 years ago by mkoeppe

  • Description modified (diff)

I've added a comment on the ticket summary.

comment:28 Changed 2 years ago by mkoeppe

  • Cc isuruf added
  • Description modified (diff)

comment:29 follow-up: Changed 2 years ago by isuruf

Can you use just one file? Or put them into a new folder inside?

comment:30 Changed 2 years ago by git

  • Commit changed from 3c382c91651a18d05cc24cc00e6080ccf507306e to 6bba0237d554692663407bc4e419e69ec8758b17

Branch pushed to git repo; I updated commit sha1. New commits:

985d036add info on cbc, gf2x, gfan, nauty
6bba023build/bin/write-dockerfile.sh: test all BLAS-using packages

comment:31 in reply to: ↑ 29 Changed 2 years ago by mkoeppe

Replying to isuruf:

Can you use just one file? Or put them into a new folder inside?

Good suggestion, I'll put them into a new subfolder.

I'm also looking for a good solution for distro versioning. For example, ubuntu before bionic is missing many packages

comment:32 Changed 2 years ago by git

  • Commit changed from 6bba0237d554692663407bc4e419e69ec8758b17 to dfd5bf6cb0c5db735aa3411b614135dcf2eddaa0

Branch pushed to git repo; I updated commit sha1. New commits:

dfd5bf6Move files into build/pkgs/SPKG/distros/debian.txt etc.

comment:33 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:34 follow-up: Changed 2 years ago by isuruf

  • Branch changed from u/mkoeppe/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time to u/isuruf/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time
  • Commit changed from dfd5bf6cb0c5db735aa3411b614135dcf2eddaa0 to 1cfa89b91b766030c361ef0e04474b7a4a639258

Added conda as well. We can generate the environment.yml from this.


New commits:

1cfa89bAdd conda.txt

comment:35 follow-up: Changed 2 years ago by isuruf

There are some packages here that don't have spkg-configure.m4. Should those be removed?

comment:36 follow-up: Changed 2 years ago by isuruf

Did you use a script to generate src/doc/en/installation/fedora.txt?

comment:37 in reply to: ↑ 35 Changed 2 years ago by mkoeppe

Replying to isuruf:

There are some packages here that don't have spkg-configure.m4. Should those be removed?

No, that's OK and good preparation for when that is added. I'll update the scripts to only use packages that do have spkg-configure.m4`.

comment:38 in reply to: ↑ 34 Changed 2 years ago by mkoeppe

Replying to isuruf:

Added conda as well. We can generate the environment.yml from this.


New commits:

1cfa89bAdd conda.txt

Great! Thanks very much

comment:39 in reply to: ↑ 36 Changed 2 years ago by mkoeppe

Replying to isuruf:

Did you use a script to generate src/doc/en/installation/fedora.txt?

No, not yet. This is only taken out from the manual. I haven't written the formatting script yet.

comment:40 Changed 2 years ago by mkoeppe

  • Branch changed from u/isuruf/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time to u/mkoeppe/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time

comment:41 Changed 2 years ago by mkoeppe

  • Commit changed from 1cfa89b91b766030c361ef0e04474b7a4a639258 to 0f2b16ed34c87b583bb75791a8ff843cf52af530

More changes at https://github.com/mkoeppe/sage/commits/test-29053-on-29051


New commits:

80d2f90build/pkgs/debian-bootstrap.txt: Add pkg-config
d31441eremove nonexisting fedora package
e4c2bf8build/pkgs/arch.txt: Add gcc
c55071fbuild/pkgs/debian.txt: add g++
ea04ee7build/pkgs/debian.txt: python3-minimal->python3
b4e90a0build/bin/write-dockerfile.sh: Only install a package if it has spkg-configure.m4
0f2b16ebuild/bin/sage-spkg: Explain to user what will be done with the system package

comment:42 follow-up: Changed 2 years ago by mkoeppe

@isuruf: Compiling python3 on conda-forge gives me an error that looks like it's related to ffi - https://github.com/mkoeppe/sage/runs/410329852

comment:43 follow-up: Changed 2 years ago by arojas

Arch package sage-data-elliptic_curves corresponds to Sage's elliptic_curves, not to the optional database_cremona_ellcurve

comment:44 in reply to: ↑ 42 Changed 2 years ago by isuruf

Replying to mkoeppe:

@isuruf: Compiling python3 on conda-forge gives me an error that looks like it's related to ffi - https://github.com/mkoeppe/sage/runs/410329852

Looks like you are not activating the conda environment.

comment:45 Changed 2 years ago by git

  • Commit changed from 0f2b16ed34c87b583bb75791a8ff843cf52af530 to ddcac277d5f5ca60c1c9165c729a47d0e2329025

Branch pushed to git repo; I updated commit sha1. New commits:

ddcac27Squashed commit of the following:

comment:46 Changed 2 years ago by git

  • Commit changed from ddcac277d5f5ca60c1c9165c729a47d0e2329025 to 8c30f07c1a4bd4231989d8b71de48e2d94713bb4

Branch pushed to git repo; I updated commit sha1. New commits:

8c30f07Arch package sage-data-elliptic_curves corresponds to Sage's elliptic_curves, not to the optional database_cremona_ellcurve

comment:47 in reply to: ↑ 43 Changed 2 years ago by mkoeppe

Replying to arojas:

Arch package sage-data-elliptic_curves corresponds to Sage's elliptic_curves, not to the optional database_cremona_ellcurve

Thank you! Fixed.

comment:48 Changed 2 years ago by mkoeppe

  • Description modified (diff)
Note: See TracTickets for help on using tickets.