Opened 3 years ago

Closed 2 years ago

Last modified 2 years ago

#26964 closed defect (fixed)

./bootstrap: Format build/pkgs/SPKG/distros/ information to produce the apt-get/yum command lines shown in the installation manual

Reported by: embray Owned by:
Priority: minor Milestone: sage-9.1
Component: documentation Keywords:
Cc: dimpase, embray, jhpalmieri Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 688c68e (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

When trying to build Sage on a very bare fedora install (e.g. a Docker container) the packages listed in the build instructions at:

https://doc.sagemath.org/html/en/installation/source.html#linux-prerequisite-installation

are insufficient. At the very least it also needs findutils and which, and currently it also requires python2, though that should be fixed as part of #26953. The need for which comes, at the very least, from MPIR's configure script, which uses which at least on Linux.

With #29053, the necessary distribution packages for an installation are cataloged in build/pkgs/fedora.txt, build/pkgs/SPKG/distros/fedora.txt and similar for other distributions.

In this ticket, we add to bootstrap some code to keep the yum (and apt-get for debian) command-lines in the installation manual up to date. #29053 prepared this by isolating these command lines in separate .txt files in src/doc.

Change History (49)

comment:1 Changed 3 years ago by embray

  • Summary changed from missing packages for building Sage on fedora to missing packages in instructions for building Sage on fedora

comment:2 Changed 3 years ago by embray

  • Milestone changed from sage-8.6 to sage-8.7

Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.

comment:3 Changed 3 years ago by embray

  • Milestone changed from sage-8.7 to sage-pending

Removing most of the rest of my open tickets out of the 8.7 milestone, which should be closed.

comment:4 Changed 2 years ago by mkoeppe

  • Dependencies set to #29053

Added these 2 packages to #29053.

comment:5 Changed 2 years ago by mkoeppe

  • Cc dimpase embray added
  • Dependencies #29053 deleted
  • Description modified (diff)
  • Milestone changed from sage-pending to sage-9.1
  • Summary changed from missing packages in instructions for building Sage on fedora to ./bootstrap: Format build/pkgs/SPKG/distros/ information to produce the apt-get/yum command lines shown in the installation manual

comment:6 Changed 2 years ago by embray

I'm not sure how I feel about using the bootstrap script for this purpose. I think I would prefer if it were done by configure instead. Other than that, I think this is a great idea!

Version 0, edited 2 years ago by embray (next)

comment:7 follow-up: Changed 2 years ago by mkoeppe

  • Status changed from new to needs_review

No, no, bootstrap is exactly the right phase for this, not configure. Let me explain more.

bootstrap will prepare a source tree so that

  • make sdist can be run
  • configure can be run by a user (who does not have autotools)
  • src/setup.py sdist can be run.
  • src/configure can be run (#29119)

None of this depends on the configuration determined by configure!

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

comment:8 in reply to: ↑ 7 ; follow-up: Changed 2 years ago by embray

  • Status changed from needs_review to needs_work

Replying to mkoeppe:

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?

I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a src/bootstrap script for sagelib itself, so maybe something like this would make more sense to go there.

Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.

comment:9 follow-up: Changed 2 years ago by embray

Couldn't this also be done as part of make sdist?

comment:10 in reply to: ↑ 8 Changed 2 years ago by mkoeppe

Replying to embray:

Replying to mkoeppe:

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?

Both.

Currently, of course, only sage-the-distribution has a working sdist.

sagelib's sdist needs fixing. You might remember ticket #21516 - Fix sagelib sdist (src/setup.py sdist).

I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a src/bootstrap script for sagelib itself, so maybe something like this would make more sense to go there.

In my opinion, there should only be one developer-invoked bootstrap script (the one for the distribution) -- because everything will depend on the distribution's build/pkgs files.

SAGE_ROOT/bootstrap could certainly call a new script src/bootstrap for generating files that end up in src/. If you create that in #29119, fine.

Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.

Sure. #29041 (src/requirements.txt, src/constraints.txt, src/setup.cfg) has a branch, needs review.

comment:11 in reply to: ↑ 9 ; follow-up: Changed 2 years ago by mkoeppe

Replying to embray:

Couldn't this also be done as part of make sdist?

No, because developers who build sage out of a git worktree do not run make sdist.

comment:12 Changed 2 years ago by mkoeppe

  • Branch set to u/mkoeppe/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual

comment:13 Changed 2 years ago by git

  • Commit set to 438bc7fb739b571e82674e69210f81e7ec29c5e8

Branch pushed to git repo; I updated commit sha1. New commits:

438bc7f./bootstrap, src/doc/bootstrap: Auto-generate apt-get/yum command lines

comment:14 Changed 2 years ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Status changed from needs_work to needs_review

comment:15 Changed 2 years ago by git

  • Commit changed from 438bc7fb739b571e82674e69210f81e7ec29c5e8 to 3810bc9835050073f663109d576eae1157bf4f77

Branch pushed to git repo; I updated commit sha1. New commits:

3810bc9Clean and ignore the generated files

comment:16 Changed 2 years ago by mkoeppe

  • Cc jhpalmieri added

comment:17 Changed 2 years ago by git

  • Commit changed from 3810bc9835050073f663109d576eae1157bf4f77 to bdd69c3bd85b2d33a253ef8c7edde6ca8adffb85

Branch pushed to git repo; I updated commit sha1. New commits:

bdd69c3build/bin/write-dockerfile.sh: Copy src/doc/bootstrap early

comment:18 Changed 2 years ago by git

  • Commit changed from bdd69c3bd85b2d33a253ef8c7edde6ca8adffb85 to 8be072b7244b342c6b70737cc36409121b70b41b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8be072bbuild/bin/write-dockerfile.sh: Copy src/doc/bootstrap early

comment:19 Changed 2 years ago by git

  • Commit changed from 8be072b7244b342c6b70737cc36409121b70b41b to 38741947257da00371142bd7ca4099909717cc50

Branch pushed to git repo; I updated commit sha1. New commits:

e3b5fdebootstrap: Pass SAGE_ROOT to src/doc/bootstrap
3874194src/doc/bootstrap: Create output dir if it does not exist (for Docker run)

comment:20 Changed 2 years ago by dimpase

something is not quite right:

(sage-sh) dimpase@clpc171:sage$ ./build/bin/sage-print-system-package-command debian gmp
sudo apt-get gmp 
(sage-sh) dimpase@clpc171:sage$ ./build/bin/sage-print-system-package-command conda gmp
# gmp the following packages: 
  • so debian works, conda does not...

comment:21 Changed 2 years ago by mkoeppe

you forgot the verb install in both commands

comment:22 Changed 2 years ago by git

  • Commit changed from 38741947257da00371142bd7ca4099909717cc50 to eb7a1d9b9a16c40a5a30c9451d14b5e078b88007

Branch pushed to git repo; I updated commit sha1. New commits:

eb7a1d9build/bin/sage-print-system-package-command: Print usage message if not enough arguments provided

comment:23 Changed 2 years ago by git

  • Commit changed from eb7a1d9b9a16c40a5a30c9451d14b5e078b88007 to 146a52052b48c4f97aaa6be1122f9a63d5f44e8d

Branch pushed to git repo; I updated commit sha1. New commits:

146a520build/bin/sage-print-system-package-command: Do not print a command if no packages are to be installed

comment:24 in reply to: ↑ 11 ; follow-up: Changed 2 years ago by embray

Replying to mkoeppe:

Replying to embray:

Couldn't this also be done as part of make sdist?

No, because developers who build sage out of a git worktree do not run make sdist.

Sure, but what is required for running make sdist? It seems to me this should be done in two separate cases: One when running make sdist and one when developing out of source. In either case I don't think the bootstrap script is the appropriate place for it.

The only reason you seem to be putting it there is that it would happen to cover both cases, but I think both cases should be handled separately.

comment:25 follow-up: Changed 2 years ago by embray

I think I might see part of the problem here: It is true that the information you need to generate these documentation pages lives in sage-the-distribution, above the sagelib sources. If you want to package sagelib in a stand-alone fashion you need to get access to this data somehow, but you can't, so the file is generated sort of externally to sagelib, using data from sage-the-distribution.

A similar problem applies to #29119. In the prototype for that I am working on a more general solution to the problem, which is that ./bootstrap makes hard links of files sagelib needs into the sagelib sources under src/. That way, only one copy of the relevant files needs to exist in the existing combined repository, but if sagelib is packaged separately it will retain the necessary files.

I think having that might serve as a solution here too. My feeling is that these documentation pages should be generated by the docbuilder itself. As long as it has access to information about Sage's dependencies it can do this, without the top-level ./bootstrap script having to take any special action.

In other words, what we ultimately want is to be able to build Sage's docs without direct dependency on the sage-the-distribution stuff. As it stands, I fear this PR is introducing too much direct dependency on sage-the-distribution just to be able to build Sage's docs. What we would really want is to be able to run setup.py sdist, the a source tarball just for sagelib and its docs, and later unpack it somewhere else and still be able to build the docs (and sage itself).

comment:26 follow-up: Changed 2 years ago by jhpalmieri

How is this supposed to work with an incremental build?

$ make bootstrap
make: `bootstrap' is up to date.
$ make configure
make: `configure' is up to date.
$ make
...
OSError: /Users/jpalmier/Desktop/Sage/git/sage/src/doc/en/installation/source.rst:228: WARNING: Include file '/Users/jpalmier/Desktop/Sage/git/sage/src/doc/en/installation/debian.txt' not found or reading it failed

comment:27 in reply to: ↑ 26 ; follow-up: Changed 2 years ago by mkoeppe

Replying to jhpalmieri:

How is this supposed to work with an incremental build?

Thanks for catching this -- I need to add some dependencies

comment:28 in reply to: ↑ 25 Changed 2 years ago by mkoeppe

Replying to embray:

I think I might see part of the problem here: It is true that the information you need to generate these documentation pages lives in sage-the-distribution, above the sagelib sources. If you want to package sagelib in a stand-alone fashion you need to get access to this data somehow, but you can't, so the file is generated sort of externally to sagelib, using data from sage-the-distribution.

A similar problem applies to #29119. [...]

There is a really easy solution:

  • bootstrap is only allowed in the full distribution source, not in a separate sagelib source. No hardlinks necessary. Only developers bootstrap.
  • Users don't bootstrap, they start with the configure phase.
  • sdist of the distribution and sdist of sagelib generate bootstrapped source archives.

comment:29 in reply to: ↑ 24 Changed 2 years ago by mkoeppe

Replying to embray:

Sure, but what is required for running make sdist?

Obviously, it requires exactly a bootstrapped source tree!

It seems to me this should be done in two separate cases: One when running make sdist and one when developing out of source. In either case I don't think the bootstrap script is the appropriate place for it.

The only reason you seem to be putting it there is that it would happen to cover both cases, but I think both cases should be handled separately.

bootstrap is the phase before configure, which does not depend on the configuration of the system.

comment:30 Changed 2 years ago by git

  • Commit changed from 146a52052b48c4f97aaa6be1122f9a63d5f44e8d to ce34ce1fcc5e8669c9e05b758bb8576cc8afa8d1

Branch pushed to git repo; I updated commit sha1. New commits:

ce34ce1Makefile: Add dependencies on build/pkgs/*.txt build/pkgs/*/distros/*.txt

comment:31 in reply to: ↑ 27 Changed 2 years ago by mkoeppe

Replying to mkoeppe:

Replying to jhpalmieri:

How is this supposed to work with an incremental build?

Thanks for catching this -- I need to add some dependencies

Done.

comment:32 follow-ups: Changed 2 years ago by jhpalmieri

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation. By the way, I'm not sure whether I like the old style

     $ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \
       git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \
       libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \
       pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \
       libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \
       libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \
       libpari-dev pari-gp2c libec-dev liblrcalc-dev \
       libm4ri-dev libm4rie-dev liblfunction-dev lcalc \
       libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliquer

or the new style

    $ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-devel

better.

Is there a ticket which adds homebrew information to the installation/source.rst?

comment:33 in reply to: ↑ 32 ; follow-up: Changed 2 years ago by mkoeppe

Replying to jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation.

The *generated* files are in src/doc/en/installation. The source files are in build/pkgs/. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.

comment:34 in reply to: ↑ 32 Changed 2 years ago by mkoeppe

Replying to jhpalmieri:

Is there a ticket which adds homebrew information to the installation/source.rst?

#29104 is for homebrew. I haven't added the package information there yet -- any help is very welcome!

comment:35 in reply to: ↑ 33 ; follow-up: Changed 2 years ago by jhpalmieri

Replying to mkoeppe:

Replying to jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation.

The *generated* files are in src/doc/en/installation. The source files are in build/pkgs/. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.

But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and make fails, even with your latest branch. Is there no way to avoid this?

$ git checkout t/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual 
Switched to branch 't/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual'
$ make configure
make: `configure' is up to date.

./bootstrap -d generates the correct files, but I have to run it manually.

comment:36 follow-up: Changed 2 years ago by dimpase

the idea is that one can make configure is not a good one. configure is "built" by autoconf, not by make.

can we get rid of configure target?

Last edited 2 years ago by dimpase (previous) (diff)

comment:37 in reply to: ↑ 32 Changed 2 years ago by dimpase

Replying to jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation. By the way, I'm not sure whether I like the old style

     $ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \
       git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \
       libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \
       pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \
       libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \
       libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \
       libpari-dev pari-gp2c libec-dev liblrcalc-dev \
       libm4ri-dev libm4rie-dev liblfunction-dev lcalc \
       libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliquer

or the new style

    $ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-devel

better.

the new style is built automatically, whereas the old one was manual. One can surely hack further to add insertion of \ after ~80 chars, but it's certainlty minor...

comment:38 Changed 2 years ago by git

  • Commit changed from ce34ce1fcc5e8669c9e05b758bb8576cc8afa8d1 to 688c68e7a4d0b17d0a8093585389c6c4f2fc8c27

Branch pushed to git repo; I updated commit sha1. New commits:

688c68eMakefile [configure]: Add dependency on bootstrap, src/doc/bootstrap

comment:39 in reply to: ↑ 35 Changed 2 years ago by mkoeppe

Replying to jhpalmieri:

But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and make fails, even with your latest branch. Is there no way to avoid this?

Thanks! I've added more dependencies to the make configure target that should take care of this case.

comment:40 in reply to: ↑ 36 Changed 2 years ago by mkoeppe

Replying to dimpase:

the idea is that one can make configure is not a good one. configure is "built" by autoconf, not by make.

I don't think there's anything wrong with these make targets -- also autotools have them (in maintainer mode).

comment:41 Changed 2 years ago by dimpase

a typical autotools project won't have a Makefile in unbuilt state, only Makefile.am

comment:42 Changed 2 years ago by mkoeppe

That is true, but there are Makefile targets that regenerate configure etc. from configure.ac.

comment:43 Changed 2 years ago by jhpalmieri

That works for me, thanks.

I don't in general ever use make configure explicitly, but I was trying to get the previous versions of this branch to work so I tried it. With the most recent branch, I can just check it out and run make.

comment:44 Changed 2 years ago by dimpase

is sage-print-system-package-command meant to runnable by the user, or it's a "service" script?

comment:45 Changed 2 years ago by mkoeppe

It's an internal script of sage the distribution. That's why it's in build/bin.

comment:46 Changed 2 years ago by dimpase

  • Reviewers set to John Palmieri, Dima Pasechnik
  • Status changed from needs_review to positive_review

OK, the only nitpck is that on modern Fedora's it's dnf, not yum.

comment:47 Changed 2 years ago by mkoeppe

Thank you!

comment:48 Changed 2 years ago by vbraun

  • Branch changed from u/mkoeppe/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual to 688c68e7a4d0b17d0a8093585389c6c4f2fc8c27
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:49 Changed 2 years ago by vbraun

  • Commit 688c68e7a4d0b17d0a8093585389c6c4f2fc8c27 deleted

This breaks incremental builds: #29233

Note: See TracTickets for help on using tickets.