./bootstrap: Format build/pkgs/SPKG/distros/ information to produce the apt-get/yum command lines shown in the installation manual

[embray]

When trying to build Sage on a very bare fedora install (e.g. a Docker container) the packages listed in the build instructions at:

​https://doc.sagemath.org/html/en/installation/source.html#linux-prerequisite-installation

are insufficient. At the very least it also needs findutils and which, and currently it also requires python2, though that should be fixed as part of #26953. The need for which comes, at the very least, from MPIR's configure script, which uses which at least on Linux.

With #29053, the necessary distribution packages for an installation are cataloged in build/pkgs/fedora.txt, build/pkgs/SPKG/distros/fedora.txt and similar for other distributions.

In this ticket, we add to bootstrap some code to keep the yum (and apt-get for debian) command-lines in the installation manual up to date. #29053 prepared this by isolating these command lines in separate .txt files in src/doc.

[embray]

Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.

[embray]

Removing most of the rest of my open tickets out of the 8.7 milestone, which should be closed.

[mkoeppe]

Added these 2 packages to #29053.

[embray]

I'm not sure how I feel about using the bootstrap script for this purpose. I think I would prefer if it were done by configure instead. Reason: The bootstrap script should really be limited to tasks specific to generating the configure file (whether running autoreconf or downloadng it). Other than that, I think this is a great idea!

[mkoeppe]

No, no, bootstrap is exactly the right phase for this, not configure. Let me explain more.

bootstrap will prepare a source tree so that

• make sdist can be run
• configure can be run by a user (who does not have autotools)
• src/setup.py sdist can be run.
• src/configure can be run (#29119)

None of this depends on the configuration determined by configure!

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

[embray]

Replying to mkoeppe:

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?

I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a src/bootstrap script for sagelib itself, so maybe something like this would make more sense to go there.

Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.

[embray]

Couldn't this also be done as part of make sdist?

[mkoeppe]

Replying to embray:

Replying to mkoeppe:

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?

Both.

Currently, of course, only sage-the-distribution has a working sdist.

sagelib's sdist needs fixing. You might remember ticket #21516 - Fix sagelib sdist (src/setup.py sdist).

I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a src/bootstrap script for sagelib itself, so maybe something like this would make more sense to go there.

In my opinion, there should only be one developer-invoked bootstrap script (the one for the distribution) -- because everything will depend on the distribution's build/pkgs files.

SAGE_ROOT/bootstrap could certainly call a new script src/bootstrap for generating files that end up in src/. If you create that in #29119, fine.

Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.

Sure. #29041 (src/requirements.txt, src/constraints.txt, src/setup.cfg) has a branch, needs review.

[mkoeppe]

Replying to embray:

Couldn't this also be done as part of make sdist?

No, because developers who build sage out of a git worktree do not run make sdist.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​438bc7f

./bootstrap, src/doc/bootstrap: Auto-generate apt-get/yum command lines

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3810bc9

Clean and ignore the generated files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bdd69c3

build/bin/write-dockerfile.sh: Copy src/doc/bootstrap early

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​8be072b

build/bin/write-dockerfile.sh: Copy src/doc/bootstrap early

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e3b5fde	bootstrap: Pass SAGE_ROOT to src/doc/bootstrap
​3874194	src/doc/bootstrap: Create output dir if it does not exist (for Docker run)

[dimpase]

something is not quite right:

(sage-sh) dimpase@clpc171:sage$ ./build/bin/sage-print-system-package-command debian gmp
sudo apt-get gmp 
(sage-sh) dimpase@clpc171:sage$ ./build/bin/sage-print-system-package-command conda gmp
# gmp the following packages: 

• so debian works, conda does not...

[mkoeppe]

you forgot the verb install in both commands

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​eb7a1d9

build/bin/sage-print-system-package-command: Print usage message if not enough arguments provided

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​146a520

build/bin/sage-print-system-package-command: Do not print a command if no packages are to be installed

[embray]

Replying to mkoeppe:

Replying to embray:

Couldn't this also be done as part of make sdist?

No, because developers who build sage out of a git worktree do not run make sdist.

Sure, but what is required for running make sdist? It seems to me this should be done in two separate cases: One when running make sdist and one when developing out of source. In either case I don't think the bootstrap script is the appropriate place for it.

The only reason you seem to be putting it there is that it would happen to cover both cases, but I think both cases should be handled separately.

[embray]

I think I might see part of the problem here: It is true that the information you need to generate these documentation pages lives in sage-the-distribution, above the sagelib sources. If you want to package sagelib in a stand-alone fashion you need to get access to this data somehow, but you can't, so the file is generated sort of externally to sagelib, using data from sage-the-distribution.

A similar problem applies to #29119. In the prototype for that I am working on a more general solution to the problem, which is that ./bootstrap makes hard links of files sagelib needs into the sagelib sources under src/. That way, only one copy of the relevant files needs to exist in the existing combined repository, but if sagelib is packaged separately it will retain the necessary files.

I think having that might serve as a solution here too. My feeling is that these documentation pages should be generated by the docbuilder itself. As long as it has access to information about Sage's dependencies it can do this, without the top-level ./bootstrap script having to take any special action.

In other words, what we ultimately want is to be able to build Sage's docs without direct dependency on the sage-the-distribution stuff. As it stands, I fear this PR is introducing too much direct dependency on sage-the-distribution just to be able to build Sage's docs. What we would really want is to be able to run setup.py sdist, the a source tarball just for sagelib and its docs, and later unpack it somewhere else and still be able to build the docs (and sage itself).

[jhpalmieri]

How is this supposed to work with an incremental build?

$ make bootstrap
make: `bootstrap' is up to date.
$ make configure
make: `configure' is up to date.
$ make
...
OSError: /Users/jpalmier/Desktop/Sage/git/sage/src/doc/en/installation/source.rst:228: WARNING: Include file '/Users/jpalmier/Desktop/Sage/git/sage/src/doc/en/installation/debian.txt' not found or reading it failed

[mkoeppe]

Replying to jhpalmieri:

How is this supposed to work with an incremental build?

Thanks for catching this -- I need to add some dependencies

[mkoeppe]

Replying to embray:

I think I might see part of the problem here: It is true that the information you need to generate these documentation pages lives in sage-the-distribution, above the sagelib sources. If you want to package sagelib in a stand-alone fashion you need to get access to this data somehow, but you can't, so the file is generated sort of externally to sagelib, using data from sage-the-distribution.

A similar problem applies to #29119. [...]

There is a really easy solution:

• bootstrap is only allowed in the full distribution source, not in a separate sagelib source. No hardlinks necessary. Only developers bootstrap.
• Users don't bootstrap, they start with the configure phase.
• sdist of the distribution and sdist of sagelib generate bootstrapped source archives.

[mkoeppe]

Replying to embray:

Sure, but what is required for running make sdist?

Obviously, it requires exactly a bootstrapped source tree!

It seems to me this should be done in two separate cases: One when running make sdist and one when developing out of source. In either case I don't think the bootstrap script is the appropriate place for it.

The only reason you seem to be putting it there is that it would happen to cover both cases, but I think both cases should be handled separately.

bootstrap is the phase before configure, which does not depend on the configuration of the system.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ce34ce1

Makefile: Add dependencies on build/pkgs/*.txt build/pkgs/*/distros/*.txt

[mkoeppe]

Replying to mkoeppe:

Replying to jhpalmieri:

How is this supposed to work with an incremental build?

Thanks for catching this -- I need to add some dependencies

Done.

[jhpalmieri]

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation. By the way, I'm not sure whether I like the old style

     $ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \
       git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \
       libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \
       pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \
       libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \
       libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \
       libpari-dev pari-gp2c libec-dev liblrcalc-dev \
       libm4ri-dev libm4rie-dev liblfunction-dev lcalc \
       libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliquer

or the new style

    $ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-devel

better.

Is there a ticket which adds homebrew information to the installation/source.rst?

[mkoeppe]

Replying to jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation.

The *generated* files are in src/doc/en/installation. The source files are in build/pkgs/. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.

[mkoeppe]

Replying to jhpalmieri:

Is there a ticket which adds homebrew information to the installation/source.rst?

#29104 is for homebrew. I haven't added the package information there yet -- any help is very welcome!

[jhpalmieri]

Replying to mkoeppe:

Replying to jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation.

The *generated* files are in src/doc/en/installation. The source files are in build/pkgs/. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.

But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and make fails, even with your latest branch. Is there no way to avoid this?

$ git checkout t/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual 
Switched to branch 't/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual'
$ make configure
make: `configure' is up to date.

./bootstrap -d generates the correct files, but I have to run it manually.

[dimpase]

the idea is that one can make configure is not a good one. configure is "built" by autoconf, not by make.

can we get rid of configure target?

[dimpase]

Replying to jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation. By the way, I'm not sure whether I like the old style

     $ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \
       git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \
       libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \
       pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \
       libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \
       libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \
       libpari-dev pari-gp2c libec-dev liblrcalc-dev \
       libm4ri-dev libm4rie-dev liblfunction-dev lcalc \
       libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliquer

or the new style

    $ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-devel

better.

the new style is built automatically, whereas the old one was manual. One can surely hack further to add insertion of \ after ~80 chars, but it's certainlty minor...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​688c68e

Makefile [configure]: Add dependency on bootstrap, src/doc/bootstrap

[mkoeppe]

Replying to jhpalmieri:

But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and make fails, even with your latest branch. Is there no way to avoid this?

Thanks! I've added more dependencies to the make configure target that should take care of this case.

[mkoeppe]

Replying to dimpase:

the idea is that one can make configure is not a good one. configure is "built" by autoconf, not by make.

I don't think there's anything wrong with these make targets -- also autotools have them (in maintainer mode).

[dimpase]

a typical autotools project won't have a Makefile in unbuilt state, only Makefile.am

[mkoeppe]

That is true, but there are Makefile targets that regenerate configure etc. from configure.ac.

[jhpalmieri]

That works for me, thanks.

I don't in general ever use make configure explicitly, but I was trying to get the previous versions of this branch to work so I tried it. With the most recent branch, I can just check it out and run make.

[dimpase]

is sage-print-system-package-command meant to runnable by the user, or it's a "service" script?

[mkoeppe]

It's an internal script of sage the distribution. That's why it's in build/bin.

[dimpase]

OK, the only nitpck is that on modern Fedora's it's dnf, not yum.

[mkoeppe]

Thank you!

[vbraun]

This breaks incremental builds: #29233