#26964 closed defect (fixed)
./bootstrap: Format build/pkgs/SPKG/distros/ information to produce the apt-get/yum command lines shown in the installation manual
Reported by: | embray | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | sage-9.1 |
Component: | documentation | Keywords: | |
Cc: | dimpase, embray, jhpalmieri | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | John Palmieri, Dima Pasechnik |
Report Upstream: | N/A | Work issues: | |
Branch: | 688c68e (Commits, GitHub, GitLab) | Commit: | |
Dependencies: | Stopgaps: |
Description (last modified by )
When trying to build Sage on a very bare fedora install (e.g. a Docker container) the packages listed in the build instructions at:
https://doc.sagemath.org/html/en/installation/source.html#linux-prerequisite-installation
are insufficient. At the very least it also needs findutils
and which
, and currently it also requires python2
, though that should be fixed as part of #26953. The need for which
comes, at the very least, from MPIR's configure
script, which uses which
at least on Linux.
With #29053, the necessary distribution packages for an installation are cataloged in build/pkgs/fedora.txt
, build/pkgs/SPKG/distros/fedora.txt
and similar for other distributions.
In this ticket, we add to bootstrap
some code to keep the yum
(and apt-get
for debian) command-lines in the installation manual up to date. #29053 prepared this by isolating these command lines in separate .txt files in src/doc
.
Change History (49)
comment:1 Changed 4 years ago by
- Summary changed from missing packages for building Sage on fedora to missing packages in instructions for building Sage on fedora
comment:2 Changed 4 years ago by
- Milestone changed from sage-8.6 to sage-8.7
comment:3 Changed 3 years ago by
- Milestone changed from sage-8.7 to sage-pending
Removing most of the rest of my open tickets out of the 8.7 milestone, which should be closed.
comment:5 Changed 3 years ago by
- Cc dimpase embray added
- Dependencies #29053 deleted
- Description modified (diff)
- Milestone changed from sage-pending to sage-9.1
- Summary changed from missing packages in instructions for building Sage on fedora to ./bootstrap: Format build/pkgs/SPKG/distros/ information to produce the apt-get/yum command lines shown in the installation manual
comment:6 Changed 3 years ago by
I'm not sure how I feel about using the bootstrap script for this purpose. I think I would prefer if it were done by configure instead. Reason: The bootstrap script should really be limited to tasks specific to generating the configure
file (whether running autoreconf or downloadng it). Other than that, I think this is a great idea!
comment:7 follow-up: ↓ 8 Changed 3 years ago by
- Status changed from new to needs_review
No, no, bootstrap
is exactly the right phase for this, not configure
. Let me explain more.
bootstrap
will prepare a source tree so that
make sdist
can be runconfigure
can be run by a user (who does not have autotools)src/setup.py sdist
can be run.src/configure
can be run (#29119)
None of this depends on the configuration determined by configure
!
As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.
comment:8 in reply to: ↑ 7 ; follow-up: ↓ 10 Changed 3 years ago by
- Status changed from needs_review to needs_work
Replying to mkoeppe:
As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.
I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?
I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a src/bootstrap
script for sagelib itself, so maybe something like this would make more sense to go there.
Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.
comment:9 follow-up: ↓ 11 Changed 3 years ago by
Couldn't this also be done as part of make sdist
?
comment:10 in reply to: ↑ 8 Changed 3 years ago by
Replying to embray:
Replying to mkoeppe:
As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.
I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?
Both.
Currently, of course, only sage-the-distribution has a working sdist.
sagelib's sdist needs fixing. You might remember ticket #21516 - Fix sagelib sdist (src/setup.py sdist
).
I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a
src/bootstrap
script for sagelib itself, so maybe something like this would make more sense to go there.
In my opinion, there should only be one developer-invoked bootstrap
script (the one for the distribution) -- because everything will depend on the distribution's build/pkgs
files.
SAGE_ROOT/bootstrap
could certainly call a new script src/bootstrap
for generating files that end up in src/
. If you create that in #29119, fine.
Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.
Sure. #29041 (src/requirements.txt, src/constraints.txt, src/setup.cfg) has a branch, needs review.
comment:11 in reply to: ↑ 9 ; follow-up: ↓ 24 Changed 3 years ago by
Replying to embray:
Couldn't this also be done as part of
make sdist
?
No, because developers who build sage out of a git worktree do not run make sdist
.
comment:12 Changed 3 years ago by
- Branch set to u/mkoeppe/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual
comment:13 Changed 3 years ago by
- Commit set to 438bc7fb739b571e82674e69210f81e7ec29c5e8
Branch pushed to git repo; I updated commit sha1. New commits:
438bc7f | ./bootstrap, src/doc/bootstrap: Auto-generate apt-get/yum command lines
|
comment:14 Changed 3 years ago by
- Status changed from needs_work to needs_review
comment:15 Changed 3 years ago by
- Commit changed from 438bc7fb739b571e82674e69210f81e7ec29c5e8 to 3810bc9835050073f663109d576eae1157bf4f77
Branch pushed to git repo; I updated commit sha1. New commits:
3810bc9 | Clean and ignore the generated files
|
comment:16 Changed 3 years ago by
- Cc jhpalmieri added
comment:17 Changed 3 years ago by
- Commit changed from 3810bc9835050073f663109d576eae1157bf4f77 to bdd69c3bd85b2d33a253ef8c7edde6ca8adffb85
Branch pushed to git repo; I updated commit sha1. New commits:
bdd69c3 | build/bin/write-dockerfile.sh: Copy src/doc/bootstrap early
|
comment:18 Changed 3 years ago by
- Commit changed from bdd69c3bd85b2d33a253ef8c7edde6ca8adffb85 to 8be072b7244b342c6b70737cc36409121b70b41b
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
8be072b | build/bin/write-dockerfile.sh: Copy src/doc/bootstrap early
|
comment:19 Changed 3 years ago by
- Commit changed from 8be072b7244b342c6b70737cc36409121b70b41b to 38741947257da00371142bd7ca4099909717cc50
comment:20 Changed 3 years ago by
something is not quite right:
(sage-sh) dimpase@clpc171:sage$ ./build/bin/sage-print-system-package-command debian gmp sudo apt-get gmp (sage-sh) dimpase@clpc171:sage$ ./build/bin/sage-print-system-package-command conda gmp # gmp the following packages:
- so
debian
works,conda
does not...
comment:21 Changed 3 years ago by
you forgot the verb install
in both commands
comment:22 Changed 3 years ago by
- Commit changed from 38741947257da00371142bd7ca4099909717cc50 to eb7a1d9b9a16c40a5a30c9451d14b5e078b88007
Branch pushed to git repo; I updated commit sha1. New commits:
eb7a1d9 | build/bin/sage-print-system-package-command: Print usage message if not enough arguments provided
|
comment:23 Changed 3 years ago by
- Commit changed from eb7a1d9b9a16c40a5a30c9451d14b5e078b88007 to 146a52052b48c4f97aaa6be1122f9a63d5f44e8d
Branch pushed to git repo; I updated commit sha1. New commits:
146a520 | build/bin/sage-print-system-package-command: Do not print a command if no packages are to be installed
|
comment:24 in reply to: ↑ 11 ; follow-up: ↓ 29 Changed 3 years ago by
Replying to mkoeppe:
Replying to embray:
Couldn't this also be done as part of
make sdist
?No, because developers who build sage out of a git worktree do not run
make sdist
.
Sure, but what is required for running make sdist
? It seems to me this should be done in two separate cases: One when running make sdist
and one when developing out of source. In either case I don't think the bootstrap script is the appropriate place for it.
The only reason you seem to be putting it there is that it would happen to cover both cases, but I think both cases should be handled separately.
comment:25 follow-up: ↓ 28 Changed 3 years ago by
I think I might see part of the problem here: It is true that the information you need to generate these documentation pages lives in sage-the-distribution, above the sagelib sources. If you want to package sagelib in a stand-alone fashion you need to get access to this data somehow, but you can't, so the file is generated sort of externally to sagelib, using data from sage-the-distribution.
A similar problem applies to #29119. In the prototype for that I am working on a more general solution to the problem, which is that ./bootstrap
makes hard links of files sagelib needs into the sagelib sources under src/
. That way, only one copy of the relevant files needs to exist in the existing combined repository, but if sagelib is packaged separately it will retain the necessary files.
I think having that might serve as a solution here too. My feeling is that these documentation pages should be generated by the docbuilder itself. As long as it has access to information about Sage's dependencies it can do this, without the top-level ./bootstrap
script having to take any special action.
In other words, what we ultimately want is to be able to build Sage's docs without direct dependency on the sage-the-distribution stuff. As it stands, I fear this PR is introducing too much direct dependency on sage-the-distribution just to be able to build Sage's docs. What we would really want is to be able to run setup.py sdist
, the a source tarball just for sagelib and its docs, and later unpack it somewhere else and still be able to build the docs (and sage itself).
comment:26 follow-up: ↓ 27 Changed 3 years ago by
How is this supposed to work with an incremental build?
$ make bootstrap make: `bootstrap' is up to date. $ make configure make: `configure' is up to date. $ make ... OSError: /Users/jpalmier/Desktop/Sage/git/sage/src/doc/en/installation/source.rst:228: WARNING: Include file '/Users/jpalmier/Desktop/Sage/git/sage/src/doc/en/installation/debian.txt' not found or reading it failed
comment:27 in reply to: ↑ 26 ; follow-up: ↓ 31 Changed 3 years ago by
Replying to jhpalmieri:
How is this supposed to work with an incremental build?
Thanks for catching this -- I need to add some dependencies
comment:28 in reply to: ↑ 25 Changed 3 years ago by
Replying to embray:
I think I might see part of the problem here: It is true that the information you need to generate these documentation pages lives in sage-the-distribution, above the sagelib sources. If you want to package sagelib in a stand-alone fashion you need to get access to this data somehow, but you can't, so the file is generated sort of externally to sagelib, using data from sage-the-distribution.
A similar problem applies to #29119. [...]
There is a really easy solution:
bootstrap
is only allowed in the full distribution source, not in a separate sagelib source. No hardlinks necessary. Only developers bootstrap.- Users don't bootstrap, they start with the configure phase.
- sdist of the distribution and sdist of sagelib generate bootstrapped source archives.
comment:29 in reply to: ↑ 24 Changed 3 years ago by
Replying to embray:
Sure, but what is required for running
make sdist
?
Obviously, it requires exactly a bootstrapped source tree!
It seems to me this should be done in two separate cases: One when running
make sdist
and one when developing out of source. In either case I don't think the bootstrap script is the appropriate place for it.The only reason you seem to be putting it there is that it would happen to cover both cases, but I think both cases should be handled separately.
bootstrap
is the phase before configure
, which does not depend on the configuration of the system.
comment:30 Changed 3 years ago by
- Commit changed from 146a52052b48c4f97aaa6be1122f9a63d5f44e8d to ce34ce1fcc5e8669c9e05b758bb8576cc8afa8d1
Branch pushed to git repo; I updated commit sha1. New commits:
ce34ce1 | Makefile: Add dependencies on build/pkgs/*.txt build/pkgs/*/distros/*.txt
|
comment:31 in reply to: ↑ 27 Changed 3 years ago by
Replying to mkoeppe:
Replying to jhpalmieri:
How is this supposed to work with an incremental build?
Thanks for catching this -- I need to add some dependencies
Done.
comment:32 follow-ups: ↓ 33 ↓ 34 ↓ 37 Changed 3 years ago by
That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt
but rather in src/doc/en/installation
. By the way, I'm not sure whether I like the old style
$ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \ git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \ libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \ pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \ libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \ libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \ libpari-dev pari-gp2c libec-dev liblrcalc-dev \ libm4ri-dev libm4rie-dev liblfunction-dev lcalc \ libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliquer
or the new style
$ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-devel
better.
Is there a ticket which adds homebrew information to the installation/source.rst
?
comment:33 in reply to: ↑ 32 ; follow-up: ↓ 35 Changed 3 years ago by
Replying to jhpalmieri:
That doesn't help, because the necessary text files are not
build/pkgs/*.txt build/pkgs/*/distros/*.txt
but rather insrc/doc/en/installation
.
The *generated* files are in src/doc/en/installation
. The source files are in build/pkgs/
. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.
comment:34 in reply to: ↑ 32 Changed 3 years ago by
Replying to jhpalmieri:
Is there a ticket which adds homebrew information to the
installation/source.rst
?
#29104 is for homebrew. I haven't added the package information there yet -- any help is very welcome!
comment:35 in reply to: ↑ 33 ; follow-up: ↓ 39 Changed 3 years ago by
Replying to mkoeppe:
Replying to jhpalmieri:
That doesn't help, because the necessary text files are not
build/pkgs/*.txt build/pkgs/*/distros/*.txt
but rather insrc/doc/en/installation
.The *generated* files are in
src/doc/en/installation
. The source files are inbuild/pkgs/
. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.
But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and make
fails, even with your latest branch. Is there no way to avoid this?
$ git checkout t/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual Switched to branch 't/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual' $ make configure make: `configure' is up to date.
./bootstrap -d
generates the correct files, but I have to run it manually.
comment:36 follow-up: ↓ 40 Changed 3 years ago by
the idea is that one can make configure
is not a good one. configure
is "built" by autoconf, not by make.
can we get rid of configure
target?
comment:37 in reply to: ↑ 32 Changed 3 years ago by
Replying to jhpalmieri:
That doesn't help, because the necessary text files are not
build/pkgs/*.txt build/pkgs/*/distros/*.txt
but rather insrc/doc/en/installation
. By the way, I'm not sure whether I like the old style$ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \ git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \ libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \ pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \ libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \ libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \ libpari-dev pari-gp2c libec-dev liblrcalc-dev \ libm4ri-dev libm4rie-dev liblfunction-dev lcalc \ libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliqueror the new style
$ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-develbetter.
the new style is built automatically, whereas the old one was manual.
One can surely hack further to add insertion of \
after ~80 chars, but it's certainlty minor...
comment:38 Changed 3 years ago by
- Commit changed from ce34ce1fcc5e8669c9e05b758bb8576cc8afa8d1 to 688c68e7a4d0b17d0a8093585389c6c4f2fc8c27
Branch pushed to git repo; I updated commit sha1. New commits:
688c68e | Makefile [configure]: Add dependency on bootstrap, src/doc/bootstrap
|
comment:39 in reply to: ↑ 35 Changed 3 years ago by
Replying to jhpalmieri:
But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and
make
fails, even with your latest branch. Is there no way to avoid this?
Thanks!
I've added more dependencies to the make configure
target that should take care of this case.
comment:40 in reply to: ↑ 36 Changed 3 years ago by
Replying to dimpase:
the idea is that one can
make configure
is not a good one.configure
is "built" by autoconf, not by make.
I don't think there's anything wrong with these make targets -- also autotools have them (in maintainer mode).
comment:41 Changed 3 years ago by
a typical autotools project won't have a Makefile in unbuilt state, only Makefile.am
comment:42 Changed 3 years ago by
That is true, but there are Makefile targets that regenerate configure etc. from configure.ac.
comment:43 Changed 3 years ago by
That works for me, thanks.
I don't in general ever use make configure
explicitly, but I was trying to get the previous versions of this branch to work so I tried it. With the most recent branch, I can just check it out and run make
.
comment:44 Changed 2 years ago by
is sage-print-system-package-command
meant to runnable by the user, or it's a "service" script?
comment:45 Changed 2 years ago by
It's an internal script of sage the distribution. That's why it's in build/bin.
comment:46 Changed 2 years ago by
- Reviewers set to John Palmieri, Dima Pasechnik
- Status changed from needs_review to positive_review
OK, the only nitpck is that on modern Fedora's it's dnf
, not yum
.
comment:47 Changed 2 years ago by
Thank you!
comment:48 Changed 2 years ago by
- Branch changed from u/mkoeppe/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual to 688c68e7a4d0b17d0a8093585389c6c4f2fc8c27
- Resolution set to fixed
- Status changed from positive_review to closed
comment:49 Changed 2 years ago by
- Commit 688c68e7a4d0b17d0a8093585389c6c4f2fc8c27 deleted
This breaks incremental builds: #29233
Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.