Add script packages build/pkgs/_prereq, build/pkgs/_bootstrap

[mkoeppe]

As a follow-up to #29053, we move the build/pkgs/*.txt files to a more systematic location. This will simplify some scripts.

• build/pkgs/_prereq/distros/debian.txt etc. - records the minimal requirements (system packages) for installing from source. (we move the current build/pkgs/debian.txt etc. there)
• build/pkgs/_bootstrap/distros/debian.txt etc. - would record the additional requirements for running ./bootstrap (we move the current build/pkgs/debian-bootstrap.txt there)

Related:

• #28998: Add script-type _spkg_check package that installs tox, mock, pytest etc.
• #29557: build/pkgs/_recommended - records additional recommended packages such as texlive-xetex (see ​https://wiki.sagemath.org/prerequisites/Ubuntu)

Possible follow-up:

• build/pkgs/_toolchain/dependencies - to replace some special Makefile targets currently in build/make/Makefile.in.

[mkoeppe]

for example on fedora-26-standard sagetex testsuite fails:

! LaTeX Error: File `makecmds.sty' not found.

Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)

Enter file name: 
! Emergency stop.
<read *> 
         
l.78 \RequirePackage
                    {ifpdf}^^M

[mkoeppe]

Installation manual also recommends Tcl/Tk? - what are they used for in Sage? ​http://doc.sagemath.org/html/en/installation/source.html#tcl-tk

---

New commits:

​bf2dfdf

build/pkgs/_recommended: New

[mkoeppe]

Best done on top of #30951

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​0281b21

tox.ini: Simplify, use sage-get-system-packages, sage-package list

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3abae24

tox.ini: Fixup - always include _bootstrap

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​36f62de	build/bin/write-dockerfile.sh: Remove special casing of build/pkgs/$SYSTEM.txt
​06edf3b	src/doc/en/developer/portability_testing.rst: Update paths in documentation

[gh-tobiasdiez]

I'm not sure about the naming of the local- environments. Why should I expect as a developer that local-sudo installs system packages? What about local-system-packages-ubuntu (for root) and local-sudo-system-packages-ubuntu (for sudo)?

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​0e97683	tox.ini: Simplify, use sage-get-system-packages, sage-package list
​da3f657	tox.ini: Fixup - always include _bootstrap
​8fde1dc	build/bin/write-dockerfile.sh: Remove special casing of build/pkgs/$SYSTEM.txt
​8d725b8	src/doc/en/developer/portability_testing.rst: Update paths in documentation

[mkoeppe]

I have taken out the addition of the local-root environments to keep this ticket more narrowly focused. Let's return to this in #30944.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bd7f66c

Makefile (configure): Update dependency on moved files build/pkgs/*.txt

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5e9366d	Merge tag '9.3.beta2' into t/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_
​6b9f3a3	Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b0459c3	Merge tag '9.3.beta2' into t/30951/tox_ini__build_bin_write_dockerfile_sh__simplify_by_using_new_options_of__sage__package_list_
​a8708dd	build/bin/write-dockerfile.sh: Change bash -x back to bash
​8a575d4	tox.ini (minimal): Use SAGE_PACKAGE_LIST_ARGS=_prereq
​f696cfc	Merge branch 't/30951/tox_ini__build_bin_write_dockerfile_sh__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap
​a5ecf3c	bootstrap: Remove --enable-_recommended etc.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0e35938

m4/sage_spkg_collect.m4: Remove _recommended etc. from the configure package summary

[slabbe]

Am I right to think that only the last two commits of the branch should be reviewed here?

[mkoeppe]

No, there are actually a few more commits earlier that don't come from the merged dependencies

[slabbe]

Ok. Then, could the branch be rebased on top of the dependencies, to make the changes made by this ticket consecutive and easier to read (during the review and/or after).

[mkoeppe]

Hoping we can make progress on this ticket this week - ​https://wiki.sagemath.org/days111

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​2792314	Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap
​c901ba6	Add script packages build/pkgs/_prereq, build/pkgs/_bootstrap
​5b2f4fb	build/pkgs/_prereq/spkg-configure.m4: New
​7ad8286	src/doc/bootstrap: Remove special casing of build/pkgs/$SYSTEM.txt - _prereq is now just a standard package with spkg-configure.m4
​ad24294	tox.ini: Simplify, use sage-get-system-packages, sage-package list
​1a77a6b	build/bin/write-dockerfile.sh: Remove special casing of build/pkgs/$SYSTEM.txt
​35241e7	src/doc/en/developer/portability_testing.rst: Update paths in documentation
​68baff1	Makefile (configure): Update dependency on moved files build/pkgs/*.txt
​fae4bd0	bootstrap: Remove --enable-_recommended etc.
​a9bd145	m4/sage_spkg_collect.m4: Remove _recommended etc. from the configure package summary

[mkoeppe]

Rebased on top of #30947 and current beta

[slabbe]

Thank you. I do have lots of other todos this week, but I hope to be able to review few tickets during the week. I will start with this one and the other ones you put me in cc.

[slabbe]

I am not sure if commit a9bd145 works because I still see underscore packages listed in the list of SPKGs which did not find equivalent system packages during the run of ./configure:

    notice: the following SPKGs did not find equivalent system packages:

        _recommended boost coxeter3 gp2c igraph isl libsemigroups pari_elldata pari_galpol pari_nftables pari_seadata
        
checking for the package system in use... debian
configure:

    hint: installing the following system packages, if not
    already present, is recommended and may avoid having to
    build them (though some may have to be built anyway):

      $ sudo apt-get update 
      $ sudo apt-get install  texlive-latex-extra texlive-xetex latexmk pandoc dvipng default-jdk ffmpeg libavdevice-dev libboost-dev pari-gp2c libigraph-dev libisl-dev

    After installation, re-run configure using:

      $ ./config.status --recheck && ./config.status

[slabbe]

Ok, I see, the goal of commit a9bd145 was to remove _recommended elsewhere in the log:

config.log on 9.3.beta3:

## ----------------------------------------------------------------------- ##
## Build status for each package:                                          ##
## ----------------------------------------------------------------------- ##
4ti2-1.6.7.p0:                               does not support check for system package; optional, will be installed as an SPKG
_recommended-none:                           no suitable system package; optional, use "./configure --enable-_recommended" to install
alabaster-0.7.12:                            does not support check for system package; will be installed as an SPKG

config.log with this branch on top of 9.3.beta3:

## ----------------------------------------------------------------------- ##
## Build status for each package:                                          ##
## ----------------------------------------------------------------------- ##
4ti2-1.6.7.p0:                               does not support check for system package; optional, will be installed as an SPKG
_bootstrap-none:                             does not support check for system package; optional, use "./configure --enable-_bootstrap" to install
_prereq-none:                                does not support check for system package; will be installed as an SPKG
_recommended-none:                           no suitable system package; optional, use "./configure --enable-_recommended" to install
alabaster-0.7.12:                            does not support check for system package; will be installed as an SPKG

config.log with this branch on top of 9.3.beta3 (after running make first):

## ----------------------------------------------------------------------- ##
## Build status for each package:                                          ##
## ----------------------------------------------------------------------- ##
4ti2-1.6.7.p0:                               does not support check for system package; optional, will be installed as an SPKG
alabaster-0.7.12:                            does not support check for system package; will be installed as an SPKG
appnope-0.1.0.p0:                            does not support check for system package; will be installed as an SPKG

Still, I would suggest to also remove it from the list notice: the following SPKGs did not find equivalent system packages:.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​184ae6d	Merge tag '9.3.beta3' into t/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_
​084fbf6	src/doc/bootstrap: Use ./sage
​571cc49	Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap

[mkoeppe]

Merged new version of #30947

[mkoeppe]

Replying to slabbe:

Still, I would suggest to also remove it from the list notice: the following SPKGs did not find equivalent system packages:.

I agree it's not pretty. But all of the SPKGs listed there expand/transform to system packages that are suggested to be installed. Without the _recommended, it will be a mystery where some of the suggested system packages are coming from. So I think it's better to keep it there.

[slabbe]

Ok. works for me.

[mkoeppe]

Thank you!

[vbraun]

Merge conflict

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​9c5e8a2	add references to `bin/sage-build-env` and `bin/sage-build-env-config`
​4b8933b	remove incorrect `$` in `build/bin/sage-build-env`
​071b8cc	Revert "add references to `bin/sage-build-env` and `bin/sage-build-env-config`"
​c51e752	correct comments to makefile
​aa7ecd6	fix that we might have changed $FCFLAGS during configure
​94326db	better solution for dealing with -ffree-form
​87dae61	keep FCFLAGS during gfortran configure
​7577dd4	added quotes
​3bb3099	prepare preciouos variables
​c529729	Merge commit '3bb309944b7e8542b2ac88ed3c9d9a60e68644d7' of git://trac.sagemath.org/sage into t/29124/script-packages-prereq-toolchain-bootstrap

[mkoeppe]

Merged #30375 in the hope to resolve merge conflicts

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​b73d6f0

Merge tag '9.3.beta4' into t/29124/script-packages-prereq-toolchain-bootstrap

[mkoeppe]

Merged current beta

[vbraun]

**********************************************************************
File "src/sage/misc/package.py", line 427, in sage.misc.package.standard_packages
Failed example:
    installed[0], installed[-1]  # optional - build
Expected:
    ('alabaster', 'zn_poly')
Got:
    ('_prereq', 'zn_poly')
**********************************************************************
1 item had failures:
   1 of   4 in sage.misc.package.standard_packages
    [49 tests, 1 failure, 3.31 s]
----------------------------------------------------------------------

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​97942e3	sage.misc.package.pip_installed_packages: Only normalize when optional arg normalization='spkg'
​6706182	sage.misc.package.list_packages: Return early if SAGE_PKGS is not available
​9258849	sage.misc.package.list_packages: Streamline handling of exclude_pip
​7f80e3e	sage.misc.package.list_packages: Actually add new items to the list
​78bae1f	Merge tag '9.3.beta3' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​3cf5ee4	Merge commit 'a50ddf88975086b14a49895e371477df00fd57b5' of git://trac.sagemath.org/sage into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​b4927e1	sage.misc.package: Remove/adjust non-robust doctests
​e5fe752	Merge tag '9.3.beta4' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​78ff9d5	src/sage/misc/package.py: Add one more # optional - build
​a44042f	Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap

[mkoeppe]

Merged #30940, which makes these doctests more robust

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​64bde5f	Merge tag '9.3.beta5' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available
​e9a7572	src/sage/misc/package.py: Improve source formatting
​c7bcda9	Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap
​9988c5f	ci-cygwin*.yml: Adjust to new script packages _bootstrap, _prereq

[mkoeppe]

The last commit fixes the issue observed in #29124:

moved the files build/pkgs/*.txt to new locations but forgot to update the CI scripts for Cygwin, which had led to failures like this (​https://github.com/mkoeppe/sage/runs/1561097256):

sed: can't read ./build/pkgs/cygwin.txt: No such file or directory
sed: can't read ./build/pkgs/cygwin-bootstrap.txt: No such file or directory
Chocolatey v0.10.15
Package name is required. Please pass at least one package name to install.
Error: Process completed with exit code 1.

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​6f3b864	build/pkgs/[s-t]*/SPKG.rst: Reformat title in format 'spkg_name: Short description'
​23558f8	build/pkgs/[t-z]*/SPKG.rst: Reformat title in format 'spkg_name: Short description' -- completes coverage of all standard packages
​17789df	build/pkgs/[z-z]*/SPKG.rst: Reformat title in format 'spkg_name: Short description' -- completes coverage of all optional packages
​944408d	build/pkgs/[a-z]*/SPKG.rst: Reformat title in format 'spkg_name: Short description' -- completes coverage of all experimental packages
​feafa60	src/doc/en/installation/standard_packages.rst: Remove
​58c63e1	src/doc/bootstrap: Add brief explanation of the standard/optional/experimental package types
​54d4fb1	extra line after version
​17f2f24	Merge branch 't/29655/improve_build_pkgs___spkg_rst' into t/29124/script-packages-prereq-toolchain-bootstrap
​e7a768b	build/pkgs/{_bootstrap,_prereq}/SPKG.rst: Reformat title
​6ff6545	Move build/pkgs/opensuse*.txt to build/pkgs/{_bootstrap,_prereq}/distros

[mkoeppe]

Still needs review

[dimpase]

./bootstrap is getting very slow - 3 times slower than ./configure, over 60 sec on a relatively fast machine!

[dimpase]

On this branch I'm getting

sage -t --warn-long 53.2 --random-seed=0 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 219, in sage.tests.cmdline.test_executable
Failed example:
    print(out)  # optional - build
Expected:
    Found local metadata for sqlite-...
    SQLite
    ======
    ...
    SQLite is a software library that implements a self-contained,
    serverless, zero-configuration, transactional SQL database engine.
    ...
Got:
    sqlite: An SQL database engine
    ==============================
    <BLANKLINE>
    Description
    -----------
    <BLANKLINE>
    SQLite is a software library that implements a self-contained,
    serverless, zero-configur
...

and one more similar failure in this file. Not sure whether it's coming from this ticket, or one of dependencies.

[mkoeppe]

Thanks for catching this. It's coming from #29655

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2eac7a1	src/doc/bootstrap: Use sage-spkg-info directly instead of 'sage --info', much faster now
​3fe8d4c	src/sage/tests/cmdline.py: Adjust doctests
​b57fecc	Merge branch 't/29655/improve_build_pkgs___spkg_rst' into t/29124/script-packages-prereq-toolchain-bootstrap

[dimpase]

hmm, now I see

File "src/sage/tests/cmdline.py", line 219, in sage.tests.cmdline.test_executable
Failed example:
    print(out)  # optional - build
Expected:
    SQLite
    ======
    ...
    SQLite is a software library that implements a self-contained,
    serverless, zero-configuration, transactional SQL database engine.
    ...
Got:
    sqlite: An SQL database engine
    ==============================
    <BLANKLINE>
    Description
    -----------
    <BLANKLINE>
    SQLite is a software library that implements a self-contained,
    serverless, zero-configuration, transactional SQL database engine.
    <BLANKLINE>
...

(and the same in line 232)

So more changes of these test outputs are needed.

The different heading (sqlite: An SQL database engine) comes from src/doc/en/reference/spkg/sqlite.rst, as far as me and grep can tell.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​49ad8ae	src/sage/tests/cmdline.py: Adjust doctest properly
​e8da106	Merge branch 't/29655/improve_build_pkgs___spkg_rst' into t/29124/script-packages-prereq-toolchain-bootstrap

[dimpase]

OK, good.

[mkoeppe]

Thanks!