Only ./bootstrap should glob build/pkgs

[mkoeppe]

Right now both ./bootstrap and ./configure (via m4/sage_spkg_collect.m4) ​glob build/pkgs.

In this ticket, we move this code into bootstrap, making the generated configure script more static, more straightforward to read and debug, and slightly faster because less work (and file system accesses) is done at the run time of configure.

Before:

$ time ./configure -q --without-system-ntl
real	0m40.860s
user	0m24.260s
sys	0m16.266s

After:

real	0m32.485s
user	0m20.773s
sys	0m10.462s

There are no changes in functionality.

We make a cosmetic fix to the ./configure --help output, fixing ​https://trac.sagemath.org/ticket/33345#comment:20.

Even after this ticket, package-version.txt and dependencies are processed at configure run time, not at bootstrap time. So simple package upgrades can still be done by developers without having to run ./bootstrap.

[mkoeppe]

Branch is on top of #28095.

---

New commits:

​cf5607e	Add configure options '--enable-SPKG', --disable-SPKG' for optional/experimental SPKG
​e4675d1	Actually implement --enable-SPKG and --disable-SPKG
​a3e0e47	Change type of gmp, mpir to 'standard' (they are mutually exclusive standard packages)
​4ed7b05	Fix indentation of '--with-mp=' options in configure --help
​82ee84d	sage_spkg_enable.m4: Popdef what wass pushdef-ed
​f7a84e0	Enumerate packages in ./bootstrap; only warn about missing 'type' files; use the generated list in m4/sage_spkg_collect.m4 instead of globbing

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6ec6413	Merge branch 'develop' of git://trac.sagemath.org/sage into t/29114/no-spkg-globbing-in-configure
​9295a74	sage_spkg_enable.m4: Popdef what wass pushdef-ed
​355d147	Enumerate packages in ./bootstrap; only warn about missing 'type' files; use the generated list in m4/sage_spkg_collect.m4 instead of globbing

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bbd6521

Makefile [configure]: also depend on build/pkgs/type

[dimpase]

Could you explain what "glob" means here?

[dimpase]

This reminds me of #27373 - which becomes obsolete as soon as this ticket and #29113 is done.

[embray]

It seems this is adding a bunch of previously unnecessary complication only because #29038, which I strongly objected to, was merged. I didn't step in then because I was busy with other things. Can we please roll back #29038 instead? I'm absolute convinced there's no need for it.

[embray]

See #29118 for an alternative proposal.

[dimpase]

I think we should progress, even if the route is not optimal, and Matthias has a lot of time for working on the build system now.

I hate to see ./configure running many times as I launch make, it really slows working on build system down, and I hope #29113 will happen soon.

[embray]

I should add, some of the changes in this ticket look reasonable. But it's mixed up with a bunch of other stuff (the way this is built on top of #28095 in particular is confusing). I'd like to start with #29118 and then see some fixes to individual bugs that predate #29038 .

[embray]

Replying to dimpase:

I think we should progress, even if the route is not optimal, and Matthias has a lot of time for working on the build system now.

Progress on what though? I don't think you can "progress" on unsolid footing. I should note that part of the motivation behind #29038 is to enable the approach in #27824 which I also think is overcomplicated.

I don't think we should be making complicated changes on top of complicated changes to enable even more complicated changes. I hate to sound like Jeroen here because I've been on the other side of disagreements like this with him. But he'd probably be saying the same if he were here and I'd agree this time.

I would like to know what the bigger picture is to all this before making changes that don't seem well-motivated. Don't commit a sunk cost fallacy especially when it's relatively early...

[dimpase]

If you want to talk about simplification, why don't we just drop sage -i/-f instead of trying to keep this awful hack - something I proposed a year ago, and Jeroen objected...

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​1f193bf	Issue only a warning, not an error, if there are stray files in build/pkgs
​a04733a	Merge branch 't/29120/one-line-fix-configure-too-sensitive-to-stray-files-in-build-pkgs' into t/29114/no-spkg-globbing-in-configure
​25a55e0	Enumerate packages in ./bootstrap; only warn about missing 'type' files; use the generated list in m4/sage_spkg_collect.m4 instead of globbing
​7214ed7	Makefile [configure]: also depend on build/pkgs/type

[mkoeppe]

I've rebased on top of #29120, which has the "hot fix", which may be easier to review and should go into the next beta.

The present ticket, #29114, is not so urgent.

[mjo]

Since this list changes maybe once a year, we could just write it into configure.ac and avoid that part of the bootstrap process.

[mkoeppe]

I think that may be too radical a change (although I am personally in favor of it - agreeing with the autotools policies) - Sage has traditionally used source discovery instead of listing files explicitly. That's why in this ticket I just want to put the source discovery for build/pkgs into 1 script

[mkoeppe]

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​0f59e29

Enumerate packages in ./bootstrap; only warn about missing 'type' files; use the generated list in m4/sage_spkg_collect.m4 instead of globbing

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a34f10b	bootstrap, m4/sage_spkg_collect.m4: Pass SPKG_TYPE to SAGE_SPKG
​025eb95	bootstrap: Improve the SPKG.rst title parser
​5813473	bootstrap: First SAGE_SPKG_CONFIGURE, then SAGE_SPKG_FINALIZE
​c5c6caa	m4/sage_spkg_collect.m4 (SAGE_SPKG_COLLECT_INIT): Rename from SAGE_SPKG_INIT

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​54974a3

m4/sage_spkg_collect.m4 (SAGE_SPKG_COLLECT_INIT): Rename from SAGE_SPKG_INIT

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2eba1f4	m4/sage_spkg_collect.m4 (SAGE_SPKG_FINALIZE): Make SPKG_NAME, SPKG_TYPE m4 definitions, not shell variables
​1e9c638	m4/sage_spkg_collect.m4: Distinguish cases of SPKG_TYPE at macroexpansion time, not runtime
​7310bd5	bootstrap: First SAGE_SPKG_CONFIGURE, then SAGE_SPKG_FINALIZE (fixup)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1eceb52

bootstrap: Determine SPKG_SOURCE here, pass it as a macro argument to SAGE_SPKG_FINALIZE

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a013698	m4/sage_spkg_collect.m4: Distinguish underscore packages at macroexpansion time
​8421854	m4/sage_spkg_collect.m4: Handle in_sdist at macroexpansion time
​7115a72	m4/sage_spkg_collect.m4: Suppress some whitespace in the macroexpansion

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​35fa89f	bootstrap: Determine SPKG_TREE_VAR here, pass it as a macro argument to SAGE_SPKG_FINALIZE
​8070dd9	m4/sage_spkg_collect.m4: Make want_spkg a macro definition only
​be824d6	m4/sage_spkg_collect.m4: Replace read from requirements.txt by use of macro argument

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e796fcf

m4/sage_spkg_enable.m4: Do not AC_REQUIRE([SAGE_SPKG_COLLECT_INIT]) here

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a6d70c1

m4/sage_spkg_collect.m4: Make SAGE_NEED_SYSTEM_PACKAGES_VAR a macro

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7d77884	bootstrap: More "installing..."
​3b04496	bootstrap: Consolidate output redirections
​e7b8c3b	m4/sage_spkg_collect.m4: Add missing m4_popdef

[jhpalmieri]

./configure -q is a useful command. I see this output:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named lzma
./configure: line 39392: ,: command not found

The first is probably coming from the line if "$SAGE_BOOTSTRAP_PYTHON" -c "import lzma"; then :, no big deal. The other error is from a comma all by itself on line 39392 (that's with develop; it's line 42361 with this branch). Maybe an extra comma in build/pkgs/qhull/spkg-configure.m4?

[mkoeppe]

This is unrelated to the ticket and fixed in #33247

[jhpalmieri]

In principle this looks good to me, and Sage builds. It would be great if someone more familiar with .m4 code could check the details.

[mjo]

I looked through it all and don't see anything too scary. My one question is about the grep -v ^= that was added to bootstrap. It looks like that only affects the sagemath_categories, sagemath_objects, and sage_docbuild packages. Why do those have ======... headers in their SPKG.rst to begin with?

[mkoeppe]

Replying to mjo:

It looks like that only affects the sagemath_categories, sagemath_objects, and sage_docbuild packages. Why do those have ======... headers in their SPKG.rst to begin with?

These files double as README.rst for these distributions, so this affects how it displays on PyPI.

[mjo]

Ok, I don't believe in parsing SPKG.rst for package information anyway, so I don't mind the hack. (In the futue, I would rather we quit parsing that file than try to make them all nice and consistent.)

Everything else looks reasonable. Nothing broke in my sage build and there aren't any new subtle ./configure errors logged to the console.

John, don't forget to add yourself as a reviewer if you want to.

[mkoeppe]

Thanks.