Opened 3 years ago
Last modified 17 months ago
#21566 new task
Task ticket: Make sage (the distribution) behave like a standard autotools package, to the extent possible
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-7.5 |
Component: | build | Keywords: | |
Cc: | vbraun, jdemeyer, fbissey, embray, dimpase, was, mmezzarobba, thansen, gh-timokau | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
The goal of this ticket is to be able to explain to experienced Unix users what sage-the-distribution is, as follows:
If you download sage-the-distribution and do
./configure --prefix=/SOMEWHERE && makethen that's the same as doing a sequence of about 100 times:
wget SOMEPACKAGE.tar.gz && tar xf SOMEPACKAGE.tar.gz && (cd SOMEPACKAGE \ && ./configure --prefix=/SOMEWHERE && make && make install)(and about a 50 times
pip install SOMEPACKAGE
). sage-the-distribution has figured out the right order of installing these packages, tricky configure options so that everything works, and on top has fixes for various outdated/handwritten/missing build systems of various packages.
Thus it is beyond the scope of this ticket:
- to separate
make
andmake install
. Ourmake install
is a no-op. ANY discussion of this needs to go to ticket #21495, not here.
We will implement this goal without sacrificing any of the traditional convenience features that Sage-the-distribution has provided for the casual user (such as sage -i
for installing packages; and that ./configure
is an optional step of the installation process).
We have a separate task ticket for the following:
- #21507: Make sagelib (sage-the-Python-library) a normal Python package, installable by
setup.py
,pip
, etc. and eventually even via PyPI. We will NOT turn sagelib to an automake package (which was proposed in #14807).
(There will be some interaction with some of the steps of that ticket.)
Included on this ticket are the following steps.
Implement standard features expected of an autotools build system.
- Choosing the installation hierarchy (
configure --prefix=SAGE_LOCAL
). Right now it is the subdirectorylocal
ofSAGE_ROOT
. - By allowing the user to choose the installation hierarchy, there are new requirements. What is installed there should run without requiring environment variables to be set. It should not refer to the environment variables
SAGE_LOCAL
, orSAGE_ROOT
(orSAGE_SRC_ROOT
). An exception could perhaps be made for the latter for "debugging" or "source inspection" facilities: - #21539:
make V=0
should silence the build - #21775: make distclean: Don't delete the install hierarchy (SAGE_LOCAL)
- #21589: Make
make V=0
the default by./configure --enable-silent-rules
- #21591: Replace use of
SAGE_ROOT
by more specific environment variables - #21469: Enable VPATH builds (
SAGE_SRC_ROOT/configure --srcdir=SAGE_SRC_ROOT
) - #21538:
./configure --with-packages=...
- #22646:
./configure CC=/path/to/gcc ...
- Various other
configure
options, to replace use of environment variables that influence the build.
Clean up parts of the build system to make it more standard. This is to make it straightforward for developers familiar with the autotools system to contribute to sage.
- #21532: Create
SAGE_LOCAL
directory hierarchy duringmake
, notconfigure
- #21524:
configure.ac
: writebuild/make/Makefile
within anAC_CONFIG_COMMANDS
, not during mainconfigure
Make the separation between sage-the-distribution and sagelib (sage-the-Python-library) clearer. This will be beneficial for distributions such as Debian etc.
- #21559: Move sage-the-distribution scripts from
src/bin
tobuild/bin
- #21565: Add src/README.txt and build/README.txt
Following are workarounds to enable root-owned installation hierarchies (prefix
).
Change History (24)
comment:1 Changed 3 years ago by
- Cc was added
comment:2 Changed 3 years ago by
- Cc mmezzarobba added
comment:3 follow-up: ↓ 5 Changed 3 years ago by
comment:4 Changed 3 years ago by
I wonder if this work might not also benefit from adopting portions of astropy-helpers into Sage's build tools. I don't think it would necessarily be appropriate to use directly--although there's nothing very "astropy" specific in astropy-helpers, a lot of it does assume one is using the astropy project template, or something close to it. But large swaths of it are very generic, and I put a lot of work into developing tools for build sanity of large Python projects.
Some key features that I think might benefit sage include:
- per-subpackage
setup_package.py
modules, which can contain any number of a (mostly) well-documented hooks for building compiled modules in that sub-package. This provides a clear and explicit alternative to a monolithic module_list.py (see the discussion here), and would also make it easier to break off some of Sage's subpackages into standalone packages should the time ever come for that. - one of the less-well-documented features is also that
setup_package.py
can contain hooks for setup.py commands. For example, if a module contains some generated code (a lasage_setup.autogen
), it can include apre_build_ext
hook to make sure the generated code is up to date before running the./setup.py build_ext
command (either explicitly, or implicitly as a sub-command). This makes it relatively easy for individual submodules to put hooks into the build process without having to write complicated command subclasses cluttered with subpackage-specific crud. - smarter handling of Cython modules--for example, when creating a source tarball it will make sure all Cython modules have been Cythonized, and it will include the resulting C/C++ sources in the source tarball. This ensures that the user does not need Cython to be installed in order to build from source (which is good, because even if they do have Cython it will often be the wrong version--and in Sage's case missing important patches as well).
Those are the main ones for now but I could probably think of more.
comment:5 in reply to: ↑ 3 Changed 3 years ago by
Replying to mmezzarobba:
Do you intend
./configure
to detect already installed dependencies so thatmake
skips installing them? (Eventually? as part of this task?)
That is beyond the scope of this task ticket. But I would be interested in a follow-up ticket to this effect.
comment:6 Changed 3 years ago by
- Description modified (diff)
comment:7 Changed 3 years ago by
- Description modified (diff)
comment:8 Changed 3 years ago by
- Description modified (diff)
comment:9 follow-ups: ↓ 10 ↓ 12 Changed 3 years ago by
Will this also fix how make
actually runs configure
for you, and that configure
gets invoked even if you run make clean
targets?
comment:10 in reply to: ↑ 9 ; follow-up: ↓ 11 Changed 3 years ago by
- Description modified (diff)
Replying to embray:
Will this also fix how
make
actually runsconfigure
for you,
No, probably not. For this ticket, I want to keep configure
an optional step of the installation process. (I've updated the description.)
and that
configure
gets invoked even if you runmake clean
targets?
Not sure about this one. To keep this ticket manageable, I want to change the top-level Makefile
as little as possible. But please do open a ticket that describes make clean
issues that ought to be addressed.
comment:11 in reply to: ↑ 10 Changed 3 years ago by
Replying to mkoeppe:
Replying to embray:
Will this also fix how
make
actually runsconfigure
for you,No, probably not. For this ticket, I want to keep
configure
an optional step of the installation process. (I've updated the description.)
I guess it would be fine to make it an optional step. But I don't think it should be a mandatory step for all make targets, and currently it is, or at least seems to be. For example if I run make maintainer-clean
twice in a row, the second time will run configure
only to delete its output.
This is especially annoying on Cygwin where configure
is extremely slow.
comment:12 in reply to: ↑ 9 Changed 3 years ago by
Replying to embray:
Will this also fix how
make
actually runsconfigure
for you, and thatconfigure
gets invoked even if you runmake clean
targets?
In standard autotools packages, the Makefile
gets created by configure
, so you wouldn't be able to run make clean
either without configure
.
comment:13 Changed 3 years ago by
True--in that case it should (hopefully) be a moot point. This is currently a huge annoyance for me.
comment:14 Changed 3 years ago by
- Description modified (diff)
comment:15 Changed 3 years ago by
- Description modified (diff)
comment:16 Changed 3 years ago by
- Description modified (diff)
comment:17 Changed 3 years ago by
- Cc thansen added
comment:18 follow-up: ↓ 19 Changed 3 years ago by
Do we have a ticket somewhere for adding more configure-time checks for system packages? If there is I can't find it.
It's a little outside the scope of this ticket but not entirely. There are quite a few system packages installed by sage that could be skipped if we added the appropriate scripts to check for them.
comment:19 in reply to: ↑ 18 Changed 3 years ago by
Replying to embray:
Do we have a ticket somewhere for adding more configure-time checks for system packages? If there is I can't find it.
No, we don't. Feel free to open a ticket. I would prefer one ticket for every package, not a single ticket trying to do too much.
comment:20 Changed 3 years ago by
I agree--I probably won't even make a ticket for every package right away but I will make a master ticket for coordinating the task, and then open individual tickets for packages that I think are most worth tackling.
comment:21 Changed 3 years ago by
- Description modified (diff)
comment:22 follow-up: ↓ 23 Changed 2 years ago by
Would someone who has followed recent build system changes be interested in updating the description of this meta-ticket?
comment:23 in reply to: ↑ 22 Changed 2 years ago by
Replying to mkoeppe:
Would someone who has followed recent build system changes be interested in updating the description of this meta-ticket?
Which build system changes specifically? I have a few open tickets that are making some not insignificant changes, but they don't terribly change things for this ticket. If anything they will help make it easier to implement the goals of this ticket. There is one small conflict I can see: The work I'm doing in #22509 and in #23160 will mostly supersede the work you already did in #21537. This is because the sdh_make_install
helper function I added will also handle $SAGE_SUDO
.
Right now I'm working toward the goals I outlined here for improving support for building Sage against dependencies from the system. However, I haven't even made tickets for every aspect of that plan yet. All the work I'm currently doing is toward #22509 and #22510 which I see as necessary for saner package management in Sage (they will also be very helpful for work I need to do of making it easier to install optional packages with the Windows installer, but that's otherwise an orthogonal issue).
comment:24 Changed 17 months ago by
- Cc gh-timokau added
Do you intend
./configure
to detect already installed dependencies so thatmake
skips installing them? (Eventually? as part of this task?) For example, is the idea that./configure
would finish by saying something like “the following dependencies are missing: ... please install them, or typemake
and I'll install my own copy”? (This wouldn't be quite standard, but close enough, while staying mostly compatible with the installation procedure of sage-the-distribution.)Or would sage-the-distribution still always have its private version of everything, so that users would have to choose between installing it and installing sagelib directly (e.g., via pip)?