Opened 3 years ago

Last modified 21 months ago

#21495 new task

Long-term task ticket: Support "make install" of the Sage distribution

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-wishlist
Component: build Keywords:
Cc: dimpase, felixs, jdemeyer, mmezzarobba Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description

#21479 makes SAGE_LOCAL configurable by "./configure --prefix=SAGE_LOCAL". The default is the "local" subdirectory of the sage source tree (or the separate sage build tree, if using a VPATH configuration - #21469).

On this task ticket, let's organize the steps necessary to support "make install" that is separate from "make".

  • Step 1: Autotoolize ALL packages, so that they support staged installs using DESTDIR
  • Step 2: ...
  • Step 3: Have packages install into staging hierarchy and have make install copy the staged stuff to the prefix hierarchy. Easy! ;)

The comments section of this ticket is also a place for philosophical discussions regarding the nature of build and install directories, etc.

Change History (22)

comment:1 Changed 3 years ago by felixs

step 1 Autotoolize ALL packages

wow. that's a tremendous effort. i don't think it's really necessary.

but even if: why would the next step be "staged installs"? an autotoolized package will work fine after just "make". in contrast, it will not necessarily be functional from within $DESTDIR, that's not the intent of DESTDIR.

the package transition from "install to SAGE_LOCAL (during build)" + "unable to install to $prefix" to "dont install during build" + "possibly install to $prefix" can be done one after another. by someone who needs the package installed by sage. for this to make sense, it must be possible to disable packages... that's implemented (drafted) in #14796.

think of "patch". as long as nobody wants to install sage on a platform without patch, it will not be necessary to have an installable patch package provided by sage. sage will still work on this platform the traditional way, with patch "installed" to $SAGE_LOCAL, as usual ($PATH will take care of it).

my sage-autotools draft also used "staging" for "prepare something that can be used right now before installation". the fallback/transition was "install to SAGE_LOCAL". back then, i only needed ~3 packages installable by sage. the rest was already there (hence "disabled" at top level).

i think step 0 is missing. imo it is "implement configure for sagelib". without configure it will be harder to pass things like --with-$program=$somewhere. also sagelib might need (environment?) overrides for several paths.

how does "disable packages at top level" sound to you?

comment:2 in reply to: ↑ description Changed 3 years ago by jdemeyer

Replying to mkoeppe:

On this task ticket, let's organize the steps necessary to support "make install" that is separate from "make".

What is the point of that?

comment:3 Changed 3 years ago by mkoeppe

Mostly I want 1 central place where this is discussed to keep productive tickets clear of this discussion.

comment:4 Changed 3 years ago by jdemeyer

OK, for the record: I completely disagree with this ticket.

comment:5 follow-up: Changed 3 years ago by jdemeyer

Replying to dimpase:

$SAGE_LOCAL is fine as long as you can also install things from there somewhere else.

You want to install stuff from $SAGE_LOCAL to another place? That does not really make sense. I cannot easily install stuff from /usr (say) to another directory and that's not an issue. If you think of $SAGE_LOCAL as the installation directory, analogously to /usr, then I really do not see the point.

Of course, I agree that we should support #21479. Maybe if the installation directory becomes configurable, people will start understanding that it really is an installation directory.

comment:6 follow-up: Changed 3 years ago by felixs

You want to install stuff from $SAGE_LOCAL to another place? That does not really make sense.

exactly. SAGE_LOCAL does not make sense. it is a (historic) workaround, and an implementation detail. not a requirement.

Of course, I agree that we should support #21479.

#21479 is ./configure --prefix=SAGE_LOCAL. what should (in your opinion) make (not make install) do in

    $ ./configure --prefix=SAGE_LOCAL
    $ make

?

you know i think the answer is "what every other build system would do": build stuff. no other build system requires a writable $prefix for what it does.

comment:7 in reply to: ↑ 6 Changed 3 years ago by jdemeyer

Replying to felixs:

you know i think the answer is "what every other build system would do": build stuff. no other build system requires a writable $prefix for what it does.

But Sage-the-distribution is not a program, it is a distribution. It doesn't really make sense to "build" Sage-the-distribution. A separate build/install step really only makes sense for individual packages.

So the top-level make command installs Sage-the-distribution in its installation directory, which is $SAGE_LOCAL.

comment:8 in reply to: ↑ 5 ; follow-up: Changed 3 years ago by dimpase

Replying to jdemeyer:

Replying to dimpase:

$SAGE_LOCAL is fine as long as you can also install things from there somewhere else.

You want to install stuff from $SAGE_LOCAL to another place? That does not really make sense. I cannot easily install stuff from /usr (say) to another directory and that's not an issue.

Surely, you normally never install stuff from /usr anywhere, because /usr is an installation place, not a place to build stuff.

E.g. /usr can be NFS-mounted and used by lots of users, and thus unsuitable for build work.

And BEFORE installing stuff, you want to make check.

comment:9 follow-ups: Changed 3 years ago by felixs

But Sage-the-distribution is not a program

it's a software package. and it's not the first software package with subpackages. it's just working differently (for historical reasons).

question is how to escape this dilemma. answer is: apply standard practices/terminology.

in the meantime, the transition should better not affect any of the users and developers.

i don't really grasp why you are against progress.

comment:10 in reply to: ↑ 8 Changed 3 years ago by jdemeyer

Replying to dimpase:

Surely, you normally never install stuff from /usr anywhere, because /usr is an installation place, not a place to build stuff.

Right, just like $SAGE_LOCAL.

comment:11 in reply to: ↑ 9 ; follow-up: Changed 3 years ago by jdemeyer

Replying to felixs:

i don't really grasp why you are against progress.

I am not against progress. I am against pointless changes, especially if those changes make stuff more complicated without any advantage.

comment:12 in reply to: ↑ 9 ; follow-up: Changed 3 years ago by jdemeyer

Replying to felixs:

But Sage-the-distribution is not a program

it's a software package. and it's not the first software package with subpackages.

I do not see it as one package with subpackages. It is a collection of packages, installed in a common prefix.

comment:13 in reply to: ↑ 11 Changed 3 years ago by felixs

Replying to jdemeyer:

more complicated without any advantage.

i did not say that enabling users to install sage is meant to be an advantage for you in particular. my impression is, you are a developer not only a user.

look at the packages that you use, without being a developer of that particular package. sure i don't know what they are, statistically it's stuff like an OS kernel, a web browser, a desktop environment, a mail client, some office-suite... endless list.

would you say that it's not an advantage to have all these as packages ready in a zillon of distros that you could choose from?

(this is again drifting towards being off-topic. maybe it should be relacated to sage-flame)

comment:14 in reply to: ↑ 12 Changed 3 years ago by felixs

Replying to jdemeyer:

I do not see it as one package with subpackages. It is a collection of packages, installed in a common prefix.

what you describe is stuff like "pip" or some "ebuild" package manager, or maybe "apt" (together with the corresponding package directories)

sage-the-distribution has its place, but it's nowhere close. in architecure and use it closely resembles a standard package. it's just lacking some functionality

comment:15 Changed 3 years ago by mmezzarobba

  • Cc mmezzarobba added

comment:16 Changed 2 years ago by embray

I think I wrote elsewhere before I saw that this ticket already exists, but I'll repeat the basic idea here:

I think that in order to support this functionality, what we ought to do is break up all existing spkg-installs into separate spkg-build and spkg-install scripts--possibly also an spkg-env script specifically for setting environment variables needed for that package at both "build" and "install" time.

This would also ease further elimination of duplication between spkg-installs. For example, most packages' spkg-install becomes little more than a symlink to a stub script that calls make install.

Some packages can't easily be broken into cleanly separate "build" and "install" steps. I suppose this is the main reason for point 1 in the ticket description. I think "autotoolize" all packages is a big undertaking though, and not even appropriate for truly "all" packages (even excluding Python ones). For those packages that can't be separated, having a single "install" script is probably about as good as we'll get.

Where Python packages are concerned, ironically before switching to pip there was a clearer separation--you could ./setup.py build and ./setup.py install. For most pure-Python packages there's no reason to make this distinction, but for packages with extension modules, of which we use many, one could see some small argument for a separate build phase. A simple workaround for this with pip is to do pip install into an alternative --target (wherever the VPATH builds are going, for example). Then the install phase is just a matter of copying the built install into the appropriate site-packages for the Python being used to run Sage (which may even be the system Python--the ultimate goal).

comment:17 follow-up: Changed 2 years ago by embray

(and to be clear, I agree with felix that "autotoolize all packages", even if you exclude the obvious ones where it doesn't apply, is probably more work than necessary--would be nice to do where it is easy, but I think simply separating build and install steps in the spkgs is an easier way to go)

comment:18 in reply to: ↑ 17 Changed 2 years ago by jdemeyer

Replying to embray:

separating build and install steps in the spkgs is an easier way to go

What do we gain from that? I am totally missing that from the ticket description. Usually, there is a problem that you want to solve and then you look for a solution. So, what exactly is the problem that we are trying to solve here?

comment:19 Changed 2 years ago by embray

It would actually be great if I could test building Sage along with sagelib without having to actually muck with or break things in my existing Sage "installation" (i.e. currently wherever $SAGE_LOCAL is) that I'm using to run and test development just on sagelib. It would also be great for testing if make <packagename> just tried to build it but didn't install it anywhere.

Currently the only alternatives are to constantly muck with environment variables (and inevitably make mistakes) or have about a dozen simultaneous sage source dirs as it seems I currently do...

comment:20 follow-up: Changed 2 years ago by mkoeppe

I strongly recommend not to spend time on this ticket. It is here to centralize discussion that will invariably come up.

There are plenty of build tickets which need actual work or review.

comment:21 in reply to: ↑ 20 Changed 2 years ago by embray

Replying to mkoeppe:

I strongly recommend not to spend time on this ticket. It is here to centralize discussion that will invariably come up.

Right now you mean ;)

There are plenty of build tickets which need actual work or review.

Agreed--there's lots else to be done before this should be worked on at all though I'm all for having the discussion.

comment:22 Changed 21 months ago by embray

One obvious problem with this that I'm surprised hasn't been pointed out yet (unless it has, and I'm missing it--or at least, it hasn't been pointed out in this ticket) is that so many packages in Sage require their dependencies to be installed in order to build. So you can't do a make that builds all the packages in the Sage distribution, and then a separate make install to install them since most packages aren't going to build in the first place until and unless their dependencies have been installed.

This could be worked around by looking for the dependencies in their temporary build roots, but that could also get tricky fast. The only way this can be made to work is if every package were autotoolized, but that's not going to happen and is beyond the scope of what Sage's packaging system does.

I think the work I'm doing now for #22509 will ultimately satisfy my needs for easily building packages without installing them. Right now the install step is still run unconditionally but it will be easy to add a "build only" flag in the appropriate places once this work is done.

Note: See TracTickets for help on using tickets.