Opened 15 months ago

Closed 13 days ago

#25130 closed defect (fixed)

Move sage-dist-helpers from src/bin to build/bin

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-8.9
Component: build Keywords:
Cc: embray, fbissey, dimpase Merged in:
Authors: Matthias Koeppe Reviewers: Erik Bray
Report Upstream: N/A Work issues:
Branch: abc9fa9 (Commits) Commit: abc9fa93e25a86eef191ece157158e838d2fb433
Dependencies: Stopgaps:

Description (last modified by mkoeppe)

This seems to be a script for use within build/pkgs scripts, so it should go to build/bin.

Change History (21)

comment:1 Changed 15 months ago by embray

I go back-and-forth on that. I agree with your logic completely. But I think we need to talk about where installation-related files go for use "at runtime". Because currently all build-related files live in build/ but don't get "installed" into $SAGE_LOCAL. And yet they are in a sense needed at "runtime" since a feature of Sage is the ability to install new packages--particularly optional packages.

So I think what we really need is overall better organization of what build-related files are needed at runtime and how to "install" those files in such a way that installing additional Sage packages doesn't necessarily depend on $SAGE_ROOT.

Under that rubrik I'm not sure the existence of build/bin even makes sense since most, if not all of those scripts are needed for optional package installation.

comment:2 Changed 15 months ago by mkoeppe

build is for sage-the-distribution, and therefore for everything related to installing packages. src is for sagelib.

comment:3 follow-up: Changed 15 months ago by embray

I agree, and yet that ignores everything else I wrote...

comment:4 in reply to: ↑ 3 Changed 15 months ago by mkoeppe

Replying to embray:

I agree, and yet that ignores everything else I wrote...

Sorry, I had to catch a flight.

The longer answer: I disagree with the notion of installation of optional packages being done "at runtime". There is no ticket that describes your goal of being able to install tickets even if $SAGE_ROOT is not present. Right now installation of optional packages does not even reliably work in binary distributions.

comment:5 Changed 15 months ago by embray

I think it should work--at least so long as a compiler is available. As it stands, in binary distributions the feature is not disabled, and it is built into the command-line sage interface.

Alternatively, I have also mentioned that I don't like it in the first place that build-related commands (e.g. sage -b and the like) are built into the same runtime interface that users use to start and run Sage and associated commands.

So I think there needs to be a clearer vision of what an arbitrary user should expect to be able to do from the sage command regardless of the context in which they're running it.

comment:6 Changed 15 months ago by embray

Put more succinctly, if someone wants to move this file around because it's "build-related" I have no problem with that. My point is just that I see it as a distinction without a difference since Sage doesn't clearly separate build-time from run-time in all cases.

comment:7 Changed 15 months ago by jdemeyer

I agree with @mkoeppe here. Installing Sage packages should not considered to be run-time, but build-time.

comment:8 Changed 15 months ago by embray

And yet sage -i <optional-package> is considered standard functionality of Sage that is expected to work for all users...?

comment:9 Changed 15 months ago by mkoeppe

It wouldn't be expected, for example, in a Linux distribution's sage binary; sage's optional packages would probably be supplied by the distribution's package manager instead.

I see the "sage -i" as a mere convenience which "calls out to the SAGE_ROOT" to install a package.

comment:10 Changed 15 months ago by embray

  • Cc fbissey added

I'd be curious about the sage-on-gentoo approach to this. If that is indeed the case (and I'm not necessarily disagreeing) then we should still clearly delineate what features of the command-line sage interface should and should not work outside of a Sage source tree, and clearly disable if not remove those features in cases where Sage is "installed" in another filesystem hierarchy. This should also be made clear in the documentation both for users, and for developers interesting in packaging Sage for different platforms: That they need to think about what optional packages to include, if any, and if and whether users should have means to obtain optional packages. There's also a question of whether or not "pip" should work...

I'm thinking about this because currently installing optional packages is broken in Sage for Windows. Users really have no way to install them. If sage -i shouldn't work, then there has to be some mechanism whereby users can install packages. For Windows I have a few different ideas which I'd be fine with, but I would still want to be able to either explicitly disable sage -i, or have it print some message as to how users should actually install optional packages (or maybe it could install the package, but would have to go through an alternate system-specific backend to do so). For example in the case of Windows my preferred approach is to provide binary packages that are downloaded and unpacked.

comment:11 Changed 15 months ago by fbissey

On sage-on-gentoo sage -i is not offered. I am pushing slowly but surely for is_package_installed to be removed at runtime and in the more distant future at build time. doctesting and is_package_installed are also causing me some headaches.

For information this is sage-on-gentoo misc/package.py file [ https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/sage/files/sage-7.3-package.py].

And these are the commands that I ship if you request debugging and doctesting enabled

/usr/bin/math-readline
/usr/bin/sage
/usr/bin/sage-cachegrind
/usr/bin/sage-callgrind
/usr/bin/sage-cleaner
/usr/bin/sage-cython
/usr/bin/sage-eval
/usr/bin/sage-gdb-commands
/usr/bin/sage-ipython
/usr/bin/sage-massif
/usr/bin/sage-native-execute
/usr/bin/sage-notebook
/usr/bin/sage-num-threads.py
/usr/bin/sage-omega
/usr/bin/sage-preparse
/usr/bin/sage-python
/usr/bin/sage-rst2sws
/usr/bin/sage-rst2txt
/usr/bin/sage-run
/usr/bin/sage-run-cython
/usr/bin/sage-runtests
/usr/bin/sage-startuptime.py
/usr/bin/sage-sws2rst
/usr/bin/sage-valgrind
/usr/bin/sage-version.sh

As for dealing with optional packages. In sage-on-distro we of course expect them to be installed by the distro's package manager. Not all optional packages may be on offer and filling that gap is hard. Debian has a massive amount of packages so they have a better starting point than me on this.

Next there is the problem that there are two categories of optional packages:

  • runtime packages, the functionality will be available if the package is installed and it can be installed after sage. They are easy to deal with - you can just install them whenever, in a standard path, and they should work.
  • build time packages. In the current system if those packages are found when sage is built, specific binding will be build for those packages. Those are the hard ones. If you want to support them they have to be part of your dependency tree.

Gentoo has useflags, I offer some built time optional packages through useflags. If you want any of bliss, modular_decomposition, libbraiding or libhomfly they can be enabled. cbc is hard and I failed to enable it once.

Missing packages in that category are mcqd, tdlib, coxeter3, fes, sirocco, meataxe (not going to happen anytime soon, with all due respect, meataxe packaging sucks), gurobi (in its own category, it checks for presence differently from anyone else), cplex and cbc (the only extension to require lapack).

Last edited 15 months ago by fbissey (previous) (diff)

comment:12 Changed 15 months ago by embray

Thanks for details. It's definitely clear to me we need to do some more rethinking of how optional packages are handled. Anyways, sorry to derail this ticket. Like I said in the first place I don't mind moving the current location of sage-dist-helpers. It just got me thinking about why I placed it where I did in the first place. There was certainly motivated reasoning behind it, though in practice it still didn't make sense...

comment:13 Changed 2 months ago by mkoeppe

  • Branch set to u/mkoeppe/move_sage_dist_helpers_from_src_bin_to_build_bin

comment:14 Changed 2 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Commit set to 7405dd5b05259028e4dadc981b5bd309a124fbfb
  • Description modified (diff)
  • Milestone changed from sage-8.2 to sage-8.8
  • Status changed from new to needs_review

New commits:

7405dd5Move sage-dist-helpers from src/bin to build/bin

comment:15 Changed 2 months ago by mkoeppe

  • Cc dimpase added

comment:16 Changed 8 weeks ago by embray

I'm concerned that this will be broken if $SAGE_ROOT contains spaces. It probably shouldn't and I'm not sure that there aren't other things that break if that's the case.

I think it would be fine to just put each path in double-quotes.

comment:17 Changed 8 weeks ago by git

  • Commit changed from 7405dd5b05259028e4dadc981b5bd309a124fbfb to abc9fa93e25a86eef191ece157158e838d2fb433

Branch pushed to git repo; I updated commit sha1. New commits:

abc9fa9Add quotes

comment:18 Changed 7 weeks ago by mkoeppe

Needs review.

comment:19 Changed 2 weeks ago by embray

  • Milestone changed from sage-8.8 to sage-8.9
  • Reviewers set to Erik Bray
  • Status changed from needs_review to positive_review

Thank you. Sorry this took so long.

comment:20 Changed 2 weeks ago by mkoeppe

Thank you!

comment:21 Changed 13 days ago by vbraun

  • Branch changed from u/mkoeppe/move_sage_dist_helpers_from_src_bin_to_build_bin to abc9fa93e25a86eef191ece157158e838d2fb433
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.