Opened 15 months ago
Closed 13 days ago
#25130 closed defect (fixed)
Move sagedisthelpers from src/bin to build/bin
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage8.9 
Component:  build  Keywords:  
Cc:  embray, fbissey, dimpase  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  Erik Bray 
Report Upstream:  N/A  Work issues:  
Branch:  abc9fa9 (Commits)  Commit:  abc9fa93e25a86eef191ece157158e838d2fb433 
Dependencies:  Stopgaps: 
Description (last modified by )
This seems to be a script for use within build/pkgs
scripts, so it should go to build/bin
.
Change History (21)
comment:1 Changed 15 months ago by
comment:2 Changed 15 months ago by
build is for sagethedistribution, and therefore for everything related to installing packages. src is for sagelib.
comment:3 followup: ↓ 4 Changed 15 months ago by
I agree, and yet that ignores everything else I wrote...
comment:4 in reply to: ↑ 3 Changed 15 months ago by
Replying to embray:
I agree, and yet that ignores everything else I wrote...
Sorry, I had to catch a flight.
The longer answer: I disagree with the notion of installation of optional packages being done "at runtime". There is no ticket that describes your goal of being able to install tickets even if $SAGE_ROOT is not present. Right now installation of optional packages does not even reliably work in binary distributions.
comment:5 Changed 15 months ago by
I think it should workat least so long as a compiler is available. As it stands, in binary distributions the feature is not disabled, and it is built into the commandline sage
interface.
Alternatively, I have also mentioned that I don't like it in the first place that buildrelated commands (e.g. sage b
and the like) are built into the same runtime interface that users use to start and run Sage and associated commands.
So I think there needs to be a clearer vision of what an arbitrary user should expect to be able to do from the sage
command regardless of the context in which they're running it.
comment:6 Changed 15 months ago by
Put more succinctly, if someone wants to move this file around because it's "buildrelated" I have no problem with that. My point is just that I see it as a distinction without a difference since Sage doesn't clearly separate buildtime from runtime in all cases.
comment:7 Changed 15 months ago by
I agree with @mkoeppe here. Installing Sage packages should not considered to be runtime, but buildtime.
comment:8 Changed 15 months ago by
And yet sage i <optionalpackage>
is considered standard functionality of Sage that is expected to work for all users...?
comment:9 Changed 15 months ago by
It wouldn't be expected, for example, in a Linux distribution's sage binary; sage's optional packages would probably be supplied by the distribution's package manager instead.
I see the "sage i" as a mere convenience which "calls out to the SAGE_ROOT" to install a package.
comment:10 Changed 15 months ago by
 Cc fbissey added
I'd be curious about the sageongentoo approach to this. If that is indeed the case (and I'm not necessarily disagreeing) then we should still clearly delineate what features of the commandline sage
interface should and should not work outside of a Sage source tree, and clearly disable if not remove those features in cases where Sage is "installed" in another filesystem hierarchy. This should also be made clear in the documentation both for users, and for developers interesting in packaging Sage for different platforms: That they need to think about what optional packages to include, if any, and if and whether users should have means to obtain optional packages. There's also a question of whether or not "pip" should work...
I'm thinking about this because currently installing optional packages is broken in Sage for Windows. Users really have no way to install them. If sage i
shouldn't work, then there has to be some mechanism whereby users can install packages. For Windows I have a few different ideas which I'd be fine with, but I would still want to be able to either explicitly disable sage i
, or have it print some message as to how users should actually install optional packages (or maybe it could install the package, but would have to go through an alternate systemspecific backend to do so). For example in the case of Windows my preferred approach is to provide binary packages that are downloaded and unpacked.
comment:11 Changed 15 months ago by
On sageongentoo sage i
is not offered. I am pushing slowly but surely for is_package_installed
to be removed at runtime and in the more distant future at build time. doctesting and is_package_installed
are also causing me some headaches.
For information this is sageongentoo misc/package.py
file [ https://github.com/cschwan/sageongentoo/blob/master/scimathematics/sage/files/sage7.3package.py].
And these are the commands that I ship if you request debugging and doctesting enabled
/usr/bin/mathreadline /usr/bin/sage /usr/bin/sagecachegrind /usr/bin/sagecallgrind /usr/bin/sagecleaner /usr/bin/sagecython /usr/bin/sageeval /usr/bin/sagegdbcommands /usr/bin/sageipython /usr/bin/sagemassif /usr/bin/sagenativeexecute /usr/bin/sagenotebook /usr/bin/sagenumthreads.py /usr/bin/sageomega /usr/bin/sagepreparse /usr/bin/sagepython /usr/bin/sagerst2sws /usr/bin/sagerst2txt /usr/bin/sagerun /usr/bin/sageruncython /usr/bin/sageruntests /usr/bin/sagestartuptime.py /usr/bin/sagesws2rst /usr/bin/sagevalgrind /usr/bin/sageversion.sh
As for dealing with optional packages. In sageondistro we of course expect them to be installed by the distro's package manager. Not all optional packages may be on offer and filling that gap is hard. Debian has a massive amount of packages so they have a better starting point than me on this.
Next there is the problem that there are two categories of optional packages:
 runtime packages, the functionality will be available if the package is installed and it can be installed after sage. They are easy to deal with  you can just install them whenever, in a standard path, and they should work.
 build time packages. In the current system if those packages are found when sage is built, specific binding will be build for those packages. Those are the hard ones. If you want to support them they have to be part of your dependency tree.
Gentoo has useflags, I offer some built time optional packages through useflags. If you want any of bliss
, modular_decomposition
, libbraiding
or libhomfly
they can be enabled. cbc
is hard and I failed to enable it once.
Missing packages in that category are mcqd
, tdlib
, coxeter3
, fes
, sirocco
, meataxe
(not going to happen anytime soon, with all due respect, meataxe packaging sucks), gurobi
(in its own category, it checks for presence differently from anyone else), cplex
and cbc
(the only extension to require lapack).
comment:12 Changed 15 months ago by
Thanks for details. It's definitely clear to me we need to do some more rethinking of how optional packages are handled. Anyways, sorry to derail this ticket. Like I said in the first place I don't mind moving the current location of sagedisthelpers
. It just got me thinking about why I placed it where I did in the first place. There was certainly motivated reasoning behind it, though in practice it still didn't make sense...
comment:13 Changed 2 months ago by
 Branch set to u/mkoeppe/move_sage_dist_helpers_from_src_bin_to_build_bin
comment:14 Changed 2 months ago by
 Commit set to 7405dd5b05259028e4dadc981b5bd309a124fbfb
 Description modified (diff)
 Milestone changed from sage8.2 to sage8.8
 Status changed from new to needs_review
New commits:
7405dd5  Move sagedisthelpers from src/bin to build/bin

comment:15 Changed 2 months ago by
 Cc dimpase added
comment:16 Changed 8 weeks ago by
I'm concerned that this will be broken if $SAGE_ROOT
contains spaces. It probably shouldn't and I'm not sure that there aren't other things that break if that's the case.
I think it would be fine to just put each path in doublequotes.
comment:17 Changed 8 weeks ago by
 Commit changed from 7405dd5b05259028e4dadc981b5bd309a124fbfb to abc9fa93e25a86eef191ece157158e838d2fb433
Branch pushed to git repo; I updated commit sha1. New commits:
abc9fa9  Add quotes

comment:18 Changed 7 weeks ago by
Needs review.
comment:19 Changed 2 weeks ago by
 Milestone changed from sage8.8 to sage8.9
 Reviewers set to Erik Bray
 Status changed from needs_review to positive_review
Thank you. Sorry this took so long.
comment:20 Changed 2 weeks ago by
Thank you!
comment:21 Changed 13 days ago by
 Branch changed from u/mkoeppe/move_sage_dist_helpers_from_src_bin_to_build_bin to abc9fa93e25a86eef191ece157158e838d2fb433
 Resolution set to fixed
 Status changed from positive_review to closed
I go backandforth on that. I agree with your logic completely. But I think we need to talk about where installationrelated files go for use "at runtime". Because currently all buildrelated files live in
build/
but don't get "installed" into$SAGE_LOCAL
. And yet they are in a sense needed at "runtime" since a feature of Sage is the ability to install new packagesparticularly optional packages.So I think what we really need is overall better organization of what buildrelated files are needed at runtime and how to "install" those files in such a way that installing additional Sage packages doesn't necessarily depend on
$SAGE_ROOT
.Under that rubrik I'm not sure the existence of
build/bin
even makes sense since most, if not all of those scripts are needed for optional package installation.