Opened 4 years ago
Last modified 3 weeks ago
#21707 needs_work task
Metaticket: Split sageenv into 5 to clean up sage configuration
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage9.2 
Component:  build  Keywords:  
Cc:  embray, jdemeyer, leif, fbissey, dimpase, isuruf, arojas, infinity0, ghtimokau, mjo, jhpalmieri  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  #29038, #29052  Stopgaps: 
Description (last modified by )
src/bin/sageenv
(with generated configuration in src/bin/sageenvconfig
; installed in $SAGE_LOCAL/bin
) is used in all of the following contexts:
 Sagethedistribution while building spkgs,
 Sagethedistribution for building sagelib,
 for environment variables needed to start Sage,
 as variables by Sage at runtime (through
sage.env
), and  for environment variables affecting subprocesses of Python invoked by sagelib modules.
This ticket proposes to split the configuration according to these 5 contexts. This will make the installation of Sage more modular and flexible.
1./2. Create build/bin/sagebuildenv
(with generated configuration in build/bin/sagebuildenvconfig
) for the buildtime environment variables for sagethedistribution (spkg and sagelib). It is not installed in $SAGE_LOCAL.
 Initially,
build/bin/sageenv
will callsrc/bin/sageenv
.  In
build/make/deps
and the scripts generated bybuild/bin/sagespkg
, sourcebuild/bin/sagebuildenv
in addition tosrc/bin/sageenv
. This is #29052.  Gradually, we will move settings from
src/bin/sageenv[config]
tobuild/bin/sagebuildenv[config]
that are known to be only needed for 1./2. (Note that not all compilerrelated environment variables can be moved exclusively to 1./2. – some need to be added to 5 as well for some some calls to compilers needed at runtime to support things like%cython
(?) andsage.calculus.desolvers.desolve_mintides
.) As well as users' use ofpip
to install additional Python packages!  Eventually, we will remove the call to
src/bin/sageenv
.
2./3. Obtain sagelib's buildtime configuration such as src/setup.py
's library_dirs
and runtime information (sage.env
) from sageconfig
(introduced in #29038).
 In
src/Makefile
, poison theSAGE_LOCAL
variable.  Pass
SAGE_PKGS
(if still needed at all  see #28815) viasageconfig
instead of fromsrc/Makefile
.
3./4./5. Using sage_conf
(Python module and script sageconfig
, #29038), make sage.all
fully functional when imported from a Python, without setting any environment variables (sageenv
).
 Initially, phase out
src/bin/sageenvconfig
by usingsageconfig
instead. #29384: Clean up
src/bin/sageenvconfig.in
: Move logic tosrc/bin/sageenv
, move nonenvironment configuration variables tosage_conf.py
 #29384: Clean up
 Set
sage.env
variables via the Python modulesage_conf
instead of relying on information from environment variables. Example:CYSIGNALS_CRASH_DAYS
 Set environment variables that are needed only by subprocesses invoked by sagelib in the environment of these subprocesses, rather than relying on them begin set in
sageenv
. This environment could be provided by a variablesage_conf.SAGE_SUBPROCESS_ENV
and/or more invidual variables. Example:R_PROFILE
 Finally, remove use of
sageenv
for all purposes except forsage sh
.
Aspects of downstream sage packaging:
 The goal is to enable downstream sage packagers to use a whole unmodified
src
directory to build and install sagelib.  Downstream packagers would provide their own "implementation" (one Python module sage_conf.py) of the configuration module defined by #29038, instead of patching sources. They would *not* use the reference implementation in
build/pkgs/sage_conf
, which is intended for use with sagethedistribution only.
Relation to the goal of sagelib as a standalone Python library (#21507)
 An sdist of sagelib, intended to be installed from source from PyPI, could package or depend on another implementation of
sage_conf
that calls an autoconfgenerated configure script (which reuses parts of the build/pkgs/SPKG/spkgconfigure.m4 infrastructure); see #29119.
More related tickets:
Change History (41)
comment:1 Changed 4 years ago by
comment:3 Changed 4 years ago by
 Description modified (diff)
comment:4 Changed 4 years ago by
I agree.
Consider a slight alternative: Rewrite sageenv as a Python script. This would give it much more flexibility and probably more legibility, including some commandline options for different environments (build
, especially).
The output would of course be the shell commands to actually set the variables. So rather than source sageenv
one would eval $(sageenv)
. My one concern is that it would be slower, but ideally it's something that shouldn't happen frequently anyways, but rather only when not already in sageenv.
comment:5 Changed 4 years ago by
I have a tendency to mix up sageenv
the script and sage.env.py
, my comment is really an example of that. However a lot is common between the two.
comment:6 followup: ↓ 7 Changed 4 years ago by
Rightthey could be one in the same.
comment:7 in reply to: ↑ 6 Changed 4 years ago by
Replying to embray:
Rightthey could be one in the same.
I have thought about it often. Of course that really means that you'll have to removing all the build time stuff. But in sageongentoo where I ship a very simplified sageenv (in /etc as well) they are virtually doing the same thing, one in shell, the other in python.
comment:8 Changed 4 years ago by
In either casewhether making a single script, or two separate shell scripts, this would be useful for gentoo (and probably other distros as well) then right?
comment:9 Changed 4 years ago by
Probably, the current sageenv
is a forest mostly dedicated to the build system. I just discard it and bring my own, it takes less space and maintenance than a mega patch.
comment:10 Changed 4 years ago by
Ideally, the configuration for sagelib runtime should take place in a generated Python file and not involve environment variables at all.
comment:11 Changed 4 years ago by
More likely parts of it would be static, and other parts could be generated, but yes, what I meant above in terms of rewriting sageenv
in Python was that it would not (by default) involve environment variables at all.
But it's still useful for interacting with other tools, especially for build purposes, as well as dropping into ./sage sh
to be able to set environment variables, which is why I suggest a mode to output a list of variables that can be read from the shell.
comment:12 Changed 12 months ago by
 Branch set to u/mkoeppe/split_sage_env_into_sage_build_env_and_sage_env
comment:13 Changed 12 months ago by
 Cc dimpase added
 Commit set to 9189d609fa8cc90d616cb7ea05be405ce196295e
 Dependencies set to #25130
 Milestone changed from sage7.5 to sage8.8
 Status changed from new to needs_review
comment:14 Changed 12 months ago by
Asis this is going to conflict with tickets like #27825 that are adding still more environment variables needed at buildtime to sageenvconfig.
I'm completely in favor of this separation in principle though. The question is whether we should do this first and then update tickets like #27825 on top of it, or the other way around?
comment:15 Changed 12 months ago by
comment:16 Changed 12 months ago by
The tickets would be trivial to rebase on top of this one.
comment:17 followup: ↓ 26 Changed 12 months ago by
Probably one should introduce a command sage buildsh
, which would provide the larger environment.
comment:18 Changed 11 months ago by
For the sake of a bit more terseness I wonder if we could call it just sageenvbuild.in
and sageenvbuild
. The only reason for the config
in the other one is to distinguish it from plain sageenv
.
Now that I look I'm not even entirely sure we need to separate sageenvconfig
from sageenv
.
I think we could probably have a sageenv.in
produce sageenv
and get rid of the separate sageenvconfig
entirely, though it's possible there's some subtlety to that that I'm missing.
comment:19 Changed 11 months ago by
 Branch changed from u/mkoeppe/split_sage_env_into_sage_build_env_and_sage_env to public/split_sage_env_into_sage_build_env_and_sage_env
 Commit changed from 9189d609fa8cc90d616cb7ea05be405ce196295e to ff0712125a3d05500a1f77e62c371d695d1a9794
Went ahead and rebased on current develop so it could incorporate all the additional variables we've since added.
I didn't change the name of the script yet; I would still prefer to shorten it but if anyone disagrees we can leave it asis.
New commits:
a34a9f1  Move sagedisthelpers from src/bin to build/bin

ff07121  Split out build/bin/sagebuildenvconfig from sageenvconfig

comment:20 Changed 11 months ago by
The only subtility to sageenvconfig
in my mind, is that you can touch it or even regenerate it without touching sageenv
. But I am sure you don't need a separate file to achieve your objectives in either case.
comment:21 Changed 11 months ago by
No, probably not. I might try it, though we'll leave that as a separate exercise from this ticket.
comment:22 Changed 11 months ago by
 Milestone changed from sage8.8 to sage8.9
Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).
comment:23 Changed 5 months ago by
 Milestone changed from sage8.9 to sage9.1
Ticket retargeted after milestone closed
comment:24 Changed 4 months ago by
 Status changed from needs_review to needs_work
comment:25 followup: ↓ 27 Changed 4 months ago by
The runtime environment variables of sage could as well be configured by a Python module src/sage/env_config.py
 see #29022.
comment:26 in reply to: ↑ 17 Changed 4 months ago by
Replying to mkoeppe:
Probably one should introduce a command
sage buildsh
, which would provide the larger environment.
As an extension of this, there could be sage buildsh SPKG
, which would set up the environment in which spkginstall
runs. This could be useful for testing new versions of packages.
comment:27 in reply to: ↑ 25 Changed 4 months ago by
comment:28 Changed 4 months ago by
 Branch public/split_sage_env_into_sage_build_env_and_sage_env deleted
 Cc isuruf arojas infinity0 ghtimokau added
 Commit ff0712125a3d05500a1f77e62c371d695d1a9794 deleted
 Dependencies changed from #25130 to #29038
 Description modified (diff)
 Status changed from needs_work to needs_info
 Summary changed from Split sageenv into sagebuildenv and sageenv to Split sageenv into 5
 Type changed from enhancement to task
Hoping for input from people involved in packaging.
comment:29 Changed 4 months ago by
 Status changed from needs_info to needs_review
comment:30 Changed 4 months ago by
 Description modified (diff)
comment:31 Changed 4 months ago by
 Description modified (diff)
comment:32 Changed 4 months ago by
 Dependencies changed from #29038 to #29038, #29052
comment:33 Changed 4 months ago by
 Summary changed from Split sageenv into 5 to Metaticket: Split sageenv into 5 to clean up sage configuration
comment:34 Changed 4 months ago by
 Description modified (diff)
comment:35 Changed 4 months ago by
 Description modified (diff)
comment:36 Changed 4 months ago by
 Cc mjo added
comment:37 Changed 2 months ago by
 Description modified (diff)
 Status changed from needs_review to needs_work
comment:38 Changed 6 weeks ago by
 Description modified (diff)
comment:39 Changed 5 weeks ago by
 Milestone changed from sage9.1 to sage9.2
comment:40 Changed 3 weeks ago by
 Description modified (diff)
comment:41 Changed 3 weeks ago by
 Cc jhpalmieri added
One comment. Currently
sage_setup
is installed because it needs to be installed to be doctested  like the rest of sage. But nothing insage_setup
is really needed at runtime and distro can choose not to install it. I don't in sageongentoo. I tend to push for thing not needed at runtime to be moved in there. And that's where I would have putsagebuildenv
but I am not going to fight it going intobuild/bin
although I'd like everything used bysetup.py
to be neatly undersrc
.