Opened 6 years ago
Last modified 13 days ago
#21707 needs_work task
Metaticket: Split sageenv into 5 to clean up sage configuration
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage9.7 
Component:  build  Keywords:  sd109 
Cc:  embray, jdemeyer, leif, fbissey, dimpase, isuruf, arojas, infinity0, ghtimokau, mjo, jhpalmieri, ghtobiasdiez  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  #29038, #29052  Stopgaps: 
Description (last modified by )
src/bin/sageenv
(with generated configuration in src/bin/sageenvconfig
; installed in $SAGE_LOCAL/bin
) is used in all of the following contexts:
 Sagethedistribution while building spkgs,
 Sagethedistribution for building sagelib,
 for environment variables needed to start Sage,
 as variables by Sage at runtime (through
sage.env
), and  for environment variables affecting subprocesses of Python invoked by sagelib modules.
This ticket proposes to split the configuration according to these 5 contexts. This will make the installation of Sage more modular and flexible.
1./2. Create build/bin/sagebuildenv
(with generated configuration in build/bin/sagebuildenvconfig
) for the buildtime environment variables for sagethedistribution (spkg and sagelib). It is not installed in $SAGE_LOCAL.
 In
build/make/Makefile.in
and the scripts generated bybuild/bin/sagespkg
, sourcebuild/bin/sagebuildenv
in addition tosrc/bin/sageenv
. This is #29052.  Gradually, we will move settings from
src/bin/sageenv[config]
tobuild/bin/sagebuildenv[config]
that are known to be only needed for 1./2. (Note that not all compilerrelated environment variables can be moved exclusively to 1./2. – some need to be added to 5 as well for some some calls to compilers needed at runtime to support things like%cython
(?) andsage.calculus.desolvers.desolve_mintides
.) As well as users' use ofpip
to install additional Python packages!  Eventually, we will remove the call to
src/bin/sageenv
from the build scripts.
2./3. Obtain sagelib's buildtime configuration such as src/setup.py
's library_dirs
and runtime information (sage.env
) from sageconfig
(introduced in #29038).
 In
src/Makefile
, poison theSAGE_LOCAL
variable.  Pass
SAGE_PKGS
(if still needed at all  see #28815 or #29705) viasageconfig
instead of fromsrc/Makefile
.
3./4./5. Using sage_conf
(Python module and script sageconfig
, #29038), make sage.all
fully functional when imported from a Python, without setting any environment variables (sageenv
).
 Initially, phase out
src/bin/sageenvconfig
by usingsageconfig
instead.  Set
sage.env
variables via the Python modulesage_conf
instead of relying on information from environment variables. Example:CYSIGNALS_CRASH_DAYS
 #34236
sage.env
: Give values fromsage_conf
precedence over environment variables
 #34236
 Set environment variables that are needed only by subprocesses invoked by sagelib in the environment of these subprocesses, rather than relying on them begin set in
sageenv
. This environment could be provided by a variablesage_conf.SAGE_SUBPROCESS_ENV
and/or more invidual variables. Example:R_PROFILE
 Finally, remove use of
sageenv
for all purposes except forsage sh
.
Aspects of downstream sage packaging:
 The goal is to enable downstream sage packagers to use a whole unmodified
src
directory to build and install sagelib.  Downstream packagers could provide their own "implementation" (one Python module sage_conf.py) of the configuration module defined by #29038, instead of patching sources. They would not necessarily use the reference implementation in
build/pkgs/sage_conf
, which is intended for use with sagethedistribution.
Binary packages:
 #31417 version of package
sage_conf
for relocatable binary distributions
pipinstallable Sage
 #29039 pipinstallable version of package
sage_conf
 installs nonPython bits of the Sage distribution in~/.sage/
 #31396 relocatable wheel version of package
sage_conf
More related tickets:
 cleaning of
src/bin
as described in #21570, #21559.  #25486: Discover
SAGE_SCRIPTS_DIR
to make$SAGE_LOCAL/bin/sage
work from any directory, in an environment withoutSAGE_*
variables  this adds aSAGE_ROOT
variable tosageenvconfig
 #30724:
src/bin/sageenv
,src/bin/sageenvconfig.in
: Remove PYTHON_FOR_VENV  #29133 METAMETATICKET: Build, packaging, testing improvements
Change History (64)
comment:1 Changed 6 years ago by
comment:3 Changed 6 years ago by
 Description modified (diff)
comment:4 Changed 6 years ago by
I agree.
Consider a slight alternative: Rewrite sageenv as a Python script. This would give it much more flexibility and probably more legibility, including some commandline options for different environments (build
, especially).
The output would of course be the shell commands to actually set the variables. So rather than source sageenv
one would eval $(sageenv)
. My one concern is that it would be slower, but ideally it's something that shouldn't happen frequently anyways, but rather only when not already in sageenv.
comment:5 Changed 6 years ago by
I have a tendency to mix up sageenv
the script and sage.env.py
, my comment is really an example of that. However a lot is common between the two.
comment:6 followup: ↓ 7 Changed 6 years ago by
Rightthey could be one in the same.
comment:7 in reply to: ↑ 6 Changed 6 years ago by
Replying to embray:
Rightthey could be one in the same.
I have thought about it often. Of course that really means that you'll have to removing all the build time stuff. But in sageongentoo where I ship a very simplified sageenv (in /etc as well) they are virtually doing the same thing, one in shell, the other in python.
comment:8 Changed 6 years ago by
In either casewhether making a single script, or two separate shell scripts, this would be useful for gentoo (and probably other distros as well) then right?
comment:9 Changed 6 years ago by
Probably, the current sageenv
is a forest mostly dedicated to the build system. I just discard it and bring my own, it takes less space and maintenance than a mega patch.
comment:10 Changed 6 years ago by
Ideally, the configuration for sagelib runtime should take place in a generated Python file and not involve environment variables at all.
comment:11 Changed 6 years ago by
More likely parts of it would be static, and other parts could be generated, but yes, what I meant above in terms of rewriting sageenv
in Python was that it would not (by default) involve environment variables at all.
But it's still useful for interacting with other tools, especially for build purposes, as well as dropping into ./sage sh
to be able to set environment variables, which is why I suggest a mode to output a list of variables that can be read from the shell.
comment:12 Changed 3 years ago by
 Branch set to u/mkoeppe/split_sage_env_into_sage_build_env_and_sage_env
comment:13 Changed 3 years ago by
 Cc dimpase added
 Commit set to 9189d609fa8cc90d616cb7ea05be405ce196295e
 Dependencies set to #25130
 Milestone changed from sage7.5 to sage8.8
 Status changed from new to needs_review
comment:14 Changed 3 years ago by
Asis this is going to conflict with tickets like #27825 that are adding still more environment variables needed at buildtime to sageenvconfig.
I'm completely in favor of this separation in principle though. The question is whether we should do this first and then update tickets like #27825 on top of it, or the other way around?
comment:15 Changed 3 years ago by
comment:16 Changed 3 years ago by
The tickets would be trivial to rebase on top of this one.
comment:17 followup: ↓ 26 Changed 3 years ago by
Probably one should introduce a command sage buildsh
, which would provide the larger environment.
comment:18 Changed 3 years ago by
For the sake of a bit more terseness I wonder if we could call it just sageenvbuild.in
and sageenvbuild
. The only reason for the config
in the other one is to distinguish it from plain sageenv
.
Now that I look I'm not even entirely sure we need to separate sageenvconfig
from sageenv
.
I think we could probably have a sageenv.in
produce sageenv
and get rid of the separate sageenvconfig
entirely, though it's possible there's some subtlety to that that I'm missing.
comment:19 Changed 3 years ago by
 Branch changed from u/mkoeppe/split_sage_env_into_sage_build_env_and_sage_env to public/split_sage_env_into_sage_build_env_and_sage_env
 Commit changed from 9189d609fa8cc90d616cb7ea05be405ce196295e to ff0712125a3d05500a1f77e62c371d695d1a9794
Went ahead and rebased on current develop so it could incorporate all the additional variables we've since added.
I didn't change the name of the script yet; I would still prefer to shorten it but if anyone disagrees we can leave it asis.
New commits:
a34a9f1  Move sagedisthelpers from src/bin to build/bin

ff07121  Split out build/bin/sagebuildenvconfig from sageenvconfig

comment:20 Changed 3 years ago by
The only subtility to sageenvconfig
in my mind, is that you can touch it or even regenerate it without touching sageenv
. But I am sure you don't need a separate file to achieve your objectives in either case.
comment:21 Changed 3 years ago by
No, probably not. I might try it, though we'll leave that as a separate exercise from this ticket.
comment:22 Changed 3 years ago by
 Milestone changed from sage8.8 to sage8.9
Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).
comment:23 Changed 3 years ago by
 Milestone changed from sage8.9 to sage9.1
Ticket retargeted after milestone closed
comment:24 Changed 3 years ago by
 Status changed from needs_review to needs_work
comment:25 followup: ↓ 27 Changed 3 years ago by
The runtime environment variables of sage could as well be configured by a Python module src/sage/env_config.py
 see #29022.
comment:26 in reply to: ↑ 17 Changed 3 years ago by
Replying to mkoeppe:
Probably one should introduce a command
sage buildsh
, which would provide the larger environment.
As an extension of this, there could be sage buildsh SPKG
, which would set up the environment in which spkginstall
runs. This could be useful for testing new versions of packages.
comment:27 in reply to: ↑ 25 Changed 3 years ago by
comment:28 Changed 3 years ago by
 Branch public/split_sage_env_into_sage_build_env_and_sage_env deleted
 Cc isuruf arojas infinity0 ghtimokau added
 Commit ff0712125a3d05500a1f77e62c371d695d1a9794 deleted
 Dependencies changed from #25130 to #29038
 Description modified (diff)
 Status changed from needs_work to needs_info
 Summary changed from Split sageenv into sagebuildenv and sageenv to Split sageenv into 5
 Type changed from enhancement to task
Hoping for input from people involved in packaging.
comment:29 Changed 3 years ago by
 Status changed from needs_info to needs_review
comment:30 Changed 3 years ago by
 Description modified (diff)
comment:31 Changed 3 years ago by
 Description modified (diff)
comment:32 Changed 3 years ago by
 Dependencies changed from #29038 to #29038, #29052
comment:33 Changed 3 years ago by
 Summary changed from Split sageenv into 5 to Metaticket: Split sageenv into 5 to clean up sage configuration
comment:34 Changed 3 years ago by
 Description modified (diff)
comment:35 Changed 3 years ago by
 Description modified (diff)
comment:36 Changed 3 years ago by
 Cc mjo added
comment:37 Changed 2 years ago by
 Description modified (diff)
 Status changed from needs_review to needs_work
comment:38 Changed 2 years ago by
 Description modified (diff)
comment:39 Changed 2 years ago by
 Milestone changed from sage9.1 to sage9.2
comment:40 Changed 2 years ago by
 Description modified (diff)
comment:41 Changed 2 years ago by
 Cc jhpalmieri added
comment:42 Changed 2 years ago by
 Keywords sd109 added
comment:43 Changed 2 years ago by
 Description modified (diff)
comment:44 Changed 2 years ago by
 Description modified (diff)
comment:45 Changed 2 years ago by
 Description modified (diff)
comment:46 Changed 2 years ago by
 Description modified (diff)
comment:47 Changed 22 months ago by
 Description modified (diff)
comment:48 Changed 22 months ago by
 Description modified (diff)
comment:49 Changed 22 months ago by
Not sure if this is the right place, but I would suggest to use the refactoring process to also convert the environment variables to configuration files.
The idea would be to have a toml
file (or yaml
or whatever you prefer) in some predefined folder (say src
for the start). This file could be generated e.g by the configure
script. This config file is then parsed and wrapped by a python class (natural candidate would be the current sage_conf.py
). The python class also provides reasonable default values in case the config file doesn't exist.
This has several advantages in my opinion:
 More transparent, as one sees on one glance which variables influence sage.
 Easily editable by the user/developer.
 Local (e.g. think about multiple virtual environments with different configs).
 Typesafe, i.e, it is ensured when parsing the toml file that the types of the config variables is correct.
 Easier for downstream package managers, since they only need to provide a proper config file.
 (Higher security as one cannot inject arbitrary python code, which would be the case if
sage_conf.py
should be edited.)
What do you think?
comment:50 Changed 22 months ago by
sage_conf
defines an interface already  and the plan is indeed to get rid of dependencies on environment variables using this interface.
How sage_conf
obtains the configuration is orthogonal to the defined interface.
I am not interested in defining configuration files at the moment. There is just too much potential on wasting energy on discussions regarding the "best" config file format.
comment:51 Changed 22 months ago by
 Milestone changed from sage9.2 to sage9.3
comment:52 followups: ↓ 53 ↓ 54 Changed 22 months ago by
Well, you are actually proposing to add a configuration file in the form of a python file (sage_conf.py
). One could easily bypass an extensive discussion and follow the arguments of https://www.python.org/dev/peps/pep0518/#overviewoffileformatsconsidered.
But I agree that my suggestion concerning a config file is somewhat out of line with the goal of this ticket. I would still suggest to keep this in mind while designing the interface for sage_conf
, so that it might be easy in the future to switch to a design based on a config file. In particular, I would replace the module wide variables (like VERSION
) with methods in a config class (like Config.get_version
) that can also provide suitable defaults and do slight postprocessing.
comment:53 in reply to: ↑ 52 Changed 22 months ago by
Replying to ghtobiasdiez:
Well, you are actually proposing to add a configuration file in the form of a python file (
sage_conf.py
).
Note that sage_conf.py
is already in use since Sage 9.1
comment:54 in reply to: ↑ 52 Changed 22 months ago by
Replying to ghtobiasdiez:
I would still suggest to keep this in mind while designing the interface for
sage_conf
, so that it might be easy in the future to switch to a design based on a config file. In particular, I would replace the module wide variables (likeVERSION
) with methods in a config class (likeConfig.get_version
) that can also provide suitable defaults and do slight postprocessing.
All of this discussion is certainly valid and I am sure that you (and others) have valuable insights there, but now is simply not the right time for it. To get the design right, one needs to know about objectives and constraints. We are in the process of expanding the ways of Sage installation/deployment into several directions, including pipinstallability and modularization; and also binary packaging will need an overhaul.
comment:55 Changed 20 months ago by
 Description modified (diff)
comment:56 Changed 18 months ago by
 Milestone changed from sage9.3 to sage9.4
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
comment:57 Changed 18 months ago by
 Description modified (diff)
comment:58 Changed 13 months ago by
 Milestone changed from sage9.4 to sage9.5
Setting a new milestone for this ticket based on a cursory review.
comment:59 Changed 8 months ago by
 Cc ghtobiasdiez added
From the discussion in #32904 regarding sage_conf
at build time (with pip's build isolation) vs. runtime:
 pip specifies "ignoreinstalled" while constructing the build environment and thus will build sageconf twice. https://github.com/pypa/pip/blob/main/src/pip/_internal/build_env.py#L221
 The PyPI version
sage_conf
(from #29039) keys theSAGE_ROOT
/SAGE_LOCAL
that it creates in~/.sage
to the version ofsage_conf
 So if the
sage_conf
version is not completely pinned in thesagemathstandard
build and install requirements, then different versions (and thusSAGE_ROOT
s) may end up being used at build and install ofsagemathstandard
comment:60 followup: ↓ 61 Changed 8 months ago by
From https://trac.sagemath.org/ticket/32904#comment:24:
 Split
sageconf
into build and runtime configuration. There is still an overlap (and thus duplication) but at built time sage needs less dependencies than at runtime. In addition, maybe directly incorporate the buildtimesageconf
in the sagemath package.
comment:61 in reply to: ↑ 60 Changed 8 months ago by
Replying to mkoeppe:
From ghtobiasdiez in https://trac.sagemath.org/ticket/32904#comment:24:
 Split
sageconf
into build and runtime configuration. There is still an overlap (and thus duplication) but at build time sage needs less dependencies than at runtime. In addition, maybe directly incorporate the buildtimesageconf
in the sagemath package.
I think this is an interesting direction
comment:62 Changed 8 months ago by
 Milestone changed from sage9.5 to sage9.6
comment:63 Changed 4 months ago by
 Milestone changed from sage9.6 to sage9.7
comment:64 Changed 13 days ago by
 Description modified (diff)
One comment. Currently
sage_setup
is installed because it needs to be installed to be doctested  like the rest of sage. But nothing insage_setup
is really needed at runtime and distro can choose not to install it. I don't in sageongentoo. I tend to push for thing not needed at runtime to be moved in there. And that's where I would have putsagebuildenv
but I am not going to fight it going intobuild/bin
although I'd like everything used bysetup.py
to be neatly undersrc
.