Opened 6 years ago
Last modified 3 months ago
#21469 needs_work enhancement
Enable VPATH builds (several independent build trees connected to one source tree)
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage9.7 
Component:  build: configure  Keywords:  
Cc:  felixs, jdemeyer, fbissey, embray, leif, vbraun, dimpase, chapoton, jhpalmieri  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  
Report Upstream:  N/A  Work issues:  merge conflicts 
Branch:  u/mkoeppe/vpath_build (Commits, GitHub, GitLab)  Commit:  05f3a9107a55a6c308196c85a995f867b452fb9c 
Dependencies:  Stopgaps: 
Description (last modified by )
Executive summary Currently, prefix
selects the install tree (SAGE_LOCAL
) of sagethedistribution. But often we want several builds out of (almost) the same source tree.
The vpath
mechanism proposed here selects the build tree (SAGE_ROOT
) of sagethedistribution. This includes the configuration and build artefacts such as src/build/...
.
VPATH builds are a powerful feature of autotools (and other) build systems. This feature allows the developer to build a package in a separate (initially empty) build directory tree. The source directory tree is only read from (and could even be mounted readonly) and is therefore always clean.
This is a powerful feature for the developer because from the same source tree many different configurations can be built and tested, without having to go through "make distclean" and reconfiguration. The source tree can also be shared, for example using a networked file system between different hosts, running different architectures. Another modern use case involves VMs. For example, Docker allows to mount the source directory from the host in the VM (see #21474).
For Sage, "different configurations" could mean different architectures (via VMs), different sets of installed packages, Py2 vs Py3, etc.
This is how it is used.
cd SAGE_ROOT autoreconf
and then
mkdir BUILDDIR cd BUILDDIR SRCDIR/configure srcdir=SRCDIR make
The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_VPATH. $SAGE_LOCAL defaults to BUILDDIR/local, but can of course be changed using configure prefix
.
The present patch changes the various places where $SAGE_ROOT is used.
When "./configure" detects a VPATH build, it installs patched copies of sage
, Makefile
and build/make/install
in BUILDDIR.
With the current set of patches, the compilation goes through and gives a working sage
, except for the docbuild.
Change History (114)
comment:1 Changed 6 years ago by
 Branch set to u/mkoeppe/vpath_build
comment:2 Changed 6 years ago by
 Cc leif added
 Commit set to ac27cdc02f685d79d817c71ebdcc1ee79196b4af
comment:3 Changed 6 years ago by
 Commit changed from ac27cdc02f685d79d817c71ebdcc1ee79196b4af to e9b1c0fb2a378370b454757e397d20349323434f
comment:4 Changed 6 years ago by
 Cc vbraun dimpase added
 Status changed from new to needs_review
comment:5 Changed 6 years ago by
 Commit changed from e9b1c0fb2a378370b454757e397d20349323434f to e15ada69bb22124f463820e9dab5cc16348b3993
Branch pushed to git repo; I updated commit sha1. New commits:
e15ada6  Define SAGE_SRC_ROOT for VPATH build

comment:6 Changed 6 years ago by
 Commit changed from e15ada69bb22124f463820e9dab5cc16348b3993 to 9323c011da3005c0beb46ba0da5a6d3dba13c9e7
Branch pushed to git repo; I updated commit sha1. New commits:
9323c01  Use SAGE_SRC_ROOT in package scripts

comment:7 Changed 6 years ago by
 Commit changed from 9323c011da3005c0beb46ba0da5a6d3dba13c9e7 to 7a3913a64aa1d4310f741a3e5f898635187eb38f
Branch pushed to git repo; I updated commit sha1. New commits:
7a3913a  More SAGE_SRC_ROOT

comment:8 Changed 6 years ago by
 Commit changed from 7a3913a64aa1d4310f741a3e5f898635187eb38f to 4cc3954f1ae666aa0148d69256e59d9e17ee4258
Branch pushed to git repo; I updated commit sha1. New commits:
4cc3954  More SAGE_SRC_ROOT

comment:9 Changed 6 years ago by
The current patch makes some highlevel preparations for a VPATH build, in particular the package system.
"make" runs through until it gets to sagelib, which needs some more VPATH work (the current patch tries to get away by just making a symlink from $SAGE_SRC_ROOT/src to $SAGE_ROOT/src, which of course fails).
I've set this to 'needs review' in the hope of receiving some feedback regarding the approach of this patch.
comment:10 followups: ↓ 11 ↓ 13 Changed 6 years ago by
I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH
brings me.
This is most evident in the latest commit. As a distro maintainer of sage there are already build time things in env.py
that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py
and some in a new file sage_setup/build_env.py
but that's marginal to what you are doing.
Other people may have a different opinion but there is no value in this for me so far. Possibly more work in fact.
comment:11 in reply to: ↑ 10 Changed 6 years ago by
Replying to fbissey:
I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what
VPATH
brings me.
A VPATH build makes it easy to build from the same source in many different configurations. Concretely, I've started to run some VMs (with Linux 64 bit, 32 bit) via Docker, which mount the Sage source directory from the host.
comment:12 Changed 6 years ago by
 Commit changed from 4cc3954f1ae666aa0148d69256e59d9e17ee4258 to 7bb87184fcd1a5a39914af65e12ce4d6ed30e775
Branch pushed to git repo; I updated commit sha1. New commits:
7bb8718  Make sage script in build dir executable

comment:13 in reply to: ↑ 10 ; followup: ↓ 16 Changed 6 years ago by
Replying to fbissey:
As a distro maintainer of sage there are already build time things in
env.py
that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff insage/env.py
and some in a new filesage_setup/build_env.py
but that's marginal to what you are doing.
Yes, this ticket does not address any of the issues that would arise in distribution packaging.
Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.
comment:14 followup: ↓ 15 Changed 6 years ago by
I should apologize for being so negative. Currently configure
is run by the initial call to make, it would be nice to get to the stage where we run configure
first :)
Seriously, yes VPATH
as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean
or git clone
is a good thing.
I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH
work may help to clarify it in some areas. Let's try to minimise where it would mix things further.
comment:15 in reply to: ↑ 14 Changed 6 years ago by
Replying to fbissey:
I should apologize for being so negative.
No worries.
Currently
configure
is run by the initial call to make, it would be nice to get to the stage where we runconfigure
first :)
That already works with this patch.
Seriously, yes
VPATH
as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source withoutmake distclean
orgit clone
is a good thing.I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your
VPATH
work may help to clarify it in some areas. Let's try to minimise where it would mix things further.
Sounds good. Yes, I'm hoping too that it helps clarify some things as a sideeffect.
comment:16 in reply to: ↑ 13 ; followup: ↓ 19 Changed 6 years ago by
Replying to mkoeppe:
Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.
I sort of agree with François.
Part of the problem is that Sage is a distribution (a collection of packages) as well as a Python library. The fact this this is currently mixed up in the sources (the sources of the distributionpart and the librarypart are in one repo) makes it more difficult to understand what this ticket is about.
For the distribution part, there really is no such thing as a build directory. There is a a temporary directory in local/var/tmp/sage/build
but I would not consider that a build directory in the usual sense.
For the library part, a VPATH
build is meaningful. Although, I think that VPATH
builds are unusual in the Python world. Luckily, the default build directories are platformdependendent. Anyway, it seems like this ticket isn't about that.
comment:17 Changed 6 years ago by
 Status changed from needs_review to needs_info
I think the ticket description should have a lot more information about
 which problem that this ticket is trying to solve (the why).
 what the branch actually does (the what).
Note that this are 2 independent things (which are unfortunately often confused).
comment:18 Changed 6 years ago by
 Description modified (diff)
 Status changed from needs_info to needs_review
I've updated the description.
comment:19 in reply to: ↑ 16 ; followups: ↓ 22 ↓ 24 Changed 6 years ago by
Replying to jdemeyer:
Replying to mkoeppe: Part of the problem is that Sage is a distribution (a collection of packages) as well as a Python library. The fact this this is currently mixed up in the sources (the sources of the distributionpart and the librarypart are in one repo) makes it more difficult to understand what this ticket is about.
I've now clarified the goal in the description. I have not touched sagelib
yet; but without VPATH work on sagelib, the distribution VPATH changes won't work. I could use help with the work on sagelib
.
For the distribution part, there really is no such thing as a build directory. There is a a temporary directory in
local/var/tmp/sage/build
but I would not consider that a build directory in the usual sense.
Right, local/var/tmp/sage/build
falls short of being a real build directory. because for example src/build/
has a lot of more generated stuff.
For the library part, a
VPATH
build is meaningful. Although, I think thatVPATH
builds are unusual in the Python world. Luckily, the default build directories are platformdependendent. Anyway, it seems like this ticket isn't about that.
Certainly there src/build/lib.macosx10.9x86_642.7
includes the platform name, but it does not account for other configuration differences that a VPATH build can take care of (neither does src/build/cythonized
).
comment:20 Changed 6 years ago by
 Description modified (diff)
comment:21 Changed 6 years ago by
 Description modified (diff)
comment:22 in reply to: ↑ 19 ; followup: ↓ 23 Changed 6 years ago by
Replying to jdemeyer:
For the library part, a
VPATH
build is meaningful. Although, I think thatVPATH
builds are unusual in the Python world. Luckily, the default build directories are platformdependendent. Anyway, it seems like this ticket isn't about that.
It seems that pip install
keeps the source directory clean, building instead in a temporary directory.
pip install
also offers options build
to select a build directory (though it seems as if it does not work with all packages).
comment:23 in reply to: ↑ 22 Changed 6 years ago by
Replying to mkoeppe:
It seems that
pip install
keeps the source directory clean, building instead in a temporary directory.pip install
also offers optionsbuild
to select a build directory (though it seems as if it does not work with all packages).
However, there are some pip issues: https://github.com/pypa/pip/issues/2060 https://github.com/pypa/pip/issues/2053 https://github.com/pypa/pip/issues/804
comment:24 in reply to: ↑ 19 Changed 6 years ago by
Replying to jdemeyer: For the library part, a
VPATH
build is meaningful. Although, I think thatVPATH
builds are unusual in the Python world. Luckily, the default build directories are platformdependendent. Anyway, it seems like this ticket isn't about that.
Also, in the traditional distutils ways (https://docs.python.org/2/install/), it appears that one can use buildbase
to do VPATH builds.
python setup.py build buildbase=BUILDDIR
comment:26 Changed 6 years ago by
 Commit changed from 7bb87184fcd1a5a39914af65e12ce4d6ed30e775 to 81f4306be63d6097c98c29301c6b21bc3a1f1e01
Branch pushed to git repo; I updated commit sha1. New commits:
81f4306  Fix for unconfigured nonVPATH build

comment:27 Changed 6 years ago by
 Milestone changed from sage7.4 to sage7.5
comment:28 Changed 6 years ago by
 Status changed from needs_review to needs_work
comment:29 Changed 5 years ago by
 Commit changed from 81f4306be63d6097c98c29301c6b21bc3a1f1e01 to 7e11d55e6549a678bc5a538d64353426a8b8a4b9
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
7e11d55  VPATH build

comment:30 Changed 5 years ago by
squashed into 1 commit to simplify rebasing.
comment:31 Changed 5 years ago by
 Dependencies changed from #21480 to #21535
 Milestone changed from sage7.5 to sage8.2
The next step towards VPATH builds should probably be #21535 (enabling src/setup.py build buildbase
, to replace the hardcoded "build").
comment:32 Changed 5 years ago by
 Commit changed from 7e11d55e6549a678bc5a538d64353426a8b8a4b9 to 23cc689ccfd33fff4eeb05ee5dc6ebf76ff2db9c
comment:33 Changed 5 years ago by
 Commit changed from 23cc689ccfd33fff4eeb05ee5dc6ebf76ff2db9c to 72818e5229b573b625e4b9b181828ff7dcd22a11
Branch pushed to git repo; I updated commit sha1. New commits:
72818e5  Some path fixes for VPATH

comment:34 Changed 5 years ago by
 Commit changed from 72818e5229b573b625e4b9b181828ff7dcd22a11 to d0366b0bd4506fe3e3378fc752f7df487f783365
comment:35 Changed 5 years ago by
 Cc chapoton jhpalmieri added
 Dependencies #21535 deleted
Removed #21535 dependency. setup.py buildbase
actually works already.
With the current status of the branch, make
builds a working sage, only the docbuild does not go through yet.
comment:36 Changed 5 years ago by
 Description modified (diff)
comment:37 Changed 5 years ago by
 Description modified (diff)
comment:38 Changed 5 years ago by
 Summary changed from Enable VPATH builds to Enable VPATH builds (several independent build trees connected to one source tree)
comment:39 Changed 5 years ago by
On OS X, some directories are not being created:
$ mkdir BUILD $ cd BUILD $ /path/to/sage/configure srcdir=/path/to/sage ... /path/to/sage/configure: line 7017: build/make/Makefile: No such file or directory
If I create that directory, configure
finishes but with warning messages like this
$ mkdir p build/make $ /path/to/sage/configure srcdir=/path/to/sage ... touch: /path/to/BUILD/src/bin/sageenvconfig: No such file or directory find: /path/to/BUILD/src/ext: No such file or directory cat: /path/to/BUILD/build/make/deps: No such file or directory ... $ make make: *** No targets specified and no makefile found. Stop.
comment:40 Changed 5 years ago by
Did you autoreconf?
comment:42 followup: ↓ 45 Changed 5 years ago by
With the changes

src/bin/sageenv
diff git a/src/bin/sageenv b/src/bin/sageenv index dc2bd34cc0..8aa523d7ea 100644
a b export SAGE_EXTCODE="$SAGE_SHARE/sage/ext" 313 313 export SAGE_SPKG_INST="$SAGE_LOCAL/var/lib/sage/installed" 314 314 export SAGE_LOGS="$SAGE_ROOT/logs/pkgs" 315 315 export SAGE_SRC="$SAGE_ROOT/src" 316 export SAGE_DOC_SRC="$SAGE_SRC /doc"316 export SAGE_DOC_SRC="$SAGE_SRC_ROOT/src/doc" 317 317 export SAGE_DOC="$SAGE_SHARE/doc/sage" 318 318 319 319 if [ z "${SAGE_ORIG_PATH_SET}" ]; then 
src/doc/common/conf.py
diff git a/src/doc/common/conf.py b/src/doc/common/conf.py index 2e115fab88..88f3fdfa4a 100644
a b 1 1 import sys, os, sphinx 2 from sage.env import SAGE_DOC_SRC, SAGE_DOC, SAGE_SRC , THEBE_DIR2 from sage.env import SAGE_DOC_SRC, SAGE_DOC, SAGE_SRC_ROOT, THEBE_DIR 3 3 from datetime import date 4 4 from six import iteritems 5 5 6 6 # If your extensions are in another directory, add it here. 7 sys.path.append(os.path.join(SAGE_SRC , "sage_setup", "docbuild", "ext"))7 sys.path.append(os.path.join(SAGE_SRC_ROOT, "src", "sage_setup", "docbuild", "ext")) 8 8 9 9 # General configuration 10 10 # 
the documentation starts to build, but I eventually get an error:
[dochtml] [arithgrou] /path/to/SAGE_ROOT/src/doc/en/reference/arithgroup/sage/modular/arithgroup/arithgroup_element.rst:11: WARNING: error while formatting arguments for sage.modular.arithgroup.arithgroup_element.ArithmeticSubgroupElement.b: 'NoneType' object has no attribute 'find'
I can't figure out where this is coming from.
comment:43 Changed 5 years ago by
 Commit changed from d0366b0bd4506fe3e3378fc752f7df487f783365 to e7a78e038548b6a0d34cea75432379c61759875d
comment:44 Changed 5 years ago by
Thanks, I've added your changes as a commit.
comment:45 in reply to: ↑ 42 Changed 5 years ago by
Replying to jhpalmieri:
the documentation starts to build, but I eventually get an error:
[dochtml] [arithgrou] /path/to/SAGE_ROOT/src/doc/en/reference/arithgroup/sage/modular/arithgroup/arithgroup_element.rst:11: WARNING: error while formatting arguments for sage.modular.arithgroup.arithgroup_element.ArithmeticSubgroupElement.b: 'NoneType' object has no attribute 'find'
I get the same error
comment:46 Changed 5 years ago by
One more change, to help make distclean
work:

build/make/deps
diff git a/build/make/deps b/build/make/deps index 3128f4c656..03f2c8fff1 100644
a b docpdf: $(DOC_DEPENDENCIES) 235 235 docclean: docsrcclean docoutputclean 236 236 237 237 docsrcclean: 238 cd "$(SAGE_SRC )/doc" && $(MAKE) clean238 cd "$(SAGE_SRC_ROOT)/src/doc" && $(MAKE) clean 239 239 240 240 docoutputclean: 241 241 rm rf "$(SAGE_SHARE)/doc/sage"
comment:47 Changed 5 years ago by
 Commit changed from e7a78e038548b6a0d34cea75432379c61759875d to 734fb4ba95dc62a7f6f10c30b6252fa309c6b7b0
Branch pushed to git repo; I updated commit sha1. New commits:
734fb4b  VPATH fix for docsrcclean

comment:48 Changed 5 years ago by
Perhaps the sphinx problem is related to failing source discovery for Cython modules:
sage: sage.rings.number_field.number_field_element_quadratic?? Error getting source: s must be a string Type: module ...
comment:49 Changed 5 years ago by
Yes, that looks like it.
comment:50 followup: ↓ 54 Changed 5 years ago by
Okay, how about this:

src/sage/misc/sageinspect.py
diff git a/src/sage/misc/sageinspect.py b/src/sage/misc/sageinspect.py index f9dffc1ce3..3a286c7934 100644
a b import tokenize 125 125 import types 126 126 import re 127 127 EMBEDDED_MODE = False 128 from sage.env import SAGE_SRC 128 from sage.env import SAGE_SRC_ROOT 129 129 130 130 def loadable_module_extension(): 131 131 r""" … … def _extract_embedded_position(docstring): 222 222 return None 223 223 224 224 raw_filename = res.group('FILENAME') 225 filename = os.path.join(SAGE_SRC , raw_filename)225 filename = os.path.join(SAGE_SRC_ROOT, 'src', raw_filename) 226 226 if not os.path.isfile(filename): 227 227 from sage.misc.misc import SPYX_TMP 228 228 filename = os.path.join(SPYX_TMP, '_'.join(raw_filename.split('_')[:1]), raw_filename) … … def __internal_tests(): 2308 2308 2309 2309 Test _extract_embedded_position: 2310 2310 2311 We cannot test the filename since it depends on SAGE_SRC .2311 We cannot test the filename since it depends on SAGE_SRC_ROOT. 2312 2312 2313 2313 Make sure things work with no trailing newline:: 2314 2314
comment:51 Changed 5 years ago by
Now the crossreferences are wrong or missing:
[dochtml] OSError: [parallel ] /.../sage8.1.beta3/src/doc/en/reference/parallel/index.rst:17: WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)
comment:52 Changed 5 years ago by
Also, it seems that the docbuilding doesn't recognize when a document has been built already, so everything is rebuilt every time.
comment:53 Changed 5 years ago by
 Commit changed from 734fb4ba95dc62a7f6f10c30b6252fa309c6b7b0 to 8c801f7e36d7d1d680ee175f8a5eac40e118568d
Branch pushed to git repo; I updated commit sha1. New commits:
8c801f7  VPATH: Set SAGE_SRC to actual source tree throughout

comment:54 in reply to: ↑ 50 Changed 5 years ago by
Replying to jhpalmieri:
Okay, how about this:
src/sage/misc/sageinspect.py
diff git a/src/sage/misc/sageinspect.py b/src/sage/misc/sageinspect.py index f9dffc1ce3..3a286c7934 100644
a b def _extract_embedded_position(docstring): 222 222 return None 223 223 224 224 raw_filename = res.group('FILENAME') 225 filename = os.path.join(SAGE_SRC , raw_filename)225 filename = os.path.join(SAGE_SRC_ROOT, 'src', raw_filename) 226 226 if not os.path.isfile(filename): 227 227 from sage.misc.misc import SPYX_TMP 228 228 filename = os.path.join(SPYX_TMP, '_'.join(raw_filename.split('_')[:1]), raw_filename)
Instead, I have changed SAGE_SRC
to point to the actual source tree. This was inconsistent before.
comment:55 Changed 5 years ago by
I'm now testing the VPATH build using a separate user account that cannot write into the source tree. More changes to the docbuild are necessary to separate sources from built files:
[dochtml] IOError: [Errno 13] Permission denied: '/Users/mkoeppe/s/sage/sagerebasing/vpath/B/../src/doc/en/reference/repl/sage/misc/trace.rst'
comment:56 Changed 5 years ago by
By the way, I get warnings like
touch: /.../BUILDDIR/src/bin/sageenvconfig: No such file or directory
I assume this isn't a big deal since you're planning on getting rid of this file. It also doesn't seem to interfere with the build or running Sage.
comment:57 Changed 5 years ago by
I don't plan to eliminate sageenvconfig
(perhaps you mean sagearchenv
, #23733)?
comment:58 Changed 5 years ago by
Sorry, yes, that's what I meant, and that's what most of the warnings have been about.
comment:59 Changed 5 years ago by
The sagearchenv
message is harmless.
But there's indeed also something wrong with sageenvconfig
too. Fix will come shortly.
comment:60 Changed 5 years ago by
 Commit changed from 8c801f7e36d7d1d680ee175f8a5eac40e118568d to 5530827f3266b9d43326ddba619bd3a757e0569d
Branch pushed to git repo; I updated commit sha1. New commits:
5530827  sageenvconfig: Handle special case in build/make/deps, rather than configure.ac

comment:61 Changed 5 years ago by
I'm getting this build error:
make[2]: *** No rule to make target `/.../BUILDDIR/local/share/sage/ext/doctest/invalid/syntax_error.tachyon', needed by `sagelib'. Stop.
comment:62 Changed 5 years ago by
 Commit changed from 5530827f3266b9d43326ddba619bd3a757e0569d to 64515c1bc26cf7d4cb6ac1665431afa4b87092f6
Branch pushed to git repo; I updated commit sha1. New commits:
64515c1  Fix last change

comment:63 Changed 5 years ago by
The make target "sagestarts" fails for me. This fixes it; is it the right way to fix it?

src/bin/sagestarts
diff git a/src/bin/sagestarts b/src/bin/sagestarts index 85cbae67e5..498c930810 100755
a b echo "[`date +'%Y%m%d %H:%M:%S'`] `cat VERSION.txt 2>/dev/null`"  tee a logs 14 14 # sysadmin. We use nodotsage so we don't force a .sage directory 15 15 # in the sysadmin's HOME directory. 16 16 cmd='sage.all._write_started_file(); print "Yes, Sage starts."' 17 build/bin/sagelogger "./sage nodotsage c '$cmd'" logs/start.log17 "$SAGE_SRC_ROOT"/build/bin/sagelogger "./sage nodotsage c '$cmd'" logs/start.log 18 18 19 19 if [ $? ne 0 ]; then 20 20 echo >&2 "Sage failed to start up."
comment:64 followup: ↓ 66 Changed 5 years ago by
With that change, make ptestlong
fails:
Testing that Sage starts... [20170830 09:19:29] This looks like the first time you are running Sage. Cleaning up, do not interrupt this. Done cleaning. Yes, Sage starts. real 52m33.269s user 191m2.435s sys 19m38.034s Sage build/upgrade complete! To install small scripts to directly run Sage's versions of GAP, the PARI/GP interpreter, Maxima, or Singular etc. (by typing e.g. just 'gap' or 'gp') into a standard 'bin' directory, start Sage by typing 'sage' (or './sage') and enter something like install_scripts('/usr/local/bin') at the Sage command prompt ('sage:'). /.../SAGE_SRC_ROOT/sage t p all long logfile=logs/ptestlong.log ************************************************************************ It seems that you are attempting to run Sage from an unpacked source tarball, but you have not compiled it yet (or maybe the build has not finished). You should run `make` in the Sage root directory first. If you did not intend to build Sage from source, you should download a binary tarball instead. Read README.txt for more information. ************************************************************************ make: *** [ptestlong] Error 1
I think that the make target
TESTALL = $(top_srcdir)/sage t all
needs to be changed so that it runs SAGE_ROOT/sage
instead of SAGE_SRC_ROOT/sage
.
./sage t all
also fails:
$ ./sage t all /.../SAGE_SRC_ROOT/src/bin/sageenv: line 527: /.../BUILDDIR/src/bin/sagearchenv: No such file or directory Traceback (most recent call last): File "/.../SAGE_SRC_ROOT/src/bin/sageruntests", line 97, in <module> DC = DocTestController(options, args) File "/.../BUILDDIR/local/lib/python2.7/sitepackages/sage/doctest/control.py", line 334, in __init__ for pkg in list_packages('optional', local=True).values(): File "/.../BUILDDIR/local/lib/python2.7/sitepackages/sage/misc/package.py", line 230, in list_packages for p in os.listdir(SAGE_PKGS): OSError: [Errno 2] No such file or directory: '/.../BUILDDIR/build/pkgs'
This can almost be fixed by

src/sage/env.py
diff git a/src/sage/env.py b/src/sage/env.py index 33cd754a06..f91c70ba7d 100644
a b _add_variable_or_fallback('SAGE_LIB', SITE_PACKAGES[0]) 111 111 _add_variable_or_fallback('SAGE_CYTHONIZED', opj('$SAGE_ROOT', 'src', 'build', 'cythonized')) 112 112 113 113 # Used by sage/misc/package.py. Should be SAGE_SRC_ROOT in VPATH. 114 _add_variable_or_fallback('SAGE_PKGS', opj('$SAGE_ ROOT', 'build', 'pkgs'))114 _add_variable_or_fallback('SAGE_PKGS', opj('$SAGE_SRC_ROOT', 'build', 'pkgs')) 115 115 116 116 117 117 _add_variable_or_fallback('SAGE_EXTCODE', opj('$SAGE_SHARE', 'sage', 'ext'))
but there is a bug in the function _add_variable_or_fallback
in how it parses variable names prefixed with dollar signs. I'll open a new ticket for that.
comment:65 Changed 5 years ago by
If I hack the toplevel Makefile so that make ptestlong
works, then I only get test failures in sage/tests/cmdline.py
.
comment:66 in reply to: ↑ 64 Changed 5 years ago by
Replying to jhpalmieri:
there is a bug in the function
_add_variable_or_fallback
in how it parses variable names prefixed with dollar signs. I'll open a new ticket for that.
See #23758.
comment:67 Changed 5 years ago by
 Commit changed from 64515c1bc26cf7d4cb6ac1665431afa4b87092f6 to 3e36b75f51413973d847994aed067dbe7bfef822
Branch pushed to git repo; I updated commit sha1. New commits:
bc6263c  VPATH fix for SAGE_PKGS

f9c25bc  VPATH fix for test makefile targets

ecd2f9d  docsrcclean: Clean in source directory until docbuild is fully VPATHified

60927e5  trac 23758: in env.py, _add_variable_or_fallback should depend less

c23cc1c  trac 23758: restore "import os"

343d685  trac 23758: add a doctest

3e36b75  Merge remotetracking branch 'trac/u/jhpalmieri/variable_replacement' into t/21469/vpath_build

comment:68 followup: ↓ 69 Changed 5 years ago by
I'm still getting docbuild errors of the type WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)
comment:69 in reply to: ↑ 68 Changed 5 years ago by
Replying to mkoeppe:
I'm still getting docbuild errors of the type
WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)
This is a general problem, nothing to worry in the context of this ticket.
By the way you might like to use here the upgraded sphinx on #23023, which will most probably be in the next beta.
comment:70 followups: ↓ 71 ↓ 74 Changed 5 years ago by
I agree that the docbuild warnings are not new. (My issue in comment:51 was that these were errors, causing docbuilding to actually fail. I am now able to build the docs successfully.)
I am still having the problem in comment:63 with sagestarts
.
comment:71 in reply to: ↑ 70 ; followup: ↓ 72 Changed 5 years ago by
Replying to jhpalmieri:
I agree that the docbuild warnings are not new. (My issue in comment:51 was that these were errors, causing docbuilding to actually fail. I am now able to build the docs successfully.)
Unfortunately these warnings get upgraded to errors like this:
[dochtml] OSError: [parallel ] /Users/mkoeppe/s/sage/sagerebasing/vpath/src/doc/en/reference/parallel/index.rst:17: WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)
which stops this build for me.
I'll try with #23023 as suggested by Dima.
comment:72 in reply to: ↑ 71 Changed 5 years ago by
Replying to mkoeppe:
Replying to jhpalmieri:
I agree that the docbuild warnings are not new. (My issue in comment:51 was that these were errors, causing docbuilding to actually fail. I am now able to build the docs successfully.)
Unfortunately these warnings get upgraded to errors like this:
[dochtml] OSError: [parallel ] /Users/mkoeppe/s/sage/sagerebasing/vpath/src/doc/en/reference/parallel/index.rst:17: WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)which stops this build for me.
Your change in comment:53 fixed that issue for me. (At least I think that's what fixed it.)
comment:73 Changed 5 years ago by
 Commit changed from 3e36b75f51413973d847994aed067dbe7bfef822 to 754ec81627aa01a9bfa5646e946bdea02fe220ff
Branch pushed to git repo; I updated commit sha1. New commits:
754ec81  sagestarts: Fix sagelogger invocation for VPATH build

comment:74 in reply to: ↑ 70 Changed 5 years ago by
Replying to jhpalmieri:
I am still having the problem in comment:63 with
sagestarts
.
I agree with your patch and have added a commit; but see #23769.
comment:75 Changed 5 years ago by
 Commit changed from 754ec81627aa01a9bfa5646e946bdea02fe220ff to 0c357dd5da4d70a8a3f3ca96984359a563686307
Branch pushed to git repo; I updated commit sha1. New commits:
0c357dd  Simplify VPATH patch

comment:76 Changed 5 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
comment:77 Changed 5 years ago by
I think this ticket is ready for broader review.
comment:78 Changed 5 years ago by
 Description modified (diff)
make distclean
fails because make sagelibclean
fails:
cd "/.../SAGE_SRC_ROOT/src" && make j6 clean make[2]: warning: jN forced in submake: disabling jobserver mode. make[2]: *** No rule to make target `clean'. Stop. make[1]: *** [sagelibclean] Error 2
I guess it should cd to /.../BUILDDIR/src
instead.
comment:79 Changed 5 years ago by
 Commit changed from 0c357dd5da4d70a8a3f3ca96984359a563686307 to d32598ee2cf41e37af931b0842e520eac43b9319
Branch pushed to git repo; I updated commit sha1. New commits:
d32598e  VPATH fix for sagelibclean

comment:80 Changed 5 years ago by
 Commit changed from d32598ee2cf41e37af931b0842e520eac43b9319 to 06f7d1d6914fe55bf1d3eedab04a9aa13f260dfc
Branch pushed to git repo; I updated commit sha1. New commits:
06f7d1d  Fix build/make/install change

comment:81 Changed 5 years ago by
I tried again from a distcleaned tree. Now I'm running into:
[sagelib8.1.beta3] Generating autogenerated sources [sagelib8.1.beta3] Building interpreters for fast_callable [sagelib8.1.beta3] > First build of interpreters [sagelib8.1.beta3] error: [Errno 2] No such file or directory: 'sage/ext/interpreters/interp_cc.c'
comment:82 followup: ↓ 83 Changed 5 years ago by
I haven't seen that after running make distclean
in both the original source directory and the build directory, but I could imagine it would happen if you don't have write access to the original source directory, since the interpreters directory and its contents are autogenerated (see src/sage_setup/autogen
). Where should those files be created?
comment:83 in reply to: ↑ 82 ; followups: ↓ 90 ↓ 91 Changed 5 years ago by
Replying to jhpalmieri:
I haven't seen that after running
make distclean
in both the original source directory and the build directory, but I could imagine it would happen if you don't have write access to the original source directory, since the interpreters directory and its contents are autogenerated (seesrc/sage_setup/autogen
).
Yes, this is how I am testing it (except I made src/doc
writable).
Something in the code generator is missing error checking. I suspect it's coming fromsrc/sage_setup/autogen/interpreters/__init__.py
, which has some complicated exception handling.
Where should those files be created?
They should be created in SAGE_ROOT/src/sage/ext
.
comment:84 Changed 5 years ago by
I think we need a variable for SAGE_ROOT/src
to make this patch cleaner. Suggestions for the variable name?
comment:85 followup: ↓ 86 Changed 5 years ago by
Perhaps should:
 rename SAGE_SRC_ROOT to SAGE_VPATH
 rename SAGE_SRC to SAGE_VPATH_SRC = SAGE_VPATH/src (and change many places that now use SAGE_SRC)
 create SAGE_SRC = SAGE_ROOT/src
 rename SAGE_DOC_SRC to SAGE_VPATH_SRC_DOC = SAGE_VPATH/src/doc
 create SAGE_SRC_DOC = SAGE_ROOT/src/doc
comment:86 in reply to: ↑ 85 Changed 5 years ago by
Replying to mkoeppe:
 create SAGE_SRC = SAGE_ROOT/src
How about SAGE_ROOT_SRC
? (only if you're changing SAGE_SRC_ROOT
)
 rename SAGE_DOC_SRC to SAGE_VPATH_SRC_DOC = SAGE_VPATH/src/doc
 create SAGE_SRC_DOC = SAGE_ROOT/src/doc
Or SAGE_ROOT_SRC_DOC
?
comment:87 Changed 5 years ago by
Thanks, these are good suggestions.
comment:88 Changed 5 years ago by
 Commit changed from 06f7d1d6914fe55bf1d3eedab04a9aa13f260dfc to 05f3a9107a55a6c308196c85a995f867b452fb9c
Branch pushed to git repo; I updated commit sha1. New commits:
05f3a91  Rename SAGE_SRC_ROOT > SAGE_VPATH

comment:89 Changed 5 years ago by
 Description modified (diff)
comment:90 in reply to: ↑ 83 ; followup: ↓ 92 Changed 5 years ago by
Replying to mkoeppe:
Replying to jhpalmieri:
I haven't seen that after running
make distclean
in both the original source directory and the build directory, but I could imagine it would happen if you don't have write access to the original source directory, since the interpreters directory and its contents are autogenerated (seesrc/sage_setup/autogen
).Yes, this is how I am testing it (except I made
src/doc
writable).Something in the code generator is missing error checking. I suspect it's coming from
src/sage_setup/autogen/interpreters/__init__.py
, which has some complicated exception handling.Where should those files be created?
They should be created in
SAGE_ROOT/src/sage/ext
.
It looks like they are actually created in SAGE_SRC_ROOT=SAGE_VPATH
, then copied to SAGE_ROOT/...
.
comment:91 in reply to: ↑ 83 Changed 5 years ago by
comment:92 in reply to: ↑ 90 Changed 5 years ago by
Replying to jhpalmieri:
Where should those files be created?
They should be created in
SAGE_ROOT/src/sage/ext
.It looks like they are actually created in
SAGE_SRC_ROOT=SAGE_VPATH
...
Yes, see src/sage_setup/autogen/interpreters/__main__.py
.
But I think I will defer full VPATHification of this and of the docbuild to followup tickets.
comment:93 Changed 5 years ago by
I wonder if instead of
+dnl Set up files for VPATH build +AS_IF([test x$srcdir != x.], + [AC_CONFIG_COMMANDS([vpath], + [ + SAGE_ROOT=${ac_abs_top_builddir} + SAGE_VPATH=${ac_abs_top_srcdir} + # Assume that $srcdir is readonly, so symlink existing upstream files into build directory upstream + # If new packages are downloaded, they end up in the build directory upstream. + mkdir p upstream + for a in $SAGE_VPATH/upstream/* ; do + if test f "$a" a ! f "upstream/`basename $a`" ; then + ln s "$a" upstream/ + fi + done + cat > sage <<EOF
we could simply modify whatever code looks for the upstream/
directory (presumably mostly in sagespkg
) to explicitly look for it under $SAGE_ROOT
. In other words I don't see why making symlinks in the VPATH sources are necessary if we already know where to find the upstream/
directory.
comment:94 Changed 5 years ago by
I'm seeing a lot of activity on this ticket that I haven't followed.
Anyway, I still don't quite understand what a VPATH
build even means in the context of Sage. I really think that this should be explained better, and how this differs from ./configure prefix=...
comment:95 followup: ↓ 96 Changed 5 years ago by
prefix selects the install tree (SAGE_LOCAL) of sagethedistribution.
The vpath mechanism selects the build tree (SAGE_ROOT) of sagethedistribution. This includes the configuration and build artifacts such as src/build/...
comment:96 in reply to: ↑ 95 Changed 5 years ago by
Replying to mkoeppe:
prefix selects the install tree (SAGE_LOCAL) of sagethedistribution.
The vpath mechanism selects the build tree (SAGE_ROOT) of sagethedistribution. This includes the configuration and build artifacts such as src/build/...
This short reply is much better than the long explanation in the ticket description. It's all totally clear now.
comment:97 followup: ↓ 103 Changed 5 years ago by
 Description modified (diff)
Would the old way to build still work unamended, or BUILDDIR
will need to be created separately?
comment:98 followups: ↓ 99 ↓ 101 Changed 5 years ago by
I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).
comment:99 in reply to: ↑ 98 ; followups: ↓ 100 ↓ 102 Changed 5 years ago by
Replying to dimpase:
I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).
When making a VPATH build you typically specify the path to the build tree yourself, so you could do something like VPATH=builds/$(git symbolicref short HEAD)
. But indeed it might be nice to have a way to automate that (or even enable by default).
comment:100 in reply to: ↑ 99 Changed 5 years ago by
Replying to embray:
Replying to dimpase:
I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).
Maybe gitworktree is the right git functionality to use...
When making a VPATH build you typically specify the path to the build tree yourself, so you could do something like
VPATH=builds/$(git symbolicref short HEAD)
. But indeed it might be nice to have a way to automate that (or even enable by default).
comment:101 in reply to: ↑ 98 ; followup: ↓ 104 Changed 5 years ago by
Replying to dimpase:
I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).
It would be nice if the Sage version number that it displays at startup would show the git commit from which it is built. But this is entirely unrelated to the present ticket.
comment:102 in reply to: ↑ 99 Changed 5 years ago by
Replying to embray:
When making a VPATH build you typically specify the path to the build tree yourself, so you could do something like
VPATH=builds/$(git symbolicref short HEAD)
.
Just a quick note. When one says "VPATH", it refers to the source directory, not to the build directory. https://www.gnu.org/software/make/manual/html_node/GeneralSearch.html
comment:103 in reply to: ↑ 97 Changed 5 years ago by
Replying to dimpase:
Would the old way to build still work unamended, or
BUILDDIR
will need to be created separately?
Yes, with this patch one can still do nonVPATH builds, without any change.
comment:104 in reply to: ↑ 101 Changed 5 years ago by
Replying to mkoeppe:
Replying to dimpase:
I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).
It would be nice if the Sage version number that it displays at startup would show the git commit from which it is built. But this is entirely unrelated to the present ticket.
I suppose I just don't understand what happens to a VPATH build after the source tree it is built from is changed. Will it become broken, or not?
comment:105 Changed 5 years ago by
Are there any immediate issues left here beside comment:93?
comment:106 Changed 4 years ago by
 Status changed from needs_review to needs_work
 Work issues set to merge conflicts
comment:107 Changed 2 years ago by
I actually could really use this feature right now :)
mkoeppe, should I try to resolve conflicts here? Or do you have time to do so, so I can review afterwards and maybe fix comment:93?
comment:108 Changed 2 years ago by
I hope to get back to it within the next month, but not right now. It will likely interact with some of my buildrelated tickets that are waiting for review...
If you need this earlier, please feel free to work on it.
I don't think comment 93 should hold up this ticket. Improvements could always be done in a followup ticket.
comment:109 Changed 2 years ago by
 Milestone changed from sage8.2 to sage9.2
comment:110 Changed 22 months ago by
 Milestone changed from sage9.2 to sage9.3
comment:111 Changed 15 months ago by
 Milestone changed from sage9.3 to sage9.4
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
comment:112 Changed 10 months ago by
 Milestone changed from sage9.4 to sage9.5
Setting a new milestone for this ticket based on a cursory review.
comment:113 Changed 5 months ago by
 Milestone changed from sage9.5 to sage9.6
comment:114 Changed 3 months ago by
 Milestone changed from sage9.6 to sage9.7
New commits:
configure.ac: VPATH fixes