Opened 3 years ago

Last modified 18 months ago

#21469 needs_work enhancement

Enable VPATH builds (several independent build trees connected to one source tree)

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-8.2
Component: build: configure Keywords:
Cc: felixs, jdemeyer, fbissey, embray, leif, vbraun, dimpase, chapoton, jhpalmieri Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: N/A Work issues: merge conflicts
Branch: u/mkoeppe/vpath_build (Commits) Commit: 05f3a9107a55a6c308196c85a995f867b452fb9c
Dependencies: Stopgaps:

Description (last modified by dimpase)

Executive summary Currently, --prefix selects the install tree (SAGE_LOCAL) of sage-the-distribution. But often we want several builds out of (almost) the same source tree.

The vpath mechanism proposed here selects the build tree (SAGE_ROOT) of sage-the-distribution. This includes the configuration and build artefacts such as src/build/....


VPATH builds are a powerful feature of autotools (and other) build systems. This feature allows the developer to build a package in a separate (initially empty) build directory tree. The source directory tree is only read from (and could even be mounted read-only) and is therefore always clean.

This is a powerful feature for the developer because from the same source tree many different configurations can be built and tested, without having to go through "make distclean" and reconfiguration. The source tree can also be shared, for example using a networked file system between different hosts, running different architectures. Another modern use case involves VMs. For example, Docker allows to mount the source directory from the host in the VM (see #21474).

For Sage, "different configurations" could mean different architectures (via VMs), different sets of installed packages, Py2 vs Py3, etc.

This is how it is used.

 cd SAGE_ROOT
 autoreconf

and then

 mkdir BUILDDIR
 cd BUILDDIR
 SRCDIR/configure --srcdir=SRCDIR
 make

The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_VPATH. $SAGE_LOCAL defaults to BUILDDIR/local, but can of course be changed using configure --prefix.

The present patch changes the various places where $SAGE_ROOT is used. When "./configure" detects a VPATH build, it installs patched copies of sage, Makefile and build/make/install in BUILDDIR.

With the current set of patches, the compilation goes through and gives a working sage, except for the docbuild.

Change History (106)

comment:1 Changed 3 years ago by mkoeppe

  • Branch set to u/mkoeppe/vpath_build

comment:2 Changed 3 years ago by mkoeppe

  • Cc leif added
  • Commit set to ac27cdc02f685d79d817c71ebdcc1ee79196b4af

New commits:

ac27cdcconfigure.ac: VPATH fixes

comment:3 Changed 3 years ago by git

  • Commit changed from ac27cdc02f685d79d817c71ebdcc1ee79196b4af to e9b1c0fb2a378370b454757e397d20349323434f

Branch pushed to git repo; I updated commit sha1. New commits:

5698f8eDistinguish SAGE_ROOT and SAGE_SRC_ROOT for VPATH build
a98cfa2Use AC_CONFIG_COMMANDS
e9b1c0fSymlink build/pkgs (temporary solution)

comment:4 Changed 3 years ago by mkoeppe

  • Cc vbraun dimpase added
  • Status changed from new to needs_review

comment:5 Changed 3 years ago by git

  • Commit changed from e9b1c0fb2a378370b454757e397d20349323434f to e15ada69bb22124f463820e9dab5cc16348b3993

Branch pushed to git repo; I updated commit sha1. New commits:

e15ada6Define SAGE_SRC_ROOT for VPATH build

comment:6 Changed 3 years ago by git

  • Commit changed from e15ada69bb22124f463820e9dab5cc16348b3993 to 9323c011da3005c0beb46ba0da5a6d3dba13c9e7

Branch pushed to git repo; I updated commit sha1. New commits:

9323c01Use SAGE_SRC_ROOT in package scripts

comment:7 Changed 3 years ago by git

  • Commit changed from 9323c011da3005c0beb46ba0da5a6d3dba13c9e7 to 7a3913a64aa1d4310f741a3e5f898635187eb38f

Branch pushed to git repo; I updated commit sha1. New commits:

7a3913aMore SAGE_SRC_ROOT

comment:8 Changed 3 years ago by git

  • Commit changed from 7a3913a64aa1d4310f741a3e5f898635187eb38f to 4cc3954f1ae666aa0148d69256e59d9e17ee4258

Branch pushed to git repo; I updated commit sha1. New commits:

4cc3954More SAGE_SRC_ROOT

comment:9 Changed 3 years ago by mkoeppe

The current patch makes some high-level preparations for a VPATH build, in particular the package system.

"make" runs through until it gets to sagelib, which needs some more VPATH work (the current patch tries to get away by just making a symlink from $SAGE_SRC_ROOT/src to $SAGE_ROOT/src, which of course fails).

I've set this to 'needs review' in the hope of receiving some feedback regarding the approach of this patch.

comment:10 follow-ups: Changed 3 years ago by fbissey

I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH brings me.

This is most evident in the latest commit. As a distro maintainer of sage there are already build time things in env.py that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py and some in a new file sage_setup/build_env.py but that's marginal to what you are doing.

Other people may have a different opinion but there is no value in this for me so far. Possibly more work in fact.

comment:11 in reply to: ↑ 10 Changed 3 years ago by mkoeppe

Replying to fbissey:

I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH brings me.

A VPATH build makes it easy to build from the same source in many different configurations. Concretely, I've started to run some VMs (with Linux 64 bit, 32 bit) via Docker, which mount the Sage source directory from the host.

comment:12 Changed 3 years ago by git

  • Commit changed from 4cc3954f1ae666aa0148d69256e59d9e17ee4258 to 7bb87184fcd1a5a39914af65e12ce4d6ed30e775

Branch pushed to git repo; I updated commit sha1. New commits:

7bb8718Make sage script in build dir executable

comment:13 in reply to: ↑ 10 ; follow-up: Changed 3 years ago by mkoeppe

Replying to fbissey:

As a distro maintainer of sage there are already build time things in env.py that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py and some in a new file sage_setup/build_env.py but that's marginal to what you are doing.

Yes, this ticket does not address any of the issues that would arise in distribution packaging.

Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.

comment:14 follow-up: Changed 3 years ago by fbissey

I should apologize for being so negative. Currently configure is run by the initial call to make, it would be nice to get to the stage where we run configure first :)

Seriously, yes VPATH as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean or git clone is a good thing.

I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH work may help to clarify it in some areas. Let's try to minimise where it would mix things further.

comment:15 in reply to: ↑ 14 Changed 3 years ago by mkoeppe

Replying to fbissey:

I should apologize for being so negative.

No worries.

Currently configure is run by the initial call to make, it would be nice to get to the stage where we run configure first :)

That already works with this patch.

Seriously, yes VPATH as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean or git clone is a good thing.

I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH work may help to clarify it in some areas. Let's try to minimise where it would mix things further.

Sounds good. Yes, I'm hoping too that it helps clarify some things as a side-effect.

comment:16 in reply to: ↑ 13 ; follow-up: Changed 3 years ago by jdemeyer

Replying to mkoeppe:

Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.

I sort of agree with François.

Part of the problem is that Sage is a distribution (a collection of packages) as well as a Python library. The fact this this is currently mixed up in the sources (the sources of the distribution-part and the library-part are in one repo) makes it more difficult to understand what this ticket is about.

For the distribution part, there really is no such thing as a build directory. There is a a temporary directory in local/var/tmp/sage/build but I would not consider that a build directory in the usual sense.

For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

comment:17 Changed 3 years ago by jdemeyer

  • Status changed from needs_review to needs_info

I think the ticket description should have a lot more information about

  1. which problem that this ticket is trying to solve (the why).
  1. what the branch actually does (the what).

Note that this are 2 independent things (which are unfortunately often confused).

Last edited 3 years ago by jdemeyer (previous) (diff)

comment:18 Changed 3 years ago by mkoeppe

  • Description modified (diff)
  • Status changed from needs_info to needs_review

I've updated the description.

comment:19 in reply to: ↑ 16 ; follow-ups: Changed 3 years ago by mkoeppe

Replying to jdemeyer:

Replying to mkoeppe: Part of the problem is that Sage is a distribution (a collection of packages) as well as a Python library. The fact this this is currently mixed up in the sources (the sources of the distribution-part and the library-part are in one repo) makes it more difficult to understand what this ticket is about.

I've now clarified the goal in the description. I have not touched sagelib yet; but without VPATH work on sagelib, the distribution VPATH changes won't work. I could use help with the work on sagelib.

For the distribution part, there really is no such thing as a build directory. There is a a temporary directory in local/var/tmp/sage/build but I would not consider that a build directory in the usual sense.

Right, local/var/tmp/sage/build falls short of being a real build directory. because for example src/build/ has a lot of more generated stuff.

For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

Certainly there src/build/lib.macosx-10.9-x86_64-2.7 includes the platform name, but it does not account for other configuration differences that a VPATH build can take care of (neither does src/build/cythonized).

comment:20 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:21 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:22 in reply to: ↑ 19 ; follow-up: Changed 3 years ago by mkoeppe

Replying to jdemeyer:

For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

It seems that pip install keeps the source directory clean, building instead in a temporary directory. pip install also offers options --build to select a build directory (though it seems as if it does not work with all packages).

comment:23 in reply to: ↑ 22 Changed 3 years ago by mkoeppe

Replying to mkoeppe:

It seems that pip install keeps the source directory clean, building instead in a temporary directory. pip install also offers options --build to select a build directory (though it seems as if it does not work with all packages).

However, there are some pip issues: https://github.com/pypa/pip/issues/2060 https://github.com/pypa/pip/issues/2053 https://github.com/pypa/pip/issues/804

comment:24 in reply to: ↑ 19 Changed 3 years ago by mkoeppe

Replying to jdemeyer: For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

Also, in the traditional distutils ways (https://docs.python.org/2/install/), it appears that one can use --build-base to do VPATH builds.

python setup.py build --build-base=BUILDDIR
Last edited 3 years ago by mkoeppe (previous) (diff)

comment:25 Changed 3 years ago by mkoeppe

  • Dependencies set to #21480

This is now #21480.

comment:26 Changed 3 years ago by git

  • Commit changed from 7bb87184fcd1a5a39914af65e12ce4d6ed30e775 to 81f4306be63d6097c98c29301c6b21bc3a1f1e01

Branch pushed to git repo; I updated commit sha1. New commits:

81f4306Fix for unconfigured non-VPATH build

comment:27 Changed 3 years ago by mkoeppe

  • Milestone changed from sage-7.4 to sage-7.5

comment:28 Changed 3 years ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:29 Changed 22 months ago by git

  • Commit changed from 81f4306be63d6097c98c29301c6b21bc3a1f1e01 to 7e11d55e6549a678bc5a538d64353426a8b8a4b9

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7e11d55VPATH build

comment:30 Changed 22 months ago by mkoeppe

squashed into 1 commit to simplify rebasing.

comment:31 Changed 22 months ago by mkoeppe

  • Dependencies changed from #21480 to #21535
  • Milestone changed from sage-7.5 to sage-8.2

The next step towards VPATH builds should probably be #21535 (enabling src/setup.py build --build-base, to replace the hard-coded "build").

comment:32 Changed 22 months ago by git

  • Commit changed from 7e11d55e6549a678bc5a538d64353426a8b8a4b9 to 23cc689ccfd33fff4eeb05ee5dc6ebf76ff2db9c

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ca281d2VPATH build
ae7bdf7configure.ac: Quoting fix
23cc689src/Makefile.in: Use setup.py --build-base

comment:33 Changed 22 months ago by git

  • Commit changed from 23cc689ccfd33fff4eeb05ee5dc6ebf76ff2db9c to 72818e5229b573b625e4b9b181828ff7dcd22a11

Branch pushed to git repo; I updated commit sha1. New commits:

72818e5Some path fixes for VPATH

comment:34 Changed 22 months ago by git

  • Commit changed from 72818e5229b573b625e4b9b181828ff7dcd22a11 to d0366b0bd4506fe3e3378fc752f7df487f783365

Branch pushed to git repo; I updated commit sha1. New commits:

3ffb004VPATH fixes for src/bin and src/ext installs
d0366b0build/make/deps: Fix sage-env path

comment:35 Changed 22 months ago by mkoeppe

  • Cc chapoton jhpalmieri added
  • Dependencies #21535 deleted

Removed #21535 dependency. setup.py --build-base actually works already.

With the current status of the branch, make builds a working sage, only the docbuild does not go through yet.

comment:36 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:37 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:38 Changed 22 months ago by mkoeppe

  • Summary changed from Enable VPATH builds to Enable VPATH builds (several independent build trees connected to one source tree)

comment:39 Changed 22 months ago by jhpalmieri

On OS X, some directories are not being created:

$ mkdir BUILD
$ cd BUILD
$ /path/to/sage/configure --srcdir=/path/to/sage
...
/path/to/sage/configure: line 7017: build/make/Makefile: No such file or directory

If I create that directory, configure finishes but with warning messages like this

$ mkdir -p build/make
$ /path/to/sage/configure --srcdir=/path/to/sage
...
touch: /path/to/BUILD/src/bin/sage-env-config: No such file or directory
find: /path/to/BUILD/src/ext: No such file or directory
cat: /path/to/BUILD/build/make/deps: No such file or directory
...
$ make
make: *** No targets specified and no makefile found.  Stop.

comment:40 Changed 22 months ago by mkoeppe

Did you autoreconf?

comment:41 Changed 22 months ago by jhpalmieri

  • Description modified (diff)

No, that fixed it.

comment:42 follow-up: Changed 22 months ago by jhpalmieri

With the changes

  • src/bin/sage-env

    diff --git a/src/bin/sage-env b/src/bin/sage-env
    index dc2bd34cc0..8aa523d7ea 100644
    a b export SAGE_EXTCODE="$SAGE_SHARE/sage/ext" 
    313313export SAGE_SPKG_INST="$SAGE_LOCAL/var/lib/sage/installed"
    314314export SAGE_LOGS="$SAGE_ROOT/logs/pkgs"
    315315export SAGE_SRC="$SAGE_ROOT/src"
    316 export SAGE_DOC_SRC="$SAGE_SRC/doc"
     316export SAGE_DOC_SRC="$SAGE_SRC_ROOT/src/doc"
    317317export SAGE_DOC="$SAGE_SHARE/doc/sage"
    318318
    319319if [ -z "${SAGE_ORIG_PATH_SET}" ]; then
  • src/doc/common/conf.py

    diff --git a/src/doc/common/conf.py b/src/doc/common/conf.py
    index 2e115fab88..88f3fdfa4a 100644
    a b  
    11import sys, os, sphinx
    2 from sage.env import SAGE_DOC_SRC, SAGE_DOC, SAGE_SRC, THEBE_DIR
     2from sage.env import SAGE_DOC_SRC, SAGE_DOC, SAGE_SRC_ROOT, THEBE_DIR
    33from datetime import date
    44from six import iteritems
    55
    66# If your extensions are in another directory, add it here.
    7 sys.path.append(os.path.join(SAGE_SRC, "sage_setup", "docbuild", "ext"))
     7sys.path.append(os.path.join(SAGE_SRC_ROOT, "src", "sage_setup", "docbuild", "ext"))
    88
    99# General configuration
    1010# ---------------------

the documentation starts to build, but I eventually get an error:

[dochtml] [arithgrou] /path/to/SAGE_ROOT/src/doc/en/reference/arithgroup/sage/modular/arithgroup/arithgroup_element.rst:11: WARNING: error while formatting arguments for sage.modular.arithgroup.arithgroup_element.ArithmeticSubgroupElement.b: 'NoneType' object has no attribute 'find'

I can't figure out where this is coming from.

comment:43 Changed 22 months ago by git

  • Commit changed from d0366b0bd4506fe3e3378fc752f7df487f783365 to e7a78e038548b6a0d34cea75432379c61759875d

Branch pushed to git repo; I updated commit sha1. New commits:

db34265Merge tag '8.1.beta3' into t/21469/vpath_build
e7a78e0VPATH fix for docbuild

comment:44 Changed 22 months ago by mkoeppe

Thanks, I've added your changes as a commit.

comment:45 in reply to: ↑ 42 Changed 22 months ago by mkoeppe

Replying to jhpalmieri:

the documentation starts to build, but I eventually get an error:

[dochtml] [arithgrou] /path/to/SAGE_ROOT/src/doc/en/reference/arithgroup/sage/modular/arithgroup/arithgroup_element.rst:11: WARNING: error while formatting arguments for sage.modular.arithgroup.arithgroup_element.ArithmeticSubgroupElement.b: 'NoneType' object has no attribute 'find'

I get the same error

comment:46 Changed 22 months ago by jhpalmieri

One more change, to help make distclean work:

  • build/make/deps

    diff --git a/build/make/deps b/build/make/deps
    index 3128f4c656..03f2c8fff1 100644
    a b doc-pdf: $(DOC_DEPENDENCIES) 
    235235doc-clean: doc-src-clean doc-output-clean
    236236
    237237doc-src-clean:
    238        cd "$(SAGE_SRC)/doc" && $(MAKE) clean
     238       cd "$(SAGE_SRC_ROOT)/src/doc" && $(MAKE) clean
    239239
    240240doc-output-clean:
    241241       rm -rf "$(SAGE_SHARE)/doc/sage"

comment:47 Changed 22 months ago by git

  • Commit changed from e7a78e038548b6a0d34cea75432379c61759875d to 734fb4ba95dc62a7f6f10c30b6252fa309c6b7b0

Branch pushed to git repo; I updated commit sha1. New commits:

734fb4bVPATH fix for doc-src-clean

comment:48 Changed 22 months ago by mkoeppe

Perhaps the sphinx problem is related to failing source discovery for Cython modules:

sage: sage.rings.number_field.number_field_element_quadratic??
Error getting source: s must be a string
Type:            module
...

comment:49 Changed 22 months ago by jhpalmieri

Yes, that looks like it.

comment:50 follow-up: Changed 22 months ago by jhpalmieri

Okay, how about this:

  • src/sage/misc/sageinspect.py

    diff --git a/src/sage/misc/sageinspect.py b/src/sage/misc/sageinspect.py
    index f9dffc1ce3..3a286c7934 100644
    a b import tokenize 
    125125import types
    126126import re
    127127EMBEDDED_MODE = False
    128 from sage.env import SAGE_SRC
     128from sage.env import SAGE_SRC_ROOT
    129129
    130130def loadable_module_extension():
    131131    r"""
    def _extract_embedded_position(docstring): 
    222222        return None
    223223
    224224    raw_filename = res.group('FILENAME')
    225     filename = os.path.join(SAGE_SRC, raw_filename)
     225    filename = os.path.join(SAGE_SRC_ROOT, 'src', raw_filename)
    226226    if not os.path.isfile(filename):
    227227        from sage.misc.misc import SPYX_TMP
    228228        filename = os.path.join(SPYX_TMP, '_'.join(raw_filename.split('_')[:-1]), raw_filename)
    def __internal_tests(): 
    23082308
    23092309    Test _extract_embedded_position:
    23102310
    2311     We cannot test the filename since it depends on SAGE_SRC.
     2311    We cannot test the filename since it depends on SAGE_SRC_ROOT.
    23122312
    23132313    Make sure things work with no trailing newline::
    23142314

comment:51 Changed 22 months ago by jhpalmieri

Now the cross-references are wrong or missing:

[dochtml] OSError: [parallel ] /.../sage-8.1.beta3/src/doc/en/reference/parallel/index.rst:17: WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)

comment:52 Changed 22 months ago by jhpalmieri

Also, it seems that the docbuilding doesn't recognize when a document has been built already, so everything is rebuilt every time.

comment:53 Changed 22 months ago by git

  • Commit changed from 734fb4ba95dc62a7f6f10c30b6252fa309c6b7b0 to 8c801f7e36d7d1d680ee175f8a5eac40e118568d

Branch pushed to git repo; I updated commit sha1. New commits:

8c801f7VPATH: Set SAGE_SRC to actual source tree throughout

comment:54 in reply to: ↑ 50 Changed 22 months ago by mkoeppe

Replying to jhpalmieri:

Okay, how about this:

  • src/sage/misc/sageinspect.py

    diff --git a/src/sage/misc/sageinspect.py b/src/sage/misc/sageinspect.py
    index f9dffc1ce3..3a286c7934 100644
    a b def _extract_embedded_position(docstring): 
    222222        return None
    223223
    224224    raw_filename = res.group('FILENAME')
    225     filename = os.path.join(SAGE_SRC, raw_filename)
     225    filename = os.path.join(SAGE_SRC_ROOT, 'src', raw_filename)
    226226    if not os.path.isfile(filename):
    227227        from sage.misc.misc import SPYX_TMP
    228228        filename = os.path.join(SPYX_TMP, '_'.join(raw_filename.split('_')[:-1]), raw_filename)

Instead, I have changed SAGE_SRC to point to the actual source tree. This was inconsistent before.

comment:55 Changed 22 months ago by mkoeppe

I'm now testing the VPATH build using a separate user account that cannot write into the source tree. More changes to the docbuild are necessary to separate sources from built files:

[dochtml] IOError: [Errno 13] Permission denied: '/Users/mkoeppe/s/sage/sage-rebasing/vpath/B/../src/doc/en/reference/repl/sage/misc/trace.rst'

comment:56 Changed 22 months ago by jhpalmieri

By the way, I get warnings like

touch: /.../BUILDDIR/src/bin/sage-env-config: No such file or directory

I assume this isn't a big deal since you're planning on getting rid of this file. It also doesn't seem to interfere with the build or running Sage.

comment:57 Changed 22 months ago by mkoeppe

I don't plan to eliminate sage-env-config (perhaps you mean sage-arch-env, #23733)?

comment:58 Changed 22 months ago by jhpalmieri

Sorry, yes, that's what I meant, and that's what most of the warnings have been about.

comment:59 Changed 22 months ago by mkoeppe

The sage-arch-env message is harmless. But there's indeed also something wrong with sage-env-config too. Fix will come shortly.

comment:60 Changed 22 months ago by git

  • Commit changed from 8c801f7e36d7d1d680ee175f8a5eac40e118568d to 5530827f3266b9d43326ddba619bd3a757e0569d

Branch pushed to git repo; I updated commit sha1. New commits:

5530827sage-env-config: Handle special case in build/make/deps, rather than configure.ac

comment:61 Changed 22 months ago by jhpalmieri

I'm getting this build error:

make[2]: *** No rule to make target `/.../BUILDDIR/local/share/sage/ext/doctest/invalid/syntax_error.tachyon', needed by `sagelib'.  Stop.

comment:62 Changed 22 months ago by git

  • Commit changed from 5530827f3266b9d43326ddba619bd3a757e0569d to 64515c1bc26cf7d4cb6ac1665431afa4b87092f6

Branch pushed to git repo; I updated commit sha1. New commits:

64515c1Fix last change

comment:63 Changed 22 months ago by jhpalmieri

The make target "sage-starts" fails for me. This fixes it; is it the right way to fix it?

  • src/bin/sage-starts

    diff --git a/src/bin/sage-starts b/src/bin/sage-starts
    index 85cbae67e5..498c930810 100755
    a b echo "[`date +'%Y-%m-%d %H:%M:%S'`] `cat VERSION.txt 2>/dev/null`" | tee -a logs 
    1414# sysadmin.  We use --nodotsage so we don't force a .sage directory
    1515# in the sysadmin's HOME directory.
    1616cmd='sage.all._write_started_file(); print "Yes, Sage starts."'
    17 build/bin/sage-logger "./sage --nodotsage -c '$cmd'" logs/start.log
     17"$SAGE_SRC_ROOT"/build/bin/sage-logger "./sage --nodotsage -c '$cmd'" logs/start.log
    1818
    1919if [ $? -ne 0 ]; then
    2020    echo >&2 "Sage failed to start up."

comment:64 follow-up: Changed 22 months ago by jhpalmieri

With that change, make ptestlong fails:

Testing that Sage starts...
[2017-08-30 09:19:29] 
This looks like the first time you are running Sage.
Cleaning up, do not interrupt this.
Done cleaning.
Yes, Sage starts.

real	52m33.269s
user	191m2.435s
sys	19m38.034s
Sage build/upgrade complete!

To install small scripts to directly run Sage's versions of GAP,
the PARI/GP interpreter, Maxima, or Singular etc. (by typing e.g.
just 'gap' or 'gp') into a standard 'bin' directory, start Sage
by typing 'sage' (or './sage') and enter something like

    install_scripts('/usr/local/bin')

at the Sage command prompt ('sage:').

/.../SAGE_SRC_ROOT/sage -t -p --all --long --logfile=logs/ptestlong.log
************************************************************************
It seems that you are attempting to run Sage from an unpacked source
tarball, but you have not compiled it yet (or maybe the build has not
finished). You should run `make` in the Sage root directory first.
If you did not intend to build Sage from source, you should download
a binary tarball instead. Read README.txt for more information.
************************************************************************
make: *** [ptestlong] Error 1

I think that the make target

TESTALL = $(top_srcdir)/sage -t --all

needs to be changed so that it runs SAGE_ROOT/sage instead of SAGE_SRC_ROOT/sage.

./sage -t -all also fails:

$ ./sage -t --all
/.../SAGE_SRC_ROOT/src/bin/sage-env: line 527: /.../BUILDDIR/src/bin/sage-arch-env: No such file or directory
Traceback (most recent call last):
  File "/.../SAGE_SRC_ROOT/src/bin/sage-runtests", line 97, in <module>
    DC = DocTestController(options, args)
  File "/.../BUILDDIR/local/lib/python2.7/site-packages/sage/doctest/control.py", line 334, in __init__
    for pkg in list_packages('optional', local=True).values():
  File "/.../BUILDDIR/local/lib/python2.7/site-packages/sage/misc/package.py", line 230, in list_packages
    for p in os.listdir(SAGE_PKGS):
OSError: [Errno 2] No such file or directory: '/.../BUILDDIR/build/pkgs'

This can almost be fixed by

  • src/sage/env.py

    diff --git a/src/sage/env.py b/src/sage/env.py
    index 33cd754a06..f91c70ba7d 100644
    a b _add_variable_or_fallback('SAGE_LIB', SITE_PACKAGES[0]) 
    111111_add_variable_or_fallback('SAGE_CYTHONIZED', opj('$SAGE_ROOT', 'src', 'build', 'cythonized'))
    112112
    113113# Used by sage/misc/package.py.  Should be SAGE_SRC_ROOT in VPATH.
    114 _add_variable_or_fallback('SAGE_PKGS', opj('$SAGE_ROOT', 'build', 'pkgs'))
     114_add_variable_or_fallback('SAGE_PKGS', opj('$SAGE_SRC_ROOT', 'build', 'pkgs'))
    115115
    116116
    117117_add_variable_or_fallback('SAGE_EXTCODE',    opj('$SAGE_SHARE', 'sage', 'ext'))

but there is a bug in the function _add_variable_or_fallback in how it parses variable names prefixed with dollar signs. I'll open a new ticket for that.

comment:65 Changed 22 months ago by jhpalmieri

If I hack the top-level Makefile so that make ptestlong works, then I only get test failures in sage/tests/cmdline.py.

comment:66 in reply to: ↑ 64 Changed 22 months ago by jhpalmieri

Replying to jhpalmieri:

there is a bug in the function _add_variable_or_fallback in how it parses variable names prefixed with dollar signs. I'll open a new ticket for that.

See #23758.

comment:67 Changed 22 months ago by git

  • Commit changed from 64515c1bc26cf7d4cb6ac1665431afa4b87092f6 to 3e36b75f51413973d847994aed067dbe7bfef822

Branch pushed to git repo; I updated commit sha1. New commits:

bc6263cVPATH fix for SAGE_PKGS
f9c25bcVPATH fix for test makefile targets
ecd2f9ddoc-src-clean: Clean in source directory until docbuild is fully VPATHified
60927e5trac 23758: in env.py, _add_variable_or_fallback should depend less
c23cc1ctrac 23758: restore "import os"
343d685trac 23758: add a doctest
3e36b75Merge remote-tracking branch 'trac/u/jhpalmieri/variable_replacement' into t/21469/vpath_build

comment:68 follow-up: Changed 22 months ago by mkoeppe

I'm still getting docbuild errors of the type WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)

comment:69 in reply to: ↑ 68 Changed 22 months ago by dimpase

Replying to mkoeppe:

I'm still getting docbuild errors of the type WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)

This is a general problem, nothing to worry in the context of this ticket.

By the way you might like to use here the upgraded sphinx on #23023, which will most probably be in the next beta.

comment:70 follow-ups: Changed 22 months ago by jhpalmieri

I agree that the docbuild warnings are not new. (My issue in comment:51 was that these were errors, causing docbuilding to actually fail. I am now able to build the docs successfully.)

I am still having the problem in comment:63 with sage-starts.

Last edited 22 months ago by jhpalmieri (previous) (diff)

comment:71 in reply to: ↑ 70 ; follow-up: Changed 22 months ago by mkoeppe

Replying to jhpalmieri:

I agree that the docbuild warnings are not new. (My issue in comment:51 was that these were errors, causing docbuilding to actually fail. I am now able to build the docs successfully.)

Unfortunately these warnings get upgraded to errors like this:

[dochtml] OSError: [parallel ] /Users/mkoeppe/s/sage/sage-rebasing/vpath/src/doc/en/reference/parallel/index.rst:17: WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)

which stops this build for me.

I'll try with #23023 as suggested by Dima.

comment:72 in reply to: ↑ 71 Changed 22 months ago by jhpalmieri

Replying to mkoeppe:

Replying to jhpalmieri:

I agree that the docbuild warnings are not new. (My issue in comment:51 was that these were errors, causing docbuilding to actually fail. I am now able to build the docs successfully.)

Unfortunately these warnings get upgraded to errors like this:

[dochtml] OSError: [parallel ] /Users/mkoeppe/s/sage/sage-rebasing/vpath/src/doc/en/reference/parallel/index.rst:17: WARNING: undefined label: sage.interfaces.psage (if the link has no caption the label must precede a section header)

which stops this build for me.

Your change in comment:53 fixed that issue for me. (At least I think that's what fixed it.)

comment:73 Changed 22 months ago by git

  • Commit changed from 3e36b75f51413973d847994aed067dbe7bfef822 to 754ec81627aa01a9bfa5646e946bdea02fe220ff

Branch pushed to git repo; I updated commit sha1. New commits:

754ec81sage-starts: Fix sage-logger invocation for VPATH build

comment:74 in reply to: ↑ 70 Changed 22 months ago by mkoeppe

Replying to jhpalmieri:

I am still having the problem in comment:63 with sage-starts.

I agree with your patch and have added a commit; but see #23769.

comment:75 Changed 22 months ago by git

  • Commit changed from 754ec81627aa01a9bfa5646e946bdea02fe220ff to 0c357dd5da4d70a8a3f3ca96984359a563686307

Branch pushed to git repo; I updated commit sha1. New commits:

0c357ddSimplify VPATH patch

comment:76 Changed 22 months ago by mkoeppe

  • Authors changed from Matthias Koeppe to Matthias Koeppe, John H. Palmieri
  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:77 Changed 22 months ago by mkoeppe

I think this ticket is ready for broader review.

comment:78 Changed 22 months ago by jhpalmieri

  • Authors changed from Matthias Koeppe, John H. Palmieri to Matthias Koeppe
  • Description modified (diff)

make distclean fails because make sagelib-clean fails:

cd "/.../SAGE_SRC_ROOT/src" && make -j6 clean
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: *** No rule to make target `clean'.  Stop.
make[1]: *** [sagelib-clean] Error 2

I guess it should cd to /.../BUILDDIR/src instead.

comment:79 Changed 22 months ago by git

  • Commit changed from 0c357dd5da4d70a8a3f3ca96984359a563686307 to d32598ee2cf41e37af931b0842e520eac43b9319

Branch pushed to git repo; I updated commit sha1. New commits:

d32598eVPATH fix for sagelib-clean

comment:80 Changed 22 months ago by git

  • Commit changed from d32598ee2cf41e37af931b0842e520eac43b9319 to 06f7d1d6914fe55bf1d3eedab04a9aa13f260dfc

Branch pushed to git repo; I updated commit sha1. New commits:

06f7d1dFix build/make/install change

comment:81 Changed 22 months ago by mkoeppe

I tried again from a distcleaned tree. Now I'm running into:

[sagelib-8.1.beta3] Generating auto-generated sources
[sagelib-8.1.beta3] Building interpreters for fast_callable
[sagelib-8.1.beta3] -> First build of interpreters
[sagelib-8.1.beta3] error: [Errno 2] No such file or directory: 'sage/ext/interpreters/interp_cc.c'

comment:82 follow-up: Changed 22 months ago by jhpalmieri

I haven't seen that after running make distclean in both the original source directory and the build directory, but I could imagine it would happen if you don't have write access to the original source directory, since the interpreters directory and its contents are autogenerated (see src/sage_setup/autogen). Where should those files be created?

comment:83 in reply to: ↑ 82 ; follow-ups: Changed 22 months ago by mkoeppe

Replying to jhpalmieri:

I haven't seen that after running make distclean in both the original source directory and the build directory, but I could imagine it would happen if you don't have write access to the original source directory, since the interpreters directory and its contents are autogenerated (see src/sage_setup/autogen).

Yes, this is how I am testing it (except I made src/doc writable).

Something in the code generator is missing error checking. I suspect it's coming fromsrc/sage_setup/autogen/interpreters/__init__.py, which has some complicated exception handling.

Where should those files be created?

They should be created in SAGE_ROOT/src/sage/ext.

comment:84 Changed 22 months ago by mkoeppe

I think we need a variable for SAGE_ROOT/src to make this patch cleaner. Suggestions for the variable name?

comment:85 follow-up: Changed 22 months ago by mkoeppe

Perhaps should:

  • rename SAGE_SRC_ROOT to SAGE_VPATH
  • rename SAGE_SRC to SAGE_VPATH_SRC = SAGE_VPATH/src (and change many places that now use SAGE_SRC)
  • create SAGE_SRC = SAGE_ROOT/src
  • rename SAGE_DOC_SRC to SAGE_VPATH_SRC_DOC = SAGE_VPATH/src/doc
  • create SAGE_SRC_DOC = SAGE_ROOT/src/doc

comment:86 in reply to: ↑ 85 Changed 22 months ago by jhpalmieri

Replying to mkoeppe:

  • create SAGE_SRC = SAGE_ROOT/src

How about SAGE_ROOT_SRC? (only if you're changing SAGE_SRC_ROOT)

  • rename SAGE_DOC_SRC to SAGE_VPATH_SRC_DOC = SAGE_VPATH/src/doc
  • create SAGE_SRC_DOC = SAGE_ROOT/src/doc

Or SAGE_ROOT_SRC_DOC?

comment:87 Changed 22 months ago by mkoeppe

Thanks, these are good suggestions.

comment:88 Changed 22 months ago by git

  • Commit changed from 06f7d1d6914fe55bf1d3eedab04a9aa13f260dfc to 05f3a9107a55a6c308196c85a995f867b452fb9c

Branch pushed to git repo; I updated commit sha1. New commits:

05f3a91Rename SAGE_SRC_ROOT -> SAGE_VPATH

comment:89 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:90 in reply to: ↑ 83 ; follow-up: Changed 22 months ago by jhpalmieri

Replying to mkoeppe:

Replying to jhpalmieri:

I haven't seen that after running make distclean in both the original source directory and the build directory, but I could imagine it would happen if you don't have write access to the original source directory, since the interpreters directory and its contents are autogenerated (see src/sage_setup/autogen).

Yes, this is how I am testing it (except I made src/doc writable).

Something in the code generator is missing error checking. I suspect it's coming fromsrc/sage_setup/autogen/interpreters/__init__.py, which has some complicated exception handling.

Where should those files be created?

They should be created in SAGE_ROOT/src/sage/ext.

It looks like they are actually created in SAGE_SRC_ROOT=SAGE_VPATH, then copied to SAGE_ROOT/....

comment:91 in reply to: ↑ 83 Changed 22 months ago by mkoeppe

Replying to mkoeppe:

Something in the code generator is missing error checking. I suspect it's coming fromsrc/sage_setup/autogen/interpreters/__init__.py, which has some complicated exception handling.

See #23774.

comment:92 in reply to: ↑ 90 Changed 22 months ago by mkoeppe

Replying to jhpalmieri:

Where should those files be created?

They should be created in SAGE_ROOT/src/sage/ext.

It looks like they are actually created in SAGE_SRC_ROOT=SAGE_VPATH...

Yes, see src/sage_setup/autogen/interpreters/__main__.py.

But I think I will defer full VPATHification of this and of the docbuild to follow-up tickets.

comment:93 Changed 22 months ago by embray

I wonder if instead of

+dnl Set up files for VPATH build
+AS_IF([test x$srcdir != x.],
+    [AC_CONFIG_COMMANDS([vpath],
+	[
+	    SAGE_ROOT=${ac_abs_top_builddir}
+	    SAGE_VPATH=${ac_abs_top_srcdir}
+	    # Assume that $srcdir is readonly, so symlink existing upstream files into build directory upstream
+	    # If new packages are downloaded, they end up in the build directory upstream.
+	    mkdir -p upstream
+	    for a in $SAGE_VPATH/upstream/* ; do
+		if test -f "$a" -a ! -f "upstream/`basename $a`" ; then
+		    ln -s "$a" upstream/
+		fi
+	    done
+	    cat > sage <<EOF

we could simply modify whatever code looks for the upstream/ directory (presumably mostly in sage-spkg) to explicitly look for it under $SAGE_ROOT. In other words I don't see why making symlinks in the VPATH sources are necessary if we already know where to find the upstream/ directory.

comment:94 Changed 22 months ago by jdemeyer

I'm seeing a lot of activity on this ticket that I haven't followed.

Anyway, I still don't quite understand what a VPATH build even means in the context of Sage. I really think that this should be explained better, and how this differs from ./configure --prefix=...

comment:95 follow-up: Changed 22 months ago by mkoeppe

--prefix selects the install tree (SAGE_LOCAL) of sage-the-distribution.

The vpath mechanism selects the build tree (SAGE_ROOT) of sage-the-distribution. This includes the configuration and build artifacts such as src/build/...

comment:96 in reply to: ↑ 95 Changed 22 months ago by jdemeyer

Replying to mkoeppe:

--prefix selects the install tree (SAGE_LOCAL) of sage-the-distribution.

The vpath mechanism selects the build tree (SAGE_ROOT) of sage-the-distribution. This includes the configuration and build artifacts such as src/build/...

This short reply is much better than the long explanation in the ticket description. It's all totally clear now.

comment:97 follow-up: Changed 22 months ago by dimpase

  • Description modified (diff)

Would the old way to build still work unamended, or BUILDDIR will need to be created separately?

comment:98 follow-ups: Changed 22 months ago by dimpase

I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).

comment:99 in reply to: ↑ 98 ; follow-ups: Changed 22 months ago by embray

Replying to dimpase:

I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).

When making a VPATH build you typically specify the path to the build tree yourself, so you could do something like VPATH=builds/$(git symbolic-ref --short HEAD). But indeed it might be nice to have a way to automate that (or even enable by default).

comment:100 in reply to: ↑ 99 Changed 22 months ago by dimpase

Replying to embray:

Replying to dimpase:

I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).

Maybe git-worktree is the right git functionality to use...

When making a VPATH build you typically specify the path to the build tree yourself, so you could do something like VPATH=builds/$(git symbolic-ref --short HEAD). But indeed it might be nice to have a way to automate that (or even enable by default).

comment:101 in reply to: ↑ 98 ; follow-up: Changed 22 months ago by mkoeppe

Replying to dimpase:

I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).

It would be nice if the Sage version number that it displays at startup would show the git commit from which it is built. But this is entirely unrelated to the present ticket.

comment:102 in reply to: ↑ 99 Changed 22 months ago by mkoeppe

Replying to embray:

When making a VPATH build you typically specify the path to the build tree yourself, so you could do something like VPATH=builds/$(git symbolic-ref --short HEAD).

Just a quick note. When one says "VPATH", it refers to the source directory, not to the build directory. https://www.gnu.org/software/make/manual/html_node/General-Search.html

comment:103 in reply to: ↑ 97 Changed 22 months ago by mkoeppe

Replying to dimpase:

Would the old way to build still work unamended, or BUILDDIR will need to be created separately?

Yes, with this patch one can still do non-VPATH builds, without any change.

comment:104 in reply to: ↑ 101 Changed 22 months ago by dimpase

Replying to mkoeppe:

Replying to dimpase:

I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).

It would be nice if the Sage version number that it displays at startup would show the git commit from which it is built. But this is entirely unrelated to the present ticket.

I suppose I just don't understand what happens to a VPATH build after the source tree it is built from is changed. Will it become broken, or not?

comment:105 Changed 21 months ago by jhpalmieri

Are there any immediate issues left here beside comment:93?

comment:106 Changed 18 months ago by saraedum

  • Status changed from needs_review to needs_work
  • Work issues set to merge conflicts
Note: See TracTickets for help on using tickets.