Make sagelib setup.py self-contained and independent of SAGE_ROOT

[mkoeppe]

This ticket changes the build process of sagelib in the following way:

• src/Makefile delegates ALL building to src/setup.py
• src/setup.py no longer depends on environment variables $SAGE_ROOT, $SAGE_SRC, $SAGE_DOC_SRC etc. (to demonstrate this, Makefile poisons these environment variables). It still depends on $SAGE_LOCAL and environment variables that point below it.

This ticket is meant as:

• preparation for VPATH builds of sage-the-distribution (#21469)
• working towards the goal of making sagelib pip-installable -- see #21507 for the eventual goal of having sagelib on PyPI
• making the flow of directory information at build time clearer for developers

More specifically, the goal of this ticket is that only SAGE_LOCAL needs to be set when the user does 'pip install' of the sagelib. (This ticket almost achieves this, except it also needs SAGE_PKGS and SAGE_CYTHONIZED to be set. The hope is that #20382 and other future tickets will develop better mechanisms to communicate package and directory information to the build.)

. . . . . . .

Some possibly useful information:

• Documentation on distutils (​https://docs.python.org/2/install/), describing use of --build-base to do VPATH builds.
• pip install keeps the source directory clean, building instead in a temporary directory, by copying the sources. pip install also offers options --build to select a build directory, but there are some pip issues: ​2060, ​2053, ​804 that affect this
• #14807 has some tricks to making VPATH builds work without copying all python source files. But it uses automake instead of setup.sh; we will not do this in our ticket.

configure tarball: ​http://sage.ugent.be/www/jdemeyer/sage/configure-185.tar.gz

[fbissey]

Am interesting issue is that technically cython_debug has to be installed somewhere (preferably somewhere standard) to be accessible at runtime separately from the source.

I do something in sage-on-gentoo but that's not really satisfactory.

[leif]

Replying to mkoeppe:

In preparation for VPATH builds of sage-the-distribution (#21469), let's keep src/ clean by using setup.py --build-base=$SAGE_ROOT/var/tmp/sage/build/sagelib

Please use --build-base=$SAGE_BUILD_DIR/sagelib.

[leif]

... and sagelib perhaps with a version suffix.

[mkoeppe]

Replying to leif:

Replying to mkoeppe:

In preparation for VPATH builds of sage-the-distribution (#21469), let's keep src/ clean by using setup.py --build-base=$SAGE_ROOT/var/tmp/sage/build/sagelib

Please use --build-base=$SAGE_BUILD_DIR/sagelib.

Yes, the plan *after* #21469 is to use the Sage builddir -- not just for sagelib, but also for other packages.

*Before* #21469 is merged, I want to use $SAGE_ROOT/var/tmp/sage/build/sagelib to match what other packages do.

[mkoeppe]

Ah, I see what you meant, now I've found $SAGE_BUILD_DIR. Changed description accordingly.

[leif]

Replying to mkoeppe:

Replying to leif:

Replying to mkoeppe:

In preparation for VPATH builds of sage-the-distribution (#21469), let's keep src/ clean by using setup.py --build-base=$SAGE_ROOT/var/tmp/sage/build/sagelib

Please use --build-base=$SAGE_BUILD_DIR/sagelib.

Yes, the plan *after* #21469 is to use the Sage builddir -- not just for sagelib, but also for other packages.

??? SAGE_BUILD_DIR exists since years already... (Its default is $SAGE_ROOT/var/tmp/sage/build/.)

*Before* #21469 is merged, I want to use $SAGE_ROOT/var/tmp/sage/build/sagelib to match what other packages do.

[mkoeppe]

Replying to leif:

??? SAGE_BUILD_DIR exists since years already... (Its default is $SAGE_ROOT/var/tmp/sage/build/.)

Yes, thanks, see my other comment.

[leif]

Ah ok, race condition.

[mkoeppe]

By the way, help with implementing this change would be appreciated. I haven't looked at the sagelib build system at all so far.

[mkoeppe]

New commits:

​e399bf4

First, wishful step

[jdemeyer]

Really sagelib-$SAGE_VERSION? Do you want to fill up everybody's hard disks with Sage build directories? Not to mention that this would require to rebuild everything whenever the Sage version changes.

[mkoeppe]

Fine with me without $SAGE_VERSION; I was just following leif's suggestion in comment 3.

[felixs]

i understand that SAGE_BUILD_DIR is the srcdir for toplevel configure.

as long as sagelib is rooted in $(toplevel)/src, it might be less confusing to choose $SAGE_BUILD_DIR/src (not $SAGE_BUILD_DIR/sagelib) as the builddir for sagelib.

(future: replace src by sagelib, but on both ends)

i dont know exactly how the approach using setup.py will emulate VPATH builds. i think it should imitate "what automake would do", where applicable.

[mkoeppe]

Replying to felixs:

i understand that SAGE_BUILD_DIR is the srcdir for toplevel configure.

No, it's $SAGE_ROOT/var/tmp/sage/build/

[felixs]

ok, nevermind (i meant to write "SAGE_BUILD_DIR is the *builddir* for toplevel"). that does not seem to be the case either.

still i am wondering why "src" does not (simply) translate to "src". (sure, i am slightly autotools biased).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f3f275e	Put .cython_version next to cythonized files
​94a58f1	Make SAGE_BUILD_DIR variable available in sage-env, not just in sage-spkg
​a73fa06	Use SAGE_BUILD_DIR

[mkoeppe]

Here's a first version for review. It seems to work for me.

There are still some to-do items (see comments in src/Makefile):

• I am now using --build-base, but setup.sh also depends on SAGE_CYTHONIZED, defined in src/sage/env.py.

I think it would be better if setup.sh instead inferred that location from the build-base that was passed to it.

However, setup.sh already does a lot of stuff depending on SAGE_CYTHONIZED before distutils.core.setup is even called. Can this be fixed?

I think I could use some help from the Python experts in the cc list of this ticket on this.

• sage/libs/pari/auto_gen.pxi and sage/ext/interpreters/__init__.py still need to be taken care of in preparation for the VPATH build.

[jdemeyer]

Thinking about it more, I disagree with building in $SAGE_BUILD_DIR/sagelib by default.

$SAGE_LOCAL (which contains $SAGE_BUILD_DIR) is meant as install directory, not as build directory.

Let's keep $SAGE_LOCAL to be exactly the install directory and nothing else (*).

I do agree with making the build directory configurable for VPATH builds. For typical packages however, if you do not do a VPATH build, the build directory is the same as the source directory. Sage should follow the same model. This means that the build directory should be $SAGE_SRC by default.

(*) One could argue that $SAGE_BUILD_DIR is currently used as build directory for packages. That is true, but they are only used temporarily, they are not meant to actually store stuff. So this isn't so bad.

[felixs]

Let's keep $SAGE_LOCAL to be exactly the install directory and nothing else (*).

this is part of the problem with the current "build system". SAGE_LOCAL is *not* the install directory. it is where stuff ends up during the build process.

outside sage, an install directory is typically what you pass to --prefix, and where stuff is put into by "make install". you do that (if you are a sysadmin), after assuring that "make check" passes...

yes i know why SAGE_LOCAL exists, but it should be clear that it's not necessary and that it only has "evolved" that way. having simplified things in the past, now it is falling on your feet.

note how i tried to fix/work\ around that in my attempt to autotoolize sage (both the library and the distribution)... my point: the best approach will be to *not use SAGE_LOCAL* at all. here's the chance to cleanup sagelib, as a start.

that said: keep up the good and interesting work @mkoeppe. i hope i will have some more time later this year ...

[jdemeyer]

Replying to felixs:

this is part of the problem with the current "build system". SAGE_LOCAL is *not* the install directory.

Why do you think that $SAGE_LOCAL is not the install directory? It is the directory where everything is installed, which by definition makes it the install directory. The fact that Sage does not (yet) support --prefix doesn't change this fact.

the best approach will be to *not use SAGE_LOCAL* at all.

I like to know why you think that.

that said: keep up the good and interesting work @mkoeppe. i hope i will have some more time later this year ...

To be clear: I didn't say that this branch needs to be thrown out. I am just saying: keep the build directory configurable but keep the default what it currently is.

[felixs]

Why do you think that $SAGE_LOCAL is not the install directory?

you wrote it. "--prefix" is not implemented/supported. (but it should be). there's no way to really "install" stuff in the usual sense. e.g., there is no way to *install stuff after make check has passed*. (i don't know if toplevel make check is currently implemented at all, just a thought).

where everything is installed, which by definition makes it the install directory

actually nothing gets installed. what is done is mostly overhead. working around the fact that some packages don't work right after the build alone. when i was done with the "package content lists" for spkg-install, i noticed that it was a huge waste of time... don't repeat that, better just skip the "install" step.

I like to know why you think that [SAGE_LOCAL should not be used].

every instance/use of SAGE_LOCAL breaks sagelib on (lets call it) foreign distros a bit. that's not helpful. it will as well interfere with any attempt on rewriting sage-the-distribution (be it autotools based, or pip or ebuild). the autotools approach (not a necessary step, but an example) provides a transition path to anything...

sage-the-distribution is a platform for sage (core) development. no more, no less. other platforms will come and go. what is needed is sagelib without the dependency on this (and on SAGE_LOCAL). why: because developers should be able to use sagelib and develop sage extensions on *their own platforms*, with *their own* tools and *their own* review policies.

so: please embrace contributions that reduce the use of SAGE_LOCAL.

(yes, its getting off-topic. but i hope, this answers the question.)

[mkoeppe]

Replying to jdemeyer:

Thinking about it more, I disagree with building in $SAGE_BUILD_DIR/sagelib by default.

$SAGE_LOCAL (which contains $SAGE_BUILD_DIR) is meant as install directory, not as build directory.

Let's keep $SAGE_LOCAL to be exactly the install directory and nothing else (*).

I agree with you; see #21479.

[mkoeppe]

Replying to felixs:

outside sage, an install directory is typically what you pass to --prefix, and where stuff is put into by "make install". you do that (if you are a sysadmin), after assuring that "make check" passes...

See #21479 for a discussion of --prefix and make vs. make install.

But let's keep the discussion of the present ticket focused on the task at hand.

[mkoeppe]

Replying to jdemeyer:

I do agree with making the build directory configurable for VPATH builds. For typical packages however, if you do not do a VPATH build, the build directory is the same as the source directory. Sage should follow the same model. This means that the build directory should be $SAGE_SRC by default.

I agree on this as well. #21469 (VPATH for distro) will do that.

In this ticket, I first want to make the --build-base work. In particular, get rid of SAGE_CYTHONIZED.

This also works towards the eventual goal of making 'sagelib' pip-installable.

The actual location used is a detail that will be easy to change later.

(*) One could argue that $SAGE_BUILD_DIR is currently used as build directory for packages. That is true, but they are only used temporarily, they are not meant to actually store stuff. So this isn't so bad.

Yes, the current patch follows this practice. But this is temporary until #21469 is done.

[jdemeyer]

Replying to felixs:

Why do you think that $SAGE_LOCAL is not the install directory?

you wrote it. "--prefix" is not implemented/supported.

The fact that the installation process does not implement ./configure --prefix does not mean that it's not an installation process...

every instance/use of SAGE_LOCAL breaks sagelib

But why?

Of course, it's all a matter of definition. I consider the process of copying files to $SAGE_LOCAL an "installation" and you do not (for reasons which are still unclear to me). I think things will become much simpler for you if you accept the fact that $SAGE_LOCAL is the installation directory and that copying files to $SAGE_LOCAL is an installation.

but i hope, this answers the question

I absolutely does not. You are just saying "it breaks stuff" but not explaining why.

[dimpase]

Replying to jdemeyer:

Replying to felixs:

[...]

but i hope, this answers the question

I absolutely does not. You are just saying "it breaks stuff" but not explaining why.

One reason why is that in many deployment scenarios you will want to test what you have built before installing it. And Sage does not support this.

[felixs]

indeed. in "build"-"check"-"install" there is not much room for definitions of "build" and "install".

this is drifting towards the question of whether or not sage should stick to common practices and terminology. you know i think it should.

apologies for being off-topic again, this may be more related to #15105 -- there is no ticket for just "practices and terminology".

[jdemeyer]

Replying to dimpase:

One reason why is that in many deployment scenarios you will want to test what you have built before installing it. And Sage does not support this.

How does this relate to the existence of $SAGE_LOCAL?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a854831

Put setup.sh in charge of all sagelib building

[jdemeyer]

Replying to felixs:

this is drifting towards the question of whether or not sage should stick to common practices and terminology. you know i think it should.

If those "common" practices and terminology make sense, of course it should.

Anyway, I am completely not following what you are trying to say. I feel that you are always wandering around my questions instead of answering them.

[mkoeppe]

I have created #21495 as a place for such discussions.

[dimpase]

Replying to jdemeyer:

Replying to dimpase:

One reason why is that in many deployment scenarios you will want to test what you have built before installing it. And Sage does not support this.

How does this relate to the existence of $SAGE_LOCAL?

$SAGE_LOCAL is fine as long as you can also install things from there somewhere else. Currently you cannot do this in a proper way (yes, you can certainly create symbolic links etc, but this is often not enough).

[jdemeyer]

Replying to dimpase:

$SAGE_LOCAL is fine as long as you can also install things from there somewhere else. Currently you cannot do this in a proper way

Of course you can:

$ cd $package_source
$ sage --sh
$ ./configure --prefix="$SAGE_LOCAL"
$ make install

I have done this many times as first step of testing a new package or a package upgrade.

[mkoeppe]

If I understand Dima correctly, he wants to install "from" $SAGE_LOCAL, not "to" $SAGE_LOCAL.

But really this discussion should go to #21495. Thanks!

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1a027d5

Have setup.sh derive SAGE_CYTHONIZED from --build-base

[mkoeppe]

Jeroen, how does this look to you now? (I haven't changed $SAGE_BUILD_DIR yet, but will in a moment.)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f6555ce

Fix last change - handle --build-base earlier

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​853a1aa	Make src/Makefile a generated file
​1931b43	Define SAGE_PKGS in sage.env, not sage.misc.package
​c1dc4f8	Set SAGE_CYTHONIZED earlier
​d829eb4	Build in $(abs_builddir)/build-sagelib. Poison directory env variables

[mkoeppe]

I've changed the description of this ticket, as it has changed its focus. Ready for review.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​57db699	Revert "Make SAGE_BUILD_DIR variable available in sage-env, not just in sage-spkg"
​ef0919c	Fixup for SAGE_CYTHONIZED

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8ccd440	Add src/generate_py_source.mk
​bd670af	Ignore generated files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​751bd0f

Reword TODO item

[jdemeyer]

I would prefer to do this ticket after #21479 because there might be non-trivial interactions.

[jdemeyer]

Also in the code setup.sh -> setup.py

[jdemeyer]

Regarding

(cd $(srcdir) && export SAGE_ROOT=/doesnotexist SAGE_SRC=/doesnotexist SAGE_SRC_ROOT=/doesnotexist SAGE_DOC_SRC=/doesnotexist SAGE_SCRIPTS_DIR=/doesnotexist SAGE_BUILD_DIR=/doesnotexist SAGE_CYTHONIZED=/doesnotexist SAGE_PKGS=$(abs_top_srcdir)/build/pkgs && python -u setup.py build --build-base=$(abs_builddir)/build-sagelib install)

Did you know that make supports backslash-continued lines? :-)

[jdemeyer]

For backwards compatibility, can we please keep the name build instead of changing it to build-sagelib?

[jdemeyer]

What is the rationale for splitting the Makefile in two (Makefile and generate_py_source.mk)? That seems like additional complication without reason.

In any case, using make is wrong. You need to use os.environ["MAKE"].

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3a8cc0e

Fix typo in comment

[mkoeppe]

Replying to jdemeyer:

For backwards compatibility, can we please keep the name build instead of changing it to build-sagelib?

If I keep the default, it's hard to demonstrate that "--build-base" actually works.

Moreover, there are too many things already called "build" -- better be specific.

[mkoeppe]

Replying to jdemeyer:

What is the rationale for splitting the Makefile in two (Makefile and generate_py_source.mk)? That seems like additional complication without reason.

setup.sh needs to be in charge of building everything, if sagelib is to become a well-behaved Python package.

[mkoeppe]

generate_py_source.mk could certainly be implemented in pure Python and could become a part of setup.py. But I don't want to work on this.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0dd9c50

Respect environment variable MAKE

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​17f90d8

beautification

[jdemeyer]

Replying to mkoeppe:

Replying to jdemeyer:

For backwards compatibility, can we please keep the name build instead of changing it to build-sagelib?

If I keep the default, it's hard to demonstrate that "--build-base" actually works.

Well, it's easy for the reviewer to test a different value of --build-base.

Moreover, there are too many things already called "build" -- better be specific.

But src/build is pretty clear: it is the build directory corresponding to src, which is the Sage library. I don't like to change it just for the sake of changing it.

[jdemeyer]

Replying to mkoeppe:

generate_py_source.mk could certainly be implemented in pure Python and could become a part of setup.py

That doesn't answer my question. Why do you not keep this in src/Makefile?

[jdemeyer]

Replying to mkoeppe:

setup.sh needs to be in charge of building everything, if sagelib is to become a well-behaved Python package.

Sorry, I didn't see this comment. Okay, that makes sense but it wasn't clear to me,

I guess you should add some comments to src/Makefile to explain this better than just

## All sagelib-building is done by setup.py.
## DON'T ADD ADDITIONAL STEPS TO THIS MAKEFILE.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e5f9065

More comments

[jdemeyer]

Replying to jdemeyer:

I would prefer to do this ticket after #21479 because there might be non-trivial interactions.

Or at least I will test it with #21501 applied.

[mkoeppe]

OK, I'll have #21479 ready in a few days.

[jdemeyer]

I just tested this with #21501. It mostly works but there are these two doctest failures:

sage -t src/sage_setup/find.py
**********************************************************************
File "src/sage_setup/find.py", line 107, in sage_setup.find.find_extra_files
Failed example:
    find_extra_files(["sage.modular.arithgroup"], SAGE_SRC, SAGE_CYTHONIZED, ".")
Expected:
    [('./sage/modular/arithgroup',
      ['.../src/sage/modular/arithgroup/farey.pxd', ...farey_symbol.h...])]
Got:
    [('./sage/modular/arithgroup',
      ['/usr/local/src/sage-config/src/sage/modular/arithgroup/farey.pxd'])]
**********************************************************************
File "src/sage_setup/clean.py", line 87, in sage_setup.clean._find_stale_files
Failed example:
    for f in stale_iter:
        if f.endswith(skip_extensions): continue
        print('Found stale file: ' + f)
Expected nothing
Got:
    Found stale file: sage/stats/distributions/dgs_gauss.h
    Found stale file: sage/libs/lcalc/lcalc_sage.h
    Found stale file: sage/geometry/triangulation/data.h
    Found stale file: sage/misc/cython_metaclass.h
    Found stale file: sage/rings/finite_rings/stdint.h
    Found stale file: sage/stats/distributions/dgs_misc.h
    Found stale file: sage/misc/darwin_memory_usage.h
    Found stale file: sage/stats/distributions/dgs.h
    Found stale file: sage/ext/python_debug.h
    Found stale file: sage/libs/eclsig.h
    Found stale file: sage/ext/solaris_fixes.h
    Found stale file: sage/ext/pyx_visit.h
    Found stale file: sage/matroids/minorfix.h
    Found stale file: sage/modular/arithgroup/farey_symbol.h
    Found stale file: sage/ext/mod_int.h
    Found stale file: sage/combinat/matrices/dancing_links_c.h
    Found stale file: sage/ext/interpreters/wrapper_rr.h
    Found stale file: sage/symbolic/ginac_wrap.h
    Found stale file: sage/ext/interpreters/wrapper_cdf.h
    Found stale file: sage/ext/interpreters/wrapper_el.h
    Found stale file: sage/groups/perm_gps/partn_ref2/refinement_generic.h
    Found stale file: sage/libs/polybori/pb_wrap.h
    Found stale file: sage/sat/solvers/cryptominisat/solverconf_helper.h
    Found stale file: sage/combinat/partitions_c.h
    Found stale file: sage/ext/ccobject.h
    Found stale file: sage/geometry/triangulation/triangulations.h
    Found stale file: sage/libs/ntl/ntlwrap.h
    Found stale file: sage/stats/distributions/dgs_bern.h
    Found stale file: sage/sat/solvers/cryptominisat/cryptominisat_helper.h
    Found stale file: sage/libs/pari/parisage.h
    Found stale file: sage/geometry/triangulation/functions.h
**********************************************************************

[jdemeyer]

One thing I don't like about the approach of this ticket is that SAGE_CYTHONIZED is set by the value of --build-base. I would much rather have it the other way around: that some environment variable is set by configure which is then used to define the value for --build-base and for the environment variable SAGE_CYTHONIZED.

This would be more analogous to #21501 where I want to set $SAGE_LOCAL just once and then derive everything from that.

[mkoeppe]

Replying to jdemeyer:

One thing I don't like about the approach of this ticket is that SAGE_CYTHONIZED is set by the value of --build-base. I would much rather have it the other way around: that some environment variable is set by configure which is then used to define the value for --build-base and for the environment variable SAGE_CYTHONIZED.

This would be more analogous to #21501 where I want to set $SAGE_LOCAL just once and then derive everything from that.

I've updated the description to explain better why --build-base needs to be in charge of setting all build directories.

[mkoeppe]

But yes, I'll introduce something like SAGE_SRC_BUILDBASE that can be used in src/Makefile.in and to define SAGE_CYTHONIZED for that doctest.

[jdemeyer]

Replying to mkoeppe:

I've updated the description to explain better why --build-base needs to be in charge of setting all build directories.

Sorry, but the paragraph you added does not explain this. It explains better the what but not the why. This reminds me of what I recently said at ​https://trac.sagemath.org/ticket/21469#comment:17

[jdemeyer]

Replying to mkoeppe:

But yes, I'll introduce something like SAGE_SRC_BUILDBASE that can be used in src/Makefile.in and to define SAGE_CYTHONIZED for that doctest.

Now I am even more lost...

First you say that --build-base should determine everything but this comment seems to contradict that.

[mkoeppe]

Replying to jdemeyer:

Replying to mkoeppe:

I've updated the description to explain better why --build-base needs to be in charge of setting all build directories.

Sorry, but the paragraph you added does not explain this. It explains better the what but not the why.

I've created ticket #21507 to put the technical goals of the present ticket in the context of a bigger goal. Please take a look.

[jdemeyer]

For this ticket, can you please keep SAGE_CYTHONIZED independent of --build-base (and set them both from the common environment variable you mention in 72). That will also solve the doctest failures from 68.

The current code which figures out SAGE_CYTHONIZED from the command line is quite ugly and fragile. I think that we can do a bigger reorganization of setup.py which will make things simpler. Also Cython 0.25 (not released yet) has some potential to simplify cythonization.

PS: I like the fact that you are splitting this up over many tickets, it gets things moving more quickly.

[jdemeyer]

Replying to mkoeppe:

OK, I'll have #21479 ready in a few days.

Actually, maybe this doesn't really need to depend on #21479, since it doesn't involve ./configure. If you address the points I made, then this might be good to go.

[mkoeppe]

Replying to jdemeyer:

For this ticket, can you please keep SAGE_CYTHONIZED independent of --build-base (and set them both from the common environment variable you mention in 72). That will also solve the doctest failures from 68.

I disagree. I consider setup.py deriving SAGE_CYTHONIZED from --build-base a crucial feature of this patch. What I meant above is simply that setup.py will set an intermediate variable, rather than SAGE_CYTHONIZED, and SAGE_CYTHONIZED would be determined by env.py.

The current code which figures out SAGE_CYTHONIZED from the command line is quite ugly and fragile. I think that we can do a bigger reorganization of setup.py which will make things simpler. Also Cython 0.25 (not released yet) has some potential to simplify cythonization.

I am aware that the arg parsing code is ad hoc, but it's easy to understand code.

I do not want to defer getting --build-base working to some distant future, when someone finds the time to clean up setup.py.

[jdemeyer]

Replying to mkoeppe:

I do not want to defer getting --build-base working to some distant future, when someone finds the time to clean up setup.py.

At least you shouldn't determine the directory from the command line. That's too fragile to work correctly. You should find a way to get the directory from distutils which is going to require some refactoring anyway.

[mkoeppe]

Replying to jdemeyer:

At least you shouldn't determine the directory from the command line. That's too fragile to work correctly. You should find a way to get the directory from distutils which is going to require some refactoring anyway.

The current structure of setup.py does not make this easy. SAGE_CYTHONIZED is used before setup is even invoked. It can't be the job of this ticket to bring setup.py to standard distutils behavior.

[mkoeppe]

I've created a ticket #21508 for setup.py cleanup issues.

[jdemeyer]

Replying to mkoeppe:

The current structure of setup.py does not make this easy.

I know! That exactly why I proposed in 77 to postpone this piece of your patch.

If you really want to go for #21507, you will need a lot of changes to setup.py anyway. So the cleanup will need to happen sooner or later.

[mkoeppe]

Replying to jdemeyer:

Replying to mkoeppe:

The current structure of setup.py does not make this easy.

I know! That exactly why I proposed in 77 to postpone this piece of your patch.

I want the feature of --build-base now, not in the distant future. It is easily achieved with this patch. The arg parsing is ad hoc; but given the current state of setup.py, which does not take care at all about standard distutils behavior, it is an improvement.

If you really want to go for #21507, you will need a lot of changes to setup.py anyway. So the cleanup will need to happen sooner or later.

I prefer to do one concrete step (technical goal) at a time. Let's get --build-base working right now.

[jdemeyer]

Replying to mkoeppe:

Let's get --build-base working right now.

In my opinion, this branch is not making --build-base work. This branch is adding hacks to pretend like --build-base is working.

In general, hacks can have their place, but only if there is a clear reason for them. In this case, I see no reason. This ticket has no reason to exist except for going towards #21507. And this hack that you added will get us no closer to #21507 since we will need to refactor the SAGE_CYTHONIZED code anyway.

[jdemeyer]

Anyway, I will leave this issue for other people to comment, since we probably won't come to an agreement.

[mkoeppe]

Replying to jdemeyer:

Replying to mkoeppe:

Let's get --build-base working right now.

In my opinion, this branch is not making --build-base work. This branch is adding hacks to pretend like --build-base is working.

It does implement --build-base correctly.

In general, hacks can have their place, but only if there is a clear reason for them. In this case, I see no reason. This ticket has no reason to exist except for going towards #21507. And this hack that you added will get us no closer to #21507 since we will need to refactor the SAGE_CYTHONIZED code anyway.

Apart from #21507, it is also needed for the VPATH build, as explained in the description.

[fbissey]

Hum. I am not sure myself. I install sagelib in sage-on-gentoo like a normal package (I just don't install sage_setup and do not clean before hand since this is done by the package manager) with python setup.py build and then python setup.py install --root=.... I guess pip may have extra requirements. Of course I patch a lot of the sagelib code itself to help with various issues, and this ticket does nothing for them. But given your goals, I'd say you'll have to deal with most of them.

There are definitely things to do and getting setup.py to control the autogenerated files may help me. I will have to see it in action before forming a definitive opinion.

My other concerns while they could influence design here are separate issues:

• separation of build time, doctest time and runtime variables. SAGE_SRC shouldn't be found in runtime code for example, neither does package.py belong there.
• Is there a standard place where cython_debug can be installed? It is needed for debugging and shouldn't live under SAGE_SRC.

[mkoeppe]

Replying to fbissey:

• Is there a standard place where cython_debug can be installed? It is needed for debugging and shouldn't live under SAGE_SRC.

Are you saying that it should be installed under SAGE_LOCAL? This should be a separate ticket.

[fbissey]

Agreed on a separate ticket. But I mention cython_debug because it is mentioned in the summary. I am pointing to the fact that it should be installable when it is currently not. I am on an awareness campaign :)

[jdemeyer]

Replying to mkoeppe:

Apart from #21507, it is also needed for the VPATH build, as explained in the description.

You don't need the --build-base hack for VPATH builds to work. You could just set SAGE_CYTHONIZED correctly from the Makefile.

[mkoeppe]

Replying to fbissey:

Agreed on a separate ticket. But I mention cython_debug because it is mentioned in the summary. I am pointing to the fact that it should be installable when it is currently not. I am on an awareness campaign :)

I've created #21509 for this; but I guess you should explain more on that ticket.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7791cd9	Remove --buildbase code
​74169e7	Pass SAGE_SRC to generate_py_source.mk

[mkoeppe]

I have simplified this ticket to facilitate easier review. Jeroen, please take a look.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0394333

Add new file to MANIFEST.in

[jdemeyer]

Looks good on first sight, but I need to test it.

[mkoeppe]

Thanks!

[jdemeyer]

Trouble with Jupyter...

[jdemeyer]

src/setup.py runs code from src/sage/repl/ipython_kernel/kernel.py which still uses some of the directories that you blacklist.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fdedb02

Install SAGE_LOCAL/bin/sage instead of SAGE_ROOT/sage as Jupyter kernel

[mkoeppe]

With this change I've tested that I can do sage -n jupyter and then run a kernel from the Jupyter notebook. SAGE_LOCAL/bin/sage picks up its environment variables from the server that runs it, so everything works. Does it have to work in other settings that I'm unaware of as well?

[jdemeyer]

What about the use of SAGE_DOC and SAGE_EXTCODE in that file?

[mkoeppe]

SAGE_DOC and SAGE_EXTCODE are fine as they are subdirectories of SAGE_LOCAL. The patch does not poison any environment variables that are subdirectories of SAGE_LOCAL.

[embray]

Very interesting--I'd like to take some time to review this and all the comments on it. I probably won't get to it in detail until next week but I'd really like a chance to look it over.

[jdemeyer]

Replying to mkoeppe:

With this change I've tested that I can do sage -n jupyter and then run a kernel from the Jupyter notebook.

Did you test with a custom value for SAGE_LOCAL (#21501)? Because that should be done (I could do that, but not right now).

[mkoeppe]

I would probably rather test with --prefix (#21479).

[fbissey]

Replying to mkoeppe:

I would probably rather test with --prefix (#21479).

From a distro point of view, I would also want it to be tested with the underdocumented --root.

[mkoeppe]

Replying to fbissey:

Replying to mkoeppe:

I would probably rather test with --prefix (#21479).

From a distro point of view, I would also want it to be tested with the underdocumented --root.

Is --root an option of setup.sh? (I was referring to ./configure --prefix)

[mkoeppe]

(I meant setup.py of course -- I keep making the typo "setup.sh")

[fbissey]

Yes, setup.py I should abstain from commenting until my fever goes down :(

[mkoeppe]

Replying to fbissey:

From a distro point of view, I would also want it to be tested with the underdocumented --root.

This is now #21573: Make sure src/setup.py respects --install-base and --root.

[mkoeppe]

Replying to jdemeyer:

Replying to mkoeppe:

With this change I've tested that I can do sage -n jupyter and then run a kernel from the Jupyter notebook.

Did you test with a custom value for SAGE_LOCAL (#21501)? Because that should be done (I could do that, but not right now).

I've now tested with #21501 and a fresh build with SAGE_LOCAL set to $SAGE_ROOT/very/far/away/but/still/local. The Jupyter notebook works.

[jdemeyer]

Good enough for me.

[embray]

Not that it really matters, but python -u setup.py build install is superfluous--the install command always runs build as a sub-command.

[embray]

For the sake of keeping this ticket focused I think it's fine for now, but something definitely needs to be done about using make for the autogenerated sources. My ​comment in #21507 has some suggestions toward that. Is there already a separate ticket for dealing with that? If so I can work on it.

[embray]

Also, if you'd like, I can suggest a much less "hacky" way of handling --build-base, is there already a separate ticket for that?

[jdemeyer]

Replying to embray:

Also, if you'd like, I can suggest a much less "hacky" way of handling --build-base, is there already a separate ticket for that?

#21573 I guess.

[mkoeppe]

#21573 is for --install-base and --root.

--build-base work should go on either one of the following two:

• #21508: Clean up src/setup.py to bring it to standard distutils behavior
• #21535: Make src/setup.py independent of SAGE_CYTHONIZED

[mkoeppe]

Replying to embray:

Not that it really matters, but python -u setup.py build install is superfluous--the install command always runs build as a sub-command.

I've left it like this as a little reminder that --build-base can go in between.

[embray]

For generate_py_source.mk, we could even get rid of that and not invoke make at all. For simplicity's sake I could do that as a separate ticket with this one as a dependency.

To add, I especially don't like what this branch currently does as far as unconditionally invoking make every time setup.py is run. But I'm happy to ignore it for now as long as we follow up with further work--I'm opening a new ticket now.

[jdemeyer]

Replying to fbissey:

• Is there a standard place where cython_debug can be installed? It is needed for debugging and shouldn't live under SAGE_SRC.

On the other hand, debugging also requires the sources to be available. So it is not a crazy idea to keep the debug info close to the sources.

Maybe the location for cython_debug can be improved, but I think you should only install it in SAGE_LOCAL if you also install the sources somewhere in SAGE_LOCAL.

[jdemeyer]

Replying to embray:

For generate_py_source.mk, we could even get rid of that and not invoke make at all.

But make does automatic dependency checking for you. Whatever Python alternative you come up with will be more complex than a simple makefile.

[embray]

I would disagree. In fact as the discussion in #20730 demonstrates that writing a technically correct makefile for this case results in a makefile that's "too complicated to understand". I can do this correctly in Python with about the same amount of code it takes to invoke make as a subprocess (which is also pretty non-standard to do anything in a Python setup). Not to mention the contents of the Makefile itself (no matter how "correct" it is).

Part of the idea is also not to make non-Sage developers blanche when they look inside of sage's setup.py.

[embray]

Replying to jdemeyer:

Replying to fbissey:

• Is there a standard place where cython_debug can be installed? It is needed for debugging and shouldn't live under SAGE_SRC.

On the other hand, debugging also requires the sources to be available. So it is not a crazy idea to keep the debug info close to the sources.

This is why I also like to output the C(++) sources themselves to be close to the cython sources but you said you don't like :)

[felixs]

Replying to jdemeyer:

But make does automatic dependency checking for you. Whatever Python alternative you come up with will be more complex than a simple makefile.

true.

there is a python alternative that is no more complex than a simple makefile *plus* make. it's the simple makefile + a *subset* of make implemented in python. (yes, you could use GNU make as an intermediate solution...)

but seriously: what's all the fuss about reinventing and replacing make? it should obviously be the other way round: python is not the right tool for the job, it's make. unlike python, make is *designed* for this. no matter which language you implement it in.

if there's a need for a make replacement (completely different[tm]), is sage the right community or the right project to invent it in?

[fbissey]

Replying to embray:

Replying to jdemeyer:

Replying to fbissey:

• Is there a standard place where cython_debug can be installed? It is needed for debugging and shouldn't live under SAGE_SRC.

On the other hand, debugging also requires the sources to be available. So it is not a crazy idea to keep the debug info close to the sources.

This is why I also like to output the C(++) sources themselves to be close to the cython sources but you said you don't like :)

For the record: in sage-on-gentoo I ship the .pyx files next to the matching .so files. In any case distro ship debugging symbols without the sources. Of course at some point you may need the source code but this is trivial to get and unfold separately in most cases.

[vbraun]

This ticket introduces an autotools dependency (src/Makefile.in) and fails on OSX

[jdemeyer]

The is the usual breakage of your release management scripts in case of changes to configure.

[jdemeyer]

New commits:

​2cf7352	Merge tag '7.4.beta6' into t/21480/keep_src__clean_by_using___build_base_when_building_sagelib
​78ab5d7	Bump configure version

[mkoeppe]

This now conflicts with the configure spkg version change in 7.4.rc0. Seems to me incrementing configure spkg versions is something that should be done by the release manager.

[jdemeyer]

Replying to mkoeppe:

This now conflicts with the configure spkg version change in 7.4.rc0. Seems to me incrementing configure spkg versions is something that should be done by the release manager.

I know, it is really annoying.

[jdemeyer]

This problem happens every time that you make non-trivial changes to configure.ac

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:

​2871769	Merge tag '7.4.rc0' into t/21480/keep_src__clean_by_using___build_base_when_building_sagelib
​0c2ac95	Bump configure version

[jdemeyer]

This should be merged as soon as possible to avoid further problems with configure.