Opened 6 years ago

Last modified 2 years ago

#14807 needs_work task

Transition to build system for sage (the library)

Reported by: felixs Owned by: felixs
Priority: major Milestone: sage-6.4
Component: distribution Keywords: sagelib
Cc: leif, fbissey, jondo Merged in:
Authors: Felix Salfelder Reviewers:
Report Upstream: N/A Work issues:
Branch: u/felixs/sagelib (Commits) Commit: fc097b0d3725822513f174de099c19bafbc9fe6f
Dependencies: #14726, #14834, #15102, #15112, #15117, #15123, #15126, #15176 Stopgaps:

Description (last modified by felixs)

Sage ("the library") needs a build system that works inside and outside of sage ("the distribution").

This ticket integrates the build systems for the parts of sagelib into the current toplevel build system.

Change History (37)

comment:1 Changed 6 years ago by leif

  • Cc leif added
  • Owner changed from Felix Salfelder to felixs

comment:2 Changed 6 years ago by fbissey

  • Cc fbissey added

comment:3 Changed 6 years ago by felixs

  • Dependencies changed from #14726 to #14726 #14834

comment:4 Changed 6 years ago by jondo

  • Cc jondo added

comment:5 follow-up: Changed 6 years ago by jdemeyer

  • Dependencies changed from #14726 #14834 to #14726, #14834
  • Priority changed from blocker to major

Why would this be a blocker?

comment:6 in reply to: ↑ 5 Changed 6 years ago by felixs

Replying to jdemeyer:

Why would this be a blocker?

it's not, sorry.

i have created a dist tarball from my WIP branch. it is contained in http://tool.em.cs.uni-frankfurt.de/~felix/sagelib. please comment or help to improve it.

comment:7 Changed 6 years ago by felixs

  • Branch set to u/felixs/sagelib
  • Commit set to feff956b616b9c8264e2363283f7d568e622cec1
  • Dependencies changed from #14726, #14834 to #14726, #14834, #15117
  • Description modified (diff)
  • Keywords sagelib added

comment:8 Changed 6 years ago by felixs

  • Dependencies changed from #14726, #14834, #15117 to #14726, #14834, #15102, #15112, #15117, #15123, #15126

comment:9 Changed 6 years ago by git

  • Commit changed from feff956b616b9c8264e2363283f7d568e622cec1 to afa7de70dd4f27ae76588ab13c8f36bb5cadb733

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:afa7de7]Merge branch 'python_sage_build' into sagelib
[changeset:848bef8]Merge branch 'libgap_include' into sagelib
[changeset:f77e13c]python-sage: env
[changeset:29f8f3e]python-sage build system continued/cleanup
[changeset:f2d1c37]integrate sagelib build systems into old toplevel
[changeset:f5261c6]bump libgap package patchlevel
[changeset:318138e]apply patches from patchdir
[changeset:28b0da1]python-sage build system

comment:10 Changed 6 years ago by git

  • Commit changed from afa7de70dd4f27ae76588ab13c8f36bb5cadb733 to d4e1833d483f142e3e3f47e8b882b0bf48955d76

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:d4e1833]Merge branches 'sage_doc', 'python_sage_build' and 'csage_build_system' into sagelib
[changeset:1f3f7e4]Merge branches 'cython_dbg' and 'cython_deps' into python_sage_build
[changeset:42449c1]python-sage doctest path tolerance
[changeset:ae8bb63]python-sage: env
[changeset:5a16d99]python-sage build system
[changeset:79cf599]sage-bin env
[changeset:3ff5c49]sagelib: environment for check target
[changeset:ffe9b9d]sage-doc. use SAGE_DOC_SRC as doc source path
[changeset:2df5bfc]sage-doc: build system. mostly dist
[changeset:510f8b7]Merge branch 'gmp_mpir' into csage_build_system
[changeset:0d50108]provide gmpxx_or_mpirxx.h (required by sage-python)
[changeset:1e4789f]csage: gmp_or_mpir fixup
[changeset:926e3f7]build autotools before csage
[changeset:d4e9997]Merge branch 'autotools' into csage_build_system
[changeset:f82b422]cython: patch: gdb output dir option
[changeset:72c4538]add more autotools executables
[changeset:bd11c24]cython: add -MF switch to write dependency files (#14728)

comment:11 Changed 6 years ago by felixs

  • Description modified (diff)
  • Status changed from new to needs_review
  • Summary changed from build system for sage (the library) to Transition to build system for sage (the library)

A clean sagelib package that works outside of sage (the distribution) requires something like #14796. I will open another ticket...

comment:12 Changed 6 years ago by git

  • Commit changed from d4e1833d483f142e3e3f47e8b882b0bf48955d76 to 2646f0d29a2437eaf6d61d5cef49108da0db503c

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:2646f0d]Merge branch 'runtime_paths' into sagelib
[changeset:0d3cfcd]build/deps
[changeset:30fb726]sagelib build system
[changeset:b2415da]provide sage-sh within sage-the-distribution
[changeset:8de880c]fix a typo
[changeset:39f7247]ellcurve data dir fixup
[changeset:0b9cb04]doctests in src/sage/plot/misc.py
[changeset:1ef62a1]unhardwire cython.py completed
[changeset:390e28b]qepcard doctest fix: singular might well be in /usr/bin, not .../local/bin
[changeset:0dcd721]unhardwire ELLCURVES_DATA_DIR
[changeset:591471d]python-sage: whitelist python2.7 multiprocessing in stackframes test
[changeset:c19a5f3]unhardwire sandpile
[changeset:5746289]unhardwire polymake path

comment:13 Changed 6 years ago by git

  • Commit changed from 2646f0d29a2437eaf6d61d5cef49108da0db503c to 1c895525862649c8eb07c5da02b7a31a2f0504f3

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:1c89552]cython doctest
[changeset:ef8771d]dependency chain in build/deps
[changeset:7aaf953]Merge 'trac/u/ohanar/build_system' into sagelib

comment:14 Changed 6 years ago by felixs

  • Dependencies changed from #14726, #14834, #15102, #15112, #15117, #15123, #15126 to #14726, #14834, #15102, #15112, #15117, #15123, #15126, #15176

comment:15 Changed 6 years ago by git

  • Commit changed from 1c895525862649c8eb07c5da02b7a31a2f0504f3 to 0e91a83ed0f02c1d0b29d3b94622b8224c0b2ccd

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:0e91a83]Merge branches 'sage-bin' and 'python_sage_build' into sagelib
[changeset:e429991]inject proper paths into autotools calls
[changeset:41326b4]sagelib: freeze (parts of) environment during configure
[changeset:ec442e5]always use autotools within sage ("the distribution")
[changeset:26c67b2]python-sage: make cython call more toolchain friendly
[changeset:69fda9a]Merge remote-tracking branch 'trac/u/ohanar/build_system' into sage-bin
[changeset:f9f8d16]sage-bin: remove PATH_REDUNDANCY cruft
[changeset:3e2aeb1]transition from old toplevel. create symlinks, for now.
[changeset:e0064c9]sage-bin expand SAGE_ETC correctly
[changeset:fcb670f]simplify python byte compile rules
[changeset:2af9070]GMP_VS_MPIR fixup
[changeset:45a30e1]provide unhardwired SAGE_CFLAGS
[changeset:e195725]python-sage: Makefiles
[changeset:c4562b3]python_sage: build system continued
[changeset:6d074ee]sage-bin: more config files

comment:16 Changed 6 years ago by git

  • Commit changed from 0e91a83ed0f02c1d0b29d3b94622b8224c0b2ccd to fc097b0d3725822513f174de099c19bafbc9fe6f

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:fc097b0]build/deps fixup
[changeset:88a4478]Merge branch 'python_sage_build' into sagelib
[changeset:1dacfa5]python-sage: Makefiles update for 512beta4
[changeset:cb27876]python-sage: Makefiles fix

comment:17 Changed 5 years ago by jdemeyer

  • Status changed from needs_review to needs_work

Needs to be rebased.

comment:18 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:19 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:20 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:21 Changed 3 years ago by embray

I wonder if Felix could give an update on the status of this work. Obviously it hasn't been touched in a long while and needs to be rebased. But it seems like a lot of the heavy lifting has been done and this provides a good starting point for overhauling Sage's build.

What is the status of the Sage community's interest in this sort of work?

comment:22 Changed 3 years ago by felixs

Hi Erik.

maybe you can reycle the configure.ac (and most path related patches) for sagelib. the canonical transition would be to reuse the python build system for a start (templatize setup.py, call it from make...)

i could not get anywhere before rewriting the whole thing to automake, as setup.py did/does not work (until today?). mostly due to

  • lack of automatic dependency tracking (C/C++ compilers only write makefiles, setup.py does not use them. cython -MF is not ready yet)
  • missing VPATH builds (requirement for mere sanity, also for distcheck, cf. #21469),

rendering the approach useless for debugging/development on foreign distros.

eventually all this was of no use for sage (the distribution) until a functional top-level build system would be in place. it might still facilitate packaging and use...

i am not totally up to date with the recent (last ~3 years) changes, and YMMV.

comment:23 follow-up: Changed 3 years ago by jdemeyer

@felixs: I think it is a bit strange that you're trying to use a autotools-based build system for a Python package. I don't know if that has ever been done. I agree in principle that autotools is better than whatever Python provides. However, you should weigh this against the fact that this would be very non-standard. I guess that distros would hate us for doing that.

comment:24 in reply to: ↑ 23 ; follow-up: Changed 3 years ago by felixs

Replying to jdemeyer:

@felixs: I think it is a bit strange that you're trying to use a autotools-based build system for a Python package.

hi Jeroen

the reason i pulled in autotools was initially just autoconf. if you can make autoconf+setup.py.in work, then you don't need automake. likewise, if you reimplement autoconf in python etcpp. there are many ways, most of which involve "implement a viable alternative to autotools", which in my understanding is beyond the scope of this project.

I don't know if that has ever been done.

automake supports python for a loong time. i am not "trying to use" rather than "suggesting" to use it. the cython rules i wrote for sagelib back then are now in use for another project (where, coincidentally, cython is used on some interface code).

I guess that distros would hate us for doing that.

that i don't understand. package maintainers usually keep saying things like "people, use auto{conf,tools,make}". cannot think of a way that would make packageing any easier. (care to explain?)

comment:25 in reply to: ↑ 24 Changed 3 years ago by jdemeyer

Replying to felixs:

that i don't understand. package maintainers usually keep saying things like "people, use auto{conf,tools,make}".

...as alternative to home-brew build systems. But I have never seen a package maintainer say that as alternative to setup.py.

comment:26 follow-up: Changed 3 years ago by felixs

But I have never seen a package maintainer say that as alternative to setup.py.

true/plausible. the typical python package has simple dependencies and does not use cython. no need for a chainsaw.

still, i see no evidence why someone might "hate us" for using auto* (for whichever sort of package). instead (let me repeat), the autotools implement

  • VPATH
  • (pre build) configuration
  • (compile time) dependencies
  • a make based build process

all of which immediately facilitate development, testing, debugging and thus packaging.

comment:27 in reply to: ↑ 26 Changed 3 years ago by jdemeyer

Replying to felixs:

instead (let me repeat), the autotools implement

  • VPATH
  • (pre build) configuration
  • (compile time) dependencies
  • a make based build process

all of which immediately facilitate development, testing, debugging and thus packaging.

The autotools implement that for a typical C/C++ package. They don't really implement all that for Python packages. Which means that we have to implement that ourselves.

comment:28 Changed 3 years ago by felixs

The autotools implement that for a typical C/C++ package.

at least the four bullet points are intended to work for a lot more, including python packages [1]. arguably, to compile cython stuff, some custom (make) rules are required, but we had that. and it could be changed (=moved into automake).

They don't really implement all that

what exactly do you mean? please give an example.

thanks

[1] http://ftp.gnu.org/old-gnu/Manuals/automake-1.6.1/html_node/automake_59.html

comment:29 Changed 3 years ago by jdemeyer

OK, that looks interesting. I didn't know that automake could do that.

comment:30 follow-ups: Changed 3 years ago by fbissey

Yes it wexist but it is not always implemented in a nice way for package maintainer. In most case I have seen that used, you have a main package using autoconf and then they have some python bindings. Typically if if the python bits support multiple versions of python, as a package maintainer I would prefer it in a separate package.

In a distro you will usually provide bindings for all of the versions of python you support when feasible. When it is integrated as part of a bigger package it is usually a pain to cycle through different versions of python. Once it is a separate package, autotools, cmake or setup.py, I don't really care any more, I'll easily cycle on both.

Admittedly software detection and so on is easier with something else than python...

comment:31 in reply to: ↑ 30 Changed 3 years ago by leif

Replying to fbissey:

Admittedly software detection and so on is easier with something else than python...

SConstruct! B)

comment:32 Changed 3 years ago by fbissey

<curled in a ball and rocking on his chair>I will not feed the trolls. I will not feed the trolls......

comment:33 Changed 3 years ago by leif

Jeroen loves SC...

comment:34 in reply to: ↑ 30 ; follow-up: Changed 3 years ago by jdemeyer

Replying to fbissey:

Admittedly software detection and so on is easier with something else than python...

For the record: in https://github.com/sagemath/cysignals I am using autoconf for the configuration but I am still using distutils to build and install.

comment:35 in reply to: ↑ 34 Changed 3 years ago by felixs

Replying to jdemeyer:

For the record:

here's the autotoolized sage (dist [1] + lib [2]). this is where i gave up. i never touched the sage code since then... watch out for configure.ac and various m4 scripts. note that all subdirectories in src are standalone packages.

[1] https://github.com/felix-salfelder/sage/tree/autotools

[2] https://github.com/felix-salfelder/sage/tree/autotools/src

comment:36 Changed 2 years ago by embray

I have mixed feelings about the whole thing, but it's worth noting that it's getting closer and closer to the point where what's in setup.py won't matter as much as it does now. distutils is still the "easiest" way to build Python modules, so somewhere in there there will still likely be a setup.py for a while.

But there's been some progress on better supporting binary wheels on Linux (see for example https://www.python.org/dev/peps/pep-0513/, though that obviously doesn't cut it for Sage). There's also progress being made on standardizing a "source distribution" format for Python packages that isn't necessarily tied to any one build system. This means it will be fine, for example, to use autoconf to build Sage. It's fine now, even, it just doesn't fit in as as well (yet) with the Python ecosystem.

I think regardless, it will also help to further break up sage so that Python wrappers for its various dependencies are external projects themselves. I realize that's a much bigger problem though and not always practical.

comment:37 Changed 2 years ago by embray

Also, all that said, there are examples of Pythonic build configuration too. One of the best is matplotlib's: https://github.com/matplotlib/matplotlib/blob/master/setupext.py

Matplotlib has quite a few external, non-Python dependencies--nowhere close to Sage, but more than most Python packages, and it handles this nicely. Yes, it bothers me that it's "hand rolled", but most of that stuff in matplotlib can, and probably should, be factored out to an external package (with documentation!) that others can use. And I would rather be writing Python than m4 scriptlets >_<

Numpy has a similar system (used also by Scipy), but matplotlib's is a bit cleaner (even it has a few things I would do a little differently, but the concept is sound).

Note: See TracTickets for help on using tickets.