Reorganize /build

[vbraun]

Proposal: Make /build the root for the sage_bootstrap Python module

• /build/sage_bootstrap/...
• /build/setup.py
• /build/tox.ini
• /build/bin/sage-package
• /build/bin/sage-spkg

Keep the current package metadata at

• /build/pkgs/...

Move the makefile stuff to

• /build/make/Makefile
• /build/make/deps
• /build/make/pipestatus
• /build/make/install
• /build/make/...

[vbraun]

Note to self: this will require an updated confball

---

New commits:

​b667765	Move makefile-stuff to build/make/
​a7546b1	Updating paths for build/make/

[jdemeyer]

FYI: this conflicts heavily with #18710.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f034a44	set up print and logging for sage_bootstrap
​bd61b67	add the sage-pkg entrypoint
​25022c3	move sage-download-file to the sage_bootstrap library
​0acf070	initial implementation of the sage-pkg tool
​52d0126	rename sage-pkg -> sage-package
​e4aff9a	fix indentation
​e45a0b6	Do not assume that the confball has been downloaded before in test
​da11bfc	Merge sage_bootstrap branch nto #18788
​fd25c7c	Move sage_bootstrap to /build

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f3c4445	Move various targets to build/Makefile
​b86e0bb	Fix strange mixture of TABs and spaces
​bd3149b	Merge branch 'u/jdemeyer/move_some_make_targets_to_build_makefile' of http://trac.sagemath.org/sage into #18788

[vbraun]

IMHO the pipestatus and install scripts are just a symptom of the makefile hack. They should be removed when we switch to a sane build system, so they shouldn't be in build/bin

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​51f13e8	fix documentation build
​0aa6cfb	Remove test for faux package

[vbraun]

The last commit fixes the doctest failure introduced in #18710 (but only triggered after build from scratch):

sage -t --warn-long 27.4 src/sage/misc/package.py
**********************************************************************
File "src/sage/misc/package.py", line 193, in sage.misc.package.is_package_installed
Failed example:
    is_package_installed('sage')
Expected:
    True
Got:
    False
**********************************************************************
1 item had failures:
   1 of   4 in sage.misc.package.is_package_installed
    [15 tests, 1 failure, 0.08 s]

[jdemeyer]

Replying to vbraun:

IMHO the pipestatus and install scripts are just a symptom of the makefile hack. They should be removed when we switch to a sane build system, so they shouldn't be in build/bin

For the record: I don't think makefiles are a hack. The current system surely is far from perfect, but it actually does work pretty well.

[jdemeyer]

Replying to vbraun:

The last commit fixes the doctest failure introduced in #18710 (but only triggered after build from scratch):

sage -t --warn-long 27.4 src/sage/misc/package.py
**********************************************************************
File "src/sage/misc/package.py", line 193, in sage.misc.package.is_package_installed
Failed example:
    is_package_installed('sage')
Expected:
    True
Got:
    False
**********************************************************************
1 item had failures:
   1 of   4 in sage.misc.package.is_package_installed
    [15 tests, 1 failure, 0.08 s]

I don't think this commit should be on this ticket, but on a separate blocker ticket with just this one commit.

Also: don't you test building from scratch when closing a ticket?

[jdemeyer]

In case I haven't made it sufficiently clear on #18748: I am still against the fact that you first create src/sage_bootstrap/sage_bootstrap on #18748 and then immediately move it to build/sage_bootstrap on this ticket. This is just making things needlessly complicated.

[vbraun]

Well we just saw that the current system can't do reliable incremental builds, I think that this is pretty bad. That was ok 20 years ago but today this is hardly state of the art.

For the record, I'm only doing incremental builds for each ticket on the buildbot. In addition, there is a weekly full build.

You can review both tickets at the same time if the location of sage_bootstrap irks you.

[jdemeyer]

Replying to vbraun:

Well we just saw that the current system can't do reliable incremental builds

What do you refer to?

[jdemeyer]

Replying to jdemeyer:

I don't think this commit should be on this ticket, but on a separate blocker ticket with just this one commit.

See #18806 instead.

[vbraun]

Replying to jdemeyer:

Replying to vbraun:

Well we just saw that the current system can't do reliable incremental builds

What do you refer to?

I refer to #18710 working fine on incremental builds but breaking full builds. By definition, that means our build system cannot reliably do incremental builds. Really, the build system has one job to do and it failed at it. Essentially, improper handling of build artifacts. Thats ok if you have a small library, but it hurts when a full rebuild takes an hour, and it makes truly large-scale software projects impossible (e.g. google built blaze/bazel when they hit that wall with makefiles).

[jdemeyer]

Replying to vbraun:

I refer to #18710 working fine on incremental builds but breaking full builds.

So the fact that one ticket changing the build system breaks it proves that it's not reliable?

What do you want, a build system that is impossible to break when changing it?

[vbraun]

Yes. There are various open/free implementations that do precisely that, so why shouldn't we?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a497d23

Merge 6.8.beta7 into t/18788/reorganize_build

[jdemeyer]

Squashed on top of #18748 for easier reviewing.

---

New commits:

​9e2c568	Merge branch 'develop' into t/18748/python_library_to_bootstrap_sage
​cc213dc	trac 18748: add documentation to scripts in sage_bootstrap/bin
​033b56e	Merge Sage-6.8.beta8 into #18748
​8f4fb8e	More docstrings
​f9a69a5	Convert public attributes to properties and document
​075228f	Make package.tarball a Tarball instance
​c56991b	Fix Python 2.6 test error
​60a581b	trac 18748: add URL for tox to sage_bootstrap/README
​0e168b1	Reorganize /build

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b324ff0

Move README inside sage_bootstrap directory

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​538177e	Fix paths in documentation and comments
​550b26e	Move sage-spkg to build/bin

[jdemeyer]

This looks okay to me now, what remains is testing.

[jdemeyer]

Seems to work...

I give positive_review to your commit, somebody needs to review mine.

[vbraun]

wrong path in sage-sdist

[vbraun]

New commits:

​b667765	Move makefile-stuff to build/make/
​a7546b1	Updating paths for build/make/
​da11bfc	Merge sage_bootstrap branch nto #18788
​fd25c7c	Move sage_bootstrap to /build
​bd3149b	Merge branch 'u/jdemeyer/move_some_make_targets_to_build_makefile' of http://trac.sagemath.org/sage into #18788
​51f13e8	fix documentation build
​0aa6cfb	Remove test for faux package
​a497d23	Merge 6.8.beta7 into t/18788/reorganize_build
​c1be005	Fix remaining occurence of old path in sage-spkg

[jdemeyer]

The sage-sdist fix is obviously ok.

But I'm confused with the git history, did you just completely forget about my branch? If that was a mistake, I give positive review to your last commit cherry-picked onto my branch.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​8f4fb8e	More docstrings
​f9a69a5	Convert public attributes to properties and document
​075228f	Make package.tarball a Tarball instance
​c56991b	Fix Python 2.6 test error
​60a581b	trac 18748: add URL for tox to sage_bootstrap/README
​0e168b1	Reorganize /build
​b324ff0	Move README inside sage_bootstrap directory
​538177e	Fix paths in documentation and comments
​550b26e	Move sage-spkg to build/bin
​35efc21	Fix remaining occurence of old path in sage-spkg

[vbraun]

thanks, forgot to pull...