make sagelib a script package

[mkoeppe]

... so that sagelib is just an ordinary script package whose source tree is included (hence, no checksums.ini file; see #29287).

Related sage-devel discussions:

• ​https://groups.google.com/d/msg/sage-devel/nTwhCV89FXE/_7GdzGy4BgAJ ("package sources")
• ​https://groups.google.com/d/msg/sage-devel/nTwhCV89FXE/-obd4rMDBAAJ

Related tickets:

• #29386 Install script packages via sage-spkg

[mkoeppe]

New commits:

​768a314	Merge build/make/deps into build/make/Makefile.in
​80cb1b9	Make sagelib a script package

[mkoeppe]

Not quite done yet. Next step is to get rid of src/Makefile.in, moving the contents into the spkg-install and spkg-clean scripts.

[mkoeppe]

The clean target should perhaps be transformed into setup.py clean, reference: ​https://github.com/pypa/setuptools/issues/1347

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​91e5a3d

src/Makefile.in: Merge into build/make/Makefile.in, build/pkgs/sagelib/spkg-install

[mkoeppe]

Replying to mkoeppe:

The clean target should perhaps be transformed into setup.py clean, reference: ​https://github.com/pypa/setuptools/issues/1347

That's already noted in #21508, will not happen on the present ticket

[mkoeppe]

TBD: sagelib rebuilds are no longer triggered because it now goes through a non-phony target

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​08cbbf2

build/make/Makefile.in, build/pkgs/sagelib/dependencies: Make sure sagelib spkg-install is always run

[fbissey]

Not quite what I had in mind but this is a step in the right direction. It makes sagelib less special in the build system. I believe the dependency part is OK. The makefile bits deserve a bit scrutiny and field testing.

[jhpalmieri]

The old src/Makefile.in and now the new build/pkgs/sagelib/spkg-install has this in it:

Hoping that #20382 will make this unnecessary.

#20382 has been merged, so perhaps this should be clarified, although not necessarily on this ticket.

[mkoeppe]

Right, this is now #28815.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5e915b0

build/pkgs/sagelib/spkg-install: Update references to tickets

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​53f0359	src/sage/env.py (sage_include_directories): Do not put SAGE_INC in front of the sage source include directories
​16247ad	src/setup.py: Do not put SAGE_LOCAL/lib in front of the library directories
​9a50cba	src/sage/env.py (sage_include_directories): Fixup doctest
​73fb13a	Merge branch 't/29697/src_setup_py__src_sage_env_py__sage_include_directories___do_not_add_another_copy_of_sage_inc__sage_local_lib_to_include_dirs__library_dirs' into t/29411/make_sagelib_a_script_package
​f6e3fa1	build/pkgs/sagelib/spkg-install: Also poison environment variables SAGE_LOCAL etc.

[mkoeppe]

Ready for review.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fcf2cab

build/pkgs/sagelib/spkg-install: Add comment regarding SAGE_SPKG_INST

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a973fc4

build/pkgs/sagelib/spkg-install: Also poison SAGE_PKGCONFIG, SAGE_SPKG_SCRIPTS

[dimpase]

please remove gcc from the modified comment in src/setup.py - it's a misleading artefact from the past, where gcc was needed to build Sage.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c1db7ed

src/setup.py: Update comment (not specific to gcc)

[dimpase]

What is the point of FORCE target - to always (re)build sagelib, right? But is it needed?

[mkoeppe]

Yes, without it, sagelib would not be rebuilt when source files change.

[dimpase]

hmm, but why not honestly list deps in a makefile?

[mkoeppe]

Well, instead of the dependency on FORCE that is listed in build/pkgs/sagelib/dependencies, we could make it a dependency on $(SAGE_SRC) $(SAGE_SRC)/* $(SAGE_SRC)/*/* $(SAGE_SRC)/*/*/* $(SAGE_SRC)/*/*/*/*, but I don't think this is better

[mkoeppe]

In other words, we do not have dependency tracking at all for sagelib. Neither of distutils, setuptools, cython seem to have a mechanism for generating dependencies like gcc -M. This would be certainly be nice to have; in particular if it included dependencies on "system" headers and libraries, so that sagelib could automatically rebuild when system headers/libraries change (as in, most recently, ​https://groups.google.com/d/msg/sage-release/A53tGGsJNLg/uA01oUugAQAJ). But that's outside of the scope of this ticket.

[dimpase]

OK, makes sense.

[mkoeppe]

Thanks!

[mkoeppe]

I have created #29711 (sagelib setup.py: Dependencies on header files of packages).

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. Last 10 new commits:

​e1a9413	Merge build/make/deps into build/make/Makefile.in
​da9538e	Make sagelib a script package
​9fd3f7e	src/Makefile.in: Merge into build/make/Makefile.in, build/pkgs/sagelib/spkg-install
​9ea77e5	build/make/Makefile.in, build/pkgs/sagelib/dependencies: Make sure sagelib spkg-install is always run
​c3908e9	build/pkgs/sagelib/spkg-install: Update references to tickets
​62f2841	Merge branch 't/29697/src_setup_py__src_sage_env_py__sage_include_directories___do_not_add_another_copy_of_sage_inc__sage_local_lib_to_include_dirs__library_dirs' into t/29411/make_sagelib_a_script_package
​048ce5b	build/pkgs/sagelib/spkg-install: Also poison environment variables SAGE_LOCAL etc.
​4590c4c	build/pkgs/sagelib/spkg-install: Add comment regarding SAGE_SPKG_INST
​ed312c5	build/pkgs/sagelib/spkg-install: Also poison SAGE_PKGCONFIG, SAGE_SPKG_SCRIPTS
​6ee66dd	src/setup.py: Update comment (not specific to gcc)

[mkoeppe]

Redone on top of new version of #29098

[embray]

Oh, I was just thinking about this. Thanks for picking up the work. This is something I've been wanting to do for years.

[vbraun]

I'm getting this on kucalk buildbot

make[3]: Entering directory '/var/lib/buildbot/slave/sage_git/build/build/make'
cd '/var/lib/buildbot/slave/sage_git/build' && source '/var/lib/buildbot/slave/sage_git/build/src/bin/sage-env' && source '/var/lib/buildbot/slave/sage_git/build/build/bin/sage-build-env-config' && sage-logger -p '/var/lib/buildbot/slave/sage_git/build/build/pkgs/sagelib/spkg-install' '/var/lib/buildbot/slave/sage_git/build/logs/pkgs/sagelib-9.1.rc5.log'
[sagelib-9.1.rc5] /var/lib/buildbot/slave/sage_git/build/local/bin/pkg-config: line 16: /doesnotexist/bin/pkgconf: No such file or directory
[sagelib-9.1.rc5] ************************************************************************
[sagelib-9.1.rc5] Traceback (most recent call last):
[sagelib-9.1.rc5]   File "setup.py", line 72, in <module>
[sagelib-9.1.rc5]     from module_list import ext_modules, library_order
[sagelib-9.1.rc5]   File "/var/lib/buildbot/slave/sage_git/build/src/module_list.py", line 14, in <module>
[sagelib-9.1.rc5]     cblas_pc = pkgconfig.parse('cblas')
[sagelib-9.1.rc5]   File "/var/lib/buildbot/slave/sage_git/build/local/lib/python3.7/site-packages/pkgconfig/pkgconfig.py", line 248, in parse
[sagelib-9.1.rc5]     _raise_if_not_exists(package)
[sagelib-9.1.rc5]   File "/var/lib/buildbot/slave/sage_git/build/local/lib/python3.7/site-packages/pkgconfig/pkgconfig.py", line 103, in _raise_if_not_exists
[sagelib-9.1.rc5]     raise PackageNotFoundError(package)
[sagelib-9.1.rc5] pkgconfig.pkgconfig.PackageNotFoundError: cblas not found
[sagelib-9.1.rc5] ************************************************************************
[sagelib-9.1.rc5] Error building the Sage library
[sagelib-9.1.rc5] ************************************************************************
[sagelib-9.1.rc5] Please email sage-devel (http://groups.google.com/group/sage-devel)
[sagelib-9.1.rc5] explaining the problem and including the relevant part of the log file
[sagelib-9.1.rc5]   /var/lib/buildbot/slave/sage_git/build/logs/pkgs/sagelib-9.2.beta0.log
[sagelib-9.1.rc5] Describe your computer, operating system, etc.
[sagelib-9.1.rc5] ************************************************************************
[sagelib-9.1.rc5] 
[sagelib-9.1.rc5] real	0m0.117s
[sagelib-9.1.rc5] user	0m0.100s
[sagelib-9.1.rc5] sys	0m0.012s

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1cfed7c

build/pkgs/sagelib/spkg-install: Do not poison SAGE_LOCAL, used in script installed by spkg pkgconf

[mkoeppe]

(simplification in follow-up ticket: #29779 new enhancement pkg-config installed from SPKG pkgconf should not depend on environment variable SAGE_LOCAL)

[dimpase]

kucalk buildbot -> I read on sage-devel that people suspect it to be running with --with-python=2, which of course is meaningless with Sage 9.2+. Perhaps we should make --with-python=2 trigger an error?

[mkoeppe]

Replying to dimpase:

kucalk buildbot -> I read on sage-devel that people suspect it to be running with --with-python=2, which of course is meaningless with Sage 9.2+. Perhaps we should make --with-python=2 trigger an error?

#29669 makes it an error already.

But the above is unrelated to python 2 vs python 3. This error showed up on systems where no system pkg-config is available. Fixed by 1cfed7c.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​38b6bcf	Merge tag '9.2.beta0' into t/29411/make_sagelib_a_script_package
​f9a30f6	build/pkgs/sagelib/spkg-install: Fix up error exits

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​305d8cf	Trac #29345: replace a bash array with something portable.
​d0dff56	Trac #29345: replace a few uses of "source" with "."
​5ac420b	Trac #29345: fix some bashisms in sage-env's resolvelinks() function.
​0a61795	Trac #29345: don't force SHELL=bash any longer.
​5db5318	Trac #29345: remove "break" statements from AC_SEARCH_LIBS.
​e810ad1	Trac #29345: don't use sage's config.status for the lrcalc build.
​93c9921	Trac #29345: replace the function that populates the CVXOPT_* variables.
​0e66a0a	Trac #29345: add Dima's SPKG patches for ksh compatibility.
​df3f05e	build/make/Makefile.in [SCRIPT_PACKAGE_templ]: cd into the SPKG directory; adjust spkg-install scripts
​5372065	Merge branch 't/29793/script_packages_should_cd_into_the_spkg_directory' into t/29411/make_sagelib_a_script_package

[mkoeppe]

Merged in #29793 to resolve a merge conflict. Needs review

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cc30471

build/bin/write-dockerfile.sh: Do not ADD removed file src/Makefile.in

[mkoeppe]

Ah, that's why this was failing on GitHub runs!

[mkoeppe]

Ready for review...

[dimpase]

why do I see [sagelib-9.1.rc5] while building, even though

$ cat VERSION.txt 
SageMath version 9.2.beta0, Release Date: 2020-05-28

[mkoeppe]

When this ticket is merged, src/bin/sage-update-version will update build/pkgs/sagelib/package-version.txt, where this comes from

[dimpase]

OK

[mkoeppe]

Thanks!

[dimpase]

rebased over rebased #29793

---

Last 10 new commits:

​aa206bf	src/Makefile.in: Merge into build/make/Makefile.in, build/pkgs/sagelib/spkg-install
​e052741	build/make/Makefile.in, build/pkgs/sagelib/dependencies: Make sure sagelib spkg-install is always run
​1e19861	build/pkgs/sagelib/spkg-install: Update references to tickets
​d6ee516	build/pkgs/sagelib/spkg-install: Also poison environment variables SAGE_LOCAL etc.
​8a4b60d	build/pkgs/sagelib/spkg-install: Add comment regarding SAGE_SPKG_INST
​4071eeb	build/pkgs/sagelib/spkg-install: Also poison SAGE_PKGCONFIG, SAGE_SPKG_SCRIPTS
​1707161	src/setup.py: Update comment (not specific to gcc)
​985bfca	build/pkgs/sagelib/spkg-install: Do not poison SAGE_LOCAL, used in script installed by spkg pkgconf
​b212ce8	build/pkgs/sagelib/spkg-install: Fix up error exits
​5790687	build/bin/write-dockerfile.sh: Do not ADD removed file src/Makefile.in

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​539c182

build/make/install: Do not depend on src/bin/sage-version.sh

[mkoeppe]

Pushed to wrong ticket, sorry

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:

[dimpase]

this has caused a ​breakage of jupyter notebook/server. Apparently some of its dependencies get "blessed" by ./sage -b, but no longer get blessed by make build.

---

Running 'make build' on that ticket's commit in the develop branch (0bcd001f760ff724832c32b4f65fa575fd82e000 make sagelib a script package) produces the error for me whereas doing so on the immediately previous commit (10fb62408a9761bf91a95d9eb7d7f16a104924c7 Trac #29289: Remove support for installing old-style SPKGs, deprecated in Sage 6.9) does not. Interestingly, it's been sort of hit-and-miss on whether the error shows up when doing just a './sage -b'.

On Saturday, July 11, 2020 at 5:26:02 PM UTC-7, Paul Masson wrote:

Just built a ticket based on 9.2.beta3 and the same problem occurs.

On Saturday, July 11, 2020 at 4:09:56 PM UTC-7, Paul Masson wrote:

After a successful incremental build from 9.2.beta2, trying to start a new notebook results in the browser console error "Failed to load resource: the server responded with a status of 404 (Not Found)" when trying to fetch Mathjax from ​http://localhost:8888/nbextensions/mathjax/MathJax.js. Has anyone other than Dima seen this also?

This appears to be a serious server configuration error. I don't see any ticket in either this beta or the previous one that would cause this. Am I missing some new configuration step? Sage built and runs just fine.

[mkoeppe]

This is now #30123

[jhpalmieri]

This broke sage -b: it wants to run make in a directory with no Makefile.