Install sage_setup with setuptools as a prerequisite for building/installing sagelib

[mkoeppe]

sage_setup will be used by the build system of various distribution packages (#29705).

So we make it a regular distribution package itself, with sources in pkgs/sage-setup/ (#31577) and install it with pip.

The actual Python source files are not moved because sage_setup takes part in Sage doctesting (see #31577).

sagelib's setup.py will no longer install sage_setup.

sage-setup will become a build-system requires of sagemath-standard (declared in pyproject.toml)

Follow-up:

• #29924: Do not fail if sage.env, sage.* are not available

---

References:

python - pip ignores dependency_links in setup.py - Stack Overflow ​https://stackoverflow.com/questions/12518499/pip-ignores-dependency-links-in-setup-py

dependency_links flag ignored when package exists on PyPI · Issue #987 · pypa/setuptools ​https://github.com/pypa/setuptools/issues/987

Suggest alternatives for --process-dependency-links · Issue #6162 · pypa/pip ​https://github.com/pypa/pip/issues/6162

PEP 508 -- Dependency specification for Python Software Packages | Python.org ​https://www.python.org/dev/peps/pep-0508/#pep440

PEP 440 -- Version Identification and Dependency Specification | Python.org ​https://www.python.org/dev/peps/pep-0440/#local-version-segments

[mkoeppe]

Last 10 new commits:

​eed920e	src/sage/tests: Move Extension options from src/module_list.py to distutils directives
​0d25d1d	src/sage/structure: Move Extension options from src/module_list.py to distutils directives
​808f46a	src/sage/stats: Move Extension options from src/module_list.py to distutils directives
​10f7542	src/sage/schemes: Move Extension options from src/module_list.py to distutils directives
​5867c05	src/sage/modular/pollack_stevens/dist.pyx: Add missing distutils directives
​c536daa	Remove self-listing in distutils sources directive
​b1b3787	sage.env.cython_aliases: Fix for systems without zlib pc
​2ba1d7b	Merge branch 't/29706/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files' into t/29785/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_3__get_rid_of_uname_specific_
​bb6cc0c	Merge branch 't/29785/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_3__get_rid_of_uname_specific_' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup
​4ce81bf	sage_setup.command.sage_build_cython: Use SAGE_GMP_PREFIX, SAGE_NTL_PREFIX for header dependencies instead of SAGE_INC

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​041c9e8	sage_setup.command.sage_build_cython: Use SAGE_GMP_PREFIX, SAGE_NTL_PREFIX for header dependencies instead of SAGE_INC
​4a9dd3a	Merge branch 't/29855/sagelib_setup_py__fix_dependencies_on_header_files_of_packages_gmp__ntl' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup

[mkoeppe]

[sagelib-9.1.rc5] running build
[sagelib-9.1.rc5] Generating auto-generated sources
[sagelib-9.1.rc5] Building interpreters for fast_callable
[sagelib-9.1.rc5] -> Rebuilding interpreters because /Users/mkoeppe/s/sage/sage-rebasing/src/sage_setup/autogen/interpreters/memory.py changed
[sagelib-9.1.rc5] running build_cython
[sagelib-9.1.rc5] Enabling Cython debugging support
[sagelib-9.1.rc5] Updating Cython code....
[sagelib-9.1.rc5] Compiling sage/algebras/finite_dimensional_algebras/finite_dimensional_algebra_element.pyx because it changed.
[sagelib-9.1.rc5] error: [Errno 21] Is a directory: '/Users/mkoeppe/s/sage/sage-rebasing/src/sage/cpython'

Hm....

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7582906	build/pkgs/sage_setup: New
​010c6e3	src/setup.py: Do not install sage_setup; instead, make the sage_setup package a dependency

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​18aae7f	sage_setup.command.sage_build_cython: Fix up - add list brackets
​2d0d0a7	Merge branch 't/29855/sagelib_setup_py__fix_dependencies_on_header_files_of_packages_gmp__ntl' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup

[mkoeppe]

Replying to mkoeppe:

[sagelib-9.1.rc5] error: [Errno 21] Is a directory: '/Users/mkoeppe/s/sage/sage-rebasing/src/sage/cpython'

Hm....

That was coming from the mistake in #29855.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0459006

.gitignore: Ignore generated files in build/pkgs/*/src

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c66384d

build/pkgs/sage_setup/spkg-install: Fixup use of --root

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​58956e6

build/pkgs/sage_setup/spkg-src: New

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0e17023

Merge tag '9.2.beta2' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0d22cef

build/pkgs/sage_setup/spkg-src: chmod +x

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​21eb31e

build/pkgs/sage_setup/src/tox.ini, requirements.txt: New

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​eeeaa47

build/pkgs/sage_setup/src/setup.cfg: Add nested packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4dcaf1c

build/pkgs/sage_setup/src/setup.cfg: Add remaining nested packages

[dimpase]

should we wait for #29855, #30011 merged?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​212d183	trac 30011: remove sage_setup/fpickle_setup.py
​0ef60f5	Merge branch 'u/jhpalmieri/sage_setup__remove_use_of_six' of git://trac.sagemath.org/sage into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup
​44cc65d	build/pkgs/sage_setup/dependencies: New

[mkoeppe]

This ticket is not so urgent (I've set its milestone to 9.3)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​8aa6fd9	build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it
​d369aab	build/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install
​2d435ab	build/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel
​55993b6	build/bin/sage-dist-helpers: Fixup
​0a64674	build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
​ca58693	build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
​65773f7	Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30010/split_sage_setup_docbuild_out_to_a_separate_package
​14ab446	build/pkgs/sage_setup_docbuild/spkg-install: Install via setup.py bdist_wheel
​3c5584d	Merge branch 't/30010/split_sage_setup_docbuild_out_to_a_separate_package' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup
​8d465d7	build/pkgs/sage_setup/spkg-install: Install via setup.py bdist_wheel

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​db56301

build/pkgs/sagelib/src/requirements.txt: Update

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​650c500	build/pkgs/sage_setup/src: Set version, remove sage_setup.docbuild package
​1bb5b34	build/pkgs/sage_setup/type: Change to standard

[mkoeppe]

Hoping we can make progress on this ticket this week - ​https://wiki.sagemath.org/days111

[gh-kliem]

Needs rebase.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​d3a8f5f	build/pkgs/sage_setup/spkg-src: New
​9c71594	build/pkgs/sage_setup/spkg-src: chmod +x
​23cd1d9	build/pkgs/sage_setup/src/tox.ini, requirements.txt: New
​ae23b43	build/pkgs/sage_setup/src/setup.cfg: Add nested packages
​7c7b38a	build/pkgs/sage_setup/src/setup.cfg: Add remaining nested packages
​2e9b850	build/pkgs/sage_setup/dependencies: New
​5511555	build/pkgs/sage_setup/spkg-install: Install via setup.py bdist_wheel
​3e28f39	build/pkgs/sage_setup/src: Set version, remove sage_setup.docbuild package
​4d387bb	build/pkgs/sage_setup/type: Change to standard
​2b25c06	Move sage_setup distribution source to pkgs/sage-setup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​287372a	src/pyproject.toml.m4: Add sage_setup as build-system requires
​f51ca1c	src/setup.cfg.m4: Change license_file to license_files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​35e38cd

Fixup

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​a1d33bc	build/pkgs/pillow/spkg-install.in, build/bin/sage-dist-helpers: Remove debugging code
​413f6b5	build/pkgs/{jupyter_jsmol,python_igraph,pillow}/spkg-install.in: Use --build-option, not --global-option
​1e993d4	Fixup
​9ec5bc2	pkgs/sagemath-standard/setup.py, src/setup.py, src/MANIFEST.in: Exclude sage_setup
​9ec930c	Update paths in comments
​d8d5755	pkgs/sage-setup/tox.ini: Update paths in comments
​b4b5c19	build/bin/write-dockerfile.sh: ADD pkgs
​40abff0	pkgs/sage-setup/setup.cfg: README.rst, not README.md
​782b64b	pkgs/sagemath-standard: Remove symlink to SAGE_ROOT/src/sage_setup
​cf606f7	build/bin/write-dockerfile.sh: ADD src/sage_setup before building TARGETS_PRE

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a1b8832

build/pkgs/sage_setup/spkg-src: Use python3

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9270417	tox.ini, build/bin/write-dockerfile.sh, tox*.yml: Use all-sage-local as default TARGETS_PRE
​870f9aa	build/pkgs/memory_allocator/install-requires.txt: Fix filename
​61c25a0	build/pkgs/sage_conf/install-requires.txt: New

[dimpase]

ok, fine

[mkoeppe]

Thanks!

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​7f0bd4a

Merge tag '9.4.beta4' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup

[vbraun]

***********************************************
make[1]: Entering directory '/home/release/Sage/build/make'
make --no-print-directory sagelib-SAGE_VENV-no-deps
cd '/home/release/Sage/build/pkgs/sagelib' && . '/home/release/Sage/src/bin/sage-src-env-config' && . '/home/release/Sage/src/bin/sage-env-config' && . '/home/release/Sage/src/bin/sage-env' && . '/home/release/Sage/build/bin/sage-build-env-config' && . '/home/release/Sage/build/bin/sage-build-env' && SAGE_SPKG_WHEELS=/home/release/Sage/local/var/lib/sage/wheels SAGE_INST_LOCAL=/home/release/Sage/local sage-logger -p '/home/release/Sage/build/pkgs/sagelib/spkg-install' '/home/release/Sage/logs/pkgs/sagelib-9.4.beta4.log'
[sagelib-9.4.beta4] Traceback (most recent call last):
[sagelib-9.4.beta4]   File "/home/release/Sage/pkgs/sagemath-standard/setup.py", line 31, in <module>
[sagelib-9.4.beta4]     from sage_setup.excepthook import excepthook
[sagelib-9.4.beta4] ModuleNotFoundError: No module named 'sage_setup.excepthook'
[sagelib-9.4.beta4] 
[sagelib-9.4.beta4] real	0m0.140s
[sagelib-9.4.beta4] user	0m0.121s
[sagelib-9.4.beta4] sys	0m0.017s
make[2]: *** [Makefile:2693: sagelib-SAGE_VENV-no-deps] Error 1
make[1]: *** [Makefile:2693: /home/release/Sage/local/var/lib/sage/installed/sagelib-9.4.beta4] Error 2
make[1]: Leaving directory '/home/release/Sage/build/make'

real	0m0.257s
user	0m0.222s
sys	0m0.037s
***************************************************************
Error building Sage.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​35b8b74

Merge tag '9.4.beta5' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup

[mkoeppe]

I can't reproduce this error. Possibly from forgetting to run bootstrap and configure.

[jhpalmieri]

I don't know if this is worth a ticket or if there is a relevant one already, but running ./bootstrap produces various warnings:

m4/sage_spkg_configures.m4:513: warning: The macro `AC_PROG_CC_C99' is obsolete.
m4/sage_spkg_configures.m4:516: warning: The macro `AC_FOREACH' is obsolete.
m4/sage_spkg_configures.m4:516: warning: The macro `AC_FOREACH' is obsolete.
m4/sage_spkg_configures.m4:530: warning: The macro `AC_FOREACH' is obsolete.
m4/sage_spkg_configures.m4:594: warning: The macro `AC_TRY_RUN' is obsolete.
m4/sage_spkg_configures.m4:594: warning: The macro `AC_TRY_LINK' is obsolete.
configure.ac:447: warning: back quotes and double quotes must not be escaped in: multiple installation records for $SPKG_NAME: m4_newline($(ls -l "$SAGE_SPKG_INST/$SPKG_NAME"-*)) m4_newline([only one should exist, so please delete some or all of these files and re-run \"$srcdir/configure\"])

[jhpalmieri]

Replying to mkoeppe:

I can't reproduce this error. Possibly from forgetting to run bootstrap and configure.

In the meantime, are you waiting for more information from Volker, or should this go back to "positive review"?

[mkoeppe]

Replying to jhpalmieri:

I don't know if this is worth a ticket or if there is a relevant one already, but running ./bootstrap produces various warnings

I think this is #30668

[mkoeppe]

Replying to jhpalmieri:

Replying to mkoeppe:

I can't reproduce this error. Possibly from forgetting to run bootstrap and configure.

In the meantime, are you waiting for more information from Volker, or should this go back to "positive review"?

Let's do "positive review". Easier to run it just as part of Volker's integration tests.

[vbraun]

Merge conflict

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​8feca2a	Merge tag '9.4.beta5' into t/32073/deprecate_sage_libs_readline
​ed14848	Remove readline lib file from the documentation.
​773ef29	Merge #32073
​28d6622	build/bin/sage-spkg: Remove dead code for old-style spkg
​3b435d9	build/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): Record name of installed distribution name
​34bd445	sdh_pip_uninstall (sdh_pip_uninstall): New helper functtion; use same flags in 'make SPKG-clean' for pip packages
​49af8aa	build/bin/sage-spkg, build/sage_bootstrap/uninstall.py: Prepare/install/use the spkg-piprm script
​ecd49fb	build/bin/sage-dist-helpers, src/doc/en/developer/packaging.rst: Document sdh_pip_uninstall
​c3eeb69	build/bin/sage-spkg: Fix typo in comment, unindent a block
​9aa1f10	Merge #29585

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​8c5ee65	build/pkgs/pip/install-requires.txt: Update lower bound
​ed5f9f0	Merge tag '9.4.beta6' into t/32046/use_pip___use_feature_in_tree_build__replace_use_of_sdh_setup_bdist_wheel_by_sdh_pip_install
​6b1b222	Merge #32046

[mkoeppe]

The branch already contained a part of #32046 but the dependency was not recorded. Fully merged the latest version of #32046. I think we can defer this ticket to 9.5

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​85d1d27	Merge tag '9.4.rc1' into t/32046/use_pip___use_feature_in_tree_build__replace_use_of_sdh_setup_bdist_wheel_by_sdh_pip_install
​050f8df	build/pkgs/numpy/spkg-install.in: Use sdh_pip_install, sdh_prefix_args
​a41b507	Merge branch 't/32046/use_pip___use_feature_in_tree_build__replace_use_of_sdh_setup_bdist_wheel_by_sdh_pip_install' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​290fa2f	build/pkgs/sagelib/src: Restore lost symlink
​478ffcd	Merge #32046

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b2368c6

pkgs/sage-setup/setup.cfg: Add pkgconfig as install_requires

[jhpalmieri]

I'm getting a lot of doctest failures:

sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/utils.py  # 15 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/find.py  # 38 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/specs/rdf.py  # 11 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/optional_extension.py  # 12 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/__init__.py  # 5 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/generator.py  # 70 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/clean.py  # 13 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/storage.py  # 97 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/memory.py  # 57 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/specs/cdf.py  # 9 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/specs/cc.py  # 20 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/specs/python.py  # 25 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/instructions.py  # 39 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/specs/rr.py  # 21 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/specs/base.py  # 22 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/autogen/interpreters/specs/element.py  # 11 doctests failed
sage -t --long --warn-long 100.8 --random-seed=0 src/sage_setup/util.py  # 6 doctests failed

The directory local/lib/python3.9/site-packages/sage_setup doesn't have much in it anymore, and I'm guessing that this causes the problems.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​db8e30d	Merge tag '9.5.beta0' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup
​37f9b0c	build/make/Makefile.in: For script packages, make sure that 'sage -f' uninstalls first
​456d8a9	build/make/Makefile.in (sage_setup-clean): Clean the build tree
​f84730e	sage_setup.clean: Do not clean the sage_setup installation in site-packages

[mkoeppe]

Thanks for catching this. When configure --enable-editable is not used, the "cleaner" erased sage_setup's files at the end of the sagelib installation. Fixed now, tamed the cleaner.

[dimpase]

seems to work for me (without --enable-editable).

[mkoeppe]

Thank you!