Modularization of sagelib: Break out a separate package sagemath-meataxe

[mkoeppe]

... in analogy of #29864 (sage-tdlib).

To try: make sagemath_meataxe

(or ./configure --enable-sagemath_meataxe && make build)

This is part of #29705: Meta-ticket: Modularize sagelib into separate distributions (distutils packages)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​16c60ca	src/setup.py: Remove meataxe - now taken care of by package sage_meataxe
​b040746	fixup

[SimonKing]

Can you elaborate on what is supposed to be done to use meataxe --- both as a C library, as collection of executables, and as optional Cython extension module?

[mkoeppe]

spkg meataxe stays as is. It installs the C library and executables. No dependency on anything from Sage.

spkg sagelib will no longer conditionally build the Cython extension module.

Instead, the new spkg sage_meataxe will build the Cython extension module. It declares both sagelib and meataxe as a dependency.

sage_meataxe, in the implementation of this ticket, does not have a source tarball (note there is no checksums.ini). Instead, it is a "script" package, whose sources lie in build/pkgs/sage_meataxe/src. It has the structure of a Python package (distribution), with setup.py doing the build and installation. The actual Cython sources remain in the src/sage tree (using symlinks). (It is possible to build a pip-installable source distribution tarball as well, using the spkg-src script. Eventually, we will upload it to PyPI.)

[SimonKing]

Replying to mkoeppe:

spkg sagelib will no longer conditionally build the Cython extension module.

Instead, the new spkg sage_meataxe will build the Cython extension module. It declares both sagelib and meataxe as a dependency.

So, in order to use it, one has to do make sage_meataxe, right? It would automatically install its dependency meataxe, would build the corresponding Cython extension module, AND would also build the documentation of the Cython extension module? And sage -testall would run the doctests from the corresponding Cython extension module if and only if sage_meataxe is installed, so that there is no need to mark all its tests as "optional"?

If the documentation and test suite would be taken care of, it would indeed be very helpful.

[mkoeppe]

Replying to SimonKing:

Replying to mkoeppe:

spkg sagelib will no longer conditionally build the Cython extension module.

Instead, the new spkg sage_meataxe will build the Cython extension module. It declares both sagelib and meataxe as a dependency.

So, in order to use it, one has to do make sage_meataxe, right? It would automatically install its dependency meataxe, would build the corresponding Cython extension module, AND would also build the documentation of the Cython extension module?

Yes.

And sage -testall would run the doctests from the corresponding Cython extension module if and only if sage_meataxe is installed, so that there is no need to mark all its tests as "optional"?

Yes, I would hope so. I have just created #30778 for this idea.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​b060462	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​d42aa12	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​29ca24b	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​1957acd	src/sage_setup/clean.py: Update comment
​8b37214	WIP
​92f1145	sage_install_and_clean: If all_distributions is a list, do not clean files that do not belong to this list
​44a4456	src/sage_setup/command/sage_install.py: Also handle modules in namespace packages
​706bf59	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe
​f096870	build/pkgs/sage_meataxe: New
​1e03fef	src/setup.py: Remove meataxe - now taken care of by package sage_meataxe

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ad9402f	sage_setup.command.sage_install: Fixup: Remove cleaning code (moved to sage_install_and_clean)
​88455dc	Merge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​aacec42	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​800f740	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe
​3823197	build/pkgs/sage_meataxe: New
​ac10389	build/pkgs/sagelib/src/setup.py: Remove meataxe - now taken care of by package sage_meataxe

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​12b22d9

fixup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​500635d

fixup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3d3cc42

build/pkgs/sage_meataxe/spkg-install: Install via bdist_wheel

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​99f3cc7	build/pkgs/sage_meataxe/src/setup.py: Set built_distributions
​a9933b6	sage_build_cython: Pass built_distributions to find_extra_files
​10e2432	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe

[mkoeppe]

The installation seems to work, but Sage cannot start because of a circular import related from sage.matrix.args. This needs a bit more work.

[SimonKing]

#28711 previously depended on this ticket. However, because of giving a talk on p_group_cohomology at the next Sage Days, I am interested in getting p_group_cohomology up and running ASAP.

Therefore, I suggest to make this ticket depend on #28711, rather than the other way around. As soon as the circular import problem is solved, one can modify SAGE_ROOT/build/pkgs/p_group_cohomology/dependencies (same for any other package depending on meataxe, if there is any); I believe this change logically belongs here, because it is THIS ticket that makes the change in the dependencies files possible and necessary.

[mkoeppe]

No objection here. This ticket certainly won't make it into Sage 9.2. (Not sure if #28711 can make it - it looks like Volker does not intend to merge non-"blocker" tickets.)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ea0d15a	sage.matrix: Resolve circular imports without using __init__.py
​e4ce720	Merge branch 't/30784/sage_matrix__resolve_circular_imports_without_using___init___py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​202914a	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​33e0eca

build/pkgs/sage_meataxe: Add metadata

[mkoeppe]

Ready for review

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6c408b5	build/pkgs/sagelib/src/setup.py: Do not clean files that do not belong to us
​6aaa958	sage_install_and_clean: Move all_distributions into class
​98abe7e	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​39be639	FileDocTestSource.in_lib: Handle namespace packages
​fb56378	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​64c5120	find_python_sources: Also collect namespace packages - so that package_data works
​ebd5ae7	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4f5a810	Upgrade p_group_cohomology to version 3.3.2
​de8adfa	Revert a change to "dependencies" of p_group_cohomology
​cf60884	Add upstream_url to p_group_cohomology
​ae50157	p_group_cohomology's upstream changed by some new commits
​bd34c47	build/pkgs/p_group_cohomology/checksums.ini: Fix upstream_url
​245560d	Fix docbuild of p_group_cohomology
​f4e5f6a	Fix a doctest in sage/tests/modular_cohomology.py
​649ba8d	Merge branch 'u/SimonKing/fix_spkg_install_of_p_group_cohomology' of git://trac.sagemath.org/sage into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe
​6cbcbec	build/pkgs/sage_meataxe: Move dependencies here from build/pkgs/p_group_cohomology/dependencies, add sage_meataxe there

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b984eeb

build/pkgs/p_group_cohomology/spkg-install.in: Remove check for sage-meataxe modules

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3c7d675	src/sage_setup/find.py (find_python_sources): Update doc, doctests
​11637a6	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7cedb3d

build/pkgs/sagelib/src/setup.py: Fix up all_distributions

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a6e8447	Merge tag '9.3.beta3' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​ebe1dd6	Merge branch 't/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_' into t/30151/modularization_of_sagelib__break_out_a_separate_package_sage_meataxe

[mkoeppe]

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

Setting a new milestone for this ticket based on a cursory review.