Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420)

[mkoeppe]

This ticket prepares the modularization of sagelib into separate distributions (pip-installable packages) by adding support for ​native namespace packages (PEP 420) to sagelib's build system.

Ordinary packages (directories of modules) are turned into namespace packages by removing the empty __init__.py file.

Then a different distribution (distutils package) can share the same namespace.

In the first stage of modularization, planned for Sage 9.3, we use this new mechanism to split out all "optional extension modules" (which depend on some optional library) from the sagelib distribution to separate pip-installable packages. This will simplify deployment and dependency management.

(The source code remains in the same place in src/sage in the git tree; the new pip-installable packages will have separate metadata in build/pkgs/SPKG.)

1. To this end, this ticket changes the following packages to namespace packages (by removing __init__.py):
   • sage.matrix (this allows the new distribution sage-meataxe introduced in #30151 to add the module sage.matrix.matrix_gfpn_dense) to the package)
   • sage.libs (for .meataxe - #30151, .sirocco, .coexter3, .fes)
   • sage.interfaces (for .primecount)
   • sage.graphs (for .bliss, .mcqd)
   • sage.graphs.graph_decompositions (for .tdlib -> sage-tdlib, #29864.)
   • sage.numerical.backends. (This is preparation for later work, in #30152.)

2. Because our source discovery mechanism in sagelib's build system (setup.py + sage_setup) distinguishes package directories from other directories containing data files by the presence of the __init__.py file, the present ticket makes several adjustments to the build system - in particular to the functions find_python_sources and find_extra_files.

To simplify the implementation and to allow for self-documentation of the directory structure, we introduce sage_setup-specific files named:

• namespace to mark a directory without __init__.py as a namespace package;
• nonamespace to mark a directory as a data directory. (This is in analogy with the nodoctest files that control the Sage doctester.)

(In the branch of this ticket, these are empty files; but they could contain comments that explain the purpose of a particular directory.)

3. Also the incremental installation mechanisms of sagelib's build system need adjustment. clean_stale_files, which deletes installed Python modules in $SAGE_LIB/sage = ..../site-packages/sage that are no longer present in the current version of the source tree, is changed so that it does not delete modules in a namespace package that are provided by a separate distribution (pip-installable package).

Follow-up tickets:

• Wishlist item: Warn or give errors if the sage_setup: distribution directives, namespace and nonamespace files indicate an inconsistent structure such as when it was forgotten to remove __init__.py.
• #30152 will make further changes to the cleaner that will allow out-of-tree namespace packages

See also:

• #29705: Meta-ticket: Modularize sagelib into separate distutils packages
• Initial discussion of namespace packages for Sage in #28175 and ​https://github.com/mkoeppe/sage-numerical-backends-coin/pull/4
• #21654: src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory

[mkoeppe]

pushing these forward to 9.2

[mkoeppe]

New commits:

​8e50bbe	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​dff6e62	Mark src/sage/ext_data as directories that are not namespace packages
​8bc5662	sage_setup.find.find_python_sources: Handle native namespace packages and 'nonamespace' files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​606a509

sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

[mkoeppe]

Next steps: adjust the cleaner, find Cython sources

[mkoeppe]

The Cython source finder could perhaps also parse a new directive that controls OptionalExtension behavior (until we get rid of OptionalExtension altogether in #29701) so that we can get rid of module_list.py completely (see #29706)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a7b2287

find_python_sources: New optional argunment sage_packages, filter by # distutils: sage_package = PACKAGE

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​194d3b4

sage_setup.find.find_python_sources: Also find Cython modules

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​adc39e3	WIP
​ad3ebad	Merge tag '9.2.beta0' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​004dd52

Merge tag '9.2.beta0' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3a68b37

src/sage_setup/find.py: Switch to parsing directive 'sage_setup: distribution = PKG'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d900139

src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6d71e74

Add doctest

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​0a7fb48	Move src/fpickle_setup.py to srs/sage_setup/
​f4a9269	src/setup.py: Move class sage_install to new module sage_setup.command.sage_install
​5660d8f	src/module_list.py: Move library_order to new module sage_setup.library_order
​43d4f83	src/setup.py: Move classes sage_build_* to new modules sage_setup.command.sage_*
​09e3297	src/setup.py: Move excepthook to new module sage_setup.excepthook
​1cfed7c	build/pkgs/sagelib/spkg-install: Do not poison SAGE_LOCAL, used in script installed by spkg pkgconf
​62c7531	Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
​b3102e9	Merge tag '9.2.beta0' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
​5066f83	Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​f670f81	WIP

[mkoeppe]

Next: Fix find_extra_files so that it finds the .pxd files in sage.numerical.backends

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​bd85827

is_package_or_namespace_package_dir: New, monkeypatch into Cython.Build.Dependencies

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​532bb21

is_package_or_namespace_package_dir: Make self-contained, patch into more of Cython

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2b42029	src/sage_setup/command/__init__.py: New
​a830304	src/sage_setup/command/sage_build_cython.py: Fix adding setup.py as dependency
​d3080a5	Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​0aa3693	Mark src as nonamespace

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​47cb578

is_package_or_namespace_package_dir: Recognize namespace packages by file 'namespace'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​381a507	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​4c51133	sage_setup/command/sage_install.py: WIP: Disable cleaning

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c0a2ed7

src/module_list.py: Remove explicit listing of sage.graphs.graph_decompositions extensions

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​a43bf07	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​7a6d12f	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files, Cython modules, directive 'sage_setup: distribution = PKG'
​f4a6c7d	src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
​cee98e7	Add doctest
​7873513	is_package_or_namespace_package_dir: New, monkeypatch into Cython
​d01495e	Mark src as nonamespace
​899dc32	is_package_or_namespace_package_dir: Recognize namespace packages by file 'namespace'
​b97b1e0	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​4b8181b	sage_setup/command/sage_install.py: WIP: Disable cleaning
​40bc473	src/module_list.py: Remove explicit listing of sage.graphs.graph_decompositions extensions

[mkoeppe]

Squashed some commits

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​226aa5a	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​906fb23	Mark src and src/sage/ext_data as directories that are not namespace packages
​4a2eee0	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files, Cython modules, directive 'sage_setup: distribution = PKG'
​a8b0316	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​92bd8ae	src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
​6647d09	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​975b4a0	sage_setup/command/sage_install.py: WIP: Disable cleaning

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​5d5803e	src/sage/graphs/planarity.pyx: Add forgotten distutils directive
​a0be9b6	Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29701/replace_use_of_module_list_optionalextension
​d0eb84e	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​c82fb01	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​d90e3b6	Mark src and src/sage/ext_data as directories that are not namespace packages
​f598e04	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​f522b09	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​50df6d5	src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
​e34c028	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​650c661	sage_setup/command/sage_install.py: WIP: Disable cleaning

[mkoeppe]

Rebased on top of #29701

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​14396af	Merge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension
​d62da15	sage_setup.command.sage_build: Add the extensions to the distribution
​06a3609	sage_setup.find.find_python_sources: Add benchmark doctest
​174626c	Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension
​034a7f7	Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension
​d3c608b	Merge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension
​d99e942	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​474b200	Merge tag '9.2.beta5' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​eaadf77	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​59151ab	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​713a838	Mark src and src/sage/ext_data as directories that are not namespace packages
​4faff43	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​de17c1c	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​6761b7d	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​65f3739	sage_setup/command/sage_install.py: WIP: Disable cleaning
​8c20f3e	Revert "sage_setup/command/sage_install.py: WIP: Disable cleaning"
​37bd1ca	sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install
​2e79d33	WIP

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​8a19fe2	build/make/Makefile.in (sagelib-clean): Clean the new build location
​ccc67b0	src/sage_setup: Update cythonized_dir in doctests
​df38027	Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30277/remove_src_module_list_py
​ef4e554	Merge branch 't/30277/remove_src_module_list_py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​6282b8d	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​7c10680	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​6e68259	Mark src and src/sage/ext_data as directories that are not namespace packages
​b71c3a6	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​3a12146	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​98a1b88	sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​32a86d2	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​742ab87	WIP

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​776c979	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​2e61ada	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​26b1ca0	Mark src and src/sage/ext_data as directories that are not namespace packages
​a733af5	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​c08c511	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​4407e5e	sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install
​a13c126	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​a967aca	WIP

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​7980130	Merge branch 't/30779/duplicate_src_setup_py' into t/30780/sage_setup__separate_installing_and_cleaning
​9b95628	sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in build/pkgs/sagelib/src/setup.py; refactor through new class sage_install
​886ef6d	Merge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​3b9559d	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​88085b1	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​9c39a76	Mark src and src/sage/ext_data as directories that are not namespace packages
​b060462	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​fe9a7d5	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​d2fe380	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​a4e5683	WIP

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d42aa12	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​29ca24b	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​1957acd	src/sage_setup/clean.py: Update comment
​8b37214	WIP

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​92f1145	sage_install_and_clean: If all_distributions is a list, do not clean files that do not belong to this list
​44a4456	src/sage_setup/command/sage_install.py: Also handle modules in namespace packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ad9402f	sage_setup.command.sage_install: Fixup: Remove cleaning code (moved to sage_install_and_clean)
​88455dc	Merge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a9933b6

sage_build_cython: Pass built_distributions to find_extra_files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ea0d15a	sage.matrix: Resolve circular imports without using __init__.py
​e4ce720	Merge branch 't/30784/sage_matrix__resolve_circular_imports_without_using___init___py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6aaa958

sage_install_and_clean: Move all_distributions into class

[mkoeppe]

Ready for review

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​39be639

FileDocTestSource.in_lib: Handle namespace packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​64c5120

find_python_sources: Also collect namespace packages - so that package_data works

[gh-tobiasdiez]

What's the advantage/reason to introduce the sage-specific namespace/nonamespace convention instead of following PEP 420 and keep the __init__.py file as a marker for namespace packages together with setuptools.find_namespace_packages()?

[mkoeppe]

Replying to gh-tobiasdiez:

What's the advantage/reason to introduce the sage-specific namespace/nonamespace convention instead of following PEP 420 and keep the __init__.py file as a marker for namespace packages together with setuptools.find_namespace_packages()?

One cannot keep __init__.py for native namespace packages. PEP 420 is quite clear about it. Sub-packages of namespace packages do have __init__.py; and also the present ticket keeps them.

[mkoeppe]

find_namespace_packages accepts any directory as a package directory:

class PEP420PackageFinder(PackageFinder):
    @staticmethod
    def _looks_like_package(path):
        return True

(​https://github.com/pypa/setuptools/blob/f991fbb3c9d0e10a0a78ae2b508b3fd99f9cdef2/setuptools/__init__.py#L110)

But our source tree has some directories that should not be considered package directories. I have marked these with the nonamespace file.

Having both nonamespace and namespace files results in simpler changes to the existing code.

[gh-tobiasdiez]

Oh ok, sorry, I've misunderstood the ticket description. You are of course right, implicit namespace packages don't have a __init__.py file.

Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages(), and then filter out unwanted folders based on a hard-coded list. The (no)namespace files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by ​PEP 382, which was rejected in favor of PEP 420.

If you do prefer to have these files, I would like to ask you to add documentation to the dev docs.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3c7d675

src/sage_setup/find.py (find_python_sources): Update doc, doctests

[mkoeppe]

Replying to gh-tobiasdiez:

Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages(), and then filter out unwanted folders based on a hard-coded list. The (no)namespace files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by ​PEP 382, which was rejected in favor of PEP 420.

Thanks for the input and the pointer to PEP 382. I'll try out if your suggestion can be implemented without major changes.

[mkoeppe]

TBD: In incremental builds, the cleaner fails to remove previously installed __init__.py files:

**********************************************************************
File "src/sage_setup/find.py", line 343, in sage_setup.find.installed_files_by_module
Failed example:
    files_by_module['sage.graphs.graph_decompositions']
Expected nothing
Got:
    {'sage/graphs/graph_decompositions/__init__.py',
     'sage/graphs/graph_decompositions/__pycache__/__init__.cpython-37.pyc'}
**********************************************************************

[mkoeppe]

Note that one annoying thing about find_namespace_packages when applied to our source tree is that we have lots of directories that do not contain any python files, leading to output like this from distutils:

[sagelib-9.2.rc2] package init file 'sage/schemes/generic/notes/__init__.py' not found (or not a regular file)

The branch on this ticket handles such directories specially (no need for nonamespace files there). If there is a solution that could silence these notes, the code could be brought closer to find_namespace_packages behavior.

[gh-tobiasdiez]

find_namespace_packages excepts include and exclude directories as arguments. So one option would be to get a list of all folders not including any python files, and pass this as exclude (or the same approach using a whitelist).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a6e8447

Merge tag '9.3.beta3' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[mkoeppe]

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

Setting a new milestone for this ticket based on a cursory review.