Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420)

[mkoeppe]

This ticket prepares the modularization of sagelib into separate distributions (pip-installable packages) by adding support for ​native namespace packages (PEP 420) to sagelib's build system.

Ordinary packages (directories of modules) can be turned into namespace packages by removing the empty __init__.py file. Then several distribution package can share the same namespace.

Because our source discovery mechanism in sagelib's build system (setup.py + sage_setup) distinguishes package directories from other directories containing data files by the presence of the __init__.py file, the present ticket makes several adjustments to the build system - in particular to the functions find_python_sources and find_extra_files.

Per convention in the Sage library, namespace packages are recognized by the presence of all.py or all__*.py files (#33033).

In this ticket, we apply it to turn sagemath-objects and sagemath-categories (from #29865) into namespace packages, in which __init__.py are removed from the MANIFEST.in files.

Because the actual files are not removed from the source tree, the normal build of the Sage distribution is not affected.

To test these two distributions:

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-objects && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-categories && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

Follow-up tickets:

• #33011 Remove the __init__.py files for packages designated to be namespace packages
• Wishlist item: Warn or give errors if the sage_setup: distribution directives, namespace and nonamespace files indicate an inconsistent structure such as when it was forgotten to remove __init__.py.
• #30152 will make further changes to the cleaner that will allow out-of-tree namespace packages

See also:

• #29705: Meta-ticket: Modularize sagelib into separate distutils packages
• Initial discussion of namespace packages for Sage in #28175 and ​https://github.com/mkoeppe/sage-numerical-backends-coin/pull/4
• #21654: src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory

[mkoeppe]

pushing these forward to 9.2

[mkoeppe]

New commits:

​8e50bbe	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​dff6e62	Mark src/sage/ext_data as directories that are not namespace packages
​8bc5662	sage_setup.find.find_python_sources: Handle native namespace packages and 'nonamespace' files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​606a509

sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

[mkoeppe]

Next steps: adjust the cleaner, find Cython sources

[mkoeppe]

The Cython source finder could perhaps also parse a new directive that controls OptionalExtension behavior (until we get rid of OptionalExtension altogether in #29701) so that we can get rid of module_list.py completely (see #29706)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a7b2287

find_python_sources: New optional argunment sage_packages, filter by # distutils: sage_package = PACKAGE

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​194d3b4

sage_setup.find.find_python_sources: Also find Cython modules

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​adc39e3	WIP
​ad3ebad	Merge tag '9.2.beta0' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​004dd52

Merge tag '9.2.beta0' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3a68b37

src/sage_setup/find.py: Switch to parsing directive 'sage_setup: distribution = PKG'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d900139

src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6d71e74

Add doctest

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​0a7fb48	Move src/fpickle_setup.py to srs/sage_setup/
​f4a9269	src/setup.py: Move class sage_install to new module sage_setup.command.sage_install
​5660d8f	src/module_list.py: Move library_order to new module sage_setup.library_order
​43d4f83	src/setup.py: Move classes sage_build_* to new modules sage_setup.command.sage_*
​09e3297	src/setup.py: Move excepthook to new module sage_setup.excepthook
​1cfed7c	build/pkgs/sagelib/spkg-install: Do not poison SAGE_LOCAL, used in script installed by spkg pkgconf
​62c7531	Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
​b3102e9	Merge tag '9.2.beta0' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
​5066f83	Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​f670f81	WIP

[mkoeppe]

Next: Fix find_extra_files so that it finds the .pxd files in sage.numerical.backends

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​bd85827

is_package_or_namespace_package_dir: New, monkeypatch into Cython.Build.Dependencies

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​532bb21

is_package_or_namespace_package_dir: Make self-contained, patch into more of Cython

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2b42029	src/sage_setup/command/__init__.py: New
​a830304	src/sage_setup/command/sage_build_cython.py: Fix adding setup.py as dependency
​d3080a5	Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​0aa3693	Mark src as nonamespace

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​47cb578

is_package_or_namespace_package_dir: Recognize namespace packages by file 'namespace'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​381a507	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​4c51133	sage_setup/command/sage_install.py: WIP: Disable cleaning

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c0a2ed7

src/module_list.py: Remove explicit listing of sage.graphs.graph_decompositions extensions

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​a43bf07	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​7a6d12f	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files, Cython modules, directive 'sage_setup: distribution = PKG'
​f4a6c7d	src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
​cee98e7	Add doctest
​7873513	is_package_or_namespace_package_dir: New, monkeypatch into Cython
​d01495e	Mark src as nonamespace
​899dc32	is_package_or_namespace_package_dir: Recognize namespace packages by file 'namespace'
​b97b1e0	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​4b8181b	sage_setup/command/sage_install.py: WIP: Disable cleaning
​40bc473	src/module_list.py: Remove explicit listing of sage.graphs.graph_decompositions extensions

[mkoeppe]

Squashed some commits

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​226aa5a	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​906fb23	Mark src and src/sage/ext_data as directories that are not namespace packages
​4a2eee0	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files, Cython modules, directive 'sage_setup: distribution = PKG'
​a8b0316	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​92bd8ae	src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
​6647d09	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​975b4a0	sage_setup/command/sage_install.py: WIP: Disable cleaning

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​5d5803e	src/sage/graphs/planarity.pyx: Add forgotten distutils directive
​a0be9b6	Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29701/replace_use_of_module_list_optionalextension
​d0eb84e	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​c82fb01	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​d90e3b6	Mark src and src/sage/ext_data as directories that are not namespace packages
​f598e04	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​f522b09	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​50df6d5	src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
​e34c028	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​650c661	sage_setup/command/sage_install.py: WIP: Disable cleaning

[mkoeppe]

Rebased on top of #29701

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​14396af	Merge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension
​d62da15	sage_setup.command.sage_build: Add the extensions to the distribution
​06a3609	sage_setup.find.find_python_sources: Add benchmark doctest
​174626c	Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension
​034a7f7	Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension
​d3c608b	Merge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension
​d99e942	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​474b200	Merge tag '9.2.beta5' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​eaadf77	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​59151ab	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​713a838	Mark src and src/sage/ext_data as directories that are not namespace packages
​4faff43	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​de17c1c	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​6761b7d	src/module_list.py: Remove explicit list of sage.numerical.backends extensions
​65f3739	sage_setup/command/sage_install.py: WIP: Disable cleaning
​8c20f3e	Revert "sage_setup/command/sage_install.py: WIP: Disable cleaning"
​37bd1ca	sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install
​2e79d33	WIP

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​8a19fe2	build/make/Makefile.in (sagelib-clean): Clean the new build location
​ccc67b0	src/sage_setup: Update cythonized_dir in doctests
​df38027	Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30277/remove_src_module_list_py
​ef4e554	Merge branch 't/30277/remove_src_module_list_py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​6282b8d	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​7c10680	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​6e68259	Mark src and src/sage/ext_data as directories that are not namespace packages
​b71c3a6	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​3a12146	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​98a1b88	sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​32a86d2	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​742ab87	WIP

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​776c979	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​2e61ada	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​26b1ca0	Mark src and src/sage/ext_data as directories that are not namespace packages
​a733af5	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​c08c511	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​4407e5e	sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install
​a13c126	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​a967aca	WIP

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​7980130	Merge branch 't/30779/duplicate_src_setup_py' into t/30780/sage_setup__separate_installing_and_cleaning
​9b95628	sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in build/pkgs/sagelib/src/setup.py; refactor through new class sage_install
​886ef6d	Merge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
​3b9559d	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
​88085b1	sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
​9c39a76	Mark src and src/sage/ext_data as directories that are not namespace packages
​b060462	sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
​fe9a7d5	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​d2fe380	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​a4e5683	WIP

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d42aa12	is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
​29ca24b	Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
​1957acd	src/sage_setup/clean.py: Update comment
​8b37214	WIP

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​92f1145	sage_install_and_clean: If all_distributions is a list, do not clean files that do not belong to this list
​44a4456	src/sage_setup/command/sage_install.py: Also handle modules in namespace packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ad9402f	sage_setup.command.sage_install: Fixup: Remove cleaning code (moved to sage_install_and_clean)
​88455dc	Merge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a9933b6

sage_build_cython: Pass built_distributions to find_extra_files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ea0d15a	sage.matrix: Resolve circular imports without using __init__.py
​e4ce720	Merge branch 't/30784/sage_matrix__resolve_circular_imports_without_using___init___py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6aaa958

sage_install_and_clean: Move all_distributions into class

[mkoeppe]

Ready for review

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​39be639

FileDocTestSource.in_lib: Handle namespace packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​64c5120

find_python_sources: Also collect namespace packages - so that package_data works

[gh-tobiasdiez]

What's the advantage/reason to introduce the sage-specific namespace/nonamespace convention instead of following PEP 420 and keep the __init__.py file as a marker for namespace packages together with setuptools.find_namespace_packages()?

[mkoeppe]

Replying to gh-tobiasdiez:

What's the advantage/reason to introduce the sage-specific namespace/nonamespace convention instead of following PEP 420 and keep the __init__.py file as a marker for namespace packages together with setuptools.find_namespace_packages()?

One cannot keep __init__.py for native namespace packages. PEP 420 is quite clear about it. Sub-packages of namespace packages do have __init__.py; and also the present ticket keeps them.

[mkoeppe]

find_namespace_packages accepts any directory as a package directory:

class PEP420PackageFinder(PackageFinder):
    @staticmethod
    def _looks_like_package(path):
        return True

(​https://github.com/pypa/setuptools/blob/f991fbb3c9d0e10a0a78ae2b508b3fd99f9cdef2/setuptools/__init__.py#L110)

But our source tree has some directories that should not be considered package directories. I have marked these with the nonamespace file.

Having both nonamespace and namespace files results in simpler changes to the existing code.

[gh-tobiasdiez]

Oh ok, sorry, I've misunderstood the ticket description. You are of course right, implicit namespace packages don't have a __init__.py file.

Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages(), and then filter out unwanted folders based on a hard-coded list. The (no)namespace files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by ​PEP 382, which was rejected in favor of PEP 420.

If you do prefer to have these files, I would like to ask you to add documentation to the dev docs.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3c7d675

src/sage_setup/find.py (find_python_sources): Update doc, doctests

[mkoeppe]

Replying to gh-tobiasdiez:

Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages(), and then filter out unwanted folders based on a hard-coded list. The (no)namespace files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by ​PEP 382, which was rejected in favor of PEP 420.

Thanks for the input and the pointer to PEP 382. I'll try out if your suggestion can be implemented without major changes.

[mkoeppe]

TBD: In incremental builds, the cleaner fails to remove previously installed __init__.py files:

**********************************************************************
File "src/sage_setup/find.py", line 343, in sage_setup.find.installed_files_by_module
Failed example:
    files_by_module['sage.graphs.graph_decompositions']
Expected nothing
Got:
    {'sage/graphs/graph_decompositions/__init__.py',
     'sage/graphs/graph_decompositions/__pycache__/__init__.cpython-37.pyc'}
**********************************************************************

[mkoeppe]

Note that one annoying thing about find_namespace_packages when applied to our source tree is that we have lots of directories that do not contain any python files, leading to output like this from distutils:

[sagelib-9.2.rc2] package init file 'sage/schemes/generic/notes/__init__.py' not found (or not a regular file)

The branch on this ticket handles such directories specially (no need for nonamespace files there). If there is a solution that could silence these notes, the code could be brought closer to find_namespace_packages behavior.

[gh-tobiasdiez]

find_namespace_packages excepts include and exclude directories as arguments. So one option would be to get a list of all folders not including any python files, and pass this as exclude (or the same approach using a whitelist).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a6e8447

Merge tag '9.3.beta3' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[mkoeppe]

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

Setting a new milestone for this ticket based on a cursory review.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​f017b3f

#28925 Modify find_python_sources to support modularization (squashed)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​bfaeb14

28925: Modify find_python_sources to support modularization (squashed)

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​c6fc111	Prettier diagram
​7d6a44c	Use :mod: as markup for packages/modules
​f83d424	Improve ABC example
​817a8d0	src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup
​b555735	src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata
​81e9c9a	Merge #32899
​ecc0886	Merge #31386
​00d5f59	build/pkgs/sagelib: Invoke install cleaner explicitly, remove it from pkgs/sagemath-standard/setup.py
​a5aab69	Merge #32927
​15813aa	Remove __init__.py files

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​a72ee3b	src/sage/all__sagemath_objects.py: Add some more imports
​f3ee49b	src/sage/all__sagemath_objects.py: Add definitions of Integer, RealNumber for doctesting
​2a4a323	Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap
​9ef0567	Merge #29039
​e0399a1	pkgs/sagemath-objects: Install a fixed set of scripts, not all of src/bin
​7519048	src/bin/sage[-env]: Put SAGE_ROOT/src/bin in front of path only if run out of this directory
​00024e6	Merge #32933
​bfc953b	src/sage/doctest/reporting.py: Indicate --environment if not default
​36d6b09	pkgs/sagemath-categories/setup.cfg.m4: Add scripts
​8648741	Merge #29865

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fcfdf1d

Restore __init__.py files for #28925

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​31b9c16

src/sage_setup/command/sage_install.py (sage_clean): Do not fail if dist.py_modules is None

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f5fd073	pkgs/sagemath-objects/MANIFEST.in: Remove __init__.py files
​67d78e7	pkgs/sagemath-categories/MANIFEST.in.m4: Remove src/sage/categories/__init__.py
​4912f15	src/sage/env.py: Get SAGE_LIB via sage.env's __file__
​03909ae	src/sage/env.py (SAGE_LIB): Fix up

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8c8c29c	WIP
​c3f6bcb	src/sage/misc/namespace_package.py (is_package_or_sage_namespace_package_dir): New
​ab23a14	src/sage/doctest/sources.py: Use is_package_or_sage_namespace_package_dir
​0f05856	src/sage/doctest/sources.py: Remove unnecessary conversion to bool
​d8aa37c	src/sage/misc/namespace_package.py: In doctests, use 'directory' instead of the single-letter variable name
​2cf8d8d	Merge #33033
​c7f8718	src/sage_setup/find.py: Use is_package_or_sage_namespace_package_dir

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f3764bc

src/sage_setup/find.py: Get rid of use of 'nonamespace' files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​19e93f0	is_package_or_sage_namespace_package_dir: Make directories with __init__.pxd package directories
​e520abb	Merge #33033
​6089ae8	src/sage_setup/find.py: Add doctest result

[mkoeppe]

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages(), and then filter out unwanted folders based on a hard-coded list. The (no)namespace files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by ​PEP 382, which was rejected in favor of PEP 420.

Thanks for the input and the pointer to PEP 382. I'll try out if your suggestion can be implemented without major changes.

Done now via #33033.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c80dd42	src/sage/{arith,ext,libs,rings,sets}/all__sagemath_objects.py
​d01d881	pkgs/sagemath-objects/MANIFEST.in: Exclude generated files
​37a9579	pkgs/sagemath-categories/MANIFEST.in.m4: Exclude generated files

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​55e258d	pkgs/sagemath-categories/MANIFEST.in.m4: Remove src/sage/categories/__init__.py
​07ffddb	src/sage/env.py: Get SAGE_LIB via sage.env's __file__
​320f2e0	src/sage/env.py (SAGE_LIB): Fix up
​85ef02e	WIP
​95a7205	src/sage_setup/find.py: Use is_package_or_sage_namespace_package_dir
​6ab1bd7	src/sage_setup/find.py: Get rid of use of 'nonamespace' files
​0b33605	src/sage_setup/find.py: Add doctest result
​2e995dd	src/sage/{arith,ext,libs,rings,sets}/all__sagemath_objects.py
​f244b57	pkgs/sagemath-objects/MANIFEST.in: Exclude generated files
​2f6d473	pkgs/sagemath-categories/MANIFEST.in.m4: Exclude generated files

[mkoeppe]

Rebased away from #32927.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cf7371f	src/sage_setup/find.py: Update import of is_package_or_namespace_package_dir
​5808f3d	src/sage/misc/package_dir.py: New file for is_package_or_sage_namespace_package_dir
​f5addb3	Merge #33033

[gh-tobiasdiez]

Replying to mkoeppe:

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages(), and then filter out unwanted folders based on a hard-coded list. The (no)namespace files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by ​PEP 382, which was rejected in favor of PEP 420.

Thanks for the input and the pointer to PEP 382. I'll try out if your suggestion can be implemented without major changes.

Done now via #33033.

Probably I'm missing something here. The new mechanism seems to still rely on indicator/marker files, just that the are now named all__. Except for the different name this seems to be exactly the approach of the rejected PEP 382. The point of my proposal was that only the setup file should know/decide on which packages it want to include and exclude. Then the general helper in #33033 would like

is_namespace_package(dir):
   return dir in setuptools.find_namespace_packages(include=..., exclude=...)

And the setup.cfg would use ​https://setuptools.pypa.io/en/latest/userguide/package_discovery.html.

[mkoeppe]

Replying to gh-tobiasdiez:

The new mechanism seems to still rely on indicator/marker files, just that the are now named all__. Except for the different name this seems to be exactly the approach of the rejected PEP 382.

That's certainly true, but at least the Sage source tree already has these files

[mkoeppe]

Replying to gh-tobiasdiez:

The point of my proposal was that only the setup file should know/decide on which packages it want to include and exclude. Then the general helper in #33033 would like

is_namespace_package(dir):
   return dir in setuptools.find_namespace_packages(include=..., exclude=...)

The helper function is added in #33033 for runtime use (doctesting). Doctesting needs to be able to work on (a) Sage source trees, (b) installed Sage packages (this is important, for example, for the downstream packagers), (c) arbitrary files. Hardcoding inclusions and exclusions in the helper function seems a bit unmodular

[gh-tobiasdiez]

In the end it comes down to who should have the control:

• Should the doctest code decide what to test, and the setup.py/cfg file what to distribute and the sage startup script what to expose
• or should the module/package be responsible to declare that it should be tested, distributed and exposed

Currently, this ticket (and the related ones) go in the direction of the second approach by using all.py. For me, using inversion of control and following the first approach seems more natural and flexible. Since you have a better overview though, I trust your judgement.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3939af7	Merge tag '9.5.rc3' into t/33033/prepare_sage_doctest_for_namespace_packages
​178d111	Merge #33033

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5c873e0

Merge tag '9.6.beta1' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​37b3299	pkgs/sagemath-{objects,categories}/setup.cfg.m4: Update python_requires
​5546c63	pkgs/sagemath-objects/MANIFEST.in: Add sage.misc.package_dir

[mkoeppe]

All dependencies are merged

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f961a7d

src/setup.py: Support namespace packages in Cython 0.x

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d2b5293

Merge tag '9.6.beta5' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​aac736c

build/pkgs/sage_setup/dependencies: Add all sage_setup source files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6258b79

Merge tag '9.6.rc2' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​37f5cb5	.github/workflows/sdist.yml: Build wheels too
​b273985	sage.env.cython_aliases: Handle ecl via default_required_modules
​6ac30fd	pkgs/sagemath-objects/MANIFEST.in: Add missing sage.misc.package_dir
​762dd70	pkgs/sagemath-objects/MANIFEST.in: Add sage.misc.sagedoc
​33a1540	pkgs/sagemath-{objects,categories}/README.rst: Mention dependencies
​e8a98b1	Merge #33803

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​32c831c	sage.env.cython_aliases: Handle ecl via default_required_modules
​4d75ff7	pkgs/sagemath-objects/MANIFEST.in: Add missing sage.misc.package_dir
​078d72d	pkgs/sagemath-objects/MANIFEST.in: Add sage.misc.sagedoc
​4da43c5	pkgs/sagemath-{objects,categories}/README.rst: Mention dependencies
​06eec85	Merge #33803

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​c5d43bd	src/sage/env.py (cython_aliases): Fix up
​4e95de9	Merge #33803

[mkoeppe]

The failure in src/sage/interfaces/expect.py shown by the patchbot is unrelated. Ready for review

[dimpase]

should one exclude not only *.so, but *.dylib, too?

[mkoeppe]

On macOS, the compiled python modules are also .so, not .dylib.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a92db46

Merge tag '9.7.beta1' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a15f173	pkgs/sagemath-categories/tox.ini: Fix instructions in comment
​8587d47	src/sage/misc/package_dir.py: New context manager cython_namespace_package_support
​bc0992d	src/sage/misc/cython.py: Activate cython_namespace_package_support
​7c98c36	src/setup.py: Fix up use of cython_namespace_package_support
​79822e4	pkgs/sagemath-objects/setup.py: Monkey-patching cython is now done by the sage_build_cython command

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​39272c1

src/sage_setup/command/sage_build_cython.py: Reduce verbosity

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​81237e1

Merge tag '9.7.beta2' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[gh-GMS103]

Sorry for my ignorance, but I am unable to merge this ticket:

% git clone -c core.symlinks=true --branch develop --tags https://github.com/sagemath/sage.git
[...]
% cd sage
% git-trac-merge 28925

#28925
remote branch: u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

From git://trac.sagemath.org/sage
 * branch                  u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_ -> FETCH_HEAD
 * [new branch]            u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_ -> trac/u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

Auto-merging build/pkgs/sage_setup/dependencies
CONFLICT (content): Merge conflict in build/pkgs/sage_setup/dependencies
Automatic merge failed; fix conflicts and then commit the result.
% 

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dd2c525

Merge tag '9.7.beta3' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[mkoeppe]

I've resolved the merge conflict

[klee]

Not used import?

pkgs/sagemath-objects/setup.py

@@ -42 +42 @@
     python_modules = []
     cython_modules = []
 else:
-    from sage_setup.find import find_python_sources, is_package_or_namespace_package_dir
+    from sage_setup.find import find_python_sources
     python_packages, python_modules, cython_modules = find_python_sources(
         '.', ['sage'])   # for now, we do the filtering using MANIFEST

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2cc9002	pkgs/sagemath-objects/setup.py: Remove unused import
​cd4d8e9	pkgs/sagemath-standard/setup.py: Monkey-patching cython is now done by the sage_build_cython command

[mkoeppe]

Thanks!

And I have applied the change that left that import unused also to another file.

[klee]

Thanks.

Another comment: clean_install_dir and find_extra_files have a new argument distributions, but the docstrings do not explain it.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​af9ac78	src/sage_setup/find.py (find_extra_files): Document new arg 'distributions'
​13e66a3	src/sage_setup/find.py (find_extra_files): Fix indentation
​9ab3200	src/sage_setup/clean.py (clean_install_dir): Document new arg 'distributions'

[klee]

Thank you for the update.

I have been testing (using) this ticket and and the sequel #33011 for a while. They works well. I checked the code as far as I can (not very far), and it looks good to me.

If we don't have further comments on this ticket, then it would better be tested by more developers in the wild by merging this now, so that we can fix problems before the next release.

I will wait for your (not Matthias) meta comments, and proceed to "positive review" in a couple of days.

[klee]

Is this

./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'  

broken? It doesn't work with (and without) this ticket.

[dimpase]

Replying to klee:

Is this

./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'  

broken? It doesn't work with (and without) this ticket.

works for me (after make build from a clean state done)

[klee]

Replying to dimpase:

Replying to klee:

Is this

./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'  

broken? It doesn't work with (and without) this ticket.

works for me (after make build from a clean state done)

I did make distclean; make -j8; ./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'. So this is clean.

This is the initial part that goes wrong:

sage -t --random-seed=251639358214975434990568781743714015030 --environment=sage.all__sagemath_categories /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx
**********************************************************************
File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx", line 58, in sage.structure.coerce_actions.GenericAction.__init__
Failed example:
    M = MatrixSpace(ZZ,2)
Exception raised:
    Traceback (most recent call last):
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.structure.coerce_actions.GenericAction.__init__[0]>", line 1, in <module>
        M = MatrixSpace(ZZ,Integer(2))
    NameError: name 'MatrixSpace' is not defined

Any clue? I am on Mac.

[dimpase]

I start from git clean -fdx - which cleans much more.

You'll probably want to move upstream/ outside and after git clean make a symbolic link to it, otherwise it will re-download things.

So it's positive review from me.

[klee]

Replying to dimpase:

I start from git clean -fdx - which cleans much more.

Thanks. But it didn't help. As far as I understand, the doctest failures are expected. The real problem is:

$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.categories.category import Category
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/categories/category.py", line 107, in <module>
    from sage.misc.c3_controlled import _cmp_key, _cmp_key_named, C3_sorted_merge
  File "sage/misc/c3_controlled.pyx", line 364, in init sage.misc.c3_controlled (build/cythonized/sage/misc/c3_controlled.c:11819)
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/structure/__init__.py", line 2, in <module>
    import sage.structure.element
  File "sage/structure/element.pyx", line 1, in init sage.structure.element (build/cythonized/sage/structure/element.c:36204)
  File "sage/structure/category_object.pyx", line 60, in init sage.structure.category_object (build/cythonized/sage/structure/category_object.c:9982)
ImportError: cannot import name Category

Strangely this works

Hera:sage-dev$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.structure.sage_object import SageObject
>>> from sage.categories.category import Category
>>> 

Anyway, this is not a problem of this ticket. The situation is the same without this ticket.

Hence my positive review to this ticket!

[dimpase]

hmm, do you have the current directory, i.e. ., in your PATH? IMHO this might mess things up.

[klee]

Replying to dimpase:

hmm, do you have the current directory, i.e. ., in your PATH? IMHO this might mess things up.

Yes. But even after removing it from the path, it's the same.

./configure shows python3-3.10.3: using system package; SPKG will not be installed, and

$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages']

$ pkgs/sagemath-categories/.tox/python/bin/python 
Python 3.10.4 (main, Apr 26 2022, 19:42:59) [Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload', '/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages']

[mkoeppe]

Replying to dimpase:

Replying to klee:

Is this

./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'  

broken? It doesn't work with (and without) this ticket.

works for me (after make build from a clean state done)

sagemath-categories (in contrast to sagemath-standard) does not need make build because all of its build and runtime dependencies can be found on PyPI.

[mkoeppe]

Replying to klee:

This is the initial part that goes wrong:

sage -t --random-seed=251639358214975434990568781743714015030 --environment=sage.all__sagemath_categories /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx
**********************************************************************
File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx", line 58, in sage.structure.coerce_actions.GenericAction.__init__
Failed example:
    M = MatrixSpace(ZZ,2)
Exception raised:
    Traceback (most recent call last):
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.structure.coerce_actions.GenericAction.__init__[0]>", line 1, in <module>
        M = MatrixSpace(ZZ,Integer(2))
    NameError: name 'MatrixSpace' is not defined

sagemath-categories as of this ticket does not contain src/sage/matrix/matrix_space.py (see pkgs/sagemath-categories/MANIFEST.in), so MatrixSpace is not available in sage.all__sagemath_categories.

Note that the tox run ends with the message "lots of doctest failures are expected". This is one of them.

[mkoeppe]

Replying to klee:

The real problem is:

$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.categories.category import Category
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/categories/category.py", line 107, in <module>
    from sage.misc.c3_controlled import _cmp_key, _cmp_key_named, C3_sorted_merge
  File "sage/misc/c3_controlled.pyx", line 364, in init sage.misc.c3_controlled (build/cythonized/sage/misc/c3_controlled.c:11819)
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/structure/__init__.py", line 2, in <module>
    import sage.structure.element
  File "sage/structure/element.pyx", line 1, in init sage.structure.element (build/cythonized/sage/structure/element.c:36204)
  File "sage/structure/category_object.pyx", line 60, in init sage.structure.category_object (build/cythonized/sage/structure/category_object.c:9982)
ImportError: cannot import name Category

Strangely this works

Hera:sage-dev$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.structure.sage_object import SageObject
>>> from sage.categories.category import Category
>>> 

The Sage library contains many cyclic imports. If one starts at the wrong place in an import cycle, one can run into problems with partially initialized modules. See #33580 (Meta-ticket: make all modules importable without sage.all)

[mkoeppe]

Thanks both for the review!

[klee]

Replying to mkoeppe:

Note that the tox run ends with the message "lots of doctest failures are expected".

Yes, it's there. It would perhaps better be colored red or green...

[vbraun]

Merge failure on top of:

843eb03e7e Updated SageMath version to 9.7.beta4

merge was not clean: conflicts in pkgs/sagemath-categories/tox.ini

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f9df1ae

Merge tag '9.7.beta4' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_