Opened 21 months ago

Last modified 2 months ago

#28925 needs_work enhancement

Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420)

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.5
Component: build Keywords: sd111
Cc: isuruf, embray, jdemeyer, dimpase, dcoudert, vdelecroix, vbraun, gh-tobiasdiez Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: N/A Work issues: comment 90
Branch: u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_ (Commits, GitHub, GitLab) Commit: a6e8447748b8a2b4107c43a92daa2bcef30ff27b
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

This ticket prepares the modularization of sagelib into separate distributions (pip-installable packages) by adding support for native namespace packages (PEP 420) to sagelib's build system.

Ordinary packages (directories of modules) are turned into namespace packages by removing the empty __init__.py file.

Then a different distribution (distutils package) can share the same namespace.

In the first stage of modularization, planned for Sage 9.3, we use this new mechanism to split out all "optional extension modules" (which depend on some optional library) from the sagelib distribution to separate pip-installable packages. This will simplify deployment and dependency management.

(The source code remains in the same place in src/sage in the git tree; the new pip-installable packages will have separate metadata in build/pkgs/SPKG.)

  1. To this end, this ticket changes the following packages to namespace packages (by removing __init__.py):
    • sage.matrix (this allows the new distribution sage-meataxe introduced in #30151 to add the module sage.matrix.matrix_gfpn_dense) to the package)
    • sage.libs (for .meataxe - #30151, .sirocco, .coexter3, .fes)
    • sage.interfaces (for .primecount)
    • sage.graphs (for .bliss, .mcqd)
    • sage.graphs.graph_decompositions (for .tdlib -> sage-tdlib, #29864.)
    • sage.numerical.backends. (This is preparation for later work, in #30152.)
  1. Because our source discovery mechanism in sagelib's build system (setup.py + sage_setup) distinguishes package directories from other directories containing data files by the presence of the __init__.py file, the present ticket makes several adjustments to the build system - in particular to the functions find_python_sources and find_extra_files.

To simplify the implementation and to allow for self-documentation of the directory structure, we introduce sage_setup-specific files named:

  • namespace to mark a directory without __init__.py as a namespace package;
  • nonamespace to mark a directory as a data directory. (This is in analogy with the nodoctest files that control the Sage doctester.)

(In the branch of this ticket, these are empty files; but they could contain comments that explain the purpose of a particular directory.)

  1. Also the incremental installation mechanisms of sagelib's build system need adjustment. clean_stale_files, which deletes installed Python modules in $SAGE_LIB/sage = ..../site-packages/sage that are no longer present in the current version of the source tree, is changed so that it does not delete modules in a namespace package that are provided by a separate distribution (pip-installable package).

Follow-up tickets:

  • Wishlist item: Warn or give errors if the sage_setup: distribution directives, namespace and nonamespace files indicate an inconsistent structure such as when it was forgotten to remove __init__.py.
  • #30152 will make further changes to the cleaner that will allow out-of-tree namespace packages

See also:

Change History (98)

comment:1 Changed 18 months ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

pushing these forward to 9.2

comment:2 Changed 17 months ago by mkoeppe

  • Cc dimpase dcoudert added
  • Description modified (diff)

comment:3 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:4 Changed 17 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Modify clean_stale_files to support modularization of sagelib by namespace packages to Modify clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420)

comment:5 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:6 Changed 17 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Modify clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420) to Modify find_python_sources, clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420)

comment:7 Changed 17 months ago by mkoeppe

  • Cc vdelecroix added
  • Description modified (diff)

comment:8 Changed 17 months ago by mkoeppe

  • Cc vbraun added

comment:9 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:10 Changed 17 months ago by mkoeppe

  • Branch set to u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:11 Changed 17 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Commit set to 8bc56623abb2579ca435b99f324a9d5eef89c857

New commits:

8e50bbesage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
dff6e62Mark src/sage/ext_data as directories that are not namespace packages
8bc5662sage_setup.find.find_python_sources: Handle native namespace packages and 'nonamespace' files

comment:12 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:13 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:14 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:15 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 17 months ago by git

  • Commit changed from 8bc56623abb2579ca435b99f324a9d5eef89c857 to 606a509168111b4cf7306fa283e1d5f0cccd41d0

Branch pushed to git repo; I updated commit sha1. New commits:

606a509sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

comment:17 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:18 Changed 17 months ago by mkoeppe

Next steps: adjust the cleaner, find Cython sources

comment:19 Changed 17 months ago by mkoeppe

The Cython source finder could perhaps also parse a new directive that controls OptionalExtension behavior (until we get rid of OptionalExtension altogether in #29701) so that we can get rid of module_list.py completely (see #29706)

comment:20 Changed 17 months ago by git

  • Commit changed from 606a509168111b4cf7306fa283e1d5f0cccd41d0 to a7b2287a8b69e814781b20434849e788d00884ca

Branch pushed to git repo; I updated commit sha1. New commits:

a7b2287find_python_sources: New optional argunment sage_packages, filter by # distutils: sage_package = PACKAGE

comment:21 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:22 Changed 17 months ago by git

  • Commit changed from a7b2287a8b69e814781b20434849e788d00884ca to 194d3b430666c3bb72885a945274589ea54bd4be

Branch pushed to git repo; I updated commit sha1. New commits:

194d3b4sage_setup.find.find_python_sources: Also find Cython modules

comment:23 Changed 16 months ago by git

  • Commit changed from 194d3b430666c3bb72885a945274589ea54bd4be to ad3ebad7af2f4f1eea0f4a771dc029382ce60d96

Branch pushed to git repo; I updated commit sha1. New commits:

adc39e3WIP
ad3ebadMerge tag '9.2.beta0' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:24 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:25 Changed 16 months ago by git

  • Commit changed from ad3ebad7af2f4f1eea0f4a771dc029382ce60d96 to 004dd5240961beb9e2c4fab328d77688df57a862

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

004dd52Merge tag '9.2.beta0' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:26 Changed 16 months ago by git

  • Commit changed from 004dd5240961beb9e2c4fab328d77688df57a862 to 3a68b37aba6be39b55a404874f2f66dc1ed4b0eb

Branch pushed to git repo; I updated commit sha1. New commits:

3a68b37src/sage_setup/find.py: Switch to parsing directive 'sage_setup: distribution = PKG'

comment:27 Changed 16 months ago by git

  • Commit changed from 3a68b37aba6be39b55a404874f2f66dc1ed4b0eb to d900139267b5aeceaa399ee6ffa190e4fd821748

Branch pushed to git repo; I updated commit sha1. New commits:

d900139src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list

comment:28 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:29 Changed 16 months ago by git

  • Commit changed from d900139267b5aeceaa399ee6ffa190e4fd821748 to 6d71e74ac3a6ed2808e927664c593d70f11d9869

Branch pushed to git repo; I updated commit sha1. New commits:

6d71e74Add doctest

comment:30 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:31 Changed 16 months ago by git

  • Commit changed from 6d71e74ac3a6ed2808e927664c593d70f11d9869 to f670f81d1db80ab3334292b7646bc4fd034b32b0

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

0a7fb48Move src/fpickle_setup.py to srs/sage_setup/
f4a9269src/setup.py: Move class sage_install to new module sage_setup.command.sage_install
5660d8fsrc/module_list.py: Move library_order to new module sage_setup.library_order
43d4f83src/setup.py: Move classes sage_build_* to new modules sage_setup.command.sage_*
09e3297src/setup.py: Move excepthook to new module sage_setup.excepthook
1cfed7cbuild/pkgs/sagelib/spkg-install: Do not poison SAGE_LOCAL, used in script installed by spkg pkgconf
62c7531Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
b3102e9Merge tag '9.2.beta0' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
5066f83Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
f670f81WIP

comment:32 Changed 16 months ago by mkoeppe

Next: Fix find_extra_files so that it finds the .pxd files in sage.numerical.backends

comment:33 Changed 16 months ago by mkoeppe

  • Dependencies set to #29702

comment:34 Changed 16 months ago by git

  • Commit changed from f670f81d1db80ab3334292b7646bc4fd034b32b0 to bd858270985630f780a0d367c2b4b1581c755404

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

bd85827is_package_or_namespace_package_dir: New, monkeypatch into Cython.Build.Dependencies

comment:35 Changed 16 months ago by git

  • Commit changed from bd858270985630f780a0d367c2b4b1581c755404 to 532bb21b12f289e579c2dc033dbc41dab47c4330

Branch pushed to git repo; I updated commit sha1. New commits:

532bb21is_package_or_namespace_package_dir: Make self-contained, patch into more of Cython

comment:36 Changed 16 months ago by git

  • Commit changed from 532bb21b12f289e579c2dc033dbc41dab47c4330 to 0aa36935e2656eeb0b8139f98d7352afed8221cb

Branch pushed to git repo; I updated commit sha1. New commits:

2b42029src/sage_setup/command/__init__.py: New
a830304src/sage_setup/command/sage_build_cython.py: Fix adding setup.py as dependency
d3080a5Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
0aa3693Mark src as nonamespace

comment:37 Changed 16 months ago by git

  • Commit changed from 0aa36935e2656eeb0b8139f98d7352afed8221cb to 47cb57898d62e3d71ad18a4b27ea0a99ee3a982e

Branch pushed to git repo; I updated commit sha1. New commits:

47cb578is_package_or_namespace_package_dir: Recognize namespace packages by file 'namespace'

comment:38 Changed 16 months ago by git

  • Commit changed from 47cb57898d62e3d71ad18a4b27ea0a99ee3a982e to 4c51133f6e3458abf3f9303d49303e49b1b1cbb8

Branch pushed to git repo; I updated commit sha1. New commits:

381a507src/module_list.py: Remove explicit list of sage.numerical.backends extensions
4c51133sage_setup/command/sage_install.py: WIP: Disable cleaning

comment:39 Changed 16 months ago by git

  • Commit changed from 4c51133f6e3458abf3f9303d49303e49b1b1cbb8 to c0a2ed74a42cf8b575fa2cf26228e513bd69dfbd

Branch pushed to git repo; I updated commit sha1. New commits:

c0a2ed7src/module_list.py: Remove explicit listing of sage.graphs.graph_decompositions extensions

comment:40 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:41 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:42 Changed 16 months ago by mkoeppe

  • Work issues set to rebase on top of #29701

comment:43 Changed 16 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:44 Changed 16 months ago by mkoeppe

  • Dependencies changed from #29702 to #29702, #29701

comment:45 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:46 Changed 16 months ago by git

  • Commit changed from c0a2ed74a42cf8b575fa2cf26228e513bd69dfbd to 40bc47392ab35b9a7e4059bb46a455828d5d7673

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

a43bf07sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
7a6d12fsage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files, Cython modules, directive 'sage_setup: distribution = PKG'
f4a6c7dsrc/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
cee98e7Add doctest
7873513is_package_or_namespace_package_dir: New, monkeypatch into Cython
d01495eMark src as nonamespace
899dc32is_package_or_namespace_package_dir: Recognize namespace packages by file 'namespace'
b97b1e0src/module_list.py: Remove explicit list of sage.numerical.backends extensions
4b8181bsage_setup/command/sage_install.py: WIP: Disable cleaning
40bc473src/module_list.py: Remove explicit listing of sage.graphs.graph_decompositions extensions

comment:47 Changed 16 months ago by mkoeppe

Squashed some commits

comment:48 Changed 16 months ago by git

  • Commit changed from 40bc47392ab35b9a7e4059bb46a455828d5d7673 to 975b4a0d1504887dafac780eb66120b58f204440

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

226aa5asage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
906fb23Mark src and src/sage/ext_data as directories that are not namespace packages
4a2eee0sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files, Cython modules, directive 'sage_setup: distribution = PKG'
a8b0316is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
92bd8aesrc/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
6647d09src/module_list.py: Remove explicit list of sage.numerical.backends extensions
975b4a0sage_setup/command/sage_install.py: WIP: Disable cleaning

comment:49 Changed 16 months ago by git

  • Commit changed from 975b4a0d1504887dafac780eb66120b58f204440 to 650c661e50e10f57da6f121942158f654de835ec

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

5d5803esrc/sage/graphs/planarity.pyx: Add forgotten distutils directive
a0be9b6Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29701/replace_use_of_module_list_optionalextension
d0eb84esage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
c82fb01sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
d90e3b6Mark src and src/sage/ext_data as directories that are not namespace packages
f598e04sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
f522b09is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
50df6d5src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list
e34c028src/module_list.py: Remove explicit list of sage.numerical.backends extensions
650c661sage_setup/command/sage_install.py: WIP: Disable cleaning

comment:50 Changed 16 months ago by mkoeppe

  • Work issues rebase on top of #29701 deleted

Rebased on top of #29701

comment:51 Changed 16 months ago by mkoeppe

  • Dependencies changed from #29702, #29701 to #29701

comment:52 Changed 15 months ago by mkoeppe

  • Description modified (diff)
  • Work issues set to move cleaning modifications to #30152, break sage_install into several classes for the different use cases

comment:53 Changed 15 months ago by git

  • Commit changed from 650c661e50e10f57da6f121942158f654de835ec to 474b2007d3b68cbf14d10feb1cc3b2750bf0c02e

Branch pushed to git repo; I updated commit sha1. New commits:

14396afMerge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension
d62da15sage_setup.command.sage_build: Add the extensions to the distribution
06a3609sage_setup.find.find_python_sources: Add benchmark doctest
174626cMerge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension
034a7f7Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension
d3c608bMerge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension
d99e942Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
474b200Merge tag '9.2.beta5' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:54 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:55 Changed 14 months ago by git

  • Commit changed from 474b2007d3b68cbf14d10feb1cc3b2750bf0c02e to 2e79d33db38da4da9a87561f534f6b6ca813904a

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

eaadf77sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
59151absage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
713a838Mark src and src/sage/ext_data as directories that are not namespace packages
4faff43sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
de17c1cis_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
6761b7dsrc/module_list.py: Remove explicit list of sage.numerical.backends extensions
65f3739sage_setup/command/sage_install.py: WIP: Disable cleaning
8c20f3eRevert "sage_setup/command/sage_install.py: WIP: Disable cleaning"
37bd1casage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install
2e79d33WIP

comment:56 Changed 14 months ago by git

  • Commit changed from 2e79d33db38da4da9a87561f534f6b6ca813904a to 98a1b883f8cab1fc5d3ec829a937acfa148b6347

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

8a19fe2build/make/Makefile.in (sagelib-clean): Clean the new build location
ccc67b0src/sage_setup: Update cythonized_dir in doctests
df38027Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30277/remove_src_module_list_py
ef4e554Merge branch 't/30277/remove_src_module_list_py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
6282b8dsage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
7c10680sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
6e68259Mark src and src/sage/ext_data as directories that are not namespace packages
b71c3a6sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
3a12146is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
98a1b88sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install

comment:57 Changed 14 months ago by mkoeppe

  • Dependencies changed from #29701 to #30277
  • Work issues move cleaning modifications to #30152, break sage_install into several classes for the different use cases deleted

comment:58 Changed 14 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Modify find_python_sources, clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420) to Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420)

comment:59 Changed 14 months ago by git

  • Commit changed from 98a1b883f8cab1fc5d3ec829a937acfa148b6347 to 742ab8777c8010a2e3d6435428c8baa7acdea128

Branch pushed to git repo; I updated commit sha1. New commits:

32a86d2Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
742ab87WIP

comment:60 Changed 14 months ago by mkoeppe

  • Work issues set to installed_files_by_module

comment:61 Changed 14 months ago by mkoeppe

  • Description modified (diff)

comment:62 Changed 13 months ago by mkoeppe

  • Cc gh-tobiasdiez added

comment:63 Changed 13 months ago by git

  • Commit changed from 742ab8777c8010a2e3d6435428c8baa7acdea128 to a967aca4491de98118612f30cbdf8bd75ab4efa9

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

776c979sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
2e61adasage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
26b1ca0Mark src and src/sage/ext_data as directories that are not namespace packages
a733af5sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
c08c511is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
4407e5esage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install
a13c126Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
a967acaWIP

comment:64 Changed 13 months ago by mkoeppe

  • Dependencies #30277 deleted

comment:65 Changed 13 months ago by mkoeppe

  • Dependencies set to #30578

comment:66 Changed 13 months ago by mkoeppe

  • Work issues changed from installed_files_by_module to installed_files_by_module; rebase on top of #30578

comment:67 Changed 12 months ago by mkoeppe

  • Dependencies changed from #30578 to #30780
  • Work issues changed from installed_files_by_module; rebase on top of #30578 to installed_files_by_module; rebase on top of #30780

comment:68 Changed 12 months ago by git

  • Commit changed from a967aca4491de98118612f30cbdf8bd75ab4efa9 to a4e5683b1ecf6121bc766bf60bd90d312371b9cf

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

7980130Merge branch 't/30779/duplicate_src_setup_py' into t/30780/sage_setup__separate_installing_and_cleaning
9b95628sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in build/pkgs/sagelib/src/setup.py; refactor through new class sage_install
886ef6dMerge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
3b9559dsage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
88085b1sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)
9c39a76Mark src and src/sage/ext_data as directories that are not namespace packages
b060462sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files
fe9a7d5is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
d2fe380Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
a4e5683WIP

comment:69 Changed 12 months ago by mkoeppe

  • Work issues changed from installed_files_by_module; rebase on top of #30780 to installed_files_by_module

comment:70 Changed 12 months ago by git

  • Commit changed from a4e5683b1ecf6121bc766bf60bd90d312371b9cf to 8b37214c8c3bad2a3e046798f3b6ada33580dd31

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d42aa12is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython
29ca24bMake sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages
1957acdsrc/sage_setup/clean.py: Update comment
8b37214WIP

comment:71 Changed 12 months ago by git

  • Commit changed from 8b37214c8c3bad2a3e046798f3b6ada33580dd31 to 44a44560071ce851fce1e415000b13bc0cda3d90

Branch pushed to git repo; I updated commit sha1. New commits:

92f1145sage_install_and_clean: If all_distributions is a list, do not clean files that do not belong to this list
44a4456src/sage_setup/command/sage_install.py: Also handle modules in namespace packages

comment:72 Changed 12 months ago by mkoeppe

  • Work issues installed_files_by_module deleted

comment:73 Changed 12 months ago by mkoeppe

  • Status changed from new to needs_review

comment:74 Changed 12 months ago by git

  • Commit changed from 44a44560071ce851fce1e415000b13bc0cda3d90 to 88455dc42bfe6639c6f8dfc91f3d2fca3c51fb3e

Branch pushed to git repo; I updated commit sha1. New commits:

ad9402fsage_setup.command.sage_install: Fixup: Remove cleaning code (moved to sage_install_and_clean)
88455dcMerge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:75 Changed 12 months ago by git

  • Commit changed from 88455dc42bfe6639c6f8dfc91f3d2fca3c51fb3e to a9933b6b1282992d49976ce0ec586aa97e1d4c26

Branch pushed to git repo; I updated commit sha1. New commits:

a9933b6sage_build_cython: Pass built_distributions to find_extra_files

comment:76 Changed 12 months ago by mkoeppe

  • Dependencies changed from #30780 to #30780, #30784

comment:77 Changed 12 months ago by git

  • Commit changed from a9933b6b1282992d49976ce0ec586aa97e1d4c26 to e4ce7207371e0d27eb1747fbe03855af108beb57

Branch pushed to git repo; I updated commit sha1. New commits:

ea0d15asage.matrix: Resolve circular imports without using __init__.py
e4ce720Merge branch 't/30784/sage_matrix__resolve_circular_imports_without_using___init___py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:78 Changed 12 months ago by mkoeppe

  • Description modified (diff)

comment:79 Changed 12 months ago by git

  • Commit changed from e4ce7207371e0d27eb1747fbe03855af108beb57 to 6aaa95887be0f4f5257208e6662cd3ca821bc0bc

Branch pushed to git repo; I updated commit sha1. New commits:

6aaa958sage_install_and_clean: Move all_distributions into class

comment:80 Changed 12 months ago by mkoeppe

Ready for review

comment:81 Changed 12 months ago by git

  • Commit changed from 6aaa95887be0f4f5257208e6662cd3ca821bc0bc to 39be6390197b4c39382420e27653c83dd99a3503

Branch pushed to git repo; I updated commit sha1. New commits:

39be639FileDocTestSource.in_lib: Handle namespace packages

comment:82 Changed 12 months ago by git

  • Commit changed from 39be6390197b4c39382420e27653c83dd99a3503 to 64c5120b0a03c5a907cb56f63bf9df360976bd44

Branch pushed to git repo; I updated commit sha1. New commits:

64c5120find_python_sources: Also collect namespace packages - so that package_data works

comment:83 Changed 12 months ago by mkoeppe

  • Description modified (diff)

comment:84 follow-up: Changed 12 months ago by gh-tobiasdiez

What's the advantage/reason to introduce the sage-specific namespace/nonamespace convention instead of following PEP 420 and keep the __init__.py file as a marker for namespace packages together with setuptools.find_namespace_packages()?

comment:85 in reply to: ↑ 84 Changed 11 months ago by mkoeppe

Replying to gh-tobiasdiez:

What's the advantage/reason to introduce the sage-specific namespace/nonamespace convention instead of following PEP 420 and keep the __init__.py file as a marker for namespace packages together with setuptools.find_namespace_packages()?

One cannot keep __init__.py for native namespace packages. PEP 420 is quite clear about it. Sub-packages of namespace packages do have __init__.py; and also the present ticket keeps them.

comment:86 Changed 11 months ago by mkoeppe

find_namespace_packages accepts any directory as a package directory:

class PEP420PackageFinder(PackageFinder):
    @staticmethod
    def _looks_like_package(path):
        return True

(https://github.com/pypa/setuptools/blob/f991fbb3c9d0e10a0a78ae2b508b3fd99f9cdef2/setuptools/__init__.py#L110)

But our source tree has some directories that should not be considered package directories. I have marked these with the nonamespace file.

Having both nonamespace and namespace files results in simpler changes to the existing code.

comment:87 follow-up: Changed 11 months ago by gh-tobiasdiez

Oh ok, sorry, I've misunderstood the ticket description. You are of course right, implicit namespace packages don't have a __init__.py file.

Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages(), and then filter out unwanted folders based on a hard-coded list. The (no)namespace files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by PEP 382, which was rejected in favor of PEP 420.

If you do prefer to have these files, I would like to ask you to add documentation to the dev docs.

comment:88 Changed 11 months ago by git

  • Commit changed from 64c5120b0a03c5a907cb56f63bf9df360976bd44 to 3c7d6751b4bdc7c419ecc94a2add7b1da21be212

Branch pushed to git repo; I updated commit sha1. New commits:

3c7d675src/sage_setup/find.py (find_python_sources): Update doc, doctests

comment:89 in reply to: ↑ 87 Changed 11 months ago by mkoeppe

Replying to gh-tobiasdiez:

Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages(), and then filter out unwanted folders based on a hard-coded list. The (no)namespace files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by PEP 382, which was rejected in favor of PEP 420.

Thanks for the input and the pointer to PEP 382. I'll try out if your suggestion can be implemented without major changes.

comment:90 Changed 11 months ago by mkoeppe

  • Status changed from needs_review to needs_work

TBD: In incremental builds, the cleaner fails to remove previously installed __init__.py files:

**********************************************************************
File "src/sage_setup/find.py", line 343, in sage_setup.find.installed_files_by_module
Failed example:
    files_by_module['sage.graphs.graph_decompositions']
Expected nothing
Got:
    {'sage/graphs/graph_decompositions/__init__.py',
     'sage/graphs/graph_decompositions/__pycache__/__init__.cpython-37.pyc'}
**********************************************************************
Last edited 11 months ago by mkoeppe (previous) (diff)

comment:91 Changed 11 months ago by mkoeppe

  • Description modified (diff)

comment:92 Changed 11 months ago by mkoeppe

Note that one annoying thing about find_namespace_packages when applied to our source tree is that we have lots of directories that do not contain any python files, leading to output like this from distutils:

[sagelib-9.2.rc2] package init file 'sage/schemes/generic/notes/__init__.py' not found (or not a regular file)

The branch on this ticket handles such directories specially (no need for nonamespace files there). If there is a solution that could silence these notes, the code could be brought closer to find_namespace_packages behavior.

comment:93 Changed 11 months ago by gh-tobiasdiez

find_namespace_packages excepts include and exclude directories as arguments. So one option would be to get a list of all folders not including any python files, and pass this as exclude (or the same approach using a whitelist).

comment:94 Changed 10 months ago by mkoeppe

  • Keywords sd111 added

comment:95 Changed 10 months ago by git

  • Commit changed from 3c7d6751b4bdc7c419ecc94a2add7b1da21be212 to a6e8447748b8a2b4107c43a92daa2bcef30ff27b

Branch pushed to git repo; I updated commit sha1. New commits:

a6e8447Merge tag '9.3.beta3' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:96 Changed 10 months ago by mkoeppe

  • Dependencies #30780, #30784 deleted
  • Work issues set to comment 90

comment:97 Changed 8 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:98 Changed 2 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

Setting a new milestone for this ticket based on a cursory review.

Note: See TracTickets for help on using tickets.