Opened 2 years ago
Last modified 3 days ago
#28925 needs_review enhancement
Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420)
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage9.6 
Component:  build  Keywords:  sd111 
Cc:  isuruf, embray, jdemeyer, dimpase, dcoudert, vdelecroix, vbraun, ghtobiasdiez  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  
Report Upstream:  N/A  Work issues:  
Branch:  u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_ (Commits, GitHub, GitLab)  Commit:  178d111960e751d5d34bf3097ddd39c65f24c42d 
Dependencies:  #29865, #33033  Stopgaps: 
Description (last modified by )
This ticket prepares the modularization of sagelib into separate distributions (pipinstallable packages) by adding support for native namespace packages (PEP 420) to sagelib's build system.
Ordinary packages (directories of modules) can be turned into namespace packages by removing the empty __init__.py
file.
Then several distribution package can share the same namespace.
Because our source discovery mechanism in sagelib's build system (setup.py
+ sage_setup
) distinguishes package directories from other directories containing data files by the presence of the __init__.py
file, the present ticket makes several adjustments to the build system  in particular to the functions find_python_sources
and find_extra_files
.
Per convention in the Sage library, namespace packages are recognized by the presence of all.py
or all__*.py
files (#33033).
In this ticket, we apply it to turn sagemathobjects and sagemathcategories (from #29865) into namespace packages,
in which __init__.py
are removed from the MANIFEST.in
files.
Because the actual files are not removed from the source tree, the normal build of the Sage distribution is not affected.
To test these two distributions:
./bootstrap && ./sage sh c '(cd pkgs/sagemathobjects && SAGE_NUM_THREADS=16 tox r v v v e py39)' ./bootstrap && ./sage sh c '(cd pkgs/sagemathcategories && SAGE_NUM_THREADS=16 tox r v v v e py39)'
Followup tickets:
 #33011 Remove the
__init__.py
files for packages designated to be namespace packages  Wishlist item: Warn or give errors if the
sage_setup: distribution
directives,namespace
andnonamespace
files indicate an inconsistent structure such as when it was forgotten to remove__init__.py
.  #30152 will make further changes to the cleaner that will allow outoftree namespace packages
See also:
 #29705: Metaticket: Modularize sagelib into separate distutils packages
 Initial discussion of namespace packages for Sage in #28175 and https://github.com/mkoeppe/sagenumericalbackendscoin/pull/4
 #21654:
src/setup.py
: Disentangle cleaning of stale installed files in build directory and in install directory
Change History (126)
comment:1 Changed 22 months ago by
 Milestone changed from sage9.1 to sage9.2
comment:2 Changed 21 months ago by
 Cc dimpase dcoudert added
 Description modified (diff)
comment:3 Changed 21 months ago by
 Description modified (diff)
comment:4 Changed 21 months ago by
 Description modified (diff)
 Summary changed from Modify clean_stale_files to support modularization of sagelib by namespace packages to Modify clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420)
comment:5 Changed 21 months ago by
 Description modified (diff)
comment:6 Changed 21 months ago by
 Description modified (diff)
 Summary changed from Modify clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420) to Modify find_python_sources, clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420)
comment:7 Changed 21 months ago by
 Cc vdelecroix added
 Description modified (diff)
comment:8 Changed 21 months ago by
 Cc vbraun added
comment:9 Changed 21 months ago by
 Description modified (diff)
comment:10 Changed 21 months ago by
 Branch set to u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_
comment:11 Changed 21 months ago by
 Commit set to 8bc56623abb2579ca435b99f324a9d5eef89c857
New commits:
8e50bbe  sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)

dff6e62  Mark src/sage/ext_data as directories that are not namespace packages

8bc5662  sage_setup.find.find_python_sources: Handle native namespace packages and 'nonamespace' files

comment:12 Changed 21 months ago by
 Description modified (diff)
comment:13 Changed 21 months ago by
 Description modified (diff)
comment:14 Changed 21 months ago by
 Description modified (diff)
comment:15 Changed 21 months ago by
 Description modified (diff)
comment:16 Changed 21 months ago by
 Commit changed from 8bc56623abb2579ca435b99f324a9d5eef89c857 to 606a509168111b4cf7306fa283e1d5f0cccd41d0
Branch pushed to git repo; I updated commit sha1. New commits:
606a509  sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

comment:17 Changed 21 months ago by
 Description modified (diff)
comment:18 Changed 21 months ago by
Next steps: adjust the cleaner, find Cython sources
comment:19 Changed 21 months ago by
comment:20 Changed 20 months ago by
 Commit changed from 606a509168111b4cf7306fa283e1d5f0cccd41d0 to a7b2287a8b69e814781b20434849e788d00884ca
Branch pushed to git repo; I updated commit sha1. New commits:
a7b2287  find_python_sources: New optional argunment sage_packages, filter by # distutils: sage_package = PACKAGE

comment:21 Changed 20 months ago by
 Description modified (diff)
comment:22 Changed 20 months ago by
 Commit changed from a7b2287a8b69e814781b20434849e788d00884ca to 194d3b430666c3bb72885a945274589ea54bd4be
Branch pushed to git repo; I updated commit sha1. New commits:
194d3b4  sage_setup.find.find_python_sources: Also find Cython modules

comment:23 Changed 20 months ago by
 Commit changed from 194d3b430666c3bb72885a945274589ea54bd4be to ad3ebad7af2f4f1eea0f4a771dc029382ce60d96
comment:24 Changed 20 months ago by
 Description modified (diff)
comment:25 Changed 20 months ago by
 Commit changed from ad3ebad7af2f4f1eea0f4a771dc029382ce60d96 to 004dd5240961beb9e2c4fab328d77688df57a862
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
004dd52  Merge tag '9.2.beta0' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:26 Changed 20 months ago by
 Commit changed from 004dd5240961beb9e2c4fab328d77688df57a862 to 3a68b37aba6be39b55a404874f2f66dc1ed4b0eb
Branch pushed to git repo; I updated commit sha1. New commits:
3a68b37  src/sage_setup/find.py: Switch to parsing directive 'sage_setup: distribution = PKG'

comment:27 Changed 20 months ago by
 Commit changed from 3a68b37aba6be39b55a404874f2f66dc1ed4b0eb to d900139267b5aeceaa399ee6ffa190e4fd821748
Branch pushed to git repo; I updated commit sha1. New commits:
d900139  src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list

comment:28 Changed 20 months ago by
 Description modified (diff)
comment:29 Changed 20 months ago by
 Commit changed from d900139267b5aeceaa399ee6ffa190e4fd821748 to 6d71e74ac3a6ed2808e927664c593d70f11d9869
Branch pushed to git repo; I updated commit sha1. New commits:
6d71e74  Add doctest

comment:30 Changed 20 months ago by
 Description modified (diff)
comment:31 Changed 20 months ago by
 Commit changed from 6d71e74ac3a6ed2808e927664c593d70f11d9869 to f670f81d1db80ab3334292b7646bc4fd034b32b0
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
0a7fb48  Move src/fpickle_setup.py to srs/sage_setup/

f4a9269  src/setup.py: Move class sage_install to new module sage_setup.command.sage_install

5660d8f  src/module_list.py: Move library_order to new module sage_setup.library_order

43d4f83  src/setup.py: Move classes sage_build_* to new modules sage_setup.command.sage_*

09e3297  src/setup.py: Move excepthook to new module sage_setup.excepthook

1cfed7c  build/pkgs/sagelib/spkginstall: Do not poison SAGE_LOCAL, used in script installed by spkg pkgconf

62c7531  Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup

b3102e9  Merge tag '9.2.beta0' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup

5066f83  Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

f670f81  WIP

comment:32 Changed 20 months ago by
Next: Fix find_extra_files
so that it finds the .pxd
files in sage.numerical.backends
comment:33 Changed 20 months ago by
 Dependencies set to #29702
comment:34 Changed 20 months ago by
 Commit changed from f670f81d1db80ab3334292b7646bc4fd034b32b0 to bd858270985630f780a0d367c2b4b1581c755404
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
bd85827  is_package_or_namespace_package_dir: New, monkeypatch into Cython.Build.Dependencies

comment:35 Changed 20 months ago by
 Commit changed from bd858270985630f780a0d367c2b4b1581c755404 to 532bb21b12f289e579c2dc033dbc41dab47c4330
Branch pushed to git repo; I updated commit sha1. New commits:
532bb21  is_package_or_namespace_package_dir: Make selfcontained, patch into more of Cython

comment:36 Changed 20 months ago by
 Commit changed from 532bb21b12f289e579c2dc033dbc41dab47c4330 to 0aa36935e2656eeb0b8139f98d7352afed8221cb
Branch pushed to git repo; I updated commit sha1. New commits:
2b42029  src/sage_setup/command/__init__.py: New

a830304  src/sage_setup/command/sage_build_cython.py: Fix adding setup.py as dependency

d3080a5  Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

0aa3693  Mark src as nonamespace

comment:37 Changed 20 months ago by
 Commit changed from 0aa36935e2656eeb0b8139f98d7352afed8221cb to 47cb57898d62e3d71ad18a4b27ea0a99ee3a982e
Branch pushed to git repo; I updated commit sha1. New commits:
47cb578  is_package_or_namespace_package_dir: Recognize namespace packages by file 'namespace'

comment:38 Changed 20 months ago by
 Commit changed from 47cb57898d62e3d71ad18a4b27ea0a99ee3a982e to 4c51133f6e3458abf3f9303d49303e49b1b1cbb8
comment:39 Changed 20 months ago by
 Commit changed from 4c51133f6e3458abf3f9303d49303e49b1b1cbb8 to c0a2ed74a42cf8b575fa2cf26228e513bd69dfbd
Branch pushed to git repo; I updated commit sha1. New commits:
c0a2ed7  src/module_list.py: Remove explicit listing of sage.graphs.graph_decompositions extensions

comment:40 Changed 20 months ago by
 Description modified (diff)
comment:41 Changed 20 months ago by
 Description modified (diff)
comment:42 Changed 20 months ago by
 Work issues set to rebase on top of #29701
comment:43 Changed 20 months ago by
 Milestone changed from sage9.2 to sage9.3
comment:44 Changed 20 months ago by
 Dependencies changed from #29702 to #29702, #29701
comment:45 Changed 20 months ago by
 Description modified (diff)
comment:46 Changed 20 months ago by
 Commit changed from c0a2ed74a42cf8b575fa2cf26228e513bd69dfbd to 40bc47392ab35b9a7e4059bb46a455828d5d7673
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
a43bf07  sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

7a6d12f  sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files, Cython modules, directive 'sage_setup: distribution = PKG'

f4a6c7d  src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list

cee98e7  Add doctest

7873513  is_package_or_namespace_package_dir: New, monkeypatch into Cython

d01495e  Mark src as nonamespace

899dc32  is_package_or_namespace_package_dir: Recognize namespace packages by file 'namespace'

b97b1e0  src/module_list.py: Remove explicit list of sage.numerical.backends extensions

4b8181b  sage_setup/command/sage_install.py: WIP: Disable cleaning

40bc473  src/module_list.py: Remove explicit listing of sage.graphs.graph_decompositions extensions

comment:47 Changed 20 months ago by
Squashed some commits
comment:48 Changed 20 months ago by
 Commit changed from 40bc47392ab35b9a7e4059bb46a455828d5d7673 to 975b4a0d1504887dafac780eb66120b58f204440
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
226aa5a  sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

906fb23  Mark src and src/sage/ext_data as directories that are not namespace packages

4a2eee0  sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files, Cython modules, directive 'sage_setup: distribution = PKG'

a8b0316  is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython

92bd8ae  src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list

6647d09  src/module_list.py: Remove explicit list of sage.numerical.backends extensions

975b4a0  sage_setup/command/sage_install.py: WIP: Disable cleaning

comment:49 Changed 20 months ago by
 Commit changed from 975b4a0d1504887dafac780eb66120b58f204440 to 650c661e50e10f57da6f121942158f654de835ec
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
5d5803e  src/sage/graphs/planarity.pyx: Add forgotten distutils directive

a0be9b6  Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29701/replace_use_of_module_list_optionalextension

d0eb84e  sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)

c82fb01  sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

d90e3b6  Mark src and src/sage/ext_data as directories that are not namespace packages

f598e04  sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files

f522b09  is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython

50df6d5  src/module_list.py: Remove sage.graphs.graph_decompositions.* from explicit Cython module list

e34c028  src/module_list.py: Remove explicit list of sage.numerical.backends extensions

650c661  sage_setup/command/sage_install.py: WIP: Disable cleaning

comment:50 Changed 20 months ago by
 Work issues rebase on top of #29701 deleted
Rebased on top of #29701
comment:51 Changed 20 months ago by
 Dependencies changed from #29702, #29701 to #29701
comment:52 Changed 19 months ago by
 Description modified (diff)
 Work issues set to move cleaning modifications to #30152, break sage_install into several classes for the different use cases
comment:53 Changed 19 months ago by
 Commit changed from 650c661e50e10f57da6f121942158f654de835ec to 474b2007d3b68cbf14d10feb1cc3b2750bf0c02e
Branch pushed to git repo; I updated commit sha1. New commits:
14396af  Merge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension

d62da15  sage_setup.command.sage_build: Add the extensions to the distribution

06a3609  sage_setup.find.find_python_sources: Add benchmark doctest

174626c  Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension

034a7f7  Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension

d3c608b  Merge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension

d99e942  Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

474b200  Merge tag '9.2.beta5' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:54 Changed 19 months ago by
 Description modified (diff)
comment:55 Changed 18 months ago by
 Commit changed from 474b2007d3b68cbf14d10feb1cc3b2750bf0c02e to 2e79d33db38da4da9a87561f534f6b6ca813904a
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
eaadf77  sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)

59151ab  sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

713a838  Mark src and src/sage/ext_data as directories that are not namespace packages

4faff43  sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files

de17c1c  is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython

6761b7d  src/module_list.py: Remove explicit list of sage.numerical.backends extensions

65f3739  sage_setup/command/sage_install.py: WIP: Disable cleaning

8c20f3e  Revert "sage_setup/command/sage_install.py: WIP: Disable cleaning"

37bd1ca  sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install

2e79d33  WIP

comment:56 Changed 18 months ago by
 Commit changed from 2e79d33db38da4da9a87561f534f6b6ca813904a to 98a1b883f8cab1fc5d3ec829a937acfa148b6347
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
8a19fe2  build/make/Makefile.in (sagelibclean): Clean the new build location

ccc67b0  src/sage_setup: Update cythonized_dir in doctests

df38027  Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30277/remove_src_module_list_py

ef4e554  Merge branch 't/30277/remove_src_module_list_py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

6282b8d  sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)

7c10680  sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

6e68259  Mark src and src/sage/ext_data as directories that are not namespace packages

b71c3a6  sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files

3a12146  is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython

98a1b88  sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install

comment:57 Changed 18 months ago by
 Dependencies changed from #29701 to #30277
 Work issues move cleaning modifications to #30152, break sage_install into several classes for the different use cases deleted
comment:58 Changed 18 months ago by
 Description modified (diff)
 Summary changed from Modify find_python_sources, clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420) to Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420)
comment:59 Changed 18 months ago by
 Commit changed from 98a1b883f8cab1fc5d3ec829a937acfa148b6347 to 742ab8777c8010a2e3d6435428c8baa7acdea128
comment:60 Changed 18 months ago by
 Work issues set to installed_files_by_module
comment:61 Changed 18 months ago by
 Description modified (diff)
comment:62 Changed 17 months ago by
 Cc ghtobiasdiez added
comment:63 Changed 17 months ago by
 Commit changed from 742ab8777c8010a2e3d6435428c8baa7acdea128 to a967aca4491de98118612f30cbdf8bd75ab4efa9
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
776c979  sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)

2e61ada  sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

26b1ca0  Mark src and src/sage/ext_data as directories that are not namespace packages

a733af5  sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files

c08c511  is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython

4407e5e  sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install

a13c126  Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages

a967aca  WIP

comment:64 Changed 17 months ago by
 Dependencies #30277 deleted
comment:65 Changed 17 months ago by
 Dependencies set to #30578
comment:66 Changed 17 months ago by
 Work issues changed from installed_files_by_module to installed_files_by_module; rebase on top of #30578
comment:67 Changed 16 months ago by
 Dependencies changed from #30578 to #30780
 Work issues changed from installed_files_by_module; rebase on top of #30578 to installed_files_by_module; rebase on top of #30780
comment:68 Changed 16 months ago by
 Commit changed from a967aca4491de98118612f30cbdf8bd75ab4efa9 to a4e5683b1ecf6121bc766bf60bd90d312371b9cf
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
7980130  Merge branch 't/30779/duplicate_src_setup_py' into t/30780/sage_setup__separate_installing_and_cleaning

9b95628  sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in build/pkgs/sagelib/src/setup.py; refactor through new class sage_install

886ef6d  Merge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

3b9559d  sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)

88085b1  sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

9c39a76  Mark src and src/sage/ext_data as directories that are not namespace packages

b060462  sage_setup.find.find_python_sources: Handle native namespace packages, 'nonamespace' files

fe9a7d5  is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython

d2fe380  Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages

a4e5683  WIP

comment:69 Changed 16 months ago by
 Work issues changed from installed_files_by_module; rebase on top of #30780 to installed_files_by_module
comment:70 Changed 16 months ago by
 Commit changed from a4e5683b1ecf6121bc766bf60bd90d312371b9cf to 8b37214c8c3bad2a3e046798f3b6ada33580dd31
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
d42aa12  is_package_or_namespace_package_dir: New, recognize namespace packages by file 'namespace', monkeypatch into Cython

29ca24b  Make sage.graphs, sage.matrix, sage.libs, sage.interfaces namespace packages

1957acd  src/sage_setup/clean.py: Update comment

8b37214  WIP

comment:71 Changed 16 months ago by
 Commit changed from 8b37214c8c3bad2a3e046798f3b6ada33580dd31 to 44a44560071ce851fce1e415000b13bc0cda3d90
comment:72 Changed 16 months ago by
 Work issues installed_files_by_module deleted
comment:73 Changed 16 months ago by
 Status changed from new to needs_review
comment:74 Changed 16 months ago by
 Commit changed from 44a44560071ce851fce1e415000b13bc0cda3d90 to 88455dc42bfe6639c6f8dfc91f3d2fca3c51fb3e
Branch pushed to git repo; I updated commit sha1. New commits:
ad9402f  sage_setup.command.sage_install: Fixup: Remove cleaning code (moved to sage_install_and_clean)

88455dc  Merge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:75 Changed 16 months ago by
 Commit changed from 88455dc42bfe6639c6f8dfc91f3d2fca3c51fb3e to a9933b6b1282992d49976ce0ec586aa97e1d4c26
Branch pushed to git repo; I updated commit sha1. New commits:
a9933b6  sage_build_cython: Pass built_distributions to find_extra_files

comment:76 Changed 16 months ago by
 Dependencies changed from #30780 to #30780, #30784
comment:77 Changed 16 months ago by
 Commit changed from a9933b6b1282992d49976ce0ec586aa97e1d4c26 to e4ce7207371e0d27eb1747fbe03855af108beb57
Branch pushed to git repo; I updated commit sha1. New commits:
ea0d15a  sage.matrix: Resolve circular imports without using __init__.py

e4ce720  Merge branch 't/30784/sage_matrix__resolve_circular_imports_without_using___init___py' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:78 Changed 16 months ago by
 Description modified (diff)
comment:79 Changed 16 months ago by
 Commit changed from e4ce7207371e0d27eb1747fbe03855af108beb57 to 6aaa95887be0f4f5257208e6662cd3ca821bc0bc
Branch pushed to git repo; I updated commit sha1. New commits:
6aaa958  sage_install_and_clean: Move all_distributions into class

comment:80 Changed 16 months ago by
Ready for review
comment:81 Changed 16 months ago by
 Commit changed from 6aaa95887be0f4f5257208e6662cd3ca821bc0bc to 39be6390197b4c39382420e27653c83dd99a3503
Branch pushed to git repo; I updated commit sha1. New commits:
39be639  FileDocTestSource.in_lib: Handle namespace packages

comment:82 Changed 15 months ago by
 Commit changed from 39be6390197b4c39382420e27653c83dd99a3503 to 64c5120b0a03c5a907cb56f63bf9df360976bd44
Branch pushed to git repo; I updated commit sha1. New commits:
64c5120  find_python_sources: Also collect namespace packages  so that package_data works

comment:83 Changed 15 months ago by
 Description modified (diff)
comment:84 followup: ↓ 85 Changed 15 months ago by
What's the advantage/reason to introduce the sagespecific namespace/nonamespace convention instead of following PEP 420 and keep the __init__.py
file as a marker for namespace packages together with setuptools.find_namespace_packages()
?
comment:85 in reply to: ↑ 84 Changed 15 months ago by
Replying to ghtobiasdiez:
What's the advantage/reason to introduce the sagespecific namespace/nonamespace convention instead of following PEP 420 and keep the
__init__.py
file as a marker for namespace packages together withsetuptools.find_namespace_packages()
?
One cannot keep __init__.py
for native namespace packages. PEP 420 is quite clear about it. Subpackages of namespace packages do have __init__.py
; and also the present ticket keeps them.
comment:86 Changed 15 months ago by
find_namespace_packages
accepts any directory as a package directory:
class PEP420PackageFinder(PackageFinder): @staticmethod def _looks_like_package(path): return True
But our source tree has some directories that should not be considered package directories. I have marked these with the nonamespace
file.
Having both nonamespace
and namespace
files results in simpler changes to the existing code.
comment:87 followup: ↓ 89 Changed 15 months ago by
Oh ok, sorry, I've misunderstood the ticket description. You are of course right, implicit namespace packages don't have a __init__.py
file.
Concerning the package discovery, I would still prefer using setuptools.find_namespace_packages()
, and then filter out unwanted folders based on a hardcoded list. The (no)namespace
files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by PEP 382, which was rejected in favor of PEP 420.
If you do prefer to have these files, I would like to ask you to add documentation to the dev docs.
comment:88 Changed 15 months ago by
 Commit changed from 64c5120b0a03c5a907cb56f63bf9df360976bd44 to 3c7d6751b4bdc7c419ecc94a2add7b1da21be212
Branch pushed to git repo; I updated commit sha1. New commits:
3c7d675  src/sage_setup/find.py (find_python_sources): Update doc, doctests

comment:89 in reply to: ↑ 87 ; followup: ↓ 115 Changed 15 months ago by
Replying to ghtobiasdiez:
Concerning the package discovery, I would still prefer using
setuptools.find_namespace_packages()
, and then filter out unwanted folders based on a hardcoded list. The(no)namespace
files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by PEP 382, which was rejected in favor of PEP 420.
Thanks for the input and the pointer to PEP 382. I'll try out if your suggestion can be implemented without major changes.
comment:90 Changed 15 months ago by
 Status changed from needs_review to needs_work
TBD: In incremental builds, the cleaner fails to remove previously installed __init__.py
files:
********************************************************************** File "src/sage_setup/find.py", line 343, in sage_setup.find.installed_files_by_module Failed example: files_by_module['sage.graphs.graph_decompositions'] Expected nothing Got: {'sage/graphs/graph_decompositions/__init__.py', 'sage/graphs/graph_decompositions/__pycache__/__init__.cpython37.pyc'} **********************************************************************
comment:91 Changed 15 months ago by
 Description modified (diff)
comment:92 Changed 15 months ago by
Note that one annoying thing about find_namespace_packages
when applied to our source tree is that we have lots of directories that do not contain any python files, leading to output like this from distutils:
[sagelib9.2.rc2] package init file 'sage/schemes/generic/notes/__init__.py' not found (or not a regular file)
The branch on this ticket handles such directories specially (no need for nonamespace
files there). If there is a solution that could silence these notes, the code could be brought closer to find_namespace_packages
behavior.
comment:93 Changed 15 months ago by
find_namespace_packages
excepts include
and exclude
directories as arguments. So one option would be to get a list of all folders not including any python files, and pass this as exclude
(or the same approach using a whitelist).
comment:94 Changed 14 months ago by
 Keywords sd111 added
comment:95 Changed 14 months ago by
 Commit changed from 3c7d6751b4bdc7c419ecc94a2add7b1da21be212 to a6e8447748b8a2b4107c43a92daa2bcef30ff27b
Branch pushed to git repo; I updated commit sha1. New commits:
a6e8447  Merge tag '9.3.beta3' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

comment:96 Changed 14 months ago by
 Dependencies #30780, #30784 deleted
 Work issues set to comment 90
comment:97 Changed 12 months ago by
 Milestone changed from sage9.3 to sage9.4
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
comment:98 Changed 6 months ago by
 Milestone changed from sage9.4 to sage9.5
Setting a new milestone for this ticket based on a cursory review.
comment:99 Changed 2 months ago by
 Commit changed from a6e8447748b8a2b4107c43a92daa2bcef30ff27b to f017b3f879f77e70425ac65174982ccf9044baf4
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
f017b3f  #28925 Modify find_python_sources to support modularization (squashed)

comment:100 Changed 2 months ago by
 Commit changed from f017b3f879f77e70425ac65174982ccf9044baf4 to bfaeb1488e1048c3e51a898bd6eddbbb8d9a7e13
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
bfaeb14  28925: Modify find_python_sources to support modularization (squashed)

comment:101 Changed 2 months ago by
 Dependencies set to #32927
comment:102 Changed 2 months ago by
 Commit changed from bfaeb1488e1048c3e51a898bd6eddbbb8d9a7e13 to 15813aa5f45d8cecf80ce12bbb1966ece00cace9
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
c6fc111  Prettier diagram

7d6a44c  Use :mod: as markup for packages/modules

f83d424  Improve ABC example

817a8d0  src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup

b555735  src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata

81e9c9a  Merge #32899

ecc0886  Merge #31386

00d5f59  build/pkgs/sagelib: Invoke install cleaner explicitly, remove it from pkgs/sagemathstandard/setup.py

a5aab69  Merge #32927

15813aa  Remove __init__.py files

comment:103 Changed 7 weeks ago by
 Dependencies changed from #32927 to #32927, #29865
comment:104 Changed 7 weeks ago by
 Description modified (diff)
comment:105 Changed 7 weeks ago by
 Commit changed from 15813aa5f45d8cecf80ce12bbb1966ece00cace9 to 864874173a842a21f1087fab2367f4bec18bd351
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
a72ee3b  src/sage/all__sagemath_objects.py: Add some more imports

f3ee49b  src/sage/all__sagemath_objects.py: Add definitions of Integer, RealNumber for doctesting

2a4a323  Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap

9ef0567  Merge #29039

e0399a1  pkgs/sagemathobjects: Install a fixed set of scripts, not all of src/bin

7519048  src/bin/sage[env]: Put SAGE_ROOT/src/bin in front of path only if run out of this directory

00024e6  Merge #32933

bfc953b  src/sage/doctest/reporting.py: Indicate environment if not default

36d6b09  pkgs/sagemathcategories/setup.cfg.m4: Add scripts

8648741  Merge #29865

comment:106 Changed 7 weeks ago by
 Commit changed from 864874173a842a21f1087fab2367f4bec18bd351 to fcfdf1de2b481d149f2d9b17cff862f997132a43
Branch pushed to git repo; I updated commit sha1. New commits:
fcfdf1d  Restore __init__.py files for #28925

comment:107 Changed 7 weeks ago by
 Commit changed from fcfdf1de2b481d149f2d9b17cff862f997132a43 to 31b9c16578aec755cad21567fa9c5812c1d85a53
Branch pushed to git repo; I updated commit sha1. New commits:
31b9c16  src/sage_setup/command/sage_install.py (sage_clean): Do not fail if dist.py_modules is None

comment:108 Changed 7 weeks ago by
 Commit changed from 31b9c16578aec755cad21567fa9c5812c1d85a53 to 03909ae71dc00824e1e4a67f468b20a0553169d3
Branch pushed to git repo; I updated commit sha1. New commits:
f5fd073  pkgs/sagemathobjects/MANIFEST.in: Remove __init__.py files

67d78e7  pkgs/sagemathcategories/MANIFEST.in.m4: Remove src/sage/categories/__init__.py

4912f15  src/sage/env.py: Get SAGE_LIB via sage.env's __file__

03909ae  src/sage/env.py (SAGE_LIB): Fix up

comment:109 Changed 6 weeks ago by
 Milestone changed from sage9.5 to sage9.6
comment:110 Changed 6 weeks ago by
 Dependencies changed from #32927, #29865 to #32927, #29865, #33033
comment:111 Changed 6 weeks ago by
 Commit changed from 03909ae71dc00824e1e4a67f468b20a0553169d3 to c7f871899760c1d9c750c712a74d667b16c37e02
Branch pushed to git repo; I updated commit sha1. New commits:
8c8c29c  WIP

c3f6bcb  src/sage/misc/namespace_package.py (is_package_or_sage_namespace_package_dir): New

ab23a14  src/sage/doctest/sources.py: Use is_package_or_sage_namespace_package_dir

0f05856  src/sage/doctest/sources.py: Remove unnecessary conversion to bool

d8aa37c  src/sage/misc/namespace_package.py: In doctests, use 'directory' instead of the singleletter variable name

2cf8d8d  Merge #33033

c7f8718  src/sage_setup/find.py: Use is_package_or_sage_namespace_package_dir

comment:112 Changed 6 weeks ago by
 Commit changed from c7f871899760c1d9c750c712a74d667b16c37e02 to f3764bc2c73149b9a369027fc0c001f8243918f9
Branch pushed to git repo; I updated commit sha1. New commits:
f3764bc  src/sage_setup/find.py: Get rid of use of 'nonamespace' files

comment:113 Changed 6 weeks ago by
 Commit changed from f3764bc2c73149b9a369027fc0c001f8243918f9 to 6089ae892c72a41033524c137a6e6cecc853b4b6
comment:114 Changed 6 weeks ago by
 Description modified (diff)
comment:115 in reply to: ↑ 89 ; followup: ↓ 122 Changed 6 weeks ago by
Replying to mkoeppe:
Replying to ghtobiasdiez:
Concerning the package discovery, I would still prefer using
setuptools.find_namespace_packages()
, and then filter out unwanted folders based on a hardcoded list. The(no)namespace
files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by PEP 382, which was rejected in favor of PEP 420.Thanks for the input and the pointer to PEP 382. I'll try out if your suggestion can be implemented without major changes.
Done now via #33033.
comment:116 Changed 6 weeks ago by
 Work issues comment 90 deleted
comment:117 Changed 6 weeks ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
comment:118 Changed 6 weeks ago by
 Commit changed from 6089ae892c72a41033524c137a6e6cecc853b4b6 to 37a957915d1bac88f011cd185f400b9483a455b6
comment:119 Changed 6 weeks ago by
 Commit changed from 37a957915d1bac88f011cd185f400b9483a455b6 to 2f6d47334df875115f6d65611b83a00af21314d5
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
55e258d  pkgs/sagemathcategories/MANIFEST.in.m4: Remove src/sage/categories/__init__.py

07ffddb  src/sage/env.py: Get SAGE_LIB via sage.env's __file__

320f2e0  src/sage/env.py (SAGE_LIB): Fix up

85ef02e  WIP

95a7205  src/sage_setup/find.py: Use is_package_or_sage_namespace_package_dir

6ab1bd7  src/sage_setup/find.py: Get rid of use of 'nonamespace' files

0b33605  src/sage_setup/find.py: Add doctest result

2e995dd  src/sage/{arith,ext,libs,rings,sets}/all__sagemath_objects.py

f244b57  pkgs/sagemathobjects/MANIFEST.in: Exclude generated files

2f6d473  pkgs/sagemathcategories/MANIFEST.in.m4: Exclude generated files

comment:120 Changed 6 weeks ago by
 Dependencies changed from #32927, #29865, #33033 to #29865, #33033
 Description modified (diff)
Rebased away from #32927.
comment:121 Changed 6 weeks ago by
 Commit changed from 2f6d47334df875115f6d65611b83a00af21314d5 to f5addb34e5dc66d461a757df1f9620dbe5184230
comment:122 in reply to: ↑ 115 ; followups: ↓ 123 ↓ 124 Changed 6 weeks ago by
Replying to mkoeppe:
Replying to mkoeppe:
Replying to ghtobiasdiez:
Concerning the package discovery, I would still prefer using
setuptools.find_namespace_packages()
, and then filter out unwanted folders based on a hardcoded list. The(no)namespace
files just add complexity (especially for new developers). They are also very similar to the explicit declaration of namespace packages proposed by PEP 382, which was rejected in favor of PEP 420.Thanks for the input and the pointer to PEP 382. I'll try out if your suggestion can be implemented without major changes.
Done now via #33033.
Probably I'm missing something here. The new mechanism seems to still rely on indicator/marker files, just that the are now named all__
. Except for the different name this seems to be exactly the approach of the rejected PEP 382.
The point of my proposal was that only the setup file should know/decide on which packages it want to include and exclude. Then the general helper in #33033 would like
is_namespace_package(dir): return dir in setuptools.find_namespace_packages(include=..., exclude=...)
And the setup.cfg would use https://setuptools.pypa.io/en/latest/userguide/package_discovery.html.
comment:123 in reply to: ↑ 122 Changed 6 weeks ago by
Replying to ghtobiasdiez:
The new mechanism seems to still rely on indicator/marker files, just that the are now named
all__
. Except for the different name this seems to be exactly the approach of the rejected PEP 382.
That's certainly true, but at least the Sage source tree already has these files
comment:124 in reply to: ↑ 122 Changed 6 weeks ago by
Replying to ghtobiasdiez:
The point of my proposal was that only the setup file should know/decide on which packages it want to include and exclude. Then the general helper in #33033 would like
is_namespace_package(dir): return dir in setuptools.find_namespace_packages(include=..., exclude=...)
The helper function is added in #33033 for runtime use (doctesting). Doctesting needs to be able to work on (a) Sage source trees, (b) installed Sage packages (this is important, for example, for the downstream packagers), (c) arbitrary files. Hardcoding inclusions and exclusions in the helper function seems a bit unmodular
comment:125 Changed 6 weeks ago by
In the end it comes down to who should have the control:
 Should the doctest code decide what to test, and the setup.py/cfg file what to distribute and the sage startup script what to expose
 or should the module/package be responsible to declare that it should be tested, distributed and exposed
Currently, this ticket (and the related ones) go in the direction of the second approach by using all.py
. For me, using inversion of control and following the first approach seems more natural and flexible. Since you have a better overview though, I trust your judgement.
comment:126 Changed 3 days ago by
 Commit changed from f5addb34e5dc66d461a757df1f9620dbe5184230 to 178d111960e751d5d34bf3097ddd39c65f24c42d
pushing these forward to 9.2