Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories

[mkoeppe]

This ticket creates two experimental packages ​https://pypi.org/project/sagemath-objects/ and ​https://pypi.org/project/sagemath-categories/, outlined in Meta-ticket #29705, that consist of small subsets of the modules of sagelib, making Sage objects, the element/parent framework, categories, the coercion system and the related metaclasses available.

To build an sdist:

./sage -sh build/pkgs/sagemath_objects/spkg-src
./sage -sh build/pkgs/sagemath_categories/spkg-src

This creates a pip-installable source distribution sagemath-objects-VERSION.tar.gz (or sagemath-categories-VERSION.tar.gz) in upstream/.

The distributions depend only on Cython, pkgconfig, cysignals, gmpy2. The latter, of course, requires various headers and libraries, so they are not quite "purely" pip-installable packages.

The goal is for all modules of sagemath-objects and sagemath-categories to be importable even when sagelib is not installed. This can be tested (with tox) as follows:

./bootstrap && ./sage -sh -c '(cd build/pkgs/sagemath_objects/src && tox -v -v -e py39)'
./bootstrap && ./sage -sh -c '(cd build/pkgs/sagemath_categories/src && tox -v -v -e py39)'

sagemath-categories is intended to have sagemath-objects as a prerequisite. However, currently, sagemath-categories is an independent distribution that provides a superset of the modules of sagemath-objects.

Recombining it with the rest of sagelib via namespace package magic as outlined in #29705 is not implemented on this ticket. This will be done in follow-up tickets.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​889f0c6	sage.env.sage_include_directories: Do not fail if numpy cannot be imported
​dd3dc15	build/pkgs/sage_objects: Add dependencies, get minimal tox working

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4a99c1c	Add comment
​b72264e	WIP: More stuff

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​23aaec1

Version with sage.structure, sage.categories that builds OK; uses gmpy2

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​cbc5555	sage.misc.misc, sage.misc.persist, sage.structure.formal_sum: Make some imports local to functions
​0b7e11f	On the way to get sage.structure.all importing
​19f86dc	sage.env: Do not die if DOT_SAGE is None
​a795d56	sage.structure.factorization: Make import of sage.rings.integer.Integer local to methods
​0ce9916	sage.structure.sequence: Make import of sage.misc.latex local to the latex method
​fa22e19	sage.structure.all: Import .formal_sum lazily to avoid pulling in sage.modules
​acf1ba3	sage.structure.factorization: Make import of sage.misc.misc_c.prod local to methods
​14b1e55	sage.structure.formal_sum, .parent_gens: Fixup: Remove imports
​a5453bf	Fixup: Add src/sage/misc/call.py
​29e30a4	Merge branch 't/29869/move_attrcall_and_friends_from_sage_misc_misc_to_new_module_sage_misc_call' into t/29865-modularizarion-sage-objects

[mkoeppe]

import sage.structure.all succeeds now

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c4a68e3

build/pkgs/sage_objects/src/tox.ini: Add comments, test more

[mkoeppe]

Some of the Python modules in sage.categories seem to be sensitive to import order.

$ build/pkgs/sage_objects/src/.tox/python/bin/python -c 'import sage.categories.map'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "sage/categories/map.pyx", line 1, in init sage.categories.map (build/cythonized/sage/categories/map.c:16980)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sage_objects/src/.tox/python/lib/python3.7/site-packages/sage/structure/__init__.py", line 2, in <module>
    import sage.structure.element
  File "sage/structure/parent.pxd", line 12, in init sage.structure.element (build/cythonized/sage/structure/element.c:34583)
  File "sage/categories/map.pxd", line 4, in init sage.structure.parent (build/cythonized/sage/structure/parent.c:27689)
AttributeError: module 'sage.categories.map' has no attribute 'Map'

vs.

$ build/pkgs/sage_objects/src/.tox/python/bin/python -c 'import sage.structure.all, sage.categories.map'
# succeeds

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1963567	sage.categories.crystals: Make import of sage.misc.latex local to a method
​c557714	sage.categories: Make imports from sage.rings, .sets, .combinat, .plot, .matrix local to methods
​7cca294	Now we can import sage.categories.all

[mkoeppe]

$ build/pkgs/sage_objects/src/.tox/python/bin/python -c 'import sage.structure.all, sage.categories.all'
# succeeds

but does not succeed without importing sage.structure.all first...

[mkoeppe]

build/pkgs/sage_objects/src/.tox/python/bin/python -c 'import sage.structure.all, sage.categories.all, sage.categories.facade_sets; sage.categories.facade_sets.FacadeSets().example()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "sage/misc/classcall_metaclass.pyx", line 322, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1717)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sage_objects/src/.tox/python/lib/python3.7/site-packages/sage/categories/category_with_axiom.py", line 1993, in __classcall__
    (base_category_class, axiom) = cls._base_category_class_and_axiom
  File "sage/misc/lazy_attribute.pyx", line 699, in sage.misc.lazy_attribute.lazy_class_attribute.__get__ (build/cythonized/sage/misc/lazy_attribute.c:2894)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sage_objects/src/.tox/python/lib/python3.7/site-packages/sage/categories/category_with_axiom.py", line 1933, in _base_category_class_and_axiom
    base_category_class, axiom = base_category_class_and_axiom(cls)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sage_objects/src/.tox/python/lib/python3.7/site-packages/sage/categories/category_with_axiom.py", line 1790, in base_category_class_and_axiom
    assert getattr(base_category_class, axiom, None) is cls, \
  File "sage/misc/lazy_import.pyx", line 498, in sage.misc.lazy_import.LazyImport.__get__ (build/cythonized/sage/misc/lazy_import.c:4236)
  File "sage/misc/lazy_import.pyx", line 188, in sage.misc.lazy_import.LazyImport.get_object (build/cythonized/sage/misc/lazy_import.c:2340)
  File "sage/misc/lazy_import.pyx", line 217, in sage.misc.lazy_import.LazyImport._get_object (build/cythonized/sage/misc/lazy_import.c:2482)
RuntimeError: resolving lazy import FacadeSets during startup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2c2ddb7

build/pkgs/sage_objects/src/setup.py: Remove unused imports

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​5d697ce	sage.structure.sequence: Make import of sage.misc.latex local to the latex method
​dcb7c73	sage.structure.all: Import .formal_sum lazily to avoid pulling in sage.modules
​30d3647	sage.structure.factorization: Make import of sage.misc.misc_c.prod local to methods
​24b26d8	sage.structure.formal_sum, .parent_gens: Fixup: Remove imports
​5b98ba3	build/pkgs/sage_objects/src/tox.ini: Add comments, test more
​21baff3	sage.categories.crystals: Make import of sage.misc.latex local to a method
​90d3f6d	sage.categories: Make imports from sage.rings, .sets, .combinat, .plot, .matrix local to methods
​e88ceb4	Now we can import sage.categories.all
​ecc9415	build/pkgs/sage_objects/src/setup.py: Remove unused imports
​719cb6d	build/pkgs/sage_objects/src/setup.py: Remove code for namespace packages and distribution filtering

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b9d2a51

build/pkgs/sage_objects/: Add/update metadata

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​354a07f

build/pkgs/sage_objects/src/README.md: Add doc links

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5ec6042

build/pkgs/sage_objects/spkg-install: Replace by the script from sage_tdlib

[mkoeppe]

As part of this ticket I made a number of changes to import statements throughout sage.categories. ​https://git.sagemath.org/sage.git/diff/src/sage/categories/?id=5ec60422cff0e536e913c8e6306de45147b2f129&id2=e2dcdeeabb578c37bcf0361c0be3079315e9252c

Should I put them on a separate ticket to facilitate review?

[mkoeppe]

Replying to mkoeppe:

As part of this ticket I made a number of changes to import statements throughout sage.categories. ​https://git.sagemath.org/sage.git/diff/src/sage/categories/?id=5ec60422cff0e536e913c8e6306de45147b2f129&id2=e2dcdeeabb578c37bcf0361c0be3079315e9252c

Should I put them on a separate ticket to facilitate review?

This is now #29873

[mjo]

Can all of the functions/methods in sage-objects be run after a pip install, or is it only the import that works? (The existing doctests use various things from sage.all, so no one expects those to work.)

[mkoeppe]

This is unknown. Activating a part of the testsuite that can/does work, and finding the best way to test this, is TBD. I hope some of it can be done through mocking (see #29874).

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​3060ddd	On the way to get sage.structure.all importing
​db1c595	sage.env: Do not die if DOT_SAGE is None
​8b134c0	sage.structure.all: Import .formal_sum lazily to avoid pulling in sage.modules
​cb0b850	build/pkgs/sage_objects/src/tox.ini: Add comments, test more
​ad502bb	Now we can import sage.categories.all
​d030578	build/pkgs/sage_objects/src/setup.py: Remove unused imports
​2cf8312	build/pkgs/sage_objects/src/setup.py: Remove code for namespace packages and distribution filtering
​3790abe	build/pkgs/sage_objects/: Add/update metadata
​d67d4ed	build/pkgs/sage_objects/src/README.md: Add doc links
​907c9f9	build/pkgs/sage_objects/spkg-install: Replace by the script from sage_tdlib

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dc49119

.github/workflows/tox-optional.yml: Test sage_objects

[mkoeppe]

Tests run at ​https://github.com/mkoeppe/sage/actions/runs/139194878

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3f5b6ea	Move sage.misc.misc.coeff_repr, repr_lincomb to new module sage.misc.repr
​08fedfa	src/sage/combinat/root_system/type_dual.py: Remove unused variable to fix pyflakes warning
​05efc11	sage.misc.repr.coeff_repr: Add doctest, adapted from sage.misc.latex.coeff_repr
​34d38a5	Merge branch 't/29892/move_sage_misc_misc_coeff_repr__repr_lincomb_to_new_module_sage_misc_repr' into t/29865-modularizarion-sage-objects
​97199dd	sage.structure.parent_gens: Remove imports
​bb4c6a8	sage.structure.sequence: Make import of sage.misc.latex local to the latex method
​0820283	sage.structure.factorization: Make import of sage.rings.integer.Integer local to methods
​7991b64	sage.structure.factorization: Make import of sage.misc.misc_c.prod local to methods
​8ec06f0	Merge branch 't/29883/sage_structure__remove_some_module_level_imports' into t/29865-modularizarion-sage-objects
​152dac5	build/pkgs/sage_objects/dependencies: Add more

[mkoeppe]

​https://github.com/mkoeppe/sage/runs/785565655:

  [sage_objects-9.1.rc5]     File "setup.py", line 6, in <module>
  [sage_objects-9.1.rc5]       from sage_setup.command.sage_build_cython import sage_build_cython
  [sage_objects-9.1.rc5]     File "/sage/build/pkgs/sage_objects/src/sage_setup/command/sage_build_cython.py", line 19, in <module>
  [sage_objects-9.1.rc5]       from sage_setup.library_order import library_order
  [sage_objects-9.1.rc5]     File "/sage/build/pkgs/sage_objects/src/sage_setup/library_order.py", line 54, in <module>
  [sage_objects-9.1.rc5]       m4ri_pc = pkgconfig.parse('m4ri')
  [sage_objects-9.1.rc5]     File "/sage/local/lib/python3.7/site-packages/pkgconfig/pkgconfig.py", line 248, in parse
  [sage_objects-9.1.rc5]       _raise_if_not_exists(package)
  [sage_objects-9.1.rc5]     File "/sage/local/lib/python3.7/site-packages/pkgconfig/pkgconfig.py", line 103, in _raise_if_not_exists
  [sage_objects-9.1.rc5]       raise PackageNotFoundError(package)
  [sage_objects-9.1.rc5]   pkgconfig.pkgconfig.PackageNotFoundError: m4ri not found
  [sage_objects-9.1.rc5] Full log file: /sage/logs/pkgs/sage_objects-9.1.rc5.log

[mkoeppe]

Also, of course, installation of sage_objects breaks other things in the tests:

(build/cythonized/sage/plot/plot3d/shapes.c:12604)
  File "sage/ext/interpreters/wrapper_rdf.pxd", line 7, in init sage.plot.plot3d.parametric_surface (build/cythonized/sage/plot/plot3d/parametric_surface.c:12057)
ModuleNotFoundError: No module named 'sage.ext.interpreters.wrapper_rdf'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4867555	build/pkgs/sage_objects: Do not install, only test with tox, for now
​12788dd	sage.misc.misc, sage.misc.persist, sage.structure.formal_sum: Make some imports local to functions

[mkoeppe]

I'll probably make sage-objects smaller by only including sage.categories.category rather than all of sage.categories; and add a package sage-categories that is larger than the current version, including also sage.rings.ring and similar abstract classes. (On this ticket, sage-categories would just be a larger package whose MANIFEST.in is a superset of that of sage-objects. It will be done by dependencies later.)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c845235	sage.categories.all: Resolve a circular import
​b982fdc	build/pkgs/sage_objects/src/tox.ini: Add test

[mkoeppe]

This will need #29916 (sage.structure: Replace import of sage.categories.all by more specific imports)

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​94ec3c9	Add sage --package update-latest command
​cb7a291	Fixup rebase
​1d36a2e	build/bin/sage-system-python: Try python first
​a91ecab	Merge branch 't/29890/build_bin_sage_system_python__try__python__first' into t/20104/automatic_updates_of_pip_type_packages
​ba6fbb5	Merge branch 't/20104/automatic_updates_of_pip_type_packages' into t/29901/_github_workflows___test_all_optional_packages__add_workflow_for_testing_all_experimental_packages
​4dc7367	.github/workflows: Replace explicit optional/experimental lists by using sage-package
​efb14c3	Suppress linebreaks
​bd703e4	.github/workflows/tox-optional.yml: Do not test huge packages
​d6e56e8	Merge branch 't/29901/_github_workflows___test_all_optional_packages__add_workflow_for_testing_all_experimental_packages' into t/29865-modularizarion-sage-objects
​0a5ded7	build/pkgs/sage_categories: New (lightly edited copy of sage_objects)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​77cae45

build/pkgs/sage_objects: Only include the most basic categories

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​468c088	build/pkgs/sage_objects: Install some scripts from bin
​f46acbd	build/pkgs/sage_objects: Add the doctester

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​210c1a7	sage.plot: Import sage.misc.verbose
​8a5e384	Fix remaining imports of *verbose*
​5632556	src/sage/modular/modsym/relation_matrix_pyx.pyx: Fixup import
​41d42e9	sage.misc.verbose: Fix imports
​b3af9e2	sage.misc.verbose: Fix doctest
​d8abdb0	Fixup various imports and uses of verbose
​db40fd5	Fix pyflakes warnings regarding imports
​b7da100	Merge branch 't/29895/move_sage_misc_misc__verbose__to_new_module_sage_misc_verbose' into t/29865-modularizarion-sage-objects
​9f7f65c	Break out *.all__sage_objects from sage.all, sage.misc.all, sage.categories.all
​386bf91	build/pkgs/sage_objects: WIP: Run doctester

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​ad00b40	sage.doctest, src/bin/sage-runtests: Remove handling of sagenb
​faef943	Merge branch 't/29940/sage_doctest__remove_handling_of_sagenb' into t/29922/sage_doctest__make_imports_more_specific__make_global_environment_for_tests_customizable
​67db00f	src/bin/sage-runtests, sage.doctest: Handle option --environment
​9bf6f88	sage.doctest.control: Remove unused import
​89988cf	sage.doctest.forker.init_sage: Actually handle the optional arg
​2664865	sage.misc.package: Do not fail if SAGE_PKGS, SAGE_SPKG_INST do not exist
​2a602df	sage.repl.display.fancy_repr: Do not fail if sage.matrix.matrix1 canno be imported
​9e808da	Merge branch '29922'
​bdb162d	src/sage: Move more to all__sage_objects.py
​224ad51	build/pkgs/sage_objects: Make doctesting work

[mkoeppe]

Doctesting sage-objects now "works" thanks to #29922. Of course, most doctests fail because all the concrete classes that are used in the examples are not defined. And the preparsed language also requires Integer and RealNumber, which we do not have.

[mkoeppe]

Replying to mkoeppe:

I'll probably make sage-objects smaller by only including sage.categories.category rather than all of sage.categories;

this is done (but I temporarily added modules to do doctesting)

and add a package sage-categories that is larger than the current version, including also sage.rings.ring and similar abstract classes.

This is probably best to do if we can include Integer as well, so that depends on #29911 (remove ntl dependency)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​14396af	Merge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension
​5eafbc7	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29865-modularizarion-sage-objects

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​a5bc828	src/sage/misc/sageinspect.py: Fixup fixup
​1baaa68	src/sage/misc/sageinspect.py: Remove unused import
​6734969	Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29911/sage_rings_integer__integer_ring__remove_dependencies_on_sage_libs_ntl
​5e9f00c	src/sage/rings/integer*.pyx: Remove 'libraries = ntl'
​a9d4575	Merge branch 't/29911/sage_rings_integer__integer_ring__remove_dependencies_on_sage_libs_ntl' into t/29865-modularizarion-sage-objects
​e00aaec	Add from sage_objects MANIFEST.in
​739b401	Update from sage_objects tox.ini
​8f78aa8	sage.rings.ideal: Do not import sage.interfaces.singular at load time
​bac135f	build/pkgs/sage_categories/src/setup.cfg: Fix up
​1e82b2d	build/pkgs/sage_categories/src/MANIFEST.in: Add a bit from sage.rings

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​50a112c	build/pkgs/sage_categories/src/bin: New symlink
​fd81024	build/pkgs/sage_categories/src/MANIFEST.in: More tweaks

[mkoeppe]

Before we can add sage.rings.integer, we need #30022 (sage.rings.integer, rational: Remove compile-time dependency on cypari2 and flint).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4271679

Set up all__sage_categories.py

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​0915137	sage.rings.integer_ring.IntegerRing_class._coerce_ZZ: Remove, replaces uses by calls to ZZ_to_mpz
​8468ad1	src/sage/rings/integer*.pyx: Remove 'libraries = ntl'
​62e2aa9	Merge tag '9.2.beta6' into t/29911/sage_rings_integer__integer_ring__remove_dependencies_on_sage_libs_ntl
​0f3cd7b	Merge branch 't/29911/sage_rings_integer__integer_ring__remove_dependencies_on_sage_libs_ntl' into t/29865/t/29865-modularizarion-sage-objects
​d62da15	sage_setup.command.sage_build: Add the extensions to the distribution
​06a3609	sage_setup.find.find_python_sources: Add benchmark doctest
​174626c	Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension
​034a7f7	Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension
​d3c608b	Merge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension
​ef8b3a5	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29865/t/29865-modularizarion-sage-objects

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4bdafb7

build/pkgs/sage_{categories,objects}/src/MANIFEST.in: Add sage/misc/superseded

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​e40b198	build/pkgs/sage_objects: Make doctesting work
​1b64463	Add from sage_objects MANIFEST.in
​9ed18dd	Update from sage_objects tox.ini
​5627a0d	sage.rings.ideal: Do not import sage.interfaces.singular at load time
​8b74269	build/pkgs/sage_categories/src/setup.cfg: Fix up
​97fdb43	build/pkgs/sage_categories/src/MANIFEST.in: Add a bit from sage.rings
​7a180bb	build/pkgs/sage_categories/src/bin: New symlink
​691ebc0	build/pkgs/sage_categories/src/MANIFEST.in: More tweaks
​e947edf	Set up all__sage_categories.py
​41f67fb	build/pkgs/sage_{categories,objects}/src/MANIFEST.in: Add sage/misc/superseded

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cd12d4c

build/pkgs/sage_objects/src/requirements.txt: Update

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6bfe8d7

Merge tag '9.2' into t/29865/t/29865-modularizarion-sage-objects

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fa7b631	src/sage/doctest/control.py: Do not fail if sage.libs.arb cannot be imported
​1f6bc4c	build/pkgs/{sage_objects,sage_categories}/src/MANIFEST.in: Update for 9.2

[embray]

Big +1 from me. This would be very useful for #31297 and for gappy in general.

[embray]

Replying to mkoeppe:

I'll probably make sage-objects smaller by only including sage.categories.category rather than all of sage.categories; and add a package sage-categories that is larger than the current version, including also sage.rings.ring and similar abstract classes. (On this ticket, sage-categories would just be a larger package whose MANIFEST.in is a superset of that of sage-objects. It will be done by dependencies later.)

Okay. I was going to say, it seems like sage-objects still depends too heavily on too much of sage.categories. For my purposes I would want it to be as light-weight as possible.

I think it might also be worth it to further divide SageObject, Parent, and Element into some SageObjectBase, ParentBase, and ElementBase that only provide the minimal C-level type structs necessary to bootstrap the rest of the hierarchy.

For SageObject this is trivial because its type struct is empty. For Parent and Element the Base classes need to consist primarily of any cdef attributes which define the type struct. Anything else is optional and can be bolted on in a subclass.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9d3df20

Merge tag '9.3.beta6' into t/29865/t/29865-modularizarion-sage-objects

[mkoeppe]

Yes, I also hope that sagemath-objects can be eventually be made even smaller. But the coercion system pulls in quite a bit of the category system (in particular via pushout)

[nthiery]

This is really cool :-)

I won't alas have time to seriously follow the discussion. But if at some point it can be useful, I certainly can free myself for a live discussion.

[mkoeppe]

Thanks, Nicolas!

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​91559df

build/pkgs/sage_objects/src/setup.py: Add macOS multiprocessing fix from src/setup.py

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2cf7b04

Rename sage_objects, sage_categories to sagemath_objects, sagemath_categories, update metadata

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​991e547

Rename sage_objects, sage_categories to sagemath_objects, sagemath_categories, update metadata

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​86024cd

build/pkgs/sagemath_{objects,categories}/src/MANIFEST.in: Prune/exclude more

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8366cfd

build/pkgs/sagemath_{objects,categories}: Update README

[dimpase]

what's the status here?

[mkoeppe]

It works but I will want to update it according to the proposed new layout for embedded source trees (#31577)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​a1d1bb0	sage.misc.misc, sage.misc.persist, sage.structure.formal_sum: Make some imports local to functions
​3236920	sage.categories.all: Resolve a circular import
​4b33873	src/sage: Move more to all__sage_objects.py
​0a7f177	build/pkgs/sage_objects: Make doctesting work
​e1be016	Set up all__sage_categories.py
​1850deb	build/pkgs/sage_{categories,objects}/src/MANIFEST.in: Add sage/misc/superseded
​a4a29f5	build/pkgs/sage_objects/src/setup.py: Add macOS multiprocessing fix from src/setup.py
​d242689	Rename sage_objects, sage_categories to sagemath_objects, sagemath_categories, update metadata
​6df33cd	build/pkgs/sagemath_{objects,categories}/src/MANIFEST.in: Prune/exclude more
​a94f39e	build/pkgs/sagemath_{objects,categories}: Update README

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​9a6a4b2	sage.env: Do not die if DOT_SAGE is None
​206a6dc	sage.misc.misc, sage.misc.persist, sage.structure.formal_sum: Make some imports local to functions
​edb00b2	build/pkgs/sage_objects: New
​83ed0ca	sage.structure.all: Import .formal_sum lazily to avoid pulling in sage.modules
​d876c55	src/sage: Move more to all__sage_objects.py
​6ebf4f3	build/pkgs/sage_objects: Make doctesting work
​7e1d731	Set up all__sage_categories.py
​b7b8cf3	build/pkgs/sage_{categories,objects}/src/MANIFEST.in: Add sage/misc/superseded
​e1961d8	Move source trees from build/pkgs/sagemath_*/src to pkgs/sagemath-*

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​ae1f4ef

Move source trees from build/pkgs/sagemath_*/src to pkgs/sagemath-*

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7cc764a	Move source trees from build/pkgs/sagemath_*/src to pkgs/sagemath-*
​4550eb6	sage.env.sage_include_directories: Do not fail if numpy cannot be imported
​8209be8	Merge #31333

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1e80ee1

Merge tag '9.4.rc1' into t/29865/t/29865-modularizarion-sage-objects

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​df35a7c

Merge tag '9.4.rc1' into t/29865/t/29865-modularizarion-sage-objects

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​3299937	Makefile (pypi-sdists): Build sagemath_objects, sagemath_categories sdists
​290fa2f	build/pkgs/sagelib/src: Restore lost symlink
​478ffcd	Merge #32046
​b011389	Merge branch 't/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup' into t/29039/pip_installable_package_sage_bootstrap
​d800eea	Merge #29039
​bf09386	build/pkgs/sagemath_categories/spkg-src: Replace sage-python23 by python3
​5ab4e7c	pkgs/sagemath-{objects,categories}: Generate pyproject.toml, setup.cfg with m4; remove requirements.txt
​e38644a	Makefile (pypi-sdists): Add sage_setup
​e6b731c	Merge #29039
​4695948	pkgs/sagemath-objects/pyproject.toml.m4: Remove sage-conf from build-system requires

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​61ef2de

Merge tag '9.4.rc2' into t/29865/t/29865-modularizarion-sage-objects

[mkoeppe]

    File "setup.py", line 24, in <module>
      from sage.env import SAGE_VERSION
  ModuleNotFoundError: No module named 'sage'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7542209	pkgs/sagemath-{objects,categories}: Get version from VERSION.txt
​6777af1	pkgs/sagemath-objects/setup.py: Give sage_setup access to sage.env
​037ed2b	pkgs/sagemath-{objects,categories}/tox.ini: Remove use of requirements.txt

[mkoeppe]

    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-build-_o5ymrt1/sage_setup/library_order.py", line 14, in <module>
      aliases = cython_aliases()
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-build-_o5ymrt1/sage/env.py", line 451, in cython_aliases
      import pkgconfig
  ModuleNotFoundError: No module named 'pkgconfig'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b2368c6	pkgs/sage-setup/setup.cfg: Add pkgconfig as install_requires
​bad9189	Merge #29847
​8bdb61e	Merge #29039

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c66d5fc

pkgs/sagemath-{objects,categories}/MANIFEST.in: include VERSION.txt, remove vendoring of sage_setup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​20f29be

pkgs/sagemath-{objects,categories}/sage_setup: Restore symlink - needed for the tox sdist phase

[mkoeppe]

    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-build-env-f9779kgd/overlay/lib/python3.9/site-packages/sage_setup/command/sage_build_cython.py", line 208, in run
      from ..library_order import library_order
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-build-env-f9779kgd/overlay/lib/python3.9/site-packages/sage_setup/library_order.py", line 14, in <module>
      aliases = cython_aliases()
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-build-8_9lsvo_/sage/env.py", line 472, in cython_aliases
      aliases[var + "CFLAGS"] = pkgconfig.cflags(lib).split()
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-build-env-f9779kgd/overlay/lib/python3.9/site-packages/pkgconfig/pkgconfig.py", line 144, in cflags
      _raise_if_not_exists(package)
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-build-env-f9779kgd/overlay/lib/python3.9/site-packages/pkgconfig/pkgconfig.py", line 103, in _raise_if_not_exists
      raise PackageNotFoundError(package)
  pkgconfig.pkgconfig.PackageNotFoundError: fflas-ffpack not found

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4aff8a4	sage_setup.library_order: Skip libraries that were not detected
​796ef71	Merge #32373

[mkoeppe]

More of the same:

    File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-build-z0x28qby/sage_setup/command/sage_build_cython.py", line 225, in run
      aliases=cython_aliases(),
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-build-z0x28qby/sage/env.py", line 472, in cython_aliases
      aliases[var + "CFLAGS"] = pkgconfig.cflags(lib).split()
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-build-env-ti7oa28i/overlay/lib/python3.9/site-packages/pkgconfig/pkgconfig.py", line 144, in cflags
      _raise_if_not_exists(package)
    File "/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-build-env-ti7oa28i/overlay/lib/python3.9/site-packages/pkgconfig/pkgconfig.py", line 103, in _raise_if_not_exists
      raise PackageNotFoundError(package)
  pkgconfig.pkgconfig.PackageNotFoundError: fflas-ffpack not found

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​676111f	sage_setup.library_order: Take module list from new variables sage.env.default_{required,optional}_modules
​e5ae08c	Merge #32373
​4b5cee7	Revert "pkgs/sagemath-{objects,categories}/tox.ini: Remove use of requirements.txt"
​5bff677	pkgs/sagemath-{objects,categories}: Generate requirements.txt from ...m4 (for tox)
​6f512ef	pkgs/sagemath-objects/setup.py: Do not require any pkgconfig modules
​7f84f82	pkgs/sagemath-objects/pyproject.toml.m4: Add gmpy2, cysignals
​e3ec108	sage.misc.misc.cputime: Make import from sage.interfaces.quit local to this function
​d145883	pkgs/sagemath-objects/tox.ini: Ignore doctest failures
​bc767b1	pkgs/sagemath-categories/MANIFEST.in: Generate from ....m4
​4af99dd	pkgs/sagemath-categories/tox.ini: Ignore doctest failures

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bce252f

Merge tag '9.4' into t/29865/t/29865-modularizarion-sage-objects

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dda451f

Rename all__sage_objects.py, all__sage_categories.py to all__sagemath_objects.py, all__sagemath_categories.py

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5bb6e0d

Merge tag '9.5.beta1' into t/29865/t/29865-modularizarion-sage-objects

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​a23b1e1	pkgs/sage-conf_pypi/MANIFEST.in: Exclude build artifacts
​2f5be2d	configure.ac, build/pkgs/python3/spkg-configure.m4: New default --with-sage-venv=auto
​f700640	configure.ac: In config.status, create convenience symlinks prefix, venv
​b4a3845	m4/sage_spkg_collect.m4: If we install python3 spkg, install it in SAGE_VENV
​12cbbda	configure.ac: Keep old behavior of SAGE_VENV in incremental builds
​117fe3a	build/make/Makefile.in (...-clean): Do not fail trying to uninstall if SAGE_VENV has not been created yet
​ce1c344	Merge #32442
​e71b8a6	pkgs/sage-conf_pypi/setup.py: Key SAGE_ROOT only to Sage version and architecture
​9dcc189	pkgs/sage-conf_pypi/setup.py: Use --with-sage-venv
​5cf5e4d	Merge #29039

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d626aa8	build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no
​82ce539	configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.
​02505ca	pkgs/sage-conf_pypi/MANIFEST.in: Update from #31396
​aeca1a0	Makefile (pypi-sdists): Add sage_setup
​cd3dce9	pkgs/sage-conf_pypi/setup.py: Update directory of configured sage_conf.py
​c695921	pkgs/sage-conf/sage_conf.py.in (SAGE_SPKG_WHEELS): Update so it works if SAGE_VENV != SAGE_LOCAL
​96bcf03	Merge #29039

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e9fc48d	Merge tag '9.5.beta3' into t/29865/t/29865-modularizarion-sage-objects
​f79f0f8	pkgs/sagemath-objects/setup.py: On sdist or egg_info, do not run find_python_sources
​9600217	pkgs/sagemath-objects/tox.ini: passenv SAGE_NUM_THREADS*

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6bd807b

pkgs/sagemath-categories/tox.ini: passenv SAGE_NUM_THREADS*

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cebcda3	Merge tag '9.5.beta4' into t/29039/pip_installable_package_sage_bootstrap
​0026892	Merge tag '9.5.beta5' into t/29039/pip_installable_package_sage_bootstrap
​e068bd9	Merge #29039

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6187f93

Merge tag '9.5.beta6' into t/29865/t/29865-modularizarion-sage-objects

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a6cc585	build/pkgs/sage_setup/install-requires.txt: Fix version
​743e004	pkgs/sagemath-objects/MANIFEST.in: Add sage.misc.namespace_package

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a72ee3b	src/sage/all__sagemath_objects.py: Add some more imports
​f3ee49b	src/sage/all__sagemath_objects.py: Add definitions of Integer, RealNumber for doctesting

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2a4a323	Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap
​9ef0567	Merge #29039

[mkoeppe]

Let's get this in please

[klee]

This command

./bootstrap && ./sage -sh -c '(cd build/pkgs/sagemath_objects/src && tox -v -v -e py39)'

results in lots of doctest failures. This is expected. Right?

Then this command

pkgs/sagemath-objects/.tox/py39/bin/sage

should run the sage with only modules installed by sagemath-objects. But then still with this sage, I get

sage: GF(9)
Finite Field in z2 of size 3^2

Is this normal? What code would fail?

[mkoeppe]

Replying to klee:

This command

./bootstrap && ./sage -sh -c '(cd build/pkgs/sagemath_objects/src && tox -v -v -e py39)'

results in lots of doctest failures. This is expected. Right?

Yes.

[mkoeppe]

Replying to klee:

Then this command

pkgs/sagemath-objects/.tox/py39/bin/sage

should run the sage with only modules installed by sagemath-objects. But then still with this sage, I get

sage: GF(9)
Finite Field in z2 of size 3^2

Is this normal? What code would fail?

There was something wrong with which helper scripts are shipped as part of sagemath-objects. I am working on a fix

[klee]

After testing with the two experimental distributions, I could confirm that there are no other doctest failures with the main sage than it already had on develop branch. Hence the two experimental distributions are at least safe to have.

Anyway the failed doctests are

sage -t --warn-long 70.5 --random-seed=328015936148055540834358934757959169042 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 325, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    0
Got:
    4
**********************************************************************
File "src/sage/tests/cmdline.py", line 330, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    0
Got:
    2

[mkoeppe]

Replying to mkoeppe:

There was something wrong with which helper scripts are shipped as part of sagemath-objects. I am working on a fix

Opened #32933 as part of the fix

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e0399a1	pkgs/sagemath-objects: Install a fixed set of scripts, not all of src/bin
​7519048	src/bin/sage[-env]: Put SAGE_ROOT/src/bin in front of path only if run out of this directory
​00024e6	Merge #32933

[mkoeppe]

It now correctly fails:

$ pkgs/sagemath-objects/.tox/py39/bin/sage -c 'GF(9)'
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/pkgs/sagemath-objects/.tox/py39/bin/sage-eval", line 4, in <module>
    from sage.all import *
ModuleNotFoundError: No module named 'sage.all'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bfc953b

src/sage/doctest/reporting.py: Indicate --environment if not default

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​36d6b09

pkgs/sagemath-categories/setup.cfg.m4: Add scripts

[klee]

I guess this is normal

$ pkgs/sagemath-objects/.tox/py39/bin/sage
Traceback (most recent call last):
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-objects/.tox/py39/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
ModuleNotFoundError: No module named 'sage.misc.banner'

but this works well

$ pkgs/sagemath-objects/.tox/py39/bin/python3 
Python 3.9.7 (default, Oct 13 2021, 06:45:31) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.structure.sage_object import SageObject
>>> SageObject()
<sage.structure.sage_object.SageObject object at 0x10c977f60>
>>> a = SageObject()
>>> a.category()
Category of objects
>>> a
<sage.structure.sage_object.SageObject object at 0x10c977f50>
>>> a.dumps()
b'x\x9ck`J.NLO\xd5+.)*M.)-\x02\xb2\x80\xdc\xf8\xfc\xa4\xac\xd4\xe4\x12\xae` \xdb\x1f\xc2,d\xd0l,d\xd4\x03\x00\xb7P\x10\xef'
>>> a.parent()
<class 'sage.structure.sage_object.SageObject'>

[klee]

Likewise this works well

$ pkgs/sagemath-categories/.tox/py39/bin/python3
Python 3.9.7 (default, Oct 13 2021, 06:45:31) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.categories.category import Category
>>> from sage.categories.algebras import Algebras
>>> Algebras(QQ)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'QQ' is not defined
>>> Algebras(ZZ)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'ZZ' is not defined
>>> from sage.categories.groups import Groups
>>> Groups()
Category of groups
>>> c = Groups()
>>> c.example()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/categories/groups.py", line 48, in example
    from sage.rings.rational_field import QQ
ModuleNotFoundError: No module named 'sage.rings.rational_field'
>>> from sage.rings.infinity import infinity    
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sage.rings.infinity'
>>> from sage.rings.integer_ring import ZZ     
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sage.rings.integer_ring'
>>> from sage.categories.rings import Rings
>>> r = Rings()
>>> r
Category of rings
>>> r.example()
NotImplemented

It seems that there is not much to do with just categories with no constructible objects. This is of course expected.

[mkoeppe]

In #32432 I am adding a few more things to sagemath-categories so that some runtime testing becomes possible

[klee]

After I started to experiment with the branch of this ticket, I get

$ sage -tp --all
...
...
sage -t --warn-long 67.3 --random-seed=176797787270485350725074395342233660735 src/doc/ja/a_tour_of_sage/conf.py
    [0 tests, 0.00 s]
----------------------------------------------------------------------
sage -t --warn-long 67.3 --random-seed=176797787270485350725074395342233660735 src/sage/tests/cmdline.py  # 2 doctests failed
----------------------------------------------------------------------
Total time for all tests: 1819.6 seconds
    cpu time: 11313.2 seconds
    cumulative wall time: 13913.3 seconds
============================================================================ test session starts ============================================================================
platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/kwankyu/GitHub/sage-dev/src, configfile: tox.ini
collected 10 items / 1 error / 9 selected                                                                                                                                   

================================================================================== ERRORS ===================================================================================
____________________________________________________________ ERROR collecting sage/structure/sage_object_test.py ____________________________________________________________
ImportError while importing test module '/Users/kwankyu/GitHub/sage-dev/src/sage/structure/sage_object_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
src/sage/structure/sage_object_test.py:3: in <module>
    from .sage_object import SageObject
E   ImportError: attempted relative import with no known parent package
========================================================================== short test summary info ==========================================================================
ERROR src/sage/structure/sage_object_test.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================= 1 error in 5.75s ==============================================================================

I have the doctest failures of src/sage/tests/cmdline.py on develop branch. So they seem not related with the present branch.

Then what are these messages starting with ===.... test session starts ...===, which shows some failure?

[mkoeppe]

Replying to klee:

Then what are these messages starting with ===.... test session starts ...===, which shows some failure?

At the end of sage -t, pytest is run; this is from #31003 and follow-up tickets. Tickets #32892 and #31924 are for fixing some of these unsightly messages

[klee]

Would you explain this line from tox.ini?

[testenv]
deps = -rrequirements.txt 

What does -r do?

[mkoeppe]

It instructs pip to install the packages listed in the file requirements.txt

[klee]

I tested with the branch of this ticket, perhaps from a user point of view. I finds it works well, as advertised in the description. So I am positive with the branch.

But I am not capable of reading the code, most of which looks to me cryptic, to give any useful comments. So I leave this task to other reviewer well versed with sage build system and python build tools.

However, if no one gives more comments, then I will set positive review once we have a green bot, as this ticket needs to be closed before sage 9.5 release.

[mkoeppe]

Thank you!

[slelievre]

Files in build/pkgs/sagemath_categories, build/pkgs/sagemath_objects, pkgs/sage-conf_pypi, pkgs/sagemath-categories, pkgs/sagemath-objects, could ideally end with a newline, even when they only have one line.

[mkoeppe]

The files for which the diff shows this:

diff --git a/build/pkgs/sagemath_categories/SPKG.rst b/build/pkgs/sagemath_categories/SPKG.rst
new file mode 120000
index 00000000..b4545b4
--- /dev/null
+++ b/build/pkgs/sagemath_categories/SPKG.rst
@@ -0,0 +1 @@
+src/README.rst
\ No newline at end of file

are actually symbolic links, and adding a newline is not an option.

[slelievre]

Never mind then! Sorry for the noise.

[klee]

Okay. Let's get this in.

[mkoeppe]

Thanks!

[klee]

Why this is not yet merged?