Distributions sagemath-{flint-arb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules

[mkoeppe]

Basically, we are making all external libraries optional, except for a minimal set of "core" libraries such as cysignals and gmpy2 (see #29865).

(from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sagemath-ntl) - but some libraries come in packs that would not make much sense to separate (such as sagemath-flint-arb-e_antic).

List of distributions:

• sagemath-flint-arb
  • 'sage/libs/arb/arb_version.pyx',
  • 'sage/libs/arb/arith.pyx',
  • 'sage/matrix/matrix_complex_ball_dense.pyx',
  • 'sage/rings/complex_arb.pyx',
  • 'sage/rings/number_field/number_field_element_quadratic.pyx',
  • 'sage/rings/polynomial/polynomial_complex_arb.pyx',
  • 'sage/rings/real_arb.pyx'
  • etc.
• sagemath-homfly: 'sage/libs/homfly.pyx'
• sagemath-giac (ex giacpy_sage in sage.libs.giac + dependency on libgiac)
  • sage/libs/giac/giac.pyx
• sagemath-gap:
  • 'sage/coding/codecan/codecan.pyx',
  • 'sage/combinat/enumeration_mod_permgroup.pyx',
  • 'sage/combinat/root_system/reflection_group_c.pyx',
  • 'sage/combinat/root_system/reflection_group_element.pyx',
  • 'sage/graphs/spanning_tree.pyx',
  • 'sage/groups/libgap_wrapper.pyx',
  • 'sage/groups/matrix_gps/group_element.pyx',
  • 'sage/groups/perm_gps/permgroup_element.pyx',
  • 'sage/groups/perm_gps/partn_ref/automorphism_group_canonical_label.pyx',
  • 'sage/groups/perm_gps/partn_ref/canonical_augmentation.pyx',
  • 'sage/groups/perm_gps/partn_ref/data_structures.pyx',
  • 'sage/groups/perm_gps/partn_ref/double_coset.pyx',
  • 'sage/groups/perm_gps/partn_ref/refinement_binary.pyx',
  • 'sage/groups/perm_gps/partn_ref/refinement_graphs.pyx',
  • 'sage/groups/perm_gps/partn_ref/refinement_lists.pyx',
  • 'sage/groups/perm_gps/partn_ref/refinement_matrices.pyx',
  • 'sage/groups/perm_gps/partn_ref/refinement_python.pyx',
  • 'sage/groups/perm_gps/partn_ref/refinement_sets.pyx',
  • 'sage/groups/perm_gps/partn_ref2/refinement_generic.pyx',
  • 'sage/libs/gap/element.pyx',
  • 'sage/libs/gap/libgap.pyx',
  • 'sage/libs/gap/util.pyx',
  • 'sage/matrix/matrix_gap.pyx',
  • 'sage/sets/disjoint_set.pyx'

To test:

• make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_flint_arb
• make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_gap
• make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_giac
• make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_homfly

In follow-up tickets:

• sagemath-brial (ex sage-brial in sage.rings.polynomial.pbori + sage.libs.polybori, sage.crypto.boolean_function)
• sagemath-ntl
• sagemath-ecl (sage.libs.ecl)
• sagemath-maxima: requires sage-ecl
• sagemath-singular: (some of these files actually may be better put in the pynac distribution)
  • 'sage/algebras/letterplace/free_algebra_element_letterplace.pyx',
  • 'sage/algebras/letterplace/free_algebra_letterplace.pyx',
  • 'sage/algebras/letterplace/letterplace_ideal.pyx',
  • 'sage/libs/pynac/constant.pyx',
  • 'sage/libs/pynac/pynac.pyx',
  • 'sage/libs/singular/function.pyx',
  • 'sage/libs/singular/groebner_strategy.pyx',
  • 'sage/libs/singular/option.pyx',
  • 'sage/libs/singular/polynomial.pyx',
  • 'sage/libs/singular/ring.pyx',
  • 'sage/libs/singular/singular.pyx',
  • 'sage/matrix/matrix_mpolynomial_dense.pyx',
  • 'sage/rings/polynomial/multi_polynomial_ideal_libsingular.pyx',
  • 'sage/rings/polynomial/multi_polynomial_libsingular.pyx',
  • 'sage/rings/polynomial/plural.pyx',
  • 'sage/symbolic/comparison.pyx',
  • 'sage/symbolic/constants_c.pyx',
  • 'sage/symbolic/expression.pyx',
  • 'sage/symbolic/function.pyx',
  • 'sage/symbolic/getitem.pyx',
  • 'sage/symbolic/ring.pyx',
  • 'sage/symbolic/series.pyx',
  • 'sage/symbolic/substitution_map.pyx',
• sagemath-pynac
• sagemath-linbox:
  • 'sage/libs/linbox/linbox_flint_interface.pyx',
  • 'sage/matrix/matrix_integer_sparse.pyx',
  • 'sage/matrix/matrix_modn_dense_double.pyx',
  • 'sage/matrix/matrix_modn_dense_float.pyx',
  • 'sage/matrix/matrix_modn_sparse.pyx'
• sagemath-znpoly (subject to removal in #32841?)
  • 'sage/modular/modsym/p1list.pyx',
  • 'sage/modular/pollack_stevens/dist.pyx',
  • 'sage/rings/fraction_field_FpT.pyx',
  • 'sage/rings/polynomial/polynomial_zmod_flint.pyx',
  • 'sage/schemes/hyperelliptic_curves/hypellfrob.pyx',
• sagemath-mari (also requiring pbori and pbori-groebner):
  • 'sage/matrix/matrix_gf2e_dense.pyx',
  • 'sage/matrix/matrix_integer_dense.pyx',
  • 'sage/matrix/matrix_mod2_dense.pyx',
  • 'sage/matrix/matrix_rational_dense.pyx',
  • 'sage/modules/vector_mod2_dense.pyx',
  • 'sage/rings/polynomial/pbori/pbori.pyx',
  • 'sage/rings/polynomial/polynomial_gf2x.pyx'

This is preparation for Meta-ticket #29705 (Modularization) and #30371.

[gh-tobiasdiez]

Good idea to separate this into a new ticket.

While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.

Also, I think, there should be distributions for GAP, CCObject, linbox, ratpoints, zn_poly and mari.

What do you think about renaming sage_setup: distribution to sage_setup: dependency? I think this better reflects the purpose. The distribution is then generated based on which dependencies are available. Another idea would be to use distutils: libraries which already contains some of the dependencies anyway (e.g. arb, gmp, flint).

[mkoeppe]

Replying to gh-tobiasdiez:

While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.

I think they should be separate because I really want to be able to make singular optional. pynac, on the other hand, is essential for our symbolics implementation.

Also, I think, there should be distributions for GAP, CCObject, linbox, ratpoints, zn_poly and mari.

Yes, please add to the list in the ticket description with explanation (what is CCObject?)

What do you think about renaming sage_setup: distribution to sage_setup: dependency?

No, I wouldn't like that. As soon as you start with declaring "dependencies", there will be combinatorial explosion from the combinations of such dependencies. I really want a well-designed partition of the source files into (relatively few) disjoint distributions. This is crucial for the modularization -- each distribution will eventually be separately compilable and installable package on PyPI.

[gh-tobiasdiez]

Ok, makes sense. Which distribution should be assigned to files that have two dependencies (e.g use pynac and singular)?

My idea was that the cython file declares which dependencies it has, and in the setup.py file you have a list of distributions and their dependencies. If all dependencies are met, that particular distribution is built. E.g. distribution "sage-singular" relies on "pynac" and "singular". (That's only a feeling, but it feels strange to have distributions determined by dependencies and not by a theme or subject, e.g. "sage-singular" vs "sage-symbolic calculation". But you have a better overview, so I trust you there.)

I've added the above distributions to the list. (Forget about CCObject, it was a missing import in #30371 that confused me.)

[mkoeppe]

Replying to gh-tobiasdiez:

Which distribution should be assigned to files that have two dependencies (e.g use pynac and singular)?

Distributions can certainly depend on other distributions. sage-singular could depend on sage-pynac. Let's include this in the ticket description. (In the modularized build, sage-singular's setup.py or setup.cfg would have install_requires = sage-pynac.)

Also, often it is relatively easy to remove some of these dependencies by small changes to a few source files, moving a few methods around. See for example #29911. We can take work on this in follow-up tickets.

it feels strange to have distributions determined by dependencies and not by a theme or subject, e.g. "sage-singular" vs "sage-symbolic calculation".

Perhaps, but there is actually an important "political" (open source community) reason for this design too. By naming these components after major libraries that provide the features, we make these libraries and their important role for Sage more visible to end users. Attribution tends to be an important motivating factor to developers of these libraries.

[mkoeppe]

To expand on this a little bit: I wouldn't rule out distributions named after topics. The present ticket only provides a "skeleton" of distributions that are (1) technically needed to take care of compile-time dependencies, (2) provides attribution. Later on (in follow-up tickets), for example, there could be a distribution sage-feature-padics or something like this that has sage-ntl as a dependency and packages sage.rings.padics.

[mkoeppe]

Replying to mkoeppe:

Replying to gh-tobiasdiez:

While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.

I think they should be separate because I really want to be able to make singular optional. pynac, on the other hand, is essential for our symbolics implementation.

When I wrote this, I didn't recall that singular's libfactory is actually a dependency of pynac. So I think you are right, it's best to have pynac and singular in one distribution (for now, at least).

[mkoeppe]

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

New commits:

​3f3bfef

Mark files # sage_setup: distribution = sagemath-flint-arb

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8c6728d	Mark files # sage_setup: distribution = sagemath-homfly
​2654291	Mark files # sage_setup: distribution = sagemath-giac
​1ab0438	Mark files # sage_setup: distribution = sagemath-gap

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3181bb0	Do not mark *.h files with # sage-setup: distribution
​52b208b	pkgs/sagemath-gap, build/pkgs/sagemath_gap: New

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​e551f34	pkgs/sagemath-gap, build/pkgs/sagemath_gap: New
​6c41717	pkgs/sagemath-homfly, build/pkgs/sagemath_homfly: New

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cad9819	build/pkgs/sagemath_{gap,homfly}/spkg-install: Symlink to sagemath_objects spkg-install
​cd8358c	pkgs/sagemath-gap/MANIFEST.in: Add VERSION.txt

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​619c8c8	Mark files # sage_setup: distribution = sagemath-flint-arb
​71b5921	Mark files # sage_setup: distribution = sagemath-homfly
​fe08ab2	Mark files # sage_setup: distribution = sagemath-giac
​640aa74	Mark files # sage_setup: distribution = sagemath-gap
​d1ef655	Do not mark *.h files with # sage-setup: distribution
​8c46aab	pkgs/sagemath-gap, build/pkgs/sagemath_gap: New
​6db9b01	pkgs/sagemath-homfly, build/pkgs/sagemath_homfly: New
​390ebb8	build/pkgs/sagemath_{gap,homfly}/spkg-install: Symlink to sagemath_objects spkg-install
​9f26fb8	pkgs/sagemath-gap/MANIFEST.in: Add VERSION.txt

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​13f030c	src/sage/libs/gap: Add # sage_setup: distribution to more files
​a73b489	Add src/sage/**/all__sagemath_gap.py
​30112b2	pkgs/sagemath-gap: WIP setup.py
​cae814a	src/sage/groups/perm_gps/partn_ref/data_structures.pyx: Remove cimport from sage.libs.flint (only used by doctesting helper functions)
​5fda124	Add src/sage/**/all__sagemath_gap.py
​a2feea0	Add src/sage/**/all__sagemath_gap.py
​665d01c	pkgs/sagemath-gap/tox.ini: New
​963fa77	pkgs/sagemath-gap/MANIFEST.in: Add deps for sage/coding/codecan

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​96b863c

pkgs/sagemath-gap: Update description

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b73a929

build/pkgs/sagemath_gap/dependencies: Update

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fc91135	pkgs/sagemath-homfly: Fix up
​0869a7f	pkgs/sagemath-environment: Add sage.misc.package_dir properly

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​31b08a1	.gitignore: Ignore generated files
​961fb3f	pkgs/sagemath-flint-arb: New

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​45cce7b	pkgs/sagemath-giac: New
​47a2519	Fixup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​513fc3f

Add tox.ini

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0f198ca

Makefile: Add to PYPI_WHEEL_PACKAGES

[dimpase]

How are all these tags like # sage_setup: distribution = sagemath-gap processed?

[mkoeppe]

They are processed by the functions in sage_setup.find, via sage.misc.package_dir.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​133e5b8

Remove copy-pasted conda.txt files

[mkoeppe]

sage -t --random-seed=146157697718740975867546676674797881386 sage/groups/perm_gps/partn_ref/data_structures.pyx
**********************************************************************
File "sage/groups/perm_gps/partn_ref/data_structures.pyx", line 1221, in sage.groups.perm_gps.partn_ref.data_structures.SC_test_list_perms
Failed example:
    for i in range(2,9):
        test_Sn_on_m_points(i,i,1,0)
Expected nothing
Got:
    ImportError: cannot import name n_is_prime
    Exception ignored in: 'sage.groups.perm_gps.partn_ref.data_structures.SC_is_giant'
    Traceback (most recent call last):
      File "<doctest sage.groups.perm_gps.partn_ref.data_structures.SC_test_list_perms[2]>", line 4, in test_Sn_on_m_points

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4c8f5b5

src/sage/sets/disjoint_set.pyx: Remove obsolete -*- coding -*- line

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2be77af

pkgs/sagemath-flint-arb: Update description

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a827f29

pkgs/sagemath-giac: Update description

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​112e29e

src/sage/groups/perm_gps/partn_ref/data_structures.pyx: In doctest support code, avoid use of FLINT

[dimpase]

there are a number of files with m4 suffix, which does not seem to be m4 files. Should they rather get in suffux?

[mkoeppe]

They are processed by m4, see build/pkgs/sagelib/bootstrap

[dimpase]

Imho they should have .in suffix. Otherwise, looks OK.

[mkoeppe]

The .in files in use with autotools are not processed by m4.

These .m4 files are processed by m4. For example, in setup.cfg.m4 you'll see the m4 macro call esyscmd.

[dimpase]

ah, OK. I misunderstood what "processed" means in this context.

[mkoeppe]

Thanks!

[fbissey]

When I build the full sagelib with this ticket included (but I do not build those sub distribution yet) in sage-on-gentoo, building the documentation fails. After inspection, I have the sources installed for sage/libs/flint, sage/libs/arb etc, but none of compiled module are installed. Is there something I am missing or doing wrong?

[mkoeppe]

Are you building from src/ or from pkgs/sagemath-standard/?

[fbissey]

src with the setup.py from sagemath-standard. This is when I build against develop or the Volker merging branch. The main reason I am starting from src in those cases, is that patch does not like links.

[fbissey]

I also delete src/sage_setup so it is not used instead of the installed one.

[mkoeppe]

OK, that's something that I'll have to fix in pkgs/sagemath-standard/setup.py. Thanks for catching this

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​790c12b	Merge tag '9.8.beta5' into t/30666/add__sage_setup__distribution__directives_to_all_cython_modules_needing_external_libraries
​8b1cf25	pkgs/sagemath-standard/setup.py: Include files marked as # sage_setup: distribution = sagemath-flint-arb etc.

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​2b39b3d	Merge #34858
​c59622a	build/pkgs/sagemath*/spkg-src: Use build
​7fc93d0	build/pkgs/sagemath*/spkg-src: Use --skip-dependency-check
​4940db4	build/make/Makefile.in: Add targets SPKG-src for script packages
​573fcfd	build/make/Makefile.in: Move targets pypi-sdists, wheel, pypi-wheels here; use SPKG-sdist targets
​88e18d0	Merge #34855
​0bd13ac	build/make/Makefile.in: Remove duplicate setting of PYPI_WHEEL_PACKAGES
​f712d67	Merge #34855
​5fc1add	build/pkgs/sagemath*/spkg-src: Use build
​1531fda	build/pkgs/sagemath_flint_arb/dependencies: Fix up