Opened 2 years ago
Last modified 4 weeks ago
#30666 needs_work enhancement
Distributions sagemath{flintarb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage9.9 
Component:  refactoring  Keywords:  sd111 
Cc:  ghtobiasdiez, dimpase  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  Dima Pasechnik 
Report Upstream:  N/A  Work issues:  
Branch:  u/mkoeppe/add__sage_setup__distribution__directives_to_all_cython_modules_needing_external_libraries (Commits, GitHub, GitLab)  Commit:  1531fda69d3c9179f72c0c71cb702f65630b8afc 
Dependencies:  #31031, #34855, #34839  Stopgaps: 
Description (last modified by )
Basically, we are making all external libraries optional, except for a minimal set of "core" libraries such as cysignals
and gmpy2
(see #29865).
(from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sagemathntl
)  but some libraries come in packs that would not make much sense to separate (such as sagemathflintarbe_antic
).
List of distributions:
 sagemathflintarb
 'sage/libs/arb/arb_version.pyx',
 'sage/libs/arb/arith.pyx',
 'sage/matrix/matrix_complex_ball_dense.pyx',
 'sage/rings/complex_arb.pyx',
 'sage/rings/number_field/number_field_element_quadratic.pyx',
 'sage/rings/polynomial/polynomial_complex_arb.pyx',
 'sage/rings/real_arb.pyx'
 etc.
 sagemathhomfly: 'sage/libs/homfly.pyx'
 sagemathgiac (ex
giacpy_sage
insage.libs.giac
+ dependency onlibgiac
) sage/libs/giac/giac.pyx
 sagemathgap:
 'sage/coding/codecan/codecan.pyx',
 'sage/combinat/enumeration_mod_permgroup.pyx',
 'sage/combinat/root_system/reflection_group_c.pyx',
 'sage/combinat/root_system/reflection_group_element.pyx',
 'sage/graphs/spanning_tree.pyx',
 'sage/groups/libgap_wrapper.pyx',
 'sage/groups/matrix_gps/group_element.pyx',
 'sage/groups/perm_gps/permgroup_element.pyx',
 'sage/groups/perm_gps/partn_ref/automorphism_group_canonical_label.pyx',
 'sage/groups/perm_gps/partn_ref/canonical_augmentation.pyx',
 'sage/groups/perm_gps/partn_ref/data_structures.pyx',
 'sage/groups/perm_gps/partn_ref/double_coset.pyx',
 'sage/groups/perm_gps/partn_ref/refinement_binary.pyx',
 'sage/groups/perm_gps/partn_ref/refinement_graphs.pyx',
 'sage/groups/perm_gps/partn_ref/refinement_lists.pyx',
 'sage/groups/perm_gps/partn_ref/refinement_matrices.pyx',
 'sage/groups/perm_gps/partn_ref/refinement_python.pyx',
 'sage/groups/perm_gps/partn_ref/refinement_sets.pyx',
 'sage/groups/perm_gps/partn_ref2/refinement_generic.pyx',
 'sage/libs/gap/element.pyx',
 'sage/libs/gap/libgap.pyx',
 'sage/libs/gap/util.pyx',
 'sage/matrix/matrix_gap.pyx',
 'sage/sets/disjoint_set.pyx'
To test:
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_flint_arb
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_gap
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_giac
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_homfly
In followup tickets:
sagemathbrial
(exsagebrial
insage.rings.polynomial.pbori
+sage.libs.polybori
,sage.crypto.boolean_function
)sagemathntl
sagemathecl
(sage.libs.ecl
)sagemathmaxima
: requires sageeclsagemathsingular
: (some of these files actually may be better put in the pynac distribution) 'sage/algebras/letterplace/free_algebra_element_letterplace.pyx',
 'sage/algebras/letterplace/free_algebra_letterplace.pyx',
 'sage/algebras/letterplace/letterplace_ideal.pyx',
 'sage/libs/pynac/constant.pyx',
 'sage/libs/pynac/pynac.pyx',
 'sage/libs/singular/function.pyx',
 'sage/libs/singular/groebner_strategy.pyx',
 'sage/libs/singular/option.pyx',
 'sage/libs/singular/polynomial.pyx',
 'sage/libs/singular/ring.pyx',
 'sage/libs/singular/singular.pyx',
 'sage/matrix/matrix_mpolynomial_dense.pyx',
 'sage/rings/polynomial/multi_polynomial_ideal_libsingular.pyx',
 'sage/rings/polynomial/multi_polynomial_libsingular.pyx',
 'sage/rings/polynomial/plural.pyx',
 'sage/symbolic/comparison.pyx',
 'sage/symbolic/constants_c.pyx',
 'sage/symbolic/expression.pyx',
 'sage/symbolic/function.pyx',
 'sage/symbolic/getitem.pyx',
 'sage/symbolic/ring.pyx',
 'sage/symbolic/series.pyx',
 'sage/symbolic/substitution_map.pyx',
sagemathpynac
sagemathlinbox
: 'sage/libs/linbox/linbox_flint_interface.pyx',
 'sage/matrix/matrix_integer_sparse.pyx',
 'sage/matrix/matrix_modn_dense_double.pyx',
 'sage/matrix/matrix_modn_dense_float.pyx',
 'sage/matrix/matrix_modn_sparse.pyx'
sagemathznpoly
(subject to removal in #32841?) 'sage/modular/modsym/p1list.pyx',
 'sage/modular/pollack_stevens/dist.pyx',
 'sage/rings/fraction_field_FpT.pyx',
 'sage/rings/polynomial/polynomial_zmod_flint.pyx',
 'sage/schemes/hyperelliptic_curves/hypellfrob.pyx',
sagemathmari
(also requiring pbori and pborigroebner): 'sage/matrix/matrix_gf2e_dense.pyx',
 'sage/matrix/matrix_integer_dense.pyx',
 'sage/matrix/matrix_mod2_dense.pyx',
 'sage/matrix/matrix_rational_dense.pyx',
 'sage/modules/vector_mod2_dense.pyx',
 'sage/rings/polynomial/pbori/pbori.pyx',
 'sage/rings/polynomial/polynomial_gf2x.pyx'
This is preparation for Metaticket #29705 (Modularization) and #30371.
Change History (69)
comment:1 Changed 2 years ago by
Description:  modified (diff) 

comment:2 Changed 2 years ago by
Description:  modified (diff) 

comment:3 Changed 2 years ago by
Description:  modified (diff) 

comment:4 Changed 2 years ago by
Description:  modified (diff) 

comment:5 followup: 6 Changed 2 years ago by
comment:6 followup: 13 Changed 2 years ago by
Replying to ghtobiasdiez:
While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.
I think they should be separate because I really want to be able to make singular optional. pynac, on the other hand, is essential for our symbolics implementation.
Also, I think, there should be distributions for GAP, CCObject, linbox, ratpoints, zn_poly and mari.
Yes, please add to the list in the ticket description with explanation (what is CCObject?)
What do you think about renaming sage_setup: distribution to sage_setup: dependency?
No, I wouldn't like that. As soon as you start with declaring "dependencies", there will be combinatorial explosion from the combinations of such dependencies. I really want a welldesigned partition of the source files into (relatively few) disjoint distributions. This is crucial for the modularization  each distribution will eventually be separately compilable and installable package on PyPI.
comment:7 Changed 2 years ago by
Description:  modified (diff) 

comment:8 followup: 9 Changed 2 years ago by
Ok, makes sense. Which distribution should be assigned to files that have two dependencies (e.g use pynac and singular)?
My idea was that the cython file declares which dependencies it has, and in the setup.py file you have a list of distributions and their dependencies. If all dependencies are met, that particular distribution is built. E.g. distribution "sagesingular" relies on "pynac" and "singular". (That's only a feeling, but it feels strange to have distributions determined by dependencies and not by a theme or subject, e.g. "sagesingular" vs "sagesymbolic calculation". But you have a better overview, so I trust you there.)
I've added the above distributions to the list. (Forget about CCObject, it was a missing import in #30371 that confused me.)
comment:9 Changed 2 years ago by
Replying to ghtobiasdiez:
Which distribution should be assigned to files that have two dependencies (e.g use pynac and singular)?
Distributions can certainly depend on other distributions. sagesingular
could depend on sagepynac
. Let's include this in the ticket description. (In the modularized build, sagesingular
's setup.py
or setup.cfg
would have install_requires = sagepynac
.)
Also, often it is relatively easy to remove some of these dependencies by small changes to a few source files, moving a few methods around. See for example #29911. We can take work on this in followup tickets.
it feels strange to have distributions determined by dependencies and not by a theme or subject, e.g. "sagesingular" vs "sagesymbolic calculation".
Perhaps, but there is actually an important "political" (open source community) reason for this design too. By naming these components after major libraries that provide the features, we make these libraries and their important role for Sage more visible to end users. Attribution tends to be an important motivating factor to developers of these libraries.
comment:10 Changed 2 years ago by
To expand on this a little bit: I wouldn't rule out distributions named after topics. The present ticket only provides a "skeleton" of distributions that are (1) technically needed to take care of compiletime dependencies, (2) provides attribution. Later on (in followup tickets), for example, there could be a distribution sagefeaturepadics
or something like this that has sagentl
as a dependency and packages sage.rings.padics
.
comment:11 Changed 2 years ago by
Description:  modified (diff) 

comment:12 Changed 2 years ago by
Description:  modified (diff) 

comment:13 Changed 2 years ago by
Replying to mkoeppe:
Replying to ghtobiasdiez:
While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.
I think they should be separate because I really want to be able to make singular optional. pynac, on the other hand, is essential for our symbolics implementation.
When I wrote this, I didn't recall that singular's libfactory
is actually a dependency of pynac. So I think you are right, it's best to have pynac and singular in one distribution (for now, at least).
comment:14 Changed 2 years ago by
Keywords:  sd111 added 

comment:15 Changed 2 years ago by
Description:  modified (diff) 

comment:16 Changed 2 years ago by
Dependencies:  → #31031 

comment:17 Changed 2 years ago by
Milestone:  sage9.3 → sage9.4 

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
comment:18 Changed 19 months ago by
Milestone:  sage9.4 → sage9.5 

comment:19 Changed 14 months ago by
Milestone:  sage9.5 → sage9.6 

comment:20 Changed 10 months ago by
Milestone:  sage9.6 → sage9.7 

comment:21 Changed 5 months ago by
Milestone:  sage9.7 → sage9.8 

comment:22 Changed 3 months ago by
Branch:  → u/mkoeppe/add__sage_setup__distribution__directives_to_all_cython_modules_needing_external_libraries 

comment:23 Changed 3 months ago by
Commit:  → 3f3bfefbac2e7fc96950a6ec9459783a561c9bce 

Description:  modified (diff) 
New commits:
3f3bfef  Mark files # sage_setup: distribution = sagemathflintarb

comment:24 Changed 3 months ago by
Description:  modified (diff) 

comment:25 Changed 3 months ago by
Commit:  3f3bfefbac2e7fc96950a6ec9459783a561c9bce → 1ab0438283b4ee4f0c5dfdb8d08a41ad739ab21b 

comment:26 Changed 3 months ago by
Commit:  1ab0438283b4ee4f0c5dfdb8d08a41ad739ab21b → 52b208bc7d280d937c341f66a673650c6ac916c8 

comment:27 Changed 3 months ago by
Description:  modified (diff) 

comment:28 Changed 3 months ago by
Commit:  52b208bc7d280d937c341f66a673650c6ac916c8 → 6c41717002434b27c7b3effe42334704175461cb 

comment:29 Changed 3 months ago by
Commit:  6c41717002434b27c7b3effe42334704175461cb → cd8358caa5e00f249a716030131e39628827cbad 

comment:30 Changed 2 months ago by
Cc:  dimpase added 

Summary:  Add "sage_setup: distribution" directives to all Cython modules needing external libraries → Distributions sagemath{flintarb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules 
comment:31 Changed 2 months ago by
Commit:  cd8358caa5e00f249a716030131e39628827cbad → 9f26fb8c102e05033b60679798afba7ad031e796 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
619c8c8  Mark files # sage_setup: distribution = sagemathflintarb

71b5921  Mark files # sage_setup: distribution = sagemathhomfly

fe08ab2  Mark files # sage_setup: distribution = sagemathgiac

640aa74  Mark files # sage_setup: distribution = sagemathgap

d1ef655  Do not mark *.h files with # sagesetup: distribution

8c46aab  pkgs/sagemathgap, build/pkgs/sagemath_gap: New

6db9b01  pkgs/sagemathhomfly, build/pkgs/sagemath_homfly: New

390ebb8  build/pkgs/sagemath_{gap,homfly}/spkginstall: Symlink to sagemath_objects spkginstall

9f26fb8  pkgs/sagemathgap/MANIFEST.in: Add VERSION.txt

comment:32 Changed 2 months ago by
Commit:  9f26fb8c102e05033b60679798afba7ad031e796 → 963fa7704055f99af12c80654f904ee8ffd21c22 

Branch pushed to git repo; I updated commit sha1. New commits:
13f030c  src/sage/libs/gap: Add # sage_setup: distribution to more files

a73b489  Add src/sage/**/all__sagemath_gap.py

30112b2  pkgs/sagemathgap: WIP setup.py

cae814a  src/sage/groups/perm_gps/partn_ref/data_structures.pyx: Remove cimport from sage.libs.flint (only used by doctesting helper functions)

5fda124  Add src/sage/**/all__sagemath_gap.py

a2feea0  Add src/sage/**/all__sagemath_gap.py

665d01c  pkgs/sagemathgap/tox.ini: New

963fa77  pkgs/sagemathgap/MANIFEST.in: Add deps for sage/coding/codecan

comment:33 Changed 2 months ago by
Description:  modified (diff) 

comment:34 Changed 2 months ago by
Commit:  963fa7704055f99af12c80654f904ee8ffd21c22 → 96b863ce7d3ce45460e81afcac5d7448c7ee720b 

Branch pushed to git repo; I updated commit sha1. New commits:
96b863c  pkgs/sagemathgap: Update description

comment:35 Changed 2 months ago by
Commit:  96b863ce7d3ce45460e81afcac5d7448c7ee720b → b73a92956789a2548063ce867556b2d81d49f2dd 

Branch pushed to git repo; I updated commit sha1. New commits:
b73a929  build/pkgs/sagemath_gap/dependencies: Update

comment:36 Changed 2 months ago by
Commit:  b73a92956789a2548063ce867556b2d81d49f2dd → 0869a7f471b569af4f73506d4ab27d5c09654e75 

comment:37 Changed 2 months ago by
Commit:  0869a7f471b569af4f73506d4ab27d5c09654e75 → 961fb3f744e09868a59ca87d7ae5b590d886e338 

comment:38 Changed 2 months ago by
Commit:  961fb3f744e09868a59ca87d7ae5b590d886e338 → 47a2519acbf15b502a619288da2f62d2c208c018 

comment:39 Changed 2 months ago by
Commit:  47a2519acbf15b502a619288da2f62d2c208c018 → 513fc3f3e0ae775cb5f92f6fc81cb9d53484851f 

Branch pushed to git repo; I updated commit sha1. New commits:
513fc3f  Add tox.ini

comment:40 Changed 2 months ago by
Commit:  513fc3f3e0ae775cb5f92f6fc81cb9d53484851f → 0f198cafdc72abcab02994e66cdbd9c917b31bc3 

Branch pushed to git repo; I updated commit sha1. New commits:
0f198ca  Makefile: Add to PYPI_WHEEL_PACKAGES

comment:41 Changed 2 months ago by
Authors:  → Matthias Koeppe 

Status:  new → needs_review 
comment:42 Changed 2 months ago by
How are all these tags like # sage_setup: distribution = sagemathgap
processed?
comment:43 Changed 2 months ago by
They are processed by the functions in sage_setup.find
, via sage.misc.package_dir
.
comment:44 Changed 2 months ago by
Commit:  0f198cafdc72abcab02994e66cdbd9c917b31bc3 → 133e5b811eac34ef3b14a1c18596cb679d3002ab 

Branch pushed to git repo; I updated commit sha1. New commits:
133e5b8  Remove copypasted conda.txt files

comment:45 Changed 2 months ago by
sage t randomseed=146157697718740975867546676674797881386 sage/groups/perm_gps/partn_ref/data_structures.pyx ********************************************************************** File "sage/groups/perm_gps/partn_ref/data_structures.pyx", line 1221, in sage.groups.perm_gps.partn_ref.data_structures.SC_test_list_perms Failed example: for i in range(2,9): test_Sn_on_m_points(i,i,1,0) Expected nothing Got: ImportError: cannot import name n_is_prime Exception ignored in: 'sage.groups.perm_gps.partn_ref.data_structures.SC_is_giant' Traceback (most recent call last): File "<doctest sage.groups.perm_gps.partn_ref.data_structures.SC_test_list_perms[2]>", line 4, in test_Sn_on_m_points
comment:46 Changed 2 months ago by
Status:  needs_review → needs_work 

comment:47 Changed 2 months ago by
Commit:  133e5b811eac34ef3b14a1c18596cb679d3002ab → 4c8f5b5db2450e99e5968d4dedc88bb0e89127ca 

Branch pushed to git repo; I updated commit sha1. New commits:
4c8f5b5  src/sage/sets/disjoint_set.pyx: Remove obsolete * coding * line

comment:48 Changed 2 months ago by
Commit:  4c8f5b5db2450e99e5968d4dedc88bb0e89127ca → 2be77afa09adf9e4f079569d1aa9aeb501de1956 

Branch pushed to git repo; I updated commit sha1. New commits:
2be77af  pkgs/sagemathflintarb: Update description

comment:49 Changed 2 months ago by
Commit:  2be77afa09adf9e4f079569d1aa9aeb501de1956 → a827f299c54650e6a71bdb6bede15b8536f06eb0 

Branch pushed to git repo; I updated commit sha1. New commits:
a827f29  pkgs/sagemathgiac: Update description

comment:50 Changed 2 months ago by
Commit:  a827f299c54650e6a71bdb6bede15b8536f06eb0 → 112e29ec892ab6901b83ce35d0b476f3af008df3 

Branch pushed to git repo; I updated commit sha1. New commits:
112e29e  src/sage/groups/perm_gps/partn_ref/data_structures.pyx: In doctest support code, avoid use of FLINT

comment:51 Changed 2 months ago by
Status:  needs_work → needs_review 

comment:52 Changed 2 months ago by
Description:  modified (diff) 

comment:53 Changed 2 months ago by
there are a number of files with m4
suffix, which does not seem to be m4
files.
Should they rather get in
suffux?
comment:56 Changed 2 months ago by
The .in files in use with autotools are not processed by m4.
These .m4 files are processed by m4. For example, in setup.cfg.m4 you'll see the m4 macro call esyscmd
.
comment:57 Changed 2 months ago by
Status:  needs_review → positive_review 

ah, OK. I misunderstood what "processed" means in this context.
comment:59 Changed 2 months ago by
Reviewers:  → Dima Pasechnik 

comment:60 Changed 8 weeks ago by
When I build the full sagelib with this ticket included (but I do not build those sub distribution yet) in sageongentoo, building the documentation fails. After inspection, I have the sources installed for sage/libs/flint
, sage/libs/arb
etc, but none of compiled module are installed. Is there something I am missing or doing wrong?
comment:62 Changed 8 weeks ago by
src
with the setup.py
from sagemathstandard
. This is when I build against develop or the Volker merging branch. The main reason I am starting from src
in those cases, is that patch does not like links.
comment:63 Changed 8 weeks ago by
I also delete src/sage_setup
so it is not used instead of the installed one.
comment:64 Changed 8 weeks ago by
Status:  positive_review → needs_work 

OK, that's something that I'll have to fix in pkgs/sagemathstandard/setup.py
.
Thanks for catching this
comment:65 Changed 8 weeks ago by
Commit:  112e29ec892ab6901b83ce35d0b476f3af008df3 → 8b1cf25446766c8befe26ca2a8ccb0890926fdb7 

Branch pushed to git repo; I updated commit sha1. New commits:
790c12b  Merge tag '9.8.beta5' into t/30666/add__sage_setup__distribution__directives_to_all_cython_modules_needing_external_libraries

8b1cf25  pkgs/sagemathstandard/setup.py: Include files marked as # sage_setup: distribution = sagemathflintarb etc.

comment:66 Changed 7 weeks ago by
Dependencies:  #31031 → #31031, #34855 

comment:67 Changed 7 weeks ago by
Commit:  8b1cf25446766c8befe26ca2a8ccb0890926fdb7 → 1531fda69d3c9179f72c0c71cb702f65630b8afc 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
2b39b3d  Merge #34858

c59622a  build/pkgs/sagemath*/spkgsrc: Use build

7fc93d0  build/pkgs/sagemath*/spkgsrc: Use skipdependencycheck

4940db4  build/make/Makefile.in: Add targets SPKGsrc for script packages

573fcfd  build/make/Makefile.in: Move targets pypisdists, wheel, pypiwheels here; use SPKGsdist targets

88e18d0  Merge #34855

0bd13ac  build/make/Makefile.in: Remove duplicate setting of PYPI_WHEEL_PACKAGES

f712d67  Merge #34855

5fc1add  build/pkgs/sagemath*/spkgsrc: Use build

1531fda  build/pkgs/sagemath_flint_arb/dependencies: Fix up

comment:68 Changed 7 weeks ago by
Dependencies:  #31031, #34855 → #31031, #34855, #34839 

comment:69 Changed 4 weeks ago by
Milestone:  sage9.8 → sage9.9 

Good idea to separate this into a new ticket.
While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.
Also, I think, there should be distributions for GAP, CCObject, linbox, ratpoints, zn_poly and mari.
What do you think about renaming sage_setup: distribution to sage_setup: dependency? I think this better reflects the purpose. The distribution is then generated based on which dependencies are available. Another idea would be to use
distutils: libraries
which already contains some of the dependencies anyway (e.g. arb, gmp, flint).