Opened 2 years ago
Last modified 4 weeks ago
#30666 needs_work enhancement
Distributions sagemath-{flint-arb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.9 |
Component: | refactoring | Keywords: | sd111 |
Cc: | gh-tobiasdiez, dimpase | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | Dima Pasechnik |
Report Upstream: | N/A | Work issues: | |
Branch: | u/mkoeppe/add__sage_setup__distribution__directives_to_all_cython_modules_needing_external_libraries (Commits, GitHub, GitLab) | Commit: | 1531fda69d3c9179f72c0c71cb702f65630b8afc |
Dependencies: | #31031, #34855, #34839 | Stopgaps: |
Description (last modified by )
Basically, we are making all external libraries optional, except for a minimal set of "core" libraries such as cysignals
and gmpy2
(see #29865).
(from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sagemath-ntl
) - but some libraries come in packs that would not make much sense to separate (such as sagemath-flint-arb-e_antic
).
List of distributions:
- sagemath-flint-arb
- 'sage/libs/arb/arb_version.pyx',
- 'sage/libs/arb/arith.pyx',
- 'sage/matrix/matrix_complex_ball_dense.pyx',
- 'sage/rings/complex_arb.pyx',
- 'sage/rings/number_field/number_field_element_quadratic.pyx',
- 'sage/rings/polynomial/polynomial_complex_arb.pyx',
- 'sage/rings/real_arb.pyx'
- etc.
- sagemath-homfly: 'sage/libs/homfly.pyx'
- sagemath-giac (ex
giacpy_sage
insage.libs.giac
+ dependency onlibgiac
)- sage/libs/giac/giac.pyx
- sagemath-gap:
- 'sage/coding/codecan/codecan.pyx',
- 'sage/combinat/enumeration_mod_permgroup.pyx',
- 'sage/combinat/root_system/reflection_group_c.pyx',
- 'sage/combinat/root_system/reflection_group_element.pyx',
- 'sage/graphs/spanning_tree.pyx',
- 'sage/groups/libgap_wrapper.pyx',
- 'sage/groups/matrix_gps/group_element.pyx',
- 'sage/groups/perm_gps/permgroup_element.pyx',
- 'sage/groups/perm_gps/partn_ref/automorphism_group_canonical_label.pyx',
- 'sage/groups/perm_gps/partn_ref/canonical_augmentation.pyx',
- 'sage/groups/perm_gps/partn_ref/data_structures.pyx',
- 'sage/groups/perm_gps/partn_ref/double_coset.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_binary.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_graphs.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_lists.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_matrices.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_python.pyx',
- 'sage/groups/perm_gps/partn_ref/refinement_sets.pyx',
- 'sage/groups/perm_gps/partn_ref2/refinement_generic.pyx',
- 'sage/libs/gap/element.pyx',
- 'sage/libs/gap/libgap.pyx',
- 'sage/libs/gap/util.pyx',
- 'sage/matrix/matrix_gap.pyx',
- 'sage/sets/disjoint_set.pyx'
To test:
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_flint_arb
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_gap
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_giac
make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_homfly
In follow-up tickets:
sagemath-brial
(exsage-brial
insage.rings.polynomial.pbori
+sage.libs.polybori
,sage.crypto.boolean_function
)sagemath-ntl
sagemath-ecl
(sage.libs.ecl
)sagemath-maxima
: requires sage-eclsagemath-singular
: (some of these files actually may be better put in the pynac distribution)- 'sage/algebras/letterplace/free_algebra_element_letterplace.pyx',
- 'sage/algebras/letterplace/free_algebra_letterplace.pyx',
- 'sage/algebras/letterplace/letterplace_ideal.pyx',
- 'sage/libs/pynac/constant.pyx',
- 'sage/libs/pynac/pynac.pyx',
- 'sage/libs/singular/function.pyx',
- 'sage/libs/singular/groebner_strategy.pyx',
- 'sage/libs/singular/option.pyx',
- 'sage/libs/singular/polynomial.pyx',
- 'sage/libs/singular/ring.pyx',
- 'sage/libs/singular/singular.pyx',
- 'sage/matrix/matrix_mpolynomial_dense.pyx',
- 'sage/rings/polynomial/multi_polynomial_ideal_libsingular.pyx',
- 'sage/rings/polynomial/multi_polynomial_libsingular.pyx',
- 'sage/rings/polynomial/plural.pyx',
- 'sage/symbolic/comparison.pyx',
- 'sage/symbolic/constants_c.pyx',
- 'sage/symbolic/expression.pyx',
- 'sage/symbolic/function.pyx',
- 'sage/symbolic/getitem.pyx',
- 'sage/symbolic/ring.pyx',
- 'sage/symbolic/series.pyx',
- 'sage/symbolic/substitution_map.pyx',
sagemath-pynac
sagemath-linbox
:- 'sage/libs/linbox/linbox_flint_interface.pyx',
- 'sage/matrix/matrix_integer_sparse.pyx',
- 'sage/matrix/matrix_modn_dense_double.pyx',
- 'sage/matrix/matrix_modn_dense_float.pyx',
- 'sage/matrix/matrix_modn_sparse.pyx'
sagemath-znpoly
(subject to removal in #32841?)- 'sage/modular/modsym/p1list.pyx',
- 'sage/modular/pollack_stevens/dist.pyx',
- 'sage/rings/fraction_field_FpT.pyx',
- 'sage/rings/polynomial/polynomial_zmod_flint.pyx',
- 'sage/schemes/hyperelliptic_curves/hypellfrob.pyx',
sagemath-mari
(also requiring pbori and pbori-groebner):- 'sage/matrix/matrix_gf2e_dense.pyx',
- 'sage/matrix/matrix_integer_dense.pyx',
- 'sage/matrix/matrix_mod2_dense.pyx',
- 'sage/matrix/matrix_rational_dense.pyx',
- 'sage/modules/vector_mod2_dense.pyx',
- 'sage/rings/polynomial/pbori/pbori.pyx',
- 'sage/rings/polynomial/polynomial_gf2x.pyx'
This is preparation for Meta-ticket #29705 (Modularization) and #30371.
Change History (69)
comment:1 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:2 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:3 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:4 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:5 follow-up: 6 Changed 2 years ago by
comment:6 follow-up: 13 Changed 2 years ago by
Replying to gh-tobiasdiez:
While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.
I think they should be separate because I really want to be able to make singular optional. pynac, on the other hand, is essential for our symbolics implementation.
Also, I think, there should be distributions for GAP, CCObject, linbox, ratpoints, zn_poly and mari.
Yes, please add to the list in the ticket description with explanation (what is CCObject?)
What do you think about renaming sage_setup: distribution to sage_setup: dependency?
No, I wouldn't like that. As soon as you start with declaring "dependencies", there will be combinatorial explosion from the combinations of such dependencies. I really want a well-designed partition of the source files into (relatively few) disjoint distributions. This is crucial for the modularization -- each distribution will eventually be separately compilable and installable package on PyPI.
comment:7 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:8 follow-up: 9 Changed 2 years ago by
Ok, makes sense. Which distribution should be assigned to files that have two dependencies (e.g use pynac and singular)?
My idea was that the cython file declares which dependencies it has, and in the setup.py file you have a list of distributions and their dependencies. If all dependencies are met, that particular distribution is built. E.g. distribution "sage-singular" relies on "pynac" and "singular". (That's only a feeling, but it feels strange to have distributions determined by dependencies and not by a theme or subject, e.g. "sage-singular" vs "sage-symbolic calculation". But you have a better overview, so I trust you there.)
I've added the above distributions to the list. (Forget about CCObject, it was a missing import in #30371 that confused me.)
comment:9 Changed 2 years ago by
Replying to gh-tobiasdiez:
Which distribution should be assigned to files that have two dependencies (e.g use pynac and singular)?
Distributions can certainly depend on other distributions. sage-singular
could depend on sage-pynac
. Let's include this in the ticket description. (In the modularized build, sage-singular
's setup.py
or setup.cfg
would have install_requires = sage-pynac
.)
Also, often it is relatively easy to remove some of these dependencies by small changes to a few source files, moving a few methods around. See for example #29911. We can take work on this in follow-up tickets.
it feels strange to have distributions determined by dependencies and not by a theme or subject, e.g. "sage-singular" vs "sage-symbolic calculation".
Perhaps, but there is actually an important "political" (open source community) reason for this design too. By naming these components after major libraries that provide the features, we make these libraries and their important role for Sage more visible to end users. Attribution tends to be an important motivating factor to developers of these libraries.
comment:10 Changed 2 years ago by
To expand on this a little bit: I wouldn't rule out distributions named after topics. The present ticket only provides a "skeleton" of distributions that are (1) technically needed to take care of compile-time dependencies, (2) provides attribution. Later on (in follow-up tickets), for example, there could be a distribution sage-feature-padics
or something like this that has sage-ntl
as a dependency and packages sage.rings.padics
.
comment:11 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:12 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:13 Changed 2 years ago by
Replying to mkoeppe:
Replying to gh-tobiasdiez:
While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.
I think they should be separate because I really want to be able to make singular optional. pynac, on the other hand, is essential for our symbolics implementation.
When I wrote this, I didn't recall that singular's libfactory
is actually a dependency of pynac. So I think you are right, it's best to have pynac and singular in one distribution (for now, at least).
comment:14 Changed 2 years ago by
Keywords: | sd111 added |
---|
comment:15 Changed 2 years ago by
Description: | modified (diff) |
---|
comment:16 Changed 2 years ago by
Dependencies: | → #31031 |
---|
comment:17 Changed 2 years ago by
Milestone: | sage-9.3 → sage-9.4 |
---|
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
comment:18 Changed 19 months ago by
Milestone: | sage-9.4 → sage-9.5 |
---|
comment:19 Changed 14 months ago by
Milestone: | sage-9.5 → sage-9.6 |
---|
comment:20 Changed 10 months ago by
Milestone: | sage-9.6 → sage-9.7 |
---|
comment:21 Changed 5 months ago by
Milestone: | sage-9.7 → sage-9.8 |
---|
comment:22 Changed 3 months ago by
Branch: | → u/mkoeppe/add__sage_setup__distribution__directives_to_all_cython_modules_needing_external_libraries |
---|
comment:23 Changed 3 months ago by
Commit: | → 3f3bfefbac2e7fc96950a6ec9459783a561c9bce |
---|---|
Description: | modified (diff) |
New commits:
3f3bfef | Mark files # sage_setup: distribution = sagemath-flint-arb
|
comment:24 Changed 3 months ago by
Description: | modified (diff) |
---|
comment:25 Changed 3 months ago by
Commit: | 3f3bfefbac2e7fc96950a6ec9459783a561c9bce → 1ab0438283b4ee4f0c5dfdb8d08a41ad739ab21b |
---|
comment:26 Changed 3 months ago by
Commit: | 1ab0438283b4ee4f0c5dfdb8d08a41ad739ab21b → 52b208bc7d280d937c341f66a673650c6ac916c8 |
---|
comment:27 Changed 3 months ago by
Description: | modified (diff) |
---|
comment:28 Changed 3 months ago by
Commit: | 52b208bc7d280d937c341f66a673650c6ac916c8 → 6c41717002434b27c7b3effe42334704175461cb |
---|
comment:29 Changed 3 months ago by
Commit: | 6c41717002434b27c7b3effe42334704175461cb → cd8358caa5e00f249a716030131e39628827cbad |
---|
comment:30 Changed 2 months ago by
Cc: | dimpase added |
---|---|
Summary: | Add "sage_setup: distribution" directives to all Cython modules needing external libraries → Distributions sagemath-{flint-arb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules |
comment:31 Changed 2 months ago by
Commit: | cd8358caa5e00f249a716030131e39628827cbad → 9f26fb8c102e05033b60679798afba7ad031e796 |
---|
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
619c8c8 | Mark files # sage_setup: distribution = sagemath-flint-arb
|
71b5921 | Mark files # sage_setup: distribution = sagemath-homfly
|
fe08ab2 | Mark files # sage_setup: distribution = sagemath-giac
|
640aa74 | Mark files # sage_setup: distribution = sagemath-gap
|
d1ef655 | Do not mark *.h files with # sage-setup: distribution
|
8c46aab | pkgs/sagemath-gap, build/pkgs/sagemath_gap: New
|
6db9b01 | pkgs/sagemath-homfly, build/pkgs/sagemath_homfly: New
|
390ebb8 | build/pkgs/sagemath_{gap,homfly}/spkg-install: Symlink to sagemath_objects spkg-install
|
9f26fb8 | pkgs/sagemath-gap/MANIFEST.in: Add VERSION.txt
|
comment:32 Changed 2 months ago by
Commit: | 9f26fb8c102e05033b60679798afba7ad031e796 → 963fa7704055f99af12c80654f904ee8ffd21c22 |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
13f030c | src/sage/libs/gap: Add # sage_setup: distribution to more files
|
a73b489 | Add src/sage/**/all__sagemath_gap.py
|
30112b2 | pkgs/sagemath-gap: WIP setup.py
|
cae814a | src/sage/groups/perm_gps/partn_ref/data_structures.pyx: Remove cimport from sage.libs.flint (only used by doctesting helper functions)
|
5fda124 | Add src/sage/**/all__sagemath_gap.py
|
a2feea0 | Add src/sage/**/all__sagemath_gap.py
|
665d01c | pkgs/sagemath-gap/tox.ini: New
|
963fa77 | pkgs/sagemath-gap/MANIFEST.in: Add deps for sage/coding/codecan
|
comment:33 Changed 2 months ago by
Description: | modified (diff) |
---|
comment:34 Changed 2 months ago by
Commit: | 963fa7704055f99af12c80654f904ee8ffd21c22 → 96b863ce7d3ce45460e81afcac5d7448c7ee720b |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
96b863c | pkgs/sagemath-gap: Update description
|
comment:35 Changed 2 months ago by
Commit: | 96b863ce7d3ce45460e81afcac5d7448c7ee720b → b73a92956789a2548063ce867556b2d81d49f2dd |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
b73a929 | build/pkgs/sagemath_gap/dependencies: Update
|
comment:36 Changed 2 months ago by
Commit: | b73a92956789a2548063ce867556b2d81d49f2dd → 0869a7f471b569af4f73506d4ab27d5c09654e75 |
---|
comment:37 Changed 2 months ago by
Commit: | 0869a7f471b569af4f73506d4ab27d5c09654e75 → 961fb3f744e09868a59ca87d7ae5b590d886e338 |
---|
comment:38 Changed 2 months ago by
Commit: | 961fb3f744e09868a59ca87d7ae5b590d886e338 → 47a2519acbf15b502a619288da2f62d2c208c018 |
---|
comment:39 Changed 2 months ago by
Commit: | 47a2519acbf15b502a619288da2f62d2c208c018 → 513fc3f3e0ae775cb5f92f6fc81cb9d53484851f |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
513fc3f | Add tox.ini
|
comment:40 Changed 2 months ago by
Commit: | 513fc3f3e0ae775cb5f92f6fc81cb9d53484851f → 0f198cafdc72abcab02994e66cdbd9c917b31bc3 |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
0f198ca | Makefile: Add to PYPI_WHEEL_PACKAGES
|
comment:41 Changed 2 months ago by
Authors: | → Matthias Koeppe |
---|---|
Status: | new → needs_review |
comment:42 Changed 2 months ago by
How are all these tags like # sage_setup: distribution = sagemath-gap
processed?
comment:43 Changed 2 months ago by
They are processed by the functions in sage_setup.find
, via sage.misc.package_dir
.
comment:44 Changed 2 months ago by
Commit: | 0f198cafdc72abcab02994e66cdbd9c917b31bc3 → 133e5b811eac34ef3b14a1c18596cb679d3002ab |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
133e5b8 | Remove copy-pasted conda.txt files
|
comment:45 Changed 2 months ago by
sage -t --random-seed=146157697718740975867546676674797881386 sage/groups/perm_gps/partn_ref/data_structures.pyx ********************************************************************** File "sage/groups/perm_gps/partn_ref/data_structures.pyx", line 1221, in sage.groups.perm_gps.partn_ref.data_structures.SC_test_list_perms Failed example: for i in range(2,9): test_Sn_on_m_points(i,i,1,0) Expected nothing Got: ImportError: cannot import name n_is_prime Exception ignored in: 'sage.groups.perm_gps.partn_ref.data_structures.SC_is_giant' Traceback (most recent call last): File "<doctest sage.groups.perm_gps.partn_ref.data_structures.SC_test_list_perms[2]>", line 4, in test_Sn_on_m_points
comment:46 Changed 2 months ago by
Status: | needs_review → needs_work |
---|
comment:47 Changed 2 months ago by
Commit: | 133e5b811eac34ef3b14a1c18596cb679d3002ab → 4c8f5b5db2450e99e5968d4dedc88bb0e89127ca |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
4c8f5b5 | src/sage/sets/disjoint_set.pyx: Remove obsolete -*- coding -*- line
|
comment:48 Changed 2 months ago by
Commit: | 4c8f5b5db2450e99e5968d4dedc88bb0e89127ca → 2be77afa09adf9e4f079569d1aa9aeb501de1956 |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
2be77af | pkgs/sagemath-flint-arb: Update description
|
comment:49 Changed 2 months ago by
Commit: | 2be77afa09adf9e4f079569d1aa9aeb501de1956 → a827f299c54650e6a71bdb6bede15b8536f06eb0 |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
a827f29 | pkgs/sagemath-giac: Update description
|
comment:50 Changed 2 months ago by
Commit: | a827f299c54650e6a71bdb6bede15b8536f06eb0 → 112e29ec892ab6901b83ce35d0b476f3af008df3 |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
112e29e | src/sage/groups/perm_gps/partn_ref/data_structures.pyx: In doctest support code, avoid use of FLINT
|
comment:51 Changed 2 months ago by
Status: | needs_work → needs_review |
---|
comment:52 Changed 2 months ago by
Description: | modified (diff) |
---|
comment:53 Changed 2 months ago by
there are a number of files with m4
suffix, which does not seem to be m4
files.
Should they rather get in
suffux?
comment:56 Changed 2 months ago by
The .in files in use with autotools are not processed by m4.
These .m4 files are processed by m4. For example, in setup.cfg.m4 you'll see the m4 macro call esyscmd
.
comment:57 Changed 2 months ago by
Status: | needs_review → positive_review |
---|
ah, OK. I misunderstood what "processed" means in this context.
comment:59 Changed 2 months ago by
Reviewers: | → Dima Pasechnik |
---|
comment:60 Changed 8 weeks ago by
When I build the full sagelib with this ticket included (but I do not build those sub distribution yet) in sage-on-gentoo, building the documentation fails. After inspection, I have the sources installed for sage/libs/flint
, sage/libs/arb
etc, but none of compiled module are installed. Is there something I am missing or doing wrong?
comment:62 Changed 8 weeks ago by
src
with the setup.py
from sagemath-standard
. This is when I build against develop or the Volker merging branch. The main reason I am starting from src
in those cases, is that patch does not like links.
comment:63 Changed 8 weeks ago by
I also delete src/sage_setup
so it is not used instead of the installed one.
comment:64 Changed 8 weeks ago by
Status: | positive_review → needs_work |
---|
OK, that's something that I'll have to fix in pkgs/sagemath-standard/setup.py
.
Thanks for catching this
comment:65 Changed 8 weeks ago by
Commit: | 112e29ec892ab6901b83ce35d0b476f3af008df3 → 8b1cf25446766c8befe26ca2a8ccb0890926fdb7 |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
790c12b | Merge tag '9.8.beta5' into t/30666/add__sage_setup__distribution__directives_to_all_cython_modules_needing_external_libraries
|
8b1cf25 | pkgs/sagemath-standard/setup.py: Include files marked as # sage_setup: distribution = sagemath-flint-arb etc.
|
comment:66 Changed 7 weeks ago by
Dependencies: | #31031 → #31031, #34855 |
---|
comment:67 Changed 7 weeks ago by
Commit: | 8b1cf25446766c8befe26ca2a8ccb0890926fdb7 → 1531fda69d3c9179f72c0c71cb702f65630b8afc |
---|
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
2b39b3d | Merge #34858
|
c59622a | build/pkgs/sagemath*/spkg-src: Use build
|
7fc93d0 | build/pkgs/sagemath*/spkg-src: Use --skip-dependency-check
|
4940db4 | build/make/Makefile.in: Add targets SPKG-src for script packages
|
573fcfd | build/make/Makefile.in: Move targets pypi-sdists, wheel, pypi-wheels here; use SPKG-sdist targets
|
88e18d0 | Merge #34855
|
0bd13ac | build/make/Makefile.in: Remove duplicate setting of PYPI_WHEEL_PACKAGES
|
f712d67 | Merge #34855
|
5fc1add | build/pkgs/sagemath*/spkg-src: Use build
|
1531fda | build/pkgs/sagemath_flint_arb/dependencies: Fix up
|
comment:68 Changed 7 weeks ago by
Dependencies: | #31031, #34855 → #31031, #34855, #34839 |
---|
comment:69 Changed 4 weeks ago by
Milestone: | sage-9.8 → sage-9.9 |
---|
Good idea to separate this into a new ticket.
While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.
Also, I think, there should be distributions for GAP, CCObject, linbox, ratpoints, zn_poly and mari.
What do you think about renaming sage_setup: distribution to sage_setup: dependency? I think this better reflects the purpose. The distribution is then generated based on which dependencies are available. Another idea would be to use
distutils: libraries
which already contains some of the dependencies anyway (e.g. arb, gmp, flint).