Opened 20 months ago
Last modified 5 weeks ago
#30914 new enhancement
Meta-ticket: Create upstream repositories, pip-installable packages for database packages
Reported by: | gh-tobiasdiez | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.7 |
Component: | build | Keywords: | sd111 |
Cc: | mkoeppe, jhpalmieri, fbissey, isuruf, arojas, dimpase, slelievre, thansen, gh-jamesjer, soehms, vdelecroix, was, roed | Merged in: | |
Authors: | Reviewers: | ||
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
The Sage distribution contains a number of "database packages", many of which do not seem to have a "real upstream" (or installation procedures other than our scripts in build/pkgs).
We transform them into pip-installable packages and publish them to PyPI so that they can be installed using standard Python tools (and become Python dependencies of sagelib) and then discovered by libraries using standard Python facilities.
Role model: https://pypi.org/project/database-knotinfo/
For those that do not have a real upstream, we create separate git repositories in github.com/sagemath/ to serve as new upstream.
List of packages/tickets:
Architecture independent:
- #33881 New upstream for
database_stein_watkins
combinatorial_designs
(single text fileMOLS_table.txt
), 8 kB, https://repology.org/project/sagemath-combinatorial-designs/versionsgraphs
(graphs.db
,brouwer_srg_database.json
,smallgraphs.txt
,isgci_sage.xml
), 336 kB; theSPKG.rst
explains: The code used to parse the data from Andries E. Brouwer's website is available at https://github.com/nathanncohen/strongly_regular_graphs_databasepolytopes_db
(various files), 41 kBpolytopes_db_4d
(various files), 878 MB
Prepared files:
- #32747:
conway_polynomials
- elliptic_curves
New data sources:
- #30352 Interface to the KnotInfo and LinkInfo databases
database_graver_ppi
- https://www.math.ucdavis.edu/~mkoeppe/art/ppi/index.html- https://mathdb.mathhub.info/
- https://swmath.org/
Related (other stuff written by the Sage distribution into $SAGE_SHARE):
Techniques:
importlib.resources
- would help making a distributionzip_safe
(#31306)- could consider advertise install location using a pkgconfig file: https://trac.sagemath.org/ticket/30787#comment:24
- Reduce local duplication of large database package when venvs are in use: https://pypi.org/project/pydupes/ (similar to rdfind)
- Avoid duplicate installation by using PEP 660 editable wheels. Use a build backend that supports it: flit, pdm, hatchling, poetry, not setuptools. https://discuss.python.org/t/pep-660-and-setuptools/14855
- However there are implementation restrictions in the interaction of editable wheels and implicit namespace packages
Change History (37)
comment:1 Changed 20 months ago by
comment:2 Changed 20 months ago by
If you are talking about database packages such as build/pkgs/conway_polynomials
, in particular those that do not seem to have a "real upstream" (or installation procedures other than our scripts in build/pkgs), then yes, I agree that a possible direction would be to create pip-installable packages for them and to publish them to PyPI. They should be maintained outside of the source tree, each in a separate git repository.
comment:3 Changed 20 months ago by
- Cc jhpalmieri fbissey added
If this is what you had in mind, let's transform this ticket into a meta-ticket and get the downstream packagers of Sage on board -- they may have insights on the upstream status of some of these packages, which we may have forgotten about.
comment:4 follow-ups: ↓ 6 ↓ 11 Changed 20 months ago by
Yes, a confirmation of which bits would be nice. If @mkoeppe is right you are talking about
- combinatorial_designs
- conway_polynomials
- elliptic_curves
- graphs
- polytopes_db
- plus some other optional packages
They are indeed static but come in two flavors.
Architecture independent:
- combinatorial_designs
- graphs
- polytopes_db
which are really file copying.
Prepared files
- conway_polynomials
- elliptic_curves
where the files are processed before install with some python code that may be depending on python or some other software.
conway_polynomials
uses pickle.dump
which could pose problems across python versions?
elliptic_curves
is put into a SQL database with sqlite, this should be a stable format. Don't know if that's portable between, say, x86_64 and ppc64le?
comment:5 Changed 20 months ago by
- Cc isuruf arojas dimpase slelievre added
- Description modified (diff)
- Summary changed from Generate sage/share files in sage/setup.py? to Meta-ticket: Create upstream repositories, pip-installable packages for database packages
OK, I've taken the liberty to rewrite the ticket description
comment:6 in reply to: ↑ 4 Changed 20 months ago by
Replying to fbissey:
Prepared files
- conway_polynomials
- elliptic_curves
where the files are processed before install with some python code that may be depending on python or some other software.
conway_polynomials
usespickle.dump
which could pose problems across python versions?
elliptic_curves
is put into a SQL database with sqlite, this should be a stable format. Don't know if that's portable between, say, x86_64 and ppc64le?
For these ones, we should make the actual source distribution architecture-independent. The architecture-dependent products could be built during setup.py - and published as wheels.
comment:7 follow-up: ↓ 10 Changed 20 months ago by
- Description modified (diff)
For example database_stein_watkins_mini
's SPKG.rst lists http://modular.math.washington.edu/papers/stein-watkins/ as upstream or source, which is defunct
comment:8 Changed 20 months ago by
One thing that may need discussion is that for huge databases, one may want to avoid installing multiple copies
comment:9 Changed 20 months ago by
- Description modified (diff)
comment:10 in reply to: ↑ 7 ; follow-up: ↓ 35 Changed 20 months ago by
Replying to mkoeppe:
For example
database_stein_watkins_mini
's SPKG.rst lists http://modular.math.washington.edu/papers/stein-watkins/ as upstream or source, which is defunct
comment:11 in reply to: ↑ 4 Changed 20 months ago by
Replying to fbissey:
Yes, a confirmation of which bits would be nice. If @mkoeppe is right you are talking about ...
Yes, that's exactly what I meant! Thanks everybody for their input, that's already way better than what I've imagined originally.
comment:12 Changed 20 months ago by
- Cc thansen gh-jamesjer added
comment:13 Changed 19 months ago by
- Keywords sd111 added
comment:14 Changed 17 months ago by
- Description modified (diff)
comment:15 Changed 17 months ago by
- Cc soehms added
- Description modified (diff)
comment:16 Changed 17 months ago by
- Description modified (diff)
comment:17 Changed 17 months ago by
- Description modified (diff)
comment:18 Changed 17 months ago by
- Description modified (diff)
comment:19 Changed 17 months ago by
- Description modified (diff)
comment:20 Changed 17 months ago by
- Description modified (diff)
comment:21 Changed 15 months ago by
- Milestone changed from sage-9.3 to sage-9.4
Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.
comment:22 Changed 14 months ago by
- Cc vdelecroix added
comment:23 Changed 11 months ago by
- Milestone changed from sage-9.4 to sage-9.5
comment:24 Changed 9 months ago by
For #32432 (sagemath-polyhedra), pip-installable polytopes_db
, polytopes_db_4d
would be good
comment:25 Changed 8 months ago by
- Description modified (diff)
comment:26 Changed 7 months ago by
- Milestone changed from sage-9.5 to sage-9.6
comment:27 Changed 4 months ago by
- Milestone changed from sage-9.6 to sage-9.7
comment:28 Changed 5 weeks ago by
- Description modified (diff)
comment:29 Changed 5 weeks ago by
- Cc was added
comment:30 Changed 5 weeks ago by
- Cc roed added
comment:31 Changed 5 weeks ago by
- Description modified (diff)
comment:32 Changed 5 weeks ago by
- Description modified (diff)
comment:33 Changed 5 weeks ago by
- Description modified (diff)
comment:34 Changed 5 weeks ago by
- Description modified (diff)
comment:35 in reply to: ↑ 10 Changed 5 weeks ago by
Replying to slelievre:
Replying to mkoeppe:
For example
database_stein_watkins_mini
's SPKG.rst lists http://modular.math.washington.edu/papers/stein-watkins/ as upstream or source, which is defunct
The link "Click here for the database" (https://wstein.org/ecdb) and "Our database" (https://wstein.org/papers/stein-watkins/ecdb) are both broken
comment:36 Changed 5 weeks ago by
- Description modified (diff)
comment:37 Changed 5 weeks ago by
- Description modified (diff)
Which files, specifically?