Opened 2 years ago

Last modified 3 months ago

#30914 new enhancement

Meta-ticket: Create upstream repositories, pip-installable packages for database packages

Reported by: Tobias Diez Owned by:
Priority: major Milestone: sage-9.8
Component: build Keywords: sd111
Cc: Matthias Köppe, John Palmieri, François Bissey, Isuru Fernando, Antonio Rojas, Dima Pasechnik, Samuel Lelièvre, Tobias Hansen, Jerry James, Sebastian Oehms, Vincent Delecroix, William Stein, David Roe Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

The Sage distribution contains a number of "database packages", many of which do not seem to have a "real upstream" (or installation procedures other than our scripts in build/pkgs).

We transform them into pip-installable packages and publish them to PyPI so that they can be installed using standard Python tools (and become Python dependencies of sagelib) and then discovered by libraries using standard Python facilities.

Role model: https://pypi.org/project/database-knotinfo/

For those that do not have a real upstream, we create separate git repositories in github.com/sagemath/ to serve as new upstream.

List of packages/tickets:

Architecture independent:

Prepared files:

  • #32747: conway_polynomials
  • elliptic_curves

New data sources:

Related (other stuff written by the Sage distribution into $SAGE_SHARE):

Techniques:

Change History (38)

comment:1 Changed 2 years ago by Matthias Köppe

Which files, specifically?

comment:2 Changed 2 years ago by Matthias Köppe

If you are talking about database packages such as build/pkgs/conway_polynomials, in particular those that do not seem to have a "real upstream" (or installation procedures other than our scripts in build/pkgs), then yes, I agree that a possible direction would be to create pip-installable packages for them and to publish them to PyPI. They should be maintained outside of the source tree, each in a separate git repository.

comment:3 Changed 2 years ago by Matthias Köppe

Cc: John Palmieri François Bissey added

If this is what you had in mind, let's transform this ticket into a meta-ticket and get the downstream packagers of Sage on board -- they may have insights on the upstream status of some of these packages, which we may have forgotten about.

comment:4 Changed 2 years ago by François Bissey

Yes, a confirmation of which bits would be nice. If @mkoeppe is right you are talking about

  • combinatorial_designs
  • conway_polynomials
  • elliptic_curves
  • graphs
  • polytopes_db
  • plus some other optional packages

They are indeed static but come in two flavors.

Architecture independent:

  • combinatorial_designs
  • graphs
  • polytopes_db

which are really file copying.

Prepared files

  • conway_polynomials
  • elliptic_curves

where the files are processed before install with some python code that may be depending on python or some other software.

conway_polynomials uses pickle.dump which could pose problems across python versions?

elliptic_curves is put into a SQL database with sqlite, this should be a stable format. Don't know if that's portable between, say, x86_64 and ppc64le?

comment:5 Changed 2 years ago by Matthias Köppe

Cc: Isuru Fernando Antonio Rojas Dima Pasechnik Samuel Lelièvre added
Description: modified (diff)
Summary: Generate sage/share files in sage/setup.py?Meta-ticket: Create upstream repositories, pip-installable packages for database packages

OK, I've taken the liberty to rewrite the ticket description

comment:6 in reply to:  4 Changed 2 years ago by Matthias Köppe

Replying to fbissey:

Prepared files

  • conway_polynomials
  • elliptic_curves

where the files are processed before install with some python code that may be depending on python or some other software.

conway_polynomials uses pickle.dump which could pose problems across python versions?

elliptic_curves is put into a SQL database with sqlite, this should be a stable format. Don't know if that's portable between, say, x86_64 and ppc64le?

For these ones, we should make the actual source distribution architecture-independent. The architecture-dependent products could be built during setup.py - and published as wheels.

comment:7 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

For example database_stein_watkins_mini's SPKG.rst lists http://modular.math.washington.edu/papers/stein-watkins/ as upstream or source, which is defunct

comment:8 Changed 2 years ago by Matthias Köppe

One thing that may need discussion is that for huge databases, one may want to avoid installing multiple copies

comment:9 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:10 in reply to:  7 ; Changed 2 years ago by Samuel Lelièvre

Replying to mkoeppe:

For example database_stein_watkins_mini's SPKG.rst lists http://modular.math.washington.edu/papers/stein-watkins/ as upstream or source, which is defunct

It's at https://wstein.org/papers/stein-watkins/

comment:11 in reply to:  4 Changed 2 years ago by Tobias Diez

Replying to fbissey:

Yes, a confirmation of which bits would be nice. If @mkoeppe is right you are talking about ...

Yes, that's exactly what I meant! Thanks everybody for their input, that's already way better than what I've imagined originally.

comment:12 Changed 2 years ago by Matthias Köppe

Cc: Tobias Hansen Jerry James added

comment:13 Changed 2 years ago by Matthias Köppe

Keywords: sd111 added

comment:14 Changed 22 months ago by Matthias Köppe

Description: modified (diff)

comment:15 Changed 22 months ago by Matthias Köppe

Cc: Sebastian Oehms added
Description: modified (diff)

comment:16 Changed 22 months ago by Matthias Köppe

Description: modified (diff)

comment:17 Changed 22 months ago by Matthias Köppe

Description: modified (diff)

comment:18 Changed 22 months ago by Matthias Köppe

Description: modified (diff)

comment:19 Changed 22 months ago by Matthias Köppe

Description: modified (diff)

comment:20 Changed 22 months ago by Matthias Köppe

Description: modified (diff)

comment:21 Changed 21 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

comment:22 Changed 20 months ago by Matthias Köppe

Cc: Vincent Delecroix added

comment:23 Changed 17 months ago by Matthias Köppe

Milestone: sage-9.4sage-9.5

comment:24 Changed 14 months ago by Matthias Köppe

For #32432 (sagemath-polyhedra), pip-installable polytopes_db, polytopes_db_4d would be good

comment:25 Changed 14 months ago by Matthias Köppe

Description: modified (diff)

comment:26 Changed 12 months ago by Matthias Köppe

Milestone: sage-9.5sage-9.6

comment:27 Changed 9 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:28 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:29 Changed 7 months ago by Matthias Köppe

Cc: William Stein added

comment:30 Changed 7 months ago by Matthias Köppe

Cc: David Roe added

comment:31 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:32 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:33 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:34 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:35 in reply to:  10 Changed 7 months ago by Matthias Köppe

Replying to slelievre:

Replying to mkoeppe:

For example database_stein_watkins_mini's SPKG.rst lists http://modular.math.washington.edu/papers/stein-watkins/ as upstream or source, which is defunct

It's at https://wstein.org/papers/stein-watkins/

The link "Click here for the database" (https://wstein.org/ecdb) and "Our database" (https://wstein.org/papers/stein-watkins/ecdb) are both broken

comment:36 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:37 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:38 Changed 3 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8
Note: See TracTickets for help on using tickets.