Opened 2 years ago

Last modified 3 months ago

#27330 new task

Meta-ticket: spkg-configure: Try to use as many system packages as possible

Reported by: embray Owned by:
Priority: major Milestone: sage-9.4
Component: build Keywords: spkg-configure, sd111
Cc: tmonteil, mjo, gh-timokau, fbissey, isuruf, mmezzarobba, thansen, gh-tobiasdiez Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by dimpase)

#24919 implemented a mechanism for checking for checking for sagelib's direct and indirect dependencies at the time of running the main ./configure script. This was already done in an ad-hoc manner for a handful of packages (e.g. gcc, git, yasm, curl), but we wanted to be able to do the same for other packages, especially those that are common on Linux-like systems.

This has now been implemented for many other packages, or is in the process of being implemented. Those packages are listed below. However, there are many others yet to be done, and this ticket will attempt to list them in some rough order of priority.

For use of Python packages: See #29023 Meta-ticket: In a python 3 build, use system Python packages

Already Done

The following packages already have spkg-configure.m4 checks that are working on most or all supported platforms:

In Progress

The following packages have in-progress tickets to provide configure-time detection on the system:

Have Tickets

The following packages have tickets for their conversions, but don't have working prototypes (i.e. branches) yet:

  • BLAS (atlas and other non-openblas): resolving how to handle finding a usable BLAS should be one of the highest priorities; it faces many of the same issues we faced with MPIR/GMP in #27212, but perhaps with even more complexity). One can already use system's openblas, though (#27870)
  • ecl (depends on gc)
    • maxima (depends on ecl) (#29617)
  • gap (using the system GAP is obviously desirable, but non-trivial, esp. now that we need a working libgap) (#29644)
  • linbox (#29631)
  • pynac (#29542)
  • singular (#29024; dup: #29515)
  • mathjax (#30296)
  • jmol (#30316)
  • 4ti2 (#30887, optional package)
  • csdp (#31145, optional package)
  • qhull (#31161, optional package)

No Tickets yet

  • bliss (optional package)

Use of system Python packages

This will be be handled by #29023. Please do not add Python packages here.

The following are exceptions (because it does not run in the same Python as sage):

  • notebook (#30124), rst2ipynb, and their dependencies

Testing infrastructure

  • #29053: Add debian/fedora package information to build/pkgs, generate Dockerfiles and installation help
  • #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Meta infrastructure

  • #31114 Add one file build/pkgs/*/repology.txt per spkg

Documentation

Unfortunately there is still not good documentation for how to implement these configure-time dependency checks.

Bug-fixing

A number of bugs/deficiencies has been uncovered in spkg-configure.m4 files and related configuration, and a number of tickets opened/closed

Change History (148)

comment:1 Changed 2 years ago by embray

Note: The list of packages in the description is not yet exhaustive; please add others that are missing as you see fit and/or reprioritize. This is just a rough sketch.

comment:2 Changed 2 years ago by embray

  • Description modified (diff)

#27271 is the ticket for ecm.

comment:3 Changed 2 years ago by tmonteil

  • Cc tmonteil added

comment:4 Changed 2 years ago by mjo

  • Cc mjo added

comment:5 Changed 2 years ago by gh-timokau

  • Cc gh-timokau added

comment:6 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:7 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:8 Changed 2 years ago by dimpase

  • Description modified (diff)

There is also #27373, which concerns packages that change build/run environment, e.g. mpir/gmp/mpfr, that should be resolved in some way.

comment:9 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:10 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:11 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:12 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:13 Changed 2 years ago by embray

  • Milestone changed from sage-8.7 to sage-pending

Removing most of the rest of my open tickets out of the 8.7 milestone, which should be closed.

comment:14 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:15 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:16 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:17 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:18 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:19 Changed 2 years ago by dimpase

  • Cc fbissey added
  • Description modified (diff)

comment:20 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:21 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:22 Changed 2 years ago by mjo

  • Description modified (diff)

comment:23 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:24 Changed 2 years ago by embray

  • Description modified (diff)

comment:25 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:26 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:27 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:28 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:29 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:30 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:31 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:32 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:33 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:34 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:35 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:36 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:37 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:38 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:39 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:40 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:41 Changed 2 years ago by isuruf

  • Cc isuruf added

comment:42 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:43 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:44 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:45 Changed 23 months ago by dimpase

  • Description modified (diff)

comment:46 Changed 23 months ago by dimpase

  • Description modified (diff)

comment:47 Changed 23 months ago by dimpase

  • Description modified (diff)

comment:48 Changed 23 months ago by dimpase

  • Description modified (diff)

comment:49 Changed 23 months ago by dimpase

  • Description modified (diff)

comment:50 Changed 23 months ago by dimpase

  • Description modified (diff)

comment:51 Changed 23 months ago by dimpase

  • Description modified (diff)

comment:52 Changed 23 months ago by dimpase

  • Description modified (diff)

comment:53 Changed 22 months ago by jdemeyer

  • Description modified (diff)

comment:54 Changed 22 months ago by dimpase

  • Description modified (diff)

comment:55 Changed 22 months ago by mmezzarobba

  • Cc mmezzarobba added
  • Description modified (diff)

comment:56 Changed 22 months ago by mmezzarobba

  • Description modified (diff)

comment:57 Changed 22 months ago by mmezzarobba

  • Description modified (diff)

comment:58 Changed 22 months ago by dimpase

  • Description modified (diff)

comment:59 Changed 22 months ago by embray

  • Description modified (diff)

comment:60 Changed 22 months ago by dimpase

  • Description modified (diff)

comment:61 Changed 22 months ago by dimpase

  • Description modified (diff)

comment:62 Changed 22 months ago by dimpase

  • Description modified (diff)

comment:63 Changed 21 months ago by dimpase

  • Description modified (diff)

comment:64 Changed 20 months ago by dimpase

  • Description modified (diff)

comment:65 Changed 18 months ago by dimpase

  • Description modified (diff)

comment:66 Changed 18 months ago by dimpase

  • Description modified (diff)

comment:67 Changed 18 months ago by dimpase

  • Description modified (diff)

comment:68 Changed 18 months ago by dimpase

  • Description modified (diff)

comment:69 Changed 18 months ago by dimpase

  • Description modified (diff)

comment:70 Changed 18 months ago by embray

Nice to see so much progress on this. I'm sorry I haven't had more time to help, but I'm glad to see that the infrastructure I put in place has continued to be useful as support for more packages is added.

comment:71 Changed 18 months ago by dimpase

  • Description modified (diff)

comment:72 Changed 18 months ago by dimpase

  • Description modified (diff)

comment:73 Changed 18 months ago by dimpase

  • Description modified (diff)

comment:74 Changed 18 months ago by mjo

  • Description modified (diff)

comment:75 Changed 17 months ago by dimpase

  • Description modified (diff)

comment:76 Changed 17 months ago by mjo

  • Description modified (diff)

comment:77 Changed 17 months ago by mjo

  • Description modified (diff)

comment:78 follow-up: Changed 17 months ago by mkoeppe

Should we find a better way to keep track of the distribution packages that provide what spkg-configure is looking for?

Something autogenerated from new per-package information build/pkgs/* that can keep the section in the installation manual current (src/doc/en/installation/source.rst) - last updated by Dima?

Related: https://trac.sagemath.org/ticket/28745 (environment.yml for conda).

comment:79 in reply to: ↑ 78 Changed 17 months ago by dimpase

Replying to mkoeppe:

Should we find a better way to keep track of the distribution packages that provide what spkg-configure is looking for?

Something autogenerated from new per-package information build/pkgs/* that can keep the section in the installation manual current (src/doc/en/installation/source.rst) - last updated by Dima?

A good idea, I only don't know how exactly to implement this. E.g. on Debian one could create a package with all dependencies (in fact, basically almost the same as their Sage package)

Related: https://trac.sagemath.org/ticket/28745 (environment.yml for conda).

comment:80 Changed 17 months ago by mkoeppe

I would just have thought to generate a shell script with one line of apt-get install.

comment:81 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:82 Changed 17 months ago by dimpase

  • Description modified (diff)

comment:83 Changed 17 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Try to use as many system packages as possible to spkg-configure: Try to use as many system packages as possible

comment:84 Changed 17 months ago by mkoeppe

  • Summary changed from spkg-configure: Try to use as many system packages as possible to Meta-ticket: spkg-configure: Try to use as many system packages as possible

comment:85 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:86 Changed 17 months ago by dimpase

  • Description modified (diff)

comment:87 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:88 Changed 17 months ago by dimpase

  • Description modified (diff)

comment:89 Changed 16 months ago by mjo

  • Description modified (diff)

comment:90 Changed 15 months ago by mjo

  • Description modified (diff)

comment:91 Changed 15 months ago by mjo

  • Description modified (diff)

comment:92 Changed 15 months ago by mjo

  • Description modified (diff)

comment:93 Changed 15 months ago by thansen

  • Cc thansen added

comment:94 Changed 15 months ago by mjo

  • Description modified (diff)

comment:95 Changed 15 months ago by gh-thierry-FreeBSD

  • Cc thansen removed
  • Description modified (diff)

rw is listed (#27814), but since it depends on igraph, it would be fine to add igraph too.

comment:96 Changed 15 months ago by dimpase

  • Cc thansen added

comment:97 Changed 15 months ago by dimpase

  • Description modified (diff)

comment:98 Changed 15 months ago by mjo

  • Description modified (diff)

comment:99 Changed 15 months ago by mjo

  • Description modified (diff)

comment:100 Changed 15 months ago by mjo

  • Description modified (diff)

comment:101 Changed 15 months ago by mjo

  • Description modified (diff)

comment:102 Changed 14 months ago by dimpase

  • Description modified (diff)

comment:103 Changed 14 months ago by mkoeppe

spkg-configure friends, I have drafted an update on the support for using system packages for the 9.1 release notes at https://wiki.sagemath.org/ReleaseTours/sage-9.1; let me know if anything needs changing or just make edits.

comment:104 Changed 14 months ago by dimpase

  • Description modified (diff)

comment:105 Changed 14 months ago by mkoeppe

  • Description modified (diff)

comment:106 Changed 14 months ago by mkoeppe

Should "BLAS" be moved to "done"?

comment:107 Changed 14 months ago by dimpase

There should be a way to use other blas implementations from the system, not only openblas

comment:108 Changed 14 months ago by dimpase

  • Description modified (diff)

comment:109 Changed 14 months ago by gh-thierry-FreeBSD

  • Description modified (diff)

Adding a reference for giac and pynac.

comment:110 Changed 14 months ago by gh-thierry-FreeBSD

  • Description modified (diff)

comment:111 Changed 14 months ago by gh-thierry-FreeBSD

  • Description modified (diff)

Adding tickets for libhomfly and linbox.

comment:112 Changed 14 months ago by gh-thierry-FreeBSD

  • Description modified (diff)

Adding #29644 for GAP.

comment:113 Changed 14 months ago by gh-thierry-FreeBSD

  • Description modified (diff)

Adding #29649 for zn_poly.

comment:114 Changed 14 months ago by dimpase

  • Description modified (diff)

comment:115 Changed 13 months ago by gh-thierry-FreeBSD

  • Description modified (diff)

Adding cvxopt.

comment:116 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:117 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:118 Changed 13 months ago by dimpase

  • Description modified (diff)

comment:119 Changed 13 months ago by dimpase

  • Description modified (diff)

comment:120 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:121 Changed 13 months ago by dimpase

  • Description modified (diff)

comment:122 Changed 13 months ago by dimpase

  • Description modified (diff)

comment:123 Changed 13 months ago by dimpase

  • Description modified (diff)

comment:124 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:125 Changed 12 months ago by dimpase

  • Description modified (diff)

comment:126 Changed 11 months ago by dimpase

  • Description modified (diff)

comment:127 Changed 10 months ago by mkoeppe

  • Description modified (diff)

comment:128 Changed 10 months ago by mkoeppe

  • Description modified (diff)

comment:129 Changed 10 months ago by dimpase

  • Description modified (diff)

comment:130 Changed 9 months ago by mkoeppe

  • Description modified (diff)
  • Milestone changed from sage-pending to sage-9.3

comment:131 Changed 9 months ago by mkoeppe

  • Description modified (diff)

comment:132 Changed 9 months ago by mkoeppe

  • Description modified (diff)

comment:133 Changed 9 months ago by mkoeppe

  • Description modified (diff)

comment:134 Changed 9 months ago by mkoeppe

  • Description modified (diff)

comment:135 Changed 8 months ago by dimpase

  • Description modified (diff)

comment:136 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:137 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:138 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:139 Changed 7 months ago by dimpase

  • Description modified (diff)

comment:140 Changed 7 months ago by mkoeppe

  • Keywords sd111 added

comment:141 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:142 Changed 5 months ago by dimpase

  • Description modified (diff)

comment:143 Changed 5 months ago by dimpase

  • Description modified (diff)

comment:144 Changed 5 months ago by mjo

  • Description modified (diff)

comment:145 Changed 5 months ago by mjo

  • Description modified (diff)

comment:146 Changed 5 months ago by gh-tobiasdiez

  • Cc gh-tobiasdiez added

comment:147 Changed 3 months ago by dimpase

  • Description modified (diff)

comment:148 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

Note: See TracTickets for help on using tickets.