Opened 4 years ago

Last modified 3 months ago

#27330 new task

Meta-ticket: spkg-configure: Try to use as many system packages as possible

Reported by: Erik Bray Owned by:
Priority: major Milestone: sage-9.8
Component: build: configure Keywords: spkg-configure, sd111
Cc: Thierry Monteil, Michael Orlitzky, Timo Kaufmann, François Bissey, Isuru Fernando, Marc Mezzarobba, Tobias Hansen, Tobias Diez Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Dima Pasechnik)

#24919 implemented a mechanism for checking for checking for sagelib's direct and indirect dependencies at the time of running the main ./configure script. This was already done in an ad-hoc manner for a handful of packages (e.g. gcc, git, yasm, curl), but we wanted to be able to do the same for other packages, especially those that are common on Linux-like systems.

This has now been implemented for many other packages, or is in the process of being implemented. Those packages are listed below. However, there are many others yet to be done, and this ticket will attempt to list them in some rough order of priority.

For use of Python packages: See #29023 Meta-ticket: In a python 3 build, use system Python packages

Already done

The following packages already have spkg-configure.m4 checks that are working on most or all supported platforms:

In Progress

The following packages have in-progress tickets to provide configure-time detection on the system:

Have tickets

The following packages have tickets for their conversions, but don't have working prototypes (i.e. branches) yet:

  • BLAS (atlas and other non-openblas): resolving how to handle finding a usable BLAS should be one of the highest priorities; it faces many of the same issues we faced with MPIR/GMP in #27212, but perhaps with even more complexity). One can already use system's openblas, though (#27870), and we removed ATLAS in #30350.
  • csdp (#31145, optional package)
  • gap (using system GAP is desirable, but non-trivial, especially now that we need a working libgap) (#29644)
  • jmol (#30316)
  • lie (#32055, experimental package)
  • mathjax (#30296)
  • perl (#31839, script package)
  • symengine (#32421, optional package)

No ticket yet

  • bliss (optional package)

Use of system Python packages

This will be be handled by #29023. Please do not add Python packages here.

The following are exceptions (because it does not run in the same Python as sage):

  • notebook (#30124), rst2ipynb, and their dependencies

Testing infrastructure

  • #29053: Add debian/fedora package information to build/pkgs, generate Dockerfiles and installation help
  • #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Meta infrastructure

  • #31114 Add one file build/pkgs/*/repology.txt per spkg

Documentation

Unfortunately there is still not good documentation for how to implement these configure-time dependency checks.

  • #26668: add documentation for implementing spkg-configure.m4 and distros/ for spkgs
  • #29018: improve documentation of m4/spkg-configure.m4
  • #30543: Document spkg-configure.m4 and distros directory

Bug-fixing

A number of bugs/deficiencies has been uncovered in spkg-configure.m4 files and related configuration, and a number of tickets opened/closed

  • #26715: build/pkgs/gfortran/spkg-configure.m4 works incorrectly if CC and CXX are already there
  • #27219: have bootstrap fail gracefully if no pkg-config is available
  • #27373: some dummy packages must be only re-installable via ./configure --with-...
  • #27567: Add --with-system-<spkg> options to configure
  • #27641: SAGE_SPKG_CONFIGURE macro: Add new pre-check and post-check optional arguments
  • #27642: Re-run configure+make after installing an SPKG with sage -i
  • #27936: spkg-configure.m4 for pcre should test for UTF support etc
  • #27941: R installation fails on macOS with libcurl from Anaconda
  • #28095: Add --enable-OPTIONALSPKG options to configure
  • #28317: Use -print-prog-name instead of -print-file-name in gcc spkg-configure.m4
  • #28341: Allow doc to build with SAGE_PARI_CFG undefined
  • #28380: bump up givaro version in spkg-configure.m4
  • #28401: pass correct --with-flint to eclib's configure
  • #28405: correct the test for galpol package in pari's spkg-configure.m4
  • #28409: Flint configure fails when using system GMP in certain cases
  • #28906: generate libpng.pc, zlib.pc if needed
  • #28956: correct AC_CHECK_PROG call in spkg-configure for bzip2
  • #29071: verify if cblas.pc and lapack.pc should be replaced by links to openblas.pc
  • #30752: switch the default mp library to gmp
  • #31348: build/pkgs/mpfr/spkg-configure.m4: Check pkg-config first
  • #31358: python3 spkg-configure.m4: Do not reject python based on sysconfig LDFLAGS containing "-L."
  • #31429: flint spkg-configure: Reject FLINT 2.7.x
  • #31528: Cleanup spkg-configure.m4 files that mix tabs and spaces
  • #31539: conda-forge-standard (linux): python3 spkg-configure.m4 rejects conda's python3
  • #32789: Fix spkg-configure.m4 for singular
  • #33042: givaro spkg-configure: Reject too new versions
  • #33092: spkg-configure.m4 for ffmpeg/imagemagick need feature checks

Change History (161)

comment:1 Changed 4 years ago by Erik Bray

Note: The list of packages in the description is not yet exhaustive; please add others that are missing as you see fit and/or reprioritize. This is just a rough sketch.

comment:2 Changed 4 years ago by Erik Bray

Description: modified (diff)

#27271 is the ticket for ecm.

comment:3 Changed 4 years ago by Thierry Monteil

Cc: Thierry Monteil added

comment:4 Changed 4 years ago by Michael Orlitzky

Cc: Michael Orlitzky added

comment:5 Changed 4 years ago by Timo Kaufmann

Cc: Timo Kaufmann added

comment:6 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:7 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:8 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

There is also #27373, which concerns packages that change build/run environment, e.g. mpir/gmp/mpfr, that should be resolved in some way.

comment:9 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:10 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:11 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:12 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:13 Changed 4 years ago by Erik Bray

Milestone: sage-8.7sage-pending

Removing most of the rest of my open tickets out of the 8.7 milestone, which should be closed.

comment:14 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:15 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:16 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:17 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:18 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:19 Changed 4 years ago by Dima Pasechnik

Cc: François Bissey added
Description: modified (diff)

comment:20 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:21 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:22 Changed 4 years ago by Michael Orlitzky

Description: modified (diff)

comment:23 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:24 Changed 4 years ago by Erik Bray

Description: modified (diff)

comment:25 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:26 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:27 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:28 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:29 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:30 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:31 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:32 Changed 4 years ago by Dima Pasechnik

Description: modified (diff)

comment:33 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:34 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:35 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:36 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:37 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:38 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:39 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:40 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:41 Changed 3 years ago by Isuru Fernando

Cc: Isuru Fernando added

comment:42 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:43 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:44 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:45 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:46 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:47 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:48 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:49 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:50 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:51 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:52 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:53 Changed 3 years ago by Jeroen Demeyer

Description: modified (diff)

comment:54 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:55 Changed 3 years ago by Marc Mezzarobba

Cc: Marc Mezzarobba added
Description: modified (diff)

comment:56 Changed 3 years ago by Marc Mezzarobba

Description: modified (diff)

comment:57 Changed 3 years ago by Marc Mezzarobba

Description: modified (diff)

comment:58 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:59 Changed 3 years ago by Erik Bray

Description: modified (diff)

comment:60 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:61 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:62 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:63 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:64 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:65 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:66 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:67 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:68 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:69 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:70 Changed 3 years ago by Erik Bray

Nice to see so much progress on this. I'm sorry I haven't had more time to help, but I'm glad to see that the infrastructure I put in place has continued to be useful as support for more packages is added.

comment:71 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:72 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:73 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:74 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:75 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:76 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:77 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:78 Changed 3 years ago by Matthias Köppe

Should we find a better way to keep track of the distribution packages that provide what spkg-configure is looking for?

Something autogenerated from new per-package information build/pkgs/* that can keep the section in the installation manual current (src/doc/en/installation/source.rst) - last updated by Dima?

Related: https://trac.sagemath.org/ticket/28745 (environment.yml for conda).

comment:79 in reply to:  78 Changed 3 years ago by Dima Pasechnik

Replying to mkoeppe:

Should we find a better way to keep track of the distribution packages that provide what spkg-configure is looking for?

Something autogenerated from new per-package information build/pkgs/* that can keep the section in the installation manual current (src/doc/en/installation/source.rst) - last updated by Dima?

A good idea, I only don't know how exactly to implement this. E.g. on Debian one could create a package with all dependencies (in fact, basically almost the same as their Sage package)

Related: https://trac.sagemath.org/ticket/28745 (environment.yml for conda).

comment:80 Changed 3 years ago by Matthias Köppe

I would just have thought to generate a shell script with one line of apt-get install.

comment:81 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:82 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:83 Changed 3 years ago by Matthias Köppe

Description: modified (diff)
Summary: Try to use as many system packages as possiblespkg-configure: Try to use as many system packages as possible

comment:84 Changed 3 years ago by Matthias Köppe

Summary: spkg-configure: Try to use as many system packages as possibleMeta-ticket: spkg-configure: Try to use as many system packages as possible

comment:85 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:86 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:87 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:88 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:89 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:90 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:91 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:92 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:93 Changed 3 years ago by Tobias Hansen

Cc: Tobias Hansen added

comment:94 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:95 Changed 3 years ago by Thierry Thomas

Cc: Tobias Hansen removed
Description: modified (diff)

rw is listed (#27814), but since it depends on igraph, it would be fine to add igraph too.

comment:96 Changed 3 years ago by Dima Pasechnik

Cc: Tobias Hansen added

comment:97 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:98 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:99 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:100 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:101 Changed 3 years ago by Michael Orlitzky

Description: modified (diff)

comment:102 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:103 Changed 3 years ago by Matthias Köppe

spkg-configure friends, I have drafted an update on the support for using system packages for the 9.1 release notes at https://wiki.sagemath.org/ReleaseTours/sage-9.1; let me know if anything needs changing or just make edits.

comment:104 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:105 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:106 Changed 3 years ago by Matthias Köppe

Should "BLAS" be moved to "done"?

comment:107 Changed 3 years ago by Dima Pasechnik

There should be a way to use other blas implementations from the system, not only openblas

comment:108 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:109 Changed 3 years ago by Thierry Thomas

Description: modified (diff)

Adding a reference for giac and pynac.

comment:110 Changed 3 years ago by Thierry Thomas

Description: modified (diff)

comment:111 Changed 3 years ago by Thierry Thomas

Description: modified (diff)

Adding tickets for libhomfly and linbox.

comment:112 Changed 3 years ago by Thierry Thomas

Description: modified (diff)

Adding #29644 for GAP.

comment:113 Changed 3 years ago by Thierry Thomas

Description: modified (diff)

Adding #29649 for zn_poly.

comment:114 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:115 Changed 3 years ago by Thierry Thomas

Description: modified (diff)

Adding cvxopt.

comment:116 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:117 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:118 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:119 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:120 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:121 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:122 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:123 Changed 3 years ago by Dima Pasechnik

Description: modified (diff)

comment:124 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:125 Changed 2 years ago by Dima Pasechnik

Description: modified (diff)

comment:126 Changed 2 years ago by Dima Pasechnik

Description: modified (diff)

comment:127 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:128 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:129 Changed 2 years ago by Dima Pasechnik

Description: modified (diff)

comment:130 Changed 2 years ago by Matthias Köppe

Description: modified (diff)
Milestone: sage-pendingsage-9.3

comment:131 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:132 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:133 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:134 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:135 Changed 2 years ago by Dima Pasechnik

Description: modified (diff)

comment:136 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:137 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:138 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:139 Changed 2 years ago by Dima Pasechnik

Description: modified (diff)

comment:140 Changed 2 years ago by Matthias Köppe

Keywords: sd111 added

comment:141 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:142 Changed 2 years ago by Dima Pasechnik

Description: modified (diff)

comment:143 Changed 2 years ago by Dima Pasechnik

Description: modified (diff)

comment:144 Changed 2 years ago by Michael Orlitzky

Description: modified (diff)

comment:145 Changed 2 years ago by Michael Orlitzky

Description: modified (diff)

comment:146 Changed 2 years ago by Tobias Diez

Cc: Tobias Diez added

comment:147 Changed 21 months ago by Dima Pasechnik

Description: modified (diff)

comment:148 Changed 21 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

comment:149 Changed 18 months ago by Dima Pasechnik

Description: modified (diff)

comment:150 Changed 17 months ago by Dima Pasechnik

Component: buildbuild: configure
Description: modified (diff)

comment:151 Changed 16 months ago by Matthias Köppe

Milestone: sage-9.4sage-9.5

comment:152 Changed 16 months ago by Dima Pasechnik

Description: modified (diff)

comment:153 Changed 16 months ago by Dima Pasechnik

Description: modified (diff)

comment:154 Changed 14 months ago by Matthias Köppe

Description: modified (diff)

comment:155 Changed 13 months ago by Michael Orlitzky

Description: modified (diff)

comment:156 Changed 12 months ago by Dima Pasechnik

Description: modified (diff)

comment:157 Changed 12 months ago by Matthias Köppe

Milestone: sage-9.5sage-9.6

comment:158 Changed 11 months ago by Samuel Lelièvre

Description: modified (diff)

comment:159 Changed 9 months ago by Dima Pasechnik

Description: modified (diff)

comment:160 Changed 8 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:161 Changed 3 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8
Note: See TracTickets for help on using tickets.