Opened 12 months ago

Last modified 3 months ago

#29711 new enhancement

sagelib setup.py: Dependencies on header files of packages

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.4
Component: build Keywords:
Cc: dimpase, charpent, mjo, gh-kliem Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

Rebuilds should be triggered when headers of relevant packages change.

In setup.py (after #29702, sage_setup.command.sage_build_cython) we have:

# Do not put all, but only the most common libraries and their headers
# (that are likely to change on an upgrade) here:
# [At least at the moment. Make sure the headers aren't copied with "-p",
# or explicitly touch them in the respective spkg's spkg-install.]
lib_headers = { "gmp":     [ os.path.join(SAGE_INC, 'gmp.h') ],   # cf. #8664, #9896
                "gmpxx":   [ os.path.join(SAGE_INC, 'gmpxx.h') ],
                "ntl":     [ os.path.join(SAGE_INC, 'NTL', 'config.h') ]
              }

Except for this (which is much too limited and also needs updating in light of the spkg-configure mechanism), we do not have dependency tracking for sagelib. Neither of distutils, setuptools, cython seem to have a mechanism for generating dependencies like gcc -M. This would be certainly be nice to have; in particular if it included dependencies on "system" headers and libraries, so that sagelib could automatically rebuild when system headers/libraries change (as in, most recently, ​https://groups.google.com/d/msg/sage-release/A53tGGsJNLg/uA01oUugAQAJ).

(from #29411)

Change History (11)

comment:1 Changed 12 months ago by mkoeppe

  • Description modified (diff)

comment:2 follow-up: Changed 12 months ago by mjo

What you really want to do here is rebuild sagelib whenever the public API of a dependency changes in a backwards-incompatible way. We devised "subslots" in Gentoo for the purpose of telling the package manager when it needs to rebuild. The documentation looks complicated but the basic idea is that we're copy/pasting the major component of the soname into the ebuild. When it changes, the package manager knows to rebuild the things that depend on it.

Rebuilding whenever any header of any dependency changes is an OK way to fake it, but it's going to trigger unnecessary sagelib rebuilds frequently.

comment:3 Changed 11 months ago by mkoeppe

  • Description modified (diff)

comment:4 in reply to: ↑ 2 ; follow-up: Changed 11 months ago by mkoeppe

Replying to mjo:

What you really want to do here is rebuild sagelib whenever the public API of a dependency changes in a backwards-incompatible way. We devised "subslots" in Gentoo for the purpose of telling the package manager when it needs to rebuild. The documentation looks complicated

... indeed, from a quick look it seems there's a lot of Gentoo jargon there...

but the basic idea is that we're copy/pasting the major component of the soname into the ebuild. When it changes, the package manager knows to rebuild the things that depend on it.

Thanks for the pointer. It's probably a little bit more complicated for us because if I'm not mistaken, upstream packages are not always very careful with their sonames (which is then fixed in distributions)

comment:5 Changed 11 months ago by mkoeppe

The update of these paths, necessary because of the spkg-configure mechanism for GMP and NTL could use the environment variables set in sage-build-env-config.

comment:6 follow-up: Changed 11 months ago by mkoeppe

I'll do this in #29847.

comment:7 in reply to: ↑ 6 Changed 11 months ago by mkoeppe

Replying to mkoeppe:

I'll do this in #29847.

... or rather #29855.

comment:8 in reply to: ↑ 4 Changed 11 months ago by mjo

Replying to mkoeppe:

Thanks for the pointer. It's probably a little bit more complicated for us because if I'm not mistaken, upstream packages are not always very careful with their sonames (which is then fixed in distributions)

In hindsight, it wouldn't really work for sage anyway. Since the (gentoo) developer makes up the subslot, he's able to change the subslot when the upstream API changes even if upstream doesn't change their soname. But rebuild-on-subslot-change only works for us because we know that all packages will be installed from the same repository. With sage, some packages may come from the system, and then you won't have access to that information. So rebuilding every time the headers change might be the best we can do.

comment:9 Changed 11 months ago by mkoeppe

  • Cc gh-kliem added

See also #29791 comment 6

comment:10 Changed 9 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:11 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

Note: See TracTickets for help on using tickets.