Opened 5 months ago

Closed 3 months ago

#34203 closed enhancement (fixed)

Find and uninstall broken installed SPKGs and wheels

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.8
Component: build Keywords:
Cc: Dima Pasechnik, John Palmieri, Nils Bruin, Emmanuel Charpentier Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: 3c70a38 (Commits, GitHub, GitLab) Commit: 3c70a38483a15327118ccfcaeabf154e5d9f2af1
Dependencies: #29097 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

When system packages are upgraded, shared libraries that a Sage installation links to can disappear, rendering the Sage installation broken. This is a common problem, reported for example in https://groups.google.com/g/sage-devel/c/Iz8ZsmQM3Pg/m/CPl9CbHmBgAJ

To avoid a full rebuild (make distclean && make build), we can try to find the specific broken packages.

The new command make -j list-broken-packages assists the user with this:

$ make -j list-broken-packages
make --no-print-directory auditwheel_or_delocate-no-deps
sage-logger -p 'sage --pip install -r "/Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt"' '/Users/mkoeppe/s/sage/sage-rebasing/logs/pkgs/auditwheel_or_delocate.log'
[auditwheel_or_delocate] Ignoring auditwheel: markers 'sys_platform != "darwin"' don't match your environment
[auditwheel_or_delocate] Requirement already satisfied: delocate in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from -r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.10.2)
[auditwheel_or_delocate] Requirement already satisfied: wheel>=0.32.0 in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.37.1)
[auditwheel_or_delocate] Requirement already satisfied: typing-extensions in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (3.10.0.0)
make[2]: warning: -jN forced in submake: disabling jobserver mode.
# Checking /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/installed/bliss-0.73+debian-1+sage-2016-08-02.p0
...
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libumfpack.dylib'
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib'
Error during installcheck of 'suitesparse': /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib
Uninstall broken packages by typing:


    make antic-SAGE_LOCAL-uninstall;
    make e_antic-SAGE_LOCAL-uninstall;
    make lcalc-SAGE_LOCAL-uninstall;
    make ratpoints-SAGE_LOCAL-uninstall;
    make normaliz-SAGE_LOCAL-uninstall;
    make r-SAGE_LOCAL-uninstall;
    make backports_zoneinfo-SAGE_VENV-uninstall;
    make fastjsonschema-SAGE_VENV-uninstall;
    make pytz_deprecation_shim-SAGE_VENV-uninstall;
    make stack_data-SAGE_VENV-uninstall;
    make tinycss2-SAGE_VENV-uninstall;
    make tzdata-SAGE_VENV-uninstall;
    make suitesparse-SAGE_LOCAL-uninstall;

real	2m25.787s
user	4m54.270s
sys	4m0.796s

The core functionality is implemented by the new command sage-spkg-installcheck (implemented in sage_bootstrap):

$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose givaro'
Checking shared library file 'lib/libgivaro.9.dylib'
Checking shared library file 'lib/libgivaro.dylib'
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose matplotlib $SAGE_VENV'
Checking wheel file 'var/lib/sage/wheels/matplotlib-3.5.1-cp310-cp310-macosx_12_0_x86_64.whl'
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose e_antic'
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib'
ERROR [libsana|get_dependencies:137]: 
/usr/local/opt/flint/lib/libflint-16.dylib not found:
  Needed by: /Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib
Error during installcheck of 'e_antic': Could not find all dependencies.

It inspects the installation records in $SAGE_LOCAL/var/lib/sage/installed/ and all shared libraries and platform wheels listed there.

On macOS, instead of using otool directly, we use delocate-listdeps from https://pypi.org/project/delocate/

On Linux, instead of using ldd directly, we use auditwheel.lddtree.lddtree (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/lddtree.py)

(Both packages are also used by cibuildwheel)

See also: #34205 Optionally run auditwheel/delocate on wheels before installing them

Change History (64)

comment:1 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:2 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:3 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:4 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:5 Changed 5 months ago by Matthias Köppe

Branch: u/mkoeppe/find_and_uninstall_broken_installed_spkgs_and_wheels

comment:6 Changed 5 months ago by git

Commit: 562fcabd8dd959a08f5578d1cab088bbbfe05d79

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

562fcabbuild/bin/sage-spkg-installcheck, build/sage_bootstrap/installcheck.py: New

comment:7 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:8 Changed 5 months ago by git

Commit: 562fcabd8dd959a08f5578d1cab088bbbfe05d795faa8cb70b1d864b97acea0c6333a7ab29a038de

Branch pushed to git repo; I updated commit sha1. New commits:

5faa8cbbuild/pkgs/auditwheel_or_delocate: New

comment:9 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:10 Changed 5 months ago by git

Commit: 5faa8cb70b1d864b97acea0c6333a7ab29a038de92654161d6d22b1f057c468a3939ea69b4322a76

Branch pushed to git repo; I updated commit sha1. New commits:

9265416build/sage_bootstrap/installcheck.py: Implement for macOS

comment:11 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:12 Changed 5 months ago by git

Commit: 92654161d6d22b1f057c468a3939ea69b4322a76c4e33715dc092f8fcd6323f7066e3db8dcf8e941

Branch pushed to git repo; I updated commit sha1. New commits:

a18b205build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check
ad92af4build/sage_bootstrap/installcheck.py: Skip pure Python wheels
c4e3371build/make/Makefile.in (list-broken-packages): New

comment:13 Changed 5 months ago by Matthias Köppe

Authors: Matthias Koeppe
Description: modified (diff)

comment:14 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:15 Changed 5 months ago by git

Commit: c4e33715dc092f8fcd6323f7066e3db8dcf8e94108388f7e218fb2be22bc9f3c1ab8ab0313a3cde3

Branch pushed to git repo; I updated commit sha1. New commits:

dbdf33bbuild/sage_bootstrap/installcheck.py: Only warn when delocate is not available
08388f7build/make/Makefile.in (list-broken-packages): Parallelize it

comment:16 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:17 Changed 5 months ago by Matthias Köppe

Ready for testing on macOS.

comment:18 Changed 5 months ago by git

Commit: 08388f7e218fb2be22bc9f3c1ab8ab0313a3cde3f2dba639703f4e1bc27dc5d4c6c21813abd0a097

Branch pushed to git repo; I updated commit sha1. New commits:

f2dba63build/sage_bootstrap/uninstall.py: Do not refuse to uninstall packages that do not exist in the source tree

comment:19 Changed 5 months ago by git

Commit: f2dba639703f4e1bc27dc5d4c6c21813abd0a097b4394290c965ff06d7f498fa9812591fcc80e795

Branch pushed to git repo; I updated commit sha1. New commits:

a2437fcbuild/sage_bootstrap/uninstall.py: Do not override the stamp file location from environment variable
b439429build/make/Makefile.in (list-broken-packages): Show uninstallation commands that actually work

comment:20 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:21 Changed 5 months ago by git

Commit: b4394290c965ff06d7f498fa9812591fcc80e7955c104aab7a4747c1f8b1b5a49eddecfc73fdbf32

Branch pushed to git repo; I updated commit sha1. New commits:

cd7312cbuild/make/Makefile.in: New implicit targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall
5c104aabuild/make/Makefile.in (list-broken-packages): Use targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall when possible

comment:22 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:23 Changed 5 months ago by Matthias Köppe

Cc: John Palmieri Nils Bruin added

comment:24 Changed 5 months ago by git

Commit: 5c104aab7a4747c1f8b1b5a49eddecfc73fdbf321d95bbb039683c3f2068006e59401147d0d3b9ad

Branch pushed to git repo; I updated commit sha1. New commits:

1d95bbbbuild/make/Makefile.in (list-broken-packages): Fix up

comment:25 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:26 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:27 Changed 5 months ago by Matthias Köppe

Description: modified (diff)
Status: newneeds_review

comment:28 Changed 5 months ago by git

Commit: 1d95bbb039683c3f2068006e59401147d0d3b9ad37eaa92eb303529f8896333c5a0e1209afd777fe

Branch pushed to git repo; I updated commit sha1. New commits:

37eaa92trac 29097: change "make SPKG-clean" to "make SPKG-uninstall".

comment:29 Changed 5 months ago by Matthias Köppe

Authors: Matthias KoeppeMatthias Koeppe, John Palmieri

Cherry-picked / updated the commit from #29097.

comment:30 Changed 5 months ago by Matthias Köppe

Authors: Matthias Koeppe, John PalmieriMatthias Koeppe
Dependencies: #29097

comment:31 Changed 5 months ago by git

Commit: 37eaa92eb303529f8896333c5a0e1209afd777fe1c24771c2cfa1d80c1571b5cfbc901c698fcb877

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

1458ee4build/pkgs/auditwheel_or_delocate: New
8d1c6a4build/sage_bootstrap/installcheck.py: Implement for macOS
320a290build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check
f19ff62build/sage_bootstrap/installcheck.py: Skip pure Python wheels
53a98fdbuild/make/Makefile.in (list-broken-packages): New
52bd73cbuild/sage_bootstrap/installcheck.py: Only warn when delocate is not available
3bca243build/make/Makefile.in (list-broken-packages): Parallelize it
e00a064build/make/Makefile.in (list-broken-packages): Show uninstallation commands that actually work
980bbdbbuild/make/Makefile.in (list-broken-packages): Use targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall when possible
1c24771build/make/Makefile.in (list-broken-packages): Fix up

comment:32 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:33 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:34 Changed 5 months ago by Matthias Köppe

This also finds a strange bug in our suitesparse installation on macOS: The installation record (local/var/lib/sage/installed/suitesparse-5.10.1) contains paths in the temporary build directory (var/tmp/sage/build/suitesparse-5.10.1). I've opened #34206 for this

Last edited 5 months ago by Matthias Köppe (previous) (diff)

comment:35 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:36 Changed 5 months ago by git

Commit: 1c24771c2cfa1d80c1571b5cfbc901c698fcb8779d449bdf4237c0dfa4173515b0a164317fb0eb4c

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

24fb9ffbuild/sage_bootstrap/installcheck.py: Implement for macOS
3a4d2edbuild/sage_bootstrap/installcheck.py: Do not warn if there are no files to check
e36a56bbuild/sage_bootstrap/installcheck.py: Skip pure Python wheels
bebc782build/make/Makefile.in (list-broken-packages): New
0924cf1build/sage_bootstrap/installcheck.py: Only warn when delocate is not available
d5473ecbuild/make/Makefile.in (list-broken-packages): Parallelize it
75b1d96build/make/Makefile.in (list-broken-packages): Show uninstallation commands that actually work
9418c08build/make/Makefile.in (list-broken-packages): Use targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall when possible
51d2d56build/make/Makefile.in (list-broken-packages): Fix up
9d449bdbuild/sage_bootstrap/installclean.py: Clarify that the sage_local argument is SAGE_LOCAL or SAGE_VENV

comment:37 Changed 5 months ago by git

Commit: 9d449bdf4237c0dfa4173515b0a164317fb0eb4c400eaee762e33cdc28c46ef7587d6f889c5cda22

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

00ec17bbuild/sage_bootstrap/installcheck.py: Implement for macOS
b2033b9build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check
eb47780build/sage_bootstrap/installcheck.py: Skip pure Python wheels
30d9b5fbuild/make/Makefile.in (list-broken-packages): New
e6679a7build/sage_bootstrap/installcheck.py: Only warn when delocate is not available
3d94488build/make/Makefile.in (list-broken-packages): Parallelize it
6881409build/make/Makefile.in (list-broken-packages): Show uninstallation commands that actually work
b5a7d05build/make/Makefile.in (list-broken-packages): Use targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall when possible
6a73497build/make/Makefile.in (list-broken-packages): Fix up
400eaeebuild/sage_bootstrap/installclean.py: Clarify that the sage_local argument is SAGE_LOCAL or SAGE_VENV

comment:38 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:39 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:40 Changed 5 months ago by git

Commit: 400eaee762e33cdc28c46ef7587d6f889c5cda22ef9226bc1717bc579bdeddfb84f77c158a6ce516

Branch pushed to git repo; I updated commit sha1. New commits:

ef9226bbuild/sage_bootstrap/installcheck.py: Implement for Linux

comment:41 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:42 Changed 4 months ago by Matthias Köppe

Hi Nils, I developed this ticket as a solution for last time that you complained about people breaking your build. Care to review?

comment:43 Changed 3 months ago by Matthias Köppe

Cc: Emmanuel Charpentier added

comment:44 Changed 3 months ago by John Palmieri

Can you explain what "package" means in list-broken-packages? I ran the command and saw

usage: sage-spkg-installcheck [-h] [-v] spkg [sage_local]
sage-spkg-installcheck: error: argument spkg: 'stack_data' is not a known spkg

followed by the suggestion to run make stack_data-SAGE_VENV-uninstall. Is stack_data actually broken, or is the "usage" message indication of something else going wrong? And what is stack_data, since it's not an spkg?

comment:45 in reply to:  44 Changed 3 months ago by Matthias Köppe

Replying to John Palmieri:

Can you explain what "package" means in list-broken-packages?

An SPKG

comment:46 in reply to:  44 Changed 3 months ago by Matthias Köppe

Replying to John Palmieri:

Is stack_data actually broken, or is the "usage" message indication of something else going wrong? And what is stack_data, since it's not an spkg?

It's broken as follows: You have an installation of it (from some other branch), but it's not (per what is currently in build/pkgs) an SPKG.

comment:47 Changed 3 months ago by Matthias Köppe

In 9.7.rc1, stack_data of course is a package, so after merging 9.7.rc1, this will no longer be reported as "broken".

comment:48 Changed 3 months ago by git

Commit: ef9226bc1717bc579bdeddfb84f77c158a6ce5160bbba440e3342d12e6b705011ea27874e7b0e9a3

Branch pushed to git repo; I updated commit sha1. New commits:

0bbba44Merge tag '9.7.rc1' into t/34203/find_and_uninstall_broken_installed_spkgs_and_wheels

comment:49 Changed 3 months ago by Matthias Köppe

And here's that merge commit.

comment:50 Changed 3 months ago by git

Commit: 0bbba440e3342d12e6b705011ea27874e7b0e9a39a03d7fe7beb1bae71415938f7cb3ef9417c9fb1

Branch pushed to git repo; I updated commit sha1. New commits:

9a03d7fbuild/sage_bootstrap/installcheck.py: Make error message more specific

comment:51 Changed 3 months ago by John Palmieri

Should this have a nonzero return code if some packages are broken?

comment:52 Changed 3 months ago by Matthias Köppe

From its name list-broken-packages, I tend to say that it should not indicate a failure like that

comment:53 Changed 3 months ago by John Palmieri

I think I agree but I wanted to ask.

comment:54 Changed 3 months ago by John Palmieri

It is possible that some could want to do make -j list-broken-packages && do something else, but I don't see a concrete example.

Last edited 3 months ago by John Palmieri (previous) (diff)

comment:55 Changed 3 months ago by Matthias Köppe

Yes, I thought about this, but I'd say that this whole thing is a strictly manual operation, which doesn't lend itself to scripting.

comment:56 Changed 3 months ago by John Palmieri

For the new package auditwheel_or_delocate, it seems that potentially both are used on Linux, only delocate on OS X. If that's right, then the SPKG.rst should be modified.

Installing auditwheel via pip also installs pyelftools, and patchelf is listed as a dependency on the PyPI page. Do we need to create a new package pyelftools, too? Is patchelf really required, or is it only necessary if one is trying to fix files, not just detect whether they're broken?

comment:57 Changed 3 months ago by John Palmieri

The comment "and delete the rest" does not seem to be accurate (and I think it's fine not to delete the old stamp files here).

comment:58 Changed 3 months ago by git

Commit: 9a03d7fe7beb1bae71415938f7cb3ef9417c9fb16ceb44edbd932b7c772d3844c74103e2e3d6732c

Branch pushed to git repo; I updated commit sha1. New commits:

6ceb44ebuild/pkgs/auditwheel_or_delocate/SPKG.rst: Explain that we use delocate also on Linux

comment:59 in reply to:  56 Changed 3 months ago by Matthias Köppe

Replying to John Palmieri:

Installing auditwheel via pip also installs pyelftools, and patchelf is listed as a dependency on the PyPI page. Do we need to create a new package pyelftools, too?

Not needed - that's what makes optional "pip" packages so convenient.

Is patchelf really required, or is it only necessary if one is trying to fix files, not just detect whether they're broken?

Right, it's only needed for repairing wheels. But it's best not try to circumvent the declared dependencies.

The scipy upgrade (#34081) will bring a patchelf, by the way.

comment:60 Changed 3 months ago by git

Commit: 6ceb44edbd932b7c772d3844c74103e2e3d6732c3c70a38483a15327118ccfcaeabf154e5d9f2af1

Branch pushed to git repo; I updated commit sha1. New commits:

3c70a38build/sage_bootstrap/installcheck.py: Update comment on stamp files

comment:61 Changed 3 months ago by John Palmieri

Reviewers: John Palmieri
Status: needs_reviewpositive_review

Great, I think this is ready to go now.

comment:62 Changed 3 months ago by Matthias Köppe

Thank you!

comment:63 Changed 3 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8

comment:64 Changed 3 months ago by Volker Braun

Branch: u/mkoeppe/find_and_uninstall_broken_installed_spkgs_and_wheels3c70a38483a15327118ccfcaeabf154e5d9f2af1
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.