Opened 5 months ago
Closed 3 months ago
#34203 closed enhancement (fixed)
Find and uninstall broken installed SPKGs and wheels
Reported by:  Matthias Köppe  Owned by:  

Priority:  major  Milestone:  sage9.8 
Component:  build  Keywords:  
Cc:  Dima Pasechnik, John Palmieri, Nils Bruin, Emmanuel Charpentier  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  John Palmieri 
Report Upstream:  N/A  Work issues:  
Branch:  3c70a38 (Commits, GitHub, GitLab)  Commit:  3c70a38483a15327118ccfcaeabf154e5d9f2af1 
Dependencies:  #29097  Stopgaps: 
Description (last modified by )
When system packages are upgraded, shared libraries that a Sage installation links to can disappear, rendering the Sage installation broken. This is a common problem, reported for example in https://groups.google.com/g/sagedevel/c/Iz8ZsmQM3Pg/m/CPl9CbHmBgAJ
To avoid a full rebuild (make distclean && make build
), we can try to find the specific broken packages.
The new command make j listbrokenpackages
assists the user with this:
$ make j listbrokenpackages make noprintdirectory auditwheel_or_delocatenodeps sagelogger p 'sage pip install r "/Users/mkoeppe/s/sage/sagerebasing/build/pkgs/auditwheel_or_delocate/requirements.txt"' '/Users/mkoeppe/s/sage/sagerebasing/logs/pkgs/auditwheel_or_delocate.log' [auditwheel_or_delocate] Ignoring auditwheel: markers 'sys_platform != "darwin"' don't match your environment [auditwheel_or_delocate] Requirement already satisfied: delocate in /Users/mkoeppe/s/sage/sagerebasing/local/var/lib/sage/venvpython3.10/lib/python3.10/sitepackages (from r /Users/mkoeppe/s/sage/sagerebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.10.2) [auditwheel_or_delocate] Requirement already satisfied: wheel>=0.32.0 in /Users/mkoeppe/s/sage/sagerebasing/local/var/lib/sage/venvpython3.10/lib/python3.10/sitepackages (from delocate>r /Users/mkoeppe/s/sage/sagerebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.37.1) [auditwheel_or_delocate] Requirement already satisfied: typingextensions in /Users/mkoeppe/s/sage/sagerebasing/local/var/lib/sage/venvpython3.10/lib/python3.10/sitepackages (from delocate>r /Users/mkoeppe/s/sage/sagerebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (3.10.0.0) make[2]: warning: jN forced in submake: disabling jobserver mode. # Checking /Users/mkoeppe/s/sage/sagerebasing/local/var/lib/sage/installed/bliss0.73+debian1+sage20160802.p0 ... Checking shared library file '/Users/mkoeppe/s/sage/sagerebasing/local/lib/libumfpack.dylib' Checking shared library file '/Users/mkoeppe/s/sage/sagerebasing/local/var/tmp/sage/build/suitesparse5.10.1/src/lib/libsliplu.1.0.2.dylib' Error during installcheck of 'suitesparse': /Users/mkoeppe/s/sage/sagerebasing/local/var/tmp/sage/build/suitesparse5.10.1/src/lib/libsliplu.1.0.2.dylib Uninstall broken packages by typing: make anticSAGE_LOCALuninstall; make e_anticSAGE_LOCALuninstall; make lcalcSAGE_LOCALuninstall; make ratpointsSAGE_LOCALuninstall; make normalizSAGE_LOCALuninstall; make rSAGE_LOCALuninstall; make backports_zoneinfoSAGE_VENVuninstall; make fastjsonschemaSAGE_VENVuninstall; make pytz_deprecation_shimSAGE_VENVuninstall; make stack_dataSAGE_VENVuninstall; make tinycss2SAGE_VENVuninstall; make tzdataSAGE_VENVuninstall; make suitesparseSAGE_LOCALuninstall; real 2m25.787s user 4m54.270s sys 4m0.796s
The core functionality is implemented by the new command sagespkginstallcheck
(implemented in sage_bootstrap):
$ ./sage sh c 'python3 build/bin/sagespkginstallcheck verbose givaro' Checking shared library file 'lib/libgivaro.9.dylib' Checking shared library file 'lib/libgivaro.dylib' $ ./sage sh c 'python3 build/bin/sagespkginstallcheck verbose matplotlib $SAGE_VENV' Checking wheel file 'var/lib/sage/wheels/matplotlib3.5.1cp310cp310macosx_12_0_x86_64.whl' $ ./sage sh c 'python3 build/bin/sagespkginstallcheck verbose e_antic' Checking shared library file '/Users/mkoeppe/s/sage/sagerebasing/local/lib/libeantic.0.dylib' ERROR [libsanaget_dependencies:137]: /usr/local/opt/flint/lib/libflint16.dylib not found: Needed by: /Users/mkoeppe/s/sage/sagerebasing/local/lib/libeantic.0.dylib Error during installcheck of 'e_antic': Could not find all dependencies.
It inspects the installation records in $SAGE_LOCAL/var/lib/sage/installed/
and all shared libraries and platform wheels listed there.
On macOS, instead of using otool
directly, we use delocatelistdeps
from https://pypi.org/project/delocate/
On Linux, instead of using ldd
directly, we use auditwheel.lddtree.lddtree
(https://github.com/pypa/auditwheel/blob/main/src/auditwheel/lddtree.py)
(Both packages are also used by cibuildwheel)
See also: #34205 Optionally run auditwheel/delocate on wheels before installing them
Change History (64)
comment:1 Changed 5 months ago by
Description:  modified (diff) 

comment:2 Changed 5 months ago by
Description:  modified (diff) 

comment:3 Changed 5 months ago by
Description:  modified (diff) 

comment:4 Changed 5 months ago by
Description:  modified (diff) 

comment:5 Changed 5 months ago by
Branch:  → u/mkoeppe/find_and_uninstall_broken_installed_spkgs_and_wheels 

comment:6 Changed 5 months ago by
Commit:  → 562fcabd8dd959a08f5578d1cab088bbbfe05d79 

comment:7 Changed 5 months ago by
Description:  modified (diff) 

comment:8 Changed 5 months ago by
Commit:  562fcabd8dd959a08f5578d1cab088bbbfe05d79 → 5faa8cb70b1d864b97acea0c6333a7ab29a038de 

Branch pushed to git repo; I updated commit sha1. New commits:
5faa8cb  build/pkgs/auditwheel_or_delocate: New

comment:9 Changed 5 months ago by
Description:  modified (diff) 

comment:10 Changed 5 months ago by
Commit:  5faa8cb70b1d864b97acea0c6333a7ab29a038de → 92654161d6d22b1f057c468a3939ea69b4322a76 

Branch pushed to git repo; I updated commit sha1. New commits:
9265416  build/sage_bootstrap/installcheck.py: Implement for macOS

comment:11 Changed 5 months ago by
Description:  modified (diff) 

comment:12 Changed 5 months ago by
Commit:  92654161d6d22b1f057c468a3939ea69b4322a76 → c4e33715dc092f8fcd6323f7066e3db8dcf8e941 

comment:13 Changed 5 months ago by
Authors:  → Matthias Koeppe 

Description:  modified (diff) 
comment:14 Changed 5 months ago by
Description:  modified (diff) 

comment:15 Changed 5 months ago by
Commit:  c4e33715dc092f8fcd6323f7066e3db8dcf8e941 → 08388f7e218fb2be22bc9f3c1ab8ab0313a3cde3 

comment:16 Changed 5 months ago by
Description:  modified (diff) 

comment:18 Changed 5 months ago by
Commit:  08388f7e218fb2be22bc9f3c1ab8ab0313a3cde3 → f2dba639703f4e1bc27dc5d4c6c21813abd0a097 

Branch pushed to git repo; I updated commit sha1. New commits:
f2dba63  build/sage_bootstrap/uninstall.py: Do not refuse to uninstall packages that do not exist in the source tree

comment:19 Changed 5 months ago by
Commit:  f2dba639703f4e1bc27dc5d4c6c21813abd0a097 → b4394290c965ff06d7f498fa9812591fcc80e795 

comment:20 Changed 5 months ago by
Description:  modified (diff) 

comment:21 Changed 5 months ago by
Commit:  b4394290c965ff06d7f498fa9812591fcc80e795 → 5c104aab7a4747c1f8b1b5a49eddecfc73fdbf32 

comment:22 Changed 5 months ago by
Description:  modified (diff) 

comment:23 Changed 5 months ago by
Cc:  John Palmieri Nils Bruin added 

comment:24 Changed 5 months ago by
Commit:  5c104aab7a4747c1f8b1b5a49eddecfc73fdbf32 → 1d95bbb039683c3f2068006e59401147d0d3b9ad 

Branch pushed to git repo; I updated commit sha1. New commits:
1d95bbb  build/make/Makefile.in (listbrokenpackages): Fix up

comment:25 Changed 5 months ago by
Description:  modified (diff) 

comment:26 Changed 5 months ago by
Description:  modified (diff) 

comment:27 Changed 5 months ago by
Description:  modified (diff) 

Status:  new → needs_review 
comment:28 Changed 5 months ago by
Commit:  1d95bbb039683c3f2068006e59401147d0d3b9ad → 37eaa92eb303529f8896333c5a0e1209afd777fe 

Branch pushed to git repo; I updated commit sha1. New commits:
37eaa92  trac 29097: change "make SPKGclean" to "make SPKGuninstall".

comment:29 Changed 5 months ago by
Authors:  Matthias Koeppe → Matthias Koeppe, John Palmieri 

Cherrypicked / updated the commit from #29097.
comment:30 Changed 5 months ago by
Authors:  Matthias Koeppe, John Palmieri → Matthias Koeppe 

Dependencies:  → #29097 
comment:31 Changed 5 months ago by
Commit:  37eaa92eb303529f8896333c5a0e1209afd777fe → 1c24771c2cfa1d80c1571b5cfbc901c698fcb877 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
1458ee4  build/pkgs/auditwheel_or_delocate: New

8d1c6a4  build/sage_bootstrap/installcheck.py: Implement for macOS

320a290  build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check

f19ff62  build/sage_bootstrap/installcheck.py: Skip pure Python wheels

53a98fd  build/make/Makefile.in (listbrokenpackages): New

52bd73c  build/sage_bootstrap/installcheck.py: Only warn when delocate is not available

3bca243  build/make/Makefile.in (listbrokenpackages): Parallelize it

e00a064  build/make/Makefile.in (listbrokenpackages): Show uninstallation commands that actually work

980bbdb  build/make/Makefile.in (listbrokenpackages): Use targets %SAGE_LOCALuninstall, %SAGE_VENVuninstall when possible

1c24771  build/make/Makefile.in (listbrokenpackages): Fix up

comment:32 Changed 5 months ago by
Description:  modified (diff) 

comment:33 Changed 5 months ago by
Description:  modified (diff) 

comment:34 Changed 5 months ago by
This also finds a strange bug in our suitesparse
installation on macOS:
The installation record (local/var/lib/sage/installed/suitesparse5.10.1
) contains paths in the temporary build directory (var/tmp/sage/build/suitesparse5.10.1
).
I've opened #34206 for this
comment:35 Changed 5 months ago by
Description:  modified (diff) 

comment:36 Changed 5 months ago by
Commit:  1c24771c2cfa1d80c1571b5cfbc901c698fcb877 → 9d449bdf4237c0dfa4173515b0a164317fb0eb4c 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
24fb9ff  build/sage_bootstrap/installcheck.py: Implement for macOS

3a4d2ed  build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check

e36a56b  build/sage_bootstrap/installcheck.py: Skip pure Python wheels

bebc782  build/make/Makefile.in (listbrokenpackages): New

0924cf1  build/sage_bootstrap/installcheck.py: Only warn when delocate is not available

d5473ec  build/make/Makefile.in (listbrokenpackages): Parallelize it

75b1d96  build/make/Makefile.in (listbrokenpackages): Show uninstallation commands that actually work

9418c08  build/make/Makefile.in (listbrokenpackages): Use targets %SAGE_LOCALuninstall, %SAGE_VENVuninstall when possible

51d2d56  build/make/Makefile.in (listbrokenpackages): Fix up

9d449bd  build/sage_bootstrap/installclean.py: Clarify that the sage_local argument is SAGE_LOCAL or SAGE_VENV

comment:37 Changed 5 months ago by
Commit:  9d449bdf4237c0dfa4173515b0a164317fb0eb4c → 400eaee762e33cdc28c46ef7587d6f889c5cda22 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
00ec17b  build/sage_bootstrap/installcheck.py: Implement for macOS

b2033b9  build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check

eb47780  build/sage_bootstrap/installcheck.py: Skip pure Python wheels

30d9b5f  build/make/Makefile.in (listbrokenpackages): New

e6679a7  build/sage_bootstrap/installcheck.py: Only warn when delocate is not available

3d94488  build/make/Makefile.in (listbrokenpackages): Parallelize it

6881409  build/make/Makefile.in (listbrokenpackages): Show uninstallation commands that actually work

b5a7d05  build/make/Makefile.in (listbrokenpackages): Use targets %SAGE_LOCALuninstall, %SAGE_VENVuninstall when possible

6a73497  build/make/Makefile.in (listbrokenpackages): Fix up

400eaee  build/sage_bootstrap/installclean.py: Clarify that the sage_local argument is SAGE_LOCAL or SAGE_VENV

comment:38 Changed 5 months ago by
Description:  modified (diff) 

comment:39 Changed 5 months ago by
Description:  modified (diff) 

comment:40 Changed 5 months ago by
Commit:  400eaee762e33cdc28c46ef7587d6f889c5cda22 → ef9226bc1717bc579bdeddfb84f77c158a6ce516 

Branch pushed to git repo; I updated commit sha1. New commits:
ef9226b  build/sage_bootstrap/installcheck.py: Implement for Linux

comment:41 Changed 5 months ago by
Description:  modified (diff) 

comment:42 Changed 4 months ago by
Hi Nils, I developed this ticket as a solution for last time that you complained about people breaking your build. Care to review?
comment:43 Changed 3 months ago by
Cc:  Emmanuel Charpentier added 

comment:44 followups: 45 46 Changed 3 months ago by
Can you explain what "package" means in listbrokenpackages
? I ran the command and saw
usage: sagespkginstallcheck [h] [v] spkg [sage_local] sagespkginstallcheck: error: argument spkg: 'stack_data' is not a known spkg
followed by the suggestion to run make stack_dataSAGE_VENVuninstall
. Is stack_data
actually broken, or is the "usage" message indication of something else going wrong? And what is stack_data
, since it's not an spkg?
comment:45 Changed 3 months ago by
comment:46 Changed 3 months ago by
Replying to John Palmieri:
Is
stack_data
actually broken, or is the "usage" message indication of something else going wrong? And what isstack_data
, since it's not an spkg?
It's broken as follows: You have an installation of it (from some other branch), but it's not (per what is currently in build/pkgs
) an SPKG.
comment:47 Changed 3 months ago by
In 9.7.rc1, stack_data
of course is a package, so after merging 9.7.rc1, this will no longer be reported as "broken".
comment:48 Changed 3 months ago by
Commit:  ef9226bc1717bc579bdeddfb84f77c158a6ce516 → 0bbba440e3342d12e6b705011ea27874e7b0e9a3 

Branch pushed to git repo; I updated commit sha1. New commits:
0bbba44  Merge tag '9.7.rc1' into t/34203/find_and_uninstall_broken_installed_spkgs_and_wheels

comment:50 Changed 3 months ago by
Commit:  0bbba440e3342d12e6b705011ea27874e7b0e9a3 → 9a03d7fe7beb1bae71415938f7cb3ef9417c9fb1 

Branch pushed to git repo; I updated commit sha1. New commits:
9a03d7f  build/sage_bootstrap/installcheck.py: Make error message more specific

comment:51 Changed 3 months ago by
Should this have a nonzero return code if some packages are broken?
comment:52 Changed 3 months ago by
From its name listbrokenpackages
, I tend to say that it should not indicate a failure like that
comment:54 Changed 3 months ago by
It is possible that some could want to do make j listbrokenpackages && do something else
, but I don't see a concrete example.
comment:55 Changed 3 months ago by
Yes, I thought about this, but I'd say that this whole thing is a strictly manual operation, which doesn't lend itself to scripting.
comment:56 followup: 59 Changed 3 months ago by
For the new package auditwheel_or_delocate
, it seems that potentially both are used on Linux, only delocate
on OS X. If that's right, then the SPKG.rst
should be modified.
Installing auditwheel
via pip also installs pyelftools
, and patchelf
is listed as a dependency on the PyPI page. Do we need to create a new package pyelftools
, too? Is patchelf
really required, or is it only necessary if one is trying to fix files, not just detect whether they're broken?
comment:57 Changed 3 months ago by
The comment "and delete the rest" does not seem to be accurate (and I think it's fine not to delete the old stamp files here).
comment:58 Changed 3 months ago by
Commit:  9a03d7fe7beb1bae71415938f7cb3ef9417c9fb1 → 6ceb44edbd932b7c772d3844c74103e2e3d6732c 

Branch pushed to git repo; I updated commit sha1. New commits:
6ceb44e  build/pkgs/auditwheel_or_delocate/SPKG.rst: Explain that we use delocate also on Linux

comment:59 Changed 3 months ago by
Replying to John Palmieri:
Installing
auditwheel
via pip also installspyelftools
, andpatchelf
is listed as a dependency on the PyPI page. Do we need to create a new packagepyelftools
, too?
Not needed  that's what makes optional "pip" packages so convenient.
Is
patchelf
really required, or is it only necessary if one is trying to fix files, not just detect whether they're broken?
Right, it's only needed for repairing wheels. But it's best not try to circumvent the declared dependencies.
The scipy upgrade (#34081) will bring a patchelf, by the way.
comment:60 Changed 3 months ago by
Commit:  6ceb44edbd932b7c772d3844c74103e2e3d6732c → 3c70a38483a15327118ccfcaeabf154e5d9f2af1 

Branch pushed to git repo; I updated commit sha1. New commits:
3c70a38  build/sage_bootstrap/installcheck.py: Update comment on stamp files

comment:61 Changed 3 months ago by
Reviewers:  → John Palmieri 

Status:  needs_review → positive_review 
Great, I think this is ready to go now.
comment:63 Changed 3 months ago by
Milestone:  sage9.7 → sage9.8 

comment:64 Changed 3 months ago by
Branch:  u/mkoeppe/find_and_uninstall_broken_installed_spkgs_and_wheels → 3c70a38483a15327118ccfcaeabf154e5d9f2af1 

Resolution:  → fixed 
Status:  positive_review → closed 
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
build/bin/sagespkginstallcheck, build/sage_bootstrap/installcheck.py: New