lots of warnings when uninstalling spkgs

[jhpalmieri]

When upgrading Sage from 8.3.rc2 to 8.4.beta0: mpir-3.0.0-644faf502c56f97d9accd301965fc57d6ec70868.p0.log:

Uninstalling existing 'mpir'
Warning: File '/share/info/dir' not found
Warning: Directory '/share/info' not found
Warning: File '/share/info/mpir.info' not found
Warning: Directory '/share/info' not found
Warning: File '/share/info/mpir.info-1' not found
Warning: Directory '/share/info' not found
Warning: File '/share/info/mpir.info-2' not found
Warning: Directory '/share/info' not found
Warning: File '/include/gmp.h' not found
Warning: Directory '/include' not found
Warning: File '/include/gmpxx.h' not found
Warning: Directory '/include' not found
Warning: File '/include/mpir.h' not found
Warning: Directory '/include' not found
Warning: File '/include/mpirxx.h' not found
Warning: Directory '/include' not found
Warning: File '/lib/libgmp.23.dylib' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libgmp.a' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libgmp.dylib' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libgmpxx.8.dylib' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libgmpxx.a' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libgmpxx.dylib' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libmpir.23.dylib' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libmpir.a' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libmpir.dylib' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libmpirxx.8.dylib' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libmpirxx.a' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libmpirxx.dylib' not found
Warning: Directory '/lib' not found

mpc-1.1.0.log:

Uninstalling existing 'mpc'
Warning: File '/share/info/dir' not found
Warning: Directory '/share/info' not found
Warning: File '/share/info/mpc.info' not found
Warning: Directory '/share/info' not found
Warning: File '/include/mpc.h' not found
Warning: Directory '/include' not found
Warning: File '/lib/libmpc.3.dylib' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libmpc.a' not found
Warning: Directory '/lib' not found
Warning: File '/lib/libmpc.dylib' not found
Warning: Directory '/lib' not found

and similarly in many other log files. As far as I can tell, files are not being uninstalled before the new versions are built. Is the problem just that the path names should have SAGE_LOCAL prepended?

[jdemeyer]

Replying to jhpalmieri:

Is the problem just that the path names should have SAGE_LOCAL prepended?

Indeed. However, it works for me.

[jhpalmieri]

It happens on two different machines. Maybe something is broken. I added some print statements, and they do not clarify things. Here is the change I made:

build/sage_bootstrap/uninstall.py

@@ -187 +187 @@
     # Remove the files; if a directory is empty after removing a file
     # from it, remove the directory too.
     for filename in files:
+        print("  * sage_local = {}".format(sage_local))
+        print("  * orig filename = {}".format(filename))
         filename = pth.join(sage_local, filename)
+        print("  ** new filename = {}".format(filename))
         dirname = pth.dirname(filename)
         if os.path.exists(filename):
             if verbose:

Here is typical output:

Uninstalling existing 'glpk'
  * sage_local = /Users/jpalmier/Desktop/Sage_stuff/git/sage/local
  * orig filename = /bin/glpsol
  ** new filename = /bin/glpsol

I don't understand.

[jhpalmieri]

Oh, I do understand. From the ​documentation for os.path.join:

If a component is an absolute path, all previous components are thrown away and joining continues from the absolute path component.

So we need to strip os.sep from the left end of filename.

[jhpalmieri]

New commits:

​f1aa849	trac 26022: "![ -z ...]" is not valid, at least on OS X
​b31446a	trac 26013: use filename.lstrip(os.sep) instead of filename

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​40e0cfb

trac 26013: apply lstrip(os.sep) to filename in os.path.join(..., filename)

[embray]

Some other OSX users have pointed this issue out before, but I could never figure out why it was happening. Let me look into this a bit more for a sec because there shouldn't be absolute paths in those filenames in the first place. They should be written as relative paths. For example I have:

$ cat local/var/lib/sage/installed/mpc-1.1.0
{
    "package_name": "mpc",
    "package_version": "1.1.0",
    "install_date": "Thu Jul 19 08:10:31 UTC 2018",
    "system_uname": "Linux sage-docker 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux",
    "sage_version": "SageMath version 8.3.rc1, Release Date: 2018-07-14",
    "test_result": "",
    "files": [
        "include/mpc.h",
        "lib/libmpc.a",
        "lib/libmpc.so",
        "lib/libmpc.so.3",
        "lib/libmpc.so.3.1.0",
        "share/info/dir",
        "share/info/mpc.info"
    ]
}

Do you see absolute paths being written into those files?

[embray]

Unfortunately the OSX machine I used to have access to appears to be down, and everyone here is on vacation...

[embray]

The relevant code is in sage-spkg (yet more reason this should probably be rewritten in Python as it is growing in complexity, though as Jeroen wrote elsewhere it would be a tedious task...). A summarized version is as follows:

PREFIX="${SAGE_DESTDIR_LOCAL%/}/"
old_IFS="$IFS"; IFS=$'\n'
for filename in $(find "$PREFIX" -type f -o -type l | sort); do
    filename="${filename#$PREFIX}"
    echo "$filename"
done

in the above, $PREFIX should end with a /, and ${filename#$PREFIX} should have the effect of stripping $PREFIX (up to and including the final /) from the front of $filename. But maybe there's a bug or some other subtlety with that I was unaware of.

[embray]

Another way maybe it could fail is if your $SAGE_LOCAL somehow had a superfluous double-slash at the end of it...?

[embray]

If nothing else, something like this should do the trick. But I think I will be rewriting a chunk of this code anyways...

build/bin/sage-spkg

@@ -889 +889 @@
     old_IFS="$IFS"; IFS=$'\n'
     for filename in $(find "$PREFIX" -type f -o -type l | sort); do
         filename="${filename#$PREFIX}"
+        # Make really sure there are still no leading slashes on the filename
+        # (which could happen if enough superfluous slashes were added
+        # elsewhere)
+        while [[ "$filename" == /* ]]; do
+            filename="${filename#/}"
+        done
+
         if [ $FIRST -eq 1 ]; then
             FILE_LIST="\"$filename\""
             FIRST=0

[jhpalmieri]

I added echo "$filename" in sage-spkg, and there are indeed leading backslashes. My $SAGE_LOCAL has no trailing backslashes (at least when I do sage -sh and echo $SAGE_LOCAL), and the $PREFIX variable in sage-spkg has one. As you probably know, bash behavior can vary depending on whether it's GNU/linux or BSD. Maybe that's what's going on. Is there an issue with just using lstrip(os.sep) as my branch does?

[embray]

I'd be curious if you could figure out exactly which part is behaving differently in your bash? What happens with filename="${filename#$PREFIX}. Not even necessarily in this script, but just in general? What does something like:

PREFIX=/a/
filename=/a/b
filename=${filename#$PREFIX}
echo "$filename"

If that returns anything other than just b this is a bug in your bash I think.

Is there an issue with just using lstrip(os.sep) as my branch does?

The point is that shouldn't even be necessary. It might not be a bad idea anyways just as a sanity check, though I might also issue a warning. The intent is that these files contain a list of relative paths. If they don't then the files are malformatted, which means the real problem is in the code producing those files (whether the bug ultimately originates from a buggy bash or not).

[embray]

Alternatively, it's also possible that your find is slipping in some superfluous slashes.

[embray]

This seems to suggest that it is in fact the find at fault: ​https://apple.stackexchange.com/a/254224/44101

[embray]

This patch should do it:

build/bin/sage-spkg

@@ -869 +869 @@
 # case DESTDIR=$SAGE_DESTDIR installation was not used
 echo "Copying package files from temporary location $SAGE_DESTDIR to $SAGE_LOCAL"
 if [ -d "$SAGE_DESTDIR" ]; then
-    PREFIX="${SAGE_DESTDIR_LOCAL%/}/"
+    # Some `find` implementations will put superfluous slashes in the
+    # output if we give them a directory name with a slash; so make sure
+    # any trailing slash is removed; https://trac.sagemath.org/ticket/26013
+    PREFIX="${SAGE_DESTDIR_LOCAL%/}"
 
-    rm -f "$PREFIX"lib/*.la
+    rm -f "$PREFIX"/lib/*.la
     if [ $? -ne 0 ]; then
         error_msg "Error deleting unnecessary libtool archive files"
         exit 1
@@ -882 +885 @@
     FIRST=1
     old_IFS="$IFS"; IFS=$'\n'
     for filename in $(find "$PREFIX" -type f -o -type l | sort); do
-        filename="${filename#$PREFIX}"
+        filename="${filename#$PREFIX/}"
         if [ $FIRST -eq 1 ]; then
             FILE_LIST="\"$filename\""
             FIRST=0
@@ -893 +896 @@
         if [ ! -d "$SAGE_LOCAL/$(dirname "$filename")" ]; then
             $SAGE_SUDO mkdir -p "$SAGE_LOCAL/$(dirname "$filename")"
         fi
-        $SAGE_SUDO mv "$PREFIX$filename" "${SAGE_LOCAL%/}/$filename"
+        $SAGE_SUDO mv "$PREFIX/$filename" "${SAGE_LOCAL%/}/$filename"
         if [ $? -ne 0 ]; then
             error_msg "Error moving files for $PKG_NAME."
             exit 1

I'm fine with adding this to your existing patch to uninstall.py as long a comment is also added pointing back to this ticket (and assuming this patch does in fact work).

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​48f7cdc

trac 26013: make sure that sage-spkg generates relative filenames for

[jhpalmieri]

Your patch works for me. Here is a new branch.

[embray]

Thanks!