Opened 3 years ago

Last modified 6 months ago

#22509 needs_work enhancement

Install packages in temporary root before copying to $SAGE_LOCAL

Reported by: embray Owned by:
Priority: major Milestone: sage-8.9
Component: build Keywords:
Cc: mkoeppe Merged in:
Authors: Erik Bray Reviewers:
Report Upstream: N/A Work issues:
Branch: u/embray/build/destdir (Commits) Commit: c1d5453c2735cdda78d53dc36b56fb54ec7dd336
Dependencies: #23059 #23096 #23160 #23781 Stopgaps:

Description (last modified by embray)

Currently we build packages, in most cases, by setting --prefix=$SAGE_LOCAL (in ./configure, or the equivalent in non-autoconf packages), and then installing directly to that prefix.

This is fine to keep as is. But with automake packages we can also set DESTDIR (https://www.gnu.org/software/automake/manual/html_node/DESTDIR.html#DESTDIR) to an alternative (typically temporary) directory into which to install. This has a couple advantages:

  • Although it's not as usual for make install to fail after a successful make, it can happen. This prevents messy partial installs in the case of a failed make install run.
  • It gives us the opportunity to make an exact list of the files that were installed. This is a prerequisite to improving package uninstallation/reinstallation in sage-dist (see #22510).

This is also standard operating procedure in most other packaging systems, so it would be good for Sage to adopt too. The general approach is the same as that used for converting an arbitrary Makefile to support this convention--everywhere a file is installed, prefix the installation path with $(DESTDIR) (or in this case a variable we'll call $SAGE_DESTDIR).

For packages that already support the DESTDIR convention, then, we pass DESTDIR=$SAGE_DESTDIR to make install. It is also already possible for Python packages (e.g. pip install --root).

Since the introduction of the build helper scripts in #23160, it makes the most sense to make these changes directly in the helper scripts, and then update more packages to use those helper scripts where possible.

Dependent tickets:

  • #24106 - implement the basic functionality of supporting staged installs through sage-spkg
  • #24024 - update more spkg-build/install scripts to use the sdh_ helper functions, since that is the fastest way to add DESTDIR support for many packages.
  • #24646 - add support for --root=${SAGE_DESTDIR} to Python spkgs
  • #24645 - add support for post-install scripts; this can be useful for cases where, for example, local files are generated after the package is installed, or to handle cases where one file is shared by multiple packages and needs to be updated when a package that uses it is installed (see e.g. https://trac.sagemath.org/ticket/22510#comment:5)
  • #????? - convert all spkgs to support staged-install; for many packages this can be covered by using the helper functions, but some packages require additional steps; in particular, packages that have files that are manually copied into $SAGE_LOCAL, rather than through a standard installer

Change History (110)

comment:1 Changed 3 years ago by mkoeppe

  • Cc mkoeppe added

comment:2 Changed 3 years ago by embray

  • Description modified (diff)
  • Milestone changed from sage-7.6 to sage-wishlist

comment:3 Changed 3 years ago by embray

  • Authors set to Erik Bray
  • Branch set to u/embray/build/destdir
  • Commit set to a25a42a02a89a20058d41d95c56efc1cd0875f26
  • Dependencies set to #23096
  • Status changed from new to needs_review

Started work on this, and converted I think all standard packages to work with a staged installation (install first to a temporary directory, then copy from the temp directory into $SAGE_LOCAL).

As I wrote in the description, most packages are easy--either they use automake, or at least a Makefile that's compatible with the DESTDIR convention, or they are Python packages in which case the --root option does the same thing. A handful of packages needed hand-tweaking.

This is currently implemented in such a way that if a package doesn't yet support staged install, it will still install just fine. The only major downside is that the installed files are not recorded yet, so they wouldn't be able to take advantage of improved uninstall.

That said, I'm going to continue converting the remaining packages. My only question is whether to continue work in this ticket, or separate the general changes (i.e. to sage-spkg) from the updates to the packages themselves.


Last 10 new commits:

3fa088aUpdate pip install to use --root
30aad2fUpdate freetype and pkgconf to use DESTDIR
b1aabaaUpdated various packages with non-standard installers or that are otherwise
223608fUpdate cephes to work with staged install.
27878c3Update openblas to use staged install
ecd214cUpdate lcalc to support staged install
3d64db8Update zn_poly, which doesn't really have a proper installer, to use staged install
609393aRoll back accidental change from dc46ea2dfa97be79d7ef8ddbe9130d4c717acc3d that broke build of Python 2 on Cygwin.
03424e1Merge remote-tracking branch 'upstream/u/embray/cygwin/ticket-23059' into u/embray/build/destdir
a25a42aUpdated the python2/3 pacakges to use staged install.

comment:4 Changed 3 years ago by embray

  • Dependencies changed from #23096 to #23059 #23096

comment:5 Changed 3 years ago by embray

By way of explanation, the idea here is that there is a variable exported by sage-spkg called $SAGE_INST_TEMP (the name is open to bikeshedding), so this is expected to be passed to each spkg-install to use an alternative install root. In principle this could be defined in each spkg-install individually, but it's less trouble to standardize in one place, and in any case sage-spkg still needs to know this path to copy files from it to $SAGE_LOCAL.

Of course, if we take this approach, the developer docs will also need to be updated with some guidance on how to implement staged installation for new packages.

comment:6 Changed 3 years ago by git

  • Commit changed from a25a42a02a89a20058d41d95c56efc1cd0875f26 to 8a621af7a5f3e10d1337c07b42d04edee6a8f1b8

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3a32187Roll back accidental change from dc46ea2dfa97be79d7ef8ddbe9130d4c717acc3d that broke build of Python 2 on Cygwin.
8a621afUpdated the python2/3 pacakges to use staged install.

comment:7 Changed 3 years ago by git

  • Commit changed from 8a621af7a5f3e10d1337c07b42d04edee6a8f1b8 to 1391089b36f43b45fa0eb7bf04e83fc1e351dfea

Branch pushed to git repo; I updated commit sha1. New commits:

1391089Remove the '-v' from the 'cp' command.

comment:8 Changed 3 years ago by git

  • Commit changed from 1391089b36f43b45fa0eb7bf04e83fc1e351dfea to e2941230aea00481d6cf1f58992faa006a7b63f2

Branch pushed to git repo; I updated commit sha1. New commits:

e294123Add staged install support for atlas

comment:9 Changed 3 years ago by embray

  • Description modified (diff)
Last edited 3 years ago by embray (previous) (diff)

comment:10 Changed 3 years ago by embray

  • Description modified (diff)

comment:11 Changed 3 years ago by git

  • Commit changed from e2941230aea00481d6cf1f58992faa006a7b63f2 to 00f6de8d173a25b865c26c6af3f588b49e8401eb

Branch pushed to git repo; I updated commit sha1. New commits:

00f6de8This file should be deleted from the temp install dir during staged install on OSX

comment:12 Changed 3 years ago by git

  • Commit changed from 00f6de8d173a25b865c26c6af3f588b49e8401eb to 48e9ffbf5fe86d3186a505ae1e9ab1b13c8195f2

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

16bc700Update freetype and pkgconf to use DESTDIR
eb660c7Updated various packages with non-standard installers or that are otherwise
8de16d2Update cephes to work with staged install.
73c45e6Update openblas to use staged install
e1ad6d5Update lcalc to support staged install
f07e056Update zn_poly, which doesn't really have a proper installer, to use staged install
9997a37Updated the python2/3 pacakges to use staged install.
f9cfa6fRemove the '-v' from the 'cp' command.
e51f578Add staged install support for atlas
48e9ffbThis file should be deleted from the temp install dir during staged install on OSX

comment:13 follow-up: Changed 3 years ago by jdemeyer

  • Milestone changed from sage-wishlist to sage-8.1
  • Status changed from needs_review to needs_work

This already fails on the first package... patch:

Making install in src
make[5]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/src'
make[6]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/src'
 /bin/mkdir -p '/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/inst/opt/sage/sage-git/local/bin'
  /usr/bin/install -c patch '/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/inst/opt/sage/sage-git/local/bin'
make[6]: Nothing to be done for `install-data-am'.
make[6]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/src'
make[5]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/src'
Making install in tests
make[5]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/tests'
make[6]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/tests'
make[6]: Nothing to be done for `install-exec-am'.
make[6]: Nothing to be done for `install-data-am'.
make[6]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/tests'
make[5]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/tests'
make[5]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[6]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[6]: Nothing to be done for `install-exec-am'.
 /bin/mkdir -p '/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/inst/opt/sage/sage-git/local/share/man/man1'
 /usr/bin/install -c -m 644 'patch.man' '/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/inst/opt/sage/sage-git/local/share/man/man1/patch.1'
make[6]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[5]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[4]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[3]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
Cannot find the patch program we just installed

comment:14 follow-ups: Changed 3 years ago by jdemeyer

I see ${SAGE_INST_TEMP}${SAGE_LOCAL} occuring in various places, maybe you should define a new variable for this.

comment:15 follow-up: Changed 3 years ago by jdemeyer

Let me say that this a very brave patch :-)

comment:16 in reply to: ↑ 14 ; follow-up: Changed 3 years ago by embray

Replying to jdemeyer:

I see ${SAGE_INST_TEMP}${SAGE_LOCAL} occuring in various places, maybe you should define a new variable for this.

Yeah, I keep thinking about that. At first I was just going to redefine $SAGE_LOCAL to that, but that doesn't work because one needs the bare $SAGE_LOCAL, e.g., for the prefix. I think I just need a good name for the combined path. Maybe $SAGE_LOCAL_TEMP?

comment:17 in reply to: ↑ 15 Changed 3 years ago by embray

Replying to jdemeyer:

Let me say that this a very brave patch :-)

Thanks! I just hope you agree with it in principle, because I really think it should be done. Open to ideas on the specific details, although I think this is already pretty close. There's a lot of things we could be doing better packaging-wise in the first place of course, but I'm still just trying to make the least disruptive changes I can.

comment:18 in reply to: ↑ 16 ; follow-up: Changed 3 years ago by jdemeyer

Replying to embray:

Maybe $SAGE_LOCAL_TEMP?

I would prefer $SAGE_TEMP_LOCAL simply because that's consistent with the order (first SAGE_INST_TEMP, then SAGE_LOCAL).

comment:19 Changed 3 years ago by embray

That's fine. But before I do a mass rename on this, perhaps you (and/or others) could have a look at #23160. It would be very nice to have this, or something like it, as a dependency on this ticket so that I can make a few bulk changes once (in the case of #23160 it would be replacing all calls to $MAKE/make install with sdh_make_install, and then many future tweaks can be made in one place--the definition of sdh_make_install).

comment:20 Changed 3 years ago by git

  • Commit changed from 48e9ffbf5fe86d3186a505ae1e9ab1b13c8195f2 to 7bcbccb7a1b223017eb45d334cb097f365632c48

Branch pushed to git repo; I updated commit sha1. New commits:

d794f97Updates the autotools spkg to use staged install
7bcbccbFix installation of files containing spaces in their names

comment:21 Changed 3 years ago by git

  • Commit changed from 7bcbccb7a1b223017eb45d334cb097f365632c48 to 9eb5876c6793718ba6225aaba0e07fcf71eab432

Branch pushed to git repo; I updated commit sha1. New commits:

9eb5876Remove this sanity check

comment:22 in reply to: ↑ 13 Changed 3 years ago by embray

Replying to jdemeyer:

This already fails on the first package... patch:

Just a peculiarity of that package's spkg-install. I removed the seemingly pointless check for now.

comment:23 Changed 3 years ago by git

  • Commit changed from 9eb5876c6793718ba6225aaba0e07fcf71eab432 to 5f7d07967411a4c2ebdf2a837f6427bc210a761e

Branch pushed to git repo; I updated commit sha1. New commits:

5f7d079Fix sagenb installation

comment:24 Changed 3 years ago by git

  • Commit changed from 5f7d07967411a4c2ebdf2a837f6427bc210a761e to 23799c8b38066712b67354916bba6c265b4f15d2

Branch pushed to git repo; I updated commit sha1. New commits:

23799c8Actually fix sagenb install

comment:25 Changed 2 years ago by git

  • Commit changed from 23799c8b38066712b67354916bba6c265b4f15d2 to 84a6827229bbaa3a26154b1202d4f12fe4acad43

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

f30f964Remove this sanity check
637d136Fix sagenb installation
03b50feActually fix sagenb install
2095d1dMake sure the path these symlinks are installed in exists first.
48304f2Some more enhancements to the sage-dist-helpers:
5f0637dInclude the PKG_NAME, PKG_BASE, and PKG_VER variables directly in the install
b5633e1Now that sdh_make_install implements DESTDIR we can replace all manual
f41ab62A few more standard autotools packages that are easily updated but I missed previously
06e7339Disable pip installation when installing Python 3; see https://trac.sagemath.org/ticket/23398
84a6827Don't override the valgrind suppression file from python2; add a separate one for Python 3

comment:26 Changed 2 years ago by embray

  • Dependencies changed from #23059 #23096 to #23059 #23096 #23160

Now depends on #23160. Updates a bunch of packages to use the helper functions--once that change is made once it will be much easier to make mass updates simply by updating the helper functions.

comment:27 Changed 2 years ago by git

  • Commit changed from 84a6827229bbaa3a26154b1202d4f12fe4acad43 to 0885622958923d3f9f00725076b6553e10b8de4b

Branch pushed to git repo; I updated commit sha1. New commits:

0885622Handle the 2to3 script better so that Python2 and Python3 don't overwrite each other

comment:28 in reply to: ↑ 18 ; follow-up: Changed 2 years ago by embray

Replying to jdemeyer:

Replying to embray:

Maybe $SAGE_LOCAL_TEMP?

I would prefer $SAGE_TEMP_LOCAL simply because that's consistent with the order (first SAGE_INST_TEMP, then SAGE_LOCAL).

Actually, I think I'm going to rename $SAGE_INST_TEMP to $SAGE_DESTDIR for consistency with the DESTDIR convention. I think that makes it clear what it's for. Then the $SAGE_DESTDIR$SAGE_LOCAL combination can be called $SAGE_DESTDIR_LOCAL.

comment:29 Changed 2 years ago by git

  • Commit changed from 0885622958923d3f9f00725076b6553e10b8de4b to 62e916b7cda969e7798fb096672923a2c049fa10

Branch pushed to git repo; I updated commit sha1. New commits:

57ea8c9Added the ability for packages to have a post-install script called spkg-postinst.
76703a4Add special post-install handling for the backports/__init__.py shared
be4f47aRemove some no-longer used local variables
62e916bAdd handling for packages that install .info files and update share/info/dir

comment:30 in reply to: ↑ 14 Changed 2 years ago by embray

Replying to jdemeyer:

I see ${SAGE_INST_TEMP}${SAGE_LOCAL} occuring in various places, maybe you should define a new variable for this.

Actually on second thought, there was a reason for this: The idea was that if $SAGE_INST_TEMP isn't set then installation straight into $SAGE_LOCAL will still work. The way this works currently is $SAGE_INST_TEMP is set by sage-spkg which handles the details of how to copy from the temporary install path into SAGE_LOCAL, and record the file manifest.

But if you're just running a package's spkg-install directly, then $SAGE_INST_TEMP is not set. With #23179 in place I could make it so that $SAGE_INST_TEMP is always set in the script wrappers. But I'm not necessarily sure that's desirable, since if you're running the script outside of sage-spkg the implementation details of staged installation are not meaningful, and it may be more useful to install directly into SAGE_LOCAL assuming it's for testing purposes. Of course, one can always manually pass in a value for $SAGE_INST_TEMP.

So I think it would be better *not* to combine $SAGE_INST_TEMP$SAGE_LOCAL into a single variable, unless others are convinced that there is no use case for installing without $SAGE_INST_TEMP set.

(I do still want to rename it $SAGE_DESTDIR--in this way it is used quite analogously to the way $(DESTDIR) is used in Makefiles.)

comment:31 Changed 2 years ago by embray

If it helps makes things more readable I could use the curly-brace syntax like ${SAGE_INST_TEMP}${SAGE_LOCAL} rather then running them directly together.

comment:32 Changed 2 years ago by git

  • Commit changed from 62e916b7cda969e7798fb096672923a2c049fa10 to 01c05f0c3bdd164d07846c72324e15f897328d6b

Branch pushed to git repo; I updated commit sha1. New commits:

01c05f0Rename $SAGE_INST_TEMP to $SAGE_DESTDIR

comment:33 follow-up: Changed 2 years ago by embray

  • Status changed from needs_work to needs_review

I think that, pending #23160, this is basically ready. I've tested it quite extensively myself on Linux and Cygwin and haven't encountered any more obvious problems.

The only exception would be that there are still some packages that write files that clobber each other, but they are only cases of optional/experimental packages clobbering standard packages, or optional/experimental packages clobbering each other. These include:

  • openblas / atlas
  • mpir / gmp
  • boost_cropped / boost
  • pari_seadata_small / database_pari
  • database_stein_watkins_small / database_stein_watkins
  • gcc / compilerwrapper

However, these incompatibilities were already the status quo. While it will be possible in the future to improve the status quo (#23465 is work in that direction), I think these cases can be fixed independently of this ticket. This ticket otherwise fixes all examples among standard (and some optional packages) where files were overwriting each other.

Last edited 2 years ago by embray (previous) (diff)

comment:34 Changed 2 years ago by jhpalmieri

  • Status changed from needs_review to needs_work

On OS X, I get this with the patch package:

cp: illegal option -- u
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
cp: illegal option -- u
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
Successfully installed patch-2.7.5

A little later the same error causes yasm not to be installed, so mpir won't install:

[mpir-3.0.0.p0] configure: error: no system-wide yasm found

In addition to fixing the cp command in sage-spkg, should you also check for errors after executing commands like this so as to avoid "Successfully installed patch-2.7.5" when it didn't actually successfully install?

comment:36 Changed 2 years ago by embray

Thanks for the quick feedback! I'll have to take a look at the OSX manpage 😞. This is fairly simple functionality to duplicate.

comment:37 Changed 2 years ago by embray

Honestly, the -u isn't even necessary. In most cases it's going to be superfluous so I'd probably just remove it.

comment:38 Changed 2 years ago by git

  • Commit changed from 01c05f0c3bdd164d07846c72324e15f897328d6b to d9500366634b7eb5b24eea91dbf5fd78a50848e3

Branch pushed to git repo; I updated commit sha1. New commits:

d950036Don't use the -u option to cp. It's not portable to BSD's cp, and is

comment:39 Changed 2 years ago by embray

  • Status changed from needs_work to needs_review

comment:40 Changed 2 years ago by jhpalmieri

  • Status changed from needs_review to needs_work

ncurses fails to install for me (on OS X):

Copying package files from temporary location /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.rc0/local/var/tmp/sage/build/ncurses-6.0/inst to /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.rc0/local
cp: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.rc0/local/var/tmp/sage/build/ncurses-6.0/inst/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.rc0/local//lib/terminfo is a directory (not copied).
************************************************************************
Error copying files for ncurses.
************************************************************************

comment:41 Changed 2 years ago by embray

Do you know the correct way to copy a symbolic link without dereferencing it on OSX? It should be -P, but according to that man page it only respects -P if -R is used also. But this is copying files one at a time. I could use -R but would still have to loop over all the files anyways in order to build the manifest. Also that man page says:

In -R mode, cp will continue copying even if errors are detected.

which isn't necessarily what I want either.

comment:42 Changed 2 years ago by git

  • Commit changed from d9500366634b7eb5b24eea91dbf5fd78a50848e3 to d605943e4ee29256ea30180803b96aa6a2d6917c

Branch pushed to git repo; I updated commit sha1. New commits:

d605943Just use 'cp -a' which seems to be compatible across most cp's.

comment:43 Changed 2 years ago by embray

  • Status changed from needs_work to needs_review
Last edited 2 years ago by embray (previous) (diff)

comment:44 Changed 2 years ago by embray

It occurs to me that the pkgconf package isn't being handled quite correctly. This leads to problems when re-installing / upgrading that package.

Nevermind, for the purposes of this ticket it's fine. It's only a problem in #22510.

Last edited 2 years ago by embray (previous) (diff)

comment:45 Changed 2 years ago by jhpalmieri

Now python2 and python3 don't install:

Installing valgrind suppression file...
./spkg-install: line 140: ./python: is a directory
Testing importing of various modules...
./spkg-install: line 162: ./python: is a directory
ctypes module failed to import
./spkg-install: line 162: ./python: is a directory
math module failed to import
./spkg-install: line 162: ./python: is a directory
hashlib module failed to import
./spkg-install: line 162: ./python: is a directory
crypt module failed to import
./spkg-install: line 162: ./python: is a directory
readline module failed to import
./spkg-install: line 162: ./python: is a directory
socket module failed to import
./spkg-install: line 171: ./python: is a directory
_scproxy module failed to import
Error: One or more modules failed to import.

comment:46 follow-up: Changed 2 years ago by embray

Does OSX have case-insensitivity or something?

comment:47 Changed 2 years ago by embray

Or in other words, if I have in the same directory an executable called python and a directory called Python (as is the case in the CPython source tree) what will happen if I run ./python on OSX?

comment:48 Changed 2 years ago by embray

Coincidentally, I just discovered that this also fixes a rare race condition that results from the fact that multiple packages write to $SAGE_LOCAL/usr/share/info/dir. If two packages that write to that file are installed in parallel there are probably a number of ways that can go wrong, such as (what just happened to me):

A) [ ! -d $(DESTDIR)$info_dir ] ...
B) mkdir $(DESTDIR)$info_dir
A) ... || mkdir $(DESTDIR)$info_dir

resulting in directory already exists error.

Last edited 2 years ago by embray (previous) (diff)

comment:49 in reply to: ↑ 46 Changed 2 years ago by embray

Replying to embray:

Does OSX have case-insensitivity or something?

It turns out Python's ./configure explicitly checks this. Does the build log say something like

checking for case-insensitive build directory...yes

?

comment:50 Changed 2 years ago by embray

Indeed, it looks like on any OS (including OSX) if a case-insensitive filesystem is being built on it outputs the Python executable in the build dir as python.exe so we just need to look for that I guess.

comment:51 Changed 2 years ago by git

  • Commit changed from d605943e4ee29256ea30180803b96aa6a2d6917c to dbdb94788f6275e19c5fd150bec8a889188f3164

Branch pushed to git repo; I updated commit sha1. New commits:

dbdb947Fix Python build on non-Cygwin systems (namely OSX) with case-insensitive filesystems

comment:52 Changed 2 years ago by embray

Give that a try. I can't imagine there's any other reason for this.

comment:53 Changed 2 years ago by jhpalmieri

Yes, OS X has a type of case-insensitivity. The current branch gets a little farther, but still fails. With python2:

Installing valgrind suppression file...
Testing importing of various modules...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/Lib/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: dlopen(/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_ctypes.so, 2): Symbol not found: _PyUnicodeUCS4_AsEncodedString
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_ctypes.so
  Expected in: flat namespace
 in /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_ctypes.so
ctypes module failed to import
math module imported OK
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/Lib/hashlib.py", line 158, in <module>
    import struct
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/Lib/struct.py", line 1, in <module>
    from _struct import *
ImportError: dlopen(/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_struct.so, 2): Symbol not found: _PyUnicodeUCS4_AsEncodedString
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_struct.so
  Expected in: flat namespace
 in /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_struct.so
hashlib module failed to import
crypt module imported OK
readline module imported OK
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/Lib/socket.py", line 47, in <module>
    import _socket
ImportError: dlopen(/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_socket.so, 2): Symbol not found: _PyUnicodeUCS4_AsEncodedString
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_socket.so
  Expected in: flat namespace
 in /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python2-2.7.13.p1/src/build/lib.macosx-10.9-x86_64-2.7/_socket.so
socket module failed to import
_scproxy module imported OK
Error: One or more modules failed to import.

It's odd that some of the modules succeeded while others failed.

With python3:

Installing valgrind suppression file...
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
Testing importing of various modules...
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
./spkg-install: line 160: 84992 Abort trap: 6           $PYTHON -c "import $module"
ctypes module failed to import
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
./spkg-install: line 160: 84993 Abort trap: 6           $PYTHON -c "import $module"
math module failed to import
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
./spkg-install: line 160: 84995 Abort trap: 6           $PYTHON -c "import $module"
hashlib module failed to import
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
./spkg-install: line 160: 84997 Abort trap: 6           $PYTHON -c "import $module"
crypt module failed to import
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
./spkg-install: line 160: 84998 Abort trap: 6           $PYTHON -c "import $module"
readline module failed to import
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
./spkg-install: line 160: 85000 Abort trap: 6           $PYTHON -c "import $module"
socket module failed to import
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/lib/libpython3.6m.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/python3-3.6.1/src/./python.exe
  Reason: image not found
./spkg-install: line 176: 85001 Abort trap: 6           $PYTHON -c "import _scproxy"
_scproxy module failed to import
Error: One or more modules failed to import.

comment:54 Changed 2 years ago by embray

I see. This probably means that some DYLD_LIBRARY_PATH or similar needs to be set in order for Python being run out of the source tree to link with the correct libpython.

Thanks for your testing / patience--I need to bother nthiery to get me an OSX machine I can test on so that this will be easier in the future.

Other than this Python thing I don't anticipate too many other issues with this. The only reason Python is being stubborn is that I made a somewhat non-trivial change to how the Python self-tests are performed. Very few of the other packages otherwise have significant changes to how they're installed (though a few do).

comment:55 Changed 2 years ago by jhpalmieri

Python has also always been finicky to install on OS X. Anyway, imitating what was there already for linux works on this machine: adding

if [ "$UNAME" = "Darwin" ]; then
   export DYLD_LIBRARY_PATH="."
fi

right before echo "Testing importing of various modules...", for example, fixed both python2 and python3.

Other issues: this one is strange and non-repeatable, but I saw this once with maxima:

Found local metadata for maxima-5.39.0.p0
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
ImportError: No module named site
************************************************************************
Error downloading maxima-5.39.0.tar.gz
************************************************************************

In this case, the maxima tarball had already been downloaded to the upstream directory. When I ran "make" again, it went smoothly. I don't know if this is worth investigating.

More importantly, I get build errors with gsl and numpy. For gsl:

checking build system type... x86_64-apple-darwin16.7.0
checking host system type... x86_64-apple-darwin16.7.0
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... configure: error: in `/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta0/local/var/tmp/sage/build/gsl-2.3/src':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details

config.log says

configure:3480: checking whether the C compiler works
configure:3502: gcc -g -O2   -L/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib  conftest.c -lopenblas   -lm >&5
configure:3506: $? = 0
configure:3554: result: yes
configure:3557: checking for C compiler default output file name
configure:3559: result: a.out
configure:3565: checking for suffix of executables
configure:3572: gcc -o conftest -g -O2   -L/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib  conftest.c -lopenblas   -lm >&5
configure:3576: $? = 0
configure:3598: result: 
configure:3620: checking whether we are cross compiling
configure:3628: gcc -o conftest -g -O2   -L/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib  conftest.c -lopenblas   -lm >&5
configure:3632: $? = 0
configure:3639: ./conftest
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/openblas-0.2.19.p0/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/libopenblas_haswell-r0.2.19.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/gsl-2.3/src/./conftest
  Reason: image not found
./configure: line 3641:  5891 Abort trap: 6           ./conftest$ac_cv_exeext
configure:3643: $? = 134
configure:3650: error: in `/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/gsl-2.3/src':
configure:3652: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.

I don't know why it seems to be looking for libopenblas_haswell-r0.2.19.dylib in the temporary directory; the command invoking ./configure has the correct --prefix and --libdir.

For numpy:

Copying numpy.egg-info to /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/numpy-1.12.1-py2.7.egg-info
running install_scripts
creating /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/bin
copying build/scripts.macosx-10.9-x86_64-2.7/f2py2 -> /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/bin
changing mode of /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/bin/f2py2 to 755
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy-1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 24, in <module>
    raise ImportError(msg)
ImportError: 
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

comment:56 Changed 2 years ago by embray

I don't get any of these problems. I will probably need an OSX machine to investigate.

It could have something to do with the

LIBS="`pkg-config --libs-only-l cblas` -lm"

passed to configure in gsl's spkg-install. But that *shouldn't* be a problem.

comment:57 Changed 2 years ago by git

  • Commit changed from dbdb94788f6275e19c5fd150bec8a889188f3164 to ce38d0756f226ec9a87b55890e716edd80f0cbdc

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

4360f31Added the ability for packages to have a post-install script called spkg-postinst.
0972a41Add special post-install handling for the backports/__init__.py shared
41c73a3Remove some no-longer used local variables
750943aAdd handling for packages that install .info files and update share/info/dir
53a9c47Rename $SAGE_INST_TEMP to $SAGE_DESTDIR
7615714Don't use the -u option to cp. It's not portable to BSD's cp, and is
d4bb6f7Just use 'cp -a' which seems to be compatible across most cp's.
ed89571Fix Python build on non-Cygwin systems (namely OSX) with case-insensitive filesystems
44228b2On OSX DYLD_LIBRARY_PATH should be set a la LD_LIBRARY_PATH to run Python from its build directory
ce38d07Work around bug in OpenBLAS's Makefile with its use of DESTDIR with

comment:58 Changed 2 years ago by embray

Rebased on the latest (positively reviewed) #23160.

I added the DYLD_LIBRARY_PATH fix needed for Python installation.

Also added an untested possible fix for the gsl installation issue. I think the issue here was a bug in OpenBLAS's Makefile on OSX with how it uses install_name_tool, and this should fix it but I'm not positive. Chances are this would also fix the problem with numpy which is probably trying to link (unsuccessfully) to openblas.

comment:59 Changed 2 years ago by jhpalmieri

This works now, Sage builds and all tests pass.

comment:60 Changed 2 years ago by jhpalmieri

Several packages fail their test suites. flint:

gcc -fno-common -ansi -pedantic -Wall -O2 -funroll-loops -g -mpopcnt    -c -o test_helpers.o test_helpers.c
make  build/test/t-add_ssaaaa  build/test/t-add_sssaaaaaa  build/test/t-count_leading_zeros  build/test/t-count_trailing_zeros  build/test/t-sdiv_qrnnd  build/test/t-smul_ppmm  build/test/t-sub_ddmmss  build/test/t-udiv_qrnnd  build/test/t-udiv_qrnnd_preinv  build/test/t-umul_ppmm
mkdir -p build/test
gcc -fno-common -ansi -pedantic -Wall -O2 -funroll-loops -g -mpopcnt  -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/flint-2.5.2.p1/src -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib  test/t-add_ssaaaa.c -o build/test/t-add_ssaaaa -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/flint-2.5.2.p1/src -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -lflint -lmpfr -lgmp -lm -lntl -lpthread 

[snip]

gcc -fno-common -ansi -pedantic -Wall -O2 -funroll-loops -g -mpopcnt  -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/flint-2.5.2.p1/src -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib  test/t-udiv_qrnnd_preinv.c -o build/test/t-udiv_qrnnd_preinv -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/flint-2.5.2.p1/src -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -lflint -lmpfr -lgmp -lm -lntl -lpthread 
gcc -fno-common -ansi -pedantic -Wall -O2 -funroll-loops -g -mpopcnt  -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/flint-2.5.2.p1/src -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/include -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib  test/t-umul_ppmm.c -o build/test/t-umul_ppmm -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/flint-2.5.2.p1/src -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib -lflint -lmpfr -lgmp -lm -lntl -lpthread 
(BUILD_DIR=build; export BUILD_DIR; make -f Makefile.subdirs -C . check || exit$?;)
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib/libflint-13.5.2.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/flint-2.5.2.p1/src/build/test/t-add_ssaaaa
  Reason: image not found
make[4]: *** [build/test/t-add_ssaaaa_RUN] Abort trap: 6
/bin/sh: exit2: command not found
make[3]: *** [check] Error 127
Error: FLINT failed to pass its test suite.

arb:

Running the test suite for arb-2.8.1.p1...
make[4]: Nothing to be done for `shared'.
make[5]: Nothing to be done for `shared'.
    CC   ../build/fmpr/test/t-abs_bound_le_2exp_fmpz
    CC   ../build/fmpr/test/t-abs_bound_lt_2exp_fmpz
    CC   ../build/fmpr/test/t-abs_bound_lt_2exp_si
    CC   ../build/fmpr/test/t-add
    CC   ../build/fmpr/test/t-add_naive
    CC   ../build/fmpr/test/t-cmp
    CC   ../build/fmpr/test/t-cmp_2exp_si
    CC   ../build/fmpr/test/t-cmpabs
    CC   ../build/fmpr/test/t-cmpabs_2exp_si
    CC   ../build/fmpr/test/t-div
    CC   ../build/fmpr/test/t-divappr_abs_ubound
    CC   ../build/fmpr/test/t-exp
    CC   ../build/fmpr/test/t-expm1
    CC   ../build/fmpr/test/t-get_d
    CC   ../build/fmpr/test/t-get_fmpz
    CC   ../build/fmpr/test/t-get_mpfr
    CC   ../build/fmpr/test/t-log
    CC   ../build/fmpr/test/t-log1p
    CC   ../build/fmpr/test/t-mul
    CC   ../build/fmpr/test/t-mul_fmpz
    CC   ../build/fmpr/test/t-mul_naive
    CC   ../build/fmpr/test/t-mul_si
    CC   ../build/fmpr/test/t-mul_ui
    CC   ../build/fmpr/test/t-normalise
    CC   ../build/fmpr/test/t-root
    CC   ../build/fmpr/test/t-rsqrt
    CC   ../build/fmpr/test/t-set_fmpq
    CC   ../build/fmpr/test/t-set_fmpz_2exp
    CC   ../build/fmpr/test/t-set_round_mpn
    CC   ../build/fmpr/test/t-set_round_ui_2exp_fmpz
    CC   ../build/fmpr/test/t-set_round_uiui_2exp_fmpz
    CC   ../build/fmpr/test/t-sqrt
    CC   ../build/fmpr/test/t-sub
    CC   ../build/fmpr/test/t-sum
    CC   ../build/fmpr/test/t-ulp
dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib/libarb-1.1.1.dylib
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/arb-2.8.1.p1/src/fmpr/../build/fmpr/test/t-abs_bound_le_2exp_fmpz
  Reason: image not found
make[4]: *** [../build/fmpr/test/t-abs_bound_le_2exp_fmpz_RUN] Abort trap: 6
make[3]: *** [check] Error 2

R:

Running the test suite for r-3.4.0.p0...
../../bin/R: line 246: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib/R//etc/ldpaths: No such file or directory
make[6]: *** [test-Examples-Base] Error 1
make[5]: *** [test-Examples] Error 2
make[4]: *** [test-all-basics] Error 1
make[3]: *** [check] Error 2
Error while running the R testsuite ... exiting

Singular:

Running the test suite for singular-4.1.0p3.p0...
./spkg-check: line 33: Singular: command not found
Error loading the freegb library in Singular.

pycrypto:

Running the test suite for pycrypto-2.6.1.p2...
PyCrypto will now be tested
running test
Traceback (most recent call last):
  File "setup.py", line 456, in <module>
    core.setup(**kw)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "setup.py", line 321, in run
    from Crypto import SelfTest
ImportError: No module named Crypto
Error: PyCrypto failed to pass its test suite.

cvxopt:

Running the test suite for cvxopt-1.1.8.p2...
Testing in /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/cvxopt-1.1.8.p2/src/examples/doc/chap10
Testing l1svc.py ...
Traceback (most recent call last):
  File "l1svc.py", line 3, in <module>
    from cvxopt import normal, setseed
ImportError: No module named cvxopt
Error: test /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.1.beta1/local/var/tmp/sage/build/cvxopt-1.1.8.p2/src/examples/doc/chap10/l1svc.py failed

comment:61 Changed 2 years ago by embray

How are you running these? I don't know why it would work any differently.

comment:62 follow-up: Changed 2 years ago by jhpalmieri

I get failures from make distclean; export SAGE_CHECK=yes; make. I get some (but not all) of these failures from ./sage -f -c PKG_NAME; this is after unsetting SAGE_CHECK and running make. But once the package has been installed successfully, then ./sage -f -c PKG_NAME may behave differently, because it may be using installed versions of libraries, binaries, etc.; if it hasn't been installed, it won't be installed until spkg-check finishes successfully.

comment:63 follow-up: Changed 2 years ago by jdemeyer

In build/pkgs/widgetsnbextension/spkg-install you write

# Note: Setting PYTHONPATH here is something of a hack to support
# installation into $SAGE_DESTDIR.  This might be better handled
# as a post-install step
SITE_PACKAGES="$(sage-python23 -c 'import site; print(site.getsitepackages()[0])')"
PYTHONPATH="$SAGE_DESTDIR$SITE_PACKAGES" jupyter nbextension enable --sys-prefix --py widgetsnbextension

I agree that this looks more like a post-install step, so why didn't you implement it that way?

comment:64 follow-up: Changed 2 years ago by jdemeyer

I am a bit lost where the DESTDIR stuff for autoconf packages is actually implemented. For Python packages, I see that you added the --root option to PIP_INSTALL.

comment:65 in reply to: ↑ 33 ; follow-up: Changed 2 years ago by jdemeyer

Replying to embray:

This ticket otherwise fixes all examples among standard (and some optional packages) where files were overwriting each other.

What is the relevance of that for this ticket?

All those packages worked before. If such clobbering is no longer allowed, I consider that a regression introduced by this ticket.

comment:66 in reply to: ↑ 63 Changed 2 years ago by embray

Replying to jdemeyer:

I agree that this looks more like a post-install step, so why didn't you implement it that way?

When I first wrote that, support for post-install scripts wasn't added yet. Now that it's there, I could probably revisit things like that. Though I'd have to review what exactly jupyter nbextension enable does. If it just writes some files then this might be fine.

comment:67 in reply to: ↑ 62 Changed 2 years ago by embray

Replying to jhpalmieri:

I get failures from make distclean; export SAGE_CHECK=yes; make. I get some (but not all) of these failures from ./sage -f -c PKG_NAME; this is after unsetting SAGE_CHECK and running make. But once the package has been installed successfully, then ./sage -f -c PKG_NAME may behave differently, because it may be using installed versions of libraries, binaries, etc.; if it hasn't been installed, it won't be installed until spkg-check finishes successfully.

I'm having trouble building python3 with SAGE_CHECK=yes. Did that ever work in the first place?

comment:68 Changed 2 years ago by jhpalmieri

In my experience, neither python2 nor python3 passed their self tests. With python2 at least, this was true on a range of platforms. We should probably change the default value of SAGE_CHECK_PACKAGES to exclude python3; it already excludes python2.

Edit: I created #23612.

Last edited 2 years ago by jhpalmieri (previous) (diff)

comment:69 Changed 2 years ago by embray

On the topic of the other SAGE_CHECK failures, the problem is probably similar to the problems I was having with Python: The checks are being performed before the files are installed from SAGE_DESTDIR into SAGE_LOCAL, so at the very least it might be necessary to set LD_LIBRARY_PATH/DYLIB_LIBRARY_PATH, but possibly other paths as well?

Rather than trying to fix every spkg-check, which may be difficult especially if I can't reproduce all the problems, it might be better for the sake of this ticket if I moved execution of spkg-check until after installation to SAGE_LOCAL. This has the downside that the package still gets installed even if the check failed, but that's the existing situation anyways. Once #22510 is done (which depends on this) it will be easier to do things like make a backup of an old package--try installing the new one--and revert to the old one if the installation failed.

comment:70 in reply to: ↑ 64 Changed 2 years ago by embray

Replying to jdemeyer:

I am a bit lost where the DESTDIR stuff for autoconf packages is actually implemented. For Python packages, I see that you added the --root option to PIP_INSTALL.

Just realized I never replied to this. DESTDIR= is added in the helper function sdh_make_install (which is partly why I wanted that feature implemented for this ticket). Note that it assumes SAGE_DESTDIR is set, which it may not be if the spkg-install is not being run from sage-spkg. It might be worth noting that this is intentional--if SAGE_DESTDIR is left blank then direct installation to SAGE_LOCAL proceeds as normal, which is important and useful for testing, if nothing else.

Otherwise, SAGE_DESTDIR is set in sage-spkg, and gets passed to make install, in most cases, by way of sdh_make_install.

comment:71 Changed 2 years ago by embray

For some reason on Linux I couldn't reproduce the problems running the tests for flint or arb (despite running make distclean first). But I did get the same problems for the other packages. As I wrote here it seems to be mostly a matter of setting the required environment variables. But I think I'll still revert to the old behavior for SAGE_CHECK for now, just to keep this simpler.

comment:72 Changed 2 years ago by git

  • Commit changed from ce38d0756f226ec9a87b55890e716edd80f0cbdc to 70013431e8c8a4ac4f49191378aea54e006b1a4b

Branch pushed to git repo; I updated commit sha1. New commits:

ed2446bMove spkg-check to after the package has been installed to SAGE_LOCAL.
7001343Enable widgetsnbextension in the notebook's config file as a postinst step.

comment:73 Changed 2 years ago by embray

Just remembered there was still unfinished work on this that I hadn't pushed yet. I think now it's addressed all the issues that have been raised so far.

comment:74 follow-up: Changed 2 years ago by jdemeyer

One concern with this patch is that it's really a patch bomb, which might be hard to get reviewed and merged.

I don't think you have answered 65

comment:75 in reply to: ↑ 65 Changed 2 years ago by embray

I thought I addressed this somewhere but maybe not...

Replying to jdemeyer:

Replying to embray:

This ticket otherwise fixes all examples among standard (and some optional packages) where files were overwriting each other.

What is the relevance of that for this ticket?

All those packages worked before. If such clobbering is no longer allowed, I consider that a regression introduced by this ticket.

I wouldn't consider it a regression at all. The fact that some packages overwrote files installed by other packages is a bug, just one that hasn't had very serious effects. Most cases of this were very minor (e.g. python3 overwrote the valgrind.supp installed by python2).

The goal of this ticket is, along with #22510 (where this becomes important) is to make sagedist work a little more like a real OS packaging system in that files are belonged by packages, with any one file belonging to just one package. That way, uninstalling one package does not break another package that owns some of the same file(s).

In theory one could design a packaging system where a file can belong to multiple packages, and only be removed once all packages that use it are removed. But this is more complicated, and only works if that file is identical for all packages that use it (it may not be).

There is one particular example of a file shared between packages where this ticket would introduce a regression if it didn't carefully handle this case: There's a file info/share/dir that is updated by any package that installs an info manual. The new way packages are installed by ticket is such that each package will write a fresh info/share/dir, and then overwrite the existing one, rather than updating the existing one. This makes sure to always update info/share/dir in a post-install script for those packages.

Other than that, the other cases are mostly trivial.

comment:76 Changed 2 years ago by git

  • Commit changed from 70013431e8c8a4ac4f49191378aea54e006b1a4b to 53e3dc54db7658c6eceb50d573eeb324d886a532

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

0275058On OSX DYLD_LIBRARY_PATH should be set a la LD_LIBRARY_PATH to run Python from its build directory
8e193dbMove spkg-check to after the package has been installed to SAGE_LOCAL.
53e3dc5Enable widgetsnbextension in the notebook's config file as a postinst step.

comment:77 Changed 2 years ago by embray

Had to rebase after I realized I pop'd the completely wrong stash entry into an earlier commit.

comment:78 in reply to: ↑ 74 ; follow-up: Changed 2 years ago by embray

Replying to jdemeyer:

One concern with this patch is that it's really a patch bomb, which might be hard to get reviewed and merged.

True, but there's not much other way to handle it (I don't mind separating some changes out to separate tickets but some of them go hand-in-hand with this). John did test this extensively in OSX and got it working. The only remaining concern was that some spkg-check tests weren't working. I believe I've addressed that now, though it's hard to be 100% sure since some of those tests aren't working for me with or without this patch.

I don't think you have answered 65

Done.

Last edited 2 years ago by embray (previous) (diff)

comment:79 Changed 2 years ago by embray

  • Dependencies changed from #23059 #23096 #23160 to #23059 #23096 #23160 #21726

In an effort to split this up a little bit, I'm moving some of the fixes for the Python2/3 packages to a separate ticket, since it can work independently of this ticket: #23781

Last edited 2 years ago by embray (previous) (diff)

comment:80 Changed 2 years ago by embray

  • Dependencies changed from #23059 #23096 #23160 #21726 to #23059 #23096 #23160 #23781

Wrong ticket.

comment:81 Changed 2 years ago by git

  • Commit changed from 53e3dc54db7658c6eceb50d573eeb324d886a532 to f97fdf916d7a58006d1be37c419180c8c8ee1893

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

71cb02dAdded the ability for packages to have a post-install script called spkg-postinst.
013e6f3Add special post-install handling for the backports/__init__.py shared
b886725Remove some no-longer used local variables
f94b3ecAdd handling for packages that install .info files and update share/info/dir
f4960eaRename $SAGE_INST_TEMP to $SAGE_DESTDIR
36ff6caDon't use the -u option to cp. It's not portable to BSD's cp, and is
25c4d01Just use 'cp -a' which seems to be compatible across most cp's.
8c9a60fWork around bug in OpenBLAS's Makefile with its use of DESTDIR with
1389380Move spkg-check to after the package has been installed to SAGE_LOCAL.
f97fdf9Enable widgetsnbextension in the notebook's config file as a postinst step.

comment:82 in reply to: ↑ 78 Changed 2 years ago by jdemeyer

Replying to embray:

Replying to jdemeyer:

One concern with this patch is that it's really a patch bomb, which might be hard to get reviewed and merged.

True, but there's not much other way to handle it

Why do you think that? This doesn't have to be an all-or-nothing patch: you could first implement the general framework and then gradually fix packages.

Anyway, if you find somebody else to review this, I don't mind at all. But for me, I am quite discouraged from reviewing such a big patch. And then there is also the potential for merge conflicts if you make so many changes.

comment:83 follow-up: Changed 2 years ago by embray

I'm not sure how you would propose breaking this up. It is pretty all-or-nothing: Either DESTDIR installation works or it doesn't.

There's no sense in "gradually fix packages" because they've already all been fixed here, and in a way that is likely to result in only minor conflicts at worse (no more so than in #23179--the analysis in this comment still mostly applies).

I could break it up into multiple tickets--I already did in one case. But for the most part I don't see what that solves. If you want guidance on what to review, start by ignoring all the build/pkgs changes since the vast majority of them are completely mechanistic.

comment:84 in reply to: ↑ 83 Changed 2 years ago by jdemeyer

Replying to embray:

I'm not sure how you would propose breaking this up. It is pretty all-or-nothing: Either DESTDIR installation works or it doesn't.

You could start with "DESTDIR works for a few selected packages" or "DESTDIR works for Python packages only"

in a way that is likely to result in only minor conflicts at worse

A merge conflict is a merge conflict. It doesn't matter how minor it is.

comment:85 follow-up: Changed 2 years ago by embray

Alright, that's fair. I'll see if I can break this up a bit more.

Fact remains, there's not much benefit to this unless all packages are updated. There's no real logical way I can think of to carving it up into multiple tickets, when it can all be done at once. But for the sake of making review easier I'll try out your suggestion...

comment:86 in reply to: ↑ 85 ; follow-up: Changed 2 years ago by jdemeyer

Replying to embray:

I'll see if I can break this up a bit more.

Let me mention that this is a personal "pet peeve". If somebody else (John Palmieri for example) is happy with this ticket and wants to review it, I won't object to that.

comment:87 Changed 2 years ago by jhpalmieri

I'm not sure I understand the build process well enough to be competent enough to review it.

comment:88 in reply to: ↑ 86 ; follow-up: Changed 2 years ago by embray

Replying to jdemeyer:

Replying to embray:

I'll see if I can break this up a bit more.

Let me mention that this is a personal "pet peeve". If somebody else (John Palmieri for example) is happy with this ticket and wants to review it, I won't object to that.

It's fine--I see your point. I think it's easier to review if one patch contains just:

  • The basic structural changes
  • A few motivating examples for the different cases

and then leave the remaining boilerplate changes to a separate ticket.

comment:89 in reply to: ↑ 88 Changed 2 years ago by jdemeyer

Replying to embray:

I think it's easier to review if one patch contains just:

  • The basic structural changes
  • A few motivating examples for the different cases

Exactly. But I would prefer to finish #23781 and #21535 first.

comment:90 follow-up: Changed 2 years ago by embray

I'm not exactly sure what #21535 has to do with this.

comment:91 in reply to: ↑ 90 Changed 2 years ago by jdemeyer

Replying to embray:

I'm not exactly sure what #21535 has to do with this.

It's unrelated. I just mentioned it because I can only keep track of a limited number of build-system tickets in my brain cache.

comment:92 Changed 2 years ago by git

  • Commit changed from f97fdf916d7a58006d1be37c419180c8c8ee1893 to c1d5453c2735cdda78d53dc36b56fb54ec7dd336

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

be70ddbAdded the ability for packages to have a post-install script called spkg-postinst.
e246937Add special post-install handling for the backports/__init__.py shared
87a02acRemove some no-longer used local variables
5927318Add handling for packages that install .info files and update share/info/dir
73245deRename $SAGE_INST_TEMP to $SAGE_DESTDIR
a771cd8Don't use the -u option to cp. It's not portable to BSD's cp, and is
5125047Just use 'cp -a' which seems to be compatible across most cp's.
3e8e7b5Work around bug in OpenBLAS's Makefile with its use of DESTDIR with
58676b4Move spkg-check to after the package has been installed to SAGE_LOCAL.
c1d5453Enable widgetsnbextension in the notebook's config file as a postinst step.

comment:93 follow-up: Changed 2 years ago by embray

Went ahead and rebased on current #23781. But I'm now I'm going to do some surgery to break this into some smaller tickets for which this ticket will serve as the mothership.

comment:94 Changed 2 years ago by embray

  • Description modified (diff)
  • Status changed from needs_review to needs_work

comment:95 in reply to: ↑ 93 Changed 2 years ago by jdemeyer

Replying to embray:

Went ahead and rebased on current #23781.

This doesn't need to depend on #23781. Just leave the Python build script alone.

comment:96 Changed 2 years ago by embray

Uhh, it does need to in a manner of speaking. Eventually the Python build script needs to be updated to support this. But I'm already working on completely breaking this up in to separate tickets.

comment:97 Changed 2 years ago by embray

  • Description modified (diff)

Added #24106--a simplified ticket demonstrating the core functionality of the branch on this ticket. Will add followup tickets with additional details.

comment:98 Changed 2 years ago by embray

  • Milestone changed from sage-8.1 to sage-8.2

comment:99 Changed 23 months ago by embray

  • Description modified (diff)

comment:100 Changed 23 months ago by embray

  • Description modified (diff)

comment:101 Changed 23 months ago by jdemeyer

I am surprised now to see that #22509 and #24106 are distinct tickets. Isn't this one a "duplicate"? There is a stale branch here, is that still relevant?

comment:102 Changed 23 months ago by embray

see 93

comment:103 in reply to: ↑ 28 Changed 21 months ago by jdemeyer

Replying to embray:

Then the $SAGE_DESTDIR$SAGE_LOCAL combination can be called $SAGE_DESTDIR_LOCAL.

I don't remember if there was a reason not to do this, but I'll introduce this variable in #25001.

comment:104 Changed 21 months ago by embray

Yes, there was a reason: https://trac.sagemath.org/ticket/22509#comment:30

Though I suppose if $SAGE_DESTDIR isn't set then it's fine if $SAGE_DESTDIR_LOCAL == $SAGE_LOCAL.

comment:105 Changed 20 months ago by embray

  • Milestone changed from sage-8.2 to sage-8.3

comment:106 Changed 17 months ago by embray

  • Milestone changed from sage-8.3 to sage-8.4

I believe this issue can reasonably be addressed for Sage 8.4.

comment:107 Changed 14 months ago by embray

  • Milestone changed from sage-8.4 to sage-8.5

comment:108 Changed 12 months ago by embray

  • Milestone changed from sage-8.5 to sage-8.7

Retargeting some of my tickets (somewhat optimistically for now).

comment:109 Changed 9 months ago by embray

  • Milestone changed from sage-8.7 to sage-8.8

Moving all my in-progress tickets to 8.8 milestone.

comment:110 Changed 6 months ago by embray

  • Milestone changed from sage-8.8 to sage-8.9

Tickets still needing working or clarification should be moved to the next release milestone at the soonest (please feel free to revert if you think the ticket is close to being resolved).

Note: See TracTickets for help on using tickets.