Opened 3 years ago
Last modified 6 months ago
#22509 needs_work enhancement
Install packages in temporary root before copying to $SAGE_LOCAL
Reported by:  embray  Owned by:  

Priority:  major  Milestone:  sage8.9 
Component:  build  Keywords:  
Cc:  mkoeppe  Merged in:  
Authors:  Erik Bray  Reviewers:  
Report Upstream:  N/A  Work issues:  
Branch:  u/embray/build/destdir (Commits)  Commit:  c1d5453c2735cdda78d53dc36b56fb54ec7dd336 
Dependencies:  #23059 #23096 #23160 #23781  Stopgaps: 
Description (last modified by )
Currently we build packages, in most cases, by setting prefix=$SAGE_LOCAL
(in ./configure
, or the equivalent in nonautoconf packages), and then installing directly to that prefix.
This is fine to keep as is. But with automake packages we can also set DESTDIR
(https://www.gnu.org/software/automake/manual/html_node/DESTDIR.html#DESTDIR) to an alternative (typically temporary) directory into which to install. This has a couple advantages:
 Although it's not as usual for
make install
to fail after a successfulmake
, it can happen. This prevents messy partial installs in the case of a failedmake install
run.
 It gives us the opportunity to make an exact list of the files that were installed. This is a prerequisite to improving package uninstallation/reinstallation in sagedist (see #22510).
This is also standard operating procedure in most other packaging systems, so it would be good for Sage to adopt too. The general approach is the same as that used for converting an arbitrary Makefile to support this conventioneverywhere a file is installed, prefix the installation path with $(DESTDIR)
(or in this case a variable we'll call $SAGE_DESTDIR
).
For packages that already support the DESTDIR
convention, then, we pass DESTDIR=$SAGE_DESTDIR
to make install
. It is also already possible for Python packages (e.g. pip install root
).
Since the introduction of the build helper scripts in #23160, it makes the most sense to make these changes directly in the helper scripts, and then update more packages to use those helper scripts where possible.
Dependent tickets:
 #24106  implement the basic functionality of supporting staged installs through
sagespkg
 #24024  update more
spkgbuild/install
scripts to use thesdh_
helper functions, since that is the fastest way to addDESTDIR
support for many packages.  #24646  add support for
root=${SAGE_DESTDIR}
to Python spkgs  #24645  add support for postinstall scripts; this can be useful for cases where, for example, local files are generated after the package is installed, or to handle cases where one file is shared by multiple packages and needs to be updated when a package that uses it is installed (see e.g. https://trac.sagemath.org/ticket/22510#comment:5)
 #?????  convert all spkgs to support stagedinstall; for many packages this can be covered by using the helper functions, but some packages require additional steps; in particular, packages that have files that are manually copied into
$SAGE_LOCAL
, rather than through a standard installer
Change History (110)
comment:1 Changed 3 years ago by
 Cc mkoeppe added
comment:2 Changed 3 years ago by
 Description modified (diff)
 Milestone changed from sage7.6 to sagewishlist
comment:3 Changed 3 years ago by
 Branch set to u/embray/build/destdir
 Commit set to a25a42a02a89a20058d41d95c56efc1cd0875f26
 Dependencies set to #23096
 Status changed from new to needs_review
comment:4 Changed 3 years ago by
 Dependencies changed from #23096 to #23059 #23096
comment:5 Changed 3 years ago by
By way of explanation, the idea here is that there is a variable exported by sagespkg
called $SAGE_INST_TEMP
(the name is open to bikeshedding), so this is expected to be passed to each spkginstall
to use an alternative install root. In principle this could be defined in each spkginstall
individually, but it's less trouble to standardize in one place, and in any case sagespkg
still needs to know this path to copy files from it to $SAGE_LOCAL
.
Of course, if we take this approach, the developer docs will also need to be updated with some guidance on how to implement staged installation for new packages.
comment:6 Changed 3 years ago by
 Commit changed from a25a42a02a89a20058d41d95c56efc1cd0875f26 to 8a621af7a5f3e10d1337c07b42d04edee6a8f1b8
comment:7 Changed 3 years ago by
 Commit changed from 8a621af7a5f3e10d1337c07b42d04edee6a8f1b8 to 1391089b36f43b45fa0eb7bf04e83fc1e351dfea
Branch pushed to git repo; I updated commit sha1. New commits:
1391089  Remove the 'v' from the 'cp' command.

comment:8 Changed 3 years ago by
 Commit changed from 1391089b36f43b45fa0eb7bf04e83fc1e351dfea to e2941230aea00481d6cf1f58992faa006a7b63f2
Branch pushed to git repo; I updated commit sha1. New commits:
e294123  Add staged install support for atlas

comment:9 Changed 3 years ago by
 Description modified (diff)
comment:10 Changed 3 years ago by
 Description modified (diff)
comment:11 Changed 3 years ago by
 Commit changed from e2941230aea00481d6cf1f58992faa006a7b63f2 to 00f6de8d173a25b865c26c6af3f588b49e8401eb
Branch pushed to git repo; I updated commit sha1. New commits:
00f6de8  This file should be deleted from the temp install dir during staged install on OSX

comment:12 Changed 3 years ago by
 Commit changed from 00f6de8d173a25b865c26c6af3f588b49e8401eb to 48e9ffbf5fe86d3186a505ae1e9ab1b13c8195f2
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
16bc700  Update freetype and pkgconf to use DESTDIR

eb660c7  Updated various packages with nonstandard installers or that are otherwise

8de16d2  Update cephes to work with staged install.

73c45e6  Update openblas to use staged install

e1ad6d5  Update lcalc to support staged install

f07e056  Update zn_poly, which doesn't really have a proper installer, to use staged install

9997a37  Updated the python2/3 pacakges to use staged install.

f9cfa6f  Remove the 'v' from the 'cp' command.

e51f578  Add staged install support for atlas

48e9ffb  This file should be deleted from the temp install dir during staged install on OSX

comment:13 followup: ↓ 22 Changed 3 years ago by
 Milestone changed from sagewishlist to sage8.1
 Status changed from needs_review to needs_work
This already fails on the first package... patch
:
Making install in src make[5]: Entering directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src/src' make[6]: Entering directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src/src' /bin/mkdir p '/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/inst/opt/sage/sagegit/local/bin' /usr/bin/install c patch '/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/inst/opt/sage/sagegit/local/bin' make[6]: Nothing to be done for `installdataam'. make[6]: Leaving directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src/src' make[5]: Leaving directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src/src' Making install in tests make[5]: Entering directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src/tests' make[6]: Entering directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src/tests' make[6]: Nothing to be done for `installexecam'. make[6]: Nothing to be done for `installdataam'. make[6]: Leaving directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src/tests' make[5]: Leaving directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src/tests' make[5]: Entering directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src' make[6]: Entering directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src' make[6]: Nothing to be done for `installexecam'. /bin/mkdir p '/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/inst/opt/sage/sagegit/local/share/man/man1' /usr/bin/install c m 644 'patch.man' '/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/inst/opt/sage/sagegit/local/share/man/man1/patch.1' make[6]: Leaving directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src' make[5]: Leaving directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src' make[4]: Leaving directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src' make[3]: Leaving directory `/opt/sage/sagegit/local/var/tmp/sage/build/patch2.7.5/src' Cannot find the patch program we just installed
comment:14 followups: ↓ 16 ↓ 30 Changed 3 years ago by
I see ${SAGE_INST_TEMP}${SAGE_LOCAL}
occuring in various places, maybe you should define a new variable for this.
comment:15 followup: ↓ 17 Changed 3 years ago by
Let me say that this a very brave patch :)
comment:16 in reply to: ↑ 14 ; followup: ↓ 18 Changed 3 years ago by
Replying to jdemeyer:
I see
${SAGE_INST_TEMP}${SAGE_LOCAL}
occuring in various places, maybe you should define a new variable for this.
Yeah, I keep thinking about that. At first I was just going to redefine $SAGE_LOCAL
to that, but that doesn't work because one needs the bare $SAGE_LOCAL
, e.g., for the prefix. I think I just need a good name for the combined path. Maybe $SAGE_LOCAL_TEMP
?
comment:17 in reply to: ↑ 15 Changed 3 years ago by
Replying to jdemeyer:
Let me say that this a very brave patch :)
Thanks! I just hope you agree with it in principle, because I really think it should be done. Open to ideas on the specific details, although I think this is already pretty close. There's a lot of things we could be doing better packagingwise in the first place of course, but I'm still just trying to make the least disruptive changes I can.
comment:18 in reply to: ↑ 16 ; followup: ↓ 28 Changed 3 years ago by
Replying to embray:
Maybe
$SAGE_LOCAL_TEMP
?
I would prefer $SAGE_TEMP_LOCAL
simply because that's consistent with the order (first SAGE_INST_TEMP
, then SAGE_LOCAL
).
comment:19 Changed 3 years ago by
That's fine. But before I do a mass rename on this, perhaps you (and/or others) could have a look at #23160. It would be very nice to have this, or something like it, as a dependency on this ticket so that I can make a few bulk changes once (in the case of #23160 it would be replacing all calls to $MAKE/make install
with sdh_make_install
, and then many future tweaks can be made in one placethe definition of sdh_make_install
).
comment:20 Changed 3 years ago by
 Commit changed from 48e9ffbf5fe86d3186a505ae1e9ab1b13c8195f2 to 7bcbccb7a1b223017eb45d334cb097f365632c48
comment:21 Changed 3 years ago by
 Commit changed from 7bcbccb7a1b223017eb45d334cb097f365632c48 to 9eb5876c6793718ba6225aaba0e07fcf71eab432
Branch pushed to git repo; I updated commit sha1. New commits:
9eb5876  Remove this sanity check

comment:22 in reply to: ↑ 13 Changed 3 years ago by
Replying to jdemeyer:
This already fails on the first package...
patch
:
Just a peculiarity of that package's spkginstall
. I removed the seemingly pointless check for now.
comment:23 Changed 3 years ago by
 Commit changed from 9eb5876c6793718ba6225aaba0e07fcf71eab432 to 5f7d07967411a4c2ebdf2a837f6427bc210a761e
Branch pushed to git repo; I updated commit sha1. New commits:
5f7d079  Fix sagenb installation

comment:24 Changed 3 years ago by
 Commit changed from 5f7d07967411a4c2ebdf2a837f6427bc210a761e to 23799c8b38066712b67354916bba6c265b4f15d2
Branch pushed to git repo; I updated commit sha1. New commits:
23799c8  Actually fix sagenb install

comment:25 Changed 2 years ago by
 Commit changed from 23799c8b38066712b67354916bba6c265b4f15d2 to 84a6827229bbaa3a26154b1202d4f12fe4acad43
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
f30f964  Remove this sanity check

637d136  Fix sagenb installation

03b50fe  Actually fix sagenb install

2095d1d  Make sure the path these symlinks are installed in exists first.

48304f2  Some more enhancements to the sagedisthelpers:

5f0637d  Include the PKG_NAME, PKG_BASE, and PKG_VER variables directly in the install

b5633e1  Now that sdh_make_install implements DESTDIR we can replace all manual

f41ab62  A few more standard autotools packages that are easily updated but I missed previously

06e7339  Disable pip installation when installing Python 3; see https://trac.sagemath.org/ticket/23398

84a6827  Don't override the valgrind suppression file from python2; add a separate one for Python 3

comment:26 Changed 2 years ago by
 Dependencies changed from #23059 #23096 to #23059 #23096 #23160
Now depends on #23160. Updates a bunch of packages to use the helper functionsonce that change is made once it will be much easier to make mass updates simply by updating the helper functions.
comment:27 Changed 2 years ago by
 Commit changed from 84a6827229bbaa3a26154b1202d4f12fe4acad43 to 0885622958923d3f9f00725076b6553e10b8de4b
Branch pushed to git repo; I updated commit sha1. New commits:
0885622  Handle the 2to3 script better so that Python2 and Python3 don't overwrite each other

comment:28 in reply to: ↑ 18 ; followup: ↓ 103 Changed 2 years ago by
Replying to jdemeyer:
Replying to embray:
Maybe
$SAGE_LOCAL_TEMP
?I would prefer
$SAGE_TEMP_LOCAL
simply because that's consistent with the order (firstSAGE_INST_TEMP
, thenSAGE_LOCAL
).
Actually, I think I'm going to rename $SAGE_INST_TEMP
to $SAGE_DESTDIR
for consistency with the DESTDIR convention. I think that makes it clear what it's for. Then the $SAGE_DESTDIR$SAGE_LOCAL
combination can be called $SAGE_DESTDIR_LOCAL
.
comment:29 Changed 2 years ago by
 Commit changed from 0885622958923d3f9f00725076b6553e10b8de4b to 62e916b7cda969e7798fb096672923a2c049fa10
Branch pushed to git repo; I updated commit sha1. New commits:
57ea8c9  Added the ability for packages to have a postinstall script called spkgpostinst.

76703a4  Add special postinstall handling for the backports/__init__.py shared

be4f47a  Remove some nolonger used local variables

62e916b  Add handling for packages that install .info files and update share/info/dir

comment:30 in reply to: ↑ 14 Changed 2 years ago by
Replying to jdemeyer:
I see
${SAGE_INST_TEMP}${SAGE_LOCAL}
occuring in various places, maybe you should define a new variable for this.
Actually on second thought, there was a reason for this: The idea was that if $SAGE_INST_TEMP
isn't set then installation straight into $SAGE_LOCAL
will still work. The way this works currently is $SAGE_INST_TEMP
is set by sagespkg
which handles the details of how to copy from the temporary install path into SAGE_LOCAL, and record the file manifest.
But if you're just running a package's spkginstall directly, then $SAGE_INST_TEMP
is not set. With #23179 in place I could make it so that $SAGE_INST_TEMP
is always set in the script wrappers. But I'm not necessarily sure that's desirable, since if you're running the script outside of sagespkg the implementation details of staged installation are not meaningful, and it may be more useful to install directly into SAGE_LOCAL assuming it's for testing purposes. Of course, one can always manually pass in a value for $SAGE_INST_TEMP
.
So I think it would be better *not* to combine $SAGE_INST_TEMP$SAGE_LOCAL
into a single variable, unless others are convinced that there is no use case for installing without $SAGE_INST_TEMP
set.
(I do still want to rename it $SAGE_DESTDIR
in this way it is used quite analogously to the way $(DESTDIR)
is used in Makefiles.)
comment:31 Changed 2 years ago by
If it helps makes things more readable I could use the curlybrace syntax like ${SAGE_INST_TEMP}${SAGE_LOCAL}
rather then running them directly together.
comment:32 Changed 2 years ago by
 Commit changed from 62e916b7cda969e7798fb096672923a2c049fa10 to 01c05f0c3bdd164d07846c72324e15f897328d6b
Branch pushed to git repo; I updated commit sha1. New commits:
01c05f0  Rename $SAGE_INST_TEMP to $SAGE_DESTDIR

comment:33 followup: ↓ 65 Changed 2 years ago by
 Status changed from needs_work to needs_review
I think that, pending #23160, this is basically ready. I've tested it quite extensively myself on Linux and Cygwin and haven't encountered any more obvious problems.
The only exception would be that there are still some packages that write files that clobber each other, but they are only cases of optional/experimental packages clobbering standard packages, or optional/experimental packages clobbering each other. These include:
 openblas / atlas
 mpir / gmp
 boost_cropped / boost
 pari_seadata_small / database_pari
 database_stein_watkins_small / database_stein_watkins
 gcc / compilerwrapper
However, these incompatibilities were already the status quo. While it will be possible in the future to improve the status quo (#23465 is work in that direction), I think these cases can be fixed independently of this ticket. This ticket otherwise fixes all examples among standard (and some optional packages) where files were overwriting each other.
comment:34 Changed 2 years ago by
 Status changed from needs_review to needs_work
On OS X, I get this with the patch
package:
cp: illegal option  u usage: cp [R [H  L  P]] [fi  n] [apvX] source_file target_file cp [R [H  L  P]] [fi  n] [apvX] source_file ... target_directory cp: illegal option  u usage: cp [R [H  L  P]] [fi  n] [apvX] source_file target_file cp [R [H  L  P]] [fi  n] [apvX] source_file ... target_directory Successfully installed patch2.7.5
A little later the same error causes yasm not to be installed, so mpir won't install:
[mpir3.0.0.p0] configure: error: no systemwide yasm found
In addition to fixing the cp
command in sagespkg
, should you also check for errors after executing commands like this so as to avoid "Successfully installed patch2.7.5" when it didn't actually successfully install?
comment:35 Changed 2 years ago by
See also https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/cp.1.html (this page says it is retired, but it agrees with the man page on my Mac) and possibly also https://stackoverflow.com/questions/1132508/howtoemulatecpupdatebehavioronmacosx.
comment:36 Changed 2 years ago by
Thanks for the quick feedback! I'll have to take a look at the OSX manpage 😞. This is fairly simple functionality to duplicate.
comment:37 Changed 2 years ago by
Honestly, the u
isn't even necessary. In most cases it's going to be superfluous so I'd probably just remove it.
comment:38 Changed 2 years ago by
 Commit changed from 01c05f0c3bdd164d07846c72324e15f897328d6b to d9500366634b7eb5b24eea91dbf5fd78a50848e3
Branch pushed to git repo; I updated commit sha1. New commits:
d950036  Don't use the u option to cp. It's not portable to BSD's cp, and is

comment:39 Changed 2 years ago by
 Status changed from needs_work to needs_review
comment:40 Changed 2 years ago by
 Status changed from needs_review to needs_work
ncurses
fails to install for me (on OS X):
Copying package files from temporary location /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.0.rc0/local/var/tmp/sage/build/ncurses6.0/inst to /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.0.rc0/local cp: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.0.rc0/local/var/tmp/sage/build/ncurses6.0/inst/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.0.rc0/local//lib/terminfo is a directory (not copied). ************************************************************************ Error copying files for ncurses. ************************************************************************
comment:41 Changed 2 years ago by
Do you know the correct way to copy a symbolic link without dereferencing it on OSX? It should be P
, but according to that man page it only respects P
if R
is used also. But this is copying files one at a time. I could use R
but would still have to loop over all the files anyways in order to build the manifest. Also that man page says:
In R mode, cp will continue copying even if errors are detected.
which isn't necessarily what I want either.
comment:42 Changed 2 years ago by
 Commit changed from d9500366634b7eb5b24eea91dbf5fd78a50848e3 to d605943e4ee29256ea30180803b96aa6a2d6917c
Branch pushed to git repo; I updated commit sha1. New commits:
d605943  Just use 'cp a' which seems to be compatible across most cp's.

comment:43 Changed 2 years ago by
 Status changed from needs_work to needs_review
comment:44 Changed 2 years ago by
It occurs to me that the
pkgconf
package isn't being handled quite correctly. This leads to problems when reinstalling / upgrading that package.
Nevermind, for the purposes of this ticket it's fine. It's only a problem in #22510.
comment:45 Changed 2 years ago by
Now python2 and python3 don't install:
Installing valgrind suppression file... ./spkginstall: line 140: ./python: is a directory Testing importing of various modules... ./spkginstall: line 162: ./python: is a directory ctypes module failed to import ./spkginstall: line 162: ./python: is a directory math module failed to import ./spkginstall: line 162: ./python: is a directory hashlib module failed to import ./spkginstall: line 162: ./python: is a directory crypt module failed to import ./spkginstall: line 162: ./python: is a directory readline module failed to import ./spkginstall: line 162: ./python: is a directory socket module failed to import ./spkginstall: line 171: ./python: is a directory _scproxy module failed to import Error: One or more modules failed to import.
comment:46 followup: ↓ 49 Changed 2 years ago by
Does OSX have caseinsensitivity or something?
comment:47 Changed 2 years ago by
Or in other words, if I have in the same directory an executable called python
and a directory called Python
(as is the case in the CPython source tree) what will happen if I run ./python
on OSX?
comment:48 Changed 2 years ago by
Coincidentally, I just discovered that this also fixes a rare race condition that results from the fact that multiple packages write to $SAGE_LOCAL/usr/share/info/dir
. If two packages that write to that file are installed in parallel there are probably a number of ways that can go wrong, such as (what just happened to me):
A) [ ! d $(DESTDIR)$info_dir ] ...
B) mkdir $(DESTDIR)$info_dir
A) ...  mkdir $(DESTDIR)$info_dir
resulting in directory already exists error.
comment:49 in reply to: ↑ 46 Changed 2 years ago by
Replying to embray:
Does OSX have caseinsensitivity or something?
It turns out Python's ./configure
explicitly checks this. Does the build log say something like
checking for caseinsensitive build directory...yes
?
comment:50 Changed 2 years ago by
Indeed, it looks like on any OS (including OSX) if a caseinsensitive filesystem is being built on it outputs the Python executable in the build dir as python.exe
so we just need to look for that I guess.
comment:51 Changed 2 years ago by
 Commit changed from d605943e4ee29256ea30180803b96aa6a2d6917c to dbdb94788f6275e19c5fd150bec8a889188f3164
Branch pushed to git repo; I updated commit sha1. New commits:
dbdb947  Fix Python build on nonCygwin systems (namely OSX) with caseinsensitive filesystems

comment:52 Changed 2 years ago by
Give that a try. I can't imagine there's any other reason for this.
comment:53 Changed 2 years ago by
Yes, OS X has a type of caseinsensitivity. The current branch gets a little farther, but still fails. With python2:
Installing valgrind suppression file... Testing importing of various modules... Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/Lib/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: dlopen(/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_ctypes.so, 2): Symbol not found: _PyUnicodeUCS4_AsEncodedString Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_ctypes.so Expected in: flat namespace in /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_ctypes.so ctypes module failed to import math module imported OK Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/Lib/hashlib.py", line 158, in <module> import struct File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/Lib/struct.py", line 1, in <module> from _struct import * ImportError: dlopen(/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_struct.so, 2): Symbol not found: _PyUnicodeUCS4_AsEncodedString Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_struct.so Expected in: flat namespace in /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_struct.so hashlib module failed to import crypt module imported OK readline module imported OK Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/Lib/socket.py", line 47, in <module> import _socket ImportError: dlopen(/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_socket.so, 2): Symbol not found: _PyUnicodeUCS4_AsEncodedString Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_socket.so Expected in: flat namespace in /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python22.7.13.p1/src/build/lib.macosx10.9x86_642.7/_socket.so socket module failed to import _scproxy module imported OK Error: One or more modules failed to import.
It's odd that some of the modules succeeded while others failed.
With python3:
Installing valgrind suppression file... dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found Testing importing of various modules... dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found ./spkginstall: line 160: 84992 Abort trap: 6 $PYTHON c "import $module" ctypes module failed to import dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found ./spkginstall: line 160: 84993 Abort trap: 6 $PYTHON c "import $module" math module failed to import dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found ./spkginstall: line 160: 84995 Abort trap: 6 $PYTHON c "import $module" hashlib module failed to import dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found ./spkginstall: line 160: 84997 Abort trap: 6 $PYTHON c "import $module" crypt module failed to import dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found ./spkginstall: line 160: 84998 Abort trap: 6 $PYTHON c "import $module" readline module failed to import dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found ./spkginstall: line 160: 85000 Abort trap: 6 $PYTHON c "import $module" socket module failed to import dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/lib/libpython3.6m.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/python33.6.1/src/./python.exe Reason: image not found ./spkginstall: line 176: 85001 Abort trap: 6 $PYTHON c "import _scproxy" _scproxy module failed to import Error: One or more modules failed to import.
comment:54 Changed 2 years ago by
I see. This probably means that some DYLD_LIBRARY_PATH
or similar needs to be set in order for Python being run out of the source tree to link with the correct libpython.
Thanks for your testing / patienceI need to bother nthiery to get me an OSX machine I can test on so that this will be easier in the future.
Other than this Python thing I don't anticipate too many other issues with this. The only reason Python is being stubborn is that I made a somewhat nontrivial change to how the Python selftests are performed. Very few of the other packages otherwise have significant changes to how they're installed (though a few do).
comment:55 Changed 2 years ago by
Python has also always been finicky to install on OS X. Anyway, imitating what was there already for linux works on this machine: adding
if [ "$UNAME" = "Darwin" ]; then export DYLD_LIBRARY_PATH="." fi
right before echo "Testing importing of various modules..."
, for example, fixed both python2
and python3
.
Other issues: this one is strange and nonrepeatable, but I saw this once with maxima:
Found local metadata for maxima5.39.0.p0 Could not find platform independent libraries <prefix> Could not find platform dependent libraries <exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] ImportError: No module named site ************************************************************************ Error downloading maxima5.39.0.tar.gz ************************************************************************
In this case, the maxima tarball had already been downloaded to the upstream
directory. When I ran "make" again, it went smoothly. I don't know if this is worth investigating.
More importantly, I get build errors with gsl
and numpy
. For gsl
:
checking build system type... x86_64appledarwin16.7.0 checking host system type... x86_64appledarwin16.7.0 checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... configure: error: in `/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta0/local/var/tmp/sage/build/gsl2.3/src': configure: error: cannot run C compiled programs. If you meant to cross compile, use `host'. See `config.log' for more details
config.log says
configure:3480: checking whether the C compiler works configure:3502: gcc g O2 L/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib Wl,rpath,/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib conftest.c lopenblas lm >&5 configure:3506: $? = 0 configure:3554: result: yes configure:3557: checking for C compiler default output file name configure:3559: result: a.out configure:3565: checking for suffix of executables configure:3572: gcc o conftest g O2 L/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib Wl,rpath,/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib conftest.c lopenblas lm >&5 configure:3576: $? = 0 configure:3598: result: configure:3620: checking whether we are cross compiling configure:3628: gcc o conftest g O2 L/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib Wl,rpath,/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib conftest.c lopenblas lm >&5 configure:3632: $? = 0 configure:3639: ./conftest dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/openblas0.2.19.p0/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/libopenblas_haswellr0.2.19.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/gsl2.3/src/./conftest Reason: image not found ./configure: line 3641: 5891 Abort trap: 6 ./conftest$ac_cv_exeext configure:3643: $? = 134 configure:3650: error: in `/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/gsl2.3/src': configure:3652: error: cannot run C compiled programs. If you meant to cross compile, use `host'.
I don't know why it seems to be looking for libopenblas_haswellr0.2.19.dylib
in the temporary directory; the command invoking ./configure
has the correct prefix
and libdir
.
For numpy
:
Copying numpy.egginfo to /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/numpy1.12.1py2.7.egginfo running install_scripts creating /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/bin copying build/scripts.macosx10.9x86_642.7/f2py2 > /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/bin changing mode of /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/bin/f2py2 to 755 Traceback (most recent call last): File "<stdin>", line 3, in <module> File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/numpy/__init__.py", line 142, in <module> from . import add_newdocs File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/numpy/add_newdocs.py", line 13, in <module> from numpy.lib import add_newdoc File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/numpy/lib/__init__.py", line 8, in <module> from .type_check import * File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/numpy/lib/type_check.py", line 11, in <module> import numpy.core.numeric as _nx File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/numpy1.12.1/inst/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/numpy/core/__init__.py", line 24, in <module> raise ImportError(msg) ImportError: Importing the multiarray numpy extension module failed. Most likely you are trying to import a failed build of numpy. If you're working with a numpy git repo, try `git clean xdf` (removes all files not under version control). Otherwise reinstall numpy.
comment:56 Changed 2 years ago by
I don't get any of these problems. I will probably need an OSX machine to investigate.
It could have something to do with the
LIBS="`pkgconfig libsonlyl cblas` lm"
passed to configure
in gsl's spkginstall. But that *shouldn't* be a problem.
comment:57 Changed 2 years ago by
 Commit changed from dbdb94788f6275e19c5fd150bec8a889188f3164 to ce38d0756f226ec9a87b55890e716edd80f0cbdc
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
4360f31  Added the ability for packages to have a postinstall script called spkgpostinst.

0972a41  Add special postinstall handling for the backports/__init__.py shared

41c73a3  Remove some nolonger used local variables

750943a  Add handling for packages that install .info files and update share/info/dir

53a9c47  Rename $SAGE_INST_TEMP to $SAGE_DESTDIR

7615714  Don't use the u option to cp. It's not portable to BSD's cp, and is

d4bb6f7  Just use 'cp a' which seems to be compatible across most cp's.

ed89571  Fix Python build on nonCygwin systems (namely OSX) with caseinsensitive filesystems

44228b2  On OSX DYLD_LIBRARY_PATH should be set a la LD_LIBRARY_PATH to run Python from its build directory

ce38d07  Work around bug in OpenBLAS's Makefile with its use of DESTDIR with

comment:58 Changed 2 years ago by
Rebased on the latest (positively reviewed) #23160.
I added the DYLD_LIBRARY_PATH
fix needed for Python installation.
Also added an untested possible fix for the gsl installation issue. I think the issue here was a bug in OpenBLAS's Makefile on OSX with how it uses install_name_tool
, and this should fix it but I'm not positive. Chances are this would also fix the problem with numpy which is probably trying to link (unsuccessfully) to openblas.
comment:59 Changed 2 years ago by
This works now, Sage builds and all tests pass.
comment:60 Changed 2 years ago by
Several packages fail their test suites. flint:
gcc fnocommon ansi pedantic Wall O2 funrollloops g mpopcnt c o test_helpers.o test_helpers.c make build/test/tadd_ssaaaa build/test/tadd_sssaaaaaa build/test/tcount_leading_zeros build/test/tcount_trailing_zeros build/test/tsdiv_qrnnd build/test/tsmul_ppmm build/test/tsub_ddmmss build/test/tudiv_qrnnd build/test/tudiv_qrnnd_preinv build/test/tumul_ppmm mkdir p build/test gcc fnocommon ansi pedantic Wall O2 funrollloops g mpopcnt I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/flint2.5.2.p1/src I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib Wl,rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib test/tadd_ssaaaa.c o build/test/tadd_ssaaaa L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/flint2.5.2.p1/src L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib lflint lmpfr lgmp lm lntl lpthread [snip] gcc fnocommon ansi pedantic Wall O2 funrollloops g mpopcnt I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/flint2.5.2.p1/src I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib Wl,rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib test/tudiv_qrnnd_preinv.c o build/test/tudiv_qrnnd_preinv L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/flint2.5.2.p1/src L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib lflint lmpfr lgmp lm lntl lpthread gcc fnocommon ansi pedantic Wall O2 funrollloops g mpopcnt I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/flint2.5.2.p1/src I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/include L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib Wl,rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib test/tumul_ppmm.c o build/test/tumul_ppmm L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/flint2.5.2.p1/src L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib lflint lmpfr lgmp lm lntl lpthread (BUILD_DIR=build; export BUILD_DIR; make f Makefile.subdirs C . check  exit$?;) dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib/libflint13.5.2.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/flint2.5.2.p1/src/build/test/tadd_ssaaaa Reason: image not found make[4]: *** [build/test/tadd_ssaaaa_RUN] Abort trap: 6 /bin/sh: exit2: command not found make[3]: *** [check] Error 127 Error: FLINT failed to pass its test suite.
arb:
Running the test suite for arb2.8.1.p1... make[4]: Nothing to be done for `shared'. make[5]: Nothing to be done for `shared'. CC ../build/fmpr/test/tabs_bound_le_2exp_fmpz CC ../build/fmpr/test/tabs_bound_lt_2exp_fmpz CC ../build/fmpr/test/tabs_bound_lt_2exp_si CC ../build/fmpr/test/tadd CC ../build/fmpr/test/tadd_naive CC ../build/fmpr/test/tcmp CC ../build/fmpr/test/tcmp_2exp_si CC ../build/fmpr/test/tcmpabs CC ../build/fmpr/test/tcmpabs_2exp_si CC ../build/fmpr/test/tdiv CC ../build/fmpr/test/tdivappr_abs_ubound CC ../build/fmpr/test/texp CC ../build/fmpr/test/texpm1 CC ../build/fmpr/test/tget_d CC ../build/fmpr/test/tget_fmpz CC ../build/fmpr/test/tget_mpfr CC ../build/fmpr/test/tlog CC ../build/fmpr/test/tlog1p CC ../build/fmpr/test/tmul CC ../build/fmpr/test/tmul_fmpz CC ../build/fmpr/test/tmul_naive CC ../build/fmpr/test/tmul_si CC ../build/fmpr/test/tmul_ui CC ../build/fmpr/test/tnormalise CC ../build/fmpr/test/troot CC ../build/fmpr/test/trsqrt CC ../build/fmpr/test/tset_fmpq CC ../build/fmpr/test/tset_fmpz_2exp CC ../build/fmpr/test/tset_round_mpn CC ../build/fmpr/test/tset_round_ui_2exp_fmpz CC ../build/fmpr/test/tset_round_uiui_2exp_fmpz CC ../build/fmpr/test/tsqrt CC ../build/fmpr/test/tsub CC ../build/fmpr/test/tsum CC ../build/fmpr/test/tulp dyld: Library not loaded: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib/libarb1.1.1.dylib Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/arb2.8.1.p1/src/fmpr/../build/fmpr/test/tabs_bound_le_2exp_fmpz Reason: image not found make[4]: *** [../build/fmpr/test/tabs_bound_le_2exp_fmpz_RUN] Abort trap: 6 make[3]: *** [check] Error 2
R:
Running the test suite for r3.4.0.p0... ../../bin/R: line 246: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib/R//etc/ldpaths: No such file or directory make[6]: *** [testExamplesBase] Error 1 make[5]: *** [testExamples] Error 2 make[4]: *** [testallbasics] Error 1 make[3]: *** [check] Error 2 Error while running the R testsuite ... exiting
Singular:
Running the test suite for singular4.1.0p3.p0... ./spkgcheck: line 33: Singular: command not found Error loading the freegb library in Singular.
pycrypto:
Running the test suite for pycrypto2.6.1.p2... PyCrypto will now be tested running test Traceback (most recent call last): File "setup.py", line 456, in <module> core.setup(**kw) File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 321, in run from Crypto import SelfTest ImportError: No module named Crypto Error: PyCrypto failed to pass its test suite.
cvxopt:
Running the test suite for cvxopt1.1.8.p2... Testing in /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/cvxopt1.1.8.p2/src/examples/doc/chap10 Testing l1svc.py ... Traceback (most recent call last): File "l1svc.py", line 3, in <module> from cvxopt import normal, setseed ImportError: No module named cvxopt Error: test /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage8.1.beta1/local/var/tmp/sage/build/cvxopt1.1.8.p2/src/examples/doc/chap10/l1svc.py failed
comment:61 Changed 2 years ago by
How are you running these? I don't know why it would work any differently.
comment:62 followup: ↓ 67 Changed 2 years ago by
I get failures from make distclean; export SAGE_CHECK=yes; make
. I get some (but not all) of these failures from ./sage f c PKG_NAME
; this is after unsetting SAGE_CHECK
and running make
. But once the package has been installed successfully, then ./sage f c PKG_NAME
may behave differently, because it may be using installed versions of libraries, binaries, etc.; if it hasn't been installed, it won't be installed until spkgcheck
finishes successfully.
comment:63 followup: ↓ 66 Changed 2 years ago by
In build/pkgs/widgetsnbextension/spkginstall
you write
# Note: Setting PYTHONPATH here is something of a hack to support # installation into $SAGE_DESTDIR. This might be better handled # as a postinstall step SITE_PACKAGES="$(sagepython23 c 'import site; print(site.getsitepackages()[0])')" PYTHONPATH="$SAGE_DESTDIR$SITE_PACKAGES" jupyter nbextension enable sysprefix py widgetsnbextension
I agree that this looks more like a postinstall step, so why didn't you implement it that way?
comment:64 followup: ↓ 70 Changed 2 years ago by
I am a bit lost where the DESTDIR
stuff for autoconf packages is actually implemented. For Python packages, I see that you added the root
option to PIP_INSTALL
.
comment:65 in reply to: ↑ 33 ; followup: ↓ 75 Changed 2 years ago by
Replying to embray:
This ticket otherwise fixes all examples among standard (and some optional packages) where files were overwriting each other.
What is the relevance of that for this ticket?
All those packages worked before. If such clobbering is no longer allowed, I consider that a regression introduced by this ticket.
comment:66 in reply to: ↑ 63 Changed 2 years ago by
Replying to jdemeyer:
I agree that this looks more like a postinstall step, so why didn't you implement it that way?
When I first wrote that, support for postinstall scripts wasn't added yet. Now that it's there, I could probably revisit things like that. Though I'd have to review what exactly jupyter nbextension enable
does. If it just writes some files then this might be fine.
comment:67 in reply to: ↑ 62 Changed 2 years ago by
Replying to jhpalmieri:
I get failures from
make distclean; export SAGE_CHECK=yes; make
. I get some (but not all) of these failures from./sage f c PKG_NAME
; this is after unsettingSAGE_CHECK
and runningmake
. But once the package has been installed successfully, then./sage f c PKG_NAME
may behave differently, because it may be using installed versions of libraries, binaries, etc.; if it hasn't been installed, it won't be installed untilspkgcheck
finishes successfully.
I'm having trouble building python3 with SAGE_CHECK=yes
. Did that ever work in the first place?
comment:68 Changed 2 years ago by
In my experience, neither python2
nor python3
passed their self tests. With python2
at least, this was true on a range of platforms. We should probably change the default value of SAGE_CHECK_PACKAGES
to exclude python3
; it already excludes python2
.
Edit: I created #23612.
comment:69 Changed 2 years ago by
On the topic of the other SAGE_CHECK failures, the problem is probably similar to the problems I was having with Python: The checks are being performed before the files are installed from SAGE_DESTDIR
into SAGE_LOCAL
, so at the very least it might be necessary to set LD_LIBRARY_PATH/DYLIB_LIBRARY_PATH
, but possibly other paths as well?
Rather than trying to fix every spkgcheck
, which may be difficult especially if I can't reproduce all the problems, it might be better for the sake of this ticket if I moved execution of spkgcheck
until after installation to SAGE_LOCAL
. This has the downside that the package still gets installed even if the check failed, but that's the existing situation anyways. Once #22510 is done (which depends on this) it will be easier to do things like make a backup of an old packagetry installing the new oneand revert to the old one if the installation failed.
comment:70 in reply to: ↑ 64 Changed 2 years ago by
Replying to jdemeyer:
I am a bit lost where the
DESTDIR
stuff for autoconf packages is actually implemented. For Python packages, I see that you added theroot
option toPIP_INSTALL
.
Just realized I never replied to this. DESTDIR=
is added in the helper function sdh_make_install
(which is partly why I wanted that feature implemented for this ticket). Note that it assumes SAGE_DESTDIR
is set, which it may not be if the spkginstall
is not being run from sagespkg
. It might be worth noting that this is intentionalif SAGE_DESTDIR
is left blank then direct installation to SAGE_LOCAL
proceeds as normal, which is important and useful for testing, if nothing else.
Otherwise, SAGE_DESTDIR
is set in sagespkg
, and gets passed to make install
, in most cases, by way of sdh_make_install
.
comment:71 Changed 2 years ago by
For some reason on Linux I couldn't reproduce the problems running the tests for flint or arb (despite running make distclean
first). But I did get the same problems for the other packages. As I wrote here it seems to be mostly a matter of setting the required environment variables. But I think I'll still revert to the old behavior for SAGE_CHECK for now, just to keep this simpler.
comment:72 Changed 2 years ago by
 Commit changed from ce38d0756f226ec9a87b55890e716edd80f0cbdc to 70013431e8c8a4ac4f49191378aea54e006b1a4b
comment:73 Changed 2 years ago by
Just remembered there was still unfinished work on this that I hadn't pushed yet. I think now it's addressed all the issues that have been raised so far.
comment:74 followup: ↓ 78 Changed 2 years ago by
One concern with this patch is that it's really a patch bomb, which might be hard to get reviewed and merged.
I don't think you have answered 65
comment:75 in reply to: ↑ 65 Changed 2 years ago by
I thought I addressed this somewhere but maybe not...
Replying to jdemeyer:
Replying to embray:
This ticket otherwise fixes all examples among standard (and some optional packages) where files were overwriting each other.
What is the relevance of that for this ticket?
All those packages worked before. If such clobbering is no longer allowed, I consider that a regression introduced by this ticket.
I wouldn't consider it a regression at all. The fact that some packages overwrote files installed by other packages is a bug, just one that hasn't had very serious effects. Most cases of this were very minor (e.g. python3 overwrote the valgrind.supp installed by python2).
The goal of this ticket is, along with #22510 (where this becomes important) is to make sagedist work a little more like a real OS packaging system in that files are belonged by packages, with any one file belonging to just one package. That way, uninstalling one package does not break another package that owns some of the same file(s).
In theory one could design a packaging system where a file can belong to multiple packages, and only be removed once all packages that use it are removed. But this is more complicated, and only works if that file is identical for all packages that use it (it may not be).
There is one particular example of a file shared between packages where this ticket would introduce a regression if it didn't carefully handle this case: There's a file info/share/dir
that is updated by any package that installs an info
manual. The new way packages are installed by ticket is such that each package will write a fresh info/share/dir
, and then overwrite the existing one, rather than updating the existing one. This makes sure to always update info/share/dir
in a postinstall script for those packages.
Other than that, the other cases are mostly trivial.
comment:76 Changed 2 years ago by
 Commit changed from 70013431e8c8a4ac4f49191378aea54e006b1a4b to 53e3dc54db7658c6eceb50d573eeb324d886a532
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
0275058  On OSX DYLD_LIBRARY_PATH should be set a la LD_LIBRARY_PATH to run Python from its build directory

8e193db  Move spkgcheck to after the package has been installed to SAGE_LOCAL.

53e3dc5  Enable widgetsnbextension in the notebook's config file as a postinst step.

comment:77 Changed 2 years ago by
Had to rebase after I realized I pop'd the completely wrong stash entry into an earlier commit.
comment:78 in reply to: ↑ 74 ; followup: ↓ 82 Changed 2 years ago by
Replying to jdemeyer:
One concern with this patch is that it's really a patch bomb, which might be hard to get reviewed and merged.
True, but there's not much other way to handle it (I don't mind separating some changes out to separate tickets but some of them go handinhand with this). John did test this extensively in OSX and got it working. The only remaining concern was that some spkgcheck
tests weren't working. I believe I've addressed that now, though it's hard to be 100% sure since some of those tests aren't working for me with or without this patch.
I don't think you have answered 65
Done.
comment:79 Changed 2 years ago by
 Dependencies changed from #23059 #23096 #23160 to #23059 #23096 #23160 #21726
In an effort to split this up a little bit, I'm moving some of the fixes for the Python2/3 packages to a separate ticket, since it can work independently of this ticket: #23781
comment:80 Changed 2 years ago by
 Dependencies changed from #23059 #23096 #23160 #21726 to #23059 #23096 #23160 #23781
Wrong ticket.
comment:81 Changed 2 years ago by
 Commit changed from 53e3dc54db7658c6eceb50d573eeb324d886a532 to f97fdf916d7a58006d1be37c419180c8c8ee1893
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
71cb02d  Added the ability for packages to have a postinstall script called spkgpostinst.

013e6f3  Add special postinstall handling for the backports/__init__.py shared

b886725  Remove some nolonger used local variables

f94b3ec  Add handling for packages that install .info files and update share/info/dir

f4960ea  Rename $SAGE_INST_TEMP to $SAGE_DESTDIR

36ff6ca  Don't use the u option to cp. It's not portable to BSD's cp, and is

25c4d01  Just use 'cp a' which seems to be compatible across most cp's.

8c9a60f  Work around bug in OpenBLAS's Makefile with its use of DESTDIR with

1389380  Move spkgcheck to after the package has been installed to SAGE_LOCAL.

f97fdf9  Enable widgetsnbextension in the notebook's config file as a postinst step.

comment:82 in reply to: ↑ 78 Changed 2 years ago by
Replying to embray:
Replying to jdemeyer:
One concern with this patch is that it's really a patch bomb, which might be hard to get reviewed and merged.
True, but there's not much other way to handle it
Why do you think that? This doesn't have to be an allornothing patch: you could first implement the general framework and then gradually fix packages.
Anyway, if you find somebody else to review this, I don't mind at all. But for me, I am quite discouraged from reviewing such a big patch. And then there is also the potential for merge conflicts if you make so many changes.
comment:83 followup: ↓ 84 Changed 2 years ago by
I'm not sure how you would propose breaking this up. It is pretty allornothing: Either DESTDIR installation works or it doesn't.
There's no sense in "gradually fix packages" because they've already all been fixed here, and in a way that is likely to result in only minor conflicts at worse (no more so than in #23179the analysis in this comment still mostly applies).
I could break it up into multiple ticketsI already did in one case. But for the most part I don't see what that solves. If you want guidance on what to review, start by ignoring all the build/pkgs
changes since the vast majority of them are completely mechanistic.
comment:84 in reply to: ↑ 83 Changed 2 years ago by
Replying to embray:
I'm not sure how you would propose breaking this up. It is pretty allornothing: Either DESTDIR installation works or it doesn't.
You could start with "DESTDIR
works for a few selected packages" or "DESTDIR
works for Python packages only"
in a way that is likely to result in only minor conflicts at worse
A merge conflict is a merge conflict. It doesn't matter how minor it is.
comment:85 followup: ↓ 86 Changed 2 years ago by
Alright, that's fair. I'll see if I can break this up a bit more.
Fact remains, there's not much benefit to this unless all packages are updated. There's no real logical way I can think of to carving it up into multiple tickets, when it can all be done at once. But for the sake of making review easier I'll try out your suggestion...
comment:86 in reply to: ↑ 85 ; followup: ↓ 88 Changed 2 years ago by
Replying to embray:
I'll see if I can break this up a bit more.
Let me mention that this is a personal "pet peeve". If somebody else (John Palmieri for example) is happy with this ticket and wants to review it, I won't object to that.
comment:87 Changed 2 years ago by
I'm not sure I understand the build process well enough to be competent enough to review it.
comment:88 in reply to: ↑ 86 ; followup: ↓ 89 Changed 2 years ago by
Replying to jdemeyer:
Replying to embray:
I'll see if I can break this up a bit more.
Let me mention that this is a personal "pet peeve". If somebody else (John Palmieri for example) is happy with this ticket and wants to review it, I won't object to that.
It's fineI see your point. I think it's easier to review if one patch contains just:
 The basic structural changes
 A few motivating examples for the different cases
and then leave the remaining boilerplate changes to a separate ticket.
comment:89 in reply to: ↑ 88 Changed 2 years ago by
comment:90 followup: ↓ 91 Changed 2 years ago by
I'm not exactly sure what #21535 has to do with this.
comment:91 in reply to: ↑ 90 Changed 2 years ago by
comment:92 Changed 2 years ago by
 Commit changed from f97fdf916d7a58006d1be37c419180c8c8ee1893 to c1d5453c2735cdda78d53dc36b56fb54ec7dd336
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
be70ddb  Added the ability for packages to have a postinstall script called spkgpostinst.

e246937  Add special postinstall handling for the backports/__init__.py shared

87a02ac  Remove some nolonger used local variables

5927318  Add handling for packages that install .info files and update share/info/dir

73245de  Rename $SAGE_INST_TEMP to $SAGE_DESTDIR

a771cd8  Don't use the u option to cp. It's not portable to BSD's cp, and is

5125047  Just use 'cp a' which seems to be compatible across most cp's.

3e8e7b5  Work around bug in OpenBLAS's Makefile with its use of DESTDIR with

58676b4  Move spkgcheck to after the package has been installed to SAGE_LOCAL.

c1d5453  Enable widgetsnbextension in the notebook's config file as a postinst step.

comment:93 followup: ↓ 95 Changed 2 years ago by
Went ahead and rebased on current #23781. But I'm now I'm going to do some surgery to break this into some smaller tickets for which this ticket will serve as the mothership.
comment:94 Changed 2 years ago by
 Description modified (diff)
 Status changed from needs_review to needs_work
comment:95 in reply to: ↑ 93 Changed 2 years ago by
comment:96 Changed 2 years ago by
Uhh, it does need to in a manner of speaking. Eventually the Python build script needs to be updated to support this. But I'm already working on completely breaking this up in to separate tickets.
comment:97 Changed 2 years ago by
 Description modified (diff)
Added #24106a simplified ticket demonstrating the core functionality of the branch on this ticket. Will add followup tickets with additional details.
comment:98 Changed 2 years ago by
 Milestone changed from sage8.1 to sage8.2
comment:99 Changed 23 months ago by
 Description modified (diff)
comment:100 Changed 23 months ago by
 Description modified (diff)
comment:101 Changed 23 months ago by
comment:102 Changed 23 months ago by
see 93
comment:103 in reply to: ↑ 28 Changed 21 months ago by
comment:104 Changed 21 months ago by
Yes, there was a reason: https://trac.sagemath.org/ticket/22509#comment:30
Though I suppose if $SAGE_DESTDIR
isn't set then it's fine if $SAGE_DESTDIR_LOCAL == $SAGE_LOCAL
.
comment:105 Changed 20 months ago by
 Milestone changed from sage8.2 to sage8.3
comment:106 Changed 17 months ago by
 Milestone changed from sage8.3 to sage8.4
I believe this issue can reasonably be addressed for Sage 8.4.
comment:107 Changed 14 months ago by
 Milestone changed from sage8.4 to sage8.5
comment:108 Changed 12 months ago by
 Milestone changed from sage8.5 to sage8.7
Retargeting some of my tickets (somewhat optimistically for now).
comment:109 Changed 9 months ago by
 Milestone changed from sage8.7 to sage8.8
Moving all my inprogress tickets to 8.8 milestone.
comment:110 Changed 6 months ago by
 Milestone changed from sage8.8 to sage8.9
Tickets still needing working or clarification should be moved to the next release milestone at the soonest (please feel free to revert if you think the ticket is close to being resolved).
Started work on this, and converted I think all standard packages to work with a staged installation (install first to a temporary directory, then copy from the temp directory into
$SAGE_LOCAL
).As I wrote in the description, most packages are easyeither they use automake, or at least a Makefile that's compatible with the
DESTDIR
convention, or they are Python packages in which case theroot
option does the same thing. A handful of packages needed handtweaking.This is currently implemented in such a way that if a package doesn't yet support staged install, it will still install just fine. The only major downside is that the installed files are not recorded yet, so they wouldn't be able to take advantage of improved uninstall.
That said, I'm going to continue converting the remaining packages. My only question is whether to continue work in this ticket, or separate the general changes (i.e. to
sagespkg
) from the updates to the packages themselves.Last 10 new commits:
Update pip install to use root
Update freetype and pkgconf to use DESTDIR
Updated various packages with nonstandard installers or that are otherwise
Update cephes to work with staged install.
Update openblas to use staged install
Update lcalc to support staged install
Update zn_poly, which doesn't really have a proper installer, to use staged install
Roll back accidental change from dc46ea2dfa97be79d7ef8ddbe9130d4c717acc3d that broke build of Python 2 on Cygwin.
Merge remotetracking branch 'upstream/u/embray/cygwin/ticket23059' into u/embray/build/destdir
Updated the python2/3 pacakges to use staged install.