Opened 8 years ago
Closed 8 years ago
#16017 closed defect (fixed)
Get libs.mwrank to compile on OS X 10.4
Reported by: | kcrisman | Owned by: | |
---|---|---|---|
Priority: | blocker | Milestone: | sage-6.2 |
Component: | packages: standard | Keywords: | Darwin 8 linker error install_name libflint.dylib |
Cc: | leif, vbraun, jpflori | Merged in: | |
Authors: | François Bissey | Reviewers: | Karl-Dieter Crisman, Volker Braun |
Report Upstream: | N/A | Work issues: | |
Branch: | 6c25897 (Commits, GitHub, GitLab) | Commit: | 6c25897014779846c65035baca4b0110bf6a056e |
Dependencies: | Stopgaps: |
Description (last modified by )
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include/python2.7 -c sage/libs/mwrank/mwrank.c -o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o -fno-strict-aliasing -w gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include/python2.7 -c sage/libs/mwrank/wrap.cc -o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -fno-strict-aliasing -w g++ -bundle -undefined dynamic_lookup -L/Users/student/Desktop/sage-6.2.beta4/local/lib build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -L/Users/student/Desktop/sage-6.2.beta4/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.so /usr/bin/ld: warning can't open dynamic library: libflint.dylib referenced from: /Users/student/Desktop/sage-6.2.beta4/local/lib/libec.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2) /usr/bin/ld: Undefined symbols: _nmod_mat_clear referenced from libec expected to be defined in libflint.dylib _nmod_mat_init referenced from libec expected to be defined in libflint.dylib _nmod_mat_rref referenced from libec expected to be defined in libflint.dylib collect2: error: ld returned 1 exit status error: command 'g++' failed with exit status 1
See lots of discussion at this sage-release thread. Best guesses are either this Flint bug and leif's idea that there is a missing linker flag on Darwin (see the sage-release thread).
This is not eclib's fault, but due to incomplete / wrong "install_name
s" (meant to be absolute paths) in libflint.dylib (and also libpari-gmp.dylib).
On Darwin 10 (MacOS X 10.6) and later, apparently the linker is smarter (or more tolerant), and does find the indirectly used libraries in $SAGE_LOCAL/lib/
, despite their wrong install_name
s.
We can solve this by upgrading flint to 2.4.3 which includes the necessary fix.
New sources:
Change History (60)
comment:1 Changed 8 years ago by
- Cc jpflori added
comment:2 Changed 8 years ago by
Also required for sage.libs.cremona extensions. Given that on Cygwin we will probably need this too, I'm suggesting that this is the right solution now.
comment:3 Changed 8 years ago by
I'm pretty curious whether -- with the "superfluous" -lflint
added -- you're still getting the weird
/usr/bin/ld: warning can't open dynamic library: libflint.dylib referenced from: /Users/student/Desktop/sage-6.2.beta4/local/lib/libec.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)
The addition is IMHO just a work-around (on Darwin 8; still probably necessary on Cygwin); it doesn't address the root of the problem (provided we can trust the linker messages). (Cf. my recent comment on the Sage 6.2.beta4 thread on sage-release.)
comment:4 Changed 8 years ago by
... in other words:
Where the heck does the linker try to find the libflint.dylib
libec.dylib
refers to, and does libec.dylib
really refer to $SAGE_ROOT/local/lib/libflint.dylib
(modulo symlinks to versioned versions of the libraries)?
Could Apple's otool
enlighten us? (Not sure whether passing -t
to the Darwin linker would.)
comment:5 Changed 8 years ago by
No, no warnings if I use that. I really think it just doesn't know where to find it. This happened on Cygwin all the time, like I said. Yes, it's certainly just a workaround, but one that I can't really see a downside to. I have no idea where it is looking, and I have pretty much run out of energy on this now that I found what I think it is an acceptable solution. Not that I don't *want* a better one, but my sense is there may not be one that doesn't involve special casing a -lflint
for old Macs for no particular reason, or that doesn't involve asking John to do something annoying with eclib.
comment:6 Changed 8 years ago by
Try using https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/flint/files/flint-2.4-makefile.patch in the patches folder for flint. Then in spkg-install
export SHARE_FLAGS="-install_name ${SAGE_LOCAL}/lib/libflint.dylib"
before $MAKE Rebuild flint, eclib and see if that works. If it does we'll put that in a better shape.
comment:7 Changed 8 years ago by
- Priority changed from major to blocker
Thanks, that sounds fine too if it works, but I won't be able to try this until Monday.
Since we have several workarounds now, and since (for now) this is still a supported platform, I'm making this a blocker for 6.2.
comment:8 follow-up: ↓ 10 Changed 8 years ago by
leif@bsd:~/Sage/sage-6.2.beta5$ otool -DL local/lib/libec.dylib local/lib/libec.dylib: /Users/leif/Sage/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0) libflint.dylib (compatibility version 0.0.0, current version 0.0.0) /Users/leif/Sage/sage-6.2.beta5/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0) /Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5) /Users/leif/Sage/sage-6.2.beta5/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11) /Users/leif/Sage/sage-6.2.beta5/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
Interesting... (Note the recorded path to the PARI library!)
But we (also) already have -lpari
on the command line for linking the mwrank Python extension module (which I think by the way shouldn't be necessary either, as it isn't directly used IIRC):
g++ -bundle -undefined dynamic_lookup -L/Users/leif/Sage/sage-6.2.beta5/local/lib build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.so
And FWIW, in the build on bsd.math, libtool
also drops the -rpath
given to it when linking libec.dylib
(or, more precisely, libec.0.dylib
).
comment:9 Changed 8 years ago by
For completeness, this is how libec.0.dylib
gets linked on bsd.math (looks the same as on Karl-Dieter's MacOS X 10.4 PPC box):
/bin/sh ../libtool --tag=CXX --mode=link g++ -g -O3 -no-undefined -L/Users/leif/Sage/sage-6.2.beta5/local/lib -L/Users/leif/Sage/sage-6.2.beta5/local/lib -L/Users/leif/Sage/sage-6.2.beta5/local/lib -o libec.la -rpath /Users/leif/Sage/sage-6.2.beta5/local/lib interface.lo unimod.lo arith.lo marith.lo gpslave.lo compproc.lo vector.lo matrix.lo subspace.lo kbessel.lo mvector.lo mmatrix.lo msubspace.lo svector.lo smatrix.lo smatrix_elim.lo xsplit.lo conic.lo legendre.lo quadratic.lo illl.lo hilbert.lo timer.lo cubic.lo polys.lo realroots.lo parifact.lo p2points.lo gf.lo xsplit_data.lo threadpool.lo logger.lo curve.lo curvedata.lo curvered.lo points.lo cperiods.lo isogs.lo heights.lo mwprocs.lo lambda.lo sifter.lo sieve_search.lo htconst.lo egr.lo saturate.lo divpol.lo pointsmod.lo curvemod.lo ffmod.lo tlss.lo elog.lo getcurve.lo mequiv.lo mrank1.lo mlocsol.lo mglobsol.lo mquartic.lo mrank2.lo qc.lo sqfdiv.lo version.lo minim.lo reduce.lo transform.lo desc2.lo bitspace.lo GetOpt.lo twoadic.lo descent.lo newforms.lo symb.lo homspace.lo cusp.lo oldforms.lo fixc6.lo periods.lo moddata.lo pcprocs.lo nfd.lo curvesort.lo -lflint -lntl -lpari libtool: link: g++ -dynamiclib -o .libs/libec.0.dylib .libs/interface.o .libs/unimod.o .libs/arith.o .libs/marith.o .libs/gpslave.o .libs/compproc.o .libs/vector.o .libs/matrix.o .libs/subspace.o .libs/kbessel.o .libs/mvector.o .libs/mmatrix.o .libs/msubspace.o .libs/svector.o .libs/smatrix.o .libs/smatrix_elim.o .libs/xsplit.o .libs/conic.o .libs/legendre.o .libs/quadratic.o .libs/illl.o .libs/hilbert.o .libs/timer.o .libs/cubic.o .libs/polys.o .libs/realroots.o .libs/parifact.o .libs/p2points.o .libs/gf.o .libs/xsplit_data.o .libs/threadpool.o .libs/logger.o .libs/curve.o .libs/curvedata.o .libs/curvered.o .libs/points.o .libs/cperiods.o .libs/isogs.o .libs/heights.o .libs/mwprocs.o .libs/lambda.o .libs/sifter.o .libs/sieve_search.o .libs/htconst.o .libs/egr.o .libs/saturate.o .libs/divpol.o .libs/pointsmod.o .libs/curvemod.o .libs/ffmod.o .libs/tlss.o .libs/elog.o .libs/getcurve.o .libs/mequiv.o .libs/mrank1.o .libs/mlocsol.o .libs/mglobsol.o .libs/mquartic.o .libs/mrank2.o .libs/qc.o .libs/sqfdiv.o .libs/version.o .libs/minim.o .libs/reduce.o .libs/transform.o .libs/desc2.o .libs/bitspace.o .libs/GetOpt.o .libs/twoadic.o .libs/descent.o .libs/newforms.o .libs/symb.o .libs/homspace.o .libs/cusp.o .libs/oldforms.o .libs/fixc6.o .libs/periods.o .libs/moddata.o .libs/pcprocs.o .libs/nfd.o .libs/curvesort.o -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lflint /Users/leif/Sage/sage-6.2.beta5/local/lib/libntl.dylib -L/Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/gcc-4.7.3.p1/gcc-build/x86_64-apple-darwin10.8.0/libstdc++-v3/src -L/Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/gcc-4.7.3.p1/gcc-build/x86_64-apple-darwin10.8.0/libstdc++-v3/src/.libs -lpari -O3 -install_name /Users/leif/Sage/sage-6.2.beta5/local/lib/libec.0.dylib -compatibility_version 1 -current_version 1.0 -Wl,-single_module
So indeed the (missing or weird) install_name
s seem to be the problem. (Or libtool
... ;-) )
comment:10 in reply to: ↑ 8 Changed 8 years ago by
Replying to leif:
(Note the recorded path to the PARI library!)
But we (also) already have
-lpari
on the command line for linking the mwrank Python extension module (which I think by the way shouldn't be necessary either, as it isn't directly used IIRC) [...]
And if I remove pari
from the allegedly needed libraries of the mwrank Python extension module (in module_list.py
), this also works on bsd.math:
leif@bsd:~/Sage/sage-6.2.beta5$ touch src/sage/libs/mwrank/mwrank.pyx leif@bsd:~/Sage/sage-6.2.beta5$ ./sage -b scons: `install' is up to date. Updating Cython code.... Compiling sage/libs/mwrank/mwrank.pyx because it changed. Cythonizing sage/libs/mwrank/mwrank.pyx Finished compiling Cython code (time = 4.62841916084 seconds) running install running build running build_py running build_ext building 'sage.libs.mwrank.mwrank' extension Executing 1 command (using 1 thread) gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include/python2.7 -c sage/libs/mwrank/mwrank.c -o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o -fno-strict-aliasing -w gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include/python2.7 -c sage/libs/mwrank/wrap.cc -o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -fno-strict-aliasing -w g++ -bundle -undefined dynamic_lookup -L/Users/leif/Sage/sage-6.2.beta5/local/lib build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lcsage -lec -lntl -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.so Time to execute 1 command: 4.69843101501 seconds Total time spent compiling C/C++ extensions: 4.88040590286 seconds.
I guess doing so would trigger another "no such file or directory" warning on Karl-Dieter's machine. MacOS X 10.6's linker is simply "smarter"...
comment:11 Changed 8 years ago by
So we may also want to look at pari because it put the build time location rather than final location in install name. Not a problem if you tell sage to keep the build stuff I guess.
comment:12 follow-up: ↓ 13 Changed 8 years ago by
I actually have a patch for pari's installname that I made for pari 2.5.0 and it is still in the gentoo and sage-on-gentoo trees, I guess I should try to it merged in pari proper. https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/pari/files/pari-2.5.0-macos.patch As for Karl-Dieter's problem we should be able to see if fixing the installname solves his problem without rebuilding flint and eclib. we can use
install_name_tool -change "oldname" "newname" filename
We can alter libflint.dylib and libec.dylib with the appropriate values and see if sage compile.
comment:13 in reply to: ↑ 12 Changed 8 years ago by
Replying to fbissey:
I actually have a patch for pari's installname that I made for pari 2.5.0 and it is still in the gentoo and sage-on-gentoo trees, I guess I should try to it merged in pari proper. https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/pari/files/pari-2.5.0-macos.patch As for Karl-Dieter's problem we should be able to see if fixing the installname solves his problem without rebuilding flint and eclib. we can use
install_name_tool -change "oldname" "newname" filenameWe can alter libflint.dylib and libec.dylib with the appropriate values and see if sage compile.
Yep, but he'd have to rebuild eclib (after changing libflint's [and perhaps libpari's] install_name
) such that the proper dependency name(s) get(s) recorded in libec.0.dylib
; I don't think you can change these with install_name_tool
.
Any idea why libtool
ignores -rpath
(on both MacOS X 10.4 and 10.6)?
comment:14 follow-up: ↓ 15 Changed 8 years ago by
First yes you can.
Mirage:lib fbissey$ otool -L libec.0.dylib libec.0.dylib: /Users/fbissey/build/sage-6.2.beta4/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0) libflint.dylib (compatibility version 0.0.0, current version 0.0.0) /Users/fbissey/build/sage-6.2.beta4/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0) /Users/fbissey/build/sage-6.2.beta4/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5) /Users/fbissey/build/sage-6.2.beta4/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1) /Users/fbissey/build/sage-6.2.beta4/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0) Mirage:lib fbissey$ install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib Mirage:lib fbissey$ otool -L libec.0.dylib libec.0.dylib: /Users/fbissey/build/sage-6.2.beta4/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0) /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib (compatibility version 0.0.0, current version 0.0.0) /Users/fbissey/build/sage-6.2.beta4/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0) /Users/fbissey/build/sage-6.2.beta4/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5) /Users/fbissey/build/sage-6.2.beta4/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1) /Users/fbissey/build/sage-6.2.beta4/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0) Mirage:lib fbissey$
About libtool, not sure. I always felt rpath was redundant with the install_name mechanism but that's not really an answer to your question.
comment:15 in reply to: ↑ 14 Changed 8 years ago by
Replying to fbissey:
First yes you can.
Cool. \o/
About libtool, not sure. I always felt rpath was redundant with the install_name mechanism but that's not really an answer to your question.
Well, by default install_name
is an absolute path. You can prepend @rpath/
(and other flavours) to the install_name
to make it relative to paths set by -rpath
, similar to $ORIGIN
in an ELF object's DT_RUNPATH
. But I'm not a Darwin linker (nor Mach-O) expert; I'd expect libtool
to know how to do it though. On the other hand, libtool
did convert some of the -L/path/to/foo/ -lfoo
to absolute filenames on the linker command line (when linking libec.0.dylib
), presumably exactly [to] those with a correct, absolute install_name
.
comment:16 follow-up: ↓ 18 Changed 8 years ago by
Yikes! As long as I have a specific thing to try out on Monday I'll be happy - if you guys can agree on what I should try, I don't have infinite time with that machine ;-)
comment:17 follow-up: ↓ 21 Changed 8 years ago by
Try the equivalent for your machine of:
install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libflint.dylib
in $SAGE_LOCAL/lib. Don't forget to replace the path for SAGE_LOCAL with what's correct for you. Once that's done you can try to build sage again and see if it will go past that point. If it does we'll make the appropriate patch.
comment:18 in reply to: ↑ 16 Changed 8 years ago by
Replying to kcrisman:
Yikes! As long as I have a specific thing to try out on Monday I'll be happy - if you guys can agree on what I should try, I don't have infinite time with that machine ;-)
There are plenty of things you could try... :-)
But I think kiwifb and me agree on that you could first try to:
- Restore
module_list.py
(at least removeflint
, optionally alsopari
) from the "needed" libraries of the libs.mwrank Python extension module -- to test whether it works for one of the affected modules (you had to change about five IIRC)
- Make sure libflint's
install_name
is still justlibflint.dylib
(otherwiseinstall_name_tool
will give an error):$ cd $SAGE_ROOT $ otool -D local/lib/libflint.dylib local/lib/libflint.dylib: libflint.dylib
- Change the
install_name
(here called "id") to the library's absolute path:$ install_name_tool -id `pwd`/local/lib/libflint.dylib local/lib/libflint.dylib
- Then either rebuild/reinstall eclib (which should now record libflint's absolute path), or -- simpler -- just change the recorded name (for FLINT) in
libec.0.dylib
manually:$ otool -L local/lib/libec.0.dylib # just to check libflint's name there local/lib/libec.0.dylib: <path to>/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0) libflint.dylib (compatibility version 0.0.0, current version 0.0.0) ... $ install_name_tool -change libflint.dylib `pwd`/local/lib/libflint.dylib local/lib/libec.0.dylib # now really change the name $ otool -L local/lib/libec.0.dylib # just to verify it changed local/lib/libec.0.dylib: <path to>/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0) <path to>/sage-6.2.beta5/local/lib/libflint.dylib (compatibility version 0.0.0, current version 0.0.0) ...
- Pretend file modification to trigger the rebuild of the module(s):
$ touch src/sage/libs/mwrank/mwrank.pyx # and more if you like $ ./sage -b # and now rebuild
The linker should now be able to locate libflint.dylib
referenced by libec.dylib
.
Optionally, repeat the steps analogously for pari
(which in contrast is originally listed as a needed lib in module_list.py
, so you'd have to really remove it rather than restoring the state of vanilla beta4 or beta5); just libpari's original install_name
is a bit more complicated (with a non-existing temporary path):
$ otool -D local/lib/libpari-gmp.dylib local/lib/libpari-gmp.dylib: <path to>/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib
(libpari.dylib
is a symbolic link to libpari-gmp.dylib
.)
When you change the name (again in libec.0.dylib
), you have to specify the full original (invalid) path:
$ install_name_tool -change /Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib `pwd`/local/lib/libpari-gmp.dylib local/lib/libec.0.dylib
(Just as an example -- note that the invalid path will differ on a PowerPC, and of course your $SAGE_ROOT
.)
comment:19 Changed 8 years ago by
Just for the record: I started writing before François replied... 8)
But we (still) agree on what you could try.
comment:20 Changed 8 years ago by
Yes we do. Just using install_name_tool means that we can test that theory without rebuilding flint and eclib, which would take some time on that machine.
comment:21 in reply to: ↑ 17 Changed 8 years ago by
Replying to fbissey:
Try the equivalent for your machine of:
install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libflint.dylibin $SAGE_LOCAL/lib. Don't forget to replace the path for SAGE_LOCAL with what's correct for you.
Note that the second line wouldn't work (it's a no-op).
If you want to change the install_name
of the library itself (as opposed to the recorded install names of libraries it uses), you have to use the -id
option of install_name_tool
, which takes just the new name (plus the filename of the library to change of course).
comment:22 Changed 8 years ago by
Well spotted. I was a bit too eager there and I had completely forgotten about that important detail.
comment:23 Changed 8 years ago by
For the record, @rpath
is OSX 10.5 and later only afaik. So absolute paths it is.
comment:24 Changed 8 years ago by
Ok, I'm trying this. Thank you leif for the VERY explicit steps, which start to help me understand even what these files even are.
The otool
output was fine, and I decided to have it rebuild eclib just to make sure that worked fine. I already reverted module_list.py
and touch
ed the files so I'll let you know what happens when it's done. I'm not messing with the pari
issue, though :)
comment:25 Changed 8 years ago by
Okay, this does indeed seem to do the trick. I can't run doctests, though, because sympy isn't built yet. Weird.
comment:26 Changed 8 years ago by
Ok, I had to install sympy AND mpmath for Sage to even start - which seems odd, they shouldn't be imported on startup? Anyway, only sage/libs/cremona/newforms.pyx doesn't pass tests, but that is because the Cremona mini database apparently never got installed, unrelated, I assume. (Or is that part of the eclib package?)
comment:27 Changed 8 years ago by
I am fairly sure it is in database_cremona_ellcurve.
comment:28 follow-up: ↓ 30 Changed 8 years ago by
Actually, it ended up being installed via the elliptic_curve
spkg, which only installed later... so that should be fine as well.
Now I guess we just need a "good" way to implement your ideas, which it sounds like you all have.
comment:29 Changed 8 years ago by
In the first instance what I suggested earlier would work but I want to put it in form that's more palatable to flint upstream so they can just merge it (and I'll get rid of my patch in sage-on-gentoo).
I need to take another look at their configure script.
comment:30 in reply to: ↑ 28 ; follow-up: ↓ 33 Changed 8 years ago by
- Keywords Darwin 8 linker error install_name libflint.dylib added
Replying to kcrisman:
Now I guess we just need a "good" way to implement your ideas, which it sounds like you all have.
Which I think means fixing the FLINT (and optionally also the PARI) spkg (on Darwin at least). I don't know whether it's upstream's or probably Sage's fault, but François presumably knows, as he mentioned he had patches for both.
comment:31 follow-up: ↓ 32 Changed 8 years ago by
P.S.: Does anyone currently test on Cygwin?
comment:32 in reply to: ↑ 31 ; follow-up: ↓ 36 Changed 8 years ago by
Replying to leif:
P.S.: Does anyone currently test on Cygwin?
I could do that.
IIRC I built 6.2.beta4 without problems (except bzip2 unrelated issues) so maybe nothing is actully needed? When was eclib updated?
comment:33 in reply to: ↑ 30 ; follow-up: ↓ 34 Changed 8 years ago by
Replying to leif:
Replying to kcrisman:
Now I guess we just need a "good" way to implement your ideas, which it sounds like you all have.
Which I think means fixing the FLINT (and optionally also the PARI) spkg (on Darwin at least). I don't know whether it's upstream's or probably Sage's fault, but François presumably knows, as he mentioned he had patches for both.
Flint upstream doesn't set anything for install_name and it is straightforward to add it, I'll submit a pull request for that to them this morning hopefully. Not sure how to forward that kind of patch to pari upstream, but again they don't do anything particular and end with something that comes as default with their linking options. I posted a link to the patch in #comment:12 if we want to include it. No extra actions needed in spkg-install just apply the patch.
comment:34 in reply to: ↑ 33 Changed 8 years ago by
Replying to fbissey:
Replying to leif:
Replying to kcrisman:
Now I guess we just need a "good" way to implement your ideas, which it sounds like you all have.
Which I think means fixing the FLINT (and optionally also the PARI) spkg (on Darwin at least). I don't know whether it's upstream's or probably Sage's fault, but François presumably knows, as he mentioned he had patches for both.
Flint upstream doesn't set anything for install_name and it is straightforward to add it, I'll submit a pull request for that to them this morning hopefully. Not sure how to forward that kind of patch to pari upstream, but again they don't do anything particular and end with something that comes as default with their linking options. I posted a link to the patch in #comment:12 if we want to include it. No extra actions needed in spkg-install just apply the patch.
The PARI folk is currently releasing rc's for 2.7. It might be a very good time to forward them such a patch. I think you should post it on the pari-dev ml.
comment:35 Changed 8 years ago by
- Description modified (diff)
comment:36 in reply to: ↑ 32 Changed 8 years ago by
Replying to jpflori:
Replying to leif:
P.S.: Does anyone currently test on Cygwin?
I could do that.
IIRC I built 6.2.beta4 without problems (except bzip2 unrelated issues) so maybe nothing is actully needed? When was eclib updated?
In Sage 6.2.beta0 IIRC. (That's why I asked when K-D last successfully built Sage on that Darwin PPC box.)
comment:37 Changed 8 years ago by
I need a little bit of time to check my fix work for flint (bulding beta5 on my maverick machine)
comment:38 Changed 8 years ago by
Push request sent to upstream flint https://github.com/wbhart/flint2/pull/58 we can use the patch directly in sage. Anyone set up to make a sage commit can just grab that patch. I also tested the pari patch I quoted, needed a small adjustment for git style. I will send it and another for doc building with -j4 and over to maintonly@… which seem to be the right forum for these.
comment:39 follow-up: ↓ 45 Changed 8 years ago by
Other libs without a proper install_name:
- libLfunction.dylib (lcalc)
- libcliquer.dylib
- polybori's libraries are strange, possibly bad substitution
- libzn_poly.dylib (zn_poly)
- libcsage.dylib :) moving to autotools would solve that
And my patch for flint has been merged upstream \o/
comment:40 Changed 8 years ago by
Bill made a flint-2.4.3 release with this fix, we can just bump to that. see https://github.com/wbhart/flint2/pull/58
comment:41 Changed 8 years ago by
Any of these solutions seems good to me - thanks so much for this quick and knowledgeable work!
comment:42 follow-up: ↓ 43 Changed 8 years ago by
Going back to pari :) the install_name problem is all sage's spkg-install fault. Pari's own install target produce a correct dylib but sage does this little interesting bit:
if [ "$UNAME" = "Darwin" ]; then # The following is a hack to get the library installed correctly # on Mac OS X: echo "Applying Mac OS X hack -- deleting and reinstalling libpari.a ..." rm -f Odarwin-ppc/*lib* $MAKE install-lib-sta if [ $? -ne 0 ]; then echo >&2 "Error reinstalling PARI's static libraries (OS X trick)" exit 1 fi cd "`config/objdir`" && cp -f libpari* "$SAGE_LOCAL/lib" if [ $? -ne 0 ]; then echo >&2 "Error copying PARI's libraries (OS X trick)" exit 1 fi cd "$CUR" fi
So we use the copy from the build directory in SAGE_LOCAL/lib. But when when you do "make install" pari actually rebuilds libpari{,-gmp}.dylib in the install location (modulo DESTDIR if it is defined) with the correct install_name. But spkg-install throw all that away and uses a copy without any install_name set.
Unless someone can explain to me what this hack is all about, I think its removal should have a bug of its own on trac.
comment:43 in reply to: ↑ 42 Changed 8 years ago by
Replying to fbissey:
Going back to pari :) the install_name problem is all sage's spkg-install fault.
Doesn't really surprise me... B)
Unless someone can explain to me what this hack is all about, I think its removal should have a bug of its own on trac.
You may have to ask the BDFL; that "hack" is historically pre-Mercurial, apparently.
comment:44 Changed 8 years ago by
- Description modified (diff)
I'll try to figure the new workflow and submit an update for flint.
comment:45 in reply to: ↑ 39 Changed 8 years ago by
Replying to fbissey:
Other libs without a proper install_name:
- libLfunction.dylib (lcalc)
- libcliquer.dylib
- polybori's libraries are strange, possibly bad substitution
- libzn_poly.dylib (zn_poly)
- libcsage.dylib :) moving to autotools would solve that
And apparently the R dylibs as well - see #16044, and of course fbissey and leif discovered these too :)
comment:46 Changed 8 years ago by
- Branch set to u/fbissey/flint2.4.3
comment:47 Changed 8 years ago by
- Commit set to d86fecf3ea6fd24058aeeedcdf667224d64d625e
Branch pushed to git repo; I updated commit sha1. New commits:
d86fecf | trivial version bump for trac 16017
|
comment:48 Changed 8 years ago by
- Status changed from new to needs_review
comment:49 Changed 8 years ago by
- Reviewers set to Karl-Dieter Crisman
From my point of view this is fine - upstream does seem to have a couple other changes since the previous release so it seems good to test this on a few other machines just in case there is some problem. But presumably those problems would also be problems elsewhere.
comment:50 Changed 8 years ago by
- Status changed from needs_review to needs_work
Right. I haven't run doctests against it yet so I will. I am putting this back to "need work" because I forgot to generate the checksum :(
comment:51 follow-up: ↓ 53 Changed 8 years ago by
So... will this fix be non-movable? (I.e., on any platform, on this one...) See leif's comment on #16044.
comment:52 Changed 8 years ago by
- Commit changed from d86fecf3ea6fd24058aeeedcdf667224d64d625e to 6c25897014779846c65035baca4b0110bf6a056e
Branch pushed to git repo; I updated commit sha1. New commits:
6c25897 | checksums!
|
comment:53 in reply to: ↑ 51 Changed 8 years ago by
- Status changed from needs_work to needs_review
Replying to kcrisman:
So... will this fix be non-movable? (I.e., on any platform, on this one...) See leif's comment on #16044.
Basically yes, unless we find variables that overrides the install_names effectively. Note that once your install is moved, if we find variables overrides to make it work, you'll probably be screwed if you try to upgrade in the new location.
On linux LD_LIBRARY_PATH is a very effective override to anything you put in an elf, so regardless of Volker's comment about dangling rpath they are screened by LD_LIBRARY_PATH. Even on linux upgrading in the new location is probably leaving a mess behind, but LD_LIBRARY_PATH is a good safety blanket most of the time.
Putting this for review now that I have put the checksums.
comment:54 follow-up: ↓ 55 Changed 8 years ago by
Hmm, I don't know if I like that. I'd at least want to see what happened here. How will relocation issues be a problem with people downloading binaries? (I assume this change in flint would cause problems for all Mac platforms, not just ours, since the extra flag and hence full name comes in on all Darwin, not just this Darwin 8, based on the patch.)
comment:55 in reply to: ↑ 54 ; follow-up: ↓ 57 Changed 8 years ago by
Replying to kcrisman:
How will relocation issues be a problem with people downloading binaries? (I assume this change in flint would cause problems for all Mac platforms, not just ours, since the extra flag and hence full name comes in on all Darwin, not just this Darwin 8, based on the patch.)
For FLINT and w.r.t. relocating Sage, the patch would only (effectively) make a difference on MacOS X 10.4.
If relocation is (at least partially) broken on Darwin anyway, the change doesn't make a difference to the others, i.e., it doesn't really worsen the situation.
On Darwin, we have absolute paths in (most of) all other libraries already; fixing that is subject of another ticket I'd say.
Disclaimer: I don't know at all what the Mac app currently does; but if it behaves differently, I'd rather expect it to be smarter w.r.t. "relocation".
comment:56 follow-up: ↓ 58 Changed 8 years ago by
According to http://wiki.sagemath.org/SupportedPlatforms the Mac App doesn't work at all on OSX 10.4
comment:57 in reply to: ↑ 55 Changed 8 years ago by
Replying to leif:
If relocation is (at least partially) broken on Darwin anyway, the change doesn't make a difference to the others, i.e., it doesn't really worsen the situation.
On Darwin, we have absolute paths in (most of) all other libraries already; fixing that is subject of another ticket I'd say.
Ways to go (not excluding each other):
- Putting Sage's library folders into
DYLD_FALLBACK_LIBRARY_PATH
; rather a work-around (not to say hack).
- Rewriting all dylibs in
sage-location
(adjusting all absolute paths /install_name
s). (Raises the question whetherinstall_name_tool
is available "by default", i.e., at least shipped with something (e.g. XCode?) Sage requires anyway, for ordinary as opposed to development use. And the lengths of the new filenames might be an issue.)
- Using relative paths in
install_name
s (presumably@loader_path/...
). This could be done (just) insage-location
or some other post-inst hook as well, in the long run (also) changing build scripts (or just settings) and presumably patching a couple of upstream packages.
Ads by G**gle
Buy the compiler wrapper NOW!!!
comment:58 in reply to: ↑ 56 Changed 8 years ago by
According to http://wiki.sagemath.org/SupportedPlatforms the Mac App doesn't work at all on OSX 10.4
Huh, that surprises me. There is one built, anyway, and distributed via the mirrors for 5.13. I can check that later - it should still work okay.
comment:59 Changed 8 years ago by
- Reviewers changed from Karl-Dieter Crisman to Karl-Dieter Crisman, Volker Braun
- Status changed from needs_review to positive_review
comment:60 Changed 8 years ago by
- Branch changed from u/fbissey/flint2.4.3 to 6c25897014779846c65035baca4b0110bf6a056e
- Resolution set to fixed
- Status changed from positive_review to closed
Leif's idea seems to have been right, though I don't know that setting the DYLD_ whatever path or rpath is any easier that what seemed to do it for me - just adding an explicit dependency in src/module_list.py, as we often do for Cygwin, which is more finicky. From the thread:
But the problem is that somehow the linking isn't happening properly in sage/libs/mwrank, and indeed -lflint isn't in the line
g++ -bundle -undefined dynamic_lookup -L/Users/student/Desktop/sage-6.2.beta4/local/lib build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -L/Users/student/Desktop/sage-6.2.beta4/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.so
Maybe could I add something to src/module_list.py or whatever the appropriate file is now to force -lflint as well? On Cygwin we often had to be more explicit because of something analogous, if I recall that correctly. I'm going to try this just to see...
It seems to work! At least, things continue compiling. On Cygwin that was always the sign of victory, though :)
So Leif's idea was probably the right one. I'll see if things finish up first and then run tests - John, any files I should run tests on first to make sure this did indeed compile and link correctly?
However, I don't know the right *fix*. I don't mind including extra things in module_list.py since probably Cygwin likes them too, but perhaps there is a better fix, or maybe there's a reason that's a bad fix, or something.