Get libs.mwrank to compile on OS X 10.4

[kcrisman]

gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include/python2.7 -c sage/libs/mwrank/mwrank.c -o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o -fno-strict-aliasing -w
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include -I/Users/student/Desktop/sage-6.2.beta4/local/include/csage -I/Users/student/Desktop/sage-6.2.beta4/src -I/Users/student/Desktop/sage-6.2.beta4/src/sage/ext -I/Users/student/Desktop/sage-6.2.beta4/local/include/python2.7 -c sage/libs/mwrank/wrap.cc -o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -fno-strict-aliasing -w
g++ -bundle -undefined dynamic_lookup -L/Users/student/Desktop/sage-6.2.beta4/local/lib build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -L/Users/student/Desktop/sage-6.2.beta4/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.so
/usr/bin/ld: warning can't open dynamic library: libflint.dylib referenced from: /Users/student/Desktop/sage-6.2.beta4/local/lib/libec.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)
/usr/bin/ld: Undefined symbols:
_nmod_mat_clear referenced from libec expected to be defined in libflint.dylib
_nmod_mat_init referenced from libec expected to be defined in libflint.dylib
_nmod_mat_rref referenced from libec expected to be defined in libflint.dylib
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

See lots of discussion at ​this sage-release thread. Best guesses are either ​this Flint bug and leif's idea that there is a missing linker flag on Darwin (see the sage-release thread).

---

This is not eclib's fault, but due to incomplete / wrong "install_names" (meant to be absolute paths) in libflint.dylib (and also libpari-gmp.dylib).

On Darwin 10 (MacOS X 10.6) and later, apparently the linker is smarter (or more tolerant), and does find the indirectly used libraries in $SAGE_LOCAL/lib/, despite their wrong install_names.

We can solve this by upgrading flint to 2.4.3 which includes the necessary fix.

New sources:

• ​http://www.flintlib.org/flint-2.4.3.tar.gz

[kcrisman]

Leif's idea seems to have been right, though I don't know that setting the DYLD_ whatever path or rpath is any easier that what seemed to do it for me - just adding an explicit dependency in src/module_list.py, as we often do for Cygwin, which is more finicky. From the thread:

---

But the problem is that somehow the linking isn't happening properly in sage/libs/mwrank, and indeed -lflint isn't in the line

g++ -bundle -undefined dynamic_lookup -L/Users/student/Desktop/sage-6.2.beta4/local/lib build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.4-ppc-2.7/sage/libs/mwrank/wrap.o -L/Users/student/Desktop/sage-6.2.beta4/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.4-ppc-2.7/sage/libs/mwrank/mwrank.so

Maybe could I add something to src/module_list.py or whatever the appropriate file is now to force -lflint as well? On Cygwin we often had to be more explicit because of something analogous, if I recall that correctly. I'm going to try this just to see...

It seems to work! At least, things continue compiling. On Cygwin that was always the sign of victory, though :)

So Leif's idea was probably the right one. I'll see if things finish up first and then run tests - John, any files I should run tests on first to make sure this did indeed compile and link correctly?

However, I don't know the right *fix*. I don't mind including extra things in module_list.py since probably Cygwin likes them too, but perhaps there is a better fix, or maybe there's a reason that's a bad fix, or something.

[kcrisman]

Also required for sage.libs.cremona extensions. Given that on Cygwin we will probably need this too, I'm suggesting that this is the right solution now.

[leif]

I'm pretty curious whether -- with the "superfluous" -lflint added -- you're still getting the weird

/usr/bin/ld: warning can't open dynamic library: libflint.dylib referenced from: /Users/student/Desktop/sage-6.2.beta4/local/lib/libec.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)

The addition is IMHO just a work-around (on Darwin 8; still probably necessary on Cygwin); it doesn't address the root of the problem (provided we can trust the linker messages). (Cf. my recent comment on the Sage 6.2.beta4 thread on sage-release.)

[leif]

... in other words:

Where the heck does the linker try to find the libflint.dylib libec.dylib refers to, and does libec.dylib really refer to $SAGE_ROOT/local/lib/libflint.dylib (modulo symlinks to versioned versions of the libraries)?

Could Apple's otool enlighten us? (Not sure whether passing -t to the Darwin linker would.)

[kcrisman]

No, no warnings if I use that. I really think it just doesn't know where to find it. This happened on Cygwin all the time, like I said. Yes, it's certainly just a workaround, but one that I can't really see a downside to. I have no idea where it is looking, and I have pretty much run out of energy on this now that I found what I think it is an acceptable solution. Not that I don't *want* a better one, but my sense is there may not be one that doesn't involve special casing a -lflint for old Macs for no particular reason, or that doesn't involve asking John to do something annoying with eclib.

[fbissey]

Try using ​https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/flint/files/flint-2.4-makefile.patch in the patches folder for flint. Then in spkg-install

export SHARE_FLAGS="-install_name ${SAGE_LOCAL}/lib/libflint.dylib"

before $MAKE Rebuild flint, eclib and see if that works. If it does we'll put that in a better shape.

[kcrisman]

Thanks, that sounds fine too if it works, but I won't be able to try this until Monday.

Since we have several workarounds now, and since (for now) this is still a supported platform, I'm making this a blocker for 6.2.

[leif]

leif@bsd:~/Sage/sage-6.2.beta5$ otool -DL local/lib/libec.dylib 
local/lib/libec.dylib:
        /Users/leif/Sage/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
        /Users/leif/Sage/sage-6.2.beta5/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0)
        /Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5)
        /Users/leif/Sage/sage-6.2.beta5/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)
        /Users/leif/Sage/sage-6.2.beta5/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Interesting... (Note the recorded path to the PARI library!)

But we (also) already have -lpari on the command line for linking the mwrank Python extension module (which I think by the way shouldn't be necessary either, as it isn't directly used IIRC):

g++ -bundle -undefined dynamic_lookup -L/Users/leif/Sage/sage-6.2.beta5/local/lib build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lcsage -lec -lntl -lpari -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.so

And FWIW, in the build on bsd.math, libtool also drops the -rpath given to it when linking libec.dylib (or, more precisely, libec.0.dylib).

[leif]

For completeness, this is how libec.0.dylib gets linked on bsd.math (looks the same as on Karl-Dieter's MacOS X 10.4 PPC box):

/bin/sh ../libtool --tag=CXX   --mode=link g++  -g -O3  -no-undefined -L/Users/leif/Sage/sage-6.2.beta5/local/lib -L/Users/leif/Sage/sage-6.2.beta5/local/lib -L/Users/leif/Sage/sage-6.2.beta5/local/lib   -o libec.la -rpath /Users/leif/Sage/sage-6.2.beta5/local/lib interface.lo unimod.lo arith.lo marith.lo gpslave.lo compproc.lo vector.lo matrix.lo subspace.lo kbessel.lo mvector.lo mmatrix.lo msubspace.lo svector.lo smatrix.lo smatrix_elim.lo xsplit.lo conic.lo legendre.lo quadratic.lo illl.lo hilbert.lo timer.lo cubic.lo polys.lo realroots.lo parifact.lo p2points.lo gf.lo xsplit_data.lo threadpool.lo logger.lo curve.lo curvedata.lo curvered.lo points.lo cperiods.lo isogs.lo heights.lo mwprocs.lo lambda.lo sifter.lo sieve_search.lo htconst.lo egr.lo saturate.lo divpol.lo pointsmod.lo curvemod.lo ffmod.lo tlss.lo elog.lo getcurve.lo mequiv.lo mrank1.lo mlocsol.lo mglobsol.lo mquartic.lo mrank2.lo qc.lo sqfdiv.lo version.lo minim.lo reduce.lo transform.lo desc2.lo bitspace.lo GetOpt.lo twoadic.lo descent.lo newforms.lo symb.lo homspace.lo cusp.lo oldforms.lo fixc6.lo periods.lo moddata.lo pcprocs.lo nfd.lo curvesort.lo  -lflint -lntl -lpari
libtool: link: g++ -dynamiclib  -o .libs/libec.0.dylib  .libs/interface.o .libs/unimod.o .libs/arith.o .libs/marith.o .libs/gpslave.o .libs/compproc.o .libs/vector.o .libs/matrix.o .libs/subspace.o .libs/kbessel.o .libs/mvector.o .libs/mmatrix.o .libs/msubspace.o .libs/svector.o .libs/smatrix.o .libs/smatrix_elim.o .libs/xsplit.o .libs/conic.o .libs/legendre.o .libs/quadratic.o .libs/illl.o .libs/hilbert.o .libs/timer.o .libs/cubic.o .libs/polys.o .libs/realroots.o .libs/parifact.o .libs/p2points.o .libs/gf.o .libs/xsplit_data.o .libs/threadpool.o .libs/logger.o .libs/curve.o .libs/curvedata.o .libs/curvered.o .libs/points.o .libs/cperiods.o .libs/isogs.o .libs/heights.o .libs/mwprocs.o .libs/lambda.o .libs/sifter.o .libs/sieve_search.o .libs/htconst.o .libs/egr.o .libs/saturate.o .libs/divpol.o .libs/pointsmod.o .libs/curvemod.o .libs/ffmod.o .libs/tlss.o .libs/elog.o .libs/getcurve.o .libs/mequiv.o .libs/mrank1.o .libs/mlocsol.o .libs/mglobsol.o .libs/mquartic.o .libs/mrank2.o .libs/qc.o .libs/sqfdiv.o .libs/version.o .libs/minim.o .libs/reduce.o .libs/transform.o .libs/desc2.o .libs/bitspace.o .libs/GetOpt.o .libs/twoadic.o .libs/descent.o .libs/newforms.o .libs/symb.o .libs/homspace.o .libs/cusp.o .libs/oldforms.o .libs/fixc6.o .libs/periods.o .libs/moddata.o .libs/pcprocs.o .libs/nfd.o .libs/curvesort.o   -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lflint /Users/leif/Sage/sage-6.2.beta5/local/lib/libntl.dylib -L/Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/gcc-4.7.3.p1/gcc-build/x86_64-apple-darwin10.8.0/libstdc++-v3/src -L/Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/gcc-4.7.3.p1/gcc-build/x86_64-apple-darwin10.8.0/libstdc++-v3/src/.libs -lpari  -O3   -install_name  /Users/leif/Sage/sage-6.2.beta5/local/lib/libec.0.dylib -compatibility_version 1 -current_version 1.0 -Wl,-single_module

So indeed the (missing or weird) install_names seem to be the problem. (Or libtool... ;-) )

[leif]

Replying to leif:

(Note the recorded path to the PARI library!)

But we (also) already have -lpari on the command line for linking the mwrank Python extension module (which I think by the way shouldn't be necessary either, as it isn't directly used IIRC) [...]

And if I remove pari from the allegedly needed libraries of the mwrank Python extension module (in module_list.py), this also works on bsd.math:

leif@bsd:~/Sage/sage-6.2.beta5$ touch src/sage/libs/mwrank/mwrank.pyx 
leif@bsd:~/Sage/sage-6.2.beta5$ ./sage -b
scons: `install' is up to date.
Updating Cython code....
Compiling sage/libs/mwrank/mwrank.pyx because it changed.
Cythonizing sage/libs/mwrank/mwrank.pyx
Finished compiling Cython code (time = 4.62841916084 seconds)
running install
running build
running build_py
running build_ext
building 'sage.libs.mwrank.mwrank' extension
Executing 1 command (using 1 thread)
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include/python2.7 -c sage/libs/mwrank/mwrank.c -o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o -fno-strict-aliasing -w
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -DNTL_ALL -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include -I/Users/leif/Sage/sage-6.2.beta5/local/include/csage -I/Users/leif/Sage/sage-6.2.beta5/src -I/Users/leif/Sage/sage-6.2.beta5/src/sage/ext -I/Users/leif/Sage/sage-6.2.beta5/local/include/python2.7 -c sage/libs/mwrank/wrap.cc -o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -fno-strict-aliasing -w
g++ -bundle -undefined dynamic_lookup -L/Users/leif/Sage/sage-6.2.beta5/local/lib build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.o build/temp.macosx-10.6-x86_64-2.7/sage/libs/mwrank/wrap.o -L/Users/leif/Sage/sage-6.2.beta5/local/lib -lcsage -lec -lntl -lgmp -lgmpxx -lstdc++ -lm -lstdc++ -lntl -o build/lib.macosx-10.6-x86_64-2.7/sage/libs/mwrank/mwrank.so
Time to execute 1 command: 4.69843101501 seconds
Total time spent compiling C/C++ extensions:  4.88040590286 seconds.

I guess doing so would trigger another "no such file or directory" warning on Karl-Dieter's machine. MacOS X 10.6's linker is simply "smarter"...

[fbissey]

So we may also want to look at pari because it put the build time location rather than final location in install name. Not a problem if you tell sage to keep the build stuff I guess.

[fbissey]

I actually have a patch for pari's installname that I made for pari 2.5.0 and it is still in the gentoo and sage-on-gentoo trees, I guess I should try to it merged in pari proper. ​https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/pari/files/pari-2.5.0-macos.patch As for Karl-Dieter's problem we should be able to see if fixing the installname solves his problem without rebuilding flint and eclib. we can use

install_name_tool -change "oldname" "newname" filename

We can alter libflint.dylib and libec.dylib with the appropriate values and see if sage compile.

[leif]

Replying to fbissey:

I actually have a patch for pari's installname that I made for pari 2.5.0 and it is still in the gentoo and sage-on-gentoo trees, I guess I should try to it merged in pari proper. ​https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/pari/files/pari-2.5.0-macos.patch As for Karl-Dieter's problem we should be able to see if fixing the installname solves his problem without rebuilding flint and eclib. we can use

install_name_tool -change "oldname" "newname" filename

We can alter libflint.dylib and libec.dylib with the appropriate values and see if sage compile.

Yep, but he'd have to rebuild eclib (after changing libflint's [and perhaps libpari's] install_name) such that the proper dependency name(s) get(s) recorded in libec.0.dylib; I don't think you can change these with install_name_tool.

Any idea why libtool ignores -rpath (on both MacOS X 10.4 and 10.6)?

[fbissey]

First yes you can.

Mirage:lib fbissey$ otool -L libec.0.dylib 
libec.0.dylib:
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
Mirage:lib fbissey$ install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib 
Mirage:lib fbissey$ otool -L libec.0.dylib 
libec.0.dylib:
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libntl.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/Users/fbissey/build/sage-6.2.beta4/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib (compatibility version 2.5.0, current version 2.5.5)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
	/Users/fbissey/build/sage-6.2.beta4/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
Mirage:lib fbissey$ 

About libtool, not sure. I always felt rpath was redundant with the install_name mechanism but that's not really an answer to your question.

[leif]

Replying to fbissey:

First yes you can.

Cool. \o/

About libtool, not sure. I always felt rpath was redundant with the install_name mechanism but that's not really an answer to your question.

Well, by default install_name is an absolute path. You can prepend @rpath/ (and other flavours) to the install_name to make it relative to paths set by -rpath, similar to $ORIGIN in an ELF object's DT_RUNPATH. But I'm not a Darwin linker (nor Mach-O) expert; I'd expect libtool to know how to do it though. On the other hand, libtool did convert some of the -L/path/to/foo/ -lfoo to absolute filenames on the linker command line (when linking libec.0.dylib), presumably exactly [to] those with a correct, absolute install_name.

[kcrisman]

Yikes! As long as I have a specific thing to try out on Monday I'll be happy - if you guys can agree on what I should try, I don't have infinite time with that machine ;-)

[fbissey]

Try the equivalent for your machine of:

install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib 
install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libflint.dylib 

in $SAGE_LOCAL/lib. Don't forget to replace the path for SAGE_LOCAL with what's correct for you. Once that's done you can try to build sage again and see if it will go past that point. If it does we'll make the appropriate patch.

[leif]

Replying to kcrisman:

Yikes! As long as I have a specific thing to try out on Monday I'll be happy - if you guys can agree on what I should try, I don't have infinite time with that machine ;-)

There are plenty of things you could try... :-)

But I think kiwifb and me agree on that you could first try to:

• Restore module_list.py (at least remove flint, optionally also pari) from the "needed" libraries of the libs.mwrank Python extension module -- to test whether it works for one of the affected modules (you had to change about five IIRC)

• Make sure libflint's install_name is still just libflint.dylib (otherwise install_name_tool will give an error):

  $ cd $SAGE_ROOT
  $ otool -D local/lib/libflint.dylib
  local/lib/libflint.dylib:
  libflint.dylib
  

• Change the install_name (here called "id") to the library's absolute path:

  $ install_name_tool -id `pwd`/local/lib/libflint.dylib local/lib/libflint.dylib
  

• Then either rebuild/reinstall eclib (which should now record libflint's absolute path), or -- simpler -- just change the recorded name (for FLINT) in libec.0.dylib manually:

  $ otool -L local/lib/libec.0.dylib  # just to check libflint's name there
  local/lib/libec.0.dylib:
          <path to>/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
          libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
          ...
  
  $ install_name_tool -change libflint.dylib `pwd`/local/lib/libflint.dylib local/lib/libec.0.dylib  # now really change the name
  
  $ otool -L local/lib/libec.0.dylib  # just to verify it changed
  local/lib/libec.0.dylib:
          <path to>/sage-6.2.beta5/local/lib/libec.0.dylib (compatibility version 1.0.0, current version 1.0.0)
          <path to>/sage-6.2.beta5/local/lib/libflint.dylib (compatibility version 0.0.0, current version 0.0.0)
          ...
  
  

• Pretend file modification to trigger the rebuild of the module(s):

  $ touch src/sage/libs/mwrank/mwrank.pyx  # and more if you like
  
  $ ./sage -b  # and now rebuild
  

The linker should now be able to locate libflint.dylib referenced by libec.dylib.

Optionally, repeat the steps analogously for pari (which in contrast is originally listed as a needed lib in module_list.py, so you'd have to really remove it rather than restoring the state of vanilla beta4 or beta5); just libpari's original install_name is a bit more complicated (with a non-existing temporary path):

 $ otool -D local/lib/libpari-gmp.dylib 
local/lib/libpari-gmp.dylib:
<path to>/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib

(libpari.dylib is a symbolic link to libpari-gmp.dylib.)

When you change the name (again in libec.0.dylib), you have to specify the full original (invalid) path:

$ install_name_tool -change /Users/leif/Sage/sage-6.2.beta5/local/var/tmp/sage/build/pari-2.5.5b.p1/src/Odarwin-i386/libpari-gmp.dylib `pwd`/local/lib/libpari-gmp.dylib local/lib/libec.0.dylib

(Just as an example -- note that the invalid path will differ on a PowerPC, and of course your $SAGE_ROOT.)

[leif]

Just for the record: I started writing before François replied... 8)

But we (still) agree on what you could try.

[fbissey]

Yes we do. Just using install_name_tool means that we can test that theory without rebuilding flint and eclib, which would take some time on that machine.

[leif]

Replying to fbissey:

Try the equivalent for your machine of:

install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libec.0.dylib 
install_name_tool -change libflint.dylib /Users/fbissey/build/sage-6.2.beta4/local/lib/libflint.dylib libflint.dylib 

in $SAGE_LOCAL/lib. Don't forget to replace the path for SAGE_LOCAL with what's correct for you.

Note that the second line wouldn't work (it's a no-op).

If you want to change the install_name of the library itself (as opposed to the recorded install names of libraries it uses), you have to use the -id option of install_name_tool, which takes just the new name (plus the filename of the library to change of course).

[fbissey]

Well spotted. I was a bit too eager there and I had completely forgotten about that important detail.

[vbraun]

For the record, @rpath is OSX 10.5 and later only afaik. So absolute paths it is.

[kcrisman]

Ok, I'm trying this. Thank you leif for the VERY explicit steps, which start to help me understand even what these files even are.

The otool output was fine, and I decided to have it rebuild eclib just to make sure that worked fine. I already reverted module_list.py and touched the files so I'll let you know what happens when it's done. I'm not messing with the pari issue, though :)

[kcrisman]

Okay, this does indeed seem to do the trick. I can't run doctests, though, because sympy isn't built yet. Weird.

[kcrisman]

Ok, I had to install sympy AND mpmath for Sage to even start - which seems odd, they shouldn't be imported on startup? Anyway, only sage/libs/cremona/newforms.pyx doesn't pass tests, but that is because the Cremona mini database apparently never got installed, unrelated, I assume. (Or is that part of the eclib package?)

[fbissey]

I am fairly sure it is in database_cremona_ellcurve.

[kcrisman]

Actually, it ended up being installed via the elliptic_curve spkg, which only installed later... so that should be fine as well.

Now I guess we just need a "good" way to implement your ideas, which it sounds like you all have.

[fbissey]

In the first instance what I suggested earlier would work but I want to put it in form that's more palatable to flint upstream so they can just merge it (and I'll get rid of my patch in sage-on-gentoo).

I need to take another look at their configure script.

[leif]

Replying to kcrisman:

Now I guess we just need a "good" way to implement your ideas, which it sounds like you all have.

Which I think means fixing the FLINT (and optionally also the PARI) spkg (on Darwin at least). I don't know whether it's upstream's or probably Sage's fault, but François presumably knows, as he mentioned he had patches for both.

[leif]

P.S.: Does anyone currently test on Cygwin?

[jpflori]

Replying to leif:

P.S.: Does anyone currently test on Cygwin?

I could do that.

IIRC I built 6.2.beta4 without problems (except bzip2 unrelated issues) so maybe nothing is actully needed? When was eclib updated?

[fbissey]

Replying to leif:

Replying to kcrisman:

Now I guess we just need a "good" way to implement your ideas, which it sounds like you all have.

Which I think means fixing the FLINT (and optionally also the PARI) spkg (on Darwin at least). I don't know whether it's upstream's or probably Sage's fault, but François presumably knows, as he mentioned he had patches for both.

Flint upstream doesn't set anything for install_name and it is straightforward to add it, I'll submit a pull request for that to them this morning hopefully. Not sure how to forward that kind of patch to pari upstream, but again they don't do anything particular and end with something that comes as default with their linking options. I posted a link to the patch in #comment:12 if we want to include it. No extra actions needed in spkg-install just apply the patch.

[jpflori]

Replying to fbissey:

Replying to leif:

Replying to kcrisman:

Now I guess we just need a "good" way to implement your ideas, which it sounds like you all have.

Which I think means fixing the FLINT (and optionally also the PARI) spkg (on Darwin at least). I don't know whether it's upstream's or probably Sage's fault, but François presumably knows, as he mentioned he had patches for both.

Flint upstream doesn't set anything for install_name and it is straightforward to add it, I'll submit a pull request for that to them this morning hopefully. Not sure how to forward that kind of patch to pari upstream, but again they don't do anything particular and end with something that comes as default with their linking options. I posted a link to the patch in #comment:12 if we want to include it. No extra actions needed in spkg-install just apply the patch.

The PARI folk is currently releasing rc's for 2.7. It might be a very good time to forward them such a patch. I think you should post it on the pari-dev ml.

[leif]

Replying to jpflori:

Replying to leif:

P.S.: Does anyone currently test on Cygwin?

I could do that.

IIRC I built 6.2.beta4 without problems (except bzip2 unrelated issues) so maybe nothing is actully needed? When was eclib updated?

In Sage 6.2.beta0 IIRC. (That's why I asked when K-D last successfully built Sage on that Darwin PPC box.)

[fbissey]

I need a little bit of time to check my fix work for flint (bulding beta5 on my maverick machine)

[fbissey]

Push request sent to upstream flint ​https://github.com/wbhart/flint2/pull/58 we can use the patch directly in sage. Anyone set up to make a sage commit can just grab that patch. I also tested the pari patch I quoted, needed a small adjustment for git style. I will send it and another for doc building with -j4 and over to maintonly@… which seem to be the right forum for these.

[fbissey]

Other libs without a proper install_name:

• libLfunction.dylib (lcalc)
• libcliquer.dylib
• polybori's libraries are strange, possibly bad substitution
• libzn_poly.dylib (zn_poly)
• libcsage.dylib :) moving to autotools would solve that

And my patch for flint has been merged upstream \o/

[fbissey]

Bill made a flint-2.4.3 release with this fix, we can just bump to that. see ​https://github.com/wbhart/flint2/pull/58

[kcrisman]

Any of these solutions seems good to me - thanks so much for this quick and knowledgeable work!

[fbissey]

Going back to pari :) the install_name problem is all sage's spkg-install fault. Pari's own install target produce a correct dylib but sage does this little interesting bit:

    if [ "$UNAME" = "Darwin" ]; then
        # The following is a hack to get the library installed correctly
        # on Mac OS X:
        echo "Applying Mac OS X hack -- deleting and reinstalling libpari.a ..."
        rm -f Odarwin-ppc/*lib*
        $MAKE install-lib-sta
        if [ $? -ne 0 ]; then
            echo >&2 "Error reinstalling PARI's static libraries (OS X trick)"
            exit 1
        fi

        cd "`config/objdir`" && cp -f libpari* "$SAGE_LOCAL/lib"
        if [ $? -ne 0 ]; then
            echo >&2 "Error copying PARI's libraries (OS X trick)"
            exit 1
        fi
        cd "$CUR"
    fi

So we use the copy from the build directory in SAGE_LOCAL/lib. But when when you do "make install" pari actually rebuilds libpari{,-gmp}.dylib in the install location (modulo DESTDIR if it is defined) with the correct install_name. But spkg-install throw all that away and uses a copy without any install_name set.

Unless someone can explain to me what this hack is all about, I think its removal should have a bug of its own on trac.

[leif]

Replying to fbissey:

Going back to pari :) the install_name problem is all sage's spkg-install fault.

Doesn't really surprise me... B)

Unless someone can explain to me what this hack is all about, I think its removal should have a bug of its own on trac.

You may have to ask the BDFL; that "hack" is historically pre-Mercurial, apparently.

[fbissey]

I'll try to figure the new workflow and submit an update for flint.

[kcrisman]

Replying to fbissey:

Other libs without a proper install_name:

• libLfunction.dylib (lcalc)
• libcliquer.dylib
• polybori's libraries are strange, possibly bad substitution
• libzn_poly.dylib (zn_poly)
• libcsage.dylib :) moving to autotools would solve that

And apparently the R dylibs as well - see #16044, and of course fbissey and leif discovered these too :)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d86fecf

trivial version bump for trac 16017

[kcrisman]

From my point of view this is fine - ​upstream does seem to have a couple other changes since the previous release so it seems good to test this on a few other machines just in case there is some problem. But presumably those problems would also be problems elsewhere.

[fbissey]

Right. I haven't run doctests against it yet so I will. I am putting this back to "need work" because I forgot to generate the checksum :(

[kcrisman]

So... will this fix be non-movable? (I.e., on any platform, on this one...) See ​leif's comment on #16044.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6c25897

checksums!

[fbissey]

Replying to kcrisman:

So... will this fix be non-movable? (I.e., on any platform, on this one...) See ​leif's comment on #16044.

Basically yes, unless we find variables that overrides the install_names effectively. Note that once your install is moved, if we find variables overrides to make it work, you'll probably be screwed if you try to upgrade in the new location.

On linux LD_LIBRARY_PATH is a very effective override to anything you put in an elf, so regardless of Volker's comment about dangling rpath they are screened by LD_LIBRARY_PATH. Even on linux upgrading in the new location is probably leaving a mess behind, but LD_LIBRARY_PATH is a good safety blanket most of the time.

Putting this for review now that I have put the checksums.

[kcrisman]

Hmm, I don't know if I like that. I'd at least want to see what happened here. How will relocation issues be a problem with people downloading binaries? (I assume this change in flint would cause problems for all Mac platforms, not just ours, since the extra flag and hence full name comes in on all Darwin, not just this Darwin 8, based on the patch.)

[leif]

Replying to kcrisman:

How will relocation issues be a problem with people downloading binaries? (I assume this change in flint would cause problems for all Mac platforms, not just ours, since the extra flag and hence full name comes in on all Darwin, not just this Darwin 8, based on the patch.)

For FLINT and w.r.t. relocating Sage, the patch would only (effectively) make a difference on MacOS X 10.4.

If relocation is (at least partially) broken on Darwin anyway, the change doesn't make a difference to the others, i.e., it doesn't really worsen the situation.

On Darwin, we have absolute paths in (most of) all other libraries already; fixing that is subject of another ticket I'd say.

Disclaimer: I don't know at all what the Mac app currently does; but if it behaves differently, I'd rather expect it to be smarter w.r.t. "relocation".

[vbraun]

According to ​http://wiki.sagemath.org/SupportedPlatforms the Mac App doesn't work at all on OSX 10.4

[leif]

Replying to leif:

If relocation is (at least partially) broken on Darwin anyway, the change doesn't make a difference to the others, i.e., it doesn't really worsen the situation.

On Darwin, we have absolute paths in (most of) all other libraries already; fixing that is subject of another ticket I'd say.

Ways to go (not excluding each other):

• Putting Sage's library folders into DYLD_FALLBACK_LIBRARY_PATH; rather a work-around (not to say hack).

• Rewriting all dylibs in sage-location (adjusting all absolute paths / install_names). (Raises the question whether install_name_tool is available "by default", i.e., at least shipped with something (e.g. XCode?) Sage requires anyway, for ordinary as opposed to development use. And the lengths of the new filenames might be an issue.)

• Using relative paths in install_names (presumably @loader_path/...). This could be done (just) in sage-location or some other post-inst hook as well, in the long run (also) changing build scripts (or just settings) and presumably patching a couple of upstream packages.

---

Ads by G**gle

Buy the ​compiler wrapper NOW!!!

[kcrisman]

According to ​http://wiki.sagemath.org/SupportedPlatforms the Mac App doesn't work at all on OSX 10.4

Huh, that surprises me. There is one built, anyway, and distributed via the mirrors for 5.13. I can check that later - it should still work okay.