Opened 10 years ago
Closed 9 years ago
#10509 closed defect (invalid)
cvxopt: undefined symbol: ilaenv_
Reported by: | vbraun | Owned by: | tbd |
---|---|---|---|
Priority: | major | Milestone: | sage-duplicate/invalid/wontfix |
Component: | packages: standard | Keywords: | ATLAS, BLAS |
Cc: | dimpase, leif, kcrisman | Merged in: | |
Authors: | Reviewers: | Volker Braun | |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
After updating ATLAS (#10508), cvxopt fails with
sage: from cvxopt import lapack --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /home/vbraun/opt/sage-4.6.1.alpha3-atlas/<ipython console> in <module>() ImportError: /home/vbraun/opt/sage-4.6.1.alpha3-atlas/local/lib/python2.6/site-packages/cvxopt/lapack.so: undefined symbol: ilaenv_
I've looked into the cvxopt spkg and its lapack.so
uses the Fortran blas function ilaenv
without linking to f77blas. Adding f77blas to the libraries in setup.py
fixes the undefined symbol error.
Attached is a patch against patch/setup.py
to add the extra library.
Attachments (1)
Change History (31)
comment:1 Changed 10 years ago by
- Cc dimpase added
- Status changed from new to needs_review
comment:2 Changed 10 years ago by
- Status changed from needs_review to needs_info
comment:3 follow-up: ↓ 4 Changed 10 years ago by
Hi Dima, I'm talking about my new atlas-3.9.32.spkg from #10508. This version now builds shared libraries for everything, whereas the old atlas-3.8.3 did not provide a shared libf77blas.so
. I think that cvxopt so far dragged in the static f77blas, so you did not trigger this issue.
comment:4 in reply to: ↑ 3 Changed 10 years ago by
Replying to vbraun:
Hi Dima, I'm talking about my new atlas-3.9.32.spkg from #10508. This version now builds shared libraries for everything, whereas the old atlas-3.8.3 did not provide a shared
libf77blas.so
. I think that cvxopt so far dragged in the static f77blas, so you did not trigger this issue.
Hmm, how did cvxopt manage this? I'd rather think that something got changed in Atlas - IMHO Atlas 3.8 used to provide a complete wrapper to La@ack...
Actually, on some platforms, e.g. OSX PPC, f77blas does not even exist. On this platform, Atlas is not installed. So your patch will break this, for sure.
comment:5 Changed 10 years ago by
Atlas-3.8.3 does only provide a static lapack library, so cvxopt's lapack.so
statically linked it all:
(sage subshell) volker-desktop:sage-4.6.1.alpha0 vbraun$ ls -al local/lib/liblapack* -rw-r--r--. 1 vbraun vbraun 10254224 Nov 15 16:06 local/lib/liblapack.a SAGE_ROOT=/home/vbraun/opt/sage-4.6.1.alpha0 (sage subshell) volker-desktop:sage-4.6.1.alpha0 vbraun$ readelf -a local/lib/python/site-packages/cvxopt/lapack.so | grep ilaenv 000000340600 019000000007 R_X86_64_JUMP_SLO 00000000000d6610 ilaenv_ + 0 400: 00000000000d6610 5800 FUNC GLOBAL DEFAULT 11 ilaenv_ 361: 0000000000000000 0 FILE LOCAL DEFAULT ABS ilaenv.f 924: 00000000000d6610 5800 FUNC GLOBAL DEFAULT 11 ilaenv_ SAGE_ROOT=/home/vbraun/opt/sage-4.6.1.alpha0
Atlas-3.9.32 builds shared libraries and cvxopt now links against the shared library.
(sage subshell) volker-desktop:sage-4.6.1.alpha3-atlas vbraun$ ls -al local/lib/liblapack* local/lib/libatlas* local/lib/lib*blas* -rw-r--r--. 1 vbraun vbraun 10964328 Dec 20 22:36 local/lib/libatlas.a -rw-r--r--. 1 vbraun vbraun 6402481 Dec 20 20:39 local/lib/libatlas.so -rw-rw-r--. 1 vbraun vbraun 786962 Dec 20 22:36 local/lib/libblas.a -rw-r--r--. 1 vbraun vbraun 499528 Dec 20 22:36 local/lib/libcblas.a -rw-r--r--. 1 vbraun vbraun 157226 Dec 20 20:39 local/lib/libcblas.so -rw-r--r--. 1 vbraun vbraun 599690 Dec 20 22:36 local/lib/libf77blas.a -rw-r--r--. 1 vbraun vbraun 154978 Dec 20 20:39 local/lib/libf77blas.so -rw-r--r--. 1 vbraun vbraun 1640938 Dec 20 22:36 local/lib/libgslcblas.a -rwxr-xr-x. 1 vbraun vbraun 988 Dec 20 22:36 local/lib/libgslcblas.la lrwxrwxrwx. 1 vbraun vbraun 20 Dec 20 20:43 local/lib/libgslcblas.so -> libgslcblas.so.0.0.0 lrwxrwxrwx. 1 vbraun vbraun 20 Dec 20 20:43 local/lib/libgslcblas.so.0 -> libgslcblas.so.0.0.0 -rwxr-xr-x. 1 vbraun vbraun 896520 Dec 20 20:43 local/lib/libgslcblas.so.0.0.0 -rw-r--r--. 1 vbraun vbraun 10311776 Dec 20 22:36 local/lib/liblapack.a -rw-r--r--. 1 vbraun vbraun 5455501 Dec 20 20:39 local/lib/liblapack.so -rw-r--r--. 1 vbraun vbraun 500036 Dec 20 22:36 local/lib/libptcblas.a -rw-r--r--. 1 vbraun vbraun 157257 Dec 20 20:39 local/lib/libptcblas.so -rw-r--r--. 1 vbraun vbraun 600086 Dec 20 22:36 local/lib/libptf77blas.a -rw-r--r--. 1 vbraun vbraun 155009 Dec 20 20:39 local/lib/libptf77blas.so SAGE_ROOT=/home/vbraun/opt/sage-4.6.1.alpha3-atlas (sage subshell) volker-desktop:sage-4.6.1.alpha3-atlas vbraun$ readelf -a local/lib/python/site-packages/cvxopt/lapack.so | grep ilaenv 0000002997f8 004600000007 R_X86_64_JUMP_SLO 0000000000000000 ilaenv_ + 0 70: 0000000000000000 0 FUNC GLOBAL DEFAULT UND ilaenv_ 471: 0000000000000000 0 FUNC GLOBAL DEFAULT UND ilaenv_ SAGE_ROOT=/home/vbraun/opt/sage-4.6.1.alpha3-atlas (sage subshell) volker-desktop:sage-4.6.1.alpha3-atlas vbraun$ readelf -a local/lib/liblapack.so | grep ilaenv 00000070ff08 04e200000007 R_X86_64_JUMP_SLO 0000000000000000 clapack_ilaenv + 0 000000710ba8 070d00000007 R_X86_64_JUMP_SLO 0000000000000000 ilaenv_ + 0 1250: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND clapack_ilaenv 1805: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND ilaenv_ 3237: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND clapack_ilaenv 3792: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND ilaenv_ SAGE_ROOT=/home/vbraun/opt/sage-4.6.1.alpha3-atlas (sage subshell) volker-desktop:sage-4.6.1.alpha3-atlas vbraun$ readelf -a local/lib/libf77blas.so | grep ilaenv 00000021e288 004700000007 R_X86_64_JUMP_SLO 00000000000081b0 atl_f77wrap_ilaenv_ + 0 00000021e2b0 005100000007 R_X86_64_JUMP_SLO 0000000000000000 ATL_ilaenv + 0 71: 00000000000081b0 45 FUNC GLOBAL DEFAULT 6 atl_f77wrap_ilaenv_ 81: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND ATL_ilaenv 409: 00000000000089bc 3147 FUNC GLOBAL DEFAULT 6 ilaenv_ 12: 0000000000000000 0 FILE LOCAL DEFAULT ABS ATL_f77wrap_ilaenv.c 44: 0000000000000000 0 FILE LOCAL DEFAULT ABS ilaenv.f 400: 00000000000081b0 45 FUNC GLOBAL DEFAULT 6 atl_f77wrap_ilaenv_ 410: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND ATL_ilaenv 738: 00000000000089bc 3147 FUNC GLOBAL DEFAULT 6 ilaenv_ SAGE_ROOT=/home/vbraun/opt/sage-4.6.1.alpha3-atlas
In particular, ilaenv remains UNDefined in cvxopt lapack.so
and atlas liblapack.so
and needs to be dynamically linked in from libf77blas.so
.
I guess we should keep statically linking the static system-provided lapack on Windows and OSX, or change cvxopt to use the C version clapack_ilaenv (since it already links to libcblas.so
)
comment:6 Changed 10 years ago by
I meant: keep statically linking the static system-provided lapack on Windows and OSX, but add libf77blas.so
on real unices. Or use clapack_ilaenv
, then we are fine since we already link against the C blas.
comment:7 Changed 10 years ago by
In order to also remove the standalone blas in #10508, we need to remove -lblas
and use -lcblas -lf77blas
instead.
Changed 10 years ago by
Patch against patches/setup.py to add cblas, f77blas and remove standalone blas
comment:8 Changed 10 years ago by
- Cc leif added
comment:9 follow-up: ↓ 10 Changed 9 years ago by
Ping.
comment:10 in reply to: ↑ 9 Changed 9 years ago by
comment:11 follow-up: ↓ 12 Changed 9 years ago by
If #10508 is not an option yet, we should at least upgrade our dead old LAPACK, since people are having problems installing external packages into Sage which use it, but need a more recent version. (I.e., they get linker errors due to missing symbols.)
Also, someoneTM should provide a cvxopt spkg with the patch already applied.
comment:12 in reply to: ↑ 11 ; follow-ups: ↓ 13 ↓ 14 Changed 9 years ago by
- Cc kcrisman added
Replying to leif:
If #10508 is not an option yet, we should at least upgrade our dead old LAPACK, since people are having problems installing external packages into Sage which use it, but need a more recent version. (I.e., they get linker errors due to missing symbols.)
Also, someoneTM should provide a cvxopt spkg with the patch already applied.
the power supply of my PPC Powerbook just died, so I can't tell if this will work. :-(
Karl-Dieter, could you please test this on a MacOSX PPC ?
comment:13 in reply to: ↑ 12 Changed 9 years ago by
Replying to dimpase:
the power supply of my PPC Powerbook just died, so I can't tell if this will work. :-(
So now it's just a book, without power... :/
comment:14 in reply to: ↑ 12 ; follow-up: ↓ 15 Changed 9 years ago by
Replying to dimpase:
Replying to leif:
If #10508 is not an option yet, we should at least upgrade our dead old LAPACK, since people are having problems installing external packages into Sage which use it, but need a more recent version. (I.e., they get linker errors due to missing symbols.)
Also, someoneTM should provide a cvxopt spkg with the patch already applied.
the power supply of my PPC Powerbook just died, so I can't tell if this will work. :-(
Karl-Dieter, could you please test this on a MacOSX PPC ?
If you tell me exactly what to do and provide and spkg or two, sure. I know I won't have time while at work to make a new spkg etc. May need to wait until Thursday.
Will this potentially impact Cygwin as well? Just curious.
comment:15 in reply to: ↑ 14 Changed 9 years ago by
Replying to kcrisman:
Replying to dimpase:
Replying to leif:
If #10508 is not an option yet, we should at least upgrade our dead old LAPACK, since people are having problems installing external packages into Sage which use it, but need a more recent version. (I.e., they get linker errors due to missing symbols.)
Also, someoneTM should provide a cvxopt spkg with the patch already applied.
the power supply of my PPC Powerbook just died, so I can't tell if this will work. :-(
Karl-Dieter, could you please test this on a MacOSX PPC ?
If you tell me exactly what to do and provide and spkg or two, sure. I know I won't have time while at work to make a new spkg etc. May need to wait until Thursday.
well, apply the patch on this ticket to the cvxopt spkg, and test if it still works. Unless I don't understand something, it will not :-)
Will this potentially impact Cygwin as well? Just curious.
no, it should not.
comment:16 follow-up: ↓ 17 Changed 9 years ago by
End of very short attempt:
running build_ext building 'gsl' extension creating build/temp.macosx-10.4-ppc-2.6 creating build/temp.macosx-10.4-ppc-2.6/C gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/student/Desktop/sage-4.7.2.alpha2/local/include -I/Users/student/Desktop/sage-4.7.2.alpha2/local/include -I/Users/student/Desktop/sage-4.7.2.alpha2/local/include/python2.6 -c C/gsl.c -o build/temp.macosx-10.4-ppc-2.6/C/gsl.o gcc -L/Users/student/Desktop/sage-4.7.2.alpha2/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.4-ppc-2.6/C/gsl.o -L/Users/student/Desktop/sage-4.7.2.alpha2/local/lib -L/Users/student/Desktop/sage-4.7.2.alpha2/local/lib/gcc-lib/powerpc-apple-darwin6.8/4.0.3/ -L/Users/student/Desktop/sage-4.7.2.alpha2/local/lib -lm -llapack -lgsl -lgslcblas -lcblas -lf77blas -lf95 -lgsl -o build/lib.macosx-10.4-ppc-2.6/cvxopt/gsl.so /usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/ld: can't locate file for: -lcblas collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1 Error building/installing cvxopt real 0m7.116s user 0m0.953s sys 0m1.155s
Maybe I needed to do #10508 first? But you didn't say that ;-)
comment:17 in reply to: ↑ 16 Changed 9 years ago by
Replying to kcrisman:
End of very short attempt:
running build_ext building 'gsl' extension creating build/temp.macosx-10.4-ppc-2.6 creating build/temp.macosx-10.4-ppc-2.6/C gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/student/Desktop/sage-4.7.2.alpha2/local/include -I/Users/student/Desktop/sage-4.7.2.alpha2/local/include -I/Users/student/Desktop/sage-4.7.2.alpha2/local/include/python2.6 -c C/gsl.c -o build/temp.macosx-10.4-ppc-2.6/C/gsl.o gcc -L/Users/student/Desktop/sage-4.7.2.alpha2/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.4-ppc-2.6/C/gsl.o -L/Users/student/Desktop/sage-4.7.2.alpha2/local/lib -L/Users/student/Desktop/sage-4.7.2.alpha2/local/lib/gcc-lib/powerpc-apple-darwin6.8/4.0.3/ -L/Users/student/Desktop/sage-4.7.2.alpha2/local/lib -lm -llapack -lgsl -lgslcblas -lcblas -lf77blas -lf95 -lgsl -o build/lib.macosx-10.4-ppc-2.6/cvxopt/gsl.so /usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/ld: can't locate file for: -lcblas collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1 Error building/installing cvxopt real 0m7.116s user 0m0.953s sys 0m1.155sMaybe I needed to do #10508 first? But you didn't say that ;-)
No, 10508 updates Atlas, but on OSX we don't use Atlas at all?! Well, perhaps I was missing something: then you should start with the ticket 10508 depends on, then 10508 itself, and only then this one.
comment:18 follow-ups: ↓ 19 ↓ 21 Changed 9 years ago by
Correct, we don't. What ticket does #10508 depend on? I don't see a list.
Do we have to link to cblas instead of blas? I realize this is pretty naive, but I don't know whether that exists on Mac (yet). We would have to make it, maybe? See this ask.sagemath question.
comment:19 in reply to: ↑ 18 Changed 9 years ago by
What's the status of this ticket, given #12011 upgrading ATLAS?
Do we have to link to cblas instead of blas? I realize this is pretty naive, but I don't know whether that exists on Mac (yet). We would have to make it, maybe? See this ask.sagemath question.
See #12519, where Jeroen is pretty clearly that cblas doesn't exist on Mac, or at least not in the current situation (see #12011 also).
comment:20 Changed 9 years ago by
comment:21 in reply to: ↑ 18 Changed 9 years ago by
Replying to kcrisman:
Correct, we don't. What ticket does #10508 depend on? I don't see a list.
Do we have to link to cblas instead of blas? I realize this is pretty naive, but I don't know whether that exists on Mac (yet). We would have to make it, maybe? See this ask.sagemath question.
cblas
exists on Mac OSX in disguise. The equivalent of -lcblas
in compiler/linker options is -framework Accelerate
.
Although I have no idea how setup.py has to be modified to accommodate this.
comment:22 follow-up: ↓ 23 Changed 9 years ago by
In fact, at William's answer to that ask.sagemath.org question, he claims cblas
even exists 'normally', as /usr/lib/libcblas.dylib
(which it does on Snow Leopard, but not on Tiger). I got confused by #12519 because Jeroen was referring to it not being one of the Sage installed libraries, not that it didn't exist at all.
comment:23 in reply to: ↑ 22 Changed 9 years ago by
Replying to kcrisman:
In fact, at William's answer to that ask.sagemath.org question, he claims
cblas
even exists 'normally', as/usr/lib/libcblas.dylib
(which it does on Snow Leopard, but not on Tiger).
In which latter case I believe we use $SAGE_LOCAL/lib/libgslcblas.dylib
for such purposes.
comment:24 follow-up: ↓ 25 Changed 9 years ago by
It seems we should fall back to gslcblas on all old hardware or whenever atlas fails to build. But is it actually full-featured enough? Do all doctests pass if we just use gslcblas for everything?
comment:25 in reply to: ↑ 24 ; follow-up: ↓ 26 Changed 9 years ago by
Replying to vbraun:
It seems we should fall back to gslcblas on all old hardware or whenever atlas fails to build. But is it actually full-featured enough? Do all doctests pass if we just use gslcblas for everything?
Apparently. At any rate
find . -name "*blas.dylib" -print
in the home directory on my PPC box yields only the library mentioned above and $SAGE_LOCAL/lib/R/lib/libRblas.dylib
. So I guess this is ok, if perhaps "frickin' slow", according to an old linbox spkg-install.
comment:26 in reply to: ↑ 25 ; follow-up: ↓ 27 Changed 9 years ago by
Replying to kcrisman:
So I guess this is ok, if perhaps "frickin' slow", according to an old linbox spkg-install.
So is your PPC box ;-)
comment:27 in reply to: ↑ 26 Changed 9 years ago by
So I guess this is ok, if perhaps "frickin' slow", according to an old linbox spkg-install.
So is your PPC box ;-)
Touché! But beautiful; you wouldn't believe how many students come into the office and comment on how "you've got a really cool computer!"
comment:28 Changed 9 years ago by
- Milestone changed from sage-5.3 to sage-duplicate/invalid/wontfix
- Status changed from needs_info to needs_review
In the new atlas-3.10.0 the symbol is moved back to liblapack so we can close this ticket as invalid.
comment:29 Changed 9 years ago by
- Status changed from needs_review to positive_review
comment:30 Changed 9 years ago by
- Resolution set to invalid
- Reviewers set to Volker Braun
- Status changed from positive_review to closed
Volker, what platform do you get it on? I cannot reproduce it on Linux x86-64 Debian... Could it be an Altas hack done wrongly?