Opened 6 years ago

Closed 5 years ago

Last modified 5 years ago

#16706 closed enhancement (fixed)

Update IML to 1.0.4

Reported by: jpflori Owned by:
Priority: major Milestone: sage-6.4
Component: packages: standard Keywords: iml spkg
Cc: jdemeyer, leif, vbraun, fbissey Merged in:
Authors: Jean-Pierre Flori Reviewers: François Bissey, Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: d6d509c (Commits) Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

I've been in contact with Arne and he's very cooperative to psuh a new release on IML website.

Upstream: http://www.cs.uwaterloo.ca/~astorjoh/iml-1.0.4.tar.bz2

Should:

  • use different cblas and in particular pass easily something different from "-lcblas -latlas" to the linker,
  • and so correctly link the shared lib,
  • and so let shared libs be built on Cygwin (#15389),
  • anything else you want to fix.

Change History (75)

comment:1 Changed 6 years ago by jpflori

  • Cc jdemeyer leif vbraun added
  • Description modified (diff)

Any comments, rants, suggestions welcome.

Particular attention could be cast upon:

  • the new --with-cblas configure option/m4 macros,
  • the netlib cblas.h inclusion,
  • libtool versioning, I've just updated the revision part as should be, but before that it was naively set.

comment:2 Changed 6 years ago by jpflori

  • Cc fbissey added

comment:3 Changed 6 years ago by vbraun

Looks good to me. Do you have an eta on the official update? If its not imminent you shouldn't make your tarball look like the offical one. We't have to rename the official one later etc...

comment:4 Changed 6 years ago by jpflori

  • Description modified (diff)

Sure, it's not an official release and I don't want people to think so. It is though supposed to become the official one. Therefore I've added a .prerelease to the archive so that people trying it out will have to perform a non trivial action and be aware of what they do.

comment:5 Changed 6 years ago by jpflori

  • Description modified (diff)

comment:6 Changed 6 years ago by fbissey

Hum... So far I am not too successful with configuring with openblas(--with-cblas=-lopenblas), investigating further.

comment:7 Changed 6 years ago by jpflori

That's an important matter. Is it installed in standard location? Does it offer standard CBLAS (or whatever this should be called) API?

comment:8 Changed 6 years ago by jpflori

If not in a standrad location (or just /usr/ and /usr/lib I think, maybe even stranger for feaders, that is don't count on /usr/lib/x86_64-linux-..... which is not search for by default), you have to pass --with-blas-include= and -with-cblas-lib= (or variation therof, too lazy to checkk now, too late in my timezone).

comment:9 Changed 6 years ago by fbissey

No it's all in standard location including the headers. I checked all that stuff. It looks like there is an issue when I feed the configure script with results from pkg-config, if I do things manually there is no problems. While this may be more of an issue with pkg-config than anything else I may have a go at doing something different for clas-check.m4 I don't think you do the right thing with the AC_TRY_LINK test. It looks enough in most cases I am sure but this is not what I would do.

comment:10 follow-up: Changed 6 years ago by fbissey

Blush. OK I found the problem. While openblas install a cblas.h in the standard location it wants to load another header, not in a standard location. Not sure how I managed to configure it once without that additional header.

After further tinkering around with the cblas-check.m4 I think it is OK as it is. No need to over-engineer it.

Now one thing that tripped me is the fact that your tarball include a config.log where cblas and atlas are not found. And I kept looking at it to find what was wrong while portage was always doing an out-of-source build in a different directory (and I knew about it, why did I keep looking at the config.log in the wrong place I don't know). Could you please remove that piece of cruft that you generated on gcc1-power7.osuosl.org from the final tarball. Also, now that I have it open in front of my eyes you may want to bump the following line to 1.0.4 near the top of configure.ac

AC_INIT(IML, 1.0.3)

comment:11 Changed 6 years ago by fbissey

Dang! Your insistence on people not shooting themselves in the foot runs completely afoul of the set up I am testing. Because I am setting --with-cblas-include to grab my extra header you check --with-cblas-lib is not empty but "pkg-config --libs-only-L cblas" is empty because the library is in a standard location.

Your reasonable assumption that blas is all nicely installed under a prefix with one include file and one (possibly two) library is falling apart on my system. It wouldn't be a problem if I didn't try to script it with pkg-config (or someone didn't completely mess up how openblas is installed, it may be easier to fix that).

comment:12 follow-up: Changed 6 years ago by fbissey

After some thoughts if I fix things specifically for openblas it may blow in my face for some other proprietary implementations. There are two things:

  • I cannot think of any other packages that looks for the cblas.h header, doesn't mean it is wrong to look for it, just unusual - most other package assume it or bring it with them.
  • You could relax the need to have both variables defined at the same time and make configure spit a message asking whether both need to be set in case of failure.

Failing that, I will port the patch for iml 1.0.3 to just use pkg-config in the gentoo ebuild.

comment:13 in reply to: ↑ 10 Changed 6 years ago by jpflori

Replying to fbissey:

Blush. OK I found the problem. While openblas install a cblas.h in the standard location it wants to load another header, not in a standard location. Not sure how I managed to configure it once without that additional header.

You mean there is an "#include ..." in cblas.h which points to a header not found?? so you have to sepcify a bunch of "-I" to be able to use it?

After further tinkering around with the cblas-check.m4 I think it is OK as it is. No need to over-engineer it.

Now one thing that tripped me is the fact that your tarball include a config.log where cblas and atlas are not found. And I kept looking at it to find what was wrong while portage was always doing an out-of-source build in a different directory (and I knew about it, why did I keep looking at the config.log in the wrong place I don't know). Could you please remove that piece of cruft that you generated on gcc1-power7.osuosl.org from the final tarball.

Oups, I'll remove this for sure.

Also, now that I have it open in front of my eyes you may want to bump the following line to 1.0.4 near the top of configure.ac

AC_INIT(IML, 1.0.3)

Oups, will be fixed.

comment:14 in reply to: ↑ 12 Changed 6 years ago by jpflori

Replying to fbissey:

After some thoughts if I fix things specifically for openblas it may blow in my face for some other proprietary implementations. There are two things:

  • I cannot think of any other packages that looks for the cblas.h header, doesn't mean it is wrong to look for it, just unusual - most other package assume it or bring it with them.

Ok, I'll always fallback to the provided cblas.h if not found in standard locations.

  • You could relax the need to have both variables defined at the same time and make configure spit a message asking whether both need to be set in case of failure.

Will do that, it will ease the use of the provided cblas.h.

Failing that, I will port the patch for iml 1.0.3 to just use pkg-config in the gentoo ebuild.

Sure pkg-config would be nice, I just don't want to add that as a dependency to IML as it's not my software, I'm just helping Arne here.

comment:15 follow-up: Changed 6 years ago by fbissey

cblas.h from openblas has a line

#include "openblas_config.h"

unfortunately we install it in /usr/include/openblas. There is a convoluted logic to it and since we do everything with .pc files it normally doesn't matter - we may move away from that model in the long term.

I am not suggesting you use pkg-config to find cblas because it is not, as yet, standard. But If I patch it for Gentoo I'll save myself some trouble in the rest of the building script since all cblas you install in Gentoo will have a cblas.pc file - we add one if it is not already included.

But if you do the relaxing I don't have to patch anything.

comment:16 in reply to: ↑ 15 Changed 6 years ago by jpflori

Replying to fbissey:

cblas.h from openblas has a line

#include "openblas_config.h"

unfortunately we install it in /usr/include/openblas. There is a convoluted logic to it and since we do everything with .pc files it normally doesn't matter - we may move away from that model in the long term.

I see. And I assume this "first" cblas.h header is also in a non standard location? (Just to be sure it won't get picked up if you don't feed --with-cblas-inc anything, otherwise, I'll just make it so the provided IML cblas.h provided header is used all the time)

comment:17 Changed 6 years ago by fbissey

It's complicated. We allow people to install several blas/cblas/lapack side by side. One thing that is done to achieve that, is that headers, indeed, are put in a non-standard location (/usr/include/${somename}). Then you have a mechanism to choose one as the system default. That mechanism will link the corresponding cblas.h in /usr/include but no other headers. After the adventures of today I am not sure it even make sense to create that link.

That's the state of cblas headers as far as I know

  • atlas has an atlas.h header and a whole atlas subfolder - but cblas.h is nicely self contained.
  • gotoblas2/openblas calls an extra header
  • netlib has just cblas.h
  • gsl has gsl_cblas.h which is fortunately self contained
  • blis sort of has a (c)blas layer in blis/bli_blas.h that is full of blis subheader

That's all for open source as far as I know. I don't know what the proprietary look like (well I do for IBM ESSL but I'd rather not make this an ESSL flame).

comment:18 follow-up: Changed 6 years ago by jpflori

What is strange is that by default (and in fact even if you pass something to --with-cblas) the m4 macro looks in /usrlocalinclude for cblas.h so would have picked up your openblas cblas.h and failed to compile, but you said it didn't? didn't you?

comment:19 Changed 6 years ago by git

  • Commit changed from bcf0b32039974611af478764a733ecf5bf710176 to 24889a1038bfb7e939d09140b56c470bd628cf73

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

24889a1IML update.

comment:20 Changed 6 years ago by git

  • Commit changed from 24889a1038bfb7e939d09140b56c470bd628cf73 to 2380ff5e35628820fffe501ddba393e8702085d2

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

2380ff5IML update.

comment:21 Changed 6 years ago by jpflori

Here come a more proper archive made by "make dist-bzip2". If François is satisfied, I'll forward it to Arne so that he can do the actual release.

comment:22 in reply to: ↑ 18 Changed 6 years ago by fbissey

Replying to jpflori:

What is strange is that by default (and in fact even if you pass something to --with-cblas) the m4 macro looks in /usrlocalinclude for cblas.h so would have picked up your openblas cblas.h and failed to compile, but you said it didn't? didn't you?

I have had one success at configuring it and I am not sure how it happened. In fact I am starting to think I hallucinated that success or somehow was looking at some other code. The macro cannot possibly succeed without extra include information because the compiler cannot find openblas_config.h without it.

Refetching and retrying.

comment:23 Changed 6 years ago by fbissey

Something funny is happening. I can make it work by setting CPPFLAGS but it should work with the configuration option. I don't understand why it doesn't. I am making things more informative for my debugging.

comment:24 Changed 6 years ago by fbissey

Got it!

		if test "x${CBLAS_HEADER}" != "x/usr/include" -a "x${CBLAS_HEADER}" != "x/usr/local/include"; then
			for CBLAS_LIBRARY in ${CBLAS_LIBRARY_PATH} 
			do
				if test "x${CBLAS_LIBRARY}" != "x/usr/lib" -a "x${CBLAS_LIBRARY}" != "x/usr/local/lib"; then
					CBLAS_CFLAGS="-I${CBLAS_HEADER}"
					CBLAS_LIBS="${CBLAS_LIBS} -L${CBLAS_LIBRARY}"
				fi
			done

CBLAS_CFLAGS needs to be set outside of the test for CBLAS_LIBRARY. If you only have standard paths in CBLAS_LIBRARY_PATH, CBLAS_CFLAGS is never set. I would do

		if test "x${CBLAS_HEADER}" != "x/usr/include" -a "x${CBLAS_HEADER}" != "x/usr/local/include"; then
                        CBLAS_CFLAGS="-I${CBLAS_HEADER}"
			for CBLAS_LIBRARY in ${CBLAS_LIBRARY_PATH} 
			do
				if test "x${CBLAS_LIBRARY}" != "x/usr/lib" -a "x${CBLAS_LIBRARY}" != "x/usr/local/lib"; then
					CBLAS_LIBS="${CBLAS_LIBS} -L${CBLAS_LIBRARY}"
				fi
			done

instead. Nit-picky, should you do

CBLAS_LIBS="-L${CBLAS_LIBRARY} ${CBLAS_LIBS}"

instead of

CBLAS_LIBS="${CBLAS_LIBS} -L${CBLAS_LIBRARY}"

not sure if there are picky linker that would bother.

comment:25 follow-up: Changed 6 years ago by jpflori

Groumpf. You got me on the second one.

I must admit I saw this one and just thought "come on gcc is not so picky and does not care if the -L come after the -l". But it does.

comment:26 in reply to: ↑ 25 Changed 6 years ago by fbissey

Replying to jpflori:

Groumpf. You got me on the second one.

I must admit I saw this one and just thought "come on gcc is not so picky and does not care if the -L come after the -l". But it does.

Technically I am fairly sure that's not a complaint from gcc but from ld. gcc would pass those "as-is" to ld.

comment:27 Changed 6 years ago by fbissey

I would do the sequence of flag setting the following way

	if test -r "${CBLAS_HEADER}/cblas.h"; then
		if test "x${CBLAS_HEADER}" != "x/usr/include" -a "x${CBLAS_HEADER}" != "x/usr/local/include"; then
			CBLAS_CFLAGS="-I${CBLAS_HEADER}"
		else
			CBLAS_CFLAGS=
		fi
		for CBLAS_LIBRARY in ${CBLAS_LIBRARY_PATH} 
		do
			if test "x${CBLAS_LIBRARY}" != "x/usr/lib" -a "x${CBLAS_LIBRARY}" != "x/usr/local/lib"; then
				CBLAS_LIBS="-L${CBLAS_LIBRARY} ${CBLAS_LIBS}"
			else
				CBLAS_LIBS="${CBLAS_LIBS}"
			fi
		done

The current way, even with my small change in the previous comment, assume that if the header is in the standard place then the library is as well. This code makes no such assumption.

comment:28 Changed 6 years ago by jpflori

Thanks, Ill have a look tomorrow. Too late here for actual thinking.

Then I'll wait for your hopefully last feedback.

comment:29 Changed 6 years ago by jpflori

Your changes make sense.

I updated the tarball, please have a look.

comment:30 Changed 6 years ago by git

  • Commit changed from 2380ff5e35628820fffe501ddba393e8702085d2 to cb1b8f5e0936846bac46d93d874be282965fccab

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

cb1b8f5IML update.

comment:31 Changed 6 years ago by fbissey

My pathological case configure, compile and pass tests out of the box now :)

The only funny warning comes from IML and openblas each wanting to define "VERSION" at compilation time but that's completely harmless.

comment:32 Changed 6 years ago by fbissey

In case it wasn't clear it is ready as far as I am concerned.

comment:33 Changed 6 years ago by jpflori

Got that. I trnasmitted everything to Arne, just waiting for his feedback and official release.

comment:34 Changed 6 years ago by jpflori

We're fixing a bunch of doc stuff with Arne right now. The release will hopefully follow soon.

comment:35 Changed 6 years ago by jpflori

Official release done.

What's left here is to make sure we link to the correct CBLAS version depending on the OS. Or just do as before and link to nothing.

comment:36 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:37 Changed 6 years ago by jdemeyer

  • Description modified (diff)

comment:38 Changed 6 years ago by jdemeyer

  • Branch changed from u/jpflori/iml-1.0.4 to u/jdemeyer/ticket/16706
  • Created changed from 07/23/14 13:06:37 to 07/23/14 13:06:37
  • Modified changed from 08/20/14 09:49:43 to 08/20/14 09:49:43

comment:39 Changed 6 years ago by fbissey

  • Commit changed from cb1b8f5e0936846bac46d93d874be282965fccab to 21194b52d2239d2fef6ed497aaee700a95a07157

Somehow I missed the message about the release. I will be ready to review when it is done.


New commits:

21194b5Upgrade to IML 1.0.4

comment:40 Changed 6 years ago by jpflori

Note that if we want to correctly link IML to some CBLAS some (potentially non-trivial) work is needed (think of the linbox and similar stuff...).

The easy way is to repatch IML configure system to avoid cblas detection and don't link as before.

comment:41 Changed 6 years ago by fbissey

Well, I am not taking linbox/fflas-ffpack as a good reference but can you elaborate on what you think is the potential problem? We got iml to the point where we can configure it with any cblas we want. That means sage's ATLAS or a system ATLAS by default but also Apple's vectorize on OS X. Do you mean that you want to avoid auto-detection? That means we should always set a cblas, I certainly hope that's what we want to do.

comment:42 follow-up: Changed 6 years ago by jpflori

I mean there is no autodetection in IML configure system. So we have to be smart on the Sage side.

comment:43 in reply to: ↑ 42 ; follow-up: Changed 6 years ago by fbissey

Replying to jpflori:

I mean there is no autodetection in IML configure system. So we have to be smart on the Sage side.

That's not quite true. It still looks for atlas if nothing is provided. Doubly so.

IML_CHECK_CBLAS(,,[AC_MSG_WARN(
CBLAS not found!
Please make sure that --with-cblas=<linker flags> and optional --with-cblas-include=<path> and --with-cblas-lib=<path> are correctly set.
Trying legacy ATLAS configuration.)
IML_CHECK_ATLAS(,,[AC_MSG_ERROR(
ATLAS not found!
ATLAS version 3.0 or greater is required for this library to compile. Please make sure ATLAS is installed and specify the header and libraries location with the options -$
])
])

in configure.ac. If the check for cblas fails it runs the atlas test.

AC_ARG_WITH(cblas,
             [  --with-cblas=<linker flags>
                Specify the flags to use to link to cblas.
                If argument is <empty> that means that ATLAS is installed
                and -lcblas -latlas should be used.
              ],
              [
                CBLAS_LIBS=$withval
              ],
              [
                CBLAS_LIBS="-lcblas -latlas"
            ])

in cblas-check.m4. If we provide just "--with-cblas" atlas is assumed.

comment:44 in reply to: ↑ 43 Changed 6 years ago by jdemeyer

  • Status changed from new to needs_review

Replying to fbissey:

AC_ARG_WITH(cblas,
             [  --with-cblas=<linker flags>
                Specify the flags to use to link to cblas.
                If argument is <empty> that means that ATLAS is installed
                and -lcblas -latlas should be used.
              ],
              [
                CBLAS_LIBS=$withval
              ],
              [
                CBLAS_LIBS="-lcblas -latlas"
            ])

in cblas-check.m4. If we provide just "--with-cblas" atlas is assumed.

More precisely, the CBLAS_LIBS="-lcblas -latlas" is used when no --with-cblas option is given, which is currently the case in spkg-install. I think this is what we want, so I personally see no further obstructions for this ticket.

comment:45 follow-up: Changed 6 years ago by jpflori

Will that work on OS X? Wasn't there problems some time ago because the Apple Acceleration thingy decided to stop pretending it was ATLAS?

comment:46 Changed 6 years ago by jpflori

And note than on recent Solaris you need to link to "-lrt" as well. But I'm not sure we have access to Solaris anymore anyway.

comment:47 Changed 6 years ago by git

  • Commit changed from 21194b52d2239d2fef6ed497aaee700a95a07157 to 160fbbc740688d9283c22ece9ca2c4790e6d2752

Branch pushed to git repo; I updated commit sha1. New commits:

160fbbcRemove the special case for external ATLAS libraries

comment:48 Changed 6 years ago by jdemeyer

I have no idea if it works on other systems or not, but if it doesn't, we should try to solve the problem upstream, not work around it like for linbox.

comment:49 follow-up: Changed 6 years ago by jpflori

I don't agree. Picking up the right linker flags, include path, lmib path for any CBLAS implem is black magic. I'm not even sure the autotools suite provides anything to pick properly a working CBLAS implem on every system. Using pkg-config could ease all of that but I wouldn't say it's an option for us or IML.

comment:50 in reply to: ↑ 49 ; follow-up: Changed 6 years ago by jdemeyer

Replying to jpflori:

I'm not even sure the autotools suite provides anything to pick properly a working CBLAS implem on every system.

Google found this: http://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=blob_plain;f=m4/ax_blas.m4, but I haven't tested it...

comment:51 in reply to: ↑ 50 ; follow-up: Changed 6 years ago by fbissey

Replying to jdemeyer:

Replying to jpflori:

I'm not even sure the autotools suite provides anything to pick properly a working CBLAS implem on every system.

Google found this: http://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=blob_plain;f=m4/ax_blas.m4, but I haven't tested it...

That one is for blas (fortran) not cblas. There is a ax_cblas.m4 all these are relatively old. blas detection is non trivial because of all the possible libraries that can provide it or part of it. If everyone provided just a library even as a link or better on linux a linker script it wouldn't be a problem. In that case feeding configure appropriate stuff is the answer.

So yes currently on all systems we ship atlas the config will work without anything. I have looked at spkg-install last night and I wasn't impressed with "--with-atlas-lib=/usr/lib". Anyway --with-atlas* options are gone. So clean up is in order.

comment:52 in reply to: ↑ 51 Changed 6 years ago by jdemeyer

Replying to fbissey:

That one is for blas (fortran) not cblas.

OK, fair enough but maybe it can be adapted for C...

There is a ax_cblas.m4

...or we could use this one.

all these are relatively old.

Doesn't matter if they work (I haven't tried). If they don't work for Sage, they might serve as a good starting point.

blas detection is non trivial because of all the possible libraries that can provide it or part of it.

That's exactly why you want to solve this problem once, with one proper autoconf script. If every spkg-install script needs to be rewritten to detect ATLAS, you're doing it wrong.

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:53 Changed 6 years ago by fbissey

We are going slightly off-topic but that's something that the science group in Gentoo has studied for a while. Right any cblas/blas/lapack is installing a .pc file and every ebuild is required to use it. We are thinking of replacing those by a linker script with a standard name in the future.

On another note the current configure script from IML already borrow quite a bit of logic from ax_blas.m4. If you use something like ax_cblas.m4 you'll have to add -L$SAGE_LOCAL/lib in LDFLAGS and -I$SAGE_LOCAL/include in CPPFLAGS anyway because it assumes things to be installed in a standard location system wide.

If you write a comprehensive macro that covers all the cases that's very good, pass it on and I can tell you I will torture it to death and give you the appropriate feedback.

comment:54 in reply to: ↑ 45 Changed 6 years ago by fbissey

Replying to jpflori:

Will that work on OS X? Wasn't there problems some time ago because the Apple Acceleration thingy decided to stop pretending it was ATLAS?

I don't think spkg-install as is will work on OS X however we don't need something very involved I believe. I don't remember the logic behind the stuff fflas-pac (the layer split off linbox) but nothing that complicated should be needed.

comment:55 follow-up: Changed 6 years ago by vbraun

Upstream pulled a fast one on us:

Found local metadata for iml-1.0.4
Found local sources at /Users/vbraun/Sage/upstream/iml-1.0.4.tar.bz2
Checksum: 713a19f1076ddde07a7c3a45493440b745bd266b vs 5bfc185ce5704919ae27ef0b459298a4bd676918
Invalid checksum for /Users/vbraun/Sage/upstream/iml-1.0.4.tar.bz2

Also, at least the previous IML did not pass its testsuite on OSX because the testsuite actually tries to link against atlas (maybe just deactivate the testsuite on darwin for now). IML itself doesn't, it just leaves cblas undefined.

$ ldd local/lib/libiml.so
	linux-vdso.so.1 =>  (0x00007fff34896000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ffc1c58b000)
	libgmp.so.11 => /home/vbraun/Code/sage.git/local/lib/libgmp.so.11 (0x00007ffc1c31c000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ffc1bf5f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffc1cad2000)
$ readelf -a local/lib/libiml.so | grep -i blas
000000223098  000900000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dscal + 0
0000002230b0  000b00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dgemv + 0
000000223130  001300000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_daxpy + 0
0000002231f0  001d00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dgemm + 0
0000002232c0  002a00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dcopy + 0
0000002233c0  003d00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dger + 0
0000002233c8  003e00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dswap + 0
     9: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dscal
    11: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dgemv
    19: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_daxpy
    29: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dgemm
    42: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dcopy
    61: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dger
    62: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dswap
    89: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dscal
    92: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dgemv
   114: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_daxpy
   141: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dgemm
   170: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dcopy
   209: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dger
   210: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dswap

comment:56 Changed 6 years ago by jpflori

Sorry, I'm completely out of time these days, so no Sage devel for me before at least one week...

comment:57 Changed 6 years ago by fbissey

I need to take a look. I indicated earlier that I didn't think it would work on OS X, it doesn't do the right thing to get the accelerate libraries or even just blas.

comment:58 in reply to: ↑ 55 Changed 6 years ago by fbissey

Replying to vbraun:

Upstream pulled a fast one on us:

Found local metadata for iml-1.0.4
Found local sources at /Users/vbraun/Sage/upstream/iml-1.0.4.tar.bz2
Checksum: 713a19f1076ddde07a7c3a45493440b745bd266b vs 5bfc185ce5704919ae27ef0b459298a4bd676918
Invalid checksum for /Users/vbraun/Sage/upstream/iml-1.0.4.tar.bz2

Also, at least the previous IML did not pass its testsuite on OSX because the testsuite actually tries to link against atlas (maybe just deactivate the testsuite on darwin for now). IML itself doesn't, it just leaves cblas undefined.

$ ldd local/lib/libiml.so
	linux-vdso.so.1 =>  (0x00007fff34896000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ffc1c58b000)
	libgmp.so.11 => /home/vbraun/Code/sage.git/local/lib/libgmp.so.11 (0x00007ffc1c31c000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ffc1bf5f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffc1cad2000)
$ readelf -a local/lib/libiml.so | grep -i blas
000000223098  000900000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dscal + 0
0000002230b0  000b00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dgemv + 0
000000223130  001300000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_daxpy + 0
0000002231f0  001d00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dgemm + 0
0000002232c0  002a00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dcopy + 0
0000002233c0  003d00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dger + 0
0000002233c8  003e00000007 R_X86_64_JUMP_SLO 0000000000000000 cblas_dswap + 0
     9: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dscal
    11: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dgemv
    19: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_daxpy
    29: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dgemm
    42: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dcopy
    61: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dger
    62: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dswap
    89: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dscal
    92: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dgemv
   114: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_daxpy
   141: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dgemm
   170: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dcopy
   209: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dger
   210: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cblas_dswap

Do you have the config.log for that? The configuration should have broken down unless upstream changed something since we worked on it with Jean-Pierre.

comment:59 Changed 6 years ago by jpflori

Not sure what Volker actually reported as he states "the previous version" and indeed with the previous version we shipped (1.0.3) the IML testsuite did not pass and the IML library did not link to any cblas.

comment:60 Changed 6 years ago by vbraun

Yes I was referring to 1.0.3.

comment:61 Changed 6 years ago by fbissey

Oh yes, now I get it. It was not even OS X but linux. Yes I forgot vanilla leaves it undefined so you could plg in any cblas when you create an executable.

comment:62 Changed 6 years ago by jpflori

Nope vanilla does not do that. Vanilla used to only be able to link to ATLAS and Sage patched it so that it does not link to anything (and does not fail on systems not providing an ATLAS like linking interface).

comment:63 Changed 6 years ago by fbissey

I stand corrected.

comment:64 Changed 6 years ago by fbissey

In any case the spkg-install + iml upstream cannot configure on OS X without a modification, we need to pass -lcblas explicitly on OS X.

comment:65 Changed 6 years ago by fbissey

  • Branch changed from u/jdemeyer/ticket/16706 to u/fbissey/iml-1.0.4
  • Commit changed from 160fbbc740688d9283c22ece9ca2c4790e6d2752 to 9e4ad6b79b27192ac5e1a0fe57660e59ad2eed3e

This last commit on top of Jeroen's change should enable iml 1.0.4 to find accelerate on OS X. There may be a corner case if libclas.dylib is not available on older system. I will check on 10.5.8 system tomorrow. It is trivial to add a use case to fall back on glscblas.


New commits:

9e4ad6bMake iml 1.0.4 find system cblas on darwin

comment:66 Changed 5 years ago by fbissey

  • Reviewers set to François Bissey, Jeroen Demeyer
  • Status changed from needs_review to positive_review

I think it is time to go ahead with this. It will work with OS X 10.5.x at least.

comment:67 Changed 5 years ago by vbraun

  • Branch changed from u/fbissey/iml-1.0.4 to 9e4ad6b79b27192ac5e1a0fe57660e59ad2eed3e
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:68 Changed 5 years ago by vbraun

  • Commit 9e4ad6b79b27192ac5e1a0fe57660e59ad2eed3e deleted
  • Resolution fixed deleted
  • Status changed from closed to new

Finally the OSX buildbot works again, but IML does not on OSX:

checking for CBLAS... not found
configure: WARNING: CBLAS not found!
Please make sure that --with-cblas=<linker flags> and optional --with-cblas-include=<path> and --with-cblas-lib=<path> are correctly set.
Trying legacy ATLAS configuration.
checking for ATLAS >= 3.0... not found
configure: error: ATLAS not found!
ATLAS version 3.0 or greater is required for this library to compile. Please make sure ATLAS is installed and specify the header and libraries location with the options --with-atlas-include=<path> and --with-atlas-lib=<path> respectively when running configure.
Error configuring IML.

comment:69 Changed 5 years ago by fbissey

I thought I had tested it on my mac but I must have been logged in the wrong computer. I believe it fails to find cblas.h, will find a fix shortly.

comment:70 Changed 5 years ago by fbissey

  • Branch changed from 9e4ad6b79b27192ac5e1a0fe57660e59ad2eed3e to u/fbissey/iml-1.0.4
  • Commit set to d6d509c03b39cb34e07ef449c2329da78d0f3a1f
  • Status changed from new to needs_review

Ok used gsl_cblas.h headers to make it compile. We may want to install this file as SAGE_LOCAL/include/cblas.h on OS X just in case. Bonus: as I mentioned on the mailing list iml now pass its test suite on OS X.


New commits:

21194b5Upgrade to IML 1.0.4
160fbbcRemove the special case for external ATLAS libraries
9e4ad6bMake iml 1.0.4 find system cblas on darwin
d6d509cLast fix for OS X. There are no cblas.h headers at all on the system. Using gsl_cblas.h for compilation.

comment:71 Changed 5 years ago by fbissey

  • Status changed from needs_review to positive_review

comment:72 Changed 5 years ago by vbraun

  • Branch changed from u/fbissey/iml-1.0.4 to d6d509c03b39cb34e07ef449c2329da78d0f3a1f
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:73 Changed 5 years ago by kcrisman

  • Commit d6d509c03b39cb34e07ef449c2329da78d0f3a1f deleted

On an old Mac (slightly newer than my usual test machine, but still OS X 10.4 PPC) iml bails out with complaining about not having ATLAS 3.0 or greater. These built Sage 6.4.beta6 fine (other than rpy2) so it must be something about this upgrade that changes it - I assume iml always asked for that, see for instance this nearly 8-year-old discussion!

Could it possibly be the removal of

--with-atlas-include="`pwd`" --with-atlas-lib=/usr/lib

Reading http://web.mit.edu/sage/export/iml-1.0.2/config/atlas-check.m4 certainly seems to indicate so.

Even though in general we will have to drop easy support for that old of a machine because gcc 4.0.1 won't compile gcc 4.9.x (you'd think it would have made sense for gcc to number it 5.0 when they changed the language they wrote it in!), I do want to make one final binary for Sage 6.4.1 at least, so I would appreciate any assistance. We don't need a new ticket, I don't think. I'll continue the conversation on https://groups.google.com/forum/#!topic/sage-release/xgmJ3nAcUOY

comment:74 Changed 5 years ago by kcrisman

And now I see comment:70 but apparently this doesn't work on the older ones...

comment:75 Changed 5 years ago by kcrisman

For the record, the simple change to

EXTRA_BLAS=""
if [ $UNAME = "Darwin" ]; then
    # copy cblas headers from gsl
    cp ../patches/gsl_cblas.h cblas.h
    EXTRA_BLAS="--with-cblas=-lgslcblas"
fi

works on that older platform, apparently.

Note: See TracTickets for help on using tickets.