update M4RI to newest upstream release

[Martin Albrecht]

As the title says.

Install: ​http://sage.math.washington.edu/home/malb/spkgs/libm4ri-20120613.spkg

Apply: trac_12840_m4ri_new_version.patch​, trac_12840-rebased-on-13109.patch​

Apply to SAGE_ROOT: trac_12840_m4ri_png.patch​

[Martin Albrecht]

Updated the patch because I forgot to raise a ZeroDivisionError in __invert__. Fixed now.

[Simon King]

After applying the patch and the spkg and rebuilding the polybori spkg, sage -br failed with

gcc -pthread -shared -L/mnt/local/king/SAGE/stable/sage-5.0.beta13/local/lib build/temp.linux-x86_64-2.7/sage/matrix/matrix_integer_dense.o -L/mnt/local/king/SAGE/stable/sage-5.0.beta13/local/lib -L/mnt/local/king/SAGE/stable/sage-5.0.beta13/local/lib -lcsage -liml -lpari -lm -lgmp -lcblas -latlas -lstdc++ -lntl -lpython2.7 -o build/lib.linux-x86_64-2.7/sage/matrix/matrix_integer_dense.so
error: command 'gcc' failed with exit status 1
Error installing modified sage library code.

No error message, I am afraid.

[Simon King]

PS: That was with sage-5.0.beta13 plus a lot of patches, but none of them changing matrices.

[Simon King]

And repeating sage -br, I finally got a proper error:

In file included from /mnt/local/king/SAGE/stable/sage-5.0.beta13/local/include/m4rie/m4rie.h:56,
                 from sage/matrix/matrix_mod2e_dense.cpp:240:
/mnt/local/king/SAGE/stable/sage-5.0.beta13/local/include/m4rie/permutation.h:30:30: error: m4ri/permutation.h: Datei oder Verzeichnis nicht gefunden
error: command 'gcc' failed with exit status 1
Error installing modified sage library code.

[Simon King]

PS: permutation.h seems not to be present in the sources.

[Simon King]

PS: The same error occurs when trying to rebuild libm4rie spkg: M4RIE expects permutation.h

[Martin Albrecht]

Yes, #12841 is a dependency of this ticket.

[Simon King]

Ouch. I missed #12841. Sorry.

[Simon King]

Am I right that both spkgs have to be installed, but #12840 goes before #12841?

[Simon King]

Yippie! Applying the patches from #12840 and #12841, installing the new m4ri spkg from here, then installing the new m4rie spkg from #12841 (why is it on a different ticket? Can one live without the other?), re-installing polybori and finally sage -br, Sage starts!

[Martin Albrecht]

Replying to SimonKing:

Yippie! Applying the patches from #12840 and #12841, installing the new m4ri spkg from here, then installing the new m4rie spkg from #12841 (why is it on a different ticket? Can one live without the other?),

No, you're right, they are co-dependent.

[Simon King]

First of all: make ptestlong works with the two new spkgs (even though I have a few other patches applied...).

Here is a small timing.

Vanilla Sage-5.0.rc0:

sage: K = GF(2)
sage: %time A = random_matrix(K,20000,20000)
CPU times: user 0.29 s, sys: 0.02 s, total: 0.31 s
Wall time: 0.31 s
sage: %time B = random_matrix(K,20000,20000)
CPU times: user 0.23 s, sys: 0.03 s, total: 0.26 s
Wall time: 0.27 s
sage: %time (A*B).rank()
CPU times: user 11.70 s, sys: 0.05 s, total: 11.75 s
Wall time: 11.81 s
19937

Sage-5.0.beta13 with the new spkgs (and some other patches that shouldn't affect the timing for matrix multiplication):

sage: K = GF(2)
sage: %time A = random_matrix(K,20000,20000)
CPU times: user 0.30 s, sys: 0.03 s, total: 0.33 s
Wall time: 0.33 s
sage: %time B = random_matrix(K,20000,20000)
CPU times: user 0.32 s, sys: 0.01 s, total: 0.33 s
Wall time: 0.33 s
sage: %time (A*B).rank()
CPU times: user 11.70 s, sys: 0.10 s, total: 11.80 s
Wall time: 11.83 s
19937

Well, at least there is no slow-down...

Do you claim that the new version of M4RI is faster? You do claim a speed-up for M4RIE, on #12841.

[Martin Albrecht]

Replying to SimonKing:

Do you claim that the new version of M4RI is faster? You do claim a speed-up for M4RIE, on #12841.

No, M4RI shouldn't have changed (at least the stuff, only bugfixes & refactoring.

That said, there's a new function for inverting upper triangular matrices which is much faster than the generic method, but it's not exposed yet. I need to implement the same for lower triangular matrices, then we can speed up inversions in general.

Furthermore, M4RI ships its own PNG reader/writer now which is much faster than what we have in Sage and uses much less memory. Again, it's not used yet. This patch just gets the new version in.

[Simon King]

Since make ptestlong passes and the hg repository in the package is fine and SPKG.txt is up to date, I think I can give a positive review. However, I only tested on one platform. So, if the release manager believes that this is not enough, then please speak up!

[Jeroen Demeyer]

This fails to install on the Skynet machine "eno" (Fedora 16 x86_64):

/bin/sh ./libtool --tag=CC   --mode=link gcc -std=gnu99 -mmmx -msse -msse2 -msse3   -g -fPIC -Wall -pedantic -O2 -release 0.0.20111203 -no
-undefined  -o libm4ri.la -rpath /home/buildbot/build/sage/eno-1/eno_full/build/sage-5.1.beta1/local/lib brilliantrussian.lo misc.lo mzd.l
o graycode.lo strassen.lo mzp.lo triangular.lo triangular_russian.lo ple.lo ple_russian.lo solve.lo echelonform.lo mmc.lo debug_dump.lo io
.lo -lm -lpng
libtool: link: gcc -shared  -fPIC -DPIC  .libs/brilliantrussian.o .libs/misc.o .libs/mzd.o .libs/graycode.o .libs/strassen.o .libs/mzp.o .
libs/triangular.o .libs/triangular_russian.o .libs/ple.o .libs/ple_russian.o .libs/solve.o .libs/echelonform.o .libs/mmc.o .libs/debug_dum
p.o .libs/io.o   -lm -lpng  -mmmx -msse -msse2 -msse3 -O2   -Wl,-soname -Wl,libm4ri-0.0.20111203.so -o .libs/libm4ri-0.0.20111203.so
/usr/bin/ld: cannot find -lpng
collect2: ld returned 1 exit status
make[1]: *** [libm4ri.la] Error 1
make[1]: Leaving directory `/home/buildbot/build/sage/eno-1/eno_full/build/sage-5.1.beta1/spkg/build/libm4ri-20120415/src'
make: *** [all] Error 2
Error building libm4ri

$ ls -l local/lib/libpng*
36k -rw-r--r-- 1 buildbot sage 828k May  8 06:16 local/lib/libpng12.a
4.1k -rwxr-xr-x 1 buildbot sage 1.1k May  8 06:16 local/lib/libpng12.la*
   0 lrwxrwxrwx 1 buildbot sage   18 May  8 06:16 local/lib/libpng12.so -> libpng12.so.0.35.0*
   0 lrwxrwxrwx 1 buildbot sage   18 May  8 06:16 local/lib/libpng12.so.0 -> libpng12.so.0.35.0*
467k -rwxr-xr-x 1 buildbot sage 460k May  8 06:16 local/lib/libpng12.so.0.35.0*

$ ls -l /lib*/libpng* /usr/lib*/libpng*
ls: cannot access /lib*/libpng*: No such file or directory
   0 lrwxrwxrwx 1 root root   16 May  2 11:53 /usr/lib64/libpng.so.3 -> libpng.so.3.49.0*
173k -rwxr-xr-x 1 root root 172k Apr  7 14:33 /usr/lib64/libpng.so.3.49.0*
   0 lrwxrwxrwx 1 root root   18 May  2 11:53 /usr/lib64/libpng12.so.0 -> libpng12.so.0.49.0*
164k -rwxr-xr-x 1 root root 162k Apr  7 14:33 /usr/lib64/libpng12.so.0.49.0*

Note that there is no file /usr/lib64/libpng.so, so obviously there cannot be proper PNG support on this machine. I consider this to be an upstream (not Sage) error.

$ grep -i png spkg/logs/libm4ri-20120415.log
checking whether to build with PNG support... yes
checking for PNG... no
checking for PNG... yes
/bin/sh ./libtool --tag=CC   --mode=link gcc -std=gnu99 -mmmx -msse -msse2 -msse3   -g -fPIC -Wall -pedantic -O2 -release 0.0.20111203 -no-undefined  -o libm4ri.la -rpath /home/buildbot/build/sage/eno-1/eno_full/build/sage-5.1.beta1/local/lib brilliantrussian.lo misc.lo mzd.lo graycode.lo strassen.lo mzp.lo triangular.lo triangular_russian.lo ple.lo ple_russian.lo solve.lo echelonform.lo mmc.lo debug_dump.lo io.lo -lm -lpng
libtool: link: gcc -shared  -fPIC -DPIC  .libs/brilliantrussian.o .libs/misc.o .libs/mzd.o .libs/graycode.o .libs/strassen.o .libs/mzp.o .libs/triangular.o .libs/triangular_russian.o .libs/ple.o .libs/ple_russian.o .libs/solve.o .libs/echelonform.o .libs/mmc.o .libs/debug_dump.o .libs/io.o   -lm -lpng  -mmmx -msse -msse2 -msse3 -O2   -Wl,-soname -Wl,libm4ri-0.0.20111203.so -o .libs/libm4ri-0.0.20111203.so
/usr/bin/ld: cannot find -lpng

[Jeroen Demeyer]

Why are you using PKG_CHECK_MODULES() to check for PNG support in configure.ac? Why not AC_CHECK_LIB() or AC_SEARCH_LIBS()?

[Jeroen Demeyer]

Exactly the same problem with -lpng happens on Skynet "cleo" (RHEL 5.3 ia64)

[Martin Albrecht]

Replying to jdemeyer:

Why are you using PKG_CHECK_MODULES() to check for PNG support in configure.ac? Why not AC_CHECK_LIB() or AC_SEARCH_LIBS()?

We are doing both. We try pkg-config first and if that fails we fall back to AC_CHECK_LIB. In your log above, PKG_CHECK_MODULES() fails to find libpng and AC_CHECK_LIB() finds it.

[Martin Albrecht]

Note that stand-alone M4RI builds fine on eno.

checking pkg-config is at least version 0.9.0... yes
checking for PNG... no
checking for PNG... no
checking for PNG... no
checking for PNG... no
checking for png_create_write_struct in -lpng... no

So I don't know why it fails when built inside Sage. In any case, Sage ships libpng so we should adapt deps to make M4RI dependent on libpng. I'll provide a patch to that effect.

[Jeroen Demeyer]

Replying to malb:

In your log above, PKG_CHECK_MODULES() fails to find libpng and AC_CHECK_LIB() finds it.

Actually, it's not quite like that. PKG_CHECK_MODULES() found libpng12 but you still try to link with -lpng instead of -lpng12 (the first failure comes from the libpng14 check). AC_CHECK_LIB() isn't called (but it would fail).

[Jeroen Demeyer]

I recommend against using PKG_CHECK_MODULES() and would use only AC_SEARCH_LIBS().

[Jeroen Demeyer]

The patch you attached is probably good, but that's just part of the problem.

[Martin Albrecht]

Replying to jdemeyer:

The patch you attached is probably good, but that's just part of the problem.

Can you elaborate on that? I don't understand.

[Jeroen Demeyer]

Replying to malb:

Can you elaborate on that? I don't understand.

Well, I tried to explain in ​http://trac.sagemath.org/sage_trac/ticket/12840#comment:28. The fact that

PKG_CHECK_MODULES([PNG], [libpng12])

succeeds, does not imply that the library file libpng.so exists. I don't see what you're trying to accomplish with those PKG_CHECK_MODULES() checks.

[Jeroen Demeyer]

I can try to write a patch if you like.

[Martin Albrecht]

Replying to jdemeyer:

Replying to malb:

Can you elaborate on that? I don't understand.

PKG_CHECK_MODULES([PNG], [libpng12])

succeeds, does not imply that the library file libpng.so exists.

Ah, right. It would need -lpng12. Yep, that's a bug. I'll fix that next week-ish.

I don't see what you're trying to accomplish with those PKG_CHECK_MODULES() checks.

I am trying to use pkg-config for what it's intended. Making linking easier.

[Jeroen Demeyer]

Maybe you're using it for what it's intended, but not how it's intended.

[Martin Albrecht]

Replying to jdemeyer:

Replying to malb:

In your log above, PKG_CHECK_MODULES() fails to find libpng and AC_CHECK_LIB() finds it.

Actually, it's not quite like that. PKG_CHECK_MODULES() found libpng12 but you still try to link with -lpng instead of -lpng12 (the first failure comes from the libpng14 check). AC_CHECK_LIB() isn't called (but it would fail).

For some reason I missed this comment earlier, sorry. That makes sense & is an upstream bug which I intend to fix.

[Martin Albrecht]

Replying to jdemeyer:

Maybe you're using it for what it's intended, but not how it's intended.

This is a remarkably unhelpful comment.

[Jeroen Demeyer]

Replying to malb:

This is a remarkably unhelpful comment.

I know, but I have never done anything with pkgconfig, so I can't help more.

[Martin Albrecht]

I've updated the SPKG with a new upstream release which should fix the png issue.

​http://sage.math.washington.edu/home/malb/spkgs/libm4ri-20120522.spkg

[Martin Albrecht]

Could someone please review this. I am looking into updating LinBox and updating M4RIE is a requirement for that (long story involving Givaro).

[Simon King]

Replying to malb:

Could someone please review this. I am looking into updating LinBox and updating M4RIE is a requirement for that (long story involving Givaro).

I could test the code on one or two linuxes. But I couldn't test whether the png problems on eno (as observed by Jeroen) are fixed. Jeroen, could you finish the review?

[Jeroen Demeyer]

The previous version installed fine on eno within Sage. It seems the libpng check is totally gone now.

[Martin Albrecht]

Replying to jdemeyer:

The previous version installed fine on eno within Sage.

It seems the libpng check is totally gone now.

That's strange, it doesn't check?

I had to update SPKG because I didn't add "-I$SAGE_LOCAL/include" before but we need it now because of the PNG dependency. The newest SPKG installs fine on geom.math which doesn't have a system libpng-dev.

[Volker Braun]

I've tried this on Fedora 17 (which doesn't have libpng12 either, its up to libpng15 now) and configure works as intended:

checking whether to build with PNG support... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for PNG... yes

Jeroen, you might have overlooked this part in the log since it comes after the lengthy "checking for cache sizes...".

I'm building on eno right now and if it works I'll set this ticket to positive review.

[Martin Albrecht]

So Fedora has libpng.so then? I am asking because I am not checking for png15 explicitly.

if test "x${want_png}" = "xyes" ; then
   PKG_CHECK_MODULES([PNG], [libpng],
      [have_libpng="yes"; LIBPNG_LIBADD=`pkg-config --libs libpng`],
      [have_libpng="no"])
   if ! test "x${have_libpng}" = "xyes" ; then
      AC_CHECK_LIB([png],
         [png_create_write_struct],
         [have_libpng="yes"; LIBPNG_LIBADD="-lpng"],
         [AC_CHECK_LIB([png14],
            [png_create_write_struct],
            [have_libpng="yes"; LIBPNG_LIBADD="-lpng14"],
            [AC_CHECK_LIB([png12],
               [png_create_write_struct],
               [have_libpng="yes"; LIBPNG_LIBADD="-lpng12"],
               [AC_CHECK_LIB([png10],
                  [png_create_write_struct],
                  [have_libpng="yes"; LIBPNG_LIBADD="-lpng10"],
                  [have_libpng="no"])
               ])
            ])
        ])
   fi
   if test "x${have_libpng}" = "xno" ; then
      AC_MSG_WARN([Can not find a usuable PNG library. Make sure that CPPFLAGS and LDFLAGS are correctly set.])
   fi
fi

[Martin Albrecht]

Replying to malb:

So Fedora has libpng.so then? I am asking because I am not checking for png15 explicitly. 

Argh, I am stupid. It would find libpng12 as included in Sage. Sorry bout the noise.

[Volker Braun]

Yes the PKG_CHECK_MODULES test wins in Sage, which is the way it should be. pkg-config is the way to go imho. Though if libpng were to use standard libtool versioning we would live in a better world, too ;-)

In any case, built and tested fine on eno. Positive review.

[John Palmieri]

The deprecations here should use the format from #13109.

[Volker Braun]

The deprecations are not doctested.

[Volker Braun]

Hmm deprecations (and any other warnings) don't work in cdef methods. I guess we can't doctest them, for now.

[Jeroen Demeyer]

This needs to be rebased to sage-5.2.rc1.

[John Palmieri]

Can you clarify the rebasing issues? I have no problems applying the patches from here and from #12841 on top of sage-5.2.rc1.

[Volker Braun]

I'm with John here, I've been carrying the mari/e/givaro/linbox patches in my queue for a while now and there was no breakage in sage-5.2.rc1 as far as I can tell.

[Jeroen Demeyer]

My mistake, I only applied the second Sage library patch and that didn't work.