Opened 11 years ago
Closed 10 years ago
#12840 closed enhancement (fixed)
update M4RI to newest upstream release
Reported by: | Martin Albrecht | Owned by: | tbd |
---|---|---|---|
Priority: | major | Milestone: | sage-5.3 |
Component: | packages: standard | Keywords: | |
Cc: | Merged in: | sage-5.3.beta0 | |
Authors: | Martin Albrecht, John Palmieri | Reviewers: | Simon King, Volker Braun |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | to be merged with #12841, #13109 | Stopgaps: |
Description (last modified by )
As the title says.
Install: http://sage.math.washington.edu/home/malb/spkgs/libm4ri-20120613.spkg
Apply: trac_12840_m4ri_new_version.patch, trac_12840-rebased-on-13109.patch
Apply to SAGE_ROOT
: trac_12840_m4ri_png.patch
Attachments (3)
Change History (66)
comment:1 Changed 11 years ago by
Description: | modified (diff) |
---|
comment:2 Changed 11 years ago by
Description: | modified (diff) |
---|
comment:3 Changed 11 years ago by
Description: | modified (diff) |
---|
comment:4 Changed 11 years ago by
Dependencies: | → #12841 |
---|
comment:5 Changed 11 years ago by
Status: | new → needs_review |
---|
Changed 11 years ago by
Attachment: | trac_12840_m4ri_new_version.patch added |
---|
comment:7 Changed 11 years ago by
comment:8 Changed 11 years ago by
After applying the patch and the spkg and rebuilding the polybori spkg, sage -br failed with
gcc -pthread -shared -L/mnt/local/king/SAGE/stable/sage-5.0.beta13/local/lib build/temp.linux-x86_64-2.7/sage/matrix/matrix_integer_dense.o -L/mnt/local/king/SAGE/stable/sage-5.0.beta13/local/lib -L/mnt/local/king/SAGE/stable/sage-5.0.beta13/local/lib -lcsage -liml -lpari -lm -lgmp -lcblas -latlas -lstdc++ -lntl -lpython2.7 -o build/lib.linux-x86_64-2.7/sage/matrix/matrix_integer_dense.so error: command 'gcc' failed with exit status 1 Error installing modified sage library code.
No error message, I am afraid.
comment:9 Changed 11 years ago by
PS: That was with sage-5.0.beta13 plus a lot of patches, but none of them changing matrices.
comment:10 Changed 11 years ago by
And repeating sage -br, I finally got a proper error:
In file included from /mnt/local/king/SAGE/stable/sage-5.0.beta13/local/include/m4rie/m4rie.h:56, from sage/matrix/matrix_mod2e_dense.cpp:240: /mnt/local/king/SAGE/stable/sage-5.0.beta13/local/include/m4rie/permutation.h:30:30: error: m4ri/permutation.h: Datei oder Verzeichnis nicht gefunden error: command 'gcc' failed with exit status 1 Error installing modified sage library code.
comment:12 Changed 11 years ago by
PS: The same error occurs when trying to rebuild libm4rie spkg: M4RIE expects permutation.h
comment:15 Changed 11 years ago by
comment:16 follow-up: 17 Changed 11 years ago by
comment:17 Changed 11 years ago by
comment:18 follow-up: 19 Changed 11 years ago by
First of all: make ptestlong works with the two new spkgs (even though I have a few other patches applied...).
Here is a small timing.
Vanilla Sage-5.0.rc0:
sage: K = GF(2) sage: %time A = random_matrix(K,20000,20000) CPU times: user 0.29 s, sys: 0.02 s, total: 0.31 s Wall time: 0.31 s sage: %time B = random_matrix(K,20000,20000) CPU times: user 0.23 s, sys: 0.03 s, total: 0.26 s Wall time: 0.27 s sage: %time (A*B).rank() CPU times: user 11.70 s, sys: 0.05 s, total: 11.75 s Wall time: 11.81 s 19937
Sage-5.0.beta13 with the new spkgs (and some other patches that shouldn't affect the timing for matrix multiplication):
sage: K = GF(2) sage: %time A = random_matrix(K,20000,20000) CPU times: user 0.30 s, sys: 0.03 s, total: 0.33 s Wall time: 0.33 s sage: %time B = random_matrix(K,20000,20000) CPU times: user 0.32 s, sys: 0.01 s, total: 0.33 s Wall time: 0.33 s sage: %time (A*B).rank() CPU times: user 11.70 s, sys: 0.10 s, total: 11.80 s Wall time: 11.83 s 19937
Well, at least there is no slow-down...
Do you claim that the new version of M4RI is faster? You do claim a speed-up for M4RIE, on #12841.
comment:19 Changed 11 years ago by
Replying to SimonKing:
Do you claim that the new version of M4RI is faster? You do claim a speed-up for M4RIE, on #12841.
No, M4RI shouldn't have changed (at least the stuff, only bugfixes & refactoring.
That said, there's a new function for inverting upper triangular matrices which is much faster than the generic method, but it's not exposed yet. I need to implement the same for lower triangular matrices, then we can speed up inversions in general.
Furthermore, M4RI ships its own PNG reader/writer now which is much faster than what we have in Sage and uses much less memory. Again, it's not used yet. This patch just gets the new version in.
comment:20 Changed 11 years ago by
Reviewers: | → Simon King |
---|---|
Status: | needs_review → positive_review |
Since make ptestlong passes and the hg repository in the package is fine and SPKG.txt is up to date, I think I can give a positive review. However, I only tested on one platform. So, if the release manager believes that this is not enough, then please speak up!
comment:21 Changed 11 years ago by
Dependencies: | #12841 → to be merged wth #12841 |
---|
comment:22 Changed 11 years ago by
Dependencies: | to be merged wth #12841 → to be merged with #12841 |
---|
comment:23 Changed 11 years ago by
Status: | positive_review → needs_work |
---|
This fails to install on the Skynet machine "eno" (Fedora 16 x86_64):
/bin/sh ./libtool --tag=CC --mode=link gcc -std=gnu99 -mmmx -msse -msse2 -msse3 -g -fPIC -Wall -pedantic -O2 -release 0.0.20111203 -no -undefined -o libm4ri.la -rpath /home/buildbot/build/sage/eno-1/eno_full/build/sage-5.1.beta1/local/lib brilliantrussian.lo misc.lo mzd.l o graycode.lo strassen.lo mzp.lo triangular.lo triangular_russian.lo ple.lo ple_russian.lo solve.lo echelonform.lo mmc.lo debug_dump.lo io .lo -lm -lpng libtool: link: gcc -shared -fPIC -DPIC .libs/brilliantrussian.o .libs/misc.o .libs/mzd.o .libs/graycode.o .libs/strassen.o .libs/mzp.o . libs/triangular.o .libs/triangular_russian.o .libs/ple.o .libs/ple_russian.o .libs/solve.o .libs/echelonform.o .libs/mmc.o .libs/debug_dum p.o .libs/io.o -lm -lpng -mmmx -msse -msse2 -msse3 -O2 -Wl,-soname -Wl,libm4ri-0.0.20111203.so -o .libs/libm4ri-0.0.20111203.so /usr/bin/ld: cannot find -lpng collect2: ld returned 1 exit status make[1]: *** [libm4ri.la] Error 1 make[1]: Leaving directory `/home/buildbot/build/sage/eno-1/eno_full/build/sage-5.1.beta1/spkg/build/libm4ri-20120415/src' make: *** [all] Error 2 Error building libm4ri
$ ls -l local/lib/libpng* 36k -rw-r--r-- 1 buildbot sage 828k May 8 06:16 local/lib/libpng12.a 4.1k -rwxr-xr-x 1 buildbot sage 1.1k May 8 06:16 local/lib/libpng12.la* 0 lrwxrwxrwx 1 buildbot sage 18 May 8 06:16 local/lib/libpng12.so -> libpng12.so.0.35.0* 0 lrwxrwxrwx 1 buildbot sage 18 May 8 06:16 local/lib/libpng12.so.0 -> libpng12.so.0.35.0* 467k -rwxr-xr-x 1 buildbot sage 460k May 8 06:16 local/lib/libpng12.so.0.35.0* $ ls -l /lib*/libpng* /usr/lib*/libpng* ls: cannot access /lib*/libpng*: No such file or directory 0 lrwxrwxrwx 1 root root 16 May 2 11:53 /usr/lib64/libpng.so.3 -> libpng.so.3.49.0* 173k -rwxr-xr-x 1 root root 172k Apr 7 14:33 /usr/lib64/libpng.so.3.49.0* 0 lrwxrwxrwx 1 root root 18 May 2 11:53 /usr/lib64/libpng12.so.0 -> libpng12.so.0.49.0* 164k -rwxr-xr-x 1 root root 162k Apr 7 14:33 /usr/lib64/libpng12.so.0.49.0*
Note that there is no file /usr/lib64/libpng.so
, so obviously there cannot be proper PNG support on this machine. I consider this to be an upstream (not Sage) error.
$ grep -i png spkg/logs/libm4ri-20120415.log checking whether to build with PNG support... yes checking for PNG... no checking for PNG... yes /bin/sh ./libtool --tag=CC --mode=link gcc -std=gnu99 -mmmx -msse -msse2 -msse3 -g -fPIC -Wall -pedantic -O2 -release 0.0.20111203 -no-undefined -o libm4ri.la -rpath /home/buildbot/build/sage/eno-1/eno_full/build/sage-5.1.beta1/local/lib brilliantrussian.lo misc.lo mzd.lo graycode.lo strassen.lo mzp.lo triangular.lo triangular_russian.lo ple.lo ple_russian.lo solve.lo echelonform.lo mmc.lo debug_dump.lo io.lo -lm -lpng libtool: link: gcc -shared -fPIC -DPIC .libs/brilliantrussian.o .libs/misc.o .libs/mzd.o .libs/graycode.o .libs/strassen.o .libs/mzp.o .libs/triangular.o .libs/triangular_russian.o .libs/ple.o .libs/ple_russian.o .libs/solve.o .libs/echelonform.o .libs/mmc.o .libs/debug_dump.o .libs/io.o -lm -lpng -mmmx -msse -msse2 -msse3 -O2 -Wl,-soname -Wl,libm4ri-0.0.20111203.so -o .libs/libm4ri-0.0.20111203.so /usr/bin/ld: cannot find -lpng
comment:24 follow-up: 26 Changed 11 years ago by
Why are you using PKG_CHECK_MODULES()
to check for PNG support in configure.ac
? Why not AC_CHECK_LIB()
or AC_SEARCH_LIBS()
?
comment:25 Changed 11 years ago by
Exactly the same problem with -lpng
happens on Skynet "cleo" (RHEL 5.3 ia64)
comment:26 Changed 11 years ago by
Replying to jdemeyer:
Why are you using
PKG_CHECK_MODULES()
to check for PNG support inconfigure.ac
? Why notAC_CHECK_LIB()
orAC_SEARCH_LIBS()
?
We are doing both. We try pkg-config
first and if that fails we fall back to AC_CHECK_LIB
. In your log above, PKG_CHECK_MODULES()
fails to find libpng and AC_CHECK_LIB()
finds it.
comment:27 Changed 11 years ago by
Note that stand-alone M4RI builds fine on eno.
checking pkg-config is at least version 0.9.0... yes checking for PNG... no checking for PNG... no checking for PNG... no checking for PNG... no checking for png_create_write_struct in -lpng... no
So I don't know why it fails when built inside Sage. In any case, Sage ships libpng so we should adapt deps
to make M4RI dependent on libpng. I'll provide a patch to that effect.
Changed 11 years ago by
Attachment: | trac_12840_m4ri_png.patch added |
---|
comment:28 follow-up: 37 Changed 11 years ago by
Replying to malb:
In your log above,
PKG_CHECK_MODULES()
fails to find libpng andAC_CHECK_LIB()
finds it.
Actually, it's not quite like that. PKG_CHECK_MODULES()
found libpng12
but you still try to link with -lpng
instead of -lpng12
(the first failure comes from the libpng14
check). AC_CHECK_LIB()
isn't called (but it would fail).
comment:29 Changed 11 years ago by
Description: | modified (diff) |
---|---|
Status: | needs_work → needs_review |
comment:30 Changed 11 years ago by
I recommend against using PKG_CHECK_MODULES()
and would use only AC_SEARCH_LIBS()
.
comment:31 follow-up: 32 Changed 11 years ago by
Status: | needs_review → needs_work |
---|
The patch you attached is probably good, but that's just part of the problem.
comment:32 follow-up: 33 Changed 11 years ago by
Replying to jdemeyer:
The patch you attached is probably good, but that's just part of the problem.
Can you elaborate on that? I don't understand.
comment:33 follow-up: 35 Changed 11 years ago by
Replying to malb:
Can you elaborate on that? I don't understand.
Well, I tried to explain in http://trac.sagemath.org/sage_trac/ticket/12840#comment:28. The fact that
PKG_CHECK_MODULES([PNG], [libpng12])
succeeds, does not imply that the library file libpng.so
exists. I don't see what you're trying to accomplish with those PKG_CHECK_MODULES()
checks.
comment:35 Changed 11 years ago by
Replying to jdemeyer:
Replying to malb:
Can you elaborate on that? I don't understand.
PKG_CHECK_MODULES([PNG], [libpng12])succeeds, does not imply that the library file
libpng.so
exists.
Ah, right. It would need -lpng12
. Yep, that's a bug. I'll fix that next week-ish.
I don't see what you're trying to accomplish with those
PKG_CHECK_MODULES()
checks.
I am trying to use pkg-config for what it's intended. Making linking easier.
comment:36 follow-up: 38 Changed 11 years ago by
Maybe you're using it for what it's intended, but not how it's intended.
comment:37 Changed 11 years ago by
Replying to jdemeyer:
Replying to malb:
In your log above,
PKG_CHECK_MODULES()
fails to find libpng andAC_CHECK_LIB()
finds it.Actually, it's not quite like that.
PKG_CHECK_MODULES()
foundlibpng12
but you still try to link with-lpng
instead of-lpng12
(the first failure comes from thelibpng14
check).AC_CHECK_LIB()
isn't called (but it would fail).
For some reason I missed this comment earlier, sorry. That makes sense & is an upstream bug which I intend to fix.
comment:38 follow-up: 39 Changed 11 years ago by
Replying to jdemeyer:
Maybe you're using it for what it's intended, but not how it's intended.
This is a remarkably unhelpful comment.
comment:39 Changed 11 years ago by
Replying to malb:
This is a remarkably unhelpful comment.
I know, but I have never done anything with pkgconfig, so I can't help more.
comment:40 Changed 11 years ago by
Description: | modified (diff) |
---|
I've updated the SPKG with a new upstream release which should fix the png issue.
http://sage.math.washington.edu/home/malb/spkgs/libm4ri-20120522.spkg
comment:41 Changed 11 years ago by
Status: | needs_work → needs_review |
---|
comment:42 follow-up: 43 Changed 11 years ago by
Could someone please review this. I am looking into updating LinBox and updating M4RIE is a requirement for that (long story involving Givaro).
comment:43 Changed 11 years ago by
Replying to malb:
Could someone please review this. I am looking into updating LinBox and updating M4RIE is a requirement for that (long story involving Givaro).
I could test the code on one or two linuxes. But I couldn't test whether the png problems on eno (as observed by Jeroen) are fixed. Jeroen, could you finish the review?
comment:44 Changed 11 years ago by
Description: | modified (diff) |
---|
comment:45 Changed 11 years ago by
Description: | modified (diff) |
---|
comment:46 follow-up: 47 Changed 11 years ago by
The previous version installed fine on eno within Sage. It seems the libpng
check is totally gone now.
comment:47 Changed 11 years ago by
Replying to jdemeyer:
The previous version installed fine on eno within Sage.
It seems the
libpng
check is totally gone now.
That's strange, it doesn't check?
I had to update SPKG because I didn't add "-I$SAGE_LOCAL/include" before but we need it now because of the PNG dependency. The newest SPKG installs fine on geom.math which doesn't have a system libpng-dev.
comment:48 Changed 10 years ago by
I've tried this on Fedora 17 (which doesn't have libpng12 either, its up to libpng15 now) and configure works as intended:
checking whether to build with PNG support... yes checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for PNG... yes
Jeroen, you might have overlooked this part in the log since it comes after the lengthy "checking for cache sizes...".
I'm building on eno right now and if it works I'll set this ticket to positive review.
comment:49 follow-up: 50 Changed 10 years ago by
So Fedora has libpng.so then? I am asking because I am not checking for png15 explicitly.
if test "x${want_png}" = "xyes" ; then PKG_CHECK_MODULES([PNG], [libpng], [have_libpng="yes"; LIBPNG_LIBADD=`pkg-config --libs libpng`], [have_libpng="no"]) if ! test "x${have_libpng}" = "xyes" ; then AC_CHECK_LIB([png], [png_create_write_struct], [have_libpng="yes"; LIBPNG_LIBADD="-lpng"], [AC_CHECK_LIB([png14], [png_create_write_struct], [have_libpng="yes"; LIBPNG_LIBADD="-lpng14"], [AC_CHECK_LIB([png12], [png_create_write_struct], [have_libpng="yes"; LIBPNG_LIBADD="-lpng12"], [AC_CHECK_LIB([png10], [png_create_write_struct], [have_libpng="yes"; LIBPNG_LIBADD="-lpng10"], [have_libpng="no"]) ]) ]) ]) fi if test "x${have_libpng}" = "xno" ; then AC_MSG_WARN([Can not find a usuable PNG library. Make sure that CPPFLAGS and LDFLAGS are correctly set.]) fi fi
comment:50 Changed 10 years ago by
Replying to malb:
So Fedora has libpng.so then? I am asking because I am not checking for png15 explicitly.
Argh, I am stupid. It would find libpng12 as included in Sage. Sorry bout the noise.
comment:51 Changed 10 years ago by
Reviewers: | Simon King → Simon King, Volker Braun |
---|---|
Status: | needs_review → positive_review |
Yes the PKG_CHECK_MODULES
test wins in Sage, which is the way it should be. pkg-config is the way to go imho. Though if libpng were to use standard libtool versioning we would live in a better world, too ;-)
In any case, built and tested fine on eno. Positive review.
comment:52 Changed 10 years ago by
Milestone: | sage-5.1 → sage-5.2 |
---|
Changed 10 years ago by
Attachment: | trac_12840-rebased-on-13109.patch added |
---|
comment:53 Changed 10 years ago by
Dependencies: | to be merged with #12841 → to be merged with #12841, #13109 |
---|---|
Description: | modified (diff) |
Status: | positive_review → needs_work |
The deprecations here should use the format from #13109.
comment:54 Changed 10 years ago by
Status: | needs_work → needs_review |
---|
comment:56 Changed 10 years ago by
Authors: | Martin Albrecht → Martin Albrecht, John Palmieri |
---|---|
Status: | needs_review → positive_review |
Hmm deprecations (and any other warnings) don't work in cdef methods. I guess we can't doctest them, for now.
comment:57 Changed 10 years ago by
Milestone: | sage-5.2 → sage-pending |
---|
comment:58 Changed 10 years ago by
Milestone: | sage-pending → sage-5.3 |
---|
comment:59 Changed 10 years ago by
Status: | positive_review → needs_work |
---|
This needs to be rebased to sage-5.2.rc1.
comment:60 Changed 10 years ago by
Can you clarify the rebasing issues? I have no problems applying the patches from here and from #12841 on top of sage-5.2.rc1.
comment:61 Changed 10 years ago by
I'm with John here, I've been carrying the mari/e/givaro/linbox patches in my queue for a while now and there was no breakage in sage-5.2.rc1 as far as I can tell.
comment:62 Changed 10 years ago by
Status: | needs_work → positive_review |
---|
My mistake, I only applied the second Sage library patch and that didn't work.
comment:63 Changed 10 years ago by
Merged in: | → sage-5.3.beta0 |
---|---|
Resolution: | → fixed |
Status: | positive_review → closed |
Updated the patch because I forgot to raise a
ZeroDivisionError
in__invert__
. Fixed now.