Update LinBox to most recent upstream release

[pcpa]

The version of LinBox? used in Sage is 4 years old, we should update.

1. Observe: Dependencies
2. Install: ​http://sage.math.washington.edu/home/malb/spkgs/fflas_ffpack-1.6.0.spkg
3. Install: ​http://boxen.math.washington.edu/home/jdemeyer/spkg/linbox-1.3.2.spkg
4. Apply: matrix_modn_dense_no_linbox.patch​
5. Apply: sage-linbox.2.patch​
6. Apply: trac_12883_linbox_deps.patch​ and trac_12883-spkg-install.patch​ to SAGE_ROOT repository
7. Apply: trac_12883_hg_ignore_fflas_config.patch​ to the scripts (SAGE_LOCAL/bin) repository.

SPKG Repositories:

1. ​https://bitbucket.org/malb/fflas-ffpack-spkg
2. ​https://bitbucket.org/malb/linbox-spkg

Note for release manager: This ticket must be merged simultaneously with #13164, they depend on each other.

[leif]

What's the title supposed to tell us (if you get build errors with that version)?

For GCC 4.7.x / C++11 issues see #12762; for (problems with) updating LinBox see #11718. (I think I've read about complications when trying to upgrade to 1.2.2 elsewhere as well.) See #12865 for yet another issue, which is still present in 1.2.2.

[pcpa]

Sorry for lack of a better description. I am trying to address and submit reports to involved upstream parties about some patches and version issues of some components, while checking how far I can go building sagemath in Fedora, and using Fedora packages.

For example, the first error:

error: 'linbox_modn_dense_echelonize' was not declared in this scope

in /usr/include/linbox/linbox-sage.h I see only

EXTERN unsigned long int linbox_modn_dense_echelonize_double (double modulus, double*matrix, size_t nrows, size_t ncols);

EXTERN unsigned long int linbox_modn_dense_echelonize_float (float modulus, float * matrix,size_t nrows, size_t ncols);

Same happens for linbox_modn_dense_delete_array that now have prototypes for linbox_modn_dense_delete_array_double and linbox_modn_dense_delete_array_float.

Also linbox_integer_dense_matrix_matrix_multiply changed prototype from

/* ans must be a pre-allocated and pre-initialized array of GMP ints. */
EXTERN int linbox_integer_dense_matrix_matrix_multiply(mpz_t** ans, mpz_t **A, mpz_t **B,
			      size_t A_nr, size_t A_nc, size_t B_nr, size_t B_nc);

in the sagemath linbox-1.1.6 spkg, to

/* ans must be a pre-allocated and pre-initialized array of GMP ints. */
int linbox_integer_dense_matrix_matrix_multiply (mpz_t** ans, mpz_t **A, mpz_t **B, size_t m, size_t n, size_t k);

in linbox-1.2.2.

I am trying to have upstream aware of these issues, and if not possible to get components working, I can try to get an extra linbox 1.1.6 package in Fedora.

[pcpa]

Upstream linbox 1.2.2 should need this patch

--- linbox-1.2.2/interfaces/sage/linbox-sage.h.orig     2012-04-26 16:25:37.500132509 -0300
+++ linbox-1.2.2/interfaces/sage/linbox-sage.h  2012-04-26 16:26:00.705133519 -0300
@@ -48,7 +48,9 @@
 EXTERN void linbox_modn_dense_delete_array_double (double *f);
 EXTERN void linbox_modn_dense_delete_array_float (float *f);
 
-
+template <class Element>
+unsigned long int linbox_modn_dense_echelonize (Element modulus, Element* matrix,
+                                               size_t nrows, size_t ncols);
 EXTERN unsigned long int linbox_modn_dense_echelonize_double (double modulus, double*matrix, size_t nrows, size_t ncols);
 EXTERN unsigned long int linbox_modn_dense_echelonize_float (float modulus, float * matrix,size_t nrows, size_t ncols);
 

But it is not yet enough because the sage matrix interface was changed from "jagged pointers" to a single sequential vector.

I believe I could make a patch for sage/libs/linbox/linbox.{pxd,pyx} but I do not feel confident about other places where sage interfaces to linbox matrices.

[pcpa]

Experimental linbox 1.2.2 patch

[pcpa]

The attached patches allow continuing experimenting with a new sagemath rpm. The patches are only tested to know they make sagemath and linbox agree on api/abi, but no testing done. There were missing prototypes/templates in linbox-sage.h as well as wrong ones. I am not fully confident that some place may do wrong dereferencing of matrix fields due to upstream change from a vector of rows where every row is an allocated pointer to a single memory chunk. Another part I am not confident is that I did the correct logic in the change from A_nr, A_nc, B_nr, B_nc to test in sage code if A_nr != B_nc: ..., and convert the values of m, n, k as A_nr, A_nc, B_nr.

This is not a proposal of a final patch, but a extra call for more feedback. As this was also the first time I actually did look at linbox code, so, just patched the interfaces to make both sides agree.

[pcpa]

fflas-ffpack-64bit.patch

[pcpa]

linbox-sagemath.patch

[pcpa]

Is it reasonably to define mod_int to C long?

If not, safest approach should be to use uint32_t I believe, otherwise, should require coding 64 bit templates in linbox 1.2.2.

The new attached fflas-ffpack-64bit.patch is required to generate proper 64 bit code, and the updated linbox-sagemath.patch should be very close to the final format, but note that in 64 bit arch, linbox 1.2.2 with the attached patches needs also to have -D__LINBOX_HAVE_INT64=1 in CFLAGS and CXXFLAGS.

I should update the sage-4.8-linbox.patch at some point, as the initial patch I attached was only enough to get to another build failure in Fedora. But the next update should be for sage-5.0.

[malb]

I cut new SPKGs for LinBox and FFLAS/FFPACK:

​http://sage.math.washington.edu/home/malb/spkgs/fflas_ffpack-1.5.0.spkg

​http://sage.math.washington.edu/home/malb/spkgs/linbox-1.3.0.spkg

I am tracking SPKG repositories here:

​https://bitbucket.org/malb/fflas-ffpack-spkg

​https://bitbucket.org/malb/linbox-spkg

I will go through your patches now.

[malb]

All doctests pass on geom.math and my machine with 5.0.1.

[malb]

Needs review.

[vbraun]

matrix_modn_dense_no_linbox.patch is full of unnecessary whitespace changes that only serve to break any other patch on trac.

[malb]

Replying to vbraun:

matrix_modn_dense_no_linbox.patch is full of unnecessary whitespace changes that only serve to break any other patch on trac.

Fixed

[malb]

fixed indentation error introduced in last update

[vbraun]

What for/where is matrix_modn_dense used now? I see its commented out in matrix_space.py.

Minor nitpick: missing space in coefficients ofminpoly as a Python list.

[malb]

Replying to vbraun:

What for/where is matrix_modn_dense used now? I see its commented out in matrix_space.py.

Currently, none. But William is resurrecting it at #4968. I assume matrix_modn_dense_no_linbox.patch​ and #4968 will conflict. But I suggest to deal with this when the problem arises and when we know in which order they'll be merged.

Minor nitpick: missing space in coefficients ofminpoly as a Python list.

I'll fix that.

[malb]

Fixed.

[fbissey]

I believe it is not necessary anymore to add -DDISABLE_COMMENTATOR to CPPFLAGS. IT is added automatically in the right places when you configure with the sage interface.

[malb]

Replying to fbissey:

I believe it is not necessary anymore to add -DDISABLE_COMMENTATOR to CPPFLAGS. IT is added automatically in the right places when you configure with the sage interface.

I am inclined to postpone this change to another ticket, unless you are pretty certain it isn't needed. The reason I am suggesting this is, that if we get SIGSEGVs then on some exotic platform we are not sure whether it's the 1.3.2 update or that we dropped the -DDISABLE_COMMENTATOR. Of course, we could just try to re-add it and see what happens, but given how slow and bureaucratic Sage development is these days, this could easily take a week to accomplish, which would be annoying. What do you think?

[fbissey]

Keeping it is "mostly harmless". I just noticed when I created the ebuild for sage-on-gentoo that even so I hadn't added it to the CPPFLAGS it was used in the compilation. Adding it to the CPPFLAGS made it appear twice in those places.

I would have to test with the whole set of new packages and patches to be absolutely certain of course. That could take a few days. And of course that would be on a non-exotic platform first. Make me think I need to put a 5.1beta on power7 to cover my exotic corner.

[vbraun]

Looks good to me!

[jdemeyer]

In SPKG.txt of fflas_ffpack-1.6.0.spkg, replace

=== fflas-ffpack-1.6.0 (Martin Albrecht, 7 June 2012) ===

by

=== fflas_ffpack-1.6.0 (Martin Albrecht, 7 June 2012) ===

[leif]

Just curious -- does this fix #12865? (Won't try myself.)

[vbraun]

Yes, it does fix #12865.

[malb]

SPKG.txt issue is fixed.

[jdemeyer]

There is a weird file .swp in the linbox spkg.

[jpflori]

I've reuploaded the package without that mysterious file (which is not in spkg repo as I've checked) at ​http://perso.telecom-paristech.fr/~flori/sage/linbox-1.3.2.spkg

No other changes to the spkg. So someone should put this back to positive review (after checking I did not break everything).

Edit: Swapped 2 and 3...

[vbraun]

Initial patch

[vbraun]

The new fflas spkg installs a new script in SAGE_LOCAL/bin, which needs to be added to .hgignore. I've included a trivial patch to that extent.

Rest looks good.

[vbraun]

Rebased patch

[vbraun]

Rebased patch for sage-5.3.beta1. Just a trivial rebase in module_list.py.

[jhpalmieri]

Should the root repo patch also touch spkg/install, adding a line for fflas, like this?

spkg/install

@@ -296 +296 @@
 ECM=`newest_version ecm`
 ELLIPTIC_CURVES=`newest_version elliptic_curves`
 EXTCODE=`newest_version extcode`
+FFLASFFPACK=`newest_version fflas_ffpack`
 FLINT=`newest_version flint`
 FLINTQS=`newest_version flintqs`
 FPLLL=`newest_version libfplll`

Or is this not needed for some reason?

[vbraun]

Yes, I'm pretty sure thats required. Are you going to post a patch?

[jhpalmieri]

Sure, here's a patch. Do I need to switch it back to "needs review"?

[jdemeyer]

Why the change to the iml dependencies?

[jdemeyer]

The linbox spkg still contains the garbage I mentioned before:

? .swp

[vbraun]

Fixed. I guess vim poops into cwd when segfaulting (.swp was vim swap file), never heard of /tmp?!?

[vbraun]

PS: The iml testsuite does link against ATLAS, even though libiml doesn't depend on it.

[jhpalmieri]

For what it's worth, building from scratch with these patches works (all long tests passed) on sage.math and on my OS X Lion box.

[jdemeyer]

The LinBox? spkg needs to be rebased to linbox-1.1.6.p11 (see #13118).

[jhpalmieri]

Replying to jdemeyer:

The LinBox? spkg needs to be rebased to linbox-1.1.6.p11 (see #13118).

and also see #12762. By the way, the file SPKG.txt also has two duplicate entries in the change log for version 1.1.6.p8.

[vbraun]

I've added the changelog entries of linbox-1.1.6p9 - 11 to the SPKG.txt and removed the duplicate. None of those changes are needed any more since upstream has long fixed the issues that gcc 4.7 uncovered and Martin cleaned up the spkg-install.

I've also re-added the spkg-check as disabled-spkg-check and deleted the Makefile.am.patch. Right now the testsuite will not compile without commentator. Would be nice to have the testsuite but not on this ticket.

Likewise, it would be nice if the Sage developers could first search trac if there is already a new spkg before opening a ticket.

Since really only the description changed I'll set this ticket back to positive review.

[vbraun]

For the record, I've noted the testsuite issues on the linbox mailinglist at ​https://groups.google.com/d/topic/linbox-use/z0AlEH8Iyug/discussion

[vbraun]

Testsuite issues are fixed in svn version 4370.

[jdemeyer]

Is it intentional that the --with-gmp, --with-givaro and --with-ntl and --with-blas configure options were removed?

Secondly, by "rebasing" this way, you've lost the .hgtags (and probably the whole hg history) for some earlier versions. Not sure whether that's important, but it's something to be pointed out.

[jdemeyer]

Also, the .p11 spkg has error messages to stderr, please don't revert this.

[jdemeyer]

I think this really needs to be rebased properly. You've reverted some good changes from the .p9--p11 versions. Also, try to keep the Mercurial history.

[jdemeyer]

Also, there isn't any documentation of the fact and the reason of renaming spkg-check to disabled-spkg-check.

[vbraun]

I've wasted my coffee break in preserving the mercurial history of the bikeshedding that went into linbox-1.1.6p(N+1) despite the fact that it is all being ripped out. Hopefully future generations of archaeologists will appreciate the effort.

I've also mentioned in the SPKG.txt that spgk-check is disabled in case anybody can't figure it out from the filename.

Now that spkg-install sets --with-default we don't need --with-foo for every foo.

Now errors go to stderr again.

[jdemeyer]

This change is still a regression:

-if [ "$SAGE64" = yes ]; then
-    echo "Building a 64-bit version of LinBox."
-    if [ -z "$CFLAG64" ]; then
-        CFLAG64=-m64
-    fi
-    CFLAGS="$CFLAGS $CFLAG64"
-    CXXFLAGS="$CXXFLAGS $CFLAG64"
-    CPPFLAGS="$CPPFLAGS $CFLAG64"
-    LDFLAGS="$LDFLAGS $CFLAG64"
+if [ "$SAGE64" = "yes" ]; then
+   echo "64 bit build"
+   CFLAGS="-m64 $CFLAGS "; export CFLAGS
+   CXXFLAGS="-m64 $CXXFLAGS "; export CXXFLAGS
+   CPPFLAGS="-m64 $CPPFLAGS "; export CPPFLAGS
+   LDFLAGS="-m64 $LDFLAGS"; export LDFLAGS
 fi

[jdemeyer]

Also this:

-if [ -z "$SAGE_LOCAL" ]; then
-    echo >&2 "Error: SAGE_LOCAL undefined - exiting..."
-    echo >&2 "Maybe run 'sage -sh'?"
-    exit 1
+if [ "$SAGE_LOCAL" = "" ]; then
+   echo "SAGE_LOCAL undefined ... exiting";
+   echo "Maybe run 'sage -sh'?"
+   exit 1
 fi

[vbraun]

FYI only

[vbraun]

I did notice the removal of the undocumented and nonstandard CFLAG64 environment variable and I consider that a definite plus of the new spkg-install.

The new spkg re-adds the stderr redirection in the SAGE_LOCAL check.

[jpflori]

It would be nice to add an hg tag for the new release.

[vbraun]

I've added the tag. Of course this'll never be done consistently as long as sage -pkg doesn't check that the spkg name matches the tag.

[jdemeyer]

Replying to jpflori:

It would be nice to add an hg tag for the new release.

This is also done automatically by the merger script when the spkg is merged into Sage.

[jpflori]

Good to know. Was it mentionned anywhere in the developper guide or did I miss it or did I not read it again recently enough?

Additionnal question, what about committing changes or putting them in a queue or not committing anything and letting some script do it? Is there a prefered method to do it?

[jdemeyer]

Committing changes is always required.

Note: with the latest Mercurial, you can use

hg commit --amend

to add changes to the latest commit. So you can effectively emulate a queue with 1 patch like this.

[vbraun]

This is totally off-topic, but versioning systems are most useful if you commit often. Two logically independent changes should be two commits, even if they end up in the same ticket/spkg. Reviewer nitpicks should be another commit. Any other questions on how to use mercurial should go to sage-devel.

[jdemeyer]

Replying to vbraun:

I did notice the removal of the undocumented and nonstandard CFLAG64 environment variable and I consider that a definite plus of the new spkg-install.

Looks documented to me: ​http://sagemath.org/doc/installation/source.html#environment-variables

(I do agree that we should get rid of CFLAG64 and SAGE64 in place of using CC and CFLAGS properly, but that's not the point of this ticket)

[jdemeyer]

Something else I noted: why was the exit code for a failed patch changed from 1 to -1? Normally, programs should use exit codes in the range 0-127.

[vbraun]

Oh silly me, I only searched the developer guide for CFLAG64! I also like how the installation guide is dancing around the fact that CFLAG64 is currently useless since it is not used in all spgks, so if you were to use a compiler that doesn't understand its default -m64 value breakage will ensue. So I'll concede that it is documented, but it still doesn't sound like a good idea to me.

I've fixed the offending exit code to be +1.

[jhpalmieri]

Should the references to #9511 be changed to #13164 (in the dependencies and the ticket description?

[jdemeyer]

Replying to vbraun:

I also like how the installation guide is dancing around the fact that CFLAG64 is currently useless since it is not used in all spgks, so if you were to use a compiler that doesn't understand its default -m64 value breakage will ensue. So I'll concede that it is documented, but it still doesn't sound like a good idea to me.

This is all very true, but no good excuse for sloppy rebasing...

[jdemeyer]

Rebased from scratch, needs review.

[jdemeyer]

Diff for linbox 1.1.6.p11->1.3.2. For reference / review only.

[jdemeyer]

Could this cause the following sporadic doctest error:

sage -t  --long -force_lib devel/sage/sage/modular/modform/ambient.py
you are running out of primes. 1000 coprime primes
found**********************************************************************
File
"/release/buildbot/sage/sage-1/sage_upgrade_5.2/build/sage-5.5.beta0/devel/sage-main/sage/modular/modform/ambient.py",
line 815:
    sage: [f[0]%p for p in prime_range(100)] # long time (0s, depends on
above)
Expected:
    [0, 0, 0, 0, 1, 9, 2, 7, 0, 0, 0, 0, 1, 12, 9, 16, 37, 0, 21, 11,
70, 22, 0, 58, 76]
Got:
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1]
**********************************************************************
File
"/release/buildbot/sage/sage-1/sage_upgrade_5.2/build/sage-5.5.beta0/devel/sage-main/sage/modular/modform/ambient.py",
line 817:
    sage: [f[42]%p for p in prime_range(100)] # long time (0s, depends
on above)
Expected:
    [0, 0, 4, 0, 10, 4, 4, 8, 12, 1, 23, 13, 10, 27, 20, 13, 16, 59, 53,
41, 11, 13, 12, 6, 82]
Got:
    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0]
**********************************************************************

(the string "you are running out of primes" certainly comes from the new LinBox package).

[vbraun]

I've posted this question to the linbox-use mailinglist ​https://groups.google.com/d/topic/linbox-use/SgsXVYM7u7s/discussion

[jdemeyer]

FYI: I did 4000 doctest runs and this happened 9 times, so it is quite rare.

Because of the rarity, it doesn't have to be a sage-5.4 blocker, but of course I'd like to see it fixed.

[jdemeyer]

With previous LinBox versions, this file sometimes (equally rarely) timeouts in doctests. So maybe it's the same issue, but with a timeout before, and an error now.

[jdemeyer]

Unfortunately, no response from upstream concerning the "running out of primes" error.

[jdemeyer]

Either this or #13164 caused #14032.