Opened 7 years ago

Closed 7 years ago

#13137 closed enhancement (fixed)

Upgrade MPIR to 2.6.0

Reported by: jhpalmieri Owned by: tbd
Priority: major Milestone: sage-5.7
Component: packages: standard Keywords: mpir spkg
Cc: jpflori Merged in: sage-5.7.beta0
Authors: John Palmieri, Jean-Pierre Flori, Karl-Dieter Crisman Reviewers: Jeroen Demeyer, Leif Leonhardy, John Palmieri, R. Andrew Ohana, Karl-Dieter Crisman
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #13755 Stopgaps:

Description (last modified by jdemeyer)

As the summary says. Also, mpir-2.4.0.p5.spkg fails self-tests, moreover it's old. Upgrading to 2.6.0 fixes both issues.

This requires the fixed LinBox spkg from #13755.

Apply:

New spkg: http://boxen.math.washington.edu/home/leif/Sage/spkgs/mpir-2.6.0.p0.spkg

md5sum: ec21b55dce699d3c8ddf0492e1413107 mpir-2.6.0.p0.spkg

mpir-2.6.0.p0 (Jean-Pierre Flori, 12 January 2013)

  • Trac #13137: Update to MPIR 2.6.0.
  • Modify spkg-install to rename *,asm files to *.asm files.
  • Remove -Wl,-z,noexecstack fix which has been integrated upstream.
  • Remove old code about 32 bits Apple Darwin and use slightly modified upstream fix.

mpir-2.5.2.p0 (John Palmieri, 3 October 2012)

  • Trac #13137: Update to MPIR 2.5.2.
  • Remove the patch patches/yasm__tools__re2c__code.c.patch.
  • Do not use clang, which fails to compile MPIR, on OS X.

mpir-2.4.0.p7 (Karl-Dieter Crisman, Jean-Pierre Flori, 1 August 2012)

  • Trac #12115: let MPIR build on Cygwin.
  • Only build the shared version of the library on Cygwin.
  • Export ABI=32 on Cygwin.

Attachments (4)

trac_13137-stopgap.patch (960 bytes) - added by jhpalmieri 7 years ago.
root repo
trac_13137-mpir.patch (17.2 KB) - added by jhpalmieri 7 years ago.
patch for mpir spkg; for review only
trac_13137-mpir-2.5.2.patch (18.6 KB) - added by jhpalmieri 7 years ago.
patch for mpir 2.5.2 spkg
mpir-2.6.0.p0.diff (18.9 KB) - added by jpflori 7 years ago.
Spkg diff, for review only.

Download all attachments as: .zip

Change History (129)

Changed 7 years ago by jhpalmieri

root repo

comment:1 Changed 7 years ago by jhpalmieri

  • Authors set to John Palmieri
  • Description modified (diff)
  • Status changed from new to needs_review

comment:2 Changed 7 years ago by ohanar

This causes the libm4rie test suite to fail, as well as breaks the building of linbox (both have the same errors):

In file included from .../local/include/gmp++/gmp++.h:20:0,
                 from ../../linbox/integer.h:22,
                 from ../../linbox/randiter/abstract.h:30,
                 from ../../linbox/field/abstract.h:32,
                 from ../../linbox/field/archetype.h:42,
                 from ../../linbox/vector/vector-traits.h:35,
                 from ../../linbox/blackbox/factory.h:17,
                 from ../../linbox/matrix/sparse.h:54,
                 from ../../linbox/blackbox/sparse.h:41,
                 from linbox-sage.C:35:
.../local/include/gmpxx.h:1566:3: error: ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>::__gmp_expr(intmax_t)’ cannot be overloaded
.../local/include/gmpxx.h:1562:3: error: with ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>::__gmp_expr(long int)’
.../local/include/gmpxx.h:1567:3: error: ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>::__gmp_expr(uintmax_t)’ cannot be overloaded
.../local/include/gmpxx.h:1563:3: error: with ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>::__gmp_expr(long unsigned int)’
.../local/include/gmpxx.h:1635:16: error: ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>& __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::operator=(intmax_t)’ cannot be overloaded
.../local/include/gmpxx.h:1629:16: error: with ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>& __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::operator=(long int)’
.../local/include/gmpxx.h:1636:16: error: ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>& __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::operator=(uintmax_t)’ cannot be overloaded
.../local/include/gmpxx.h:1631:16: error: with ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>& __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::operator=(long unsigned int)’
.../local/include/gmpxx.h: In constructor ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>::__gmp_expr(intmax_t)’:
.../local/include/gmpxx.h:1566:49: error: ‘mpz_init_set_sx’ was not declared in this scope
.../local/include/gmpxx.h: In constructor ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>::__gmp_expr(uintmax_t)’:
.../local/include/gmpxx.h:1567:50: error: ‘mpz_init_set_ux’ was not declared in this scope
.../local/include/gmpxx.h: In member function ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>& __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::operator=(intmax_t)’:
.../local/include/gmpxx.h:1635:56: error: ‘mpz_set_sx’ was not declared in this scope
.../local/include/gmpxx.h: In member function ‘__gmp_expr<__mpz_struct [1], __mpz_struct [1]>& __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::operator=(uintmax_t)’:
.../local/include/gmpxx.h:1636:57: error: ‘mpz_set_ux’ was not declared in this scope
.../local/include/gmpxx.h: In member function ‘intmax_t __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::get_sx() const’:
.../local/include/gmpxx.h:1677:49: error: ‘mpz_get_sx’ was not declared in this scope
.../local/include/gmpxx.h: In member function ‘uintmax_t __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::get_ux() const’:
.../local/include/gmpxx.h:1678:50: error: ‘mpz_get_ux’ was not declared in this scope

There may be other issues, but I haven't encountered them yet.

comment:3 Changed 7 years ago by jdemeyer

There is an mpir-2.4.0.p6 at #12751, probably the new spkg should be based on that.

Also, I guess the patch patches/yasm__tools__re2c__code.c.patch can be removed completely (then also remove it from SPKG.txt)

comment:4 Changed 7 years ago by jpflori

  • Cc jpflori added

comment:5 Changed 7 years ago by jdemeyer

  • Reviewers set to Jeroen Demeyer
  • Status changed from needs_review to needs_work

See comment 3

#12751 has been merged, so this spkg must certainly be rebased.

Changed 7 years ago by jhpalmieri

patch for mpir spkg; for review only

comment:6 Changed 7 years ago by jhpalmieri

I've rebased the spkg. I haven't investigated the issues in the comment above. I'll take a look, but I may not know enough to be able to fix them.

Last edited 7 years ago by jhpalmieri (previous) (diff)

comment:7 Changed 7 years ago by jhpalmieri

On hawk (OpenSolaris): linbox fails to build, as mentioned above, and the new spkg from #12883 doesn't help. libm4rie passes its test suite, though.

On sage.math (linux, Sage's gcc): the ppl build fails to configure ("checking which instantiations are enabled... configure: error: invalid instantiation Polyhedron Error configuring the Parma Polyhedra Library."). linbox fails to build, and libm4rie fails its test suite.

On taurus (linux, gcc 4.7.0): same as with sage.math, and also pari fails its test suite.

I'm still building on OS X; I'll report the results when they're available.

I don't know enough to troubleshoot these. If someone else wants to work on this, please go ahead.

Last edited 7 years ago by jhpalmieri (previous) (diff)

comment:8 Changed 7 years ago by jhpalmieri

OS X is similar to sage.math: linbox and ppl fail to build, libm4rie fails its test suite.

comment:9 Changed 7 years ago by jhpalmieri

  • Description modified (diff)

I created an spkg out of the just-released version 2.5.2. It was more or less straightforward, except that it refuses to build using clang on OS X, so spkg-install required a little modification.

With this spkg, givaro no longer builds on sage.math or OS X, and I am seeing the same problems with ppl. Since givaro won't build, I'm not even getting to libm4rie or linbox.

Changed 7 years ago by jhpalmieri

patch for mpir 2.5.2 spkg

comment:10 Changed 7 years ago by jpflori

MPIR 2.6.0 is out and will be needed for FLINT 2.3. Should we cope with this upgrade here? Or in a subsequent ticket?

By the way, it would be nice to integrate the easy Cygwin fix of #12115 while upgrading MPIR.

comment:11 Changed 7 years ago by jhpalmieri

I have no objections to trying to upgrade to 2.6.0 instead of 2.5.2. Maybe it will fix some of the issues mentioned on this ticket. If someone else wants to put together an spkg, please go ahead.

comment:12 Changed 7 years ago by jpflori

I gave MPIR 2.6.0 a shot, and the problem with linbox remains. That's really strange, because somehow in gmpxx.h the member function get_sx is defined to invoke the C mpz_get_sx function, conditionally on the fact that stdint.h is included, and the mpz_get_sx is prototyped in mpir.h conditionally on the same fact. But somehow, the mpz_get_sx is not found. I'm asking that on mpir-devel as well because I have no clue.

Anyway, these functions are not necessary, so badly removing the definition of the member functions in gmpxx.h let you compile linbox and go on with other bits of Sage, as long as a proper solution is not found.

comment:13 Changed 7 years ago by jpflori

Maybe the problem is that mpir.h gets first included withtout stdint.h inlcuded so that when gmpxx.h gets included mpir.h is not included again and the C functions get_sx and get_ux are not defined.

comment:14 Changed 7 years ago by jpflori

Ok that's it:

In file included from linbox-sage.C:28:0:
/home/jp/boulot/sage/sage-5.5.rc0/local/include/gmp.h:1228:17: note: #pragma message: gmp wo stdint
In file included from /home/jp/boulot/sage/sage-5.5.rc0/local/include/gmp++/gmp++.h:33:0,
                 from ../../linbox/integer.h:57,
                 from ../../linbox/randiter/abstract.h:31,
                 from ../../linbox/field/abstract.h:33,
                 from ../../linbox/field/archetype.h:49,
                 from ../../linbox/vector/vector-traits.h:57,
                 from ../../linbox/blackbox/factory.h:33,
                 from ../../linbox/matrix/sparse.h:72,
                 from ../../linbox/blackbox/sparse.h:57,
                 from linbox-sage.C:38:
/home/jp/boulot/sage/sage-5.5.rc0/local/include/gmpxx.h:1710:17: note: #pragma message: gmpxx w stdint

comment:15 Changed 7 years ago by jpflori

New spkg at http://boxen.math.washington.edu/home/jpflori/mpir-2.6.0.p0.spkg. It also includes the changes from #12115.

comment:16 Changed 7 years ago by jpflori

On my linux install (Debian/experimental/gcc 4.7.?) the new MPIR spkg builds and passes its testsuite. I've upped a fixed LinBox? spkg at #13755 which should get merged with this one. I'll now have a look at libm4rie where the problem should be similar. I'll also try to build on sage.math. I don't have access to skynet, so won't try on more exotic archs.

comment:17 Changed 7 years ago by jpflori

  • Dependencies set to #13755
  • Summary changed from upgrade MPIR to 2.5.1 to upgrade MPIR to 2.6.0

With this ticket and #13755 I have no problem with libm4rie test suite, nor with PPl or pari, on the same setup as before. I'm now launching a clean build on sage.math.

And by the way, I've not put the clang changes in the spkg yet, as I cannot test if they are still needed.

comment:18 Changed 7 years ago by jpflori

On sage.math I get

../../lib/libR.so: undefined reference to `rl_sort_completion_matches'

while building the R spkg.

comment:19 Changed 7 years ago by jpflori

Apart from that the rest seems fine on sage.math (with Sage'gcc spkg). MPIR passes its test suite as PARI and libm4rie.

comment:20 Changed 7 years ago by jpflori

No problem with PPl and its testsuite as well (by the way the spkg-check from ppl lacks a nice "eerything got well, bye" message).

I'm still confused by the R issue. The problematic symbol is from readline, not sure how MPIR has influence on it.

comment:21 Changed 7 years ago by jpflori

In fact I get the same R error with a stock sage 5.5.rc0 so this is unrelated as expected.

comment:22 Changed 7 years ago by jpflori

  • Authors changed from John Palmieri to John Palmieri, Jean-Pierre Flori
  • Description modified (diff)
  • Keywords spkg added
  • Status changed from needs_work to needs_info

The discussion about headers inclusion at mpir-devel can be read at: https://groups.google.com/d/topic/mpir-devel/m__z0PR_wBw/discussion

Anyway, even if MPIR puts in a solution at some point, I think LinBox? must be fixed anyway. So #13755 makes sense and I'll put it as a dependency here.

The discussion about sage.math.washington.edu failure (which is highly likely to be unrelated): https://groups.google.com/d/topic/sage-devel/Tmg6nA1eWGk/discussion

I guess what must be done now is to retry some builds on different archs/compilers combinations (to check the clang thing for example) because this spkg is highly sensitive to that, and then we could launch the patchbots (or launch them directly but I don't know how to do that, nor how to get their results), so I put this as need_info now.

We should also decide what to do with the "history" of the spkg. For now, I've based the spkg on the one from #12115, mentionned in SPKG.txt with an hg commit plus an hg tag done by myself (if #12115 gets merged before that one, I guess we'll do a "proper" rebasing), then added mentionned of 2.5.2.p0 by John in SPKG.txt but without hg commit or hg tag, and finally mention of 2.6.0.p0 in SPKG.txt with an hg commit but no hg tag, assuming it will be automagically by Jeroen scripts.

comment:23 Changed 7 years ago by jhpalmieri

On my OS X 10.8 box, I have the same problems with clang. Trying again with the clang changes from my earlier spkg applied to jpflori's spkg: it builds and passes self-tests. Now I'm proceeding with the rest of the Sage build. I'm also trying it on the skynet machine taurus (linux, GCC 4.7.0, referred to above). I'll post results when I know more.

comment:24 Changed 7 years ago by jpflori

Thanks for the report, I'll put the clang patch back in and report upstream.

comment:25 Changed 7 years ago by jhpalmieri

No problems on either OS X (with modifications for clang) or taurus: with SAGE_CHECK=yes, all packages built and passed their tests. I ran make ptestlong and all tests passed on both platforms.

I think that this is probably ready to go, once the clang changes are reinstated. I also think what you've done with the history is fine.

comment:26 Changed 7 years ago by jpflori

  • Description modified (diff)

Ok, I'm putting back the clang stuff.

About the ,asm files, I think it's just a typo introduced here: https://github.com/wbhart/mpir/commit/295967cd2b11f7b964283c1e1bf8dd154e388af4 Waiting for confirmation here: https://groups.google.com/d/topic/mpir-devel/0uhskYIbRII/discussion As I have little doubt about this I'll add a patch to move the files and update the quote_patch patch.

(I cannot put the ticket directly as needs_work, so I'm leaving it in its current state and will put it as needs_review when the new spkg is posted.)

comment:27 Changed 7 years ago by jpflori

As it seems diff/patch lack the ability of properly detecting filename changes, I propose to manually move the ,asm files to .asm files before patching. This takes something like ten lines rather than more 1264 with a patch file which just - the *,asm and + the *.asm ones.

With the mv solution, I intend to hardcode the filenames in spkg-install, anyway this should get upstream and this part can (and will have to) then be removed. I'll report the quoting part as well while I'm at it.

Does this seem ok to everyone?

comment:28 Changed 7 years ago by jpflori

  • Status changed from needs_info to needs_review

I've upped a new spkg. It includes the fix mentioned above and I updated the patches in a minimal way in order to minimize mercurial history. I've also added a commit and a tag for 2.5.2.p0 spkg.

I've asked about the quoting stuff on mpir-devel and it should get integrated upstream at some point.

comment:29 Changed 7 years ago by jpflori

  • Status changed from needs_review to needs_work
  • Work issues set to update the remove_32...

Oops, it seems we should update the remove_32_bit_assembly_code function in spkg-install because the asm files names changed. Maybe there are new files as well which need removing, not sure about that.

comment:30 Changed 7 years ago by jpflori

The files themselves does not seem to have been modified so the problem must still exist. I was not able to find relevant information on what kind of instructions cause this problem at #5315, so if this needs to be extended to new files (in addition to the renamed files).

But who still uses such old computers...

comment:31 Changed 7 years ago by jpflori

Ok it seems this is because of @GOT. See http://lists.apple.com/archives/scitech/2006/Dec/msg00004.html which is more informative.

But now MPIR ships with YASM so I guess all of this is not needed anymore? Could someone who has access to a OS X 10.4 running on 64 bits intel processor test the spkg without the call to remove_32... function?

comment:32 Changed 7 years ago by jpflori

Thinking about it, I definitely think the fix is not needed anymore, or at least nobody encountered it for a couple of years because we don't delete the renamed problematic files since a couple of years. (the rename commit is here: https://github.com/wbhart/mpir/commit/5dad96491ba4d63613539a3124a7b63721bf4892 )

comment:33 Changed 7 years ago by jpflori

Ok and according to http://trac.sagemath.org/sage_trac/ticket/10188#comment:19 the noexecstack thing is upstream now.

comment:34 Changed 7 years ago by jpflori

  • Status changed from needs_work to needs_review
  • Work issues update the remove_32... deleted

So goes an updated spkg removing all the (hopefully) obsolete fixes.

To get the old one replace spkg by bak at the end of the file url.

comment:35 Changed 7 years ago by jhpalmieri

A minor comment: in spkg-install, perhaps the following change would be a good idea:

  • spkg-install

    diff --git a/spkg-install b/spkg-install
    a b cd src/ 
    1919# detecting filename changes. See Trac #13137.
    2020# This fix can (and will have to) be removed once integrated upstream.
    2121echo "Renaming *,asm files to *.asm..."
    22 mv mpn/x86/p6/addmul_1,asm mpn/x86/p6/addmul_1.asm
    23 mv mpn/x86/p6/submul_1,asm mpn/x86/p6/submul_1.asm
     22mv mpn/x86/p6/addmul_1,asm mpn/x86/p6/addmul_1.asm && mv mpn/x86/p6/submul_1,asm mpn/x86/p6/submul_1.asm
    2423if [ $? -ne 0 ]; then
    2524    echo >&2 "Error: moving *,asm files failed."
    2625    exit 1

comment:36 Changed 7 years ago by jpflori

Updated spkg at same address.

comment:37 Changed 7 years ago by jhpalmieri

This also builds and passes tests on skynet machine iras, which often has problems not found on other platforms. (Actually, gsl fails its self-tests, but it does that with the old mpir and linbox spkgs, so no regression there. I couldn't find a ticket for this, though...)

So I think this is pretty close. What else should be checked?

comment:38 Changed 7 years ago by jpflori

Maybe explicitely run tests on Fedora where the fix I removed is supposed to not be needed anymore. Apart from that and the OSX 10.4 (and or 10.5) on 64 intel processors (not sure it will actually be tested by anyone so let's forget OS 10.4 unless someone raises its hand), I guess this is it if you agree with all the changes made to the spkg (I do agree with the one you did before).

comment:39 Changed 7 years ago by jpflori

By the way, I (potentially again) reported all the patches we still ship upstream and they should get integrated at some point.

comment:40 Changed 7 years ago by jpflori

Reintroduced fix for Mac OS 10.4 on intel hardware as it's still needed. Same address as before.

comment:41 follow-up: Changed 7 years ago by jpflori

  • Status changed from needs_review to needs_info

Waiting for feedback from Mac Os X user on sage-devel: https://groups.google.com/d/topic/sage-devel/R43tQAKI8gA/discussion

comment:42 in reply to: ↑ 41 Changed 7 years ago by leif

Replying to jpflori:

Waiting for feedback from Mac Os X user on sage-devel: https://groups.google.com/d/topic/sage-devel/R43tQAKI8gA/discussion

I'd suggest changing the relevant lines in configure[.in] to

    # 32bit apple darwin doesn't like our PIC format asm code
    case $host in
        i[34567]86-apple-darwin*|
        pentium*-apple-darwin*|
        prescott-apple-darwin*|
        core-apple-darwin*)
            path="x86/applenopic" ;;
        *-apple-darwin*) # assume Core2 or later
            path="x86/applenopic/core2 x86/applenopic" ;;
        *)                                                      ;;
    esac

(Cf. mpir-devel)

comment:43 Changed 7 years ago by jpflori

  • Status changed from needs_info to needs_review

New spkg with the solution from mpir-devel. (The discussion is at https://groups.google.com/d/topic/mpir-devel/hsxEc_69lBo/discussion )

Back to needs_review, I think everything is sorted out now.

comment:44 Changed 7 years ago by leif

Still a bit weird he now gets -march=nocona -mtune=nocona (instead of -march=nehalem ... as previously, with -m32 of course); haven't looked at his full logs yet...

[Perhaps this was just with Apple's GCC 4.0.1, which presumably doesn't know Core i7.]

comment:45 follow-up: Changed 7 years ago by jpflori

A priori, I did not modify the configure file except for the few lines involved by the fix, so this is strange.

comment:46 in reply to: ↑ 45 ; follow-up: Changed 7 years ago by leif

Replying to jpflori:

A priori, I did not modify the configure file except for the few lines involved by the fix, so this is strange.

All files in the p2 except configure are vanilla 2.6.0 again (especially config.guess)?

Another reason for the difference might be him setting ABI=32 manually inbetween (which should matter only in the plain MPIR builds, as spkg-install does this as well, always on MacOS < 10.6 I think).

comment:47 in reply to: ↑ 46 Changed 7 years ago by leif

Replying to leif:

Replying to jpflori:

A priori, I did not modify the configure file except for the few lines involved by the fix, so this is strange.

All files in the p2 except configure are vanilla 2.6.0 again (especially config.guess)?

Another reason for the difference might be him setting ABI=32 manually inbetween (which should matter only in the plain MPIR builds, as spkg-install does this as well, always on MacOS < 10.6 I think).

I probably just confused the output of config.guess and -march=...; in his recent logs we have:

configure run "manually", Apple GCC 4.0.1:

checking build system type... nehalem-apple-darwin9.8.0
...
checking ABI=32
checking compiler gcc -m32 -O2 -fomit-frame-pointer ... yes
checking compiler gcc -m32 -O2 -fomit-frame-pointer has sizeof(long)==4... yes
checking compiler gcc -m32 -O2 -fomit-frame-pointer  -mtune=corei7... no
checking compiler gcc -m32 -O2 -fomit-frame-pointer  -mtune=core2... no
checking compiler gcc -m32 -O2 -fomit-frame-pointer  -mtune=nocona... yes
checking compiler gcc -m32 -O2 -fomit-frame-pointer -mtune=nocona  -march=corei7... no
checking compiler gcc -m32 -O2 -fomit-frame-pointer -mtune=nocona  -march=core2... no
checking compiler gcc -m32 -O2 -fomit-frame-pointer -mtune=nocona  -march=nocona... yes
...
using ABI="32"
      CC="gcc -std=gnu99"
      CFLAGS="-m32 -O2 -fomit-frame-pointer -mtune=nocona -march=nocona"
      CPPFLAGS=""
      MPN_PATH=" x86/applenopic generic"
...

configure run "manually", Apple GCC 4.2.1:

checking build system type... nehalem-apple-darwin9.8.0
...
checking ABI=32
checking compiler gcc -m32 -O2 -fomit-frame-pointer ... yes
checking compiler gcc -m32 -O2 -fomit-frame-pointer has sizeof(long)==4... yes
checking compiler gcc -m32 -O2 -fomit-frame-pointer  -mtune=corei7... no
checking compiler gcc -m32 -O2 -fomit-frame-pointer  -mtune=core2... yes
checking compiler gcc -m32 -O2 -fomit-frame-pointer -mtune=core2  -march=corei7... no
checking compiler gcc -m32 -O2 -fomit-frame-pointer -mtune=core2  -march=core2... yes
...
using ABI="32"
      CC="gcc -std=gnu99"
      CFLAGS="-m32 -O2 -fomit-frame-pointer -mtune=core2 -march=core2"
      CPPFLAGS=""
      MPN_PATH=" x86/applenopic generic"
...

MPIR 2.6.0.p2 (with sage -i ..., Apple GCC 4.2.1, build 5577):

...
Settings chosen by MPIR when configuring with CFLAGS unset:
  CC:      gcc -std=gnu99
  CFLAGS:  -m32 -O2 -fomit-frame-pointer -mtune=core2 -march=core2
...
Finally using the following settings:
  CC=gcc
  CFLAGS=-m32 -O2 -fomit-frame-pointer -mtune=core2 -march=core2  -g 
...
checking build system type... nehalem-apple-darwin9.8.0
...
checking ABI=32
checking compiler gcc -m32 -O2 -fomit-frame-pointer -mtune=core2 -march=core2  -g  ... yes
...
using ABI="32"
      CC="gcc -std=gnu99"
      CFLAGS="-m32 -O2 -fomit-frame-pointer -mtune=core2 -march=core2  -g "
      CPPFLAGS=""
      CXX="g++"
      CXXFLAGS=""
      MPN_PATH=" x86/applenopic generic"
...

(Looks like our first "dry" configure run influences the second; we should probably delete some more cache files...)

And FWIW, the -std=gnu99 still(?) gets dropped in the first place at least, for whatever reason. (It was a long-lasting MPIR bug that gmp.h -- in contrast to mpir.h -- was lacking this in the definition of __GMP_CC.)

Also, IMHO CXXFLAGS shouldn't be empty. And it doesn't look like yasm's configure would pick up the settings from MPIR's...

comment:48 follow-up: Changed 7 years ago by jpflori

Maybe we can do this in a follow up ticket "Fix spkg-install of mpir spkg"? And merge this asap, as it looks functional everywhere (I know there is some fuzz for one patch as well because of the last fix included, but I got no time to fix it once again).

comment:49 Changed 7 years ago by jpflori

I reread your message above.

The p0 is the one targeted for inclusion. It should have pristine src directory, except some useless dir for us are deleted. The patches modify configure for this apple problem, and for some fortran things which were here before. I also move "by hand" the wrongly name *,asm files.

A priori, the p2 was based on p0, with editing in a sligthly more dirty way configure (i.e. I put somethign at the end of the swith case, that's the only difference as far as configure is concerned) and adding some asm files to the applenopic. Nonetheless I don't expect that to have any influence on config.guess.

comment:50 in reply to: ↑ 48 ; follow-up: Changed 7 years ago by leif

Replying to jpflori:

Maybe we can do this in a follow up ticket "Fix spkg-install of mpir spkg"?

Sure, but then we should probably add some comments to SPKG.txt. (I haven't yet looked at the new spkg itself at all, just read Greg's logs and comments and upstream source.)

And merge this asap, as it looks functional everywhere (I know there is some fuzz for one patch as well because of the last fix included, but I got no time to fix it once again).

:-) Unfortunately we first need new FLINT and GMP-ECM spkgs as well, AFAIK; not sure what else...

(Or does just the new FLINT spkg require MPIR 2.6, but not vice versa? I only know at least GMP-ECM currently won't build with MPIR 2.6.)

comment:51 in reply to: ↑ 50 Changed 7 years ago by jpflori

Replying to leif:

Replying to jpflori:

Maybe we can do this in a follow up ticket "Fix spkg-install of mpir spkg"?

Sure, but then we should probably add some comments to SPKG.txt. (I haven't yet looked at the new spkg itself at all, just read Greg's logs and comments and upstream source.)

I think there are already some mentions of CC and CXXFLAGS in SPKG.txt, so this does not seem new.

And merge this asap, as it looks functional everywhere (I know there is some fuzz for one patch as well because of the last fix included, but I got no time to fix it once again).

:-) Unfortunately we first need new FLINT and GMP-ECM spkgs as well, AFAIK; not sure what else...

(Or does just the new FLINT spkg require MPIR 2.6, but not vice versa? I only know at least GMP-ECM currently won't build with MPIR 2.6.)

Its the new FLINT which depends on the new MPIR. The old one should build which whatever version. In fact the new FLINT release "officially" depends on MPIR 2.6.0. The real deal is that there are symbols conflicts with MPIR 2.5.x, but as Sage still ships MPIR 2.4.x, there is no problem.

I'll have a look at the GMP-ECM problems, but it indeed reminds me of something, thanks for the hint.

comment:52 Changed 7 years ago by kcrisman

This builds on Cygwin on XP.

comment:53 follow-up: Changed 7 years ago by jpflori

No problem on 64 bits Windows 7 as well.

Is there really a problem with gmp-ecm-6.4? it seems to build perfectly with MPIR 2.6.0.

I'm aware of https://groups.google.com/forum/?fromgroups=#!searchin/mpir-devel/ecm/mpir-devel/Ba5Jl_XOSHM/pciuA7eN7dQJ but doesn't it concern later svn version? The latest svn version is still broken, a fix was commited but seems only partial.

comment:54 in reply to: ↑ 53 Changed 7 years ago by kcrisman

No problem on 64 bits Windows 7 as well.

Is there really a problem with gmp-ecm-6.4? it seems to build perfectly with MPIR 2.6.0.

On Cygwin on XP ecm-6.3.p8 (the current one) builds fine with this mpir spkg.

comment:55 Changed 7 years ago by jpflori

For me everything looks fine here, any other concern from anybody?

comment:56 Changed 7 years ago by jdemeyer

  • Authors changed from John Palmieri, Jean-Pierre Flori to John Palmieri, Jean-Pierre Flori, Karl-Dieter Crisman

comment:57 Changed 7 years ago by kcrisman

I'm not having any problems with SAGE_CHECK=yes on sage.math or OS X 10.7. On Mac,

GMP-ECM's test suite passed.

and so unless I encounter problems on either of those machines (which I'd report), let's get this in. I will point out that

patching file configure.in
Hunk #1 succeeded at 1918 (offset -2 lines).
patching file yasm/Makefile.in
patching file configure
Hunk #2 succeeded at 5265 (offset -2 lines).
Hunk #3 succeeded at 5315 (offset -2 lines).
Hunk #4 succeeded at 5322 (offset -2 lines).
Hunk #5 succeeded at 5369 (offset -2 lines).
Hunk #6 succeeded at 5377 (offset -2 lines).
Hunk #7 succeeded at 5424 (offset -2 lines).
Hunk #8 succeeded at 6579 (offset -2 lines).
Hunk #9 succeeded at 6629 (offset -2 lines).
Hunk #10 succeeded at 6636 (offset -2 lines).
Hunk #11 succeeded at 6683 (offset -2 lines).
Hunk #12 succeeded at 6691 (offset -2 lines).
Hunk #13 succeeded at 6738 (offset -2 lines).
Hunk #14 succeeded at 11749 (offset -2 lines).
Hunk #15 succeeded at 14632 (offset -2 lines).
Hunk #16 succeeded at 21785 (offset -2 lines).
Hunk #17 succeeded at 26677 (offset -2 lines).
Hunk #18 succeeded at 26812 (offset -2 lines).
Hunk #19 succeeded at 26868 (offset -2 lines).
Hunk #20 succeeded at 26910 (offset -2 lines).
Hunk #21 succeeded at 27623 (offset -2 lines).
Hunk #22 succeeded at 28377 (offset -2 lines).
Hunk #23 succeeded at 29455 (offset -2 lines).
Hunk #24 succeeded at 29590 (offset -2 lines).
Hunk #25 succeeded at 29646 (offset -2 lines).
Hunk #26 succeeded at 29685 (offset -2 lines).
Hunk #27 succeeded at 30403 (offset -2 lines).
Hunk #28 succeeded at 31157 (offset -2 lines).
Hunk #29 succeeded at 32239 (offset -2 lines).
Hunk #30 succeeded at 32374 (offset -2 lines).
Hunk #31 succeeded at 32430 (offset -2 lines).
Hunk #32 succeeded at 32469 (offset -2 lines).
Hunk #33 succeeded at 33189 (offset -2 lines).
Hunk #34 succeeded at 33943 (offset -2 lines).

But I'm not exactly worried about this, though the release manager may balk at it.

comment:58 follow-up: Changed 7 years ago by kcrisman

In both cases,

The MPIR test suite passed successfully.
GMP-ECM's test suite passed.

So I think we're okay with respect to all that.

However, possibly related, on Mac, is

All tests passed.
zn_poly's *quick* test suite passed.

Now building and installing zn_poly's shared library...
gcc  -single_module -fPIC -dynamiclib -o libzn_poly.dylib src/array.o src/inver$

Now installing zn_poly's header files...
Finished installing zn_poly.

real    1m25.136s
user    1m21.802s
sys     0m1.354s
Successfully installed zn_poly-0.9.p9
Running the test suite for zn_poly-0.9.p9...

Now building zn_poly's extensive test suite (if not already built)...
make[3]: Nothing to be done for `test'.
<snip>
zn_array_mulmid_KS1()... ok
zn_array_mulmid_KS2()... ok
zn_array_mulmid_KS3()... ok
zn_array_mulmid_KS4()... ok
nuss_mul()... FAIL!

At least one test FAILED!
Error: zn_poly failed to pass its extensive test suite.

real    13m48.475s
user    13m47.253s
sys     0m0.748s
************************************************************************
Error testing package zn_poly-0.9.p9
************************************************************************

Could be a race condition, I'm seeing what happens. On sage.math,

zn_poly has passed its extensive test suite.

comment:59 follow-up: Changed 7 years ago by jpflori

Just FYI, it seems to me that we should quite quickly get rid of zn_poly spkg. Not sure the spkg will be used at all when FLINT gets updated to 2.3, see #12173 which is definitely more than ready for review.

comment:60 in reply to: ↑ 58 ; follow-up: Changed 7 years ago by kcrisman

nuss_mul()... FAIL!
> }}}

Repeatable. And when I ./sage -f zn_poly on a normal build of the same machine,

All tests passed.
zn_poly has passed its extensive test suite.

So I'm not sure what's up, but it seems to be related to this ticket. (Mac only.) Note that both sage.math and my Mac use the gcc-4.6.3 spkg, so that's not the problem.

comment:61 in reply to: ↑ 59 ; follow-up: Changed 7 years ago by kcrisman

Notwithstanding, Sage passes all (not long) tests even with that on my Mac, as well as on sage.math. Maybe that's good enough for this ticket.


Just FYI, it seems to me that we should quite quickly get rid of zn_poly spkg. Not sure the spkg will be used at all when FLINT gets updated to 2.3, see #12173 which is definitely more than ready for review.

I don't notice any talk there about getting rid of it, though I see the name changes. That said, there are only three files in module_list.py which include zn_poly in their library list, and then there is the whole sage/schemes/hyperelliptic_curves/hypellfrob/ folder, also by David Harvey, which definitely depends on it. It does appear that "zn_poly is no longer maintained" and the latest version was included in FLINT - has been since 2009, huh. I assume its functionality is still in the "complete rewrite" of FLINT 2.x?

Last edited 7 years ago by kcrisman (previous) (diff)

comment:62 in reply to: ↑ 60 ; follow-up: Changed 7 years ago by kcrisman

nuss_mul()... FAIL!

Also with one thread and this MPIR. So it's not a race condition. Bummer.

comment:63 Changed 7 years ago by leif

Looks like I should take a look at the Clang changes we have so far (I currently don't recall any of them...), but we apparently have a "new" issue (which is present in at least MPIR 2.4.0, but presumably 2.6.0 and any older version as well), namely configure failing with clang and -O0 (which gets added if SAGE_DEBUG=yes; -O1 doesn't work either).

Ivan Andrus reported this (for Sage 5.6.beta2) on sage-release, I've cc'ed mpir-devel in my [preliminary] reply.

comment:64 follow-up: Changed 7 years ago by leif

Also, is the .p0 in the description still current?

I do have different versions, i.e., older and newer versions of a .p1 (from JP IIRC), as well as of "the"(?) .p0.

comment:65 in reply to: ↑ 64 Changed 7 years ago by leif

Replying to leif:

Also, is the .p0 in the description still current?

I do have different versions, i.e., older and newer versions of a .p1 (from JP IIRC), as well as of "the"(?) .p0.

One of my "old" .p0s (fetched around December 4th) is (still) identical to the one linked to in the description, md5sum:

2fc787c7ad53fee4d921d2f387b0dfca  mpir-2.6.0.p0.spkg

comment:66 follow-up: Changed 7 years ago by leif

patches/configure.patch is pretty unreadable, and as far as I can see SPKG.txt doesn't mention Clang at all (except for John's changelog entry, which says we shouldn't [or now do not] use it...?).

comment:67 in reply to: ↑ 66 ; follow-up: Changed 7 years ago by leif

Replying to leif:

patches/configure.patch is pretty unreadable, and as far as I can see SPKG.txt doesn't mention Clang at all (except for John's changelog entry, which says we shouldn't [or now do not] use it...?).

In spkg-install, we have:

    Darwin)
        # In some cases (see SAGE_ROOT/spkg/bin/sage-env), on Darwin,
        # CC might be set to clang, but MPIR doesn't seem to build
        # with clang.
        CLANG=`command -v clang`
        GCC=`command -v gcc`
        if [ -n "$CC" ] && [ "$CC" = "$CLANG" ] && [ -n "$GCC" ] ; then
            export CC="$GCC"
        fi

Which means we use $GCC even if CC is set to clang (actually, the output of command -v clang) if GCC is available (more precisely, if command -v gcc is non-empty).

Apart from that these tests are pretty limited [to not say broken] (and CC is always set btw.), that doesn't prevent the use of Clang. So we should either issue an error message if GCC is not available (while command -v gcc might actually give a link to clang :-) ), or we should fix the build(?) issues with Clang...

comment:68 Changed 7 years ago by leif

FWIW, with vanilla MPIR 2.6.0, after fixing only the first (yet another) configure issue, so far MPIR builds, although a 32-bit version (due to another broken test; I'll fix that later), with Clang 3.1, and passes its test suite. (This is on a Linux x86_64 box.)

comment:69 in reply to: ↑ 67 Changed 7 years ago by jpflori

Replying to leif:

Replying to leif:

patches/configure.patch is pretty unreadable, and as far as I can see SPKG.txt

Yup, IIRC that's just autocrap diff.

doesn't mention Clang at all (except for John's changelog entry, which says we shouldn't [or now do not] use it...?).

I don't think we support clang so that comes as no surprise. By the way I've reportd some time ago the clang problem upstream but I saw you even proposed fixes, that's great.

In spkg-install, we have:

    Darwin)
        # In some cases (see SAGE_ROOT/spkg/bin/sage-env), on Darwin,
        # CC might be set to clang, but MPIR doesn't seem to build
        # with clang.
        CLANG=`command -v clang`
        GCC=`command -v gcc`
        if [ -n "$CC" ] && [ "$CC" = "$CLANG" ] && [ -n "$GCC" ] ; then
            export CC="$GCC"
        fi

Which means we use $GCC even if CC is set to clang (actually, the output of command -v clang) if GCC is available (more precisely, if command -v gcc is non-empty).

Indeed.

Apart from that these tests are pretty limited [to not say broken] (and CC is always set btw.), that doesn't prevent the use of Clang. So we should either issue an error message if GCC is not available (while command -v gcc might actually give a link to clang :-) ), or we should fix the build(?) issues with Clang...

I don't really know what the best solution is. Personally I don't use Apple stuff, so I would be inclined to merge the current spkg with minimal changes and deal with the clang nightmare in a later ticket.

comment:70 in reply to: ↑ 62 ; follow-ups: Changed 7 years ago by kcrisman

nuss_mul()... FAIL!

Also with one thread and this MPIR. So it's not a race condition. Bummer.

This is not 100% platform independent; I get the same failure on Cygwin on XP, with gcc 4.5.3, with this MPIR spkg. The Mac one is of course with the Sage-provided gcc.

Last edited 7 years ago by kcrisman (previous) (diff)

comment:71 in reply to: ↑ 61 Changed 7 years ago by jpflori

Replying to kcrisman:

Notwithstanding, Sage passes all (not long) tests even with that on my Mac, as well as on sage.math. Maybe that's good enough for this ticket.


Just FYI, it seems to me that we should quite quickly get rid of zn_poly spkg. Not sure the spkg will be used at all when FLINT gets updated to 2.3, see #12173 which is definitely more than ready for review.

I don't notice any talk there about getting rid of it, though I see the name changes. That said, there are only three files in module_list.py which include zn_poly in their library list, and then

For the first file, linking to zn_poly should not be needed anymore and should be fixed at #12173. In fact I'm not really sure anyway it was, there is no direct reference to zn_poly there and it must in fact use it through FLINT 1.5.2 (which ships zn_poly...).

For the second file, the sole function which still really directly use is some "experimental preview"... The real deal is operated through FLINT (which then use its own zn_poly). I think this should be removed as well and the preview function deleted (I don't think it used anywhere).

there is the whole sage/schemes/hyperelliptic_curves/hypellfrob/ folder, also by David Harvey, which definitely depends on it. It does appear that "zn_poly is no longer maintained" and the latest version was included in FLINT - has been since 2009, huh. I assume its functionality is still in the "complete rewrite" of FLINT 2.x?

Indeed the hypellfrob spkg will still depend on zn_poly. Although I think all functionalities of zn_poly should be given by FLINT 2.3 (and even 1.5.2 which ships zn_poly remember). This is more problematic as we would need to patch hyperellfrob to use what is exposed by FLINT 1.5.2 or rather 2.3...

comment:72 in reply to: ↑ 70 Changed 7 years ago by jpflori

Replying to kcrisman:

nuss_mul()... FAIL!

Also with one thread and this MPIR. So it's not a race condition. Bummer.

This is not 100% platform independent; I get the same failure on Cygwin on XP, with gcc 4.5.3, with this MPIR spkg. The Mac one is of course with the Sage-provided gcc.

Just for info as it is surely unrelated, I have also always had random segfaults while building zn_poly on my 64 bits Windows 7 Cygwin install, with the old or the new MPIR. It occured while doing some timings for multiplications. After a few tries it could go through though.

There is also a problem mentioned on the zn_poly page about GCC 4.6.3 and make tune, but I don't think we use that target anyway.

comment:73 follow-up: Changed 7 years ago by jpflori

In fact the spkg-install calls make tune so we should be affected by the problems with GCC 4.6.3 onward Drew Sutherland mentioned, strange.

comment:75 in reply to: ↑ 70 Changed 7 years ago by jpflori

Replying to kcrisman:

nuss_mul()... FAIL!

Also with one thread and this MPIR. So it's not a race condition. Bummer.

This is not 100% platform independent; I get the same failure on Cygwin on XP, with gcc 4.5.3, with this MPIR spkg. The Mac one is of course with the Sage-provided gcc.

In fact this might indeed be a bug in the new MPIR FFT, so I'm reporting that upstream.

comment:76 Changed 7 years ago by jpflori

On what exact Mac was the test failing?

comment:77 follow-up: Changed 7 years ago by jpflori

And could you please try rebuildong MPIR and then zn_poly with other GCC versions on both Cygwin and the Mac? So both Sages GCC 4.6.3 and 4.7.2 (from #13913 I advise, much smaller) and 4.7.2 on Mac?

comment:78 in reply to: ↑ 77 ; follow-up: Changed 7 years ago by kcrisman

On what exact Mac was the test failing?

OS X 10.7 building the Sage gcc package.

And could you please try rebuildong MPIR and then zn_poly with other GCC versions on both Cygwin and the Mac? So both Sages GCC 4.6.3 and 4.7.2 (from #13913 I advise, much smaller) and 4.7.2 on Mac?

No, I can't, I'm sorry - or to be more accurate, it's just too much time on the Cygwin. It really takes me a long time to get things going on there, what with rebasing and only having occasional access to that computer. Waiting for it to compile gcc spkgs... with the semester starting, I know that would be unwise.

comment:79 in reply to: ↑ 78 Changed 7 years ago by jpflori

Replying to kcrisman:

On what exact Mac was the test failing?

OS X 10.7 building the Sage gcc package.

I guess this is on top of recent Mac, so Intel 64 bits CPU?

And could you please try rebuildong MPIR and then zn_poly with other GCC versions on both Cygwin and the Mac? So both Sages GCC 4.6.3 and 4.7.2 (from #13913 I advise, much smaller) and 4.7.2 on Mac?

No, I can't, I'm sorry - or to be more accurate, it's just too much time on the Cygwin. It really takes me a long time to get things going on there, what with rebasing and only having occasional access to that computer. Waiting for it to compile gcc spkgs... with the semester starting, I know that would be unwise.

No problem, I should have some WinXP installs at my old workplace and at home on a laptop (with broken screen...). That's not the best, but I should be able to try to reproduce it.

Could you also specify the CPU used on this WinXP install? So far I assumed it was a 32 bits CPU and the install is 32 bits but I'm not sure about the former fact anymore.

comment:80 Changed 7 years ago by leif

ROFL.

...
xmlto -o . man ./frontends/yasm/yasm.xml
/usr/bin/xmlto: line 522:   917 Segmentation fault      "$XMLLINT_PATH" --noout --nonet --xinclude --postvalid --noent "$INPUT_FILE" 2> "${VALIDATION}"
xmlto: .../sage-5.6.beta1-scratch-gcc-4.4.3/spkg/build/mpir-2.6.0.p0/src/yasm/./frontends/yasm/yasm.xml does not validate (status 139)
xmlto: Fix document syntax or use --skip-validation option
make[4]: *** [yasm.1] Error 149
make[4]: Leaving directory `.../sage-5.6.beta1-scratch-gcc-4.4.3/spkg/build/mpir-2.6.0.p0/src/yasm'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `.../sage-5.6.beta1-scratch-gcc-4.4.3/spkg/build/mpir-2.6.0.p0/src/yasm'
make[2]: *** [all] Error 2
make[2]: Leaving directory `.../sage-5.6.beta1-scratch-gcc-4.4.3/spkg/build/mpir-2.6.0.p0/src/yasm'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `.../sage-5.6.beta1-scratch-gcc-4.4.3/spkg/build/mpir-2.6.0.p0/src'
make: *** [all] Error 2
Error building MPIR.

real    10m31.056s
user    4m9.180s
sys     1m23.737s
************************************************************************

comment:81 Changed 7 years ago by leif

P.S.: FWIW, this didn't happen with the MPIR 2.4.0.p6 spkg... :-)

comment:82 follow-up: Changed 7 years ago by kcrisman

I think this Mac is one of the sort-of 64-bit ones (potential, not everything is). XP must be 32 bit, the computer is ancient. But I really don't know for sure even how to check, and I wouldn't worry about it.

Leif, what kind of computer are you on - some kind of Linux?

comment:83 in reply to: ↑ 82 ; follow-up: Changed 7 years ago by leif

Replying to kcrisman:

Leif, what kind of computer are you on - some kind of Linux?

Yes. But you probably meant the machine xmlto is failing on... ;-)

That's a 32-bit Linux box I was trying to reproduce the zn_poly error on. (I've so far only built several versions of vanilla MPIR 2.6.0 on that, with different GCC versions, and at least MPIR's test suite always passed.)

comment:84 in reply to: ↑ 83 Changed 7 years ago by leif

Replying to leif:

That's a 32-bit Linux box I was trying to reproduce the zn_poly error on. (I've so far only built several versions of vanilla MPIR 2.6.0 on that, with different GCC versions, and at least MPIR's test suite always passed.)

I'm not sure whether xmlto was installed when testing the betas and RCs of MPIR 2.6.0, can't find the build logs right now. Our MPIR 2.4.0 does look for xmlto as well, finds it, but apparently doesn't try to build yasm's man pages with it, in contrast to 2.6.0.

comment:85 follow-up: Changed 7 years ago by leif

zn_poly's quick as well as its extensive test suite passed for me with the MPIR 2.6.0.p0 spkg; Pentium4 Prescott, Ubuntu GCC 4.4.3, -march=native, -O3, ...

Will retry with FSF GCC 4.7.2 later...

comment:86 in reply to: ↑ 85 ; follow-up: Changed 7 years ago by leif

Replying to leif:

zn_poly's quick as well as its extensive test suite passed for me with the MPIR 2.6.0.p0 spkg; Pentium4 Prescott, Ubuntu GCC 4.4.3, -march=native, -O3, ...

Same with FSF GCC 4.7.2; same machine, same flags.

[I didn't build or run the tests in parallel though.]

comment:87 in reply to: ↑ 86 Changed 7 years ago by leif

Replying to leif:

Replying to leif:

zn_poly's quick as well as its extensive test suite passed for me with the MPIR 2.6.0.p0 spkg; Pentium4 Prescott, Ubuntu GCC 4.4.3, -march=native, -O3, ...

Same with FSF GCC 4.7.2; same machine, same flags.

[I didn't build or run the tests in parallel though.]

Same with {C,CPP,CXX}FLAGS unset (only in the shell of course); Ubuntu GCC 4.4.3 and FSF GCC 4.7.2.

comment:88 follow-up: Changed 7 years ago by leif

So I've run zn_poly's (quick) test suite (built with MPIR 2.6.0.p0, Ubuntu GCC 4.4.3) 50 times on the already heavily-loaded 32-bit Linux box -- without any failures.

comment:89 in reply to: ↑ 88 Changed 7 years ago by leif

Replying to leif:

So I've run zn_poly's (quick) test suite (built with MPIR 2.6.0.p0, Ubuntu GCC 4.4.3) 50 times on the already heavily-loaded 32-bit Linux box -- without any failures.

... and same for the version built with GCC 4.7.2.

comment:90 Changed 7 years ago by jpflori

From at I've read on sage-devel it is not clear that the zn_poly failure is MPIR 2.6.0 specific, waiting for feedback there, we should also try on top of GMP to gather more intel.

For leif: if you want to package a clang enabled spkg, please ase it on #12115 which was closed as we thought the one here would quickly get in.

comment:91 follow-up: Changed 7 years ago by jpflori

Ok it failed with older MPIR as well, please have a look at build logs here: https://groups.google.com/d/topic/sage-release/ngw_EiBaY0Q/discussion

comment:92 in reply to: ↑ 91 ; follow-up: Changed 7 years ago by leif

Replying to jpflori:

Ok it failed with older MPIR as well, please have a look at build logs here: https://groups.google.com/d/topic/sagei-release/ngw_EiBaY0Q/discussion

Well, that's a single failure, [although] in nuss_mul()[too] (and on an Apple of courseTM)... ;-)

One would probably have to run the tuning multiple times under heavy load to reproduce this; haven't at all looked at the code...

comment:93 in reply to: ↑ 92 ; follow-up: Changed 7 years ago by jpflori

Replying to leif:

Replying to jpflori:

Ok it failed with older MPIR as well, please have a look at build logs here: https://groups.google.com/d/topic/sage-release/ngw_EiBaY0Q/discussion

Well, that's a single failure, [although] in nuss_mul()[too] (and on an Apple of courseTM)... ;-)

Hmmm, the reports on sage-release for 5.6.beta3 also point that there is a problem with zn_poly on top of MPIR 2.4.x [on OS X], and I don't know why but I have strong suspicion it could be in nuss_mul [as well].

One would probably have to run the tuning multiple times under heavy load to reproduce this; haven't at all looked at the code...

The problem is not during the tuning phase but during the quick test one.

The testing code is quite simple, it multiplies two (large) polys using FFT:

  • first with the zn_poly code,
  • then with the underlying MPIR/GMP code (using KS)

and compare the results.

So a potential culprit was the new MPIR FFT, but as the [same] bug seemed to happen with older version of MPIR and the FFT code was completely changed inbeetween, we can mostly rule out that MPIR is to blame. It looks like we have to blame zn_poly, which is unfortunate because it is unmaintained.

And remember I mentioned I get segfaults quite often in different places as well on Cygwin when building zn_poly during the tuning phase (and tuning the threshold for FFT IIRC). So it does not seem completely crazy to blame zn_poly rather than MPIR for the other problem as well.

Of course a good test would be to try to reproduce the error on the problematic systems but on top of GMP. If the problem persists, I guess we can definitely rule out that MPIR is buggy.

comment:94 Changed 7 years ago by jpflori

Other previous problems report: https://groups.google.com/d/msg/sage-devel/FP96-mDGkxE/hlcfcZksGZMJ (this one is quite scarce I admit).

So nobody with a Mac wants to test this with GMP, MPIR 2.4.x and MPIR 2.6.0? I think bsd.math is but i don't have access to it.

comment:95 Changed 7 years ago by jpflori

Other problems reports here: https://groups.google.com/d/msg/sage-devel/2ulJLgc6-ao/zrzZZ7_uOikJ

The last one is similar to the one I evoked on Cygwin. So I guess I'll have to gdb/valgrind it on Cygwin. But I'm ok with Bill's cosmic ray explanation as well.

comment:96 in reply to: ↑ 93 ; follow-up: Changed 7 years ago by leif

Replying to jpflori:

Replying to leif:

Replying to jpflori:

Ok it failed with older MPIR as well, please have a look at build logs here: https://groups.google.com/d/topic/sage-release/ngw_EiBaY0Q/discussion

Well, that's a single failure, [although] in nuss_mul()[too] (and on an Apple of courseTM)... ;-)

Hmmm, the reports on sage-release for 5.6.beta3 also point that there is a problem with zn_poly on top of MPIR 2.4.x [on OS X], and I don't know why but I have strong suspicion it could be in nuss_mul [as well].

I didn't mean to say it was MPIR's fault. (I agree it's something with zn_poly['s tuning], perhaps in conjunction with a specific GCC version [and probably also the OS]. Note that the MacOSs most probably have GCC 4.6.3 [from our spkg] in common.]


One would probably have to run the tuning multiple times under heavy load to reproduce this; haven't at all looked at the code...

The problem is not during the tuning phase but during the quick test one.

Well, it shows up during the tests, but presumably the tuning generates invalid code (or code triggering a compiler bug) under certain circumstances, like perhaps heavy load.

comment:97 Changed 7 years ago by jpflori

I agree, my point is that the zn_poly problem should not be dealt with here and should not prevent the upgrade of MPIR if it is unrelated.

comment:98 Changed 7 years ago by jpflori

Nor should the clang stuff in my opinion, it should be on a follow-up ticket.

But we should fix the xmlto problem

And patch the patches so that there is no offset, only changing the line numbers (but not the timestamps et al. in order to keep the hg history small if possible), if someone wants to write a script to do that automatically I'll be more than happy.

comment:99 in reply to: ↑ 73 Changed 7 years ago by jpflori

Replying to jpflori:

In fact the spkg-install calls make tune so we should be affected by the problems with GCC 4.6.3 onward Drew Sutherland mentioned, strange.

Ok, we don't have that one because we fixed it at #8771.

comment:100 in reply to: ↑ 96 ; follow-up: Changed 7 years ago by jpflori

Replying to leif:

Replying to jpflori:

Replying to leif:

Replying to jpflori:

Ok it failed with older MPIR as well, please have a look at build logs here: https://groups.google.com/d/topic/sage-release/ngw_EiBaY0Q/discussion

Well, that's a single failure, [although] in nuss_mul()[too] (and on an Apple of courseTM)... ;-)

Hmmm, the reports on sage-release for 5.6.beta3 also point that there is a problem with zn_poly on top of MPIR 2.4.x [on OS X], and I don't know why but I have strong suspicion it could be in nuss_mul [as well].

I didn't mean to say it was MPIR's fault. (I agree it's something with zn_poly['s tuning], perhaps in conjunction with a specific GCC version [and probably also the OS]. Note that the MacOSs most probably have GCC 4.6.3 [from our spkg] in common.]


One would probably have to run the tuning multiple times under heavy load to reproduce this; haven't at all looked at the code...

The problem is not during the tuning phase but during the quick test one.

Well, it shows up during the tests, but presumably the tuning generates invalid code (or code triggering a compiler bug) under certain circumstances, like perhaps heavy load.

From what I see after a quick look, the tuning process only chooses thresholds, and does not generates code. So maybe the code is genuinely buggy for certain random values, or there is a compiler bug (but from what we gathered I wouldn't say we always use the same GCC, bugs appear at least with FSF GCC 4.6.3 on different Os X and with Cywgin GCC 4.5.3 (maybe with later FSF GCC don't remember) on ... Cygwin) that generates code which fails on certain random values.

Or maybe under heavy load, the thresholds get chosen so that code is called on value ranges it is not meant for and boom (but only on Os X and Cygwin, if that was to be true on Linuces I guess we would have seen it before).

comment:101 in reply to: ↑ 100 ; follow-up: Changed 7 years ago by leif

Replying to jpflori:

Or maybe under heavy load, the thresholds get chosen so that code is called on value ranges it is not meant for and boom (but only on Os X and Cygwin, if that was to be true on Linuces I guess we would have seen it before).

I just did

for ((r=1;r<=100;r=r+1)); do
    tune/tune > src/tuning.c && make && test/test -quick nuss_mul || break
done

on bsd.math, with an older version of Sage (but GCC 4.6.3) though, and also zn_poly-0.9.p5 (not sure whether that matters...).

Since "unfortunately" the sysload happened to be low, I started building Sage in parallel during the 50th run, but that didn't "help" either. 8-)

I'll probably retry with the .p9...

Ceterum censeo we should open another ticket for the zn_poly issue.

comment:102 in reply to: ↑ 101 Changed 7 years ago by leif

Replying to leif:

I just did

for ((r=1;r<=100;r=r+1)); do
    tune/tune > src/tuning.c && make && test/test -quick nuss_mul || break
done

on bsd.math, with an older version of Sage (but GCC 4.6.3) though, and also zn_poly-0.9.p5 (not sure whether that matters...).

Since "unfortunately" the sysload happened to be low, I started building Sage in parallel during the 50th run, but that didn't "help" either. 8-)

I retried with Sage 5.5 (which comes with zn_poly 0.9.p9), .../test -quick all and a higher sysload, but couldn't reproduce the test failure either. (Perhaps bsd.math is still too fast...)

comment:103 Changed 7 years ago by jpflori

Updated spkg fixing the offset of configure.patch.

From my point of view the last thing to fix in this ticket is the xmlto one.

I'd prefer to postpone the clang one to another ticket and the zn_poly as well.

Changed 7 years ago by jpflori

Spkg diff, for review only.

comment:104 Changed 7 years ago by jpflori

About xmlto, I cannot reproduce the problem on my Ubuntu 12.04.1 x86_64. I did not have installed before, what was correctly detected, so I just installed it and it is detected by configure but it does not get used when I just issue make in a fresh MPIR dir or through Sage's spkg mecanism.

comment:105 Changed 7 years ago by jpflori

I spoke too fast, it gets used when installing with Sage's spkg mechanism and just works, but not in a plain upstream dir...

comment:106 Changed 7 years ago by jpflori

That must be some timestamp autotools magic because we patch some autotools file...

comment:107 Changed 7 years ago by jpflori

Or not, patching upstream with our patches and then configuring and building does not trigger the use of xmlto (although it is detected).

comment:108 Changed 7 years ago by jpflori

Ok, my bad, I must have changed some timestamp while tarring untarring stuff whence the rebuild of the man pages with the spkg (and not with upstream correctly untarred). Corrected spkg coming soon.

comment:109 Changed 7 years ago by jpflori

The just uploaded spkg should avoid using xmlto even if it's installed (and broken :)).

comment:110 Changed 7 years ago by jhpalmieri

With the old versions of the spkgs (the ones included in 5.6.beta3), on an OS X 10.8.2 machine: in one window, I can load the system by running ./sage -tp devel/sage/sage. Then in another window, I can build zn_poly, or test it by running ./test nuss_mul from spkg/build/zn_poly-0.9.p9/src/test/.

  • if the system is loaded when zn_poly is built (using ./sage -f -s spkg/standard/zn_poly-0.9.p9), then the test fails frequently. It doesn't matter whether the system is loaded while running the test, only if it was loaded while installing the spkg.
  • on the other hand, if the system is not loaded when installing the spkg, the test seems to pass consistently (regardless of the load while the test is being run).

I haven't tried this with any new versions of the spkgs.

comment:111 follow-up: Changed 7 years ago by leif

John, could you please save the tuning parameters (.../zn_poly-0.9.p9/src/src/tuning.c) from [a] failing test[s]?

Last edited 7 years ago by leif (previous) (diff)

comment:112 in reply to: ↑ 111 Changed 7 years ago by jhpalmieri

Replying to leif:

John, could you please save the tuning parameters (.../zn_poly-0.9.p9/src/src/tuning.c) from [a] failing test[s]?

I saved it. Did you want me to do anything with it? ;)

comment:113 Changed 7 years ago by jpflori

I opened #13947 for the zn_poly issue, I suggest we continue to investigate the issue there.

Same at #13948 for the clang issue.

comment:114 Changed 7 years ago by leif

Jean-Pierre, do you intend to make any further changes to the spkg here? (Or, is there anything left which has to be done here, as of now?)

comment:115 follow-up: Changed 7 years ago by jpflori

I don't see anything to change now, but I may have missed something.

comment:116 in reply to: ↑ 115 Changed 7 years ago by leif

Replying to jpflori:

I don't see anything to change now, but I may have missed something.

I'll repeat some tests with the "final" spkg (although I can't on Cygwin), do some checks, then I'll give it positive review I think.

comment:117 Changed 7 years ago by leif

A few (IMHO minor) things:

  • spkg-install

    diff --git a/spkg-install b/spkg-install
    a b  
    1919# detecting filename changes. See Trac #13137.
    2020# This fix can (and will have to) be removed once integrated upstream.
    2121echo "Renaming *,asm files to *.asm..."
    22 mv mpn/x86/p6/addmul_1,asm mpn/x86/p6/addmul_1.asm
     22mv mpn/x86/p6/addmul_1,asm mpn/x86/p6/addmul_1.asm &&
    2323mv mpn/x86/p6/submul_1,asm mpn/x86/p6/submul_1.asm
    2424if [ $? -ne 0 ]; then
    2525    echo >&2 "Error: moving *,asm files failed."
     
    202202    echo "Building a reduced version of MPIR to bootstrap GCC."
    203203    echo "MPIR will later get rebuilt (with the C++ interface and static libraries"
    204204    echo "enabled) using the new compiler."
    205     MPIR_CONFIGURE="--disable-cxx --disable-static $MPIR_CONFIGURE"
     205    MPIR_CONFIGURE="$MPIR_CONFIGURE --disable-cxx --disable-static"
    206206    SAGE_FAT_BINARY=no
    207207else
    208208    # Also build the static library to be used by e.g. ECM
    209209    # unless we are on Cygwin where we can only build a shared
    210     # or a static library but not both
     210    # or a static library but not both:
    211211    if [ "$UNAME" = "CYGWIN" ]; then
    212         echo "Building MPIR with the C++ interface."
    213         MPIR_CONFIGURE="--enable-cxx --disable-static $MPIR_CONFIGURE"
     212        echo "Building MPIR with the C++ interface and (only) shared libraries."
     213        MPIR_CONFIGURE="--enable-cxx $MPIR_CONFIGURE --disable-static"
    214214    else
    215215        echo "Building MPIR with the C++ interface and (also) static libraries."
    216216        MPIR_CONFIGURE="--enable-cxx --enable-static $MPIR_CONFIGURE"
  • The date of the changelog entry. ;-)
  • The current treatment of clang in spkg-install (as mentioned earlier), but we can postpone changes to that to #13948 (although pretty unrelated, as it is only necessary to slightly patch acinclude.m4 and configure).

comment:118 Changed 7 years ago by leif

P.S.: The spkg is otherwise clean. :-)

comment:119 Changed 7 years ago by leif

For completeness:

  • SPKG.txt

    diff --git a/SPKG.txt b/SPKG.txt
    a b  
    5050
    5151== Changelog ==
    5252
    53 === mpir-2.6.0.p0 (Jean-Pierre Flori, 25 November 2012) ===
     53=== mpir-2.6.0.p0 (Jean-Pierre Flori, 12 January 2013) ===
    5454 * Trac #13137: Update to MPIR 2.6.0.
    5555 * Modify spkg-install to rename *,asm files to *.asm files.
    5656 * Remove -Wl,-z,noexecstack fix which has been integrated upstream.
    57  * Remove old code about 32 bits Apple Darwin and use upstream fix.
     57 * Remove old code about 32 bits Apple Darwin and use slightly modified upstream fix.
    5858
    5959=== mpir-2.5.2.p0 (John Palmieri, 3 October 2012) ===
    6060 * Trac #13137: Update to MPIR 2.5.2.

An updated spkg with the above changes (to spkg-install and SPKG.txt) is here, md5sum ec21b55dce699d3c8ddf0492e1413107.

comment:120 Changed 7 years ago by leif

  • Reviewers changed from Jeroen Demeyer to Jeroen Demeyer, Leif Leonhardy

comment:121 Changed 7 years ago by leif

  • Description modified (diff)
  • Status changed from needs_review to positive_review
  • Summary changed from upgrade MPIR to 2.6.0 to Upgrade MPIR to 2.6.0

I've put the link to the spkg with my minor corrections into the description.

comment:122 Changed 7 years ago by jdemeyer

  • Milestone changed from sage-5.6 to sage-5.7

comment:123 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:124 Changed 7 years ago by kcrisman

  • Reviewers changed from Jeroen Demeyer, Leif Leonhardy to Jeroen Demeyer, Leif Leonhardy, John Palmieri, R. Andrew Ohana, Karl-Dieter Crisman

comment:125 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.7.beta0
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.