OS X Lion: pari fails self tests

[jhpalmieri]

On OS X Lion, if you set SAGE_CHECK=yes and build pari, some tests fail. See

 http://sage.math.washington.edu/home/palmieri/misc/pari-2.5.0.p3.log

for a log file.

---

New spkg:  http://sage.math.washington.edu/home/palmieri/SPKG/pari-2.5.0.p4.spkg

[jhpalmieri]

was pointed out  on sage-devel that with SAGE_DEBUG=True, self-tests pass. So it may be a compiler error; compiling with -O0 seems to work. It may be just a single file (the one containing the bezout function) causing the problems.

[kcrisman]

An exchange on #11881 about this.

I am currently at the PARI atelier and we're discussing OS X Lion. Essentially the PARI developers are saying that Apple's compiler is completely broken and that the Only Right Solution is to install a real compiler, i.e. gcc. So maybe the best solution in practice for Sage is to add a check in prereq that users aren't using Apple's broken compiler.

? So that no one using "just Xcode" can build Sage? I'm a little unclear on what would have to change for ordinary users.

Also we should have one server machine running Lion with a good compiler to build binaries.

Certainly. jhpalmieri points out on #12315 (where these comments should probably live anyway)

was pointed out on sage-devel that with SAGE_DEBUG=True, self-tests pass. So it may be a compiler error; compiling with -O0 seems to work. It may be just a single file (the one containing the bezout function) causing the problems.

So perhaps a workaround would be to check for the "bad" compiler in the spkg-install somehow and then do no optimization if that's the case. Power users who need Pari optimized would be expected to have the technical know-how to use the right one, and as you say the binary builder could have this as well.

[kcrisman]

Another datapoint, from here:

I took sage-4.8.alpha0, added in the spkg's from the various tickets here, including the Python spkg from #11967, and built Sage with SAGE_CHECK=yes. Tests failed for the Python spkg, as usual, but they passed for PARI.

Was Pari upgraded in the meantime? I assume so.

[jhpalmieri]

Replying to kcrisman:

Another datapoint, from here:

I took sage-4.8.alpha0, added in the spkg's from the various tickets here, including the Python spkg from #11967, and built Sage with SAGE_CHECK=yes. Tests failed for the Python spkg, as usual, but they passed for PARI.

I've been unable to reproduce that, so I don't know how I got Pari's self-tests to pass that one time.

It seems to me that our current viable choices are

1. Test whether the platform is Lion, and if so, turn off optimization. (Using -O0 should work; I don't know about -O1, but I'm testing that now.)

2. Test whether the platform is Lion, and if so, turn off optimization for the single file causing problems.

3. Test whether the platform is Lion and the compiler is a bad one (I'm not sure what qualifies beyond the current GCC/LLVM thing in !XCode), and if so, turn off optimization.

I can prepare an spkg doing the first of these, but I'm not sure about how to do #2 the right way, and I don't know enough about which compiler(s) are bad and which are good to do #3.

[jdemeyer]

This should be coordinated with #12363, but go ahead. I would prefer option 3. to option 1. Could you not simply blacklist the default compiler on OS X 10.7?

[kcrisman]

Could you not simply blacklist the default compiler on OS X 10.7?

Cf: William's recent comment on #11881 about whether he wants to do so on Lion:

Power users who need Pari optimized would be expected to have the technical know-how to use the right one,

I'm such a user and I don't want to have to build GCC from source.

It's pretty painful to deal with anything else on OS X. We couldn't even really use the fink or macports gcc, because we currently require people to rename or remove from their path anything in /sw.

[jdemeyer]

Replying to kcrisman:

Could you not simply blacklist the default compiler on OS X 10.7?

Cf: William's recent comment on #11881 about whether he wants to do so on Lion:

Power users who need Pari optimized would be expected to have the technical know-how to use the right one,

I'm such a user and I don't want to have to build GCC from source.

It's pretty painful to deal with anything else on OS X. We couldn't even really use the fink or macports gcc, because we currently require people to rename or remove from their path anything in /sw.

Unless we create an spkg "gcc" which works properly with Sage. I noticed somebody made an experimental package "gcc" but I haven't tried it.

[jhpalmieri]

If the problem is a single file, the file  should be src/src/kernel/none/gcdll.c. I tried compiling just that file with -O0, but I still got self test failures -- not as many, but a few:

ell-sta.dif
ell.dyn.dif

To build this way, I patched the file src/src/kernel/gmp/MakeLVL1.SH, adding in the -O0 flag.

For what it's worth, on this particular machine, it took about 19 minutes to run self tests after compiling all of Pari with -O0, and it took about 15 minutes compiling most with -O3 and just the one file with -O0.

[jdemeyer]

Please post the file ell-sta.dif (it should be inside Odarwin-* in the pari build directory).

[was]

Replying to jdemeyer:

This should be coordinated with #12363, but go ahead. I would prefer option 3.

to option 1. Could you not simply blacklist the default compiler on OS X 10.7?

I consider option 3 a total non-option unless there is a fully working alternative GCC spkg available. If there were, that would be the way to go. I posted to sage-devel to see if anybody has something like this.

I prefer option 2 by far, along with reporting the bugs to APPLE. Then just turn off 2 once the bug is fixed (but of course don't just build broken sage's by people who haven't upgraded yet).

[jhpalmieri]

Since the file is small, I'm attaching ell-sta.dif.

[jhpalmieri]

Okay, nailed it. On OS X Lion, 'gcc' is a symbolic link to '/usr/bin/llvm-gcc-4.2', but '/usr/bin/gcc-4.2' is a plain version of gcc. So if, when building Pari, we do export CC=gcc-4.2, then self-tests pass when building with standard (-O3) optimization. So we have an easy solution, but it leads to several questions:

• Is the version of OS X (Lion vs. Snow Leopard) important, or is it only the version of XCode? That is, should we just test whether 'LLVM' is in the output from gcc --version (or perhaps something more refined, like 'LLVM' is in the output from gcc --version and gcc is version 4.2.1, or gcc --version contains the string "LLVM build 2.335.15.00")?

• Should we make this change only for Pari, or should we do this when building all of Sage? I have a feeling we do it on a case-by-case basis, since I think that not all spkgs respect the CC variable. I tried it with cvxopt (see #12011), and it didn't seem to help, but it's worth exploring more...

I've posted an spkg which tests whether we're using an 'LLVM' version of gcc 4.2.1.

[was]

NICE!!!

Could you look into symmetrica too?

NICE!!!

[jhpalmieri]

Sure, I'll try. Is the problem with symmetrica only revealed by failed Sage doctests? (I don't see a ticket for symmetrica on Lion at #11881).

[was]

Replying to jhpalmieri:

Sure, I'll try. Is the problem with symmetrica only revealed by failed Sage doctests?

Yes.

[kcrisman]

Great work, jhpalmieri. Question: is there any (tiny) chance that the '/usr/bin/gcc-4.2' is a leftover from a previous Xcode you had on this machine? It would be good to test this on a machine that has only had Xcode 4 and Lion on it.

[jhpalmieri]

Replying to kcrisman:

Great work, jhpalmieri. Question: is there any (tiny) chance that the '/usr/bin/gcc-4.2' is a leftover from a previous Xcode you had on this machine? It would be good to test this on a machine that has only had Xcode 4 and Lion on it.

I think there is a tiny chance, but all of these different versions of gcc (gcc-4.2, llvm-gcc-4.2, etc) have the same modification date, so I think they all came from XCode 4.

Meanwhile, I'm having problems which I'm trying to track down: with sage-5.0.prealpha1, I get no test failures with this Pari spkg. With sage-5.0.beta1, I get a few test failures. I'm not sure what the difference is, though. Maybe the termcap spkg?

[kcrisman]

Great work, jhpalmieri. Question: is there any (tiny) chance that the '/usr/bin/gcc-4.2' is a leftover from a previous Xcode you had on this machine? It would be good to test this on a machine that has only had Xcode 4 and Lion on it.

I think there is a tiny chance, but all of these different versions of gcc (gcc-4.2, llvm-gcc-4.2, etc) have the same modification date, so I think they all came from XCode 4.

Hmm, but see  Georg's post about this on sage-devel:

Stay with the old "FSF-flavor" Apple version of gcc v4.2.1 "build 
5666.3", as has been included in XCode 3.2.6, 4.0. and 4.1 (but no 
longer from 4.2 onwards), and that gcc42 package. 

So maybe this will go away, and we might not want to rely on it.

[jhpalmieri]

Okay, at least I've figured out the problem with test failures: I based my spkg on a prerelease version of 2.5.1 (from #12363), and the installation process wasn't installing all of the required data files. This was making tests fail. In the prealpha1 installation, I must have installed an earlier version of Pari (without SAGE_CHECK set), so it had installed the data earlier.

Now I'm working on a different spkg based on the old version of Pari, and things are looking good.

[jhpalmieri]

(The posted spkg is broken because I used an incomplete tar ball. Once the spkg at #12363 is in good shape -- it's too big right now -- I'll post a new one based on that.)

[jhpalmieri]

Here's a new spkg, based on the old version of Pari. This gives us something to test with on Lion. If it works and if we're happy with the solution, the patch here should be easy to merge with the spkg at #12363.

I'm not what to do about the possible lack of gcc-4.2 in newer versions of XCode. I can't find any verification of Georg's statement about this.

[jhpalmieri]

By the way, I tried

CC=clang  # instead of gcc-4.2
export CC

but spkg-check had lots of failures.

[jhpalmieri]

Maybe I messed up last time, but using clang is working for me now. Since clang is part of XCode 4, it is guaranteed to be there (as opposed to gcc-4.2). Using clang with the standard CFLAGS is presumably faster than using the default compiler with optimization turned off.

So I've updated the spkg to use clang.

[jhpalmieri]

patch for pari spkg; for reference only

[jdemeyer]

Since PARI-2.5.0 doesn't work with gcc-4.6.2 on OS X, but PARI-2.5.1 (#12363) does work, you might want to work with the new PARI.

[jhpalmieri]

I think this ticket should be closed, since the Pari issues are taken care of via #12369.