Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#10120 closed defect (duplicate)

Errors building the PARI/GP spkg with GCC 4.4.1 (Fedora 11, openSUSE 11.2)

Reported by: mpatel Owned by: tbd
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: packages: standard Keywords: gcc pari opensuse11.2 gcc4.4.1 fedora11
Cc: jdemeyer, hivert, Koen Merged in:
Authors: Reviewers: Koen van de Sande
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by jdemeyer)

Mike Witt has reported a problem building the PARI/GP package (pari-2.4.3.svn-12577.p7) in sage-4.6.alpha3 on a 32-bit Fedora 11 system with 4 GB of RAM using gcc 4.4.1 20090725 (Red Hat 4.4.1-2). From the install.log:

gcc  -c -O3 -Wall -fno-strict-aliasing -fomit-frame-pointer  -O3 -g
-I. -I../src/headers -fPIC -o base2.o ../src/basemath/base2.c
gcc  -c -O3 -Wall -fno-strict-aliasing -fomit-frame-pointer  -O3 -g
-I. -I../src/headers -fPIC -o base3.o ../src/basemath/base3.c

cc1: out of memory allocating 248 bytes after a total of 2883715072 bytes
make[3]: *** [base3.o] Error 1
make[3]: Leaving directory
`/usr/local/sage-4.6.alpha3/spkg/build/pari-2.4.3.svn-12577.p7/src/Olinux-i686'
make[2]: *** [gp] Error 2
make[2]: Leaving directory
`/usr/local/sage-4.6.alpha3/spkg/build/pari-2.4.3.svn-12577.p7/src'
Error building GP 

See sage-release for some details. Jeroen Demeyer has suggested that this is another GCC bug uncovered by PARI.

Fixed by #10572.

Change History (30)

comment:1 Changed 11 years ago by mpatel

  • Description modified (diff)

comment:2 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:3 Changed 11 years ago by jdemeyer

Just for fun, let me point out that the following gcc bug was also found by compling PARI: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=37544

comment:4 Changed 11 years ago by wuthrich

I have the same problem with GNU compiler version 4.4.1 on SUSE 11, 64 bit. This is independent of sage as I can not compile the dev version of pari, either.

comment:5 follow-ups: Changed 11 years ago by leif

Is it worth addressing such within Sage? (I personally don't think so.)

Manually changing CFLAGS to -O2 ... (just for the PARI package) might help.

(As of course upgrading Fedora or just GCC would as well... ;-) )

comment:6 in reply to: ↑ 5 Changed 11 years ago by jdemeyer

Replying to leif:

Is it worth addressing such within Sage? (I personally don't think so.)

I don't think it's worth to special-case every broken version of gcc in Sage, espcially if the problem is fixed in newer versions of gcc.

comment:7 in reply to: ↑ 5 ; follow-up: Changed 11 years ago by leif

Replying to leif:

Manually changing CFLAGS to -O2 ... (just for the PARI package) might help.

Mike Witt reported on sage-release (for Sage 4.6.rc0) that -O2 (-fno-strict-aliasing -fomit-frame-pointer) didn't help to get around this bug; setting SAGE_DEBUG=yes instead (which sets CFLAGS to ... -O0, i.e. completely disables optimization) worked for him.

comment:8 in reply to: ↑ 7 Changed 11 years ago by leif

Replying to leif:

Replying to leif:

Manually changing CFLAGS to -O2 ... (just for the PARI package) might help.

Mike Witt reported on sage-release (for Sage 4.6.rc0) that -O2 (-fno-strict-aliasing -fomit-frame-pointer) didn't help to get around this bug; setting SAGE_DEBUG=yes instead (which sets CFLAGS to ... -O0, i.e. completely disables optimization) worked for him.

-O1 ... also worked for him.

comment:9 Changed 11 years ago by jdemeyer

  • Keywords gcc pari added

This emphasizes even more that it really is a gcc bug.

comment:10 Changed 11 years ago by leif

  • Status changed from new to needs_info

Should we document this in the Installation Guide? README.txt? Release notes of Sage 4.6?

(It's fairly out of date anyway, but updating at least some parts is IMHO overdue, since doing so hopefully prevents users again reporting known issues and should reduce potential annoyance.)

comment:11 Changed 11 years ago by leif

  • Summary changed from Error building the PARI/GP spkg on 32-bit Fedora 11 to Error building the PARI/GP spkg on 32-bit Fedora 11 (with GCC 4.4.1)

A better work-around than export SAGE_DEBUG=yes might be to let the (preferably parallel build) fail on PARI in the first place, then install PARI with modified CFLAGS, and after that continue the build by re-running make with "better" (or unset) CFLAGS:

$ export SAGE_PARALLEL_SPKG_BUILD=yes

$ export MAKE="make -j4"  # for example; using more threads/jobs than the number
                          # of cores the CPU has is usually ok, at least if the
                          # machine has enough memory (RAM) available

$ make build  # should build all packages that do not depend on PARI, but will
              # fail for PARI

$ export CFLAGS="-O1 -fno-strict-aliasing -fomit-frame-pointer"

$ ./sage -i spkg/standard/pari-2.4.3*  # build/install only the PARI spkg;
                                       # _should_ work, unless the scripts
                                       # aren't yet installed

$ unset CFLAGS  # clear them, such that the packages' defaults will be used

$ make  # continue building the remaining packages and the documentation

If the sage -i ... doesn't work (because the necessary scripts aren't yet installed), one could try once again running make build (with unset or "normal" CFLAGS), then retrying the sage -i ..., again with the modified CFLAGS as given above, after a successful build of PARI continuing with unset CFLAGS followed by make.

Another approach (untested) which makes sure the necessary scripts are installed after the build of PARI has failed in the first place is to use the -k ("keep going") option of make, i.e. replacing

$ make build

above by

$ make -k build

(Of course one could do the same with (additional) custom CFLAGS for the usual build, substituting the unset CFLAGS by export CFLAGS="...", and prepending those flags to the ones used above for building PARI.)

If one doesn't want to mess around with setting and unsetting CFLAGS many times, here's the solution with one very long line to avoid this, this time without building in parallel:

$ make -k build  # should build all packages that do not depend on PARI, but will
                 # fail for PARI (with GCC 4.4.1)

# build/install only the PARI spkg, with "temporarily" modified CFLAGS:
$ env CFLAGS="-O1 -fno-strict-aliasing -fomit-frame-pointer" ./sage -i spkg/standard/pari-2.4.3*

$ make  # continue building the remaining packages and the documentation

comment:12 Changed 11 years ago by mpatel

Finotti has inquired at AskSage about a problem compiling 4.6 on 64-bit Fedora 11:

stark.o ../src/modules/stark.c
gcc  -c -O3 -Wall -fno-strict-aliasing -fomit-frame-pointer  -O3 -g   -I. -I../src/headers -fPIC -o subfield.o ../src/modules/subfield.c
gcc  -c -O3 -Wall -fno-strict-aliasing -fomit-frame-pointer  -O3 -g   -I. -I../src/headers -fPIC -o thue.o ../src/modules/thue.c
gcc: Internal error: Killed (program cc1)
Please submit a full bug report.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
make[3]: *** [base3.o] Error 1
make[3]: Leaving directory `/work/local/sage-4.6/spkg/build/pari-2.4.3.svn-12577.p9/src/Olinux-x86_64'
make[2]: *** [gp] Error 2
make[2]: Leaving directory `/work/local/sage-4.6/spkg/build/pari-2.4.3.svn-12577.p9/src'
Error building GP

comment:13 follow-up: Changed 11 years ago by jdemeyer

Another gcc bug...

comment:14 in reply to: ↑ 13 ; follow-up: Changed 11 years ago by leif

Replying to jdemeyer:

Another gcc bug...

I'd say the same (or very similar). I don't understand why the Fedora guys don't provide an upgraded GCC package.

Mitesh, I assume you've pointed him here. (I have never looked at ask.sagemath.org.)

comment:15 in reply to: ↑ 14 Changed 11 years ago by mpatel

Replying to leif:

Replying to jdemeyer:

Another gcc bug...

I'd say the same (or very similar). I don't understand why the Fedora guys don't provide an upgraded GCC package.

Mitesh, I assume you've pointed him here. (I have never looked at ask.sagemath.org.)

Mike Witt mentioned the SAGE_DEBUG work-around and this ticket in his answer.

comment:16 follow-up: Changed 11 years ago by Koen

  • Keywords opensuse11.2 gcc4.4.1 fedora11 added

For reference: OpenSuse? 11.2 (gcc (SUSE Linux) 4.4.1 [gcc-4_4-branch revision 150839]) has the same problem when building PARI: on a machine with 64GB of RAM, it eventually fails after all memory is exhausted (takes hours). It happens when compiling the base3.c file:

gcc  -c -O3 -Wall -fno-strict-aliasing -fomit-frame-pointer  -O3 -g   -I. -I../src/headers -fPIC -o base1.o ../src/basemath/base1.c
gcc  -c -O3 -Wall -fno-strict-aliasing -fomit-frame-pointer  -O3 -g   -I. -I../src/headers -fPIC -o base2.o ../src/basemath/base2.c
gcc  -c -O3 -Wall -fno-strict-aliasing -fomit-frame-pointer  -O3 -g   -I. -I../src/headers -fPIC -o base3.o ../src/basemath/base3.c

Unfortunately, distros generally refuse to update their compiler, because all OS packages are compiled with it and they do not want to run the risk of breaking other packages.

comment:17 in reply to: ↑ 16 Changed 11 years ago by leif

  • Cc hivert added
  • Summary changed from Error building the PARI/GP spkg on 32-bit Fedora 11 (with GCC 4.4.1) to Errors building the PARI/GP spkg with GCC 4.4.1 (Fedora 11, openSUSE 11.2)

Replying to Koen:

For reference: OpenSuse? 11.2 (gcc (SUSE Linux) 4.4.1 [gcc-4_4-branch revision 150839]) has the same problem when building PARI: on a machine with 64GB of RAM, it eventually fails after all memory is exhausted (takes hours).

Hmmm, install more RAM?

What CPU type is that? I know others have built successfully on openSUSE 11.2.


Unfortunately, distros generally refuse to update their compiler, because all OS packages are compiled with it and they do not want to run the risk of breaking other packages.

This used to be different in the good old times (and some even offered source packages IIRC). Also, Fedora / RedHat? IMHO tend to include / build with "too fresh" compilers.

Perhaps we should really check the gcc version in PARI's spkg-install, and eventually switch to -O1, but IMHO this should be fixed upstream such that only the critical parts get built with that.

Can you report this upstream?

comment:18 follow-up: Changed 11 years ago by Koen

The CPU type is Nehalem (Core i7/Xeon 5500 series), specifically an X5570.

Reporting it upstream probably won't do much good, because the later release openSUSE 11.3 uses a different GCC already (4.5), and upgrading the default compiler 12 months after 11.2 release will be judged too risky...

comment:19 in reply to: ↑ 18 Changed 11 years ago by leif

Replying to Koen:

The CPU type is Nehalem (Core i7/Xeon 5500 series), specifically an X5570.

We consider "trial building" with limits, reducing the optimization level on failures (See http://trac.sagemath.org/sage_trac/ticket/10430#comment:3 ff.)

Thoughts?


Reporting it upstream probably won't do much good, because the later release openSUSE 11.3 uses a different GCC already (4.5), and upgrading the default compiler 12 months after 11.2 release will be judged too risky...

I meant rather PARI, such that the developers get more aware of these odd failures (on a potentially large number of systems).

comment:20 Changed 11 years ago by jdemeyer

At #10430, there is a spkg to test: http://sage.math.washington.edu/home/jdemeyer/spkg/pari-2.4.3.alpha.p1.spkg. It would be very useful to know whether this solves the problem reported here.

comment:21 Changed 11 years ago by Koen

I have tested it quickly (by killing the compiler process by hand when it got stuck for a few minutes on the base3.c file), and indeed the updated version falls back to -O2, then to -O1. With -O1, GCC 4.4.1 is able to successfully build the package. So the skpg solves this problem for me :)

To see how long it takes to 'fail' twice, I will now let it run for a few hours without manual intervention.

comment:22 follow-up: Changed 11 years ago by Koen

Killed it after 11GB of memory use and 17 minutes of CPU time on a single file; the fallback to -O1 works, but I'm guessing the updated spkg does not impose any limits on the compilation.

comment:23 in reply to: ↑ 22 Changed 11 years ago by jdemeyer

Replying to Koen:

Killed it after 11GB of memory use and 17 minutes of CPU time on a single file; the fallback to -O1 works, but I'm guessing the updated spkg does not impose any limits on the compilation.

Well, it should :-)

Try the following in a bash session:

$ ulimit -v
$ ( ulimit -v 1; gcc --version )

comment:24 follow-up: Changed 11 years ago by Koen

Ah, sorry for the confusion; it is a machine with a lot of memory (>64GB), and the memory limit by default is configured to 60GB. I thought the compilation process itself also specified some kind of limit.

ksande@kameleon:~> ulimit -v
61182400
ksande@kameleon:~> ulimit -v 1; gcc --version
Killed

comment:25 in reply to: ↑ 24 Changed 11 years ago by jdemeyer

Replying to Koen:

Ah, sorry for the confusion; it is a machine with a lot of memory (>64GB), and the memory limit by default is configured to 60GB. I thought the compilation process itself also specified some kind of limit.

I have made a new spkg (same location: http://sage.math.washington.edu/home/jdemeyer/spkg/pari-2.4.3.alpha.p1.spkg). Can you try with that?

comment:26 Changed 11 years ago by Koen

Great, this new version fails much sooner on -O3/-O2 (hardly any delay), and after that it finishes correctly on -O1.

gcc  -c -O3 -Wall -fno-strict-aliasing -fomit-frame-pointer  -O3 -g   -I. -I../src/headers -fPIC -o base3.o ../src/basemath/base3.c

cc1: out of memory allocating 488 bytes after a total of 341737472 bytes
make[1]: *** [base3.o] Error 1
make[1]: Leaving directory `/home/ksande/temp/sage-4.6/spkg/build/pari-2.4.3.alpha.p1/src/Olinux-x86_64'
make: *** [gp] Error 2
===========================================
Building PARI/GP with optimization flag -O2
===========================================

comment:27 Changed 11 years ago by jdemeyer

  • Description modified (diff)
  • Milestone changed from sage-4.6.1 to sage-duplicate/invalid/wontfix
  • Resolution set to duplicate
  • Status changed from needs_info to closed

comment:28 Changed 11 years ago by jdemeyer

  • Cc Koen added

Koen: could you add your real name as "reviewer" on this ticket and also add it to http://trac.sagemath.org/sage_trac/#AccountNamesmappedtoRealNames?

comment:29 Changed 11 years ago by Koen

  • Reviewers set to Koen van de Sande

comment:30 Changed 11 years ago by jdemeyer

  • Description modified (diff)
Note: See TracTickets for help on using tickets.