Opened 7 years ago

Last modified 5 years ago

#12738 needs_info defect

Singular fails to build with LTO

Reported by: SimonKing Owned by: tbd
Priority: major Milestone: sage-6.4
Component: packages: standard Keywords: singular lto gcc
Cc: leif, ncohen, malb Merged in:
Authors: Reviewers:
Report Upstream: Reported upstream. No feedback yet. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description

I took sage-5.0.beta9, replaced the ppl package by the one from #12672, added the cloog-ppl package from #12666, replaced the glpk package by the one from #12703, added a new libelf spkg, added the mpc and gcc spkgs from #12369 and modified the gcc spkg such that the gcc is built with support of loop optimization (graphite) and lto.

Then, I tried to build Sage with rather high optimization:

king@mpc622:/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9/spkg$ echo $CFLAGS
-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing -flto
king@mpc622:/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9/spkg$ echo $CXXFLAGS
-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing -flto
king@mpc622:/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9/spkg$ echo $LDFLAGS
-flto

Moreover, I attempted to build parallely (four threads).

However, Singular failed to build. Note that dropping -flto would enable to build Singular. Hence, the problem really seems to be -flto, whereas make -j4 and -march=native and -floop* seems to be fine.

It seems to me that LDFLAGS is not correctly passed. Namely, in the install log, I read

g++ -O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing -flto -O2 -g -fPIC -pipe  -fno-implicit-templates -I. -I.. -I/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.bet
a9/local  -I/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9/local/include -I/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9/local/include -I/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta9/local/include   -DNDEBUG -DOM_NDEBUG -Dx86_64_Linux -DHAVE_CONFIG_H -fpic -DPIC -Dp_Procs_FieldQ -c p_Procs_Lib.cc -o p_Procs_Lib_FieldQ.dl_o

but

ld -shared -o p_Procs_FieldQ.sog p_Procs_Lib_FieldQ.dl_og
...
ld -shared -o p_Procs_FieldQ.so p_Procs_Lib_FieldQ.dl_o

Moreover, I see that libfac/Makefile.dist defines LDFLAGS empty

  LDFLAGS=

and defines

  LD = g++ $(ALLFLAGS)

where LDFLAGS does not appear in the definition of ALLFLAGS.

Change History (29)

comment:1 Changed 7 years ago by SimonKing

  • Cc malb added

Cc to Martin, because I suppose he knows a lot about the Singular spkg...

comment:2 Changed 7 years ago by SimonKing

Leif said at #12703: "Btw., parts of Singular still seem to disrespect CXXFLAGS, i.e., some C++ sources are compiled only using CPPFLAGS." However, adding CPPFLAGS did not help.

comment:3 Changed 7 years ago by SimonKing

  • Report Upstream changed from N/A to Reported upstream. Little or no feedback.

comment:4 Changed 7 years ago by SimonKing

See #12741: There is a similar problem with R.

comment:5 follow-up: Changed 7 years ago by leif

To me this all smells more like a problem with the GCC setup (i.e., the Sage GCC spkg), as I don't have any problems compiling Sage 5.0beta{8,9} (with some spkgs replaced, see sage-release) with GCC 4.6.3 and -flto (as well as Graphite loop optimization) enabled.

comment:6 in reply to: ↑ 5 Changed 7 years ago by SimonKing

Replying to leif:

To me this all smells more like a problem with the GCC setup (i.e., the Sage GCC spkg)

Then it would probably be fair to say that it is a problem with how I modified the GCC spkg, so that it supports graphite and lto in the first place. The diff with respect to the spkg from #12369 is

  • spkg-install

    diff --git a/spkg-install b/spkg-install
    a b  
    8282../src/configure \
    8383    --prefix="$SAGE_LOCAL" \
    8484    --with-local-prefix="$SAGE_LOCAL" \
     85    --with-cloog="$SAGE_LOCAL" --with-ppl="$SAGE_LOCAL" \
     86    --enable-lto \
    8587    --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" --with-mpc="$SAGE_LOCAL" \
    8688    --with-system-zlib \
    8789    --disable-multilib \

And of course I also added a libelf spkg (not published yet, but it is just upstream sources) and a cloog-ppl spkg (#12666).

Just for the record: Singular does build with the gcc from the modified spkg, as soon as one drops -flto, while -floop* is fine.

comment:7 follow-up: Changed 7 years ago by leif

Furthermore (this is Sage 5.0.beta10, which doesn't matter here, GCC 4.6.3, but with binutils 2.22, and LDFLAGS="-flto -fuse-linker-plugin"):

lto1: internal compiler error: in lto_varpool_replace_node, at lto-symtab.c:304
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
lto-wrapper: g++-4.6.3 returned 1 exit status
/usr/local/bin/ld: lto-wrapper failed
collect2: ld returned 1 exit status
make[4]: *** [Singular] Error 1
make[4]: Leaving directory `/data/leif/Sage/sage-5.0.beta10-gcc-4.6.3-LTO/spkg/build/singular-3-1-3-3.p6/src/Singular'
make[3]: *** [install] Error 1
make[3]: Leaving directory `/data/leif/Sage/sage-5.0.beta10-gcc-4.6.3-LTO/spkg/build/singular-3-1-3-3.p6/src'
make[2]: *** [/data/leif/Sage/sage-5.0.beta10-gcc-4.6.3-LTO/local/bin/Singular-3-1-3] Error 2
make[2]: Leaving directory `/data/leif/Sage/sage-5.0.beta10-gcc-4.6.3-LTO/spkg/build/singular-3-1-3-3.p6/src'
Unable to build Singular.

But this might simply be a hidden "no space left on device" error, as /tmp is currently on my almost full root partition -- will retry later...

comment:8 in reply to: ↑ 7 Changed 7 years ago by leif

Ooops, somehow forgot to hit the submit button...

Replying to leif:

Furthermore (this is Sage 5.0.beta10, which doesn't matter here, GCC 4.6.3, but with binutils 2.22, and LDFLAGS="-flto -fuse-linker-plugin"):

lto1: internal compiler error: in lto_varpool_replace_node, at lto-symtab.c:304
...

But this might simply be a hidden "no space left on device" error, as /tmp is currently on my almost full root partition -- will retry later...

Nope, it was not. But I can currently get around that bug by omitting -fuse-linker-plugin from LDFLAGS.

comment:9 Changed 7 years ago by SimonKing

Hi! With the new setting (i.e., with a binutils-2.22.spkg), I seemingly get exactly the same problem as you:

iplib.cc:68:14: Warnung: Typ von »yylp_errlist« passt nicht zur ursprünglichen Deklaration [standardmäßig aktiviert]
libparse.l:63:13: Anmerkung: vorher hier deklariert
lto1: interner Compiler-Fehler: in lto_varpool_replace_node, bei lto-symtab.c:304
Bitte senden Sie einen vollständigen Fehlerbericht auf Englisch ein;
bearbeiten Sie die Quellen zunächst mit einem Präprozessor, wenn es
dienlich ist.
Fehler in der deutschen Übersetzung sind an translation-team-de@lists.sourceforge.net zu melden.

Gehen Sie gemäß den Hinweisen in <http://gcc.gnu.org/bugs.html> vor.
lto-wrapper: g++ gab Ende-Status 1 zurück
/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta10/local/lib/gcc/x86_64-unknown-linux-gnu/4.6.3/../../../../x86_64-unknown-linux-gnu/bin/ld: lto-wrapper failed
collect2: ld gab 1 als Ende-Status zurück
make[4]: *** [Singular] Fehler 1
make[4]: Leaving directory `/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta10/spkg/build/singular-3-1-3-3.p6/src/Singular'
make[3]: *** [install] Fehler 1
make[3]: Leaving directory `/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta10/spkg/build/singular-3-1-3-3.p6/src'
make[2]: *** [/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta10/local/bin/Singular-3-1-3] Fehler 2
make[2]: Leaving directory `/mnt/local/king/SAGE/testinggcc/lto/sage-5.0.beta10/spkg/build/singular-3-1-3-3.p6/src'
Unable to build Singular.

I am now trying to drop -fuse-linker-plugin

comment:10 follow-up: Changed 7 years ago by SimonKing

I tried to drop -fuse-linker-plugin from LDFLAGS, but it didn't help. I tried to drop it from both LDFLAGS and C(XX)FLAGS, but it didn't help either. The install log should contain all three failing sessions, I don't know if there is any visible difference between the three.

comment:11 in reply to: ↑ 10 Changed 7 years ago by leif

Replying to SimonKing:

I tried to drop -fuse-linker-plugin from LDFLAGS, but it didn't help.

That only worked for me because I had built GCC 4.6.3 with older binutils, in particular GNU ld 2.20.*.

(If a plugin-capable linker is present during the build of GCC, you later don't have to specify -fuse-linker-plugin at all.)

comment:12 Changed 7 years ago by SimonKing

In the ticket description, I mentioned that the user-defined LDFLAGS is overridden in libfac/Makefile.dist. So, I commented out the line in which that happens. Keeping my fingers crossed that it solves the problem...

comment:13 follow-up: Changed 7 years ago by SimonKing

Just as I wrote the previous comment, building Singular failed again, with the same error in lto_varpool_replace_node, bei lto-symtab.c:304.

comment:14 in reply to: ↑ 13 ; follow-up: Changed 7 years ago by leif

Replying to SimonKing:

Just as I wrote the previous comment, building Singular failed again, with the same error in lto_varpool_replace_node, bei lto-symtab.c:304.

As expected... :-(

comment:15 in reply to: ↑ 14 ; follow-up: Changed 7 years ago by SimonKing

Replying to leif:

Replying to SimonKing:

Just as I wrote the previous comment, building Singular failed again, with the same error in lto_varpool_replace_node, bei lto-symtab.c:304.

As expected... :-(

Why expected? I thought that the problem is that Singular does not take LDFLAGS into account!

comment:16 in reply to: ↑ 15 ; follow-up: Changed 7 years ago by leif

Replying to SimonKing:

Replying to leif:

Replying to SimonKing:

Just as I wrote the previous comment, building Singular failed again, with the same error in lto_varpool_replace_node, bei lto-symtab.c:304.

As expected... :-(

Why expected? I thought that the problem is that Singular does not take LDFLAGS into account!

Because dropping -flto shouldn't have any effect except that LTO simply won't happen.

Of course there are some other effects, e.g. regarding the size of the object files (and of course compile time), depending on where you drop it, if not everywhere.

-flto during compilation just prepares the generated object files for LTO (i.e., adds the extra information needed later); -flto during the link phase then triggers LTO. If some object files or libraries lack the special sections needed, they're just excluded from LTO, as there's no information available on them. This is very similar to compiling with -g, and later using the debugger.

comment:17 in reply to: ↑ 16 Changed 7 years ago by SimonKing

Replying to leif:

Replying to SimonKing:

Replying to leif:

Replying to SimonKing:

Just as I wrote the previous comment, building Singular failed again, with the same error in lto_varpool_replace_node, bei lto-symtab.c:304.

As expected... :-(

Why expected? I thought that the problem is that Singular does not take LDFLAGS into account!

Because dropping -flto shouldn't have any effect except that LTO simply won't happen.

I was not talking about "dropping -flto" but about modifying the Singular sources such that LDFLAGS is not overridden.

comment:18 follow-up: Changed 7 years ago by leif

To be more precise:

  • Singular dropping / ignoring *FLAGS (in this case, -flto, or LDFLAGS from compiler link commands) isn't nice, likewise ignoring CXXFLAGS in some compile commands, and should probably get fixed and reported upstream, but should not break the build (at least not if only -flto isn't passed to the compiler).
  • The error we get with LTO (when enabled during the link phase), which makes the build fail, is a GCC bug, and should get reported GCC upstream.
Last edited 7 years ago by leif (previous) (diff)

comment:19 in reply to: ↑ 18 Changed 7 years ago by SimonKing

Replying to leif:

  • The error we get with LTO (when enabled during the link phase), which makes the build fail, is a GCC bug, and should get reported GCC upstream.

OK. Can you do that? Because apparently the gcc spkg was automatically building a "localized" version, such that the error messages are German, but they ask for a report in English.

comment:20 follow-up: Changed 7 years ago by leif

Just successfully built Singular (3-1-3-3.p6) with GCC 4.7.0 and LTO enabled. :-)

comment:21 in reply to: ↑ 20 ; follow-up: Changed 7 years ago by SimonKing

Replying to leif:

Just successfully built Singular (3-1-3-3.p6) with GCC 4.7.0 and LTO enabled. :-)

Congrats. I just failed to build Maxima with GCC 4.6.3 and LTO enabled. Interestingly, maxima has not been a problem with -flto (if I remember correctly), but now I had -flto -fuse-linker-plugin.

comment:22 in reply to: ↑ 21 ; follow-up: Changed 7 years ago by leif

Replying to SimonKing:

Replying to leif:

Just successfully built Singular (3-1-3-3.p6) with GCC 4.7.0 and LTO enabled. :-)

Congrats. I just failed to build Maxima with GCC 4.6.3 and LTO enabled. Interestingly, maxima has not been a problem with -flto (if I remember correctly), but now I had -flto -fuse-linker-plugin.

See my comment at #12741. I've also posted to sage-release, and do have an spkg fixing this: :-)

maxima-5.26.0.p1.spkg (Changes not yet committed due to lacking ticket number, haven't yet opened one for that.)

comment:23 in reply to: ↑ 22 Changed 7 years ago by leif

Replying to leif:

Replying to SimonKing:

Replying to leif:

Just successfully built Singular (3-1-3-3.p6) with GCC 4.7.0 and LTO enabled. :-)

Congrats. I just failed to build Maxima with GCC 4.6.3 and LTO enabled. Interestingly, maxima has not been a problem with -flto (if I remember correctly), but now I had -flto -fuse-linker-plugin.

See my comment at #12741. I've also posted to sage-release, and do have an spkg fixing this: :-)

maxima-5.26.0.p1.spkg (Changes not yet committed due to lacking ticket number, haven't yet opened one for that.)

Oh, forgot: This is now #12759 (changes committed, needing review :P ).

comment:24 Changed 7 years ago by roed

  • Report Upstream changed from Reported upstream. Little or no feedback. to Reported upstream. No feedback yet.

comment:25 Changed 6 years ago by leif

  • Status changed from new to needs_info

Is this still an issue (with Singular 3-1-5, newer GCC versions...)?

comment:26 Changed 6 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:27 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:28 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:29 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4
Note: See TracTickets for help on using tickets.