Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#12369 closed task (fixed)

Add a gcc package

Reported by: jdemeyer Owned by:
Priority: major Milestone: sage-5.0
Component: packages: standard Keywords:
Cc: vbraun Merged in: sage-5.0.beta13
Authors: Jeroen Demeyer Reviewers: Simon King
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638, #12714, #12647, #12739, #12112, #12631 Stopgaps:

Description (last modified by jdemeyer)

The aim is to add a GCC package (GNU compiler collection) to Sage with compilers for C, C++ and Fortran. We don't always build it, we would install it by default only on systems where this is needed. This would replace the fortran package.

spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/gcc-4.6.3.spkg

Apply:

  1. 12369_gcc_root.patch and 12369_gcc_root_part2.patch to the SAGE_ROOT repository.
  2. 12369_scripts_hgignore.patch to the SCRIPTS repository.
  3. 12369_doc.patch to the Sage library.

See also:

  1. #12782: When building GCC, build MPIR without the C++ interface (superseded by #11616).

Attachments (4)

12369_scripts_hgignore.patch (602 bytes) - added by jdemeyer 7 years ago.
12369_doc.patch (6.5 KB) - added by jdemeyer 7 years ago.
12369_gcc_root.patch (10.9 KB) - added by jdemeyer 7 years ago.
12369_gcc_root_part2.patch (2.8 KB) - added by jdemeyer 7 years ago.

Download all attachments as: .zip

Change History (248)

comment:1 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:2 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Summary changed from Create a gcc package to Add a gcc package

comment:3 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12366, #12367, #12368 to #12366, #12367, #12368, #10492

comment:4 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:5 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12366, #12367, #12368, #10492 to #11073, #12366, #12367, #12368, #10492, #12405

comment:6 Changed 7 years ago by jdemeyer

  • Type changed from enhancement to task

comment:7 follow-up: Changed 7 years ago by kcrisman

# Install gcc if the system-provided one is older than gcc-4.4. 

But that is all Macs. This seems overkill to me; why not only do this on Lion?

Or would this eliminate having Xcode as a prerequisite to building Sage on OS X in general (even for those for which XCode is free, though sometimes annoying to get)? That might be a nice move. I assume this is a prebuilt gcc like our current Fortran package?

Could the message "installing GCC because..." occur several times here?

comment:8 Changed 7 years ago by vbraun

Updated spkg at http://www.stp.dias.ie/~vbraun/Sage/spkg/gcc-4.6.2.spkg

It turns out to be tricky to convince gcc to use lib64 for glibc but put its own libraries into lib. But I think its safe to just move the generated shared libraries to SAGE_LOCAL/lib. Also, I updated to the newest gcc, install in single-thread, and disabled multilib.

comment:9 in reply to: ↑ 7 ; follow-up: Changed 7 years ago by jdemeyer

First of all: this spkg is all pretty-much proof-of-concept. I haven't even managed to install Sage from source with this spkg.

Replying to kcrisman:

# Install gcc if the system-provided one is older than gcc-4.4. 

But that is all Macs. This seems overkill to me; why not only do this on Lion?

Maybe you are right, but my idea was to install this gcc spkg if the gcc we ship is more recent than the one the user has. I think many people would benefit from using a more recent compiler.

Or would this eliminate having Xcode as a prerequisite to building Sage on OS X in general (even for those for which XCode is free, though sometimes annoying to get)? That might be a nice move. I assume this is a prebuilt gcc like our current Fortran package?

No, this is a gcc from source. I certainly don't want to include gcc binaries for every platform, that would increase the size way too much.

Could the message "installing GCC because..." occur several times here?

Sure.

comment:10 in reply to: ↑ 9 ; follow-up: Changed 7 years ago by kcrisman

First of all: this spkg is all pretty-much proof-of-concept. I haven't even managed to install Sage from source with this spkg.

Of course; I'm just asking early questions.

But that is all Macs. This seems overkill to me; why not only do this on Lion?

Maybe you are right, but my idea was to install this gcc spkg if the gcc we ship is more recent than the one the user has. I think many people would benefit from using a more recent compiler.

Possibly. I guess what I'm wondering is how one would build this gcc if one didn't have gcc... but this may betray my ignorance. Anyway, the vast majority of Mac users will not have a way to build this gcc if they need gcc until they download/install (from a disk which they've probably thrown out, as William pointed out on sage-devel) XCode anyway, in which case we might as well only worry about Lion anyway.

Or would this eliminate having Xcode as a prerequisite to building Sage on OS X in general (even for those for which XCode is free, though sometimes annoying to get)? That might be a nice move. I assume this is a prebuilt gcc like our current Fortran package?

No, this is a gcc from source. I certainly don't want to include gcc binaries for every platform, that would increase the size way too much.

That's what I figured, but just wanted to clarify.

Could the message "installing GCC because..." occur several times here?

Sure.

Ok. I suppose it doesn't really matter, but could be confusing/alarming.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 7 years ago by jdemeyer

Replying to kcrisman:

Possibly. I guess what I'm wondering is how one would build this gcc if one didn't have gcc... but this may betray my ignorance.

My hope is that the non-gcc C compiler shipped with XCode 4 might be able to build GCC.

Anyway, the vast majority of Mac users will not have a way to build this gcc if they need gcc until they download/install

That "vast majority" of users will be happy with a binary distribution of Sage (which could actually contain GCC if I manage to make this spkg work).

comment:12 in reply to: ↑ 11 Changed 7 years ago by kcrisman

Replying to jdemeyer:

Replying to kcrisman:

Possibly. I guess what I'm wondering is how one would build this gcc if one didn't have gcc... but this may betray my ignorance.

My hope is that the non-gcc C compiler shipped with XCode 4 might be able to build GCC.

Interesting! Now I see the point of all this.

Anyway, the vast majority of Mac users will not have a way to build this gcc if they need gcc until they download/install

That "vast majority" of users will be happy with a binary distribution of Sage

Good point - until they need to change any code. But this

(which could actually contain GCC if I manage to make this spkg work).

Interesting, as I said. I wonder if this would work:

  • Sage compiled with XCode 4, including compiling gcc
  • Sage bdisted from this machine
  • Sage downloaded in that bdist to a machine WITHOUT Xcode (or MacPorts?, or any other compilers)
  • gcc included with Sage allows full Sage building fun

Good luck!

comment:13 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #12366, #12367, #12368, #10492, #12405 to #11073, #12366, #12367, #12368, #10492, #12405, #12416

comment:14 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #12366, #12367, #12368, #10492, #12405, #12416 to #11073, #12366, #12367, #12368, #10492, #12405, #12416, #12422

comment:15 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #12366, #12367, #12368, #10492, #12405, #12416, #12422 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423

comment:16 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425

comment:17 follow-ups: Changed 7 years ago by ohanar

I haven't looked at the spkg yet (which is probably old anyway), but looking at the root patch that you wrote, I have a few of requests:

  1. I agree with Karl. Do not install gcc if gcc is new enough (>=4.0.1), this will cause every mac developer to have to build gcc upon building sage (significantly increasing the build time). I'm not a mac user, but I know that if I had to do this on my system, I would be annoyed.
  1. Assume the user knows what they are doing it CC is set (like you do for CXX and SAGE_FORTRAN). While not a very common use case, it would be helpful to not have to go hack the installation file if you were trying to port sage to another compiler (like I am right now).
  1. (You might be doing this already) If fortran is the only language that needs to be built (such as on macs), please do not build gcc and g++.

And a couple comments:

  1. You assume that CC supports -dumpversion when determining if you should upgrade gcc. I'm fairly certain this is not an option supported by all compilers. Also, the value you would get would be meaningless anyway, since it wouldn't be a gcc version anyway (well, unless it is clang).
  1. For OBJC and OBJCXX, I would point to /usr/bin/cc and /usr/bin/cpp instead. If/when apple stops distributing gcc/g++, then these will continue to point to valid compilers.

In response to Karl about a replacement for Xcode (or tools on any platform). This does not provide one, even in binary distributions of sage since we need binutils and make, neither of which this would provide (and neither of which are on OSX without Xcode).

comment:18 in reply to: ↑ 17 Changed 7 years ago by kcrisman

In response to Karl about a replacement for Xcode (or tools on any platform). This does not provide one, even in binary distributions of sage since we need binutils and make, neither of which this would provide (and neither of which are on OSX without Xcode).

Right, that makes sense. I didn't know exactly what comes from Xcode and what is just pre-included.

comment:19 follow-up: Changed 7 years ago by jhpalmieri

I've been wondering for a while if we should change Sage to use autoconf. (When I say "we", I don't actually mean me, since I don't know anything about writing configure scripts, nor do I want to learn.) This could replace spkg-install (especially with the modifications at #10492 that create a Makefile on the fly). Does this make sense as an enhancement? Should I poll sage-devel about it, and then open a ticket if the response is positive?

comment:20 in reply to: ↑ 19 Changed 7 years ago by ohanar

Replying to jhpalmieri:

I've been wondering for a while if we should change Sage to use autoconf. (When I say "we", I don't actually mean me, since I don't know anything about writing configure scripts, nor do I want to learn.) This could replace spkg-install (especially with the modifications at #10492 that create a Makefile on the fly). Does this make sense as an enhancement?

I don't know how much additional functionality it would add, but I would argue for it since it is more unix user friendly than our current custom setup (SAGE_ATLAS_LIB is not as user friendly as a fairly standard --with-system-atlas argument for configure). My guess is that it would be a fairly large endeavour to implement a first version of such a change (I don't know autoconf, so I'm just guessing here), although it would hopefully be easier to maintain than spkg-install.

Should I poll sage-devel about it, and then open a ticket if the response is positive?

I don't think it could hurt. Also, that would be a better place to discuss it than on this ticket.

comment:21 in reply to: ↑ 17 Changed 7 years ago by ohanar

Replying to ohanar:

  1. Assume the user knows what they are doing it CC is set (like you do for CXX and SAGE_FORTRAN). While not a very common use case, it would be helpful to not have to go hack the installation file if you were trying to port sage to another compiler (like I am right now).

Actually, thinking a bit more about it, I think it would be better if we required SAGE_PORT to be set if CC and/or CXX is set.

comment:22 in reply to: ↑ 17 ; follow-ups: Changed 7 years ago by jdemeyer

Replying to ohanar:

I haven't looked at the spkg yet (which is probably old anyway), but looking at the root patch that you wrote, I have a few of requests:

  1. I agree with Karl. Do not install gcc if gcc is new enough (>=4.0.1), this will cause every mac developer to have to build gcc upon building sage (significantly increasing the build time). I'm not a mac user, but I know that if I had to do this on my system, I would be annoyed.

Well, we do need to build Fortran on a Mac (I'm assuming that the fortran spkg will go away). And Sage seems to require that the gcc and gfortran versions are the same. It is true that the build time of gcc is significant:

real    21m4.603s
user    15m56.183s
sys     4m26.492s
Successfully installed gcc-4.4.6

(on bsd.math, single-threaded), but it's not that bad. I think the "plus" of having a working Sage compiled with optimizations (as opposed to using XCode 4 with many packages compiled with -O0) outweighs the "minus" of 25 minutes longer build time.

  1. Assume the user knows what they are doing it CC is set (like you do for CXX and SAGE_FORTRAN). While not a very common use case, it would be helpful to not have to go hack the installation file if you were trying to port sage to another compiler (like I am right now).

Well, I was thinking anyway of adding an environment variable (or future ./configure option) to either force installation of gcc, or disable installation of gcc. Don't forget that GCC has a few dependencies, and one might set CC to build those dependencies, but still want GCC to be installed. Anyway, I don't care much about this.

  1. (You might be doing this already) If fortran is the only language that needs to be built (such as on macs), please do not build gcc and g++.

I thought gcc and gfortran needed to be the same versions?

And a couple comments:

  1. You assume that CC supports -dumpversion when determining if you should upgrade gcc. I'm fairly certain this is not an option supported by all compilers. Also, the value you would get would be meaningless anyway, since it wouldn't be a gcc version anyway (well, unless it is clang).

Agreed.

  1. For OBJC and OBJCXX, I would point to /usr/bin/cc and /usr/bin/cpp instead. If/when apple stops distributing gcc/g++, then these will continue to point to valid compilers.

Agreed (modulo the fact that you mean c++ instead of the preprocessor cpp).

In response to Karl about a replacement for Xcode (or tools on any platform). This does not provide one, even in binary distributions of sage since we need binutils and make, neither of which this would provide (and neither of which are on OSX without Xcode).

True...

comment:23 in reply to: ↑ 22 ; follow-up: Changed 7 years ago by ohanar

Replying to jdemeyer:

Well, we do need to build Fortran on a Mac (I'm assuming that the fortran spkg will go away).

Agreed, which is why I would like only fortran to be built if it is the only language needed.

I think the "plus" of having a working Sage compiled with optimizations (as opposed to using XCode 4 with many packages compiled with -O0) outweighs the "minus" of 25 minutes longer build time.

I think this is something that should be brought to the listserv. Once I get a clang port up and running (which shouldn't be too bad on osx, since atlas looks like it will be the only package that is going to be hell to port), this can be resolved by changing the default compiler on osx to clang.

  1. (You might be doing this already) If fortran is the only language that needs to be built (such as on macs), please do not build gcc and g++.

I thought gcc and gfortran needed to be the same versions?

I'm not sure about this, I'm using a mixture of clang (which dumps its version as 4.2.1) and gfortran-4.5.3, and occasionally gcc-4.2.4 and g++-4.2.4 to build sage, and currently the entire test suite is passing.

comment:24 in reply to: ↑ 23 ; follow-up: Changed 7 years ago by jdemeyer

Replying to ohanar:

Agreed, which is why I would like only fortran to be built if it is the only language needed.

Is build time your only concern here? Because if you need to build Fortran, you most likely need to build a stage 1 C compiler anyway.

comment:25 follow-up: Changed 7 years ago by jdemeyer

I would like to add that my goal is beyond OS X 10.7 but more generally all systems with bad compilers. That's also why I went for the more stable gcc-4.4.6 as opposed to the more recent (and therefore buggy) gcc-4.6.2.

comment:26 in reply to: ↑ 22 Changed 7 years ago by jhpalmieri

Replying to jdemeyer:

I think the "plus" of having a working Sage compiled with optimizations (as opposed to using XCode 4 with many packages compiled with -O0) outweighs the "minus" of 25 minutes longer build time.

To be fair, the current situation requires only one package (symmetrica) to be compiled without optimization, and then only if the extra gcc-4.2 package is not present.

comment:27 in reply to: ↑ 25 ; follow-up: Changed 7 years ago by vbraun

Replying to jdemeyer:

I would like to add that my goal is beyond OS X 10.7 but more generally all systems with bad compilers.

I have compiled gcc too often on clusters that have only old versions installed.

Personally, gcc-4.6.2 seems to work very well. I'm not saying we should jump on every .0, mind you ;-)

comment:28 in reply to: ↑ 27 ; follow-ups: Changed 7 years ago by ohanar

Replying to jdemeyer:

Is build time your only concern here? Because if you need to build Fortran, you most likely need to build a stage 1 C compiler anyway.

Yes, I guess that is true, there is probably minimal savings by not building c++. I guess then as long as we don't add this to the critical path for most systems it should be fine. Do you have a current spkg that I could look at? I would like to see if there is any quick ways to reduce build time (e.g. disabling multilib on 64-bit systems).

Replying to vbraun:

Personally, gcc-4.6.2 seems to work very well. I'm not saying we should jump on every .0, mind you ;-)

Same here (plus it compiles with clang, unlike 4.4.6, from my experimenting), however it adds mpc as a dependency.

comment:29 in reply to: ↑ 28 Changed 7 years ago by jdemeyer

  • Description modified (diff)

Replying to ohanar:

Yes, I guess that is true, there is probably minimal savings by not building c++. I guess then as long as we don't add this to the critical path for most systems it should be fine. Do you have a current spkg that I could look at?

The spkg on this ticket is current.

comment:30 in reply to: ↑ 28 ; follow-up: Changed 7 years ago by jdemeyer

Replying to ohanar:

I guess then as long as we don't add this to the critical path for most systems it should be fine.

If it gets built, it will be on the critical path (since it effectively blocks the build). However, that's not a problem since the GCC build paralellizes well: I managed to build it in 4 minutes using "make -j24":

real    4m7.170s
user    20m58.059s
sys     4m29.269s
Successfully installed gcc-4.4.6

comment:31 in reply to: ↑ 28 Changed 7 years ago by vbraun

Replying to ohanar:

any quick ways to reduce build time (e.g. disabling multilib on 64-bit systems).

My gcc-4.6.2 spkg disables multilib already

Same here (plus it compiles with clang, unlike 4.4.6, from my experimenting), however it adds mpc as a dependency.

For the record, mpc is contained in the gcc-4.6.2 spkg.

comment:32 Changed 7 years ago by ohanar

I posted a patch for the gcc-4.4.6 spkg, it disables multilib, and fixes clang build (it also disables warnings, since gcc throws so many of them, I actually noticed a small speed up from disabling them).

comment:33 in reply to: ↑ 30 Changed 7 years ago by ohanar

Replying to jdemeyer:

If it gets built, it will be on the critical path (since it effectively blocks the build).

But you also need to consider that we are building the mpir and mpfr twice as well as adding one copy of those builds to the critical path. I still think there is value to distinguishing between when gcc is needed, and when only fortran is needed. In the latter case, this should actually negligibly shorten the critical path (with the removal of the fortran spkg) plus mpir+mpfr don't have to be rebuilt.

My concern is that sage already takes a good while to build, and this spkg adds a fairly significant time to that for only a few feature gains and relatively minor performance gains (compared to implementing new/better algorithms).

comment:34 follow-up: Changed 7 years ago by jhpalmieri

By the way, I know you weren't using the word "path" this way, but regarding the path, note for example #10822: on some platforms, apparently, we may not be able to guarantee that SAGE_ROOT/local/bin comes first in the path. So in the patch for the root repo, would it better to replace

CC=gcc 

by

CC="$SAGE_LOCAL/bin/gcc"

on line 418 of sage-env, and similarly for CXX?

comment:35 Changed 7 years ago by jdemeyer

My personal opinion on the build time is the following: I don't care much about it. I much rather have a Sage which works and is fast than a Sage which builds fast.

There is no need to rebuild mpir and mpfr, except for speed gains (the newer GCC's might have better support for newer processors).

comment:36 in reply to: ↑ 34 Changed 7 years ago by jdemeyer

Replying to jhpalmieri:

CC="$SAGE_LOCAL/bin/gcc"

on line 418 of sage-env, and similarly for CXX?

Unfortunately, this would break if "$SAGE_LOCAL" contains spaces or other weird characters since $CC is usually used unquoted.

I believe the zsh "sage -sh" issues are beyond gcc (it totally breaks everything) and should be fixed on #10822, not here.

comment:37 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425, #12456, #12457

comment:38 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425, #12456, #12457 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425, #12456, #12457, #12459

comment:39 follow-up: Changed 7 years ago by jdemeyer

Why is -std=c89 needed?

comment:40 in reply to: ↑ 39 Changed 7 years ago by ohanar

Replying to jdemeyer:

Why is -std=c89 needed?

-std=gnu89 is not needed for gcc, but is needed for clang, and since I'm working on porting sage to clang, I would rather have this package work on it out of the gate (rather than make a ticket for something that could just be fixed now). Alternatively, we could use a more recent version of gcc in the spkg, since, at the very least, gcc-4.6.2 uses gnu99, the default standard for clang.

I would also suggest using making cc and c++ be the default compiler when building gcc and its dependencies. Like OBJC and OBJCXX, this is more future thinking when it comes to apple platforms (plus more forward thinking for any platform with an unsupported compiler, from which we can build gcc).

comment:41 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425, #12456, #12457, #12459 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425, #12456, #12457

comment:42 Changed 7 years ago by jdemeyer

Volker, I just had a quick look at your spkg:

  1. --enable-languages can be omitted if you ship only the source of the C, C++ and Fortran compiler (as I would do)
  2. --libdir doesn't matter since GCC doesn't respect it (don't know why).
  3. --with-ppl doesn't matter either since PPL is only used in combination with CLooG which we don't ship. I agree that it is cool that GCC can use PPL though.
  4. There is no need to install with "-j1".

comment:43 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:44 follow-up: Changed 7 years ago by jhpalmieri

Wow, compiling this spkg with SAGE_CHECK=yes is pretty painful: on my Mac (which has only two cores), it took 89 minutes.

comment:45 in reply to: ↑ 44 Changed 7 years ago by jdemeyer

Replying to jhpalmieri:

Wow, compiling this spkg with SAGE_CHECK=yes is pretty painful: on my Mac (which has only two cores), it took 89 minutes.

Note that compiling GCC with SAGE_CHECK=yes will actually make every spkg build slower, because it enables run-time checks inside GCC.

I think this is not a problem and fits with the spirit of SAGE_CHECK=yes.

comment:46 follow-up: Changed 7 years ago by vbraun

Yes, the gcc testsuite is pretty big!

comment:47 in reply to: ↑ 46 Changed 7 years ago by jdemeyer

Replying to vbraun:

Yes, the gcc testsuite is pretty big!

I'm not actually shipping nor running the gcc testsuite. It would add 6MB to the spkg size.

comment:48 Changed 7 years ago by jhpalmieri

I've built the "testing release" on OS X Lion with SAGE_CHECK=yes. The results:

  • python failed self-tests, as always.
  • ppl failed self-tests. I haven't seen this before. Near the end of the log file:
    make  check-TESTS
    PASS: ascii_dump_load1
    /bin/sh: line 1: 22675 Abort trap: 6           ${dir}$tst
    FAIL: exceptions1
    PASS: pipproblem1
    PASS: pipproblem2
    PASS: pipproblem3
    
  • cvxopt failed self-tests. This one can be tricky to notice, since if self-tests fail, spkg-check still exits with a return code of 0, so the build proceeds. But there are errors in the log file. See #12011.
  • zn_poly failed self-tests. This seems to happen frequently on Lion on a heavily loaded system, regardless of the compiler. Typically it fails if I'm executing make on Sage with MAKE="make -j2" so that other compilations are going on at the same time, but it often passes if I just do ./sage -i spkg/standard/zn_poly....
  • gsl and pari passed self-tests (they fail with the default compiler -- #12319 and #12315). The relevant file (schur.pxi) in the Sage library passes doctests, so symmetrica seems to work (#12424). This is all good news.
  • the testing release already includes the fix from #11967 (removing lib/python/config/libpython2.7.a). Do you know if that's necessary when building this way? I guess since Sage seems to function without this file, it doesn't hurt to delete it.

Doctests haven't finished yet.

comment:49 Changed 7 years ago by jdemeyer

PARI doesn't work with gcc-4.6.2 on OS X 10.6 and 10.7 systems. Luckily, the new PARI (#12363) fixes this.

comment:50 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425, #12456, #12457 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425, #12456, #12457, #12363

comment:51 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:52 Changed 7 years ago by jhpalmieri

More data: with SAGE_CHECK=yes and testing release 1 (gcc-4.4.6), I bypassed the failing packages by installing them without running self-tests. At the end there were a number of doctest failures:

	sage -t  --long -force_lib devel/sage/sage/symbolic/expression.pyx # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/matrix/matrix2.pyx # 4 doctests failed
	sage -t  --long -force_lib devel/sage/sage/rings/polynomial/polynomial_element.pyx # 1 doctests failed
	sage -t  --long -force_lib devel/sage/sage/matrix/matrix_symbolic_dense.pyx # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/calculus/wester.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/calculus/functional.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/symbolic/integration/integral.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/calculus/tests.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/functions/trig.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/functions/hyperbolic.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/symbolic/function.pyx # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/matrix/matrix_double_dense.pyx # 4 doctests failed
	sage -t  --long -force_lib devel/sage/sage/functions/min_max.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/interfaces/maxima_lib.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/libs/ppl.pyx # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/tensor/differential_form_element.py # Killed/crashed
	sage -t  --long -force_lib devel/sage/sage/tensor/differential_forms.py # Killed/crashed

The ones not marked "Killed/crashed" are the familiar ones from #11881.

Next, I tried to compile with SAGE_CHECK unset, and gd failed to build:

gcc -fPIC -g -I/Applications/sage_builds/GCC-no-check/sage-5.0.beta1-gcc/local/include -I/Applications/sage_builds/GCC-no-check/sage-5.0.beta1-gcc/local/include/freetype2/ -o .libs/gdparttopng gdparttopng.o  -L/Applications/sage_builds/GCC-no-check/sage-5.0.beta1-gcc/local/lib ./.libs/libgd.dylib -liconv /Applications/sage_builds/GCC-no-check/sage-5.0.beta1-gcc/local/lib/libfreetype.dylib /Applications/sage_builds/GCC-no-check/sage-5.0.beta1-gcc/local/lib/libpng12.dylib -lz
Undefined symbols for architecture x86_64:
  "_gdImageCreateFromGd2Part", referenced from:
      _main in gdparttopng.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
make[4]: *** [gdparttopng] Error 1
make[3]: *** [all-recursive] Error 1
make[2]: *** [all] Error 2
Error building gd.

I just did ./sage -i spkg/standard/gd-..., and it seems to have built fine. I don't know what went wrong the first time, and I haven't had time to continue the build or to start it over.

Next, I worked with testing release 2 (gcc-4.6.2). As jdemeyer has noted, pari-2.5.0 won't build, so after verifying that (actually, it claims to have built, but it failed all (!) self-tests), I dropped in the new Pari spkg from #12363. That worked. Python failed its self-tests, so I installed it with SAGE_CHECK unset. cvxopt failed self-tests, as before. Everything else passed (including ppl). Sage failed to start up, and it looks like the issue at #11967, so right now I'm rebuilding with a newer Python spkg. I'll report back when that's done.

comment:53 Changed 7 years ago by jdemeyer

gcc-4.6.2 is certainly the way to go. Sage builds, and there is only one doctest failing (with a Segmentation Fault):

$ ./sage -t --long  devel/sage/sage/rings/padics/padic_ZZ_pX_CR_element.pyx
sage -t --long "devel/sage/sage/rings/padics/padic_ZZ_pX_CR_element.pyx"
The doctested process was killed by signal 11
         [5.1 s]

----------------------------------------------------------------------
The following tests failed:


        sage -t --long "devel/sage/sage/rings/padics/padic_ZZ_pX_CR_element.pyx" # Killed/crashed
Total time for all tests: 5.1 seconds

comment:54 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #12422, #12423, #12425, #12456, #12457, #12363 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363
  • Description modified (diff)

comment:55 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12480

I created ticket #12480 for the padic_ZZ_pX_CR_element.pyx segfault.

comment:56 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12480 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363

Removing #12480 as a dependency, since it really has nothing to do with OS X 10.7.

Changed 7 years ago by jdemeyer

comment:57 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:58 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:59 Changed 7 years ago by jhpalmieri

I've tried the testing release on several machines. On all but one, everything seems fine, but on one, gcc fails to compile. Any ideas why? Here's the log file; note the lines

cc1: internal compiler error: Segmentation fault: 11
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.

comment:60 Changed 7 years ago by jhpalmieri

To clarify, I should have said that I've tried testing the release on several OS X Lion machines, and it's failing on one of them.

comment:61 follow-up: Changed 7 years ago by jdemeyer

1) Do you have MAKE="make -jN" set or such?

2) Does it fail consistently in the same spot if you try again?

3) Try building gcc with SAGE_CHECK=yes

4) Try building gcc with CFLAGS="-O0" or CFLAGS_FOR_BUILD="-O0"

comment:62 in reply to: ↑ 61 Changed 7 years ago by jhpalmieri

Replying to jdemeyer:

1) Do you have MAKE="make -jN" set or such?

I had MAKE="make -j2", but I unset it and that didn't help.

2) Does it fail consistently in the same spot if you try again?

Same spot, every time.a

3) Try building gcc with SAGE_CHECK=yes

I did that, but it didn't help: with SAGE_CHECK=yes and MAKE unset, it still fails in the same spot.

4) Try building gcc with CFLAGS="-O0" or CFLAGS_FOR_BUILD="-O0"

Do you mean just setting these variables, or do I need to modify the spkg? Just setting them (along with SAGE_CHECK and MAKE as above) doesn't help.

I also tried temporarily clearing out /usr/local in case some old files there were causing conflicts, but no help.

comment:63 Changed 7 years ago by jhpalmieri

(Actually, regarding item 2, it stops at one place if MAKE="make -j2", and another if MAKE is unset, as you might expect. But in each case, it's consistent.)

comment:64 Changed 7 years ago by jdemeyer

Could you also post a list of all your environment variables (see for example the top of SAGE_ROOT/install.log)

comment:65 Changed 7 years ago by jhpalmieri

Here's the top of install.log:

Installing GCC because a Fortran compiler is missing.
Installing GCC because your 'gcc' is not so recent.
*** ALL ENVIRONMENT VARIABLES BEFORE BUILD: ***
Apple_PubSub_Socket_Render=/tmp/launch-OIACLZ/Render
Apple_Ubiquity_Message=/tmp/launch-7Be69V/Apple_Ubiquity_Message
BIBINPUTS=/Users/palmieri/Documents/uw/tex/inputs/bibtex:.:
COMMAND_MODE=unix2003
DISPLAY=/tmp/launch-zCNApn/org.x:0
EDITOR=/usr/bin/emacs
FIGNORE=.log:.aux:.blg:.dvi:.aux:.bbl:.toc:.lof:.flc:.elc
HISTCONTROL=ignoreboth
HOME=/Users/palmieri
LANG=en_US.UTF-8
LOGNAME=palmieri
MAKE=make -j2
MAKEFLAGS=
MAKELEVEL=1
MFLAGS=
PATH=/Applications/sage_builds/clean/sage-5.0.beta3-gcc/spkg/bin:/Applications/sage_builds/clean/sage-5.0.beta3-gcc/local/bin:/Users/palmieri/bin:/opt/local/bin:/opt/local/sbin:/Applications/sage/:/usr/texbin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
PWD=/Applications/sage_builds/clean/sage-5.0.beta3-gcc/spkg
PYTHONPATH=/Applications/sage_builds/clean/sage-5.0.beta3-gcc/local
SAGEMATH=palmieri@sage.math.washington.edu:.
SAGE_DOC_JSMATH=yes
SAGE_LOCAL=/Applications/sage_builds/clean/sage-5.0.beta3-gcc/local
SAGE_LOGS=/Applications/sage_builds/clean/sage-5.0.beta3-gcc/spkg/logs
SAGE_PARALLEL_SPKG_BUILD=yes
SAGE_PORT=yes
SAGE_ROOT=/Applications/sage_builds/clean/sage-5.0.beta3-gcc
SHELL=/bin/bash
SHLVL=4
SSH_AUTH_SOCK=/tmp/launch-Y3Ezc9/Listeners
TERM=xterm-color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=303
TERM_SESSION_ID=7E69A75B-352B-4367-A920-892CA994DAB8
TEXINPUTS=/Applications/sage/local/share/texmf//:/Users/palmieri/Documents/uw/tex/inputs//:.:
TMPDIR=/var/folders/qk/gyt9gwv171df7_3lt94z5pqc0000gn/T/
TRANSFERS=jpalmier@homer.u.washington.edu:transfers
USER=palmieri
_=/usr/bin/env
__CF_USER_TEXT_ENCODING=0x1F5:0:0
***********************************************

(There is no directory /opt, so any references to it are not important. I should probably delete that part of my PATH.)

comment:66 Changed 7 years ago by jdemeyer

The $TMPDIR looks suspicious, but it seems to be added by Apple on all OS X 10.7 systems, so it can't be that. Otherwise, I don't see anything special in your environment.

comment:67 in reply to: ↑ 24 Changed 7 years ago by drkirkby

Replying to jdemeyer:

Replying to ohanar:

Agreed, which is why I would like only fortran to be built if it is the only language needed.

Is build time your only concern here? Because if you need to build Fortran, you most likely need to build a stage 1 C compiler anyway.

I'm 95% sure you will need to build the complete gcc - not just the stage 1, even if you only want fortran, so I doubt trying to build just fortran would save much, if any, time. Building g++ probably takes a non-significant amount of extra time, but I'd think it much more sensible to have all the compilers at the same version, and not a mixture.

We currently enforce the versions of gcc/g++/gfortran in Sage to be the same. Strictly speaking they don't need to be, but it increases the chances of the build going ok if the same versions are used. We can also attempt to replicate bugs reported much easier if all versions are the same.

Dave

comment:68 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12515

I removed mpc from the gcc spkg and propose to add a separate mpc spkg, which might also benefit Sage in other ways: #12515.

comment:69 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12515 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12223, #12515

comment:70 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12223, #12515 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12223, #12515, #12519

comment:71 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:72 Changed 7 years ago by jhpalmieri

Several times I've had the build fail in the middle when building with MAKE="make -j2". I think what happens is this: mpfr gets built for a second time, and during that build (or installation), its library gets deleted, and then the other spkg being built at the same time fails. For example, I just got this while building ntl:

g++ -I../include -I.  -O2 -g  -fno-common  -c  lzz_pXFactoring.c
g++ -I../include -I.  -O2 -g  -fno-common  -c  mat_GF2.c
g++ -I../include -I.  -O2 -g  -fno-common  -c  mat_GF2E.c
g++ -I../include -I.  -O2 -g  -fno-common  -c  mat_RR.c
dyld: Library not loaded: /Applications/sage_builds/GCC-clang-no-check/sage-5.0.beta4-gcc/local/lib/libmpfr.1.dylib
  Referenced from: /Applications/sage_builds/GCC-clang-no-check/sage-5.0.beta4-gcc/local/libexec/gcc/x86_64-apple-darwin11.3.0/4.6.2/cc1plus
  Reason: image not found
g++: internal compiler error: Trace/BPT trap: 5 (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
make[3]: *** [mat_RR.o] Error 4
make[2]: *** [libntl.dylib] Error 2
Failed to build NTL dylib.

So maybe mpir and mpfr should be built right after gcc, to avoid this?

comment:73 follow-up: Changed 7 years ago by jdemeyer

Thanks John, I already noticed this myself. I see two solutions:

  1. rebuild mpir, mpfr, mpc after GCC without building anything else.
  1. don't delete the old libraries in the mpir, mpfr, mpc (at least if gcc is installed).

I'm more inclined to go for option 2. I never quite understood why we have to delete old libraries anyway.

comment:74 follow-up: Changed 7 years ago by jhpalmieri

I don't know if it's the new version of Xcode (4.3) or if it's due to modifications in the latest gcc spkg, but the latest testing release builds fine with the default compiler on a laptop where it failed before and on a desktop where it succeeded before. ("Builds fine" means built and passed all of Sage's doctests. I have to rebuild with SAGE_CHECK=yes to see how that goes. I'll also try with clang.)

comment:75 in reply to: ↑ 74 Changed 7 years ago by jdemeyer

Replying to jhpalmieri:

I don't know if it's the new version of Xcode (4.3)

If you upgraded XCode on that laptop, that would be the likely reason why it suddenly works.

comment:76 in reply to: ↑ 73 Changed 7 years ago by jhpalmieri

Replying to jdemeyer:

  1. don't delete the old libraries in the mpir, mpfr, mpc (at least if gcc is installed).

I'm more inclined to go for option 2. I never quite understood why we have to delete old libraries anyway.

Right, this makes sense to me, and $MAKE install overwrites the old libraries anyway.

By the way, I've built with and without CC=clang (etc.) and with and without SAGE_CHECK=yes, on several different Lion machines, and I've had no problems except for python and cvxopt failing self-tests. I don't know what's going on with cvxopt. Could it be an atlas problem on Lion? In any case, the new Xcode seems to be a good choice for those people willing to upgrade from earlier versions of Lion to OS X 10.7.3.

comment:77 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12223, #12515, #12519 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548

comment:78 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548 to #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562

comment:79 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:80 follow-ups: Changed 7 years ago by drkirkby

There needs to be a couple of changes for Solaris - in particular, the ability to specify the assembler and linker to be used, since the Sun assembler is not suitable for gcc on x86, So when configuring gcc, one might specify

--with-gnu-as --with-as=/usr/local/binutils-2.20/bin/as --without-gnu-ld --with-ld=/usr/ccs/bin/ld

The above specifies the GNU assembler is used, where it is, the Sun linker is used, and where it is. I'd suggest we have environment variables to specify the locations of the linker and assembler. (Perhaps SAGE_PATH_TO_ASSEMBLER and SAGE_PATH_TO_LINKER or something similar). I suspect this could be useful sometimes on Linux too, as people might have more than one copy of the assembler and/or linker present.

Although I personally always specify --with-gnu-as or {{{--without-gnu-as}}], and similar for the linker, I don't think this is strictly necessary, as the gcc configure script will determine if linker and assembler specified are the GNU ones or not.

Do you want a patch on another ticket, or this one?

Dave

comment:81 in reply to: ↑ 80 Changed 7 years ago by jdemeyer

Replying to drkirkby:

I'd suggest we have environment variables to specify the locations of the linker and assembler.

GCC respects the standard $LD and $AS environment variables, so I don't see any need for this.

comment:82 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12366, #12367, #12368, #12405, #12416, #11967, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562 to #11073, #10492, #12367, #12368, #12405, #12570, #11967, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562

comment:83 Changed 7 years ago by jdemeyer

Apparently, $AS and $LD are only used to build GCC itself, so we would still need to add --with-as=$AS --with-ld=$LD.

comment:84 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12367, #12368, #12405, #12570, #11967, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562 to #11073, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562

comment:85 in reply to: ↑ 80 ; follow-up: Changed 7 years ago by jdemeyer

Replying to drkirkby:

the Sun assembler is not suitable for gcc on x86

I don't think this is true. I found only one issue with -pipe (#12562). Apart from that, Sage builds fine with the Sun assembler.

Anyway, I fixed the GCC spkg to use the AS and LD environment variables as arguments for --with-as and --with-ld.

comment:86 in reply to: ↑ 85 Changed 7 years ago by drkirkby

Replying to jdemeyer:

Replying to drkirkby:

the Sun assembler is not suitable for gcc on x86

I don't think this is true. I found only one issue with -pipe (#12562). Apart from that, Sage builds fine with the Sun assembler.

Anyway, I fixed the GCC spkg to use the AS and LD environment variables as arguments for --with-as and --with-ld.

IMHO, which is shared by the GCC documentation, the Sun assembler should be used, so unless AS is set, it would be wise to use /usr/ccs/bin/as. (On Solaris 11 and OpenSolaris, there's a link to /usr/bin/as, but that's not true on Solaris 10).

http://gcc.gnu.org/install/specific.html#ix86-x-solaris210

says on the Solaris x86 notes.


It is recommended that you configure GCC to use the GNU assembler, in /usr/sfw/bin/gas. The versions included in Solaris 10, from GNU binutils 2.15, and Solaris 11, from GNU binutils 2.19, work fine, although the current version, from GNU binutils 2.21, is known to work, too. Recent versions of the Sun assembler in /usr/ccs/bin/as work almost as well, though.

For linking, the Sun linker, is preferred.

It also says It may be necessary to configure with --without-gnu-ld --with-ld=/usr/ccs/bin/ld to guarantee use of Sun ld.


I thought the configure script would work out the --without-gnu-ld, but it might be worth adding the option unless LD is set.

I don't know what the status of t2.math is. William said he was going to switch it on a couple of weeks ago, but last time I looked it was down. Also, not sure about SkyNet?.

If we need a Solaris 11 x86 machine I have one. A bit limited in performance (pair of dual core 2.8 GHz Opterons and 8 GB RAM, but it is useable. It's running the latest release of Solaris 11. I really need to work out how to use the remote management on it, as it is very noisy, must be placed in the garage, but I don't know how to manage it properly on it's management port.

Dave

comment:87 Changed 7 years ago by jhpalmieri

With sage-5.0.beta4-gcc, CC=clang and CXX=clang++, things built fine on OS X Lion, Xcode 4.3. With sage-5.0.beta5-gcc, though, polybori fails to build when I try this: at the very start of the build, I get this:

****************************************************
Starting build...
Removing old PolyBoRi install...
Done removing old PolyBoRi install.
Running build_polybori...
scons: Reading SConscript files ...
Platform:  darwin
Detecting type sizes... no
Could not detect type sizes (maybe compile/link flags trouble)! Exiting.
Error building PolyBoRi.

I wonder what differences between beta4-gcc and beta5-gcc caused this.

I haven't seen this problem when using the default gcc instead of clang.

comment:88 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:89 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:90 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11073, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562 to #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629

comment:91 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:92 Changed 7 years ago by jdemeyer

  • Dependencies changed from #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629 to #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12011

Changed 7 years ago by jdemeyer

comment:93 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:94 Changed 7 years ago by jdemeyer

Unfortunately, it seems that GCC 4.6.3 miscompiles PARI on OS X PPC 32-bit, reported upstream at http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52517

comment:95 Changed 7 years ago by jhpalmieri

Well, the old gcc works fine on OS X PPC, so we can just keep using that, right?

comment:96 follow-up: Changed 7 years ago by kcrisman

Yeah, I really don't see the need to build gcc on older Mac platforms at all. This unnecessarily complicates things. I understand if this is necessary for Lion, but might as well stick with what works for the others.

comment:97 in reply to: ↑ 96 ; follow-up: Changed 7 years ago by jdemeyer

Replying to kcrisman:

Yeah, I really don't see the need to build gcc on older Mac platforms at all. This unnecessarily complicates things. I understand if this is necessary for Lion, but might as well stick with what works for the others.

We need to build GCC to have Fortran. Or we have to ship Fortran sources and binaries which makes no sense at all.

You could argue that we shouldn't use the gcc that we built. But I would expect gcc-4.6.3 to be better than Apple's gcc-4.0.1 (even though it contains bugs).

comment:98 Changed 7 years ago by jdemeyer

  • Dependencies changed from #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12011 to #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12011, #12638

comment:99 in reply to: ↑ 97 ; follow-up: Changed 7 years ago by kcrisman

Replying to jdemeyer:

Replying to kcrisman:

Yeah, I really don't see the need to build gcc on older Mac platforms at all. This unnecessarily complicates things. I understand if this is necessary for Lion, but might as well stick with what works for the others.

We need to build GCC to have Fortran. Or we have to ship Fortran sources and binaries which makes no sense at all.

Oh, I see - I forgot about this replacing the Fortran.

You could argue that we shouldn't use the gcc that we built. But I would expect gcc-4.6.3 to be better than Apple's gcc-4.0.1 (even though it contains bugs).

Well, but if it won't compile Pari, then maybe we should just stick with that. Why not just use the old Apple-shipped gcc for 10.4 and 10.5 or something? That seems very reasonable. And cuts down on build time for already old slow systems.

comment:100 in reply to: ↑ 99 ; follow-up: Changed 7 years ago by jdemeyer

Replying to kcrisman:

Well, but if it won't compile Pari, then maybe we should just stick with that.

In the mean time, I posted a work-around for PARI at #12638.

And cuts down on build time for already old slow systems.

Sure, it will, but I don't think "build time" really matters.

I much prefer run-time performance than build-time performance (if you really want the latter, compile everything with -O0). And one would expect run-time performance to be better with more recent GCC versions (although it will change by at most a few percent).

comment:101 Changed 7 years ago by jdemeyer

Another argument pro GCC-4.6.3 is that fixing things to work with newer GCC versions usually improves portability in general. For example, I don't think the miscompilation of PARI is specific to OS X PPC. It would probably fail on Linux PPC too.

comment:102 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from new to needs_review

I believe this is ready for review now. I don't know of any regressions, in the following sense: for every system on which vanilla sage-5.0.beta7 works, also sage-5.0.beta7-gccdeps and sage-5.0.beta7-gcc work.

It even builds and passes all tests on OS X 10.7, but there is still a serious issue with ATLAS (#12011, causing a failure in the cvxopt test suite). But I don't believe that should prevent a review of this ticket, as it's unrelated to GCC.

comment:103 in reply to: ↑ 100 Changed 7 years ago by kcrisman

Well, but if it won't compile Pari, then maybe we should just stick with that.

In the mean time, I posted a work-around for PARI at #12638.

And cuts down on build time for already old slow systems.

Sure, it will, but I don't think "build time" really matters.

Well, all fair enough. Just want to make sure I'm asking the questions, even if they're dumb :)

comment:104 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:105 Changed 7 years ago by jdemeyer

  • Dependencies changed from #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12011, #12638 to #12479, #12602, #12608, #12609, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12011, #12638

comment:106 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:107 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12479, #12602, #12608, #12609, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12011, #12638 to #12479, #12602, #12608, #12609, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638
  • Description modified (diff)

comment:108 follow-up: Changed 7 years ago by SimonKing

Building the gcc spkg really takes an awful lot of time.

One question: If I want to rebuild Sage with the compiler built by the spkg, perhaps with additional optimizations (provided that the gcc-spkg is able to pick up my system-wide CLooG or I manage to make a CLooG spkg), how could I do so?

Of course, sage -ba would recompile all Cython code in the Sage library, but it would not re-install the spkgs. Is there an easy way to install all standard packages from scratch? I am not talking about an automatic re-installation of the optional packages.

In particular, would it work to remove all entries in SAGE_ROOT/spkg/installed that correspond to standard packages, and do make again? Or perhaps better exempt the sage* spkgs from re-installation, since my Sage library is patched?

comment:109 in reply to: ↑ 108 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

Of course, sage -ba would recompile all Cython code in the Sage library, but it would not re-install the spkgs. Is there an easy way to install all standard packages from scratch? I am not talking about an automatic re-installation of the optional packages.

In particular, would it work to remove all entries in SAGE_ROOT/spkg/installed that correspond to standard packages, and do make again?

Yes, that should work.

Or perhaps better exempt the sage* spkgs from re-installation, since my Sage library is patched?

I honestly have no idea what would happen if you re-install the Sage library over a patched version.

comment:110 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12479, #12602, #12608, #12609, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638 to #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638

comment:111 follow-up: Changed 7 years ago by SimonKing

Hooray, the spkg did install, and it did early enough that I will catch the next bus. But it took long enough that I missed two buses...

On sage-devel, we discussed whether the gcc package would pick up a system wide installation of CLooG. In the install log of the spkg, I do find lines such as

gcc -c   -O0 -DIN_GCC   -W -Wall -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -Wold-style-definition -Wc++-compat -fno-common  -DHAVE_CONFIG_H -I. -I. -I../../src/gcc -I../../src/gcc/. -I../../src/gcc/../include -I../../src/gcc/../libcpp/include -I/home/simon/SAGE/sage-5.0.beta7/local/include -I/home/simon/SAGE/sage-5.0.beta7/local/include -I/home/simon/SAGE/sage-5.0.beta7/local/include  -I../../src/gcc/../libdecnumber -I../../src/gcc/../libdecnumber/bid -I../libdecnumber    ../../src/gcc/graphite-cloog-util.c -o graphite-cloog-util.o

Does that mean that CLooG/graphite will work?

Here is the log.

comment:112 Changed 7 years ago by SimonKing

On sage-devel, Jeroen said that I should look in the configure part of the log. There, I can not find CLooG mentioned. Am I right? If that is the case, then I would try to add an optional CLooG spkg, which could then use the gmp-5.0.4 spkg.

comment:113 in reply to: ↑ 111 Changed 7 years ago by jdemeyer

Replying to SimonKing:

Does that mean that CLooG/graphite will work?

No, because it can't find PPL:

checking for version 0.11 (revision 0 or later) of PPL... no

If PPL isn't found, then GCC doesn't even bother looking for CLooG.

I will change the GCC spkg such that it looks for PPL in $SAGE_LOCAL.

comment:114 follow-ups: Changed 7 years ago by jdemeyer

  • Cc vbraun added

Hmm, the problem is not with GCC, but with the PPL in Sage. It seems Sage-PPL doesn't install ppl_c.h which is needed for GCC. I don't know more about this.

comment:115 in reply to: ↑ 114 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Hmm, the problem is not with GCC, but with the PPL in Sage. It seems Sage-PPL doesn't install ppl_c.h which is needed for GCC. I don't know more about this.

:((

Anyway. I thought it would not hurt to create an optional CLooG spkg: See #12666.

The upgraded optional gmp spkg is at #12661.

comment:116 in reply to: ↑ 114 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Hmm, the problem is not with GCC, but with the PPL in Sage. It seems Sage-PPL doesn't install ppl_c.h which is needed for GCC. I don't know more about this.

There is ppl.hh provided by the PPL spkg. But I can not find ppl_c.h in the spkg's sources, and also I can not find any mention of ppl_c.h in the spkg's install log.

So, where could ppl_c.h come from?

comment:117 follow-ups: Changed 7 years ago by vbraun

The PPL spkg does not install the C interface since we only use the native C++ interface in Sage. It would be easy to also build the C interface but I don't quite understand where we are heading here. Nobody is going to have a usable CLooG/Graphite laying around without a working GCC build. On Linux its easier to install a fully working GCC than just CLooG, I daresay. And a quick grep through the install.log suggests that nothing is being built with -floop* options.

comment:118 in reply to: ↑ 117 ; follow-up: Changed 7 years ago by jdemeyer

Replying to vbraun:

I don't quite understand where we are heading here.

I guess Simon's idea is to have an optional CLooG spkg in Sage, such that people could build Sage's GCC with PPL+CLooG.

comment:119 in reply to: ↑ 118 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Replying to vbraun:

I don't quite understand where we are heading here.

I guess Simon's idea is to have an optional CLooG spkg in Sage, such that people could build Sage's GCC with PPL+CLooG.

Yes. And of course -floop* options aren't used - yet. But according to some statement on sage-devel, sometimes 30% more speed is possible with such options. So, I am curious whether I'd obtain a noticeable speed difference when I add those kind of options to CFLAGS/CXXFLAGS and do sage -ba.

comment:120 in reply to: ↑ 117 ; follow-up: Changed 7 years ago by SimonKing

Replying to vbraun:

The PPL spkg does not install the C interface since we only use the native C++ interface in Sage.

Thank you! I changed the ppl spkg's install script accordingly, and now I have ppl_c.h in SAGE_LOCAL/include.

comment:121 in reply to: ↑ 120 Changed 7 years ago by SimonKing

Replying to SimonKing:

Replying to vbraun:

The PPL spkg does not install the C interface since we only use the native C++ interface in Sage.

Thank you! I changed the ppl spkg's install script accordingly, and now I have ppl_c.h in SAGE_LOCAL/include.

It didn't help, I'm afraid.

I have ppl_c.h, I have GMP, and I have CLooG. However, when building the GCC spkg again, the log contains the line

checking for installed CLooG PPL Legacy... no

Any idea what is missing? I am afraid that I was not able to copy the log to sage.math, but I'll try again later.

comment:122 follow-up: Changed 7 years ago by SimonKing

Is it perhaps needed to build CLooG again after building PPL with C interface? At least this is what I am trying now.

comment:123 in reply to: ↑ 122 Changed 7 years ago by SimonKing

Replying to SimonKing:

Is it perhaps needed to build CLooG again after building PPL with C interface?

No, it doesn't work. It still says there is no CLooG PPL Legacy.

comment:124 Changed 7 years ago by SimonKing

Could it be that there is an important difference between cloog and cloog-ppl?

Google pointed me to cloog-ppl, but I could not find sources (although it seems to be a different package than cloog).

comment:125 follow-up: Changed 7 years ago by SimonKing

I found cloog-ppl sources. But there's the next problem. It says:

...
Can't find PolyLib.
checking for Parma Polyhedral Library (PPL)... not using PPL
...
In file included from source/block.c:41:0:
source/../include/cloog/cloog.h:47:30: schwerwiegender Fehler: polylib/missing.h: Datei oder Verzeichnis nicht gefunden
Kompilierung beendet.

PolyLib is not listed as a dependency, but ppl is.

So, why is it (still!) claimed that we are not using PPL, even though we have ppl_c.h and ppl.hh???

I apologize for the fact that it is slightly off topic - but I'd still like to learn how to build your gcc spkg with -floop* support.

comment:126 in reply to: ↑ 125 Changed 7 years ago by SimonKing

Replying to SimonKing:

Can't find PolyLib?. checking for Parma Polyhedral Library (PPL)... not using PPL

Hooray! in my cloog_ppl spkg (not submitted yet) I had simply forgotten to add --with-ppl to the configuration. And now, your gcc package says

checking for installed CLooG PPL Legacy... PPL Legacy
checking for version 0.15.5 (or later revision) of CLooG... yes

comment:127 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:128 Changed 7 years ago by SimonKing

Not good.

After building the compiler with CLooG PPL support, I tried to rebuild Sage with

CFLAGS="-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing"
export CFLAGS
CXXFLAGS="-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing"
export CXXFLAGS

This is what (I think) was suggested on sage-devel.

However, it failed very early: It was found that the compiler can not build executables. The compiler was referred to not as gcc, but as

### Compiler was: gcc -g -O3 -march=native -floop-interchange -floop-strip-mine -floop-block

So, could it be that I misunderstood and the flags have to be passed in a different way?

comment:129 Changed 7 years ago by SimonKing

In the gcc man pages, I read

This optimization applies to all the languages supported by GCC and is not limited to Fortran. To use this code transformation, GCC has to be configured with --with-ppl and --with-cloog to enable the Graphite loop transformation infrastructure.

But these to options are in fact not used in the gcc spkg!

I am trying to add them (and also providing --libdir, which in some cases seems to be needed on openSuse) and try to build gcc again.

comment:130 follow-ups: Changed 7 years ago by vbraun

I'm pretty sure that compiling everything with -O3 + graphite will break, but I applaud your heroic effort ;-)

For the record, here is Fedora's gcc configure, that's at least one data point for how to build it:

[vbraun@volker-laptop-two ~]$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.6.2/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.6.2 20111027 (Red Hat 4.6.2-1) (GCC) 

It seems like you don't need any magic beyond --with-ppl and --with-cloog

comment:131 in reply to: ↑ 130 ; follow-up: Changed 7 years ago by SimonKing

Replying to vbraun:

It seems like you don't need any magic beyond --with-ppl and --with-cloog

Exactly. But the current version of this optional gcc spkg does not contain the magic.

Anyway. I am now building gcc again (after adding the magical options to spkg-install) and will report later about my heroic failures.

comment:132 in reply to: ↑ 131 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

Exactly. But the current version of this optional gcc spkg does not contain the magic.

I don't mind adding support for this in a follow-up spkg (it's not entirely trivial, because we have to check whether CLooG is installed or not). I prefer to wait until this is reviewed and merged, to avoid rebasing in case the current spkg has problems.

comment:133 in reply to: ↑ 132 ; follow-up: Changed 7 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing: I don't mind adding support for this in a follow-up spkg (it's not entirely trivial, because we have to check whether CLooG is installed or not).

OK. But to check whether CLooG is installed should not be difficult (at least: to check whether a CLooG spkg is installed should be easy).

I prefer to wait until this is reviewed and merged, to avoid rebasing in case the current spkg has problems.

OK. But I will keep trying (and I just lost an hour compilation time because of a typo).

comment:134 in reply to: ↑ 133 Changed 7 years ago by jdemeyer

Replying to SimonKing:

But to check whether CLooG is installed should not be difficult

See #12609 which deals with this kind of checks.

comment:135 in reply to: ↑ 130 Changed 7 years ago by SimonKing

Replying to vbraun:

I'm pretty sure that compiling everything with -O3 + graphite will break, but I applaud your heroic effort ;-)

Then you might be surprised that the following worked:

  • Build gcc via the spkg
  • Build ppl (with c interface), gmp and cloog-ppl, using
    CFLAGS="-O3 -march=native"
    CXXFLAGS="-O3 -march=native"
    
  • Build gcc again, via the modified spkg (--with-ppl --with-cloog), again with -O3 -march=native.
  • Build all of Sage again (all spkgs and sage -ba), using
    CFLAGS="-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing"
    CXXFLAGS="-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing"
    
  • Run more than half of sage -testall without error.

Nevertheless, it was a heroic failure.

Here are some test times without optimisation, for sage-5.0.beta7 built with my system gcc-4.6.2:

sage -t  -force_lib "devel/sage/sage/symbolic/function.pyx" 
         [3.2 s]
sage -t  -force_lib "devel/sage/sage/symbolic/benchmark.py" 
         [8.6 s]
sage -t  -force_lib "devel/sage/sage/symbolic/function_factory.py"
         [2.5 s]
sage -t  -force_lib "devel/sage/sage/symbolic/ring.pyx"     
         [4.0 s]
sage -t  -force_lib "devel/sage/sage/symbolic/relation.py"  
         [9.4 s]
sage -t  -force_lib "devel/sage/sage/symbolic/maxima_wrapper.py"
         [2.9 s]

Here are the same tests, using full optimization, using the gcc-4.6.3 from the spkg with graphite:

sage -t  -force_lib "devel/sage/sage/symbolic/function.pyx" 
         [8.0 s]
sage -t  -force_lib "devel/sage/sage/symbolic/benchmark.py" 
         [17.4 s]
sage -t  -force_lib "devel/sage/sage/symbolic/function_factory.py"
         [5.7 s]
sage -t  -force_lib "devel/sage/sage/symbolic/ring.pyx"     
         [7.8 s]
sage -t  -force_lib "devel/sage/sage/symbolic/relation.py"  
         [23.7 s]
sage -t  -force_lib "devel/sage/sage/symbolic/maxima_wrapper.py"
         [43.5 s]

What kind of optimization is it, if it makes things MASSIVELY slower???

comment:136 follow-up: Changed 7 years ago by jdemeyer

I would be interesting to compare the timings with my GCC spkg, without PPL or CLooG.

comment:137 follow-up: Changed 7 years ago by jdemeyer

Also: can you try exactly the same without building gmp?

comment:138 in reply to: ↑ 137 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Also: can you try exactly the same without building gmp?

The first thing that I had tried was building CLooG in the Sage shell. It failed during configuration and complained that it needs GMP. So, unless there is a configuration option, I don't see how MPIR could suffice.

comment:139 Changed 7 years ago by jdemeyer

Well, it could be that GMP slowed things down, not GCC.

comment:140 in reply to: ↑ 136 Changed 7 years ago by SimonKing

Replying to jdemeyer:

I would be interesting to compare the timings with my GCC spkg, without PPL or CLooG.

I wonder: How could I apply all the patches to the sage root/extcode/lib repositories before Sage is built?

Or do you suggest that I start with my existing sage-5.0.beta7, and do the following:

  • rm $SAGE_LOCAL/*/*gcc*
  • rm -r $SAGE_LOCAL/*/*gmp*
  • rm $SAGE_ROOT/spkg/installed/*
  • do sage -f gcc-4.6.3.spkg, without setting compilation flags, and without the --with-ppl --with-cloog options; this would be built using my system's gcc, because of the first step
  • do make, which would then use the spkg's gcc

comment:141 follow-up: Changed 7 years ago by SimonKing

No, my previous sage-5.0.beta7 is totally messed up. I have to start from scratch.

I don't know how to do the following cleanly. Anyway, my plan is:

  • untar the source ball
  • type make, wait a few seconds, and interrupt, so that "./sage -sh" is available
  • replace the old pari and ppl spkgs by the new ones
  • provide C(XX)FLAGS="-O3 -march=native"
  • install the standard mpir spkg
  • install the standard (updated) ppl spkg (this is what I am doing right now)
  • install the cloog_ppl spkg
  • install the optional mpc spkg (I lost track where I got that from...).
  • install the modified gcc spkg (using --with-ppl="$SAGE_LOCAL" --with-cloog="$SAGE_LOCAL")
  • switch to C(XX)FLAGS="-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing"
  • type make and keep fingers crossed.

comment:142 in reply to: ↑ 141 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

  • type make, wait a few seconds, and interrupt, so that "./sage -sh" is available

./sage --sh should be available right after untarring. This was a consequence of #11073.

  • install the optional mpc spkg (I lost track where I got that from...)

#12515 (GCC needs it)

comment:143 follow-up: Changed 7 years ago by SimonKing

There is one aspect that I am very much disappointed about: The time for building this spkg. My laptop is now running for about 5 hours, but gcc is still not built, hence, the compilation of Sage itself didn't even start! That is not acceptable.

So, this can really only be the last resort, say, if OS X really can not provide adequate compilers.

comment:144 Changed 7 years ago by SimonKing

I just noticed - after five hours of compilation time - there is yet another round of configuration!!

How can it be that configuration is not only done in the very beginning? There is the option --config-cache used in the spkg-install script! But perhaps I am using it in the wrong way, so that it is in fact not cached and wastes time?

comment:145 in reply to: ↑ 143 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

There is one aspect that I am very much disappointed about: The time for building this spkg. My laptop is now running for about 5 hours, but gcc is still not built, hence, the compilation of Sage itself didn't even start!

You're not using SAGE_CHECK=yes by chance? That slows down GCC significantly.

comment:146 in reply to: ↑ 145 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

There is one aspect that I am very much disappointed about: The time for building this spkg. My laptop is now running for about 5 hours, but gcc is still not built, hence, the compilation of Sage itself didn't even start!

You're not using SAGE_CHECK=yes by chance? That slows down GCC significantly.

No. And, by the way, in the end it needed 509 minutes to build.

The day before yesterday, I was installing your original package (without graphite) in a fully functional Sage framework. Then, I was using the resulting compiler to build MPC, GMP, PPL, CLooG-PPL and finally GCC with graphite.

Now, I was building MPIR, MPFR, MPC, PPL, CLooG-PPL and GCC in an "empty" anvironment. So, much different setting. Next, I am re-building MPIR, MPFR, MPC, PPL and CLooG-PPL using "gcc+graphite" in high optimization -- and then I am curious to learn how long it takes to build GCC again.

Anyway. I will post the install logs soon.

comment:147 follow-up: Changed 7 years ago by jdemeyer

What system is that? This is on a Gentoo Linux laptop with a Intel(R) Core(TM)2 Duo CPU T5870 @ 2.00GHz, using 1 thread to compile the GCC spkg on this ticket:

real    70m11.765s
user    62m3.177s
sys     3m14.668s
Successfully installed gcc-4.6.3
Deleting temporary build directory
/usr/local/src/sage-5.0.beta7/spkg/build/gcc-4.6.3
Making Python scripts relocatable...
Finished installing gcc-4.6.3.spkg

comment:148 in reply to: ↑ 147 ; follow-up: Changed 7 years ago by SimonKing

Replying to jdemeyer:

What system is that?

Intel(R) Core(TM)2 Duo CPU P7450 @ 2.13GHz, openSUSE 12.1 "Asparagus", x86_64 GNU/Linux

real 70m11.765s

It took not much longer for me, the day before yesterday. Perhaps something is wrong with my computer, and I should at least reboot.

comment:149 in reply to: ↑ 148 Changed 7 years ago by SimonKing

Replying to SimonKing:

Replying to jdemeyer:

What system is that?

Intel(R) Core(TM)2 Duo CPU P7450 @ 2.13GHz, openSUSE 12.1 "Asparagus", x86_64 GNU/Linux

real 70m11.765s

It took not much longer for me, the day before yesterday. Perhaps something is wrong with my computer, and I should at least reboot.

Perhaps my laptop had just have a veeeeeeery bad day. I built gcc again (with the gcc that I had built in the 8 hour session before), and it was much faster. Also, my first attempts to build gcc (the day before yesterday) went better.

I have posted two install logs:

  • The first was from the day before yesterday, and it was in an existing Sage installation. Note that that install log contains several installations of gcc, with different parameters
  • The second is from yesterday/today and contains the very slow session and also the faster session. Hence, it is again a documentation of two installations of gcc. It was made BEFORE building Sage. That's to say, I have first manually installed the dependencies of gcc+graphite, then gcc, then installed the dependencies with higher optimization, then gcc+graphite again (just to see if it is faster), and then I typed "make". The new Sage installation is not done, yet.

Perhaps the experts (i.e., you! ) can see from the parameters documented in the install logs why it went so slow, yesterday. Note that I use configuration options different from the original options in your spkg.

comment:150 Changed 7 years ago by SimonKing

It seems that GMP was to blame for the performance loss that I observed previously in the test suite.

Here is in all detail how I proceeded:

  • untar the source ball and open a sage shell
  • replace the old pari and ppl spkgs by the new ones
  • provide C(XX)FLAGS="-O3 -march=native"
  • install the standard mpir, (updated) ppl and mpfr spkgs
  • install the optional mpc and cloog_ppl spkgs
  • install the modified gcc spkg (using
    ../src/configure \
        --prefix="$SAGE_LOCAL" \
        --with-local-prefix="$SAGE_LOCAL" \
        --libdir="$SAGE_LOCAL/lib" \
        --config-cache --enable-build-with-cxx --disable-ppl-version-check --disable-cloog-version-check \
        --with-ppl="$SAGE_LOCAL" --with-cloog="$SAGE_LOCAL" \
        --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" --with-mpc="$SAGE_LOCAL" \
        --with-system-zlib \
        --disable-multilib \
        $CONFIGURE_FLAGS "$CONFIGURE_AS" "$CONFIGURE_LD"
    
    This took more than 8 hours. But meanwhile I think that my computer had just a bad day.
  • switch to C(XX)FLAGS="-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing"
  • recompile mpir, ppl, mpfr, mpfi, mpc and cloog_ppl (to have the optimization as well)
  • recompile gcc with the new level of optimization (and just to see whether it takes another 8 hours...)
  • make. Then, the first attempt of building PolyBoRi failed. But after closing and reopening the Sage shell, make did work on PolyBoRi.
  • After a few seconds, I noticed that I had forgotten to change C(XX)FLAGS. So, I interrupted, deleted the items in spkg/installed that came after polybori, set the flags, and tried again... And it still worked!
  • Strange: ppl, mpir and mpfr, were built again, although I had installed them manually in the very beginning.
  • When make was done, I applied all the dependencies of this ticket (numerous patches to sage-root, sage-extcode and sage-lib.
  • sage -b
  • sage -testall

Result: The whole test suite took 5785.2 seconds when Sage is built with standard CFLAGS, and 5845.7 seconds when it is built with high optimization. So, there is no significant difference with or without optimization. But at least, it works, and it becomes clear that one should keep using MPIR, not GMP.

comment:151 in reply to: ↑ 142 ; follow-up: Changed 7 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

  • type make, wait a few seconds, and interrupt, so that "./sage -sh" is available

./sage --sh should be available right after untarring. This was a consequence of #11073.

It isn't. I took sage-5.0.beta8.tar, untarred it, and got:

king@mpc622:/mnt/local/king/SAGE/withgcc/sage-5.0.beta8$ ./sage --sh
./sage: no Sage installation found in $SAGE_ROOT=/mnt/local/king/lmonade/prefix-20111201/local/share/sage

So, should #11073 be re-opened?

comment:152 in reply to: ↑ 151 Changed 7 years ago by SimonKing

Replying to SimonKing:

./sage: no Sage installation found in $SAGE_ROOT=/mnt/local/king/lmonade/prefix-20111201/local/share/sage

Sorry, I accidentally started in a prefix environment. It does work. False alarm. Sorry.

comment:153 Changed 7 years ago by jdemeyer

Yes, setting SAGE_ROOT before running Sage will actually override the Sage root directory. So, if somebody installs a system-wide sage script, the user can set SAGE_ROOT himself to his preferred Sage installation.

comment:154 follow-up: Changed 7 years ago by SimonKing

Note that apparently it is needed to set something like --libdir in the configuration of one or the other package: When I tried to use the gcc spkg on another computer (not openSuse this time), the gfan spkg did not build, because libstdc++ was looked for in /usr/lib64 rather than in $SAGE_LOCAL/lib, where it can be found.

comment:155 follow-up: Changed 7 years ago by SimonKing

A (perhaps related) question: Is it normal that $SAGE_LOCAL/bin is not present when one starts to build Sage?

comment:156 in reply to: ↑ 155 Changed 7 years ago by jdemeyer

Replying to SimonKing:

A (perhaps related) question: Is it normal that $SAGE_LOCAL/bin is not present when one starts to build Sage?

Even $SAGE_LOCAL is not present (as far as I know). There is nothing wrong with this, the directories are created first thing when building Sage.

comment:157 in reply to: ↑ 154 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

Note that apparently it is needed to set something like --libdir in the configuration of one or the other package: When I tried to use the gcc spkg on another computer (not openSuse this time), the gfan spkg did not build, because libstdc++ was looked for in /usr/lib64 rather than in $SAGE_LOCAL/lib, where it can be found.

Can you give more details: which machine is this, which sources did you use, how did you build sage, was it a parallel build, what does the log file say...

comment:158 in reply to: ↑ 157 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Can you give more details: which machine is this, which sources did you use, how did you build sage, was it a parallel build, what does the log file say...

It is

king@mpc622:~$ uname -a
Linux mpc622 2.6.34.linuxpool #0 SMP PREEMPT Wed May 19 16:32:19 CEST 2010 x86_64 GNU/Linux
king@mpc622:~$ cat /etc/issue
Debian GNU/Linux 6.0 \n \l

and the four processors are

Intel(R) Core(TM) i3 CPU         530  @ 2.93GHz

Concerning the rest of your question: I think it would be best to be a bit more systematic.

My plan is to build Sage on the machine above in a couple of settings, each time with MAKE="make -j4" and SAGE_PARALLEL_SPKG_BUILD="yes":

  • Default flags and system gcc
  • -march=native and system gcc. Yesterday, I got the impression that Atlas did not build in this setting.
  • Default flags and gcc spkg as it is posted here.
  • -march=native and the gcc spkg as it is posted here.
  • -march=native and the gcc spkg modified so that it is built with graphite; change to -march=native -floop-... for building the rest of Sage. I think this was the setting in which gfan failed to build.

I think doing parallel builds, odds are that I will be able to post five install.logs this night.

comment:159 Changed 7 years ago by SimonKing

  • Status changed from needs_review to needs_work

On the machine described in my previous post, I got a total failure with the gcc package that is attached to this ticket.

I did the following:

tar -xf sage-5.0.beta8.tar
cd sage-5.0.beta8/
./sage -sh
cd spkg/standard/
wget http://boxen.math.washington.edu/home/jdemeyer/spkg/mpc-1.0.0dev1126.spkg
rm pari-2.5.1.p0.spkg
wget http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.1.p2.spkg
wget http://boxen.math.washington.edu/home/jdemeyer/spkg/gcc-4.6.3.spkg
cd ../../
sage -i spkg/standard/mpir-2.1.3.p9.spkg
sage -i spkg/standard/mpfr-3.1.0.p0.spkg
sage -i spkg/standard/mpc-1.0.0dev1126.spkg
   # note: it ended with /mnt/local/king/SAGE/experiment/gccspkg_native/sage-5.0.beta8/spkg/bin/sage-spkg: Zeile 485: cd: /mnt/local/king/SAGE/experiment/gccspkg_native/sage-5.0.beta8/local/bin: Datei oder Verzeichnis nicht gefunden
sage -i spkg/standard/gcc-4.6.3.spkg
   # fails

The reason for the failure: During configuration, it is claimed that the suffix of object files can not be determined - which is strange since the suffix has been successfully determined repeatedly during configuration.

Please have a look at the log file.

comment:160 Changed 7 years ago by SimonKing

I forgot to add: The failure both happens with MAKE="make -j4" and MAKE=make. I used CFLAGS="-march=native" and CXXFLAGS="-march=native".

comment:161 follow-up: Changed 7 years ago by jdemeyer

  • Status changed from needs_work to needs_review

You should first do

cd spkg && ./install base

to install the base packages.

comment:162 in reply to: ↑ 161 Changed 7 years ago by SimonKing

Replying to jdemeyer:

You should first do

cd spkg && ./install base

to install the base packages.

Ah! That is why it originally worked when I did "make" and waited a few seconds before hitting Ctrl-C...

OK, I am trying it again, with a parallel build and -march=native. And after that, I'll do the same with graphite.

comment:163 Changed 7 years ago by SimonKing

  • Status changed from needs_review to needs_work

I was able to reproduce the error when building gfan with the gcc from the spkg, as I have reported yesterday.

Here is the install log.

I was using MAKE="make -j4" and SAGE_PARALLEL_SPKG_BUILD="yes", the machine is as described above.

comment:164 Changed 7 years ago by SimonKing

In my next attempt, I added --libdir to the configuration of the gcc spkg. Hence, in spkg-install of the gcc spkg I have

../src/configure \
    --prefix="$SAGE_LOCAL" \
    --libdir="$SAGE_LOCAL/lib" \
    --with-local-prefix="$SAGE_LOCAL" \
    --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" --with-mpc="$SAGE_LOCAL" \
    --with-system-zlib \
    --disable-multilib \
    $CONFIGURE_FLAGS "$CONFIGURE_AS" "$CONFIGURE_LD"

But then, I again get the error

checking for suffix of object files... configure: error: in `/mnt/local/king/SAGE/experiment/gcclibdir_native/sage-5.0.beta8/spkg/build/gcc-4.6.3/gcc-build/x86_64-unknown-linux-gnu/libgcc':
configure: error: cannot compute suffix of object files: cannot compile

even though the suffix of object files has been computed repeatedly and correctly. From the config.log:

configure:4367: checking for suffix of object files
configure:4389: gcc -c -march=native  conftest.c >&5
configure:4393: $? = 0
configure:4414: result: o

So, why is providing the --libdir configuration option resulting in an error?

comment:165 Changed 7 years ago by SimonKing

comment:166 follow-ups: Changed 7 years ago by SimonKing

I could imagine that my error report was a bit confusing, since it was in fact two different reports.

The first, resulting in this install log, is with the unmodified spkg, and gfan fails to build because something can not be found where it is supposed to.

The second was with a modified spkg, in the attempt to fix the previous error by defining --libdir, but failed as in config.log or gcc-4.6.3.log

However, it seems that the second problem is actually not related with --libdir: Following Jeroen's advice, I did ./install base in SAGE_ROOT/spkg. But apparently that has not been sufficient. Now, just for fun, I did make and interrupted after a few seconds -- and when I then tried to install the modified gcc spkg (with --libdir), then the configuration worked.

Strange. So, what was missing in addition to ./install base?

comment:167 in reply to: ↑ 166 Changed 7 years ago by SimonKing

Replying to SimonKing:

The second was with a modified spkg, in the attempt to fix the previous error by defining --libdir

I am afraid it did not fix the trouble with gfan not finding GLIBCXX_3.4.15 in usr/lib64/libstdc++.so.6`. That is strange. Is it perhaps needed to explicitly rebuild the packages that I had originally built with the system gcc? Namely, these are mpir, mpfr and mpc. And at least mpc is probably not rebuilt automatically.

comment:168 follow-up: Changed 7 years ago by fbissey

It is clearly a sign that gfan is looking for libstdc++ in the wrong place. I haven't tried the new gcc spkg yet but we may have to add the location of libstdc++ in LD_LIBRARY_PATH. I am assuming that libstdc++ is not in $SAGE_LOCAL/lib and that's the problem.

comment:169 follow-up: Changed 7 years ago by fbissey

OK so in a normal configuration without modification it lands in $SAGE_LOCAL/lib64 which is addressed in #12405 which is in beta8. So it should be able to find it. At this stage I would like the output of

strace gfan

to see what exactly happens and what is opened when.

For your problem when you pass --libdir=$SAGE_LOCAL/lib you are probably looking at the wrong config.log. The config.log you posted is fine but it is also the top level config.log. The configuration file that fails is further down the building process. From the context of your build log it looks like it is related to libgcc configuring itself for a multilib install. I will try to reproduce the error.

comment:170 in reply to: ↑ 168 Changed 7 years ago by SimonKing

Replying to fbissey:

It is clearly a sign that gfan is looking for libstdc++ in the wrong place. I haven't tried the new gcc spkg yet but we may have to add the location of libstdc++ in LD_LIBRARY_PATH. I am assuming that libstdc++ is not in $SAGE_LOCAL/lib and that's the problem.

I found

sage-5.0.beta8/local/lib64/libstdc++.so.6

In other words, even though I defined --libdir="$SAGE_LOCAL/lib", the file libstdc++.so.6 is still put into the wrong folder, namely $SAGE_LOCAL/lib64. How can that be?

comment:171 in reply to: ↑ 169 Changed 7 years ago by SimonKing

Replying to fbissey:

OK so in a normal configuration without modification it lands in $SAGE_LOCAL/lib64 which is addressed in #12405 which is in beta8. So it should be able to find it. At this stage I would like the output of

strace gfan

to see what exactly happens and what is opened when.

Can you elaborate what that means?

For your problem when you pass --libdir=$SAGE_LOCAL/lib you are probably looking at the wrong config.log.

It seems to me that these are two separate problems.

First problem: If I just do ./install base, as suggested by Jeroen, then gcc fails to build, because at some point it can not determine the suffix of object files. That has nothing to do with --libdir being defined or not - it happens with or without --libdir.

Second problem: If one does make and interrupts after a few seconds, then gcc builds, regardless whether --libdir is defined or not; so, the first problem is worked around. However, even if --libdir is defined during top-level configuration, libstdc++.so.6 is put into a different folder. Note that spkg-install only contains a single call to ./configure -- that's the one I was providing with --libdir..

comment:172 Changed 7 years ago by jdemeyer

Simon, instead of adding all the patches and spkgs yourself, could you just try the testing tarball mentioned on this ticket: http://boxen.math.washington.edu/home/jdemeyer/release/sage-5.0.beta8-gcc/sage-5.0.beta8-gcc.tar

comment:173 follow-up: Changed 7 years ago by jdemeyer

Simon, try the most simple configuration: no parallel build, no CFLAGS, CXXFLAGS, using the source tarball I pointed out to you. If that fails, it truly is a problem with this ticket.

If that works, try adding "complexity" gradually, such that we can see more precisely what fails.

comment:174 in reply to: ↑ 173 ; follow-up: Changed 7 years ago by SimonKing

Replying to jdemeyer:

Simon, try the most simple configuration: no parallel build, no CFLAGS, CXXFLAGS, using the source tarball I pointed out to you. If that fails, it truly is a problem with this ticket.

OK, I am in the middle of it. gcc has successfully built after about 50 minutes (no flags, no parallel build, no additional patches).

However, I am sure that we will eventually have an error. Namely:

$ ls testinggcc/vanilla/sage-5.0.beta8-gcc/local/lib64/
libgcc_s.so           libgomp.la        libmudflap.so.0        libquadmath.so        libssp.so.0.0.0
libgcc_s.so.1         libgomp.so        libmudflap.so.0.0.0    libquadmath.so.0      libstdc++.a
libgfortran.a         libgomp.so.1      libmudflapth.a         libquadmath.so.0.0.0  libstdc++.la
libgfortran.la        libgomp.so.1.0.0  libmudflapth.la        libssp.a              libstdc++.so
libgfortran.so        libgomp.spec      libmudflapth.so        libssp.la             libstdc++.so.6
libgfortran.so.3      libiberty.a       libmudflapth.so.0      libssp_nonshared.a    libstdc++.so.6.0.16
libgfortran.so.3.0.0  libmudflap.a      libmudflapth.so.0.0.0  libssp_nonshared.la   libstdc++.so.6.0.16-gdb.py
libgfortran.spec      libmudflap.la     libquadmath.a          libssp.so             libsupc++.a
libgomp.a             libmudflap.so     libquadmath.la         libssp.so.0           libsupc++.la

All that stuff should be in local/lib, not local/lib64.

comment:175 in reply to: ↑ 174 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

However, I am sure that we will eventually have an error.

I believe this is no longer a problem with #12405.

comment:176 in reply to: ↑ 175 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

However, I am sure that we will eventually have an error.

I believe this is no longer a problem with #12405.

It was a problem yesterday, where I was using the sage-5.0.beta8 sources that contain #12405.

We will see in a few hours...

comment:177 Changed 7 years ago by SimonKing

Concerning gfan: Why is Singular not built with gfan? During configuration of the Singular spkg, it says:

checking whether to configure and build omalloc... yes
checking whether to configure and build gfan lib... no
checking size of long... 8
...
checking which apint package to use... gmp
checking gfanlib... can not build with gfan lib
checking whether to use libsvd... no

Just out of curiosity (I was grepping for gfan in the install.log).

comment:178 in reply to: ↑ 166 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

Following Jeroen's advice, I did ./install base in SAGE_ROOT/spkg. But apparently that has not been sufficient. Now, just for fun, I did make and interrupted after a few seconds -- and when I then tried to install the modified gcc spkg (with --libdir), then the configuration worked.

Strange. So, what was missing in addition to ./install base?

I have no idea. Is patch installed on your system (I mean outside of Sage)?

comment:179 in reply to: ↑ 178 ; follow-up: Changed 7 years ago by SimonKing

Replying to jdemeyer:

I have no idea. Is patch installed on your system (I mean outside of Sage)?

$ which patch
/usr/bin/patch

By the way: It seems that building Sage with gcc from the sources you provided did work. Sage is now building the reference manual. In particular, the gfan spkg did not complain, even though the gcc spkg has put stuff into lib64. That was a single thread, without CFLAGS and without CLooG-PPL.

Next, I'll try parallel build.

comment:180 follow-up: Changed 7 years ago by vbraun

Replying to SimonKing:

Concerning gfan: Why is Singular not built with gfan? During configuration of the Singular spkg, it says:

The Singular developers started their own toric varieties package and gfan is used for that. This is somewhat recent, so our Singular spkg doesn't know about it yet. Right now I think there is nothing that Andrey and I haven't already implemented for toric varieties in Sage. Of course it still would be nice to tie everything together.

comment:181 in reply to: ↑ 180 ; follow-up: Changed 7 years ago by SimonKing

Replying to vbraun:

The Singular developers started their own toric varieties package and gfan is used for that. This is somewhat recent, so our Singular spkg doesn't know about it yet. Right now I think there is nothing that Andrey and I haven't already implemented for toric varieties in Sage. Of course it still would be nice to tie everything together.

Would Singular simply pick gfan up, or would it be needed to have a different configuration in the Singular spkg? In the former case, we could simply build gfan prior to Singular, and then Singular would use gfan.

comment:182 in reply to: ↑ 181 ; follow-up: Changed 7 years ago by vbraun

Replying to SimonKing:

Would Singular simply pick gfan up, or would it be needed to have a different configuration in the Singular spkg? In the former case, we could simply build gfan prior to Singular, and then Singular would use gfan.

To the best of my knowledge they use vanilla gfan, so we just need to build it in the right order.

comment:183 in reply to: ↑ 182 Changed 7 years ago by SimonKing

Replying to vbraun:

To the best of my knowledge they use vanilla gfan, so we just need to build it in the right order.

If I want to test it: Am I right that I have to modify spkg/standard/deps, for example by adding

$(INST)/$(SINGULAR): $(BASE) $(INST)/$(PATCH) $(INST)/$(GFAN)
        +$(PIPE) "$(SAGE_SPKG) $(SINGULAR) 2>&1" "tee -a $(SAGE_LOGS)/$(SINGULAR).log"

?

comment:184 Changed 7 years ago by jdemeyer

Can you make a separate ticket for the Singular/gfan issue? Simon: yes, you need to add $(INST)/$(GFAN) which will ensure Singular is build after gfan.

comment:185 in reply to: ↑ 179 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

It seems that building Sage with gcc from the sources you provided did work.

That's what I suspected. So somehow, something went wrong with your modifications. I would be interested to know exactly what makes it break.

comment:186 follow-up: Changed 7 years ago by jdemeyer

I reproduced Simon' problem with:

tar xf sage-5.0.beta8.tar
cd sage-5.0.beta8/

cd spkg/standard/
wget http://boxen.math.washington.edu/home/jdemeyer/spkg/mpc-1.0.0dev1126.spkg
rm pari-2.5.1.p0.spkg
wget http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.1.p2.spkg
wget http://boxen.math.washington.edu/home/jdemeyer/spkg/gcc-4.6.3.spkg
cd ../../

source spkg/bin/sage-env
export CFLAGS="-march=native" CXXFLAGS="-march=native" MAKE="make -j32"
( cd spkg && ./install base )
./sage -i spkg/standard/mpir-2.1.3.p9.spkg
./sage -i spkg/standard/mpfr-3.1.0.p0.spkg
./sage -i spkg/standard/mpc-1.0.0dev1126.spkg
./sage -i spkg/standard/gcc-4.6.3.spkg

comment:187 in reply to: ↑ 185 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

It seems that building Sage with gcc from the sources you provided did work.

That's what I suspected. So somehow, something went wrong with your modifications. I would be interested to know exactly what makes it break.

I wonder, myself. In one of the posts above, I listed what I did in order to build Sage with gcc "manually".

One question: Is it possible to build Sage by providing the -floop* options? The problem is that my system gcc is not built with graphite, so that the -floop* options would not work before the gcc spkg is built. Is there some option that makes gcc ignore any option that is not supported? Then, the -floop* options would be ignored initially, but used as soon as the gcc spkg is installed.

comment:188 in reply to: ↑ 186 Changed 7 years ago by SimonKing

Replying to jdemeyer:

I reproduced Simon' problem

Good! So, what does the automatic installation do differently from my manual installation?

comment:189 follow-up: Changed 7 years ago by jdemeyer

The fact that you run everything inside one Sage shell is what caused the failures. If you don't use the Sage shell (there is no need to), it works.

sage-env has some conditional commands. One of these commands is setting LD_LIBRARY_PATH. With #12405, only existing directories are added to LD_LIBRARY_PATH. Upon entering the Sage shell, $SAGE_LOCAL/lib does not exist yet, so that directory isn't added. Since now SAGE_ENV_SOURCED=1, the variable LD_LIBRARY_PATH is not changed anymore.

comment:190 Changed 7 years ago by jdemeyer

  • Status changed from needs_work to needs_review

comment:191 in reply to: ↑ 189 Changed 7 years ago by SimonKing

Replying to jdemeyer:

sage-env has some conditional commands. One of these commands is setting LD_LIBRARY_PATH. With #12405, only existing directories are added to LD_LIBRARY_PATH. Upon entering the Sage shell, $SAGE_LOCAL/lib does not exist yet, so that directory isn't added. Since now SAGE_ENV_SOURCED=1, the variable LD_LIBRARY_PATH is not changed anymore.

Arrgh. Thank you for tracing that down!

comment:192 Changed 7 years ago by jdemeyer

See #12698 for the Sage shell issue.

comment:193 follow-up: Changed 7 years ago by SimonKing

I tried to google for an option that makes gcc ignore an option (for example, a -floop* option) if it is not supported, but I had no success. Do you know of such option?

comment:194 in reply to: ↑ 193 Changed 7 years ago by vbraun

Replying to SimonKing:

I tried to google for an option that makes gcc ignore an option (for example, a -floop* option) if it is not supported, but I had no success. Do you know of such option?

There is no such option. The easiest recourse is a wrapper shell script that conditionally removes the option.

comment:195 follow-up: Changed 7 years ago by SimonKing

Here are my experiences with the sources containing gcc. I've put SAGE_INSTALL_GCC=yes in all cases.

  1. No C(XX)FLAGS, MAKE="make": Installation went smooth, after 210m31.660s of real time was the Sage build complete.
  2. No C(XX)FLAGS, SAGE_PARALLEL_SPKG_BUILD=yes, MAKE="make -j4": Installation went smooth, after 93m32.100s of real time was the Sage build complete.
  3. C(XX)FLAGS="-O3 -march=native", SAGE_PARALLEL_SPKG_BUILD=yes, MAKE="make -j4": Installation hung for several hours during compilation of some files from the fplll spkg. Note that So, no success in that setting.

Some additional information on the failing setting: After Ctrl-C, I started make again. And now, three cc1plus processes are running for 10 minutes already, namely

19031 pts/3    R+    10:21 /usr/lib/gcc/x86_64-linux-gnu/4.4.5/cc1plus -quiet -I. -I. -I/mnt/local/king/SAGE/testinggcc/native/sage-5.0.beta8-gcc/local/include/ -I/mnt/local/king/SAGE/testinggcc/native/sage-5.0.beta8-gcc/local/include/ -MD generate.d -MF .deps/generate.Tpo -MP -MT generate.o -D_GNU_SOURCE -DPACKAGE_NAME="libfplll" -DPACKAGE_TARNAME="libfplll" -DPACKAGE_VERSION="3.0.12" -DPACKAGE_STRING="libfplll 3.0.12" -DPACKAGE_BUGREPORT="" -DPACKAGE="libfplll" -DVERSION="3.0.12" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DHAVE_LIBGMP=1 -DHAVE_LIBMPFR=1 -DSTDC_HEADERS=1 -DHAVE_FLOAT_H=1 -DHAVE_LIMITS_H=1 -DHAVE_STDLIB_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_STDBOOL_H=1 -DHAVE_FLOOR=1 -DHAVE_POW=1 -DHAVE_RINT=1 -DHAVE_SQRT=1 -DHAVE_STRTOL=1 -DHAVE_LIBGMP=1 -DHAVE_LIBMPFR=1 generate.cpp -march=core2 -mcx16 -msahf -mpopcnt -msse4.2 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256 -mtune=core2 -quiet -dumpbase generate.cpp -auxbase-strip generate.o -O3 -fPIC -o /tmp/ccPeDeVW.s
19036 pts/3    R+    10:21 /usr/lib/gcc/x86_64-linux-gnu/4.4.5/cc1plus -quiet -I. -I. -I/mnt/local/king/SAGE/testinggcc/native/sage-5.0.beta8-gcc/local/include/ -I/mnt/local/king/SAGE/testinggcc/native/sage-5.0.beta8-gcc/local/include/ -MD llldiff.d -MF .deps/llldiff.Tpo -MP -MT llldiff.o -D_GNU_SOURCE -DPACKAGE_NAME="libfplll" -DPACKAGE_TARNAME="libfplll" -DPACKAGE_VERSION="3.0.12" -DPACKAGE_STRING="libfplll 3.0.12" -DPACKAGE_BUGREPORT="" -DPACKAGE="libfplll" -DVERSION="3.0.12" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DHAVE_LIBGMP=1 -DHAVE_LIBMPFR=1 -DSTDC_HEADERS=1 -DHAVE_FLOAT_H=1 -DHAVE_LIMITS_H=1 -DHAVE_STDLIB_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_STDBOOL_H=1 -DHAVE_FLOOR=1 -DHAVE_POW=1 -DHAVE_RINT=1 -DHAVE_SQRT=1 -DHAVE_STRTOL=1 -DHAVE_LIBGMP=1 -DHAVE_LIBMPFR=1 llldiff.cpp -march=core2 -mcx16 -msahf -mpopcnt -msse4.2 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256 -mtune=core2 -quiet -dumpbase llldiff.cpp -auxbase-strip llldiff.o -O3 -fPIC -o /tmp/ccxiy3ks.s
19044 pts/3    R+    10:01 /usr/lib/gcc/x86_64-linux-gnu/4.4.5/cc1plus -quiet -I. -I. -I/mnt/local/king/SAGE/testinggcc/native/sage-5.0.beta8-gcc/local/include/ -I/mnt/local/king/SAGE/testinggcc/native/sage-5.0.beta8-gcc/local/include/ -MD fplll_micro-main.d -MF .deps/fplll_micro-main.Tpo -MP -MT fplll_micro-main.o -D_GNU_SOURCE -DPACKAGE_NAME="libfplll" -DPACKAGE_TARNAME="libfplll" -DPACKAGE_VERSION="3.0.12" -DPACKAGE_STRING="libfplll 3.0.12" -DPACKAGE_BUGREPORT="" -DPACKAGE="libfplll" -DVERSION="3.0.12" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DHAVE_LIBGMP=1 -DHAVE_LIBMPFR=1 -DSTDC_HEADERS=1 -DHAVE_FLOAT_H=1 -DHAVE_LIMITS_H=1 -DHAVE_STDLIB_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_STDBOOL_H=1 -DHAVE_FLOOR=1 -DHAVE_POW=1 -DHAVE_RINT=1 -DHAVE_SQRT=1 -DHAVE_STRTOL=1 -DHAVE_LIBGMP=1 -DHAVE_LIBMPFR=1 -DFAST_BUILD main.cpp -march=core2 -mcx16 -msahf -mpopcnt -msse4.2 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256 -mtune=core2 -quiet -dumpbase main.cpp -auxbase-strip fplll_micro-main.o -O3 -fPIC -o /tmp/ccGKc3Lc.s

I am not sure whether this means "needs work", because CFLAGS="-O3 -march=native" is probably not officially supported...

comment:196 Changed 7 years ago by SimonKing

After another Ctrl-C, I retried with "serial" make, preserving the CFLAGS. But again, it hangs:

24902 pts/3    R+    12:24 /usr/lib/gcc/x86_64-linux-gnu/4.4.5/cc1plus -quiet -I. -I. -I/mnt/local/king/SAGE/testinggcc/native/sage-5.0.beta8-gcc/local/include/ -I/mnt/local/king/SAGE/testinggcc/native/sage-5.0.beta8-gcc/local/include/ -MD generate.d -MF .deps/generate.Tpo -MP -MT generate.o -D_GNU_SOURCE -DPACKAGE_NAME="libfplll" -DPACKAGE_TARNAME="libfplll" -DPACKAGE_VERSION="3.0.12" -DPACKAGE_STRING="libfplll 3.0.12" -DPACKAGE_BUGREPORT="" -DPACKAGE="libfplll" -DVERSION="3.0.12" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DHAVE_LIBGMP=1 -DHAVE_LIBMPFR=1 -DSTDC_HEADERS=1 -DHAVE_FLOAT_H=1 -DHAVE_LIMITS_H=1 -DHAVE_STDLIB_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_STDBOOL_H=1 -DHAVE_FLOOR=1 -DHAVE_POW=1 -DHAVE_RINT=1 -DHAVE_SQRT=1 -DHAVE_STRTOL=1 -DHAVE_LIBGMP=1 -DHAVE_LIBMPFR=1 generate.cpp -march=core2 -mcx16 -msahf -mpopcnt -msse4.2 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256 -mtune=core2 -quiet -dumpbase generate.cpp -auxbase-strip generate.o -O3 -fPIC -o /tmp/cc7hcSLZ.s

But I just notice that -march=native was actually overridden by -march=core2. Anyway, I lack knowledge to analyse that myself.

comment:197 Changed 7 years ago by SimonKing

With the Sage-with-gcc version built in parallel without CFLAGS, I tried the following innocent-looking example:

u, v = var('u,v')
f_x = cos(u)*(4*sqrt(1-v^2)*sin(abs(
u))^abs(u))
f_y = sin(u) *(4*sqrt(1-v^2)*sin(abs(u))^abs(u))
f_z = v
parametric_plot3d([f_x, f_y, f_z], (u, -pi, pi), (v, -1, 1), frame=False, color="red")

Guess the result? My computer shut down!!!

Is anybody able to replicate it?

comment:198 in reply to: ↑ 195 ; follow-up: Changed 7 years ago by jdemeyer

Replying to SimonKing:

19031 pts/3 R+ 10:21 /usr/lib/gcc/x86_64-linux-gnu/4.4.5/cc1plus

This refers to gcc-4.4.5, which is not the gcc from this package.

Also, could it be that it just takes a very long time to compile those files?

comment:199 in reply to: ↑ 198 Changed 7 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

19031 pts/3 R+ 10:21 /usr/lib/gcc/x86_64-linux-gnu/4.4.5/cc1plus

This refers to gcc-4.4.5, which is not the gcc from this package.

WHAT? O my gosh, then I made a very stupid mistake and forgot to provide the SAGE_INSTALL_GCC in that case. So, I need to try again.

Also, could it be that it just takes a very long time to compile those files?

But not three hours.

comment:200 Changed 7 years ago by SimonKing

  • Owner changed from tbd to (none)

Hooray, building Sage parallelly, using C(XX)FLAGS="-O3 -march=native" apparently did work! The reference manual is already being built.

In other words, the gcc spkg allows one to use "-O3 -march=native". With my system gcc, it wouldn't be possible, since it hangs with libfplll.

comment:201 follow-up: Changed 7 years ago by SimonKing

Here some timings on Debian GNU/Linux 6.0: sage -testall in different settings

With system gcc 4.6.2 (Sage built in parallel, no CFLAGS)

All tests passed!
Total time for all tests: 4951.8 seconds

With the gcc spkg (built serially, no CFLAGS)

All tests passed!
Total time for all tests: 5089.6 seconds

With the gcc spkg (built in parallel, no CFLAGS)

All tests passed!
Total time for all tests: 4974.1 seconds

So, there is no significant difference. Need to repeat with optimization flags, though.

comment:202 in reply to: ↑ 201 Changed 7 years ago by SimonKing

  • Reviewers set to Simon King

Replying to SimonKing:

Here some timings on Debian GNU/Linux 6.0: sage -testall in different settings

...

So, there is no significant difference. Need to repeat with optimization flags, though.

Here it is, with -O3 -march=native and built in parallel:

All tests passed!
Total time for all tests: 4804.3 seconds

So, there is an improvement of 3% compared with my system's gcc without optimization. That's not significant, but at least there is no slow-down.

I can say that this gcc spkg is an interesting contribution, and it works for me. So, from my perspective, it is a positive review. However, I did not test on a machine where the gcc spkg is really needed (OS X, for example). Also, I am not sure if the vote on sage-devel on the inclusion of another standard package has come to a clear conclusion.

Hence, I vote in favour of the spkg and insert my name as one reviewer, but I do think that a second or third opinion is needed, which is why I do not just change the ticket into a positive review.

comment:203 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638 to #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638, #12714, #12647

Rebased to #12714 + #12647

comment:204 Changed 7 years ago by jhpalmieri

Regarding Simon's comment: this has worked for me on OS X Lion for quite a while. I'm far from an expert in gcc and related issues, so I can't speak to any technical aspects of the ticket. But count me for another portion of a positive review. Is this enough?

comment:205 follow-up: Changed 7 years ago by vbraun

  • Status changed from needs_review to positive_review

Enough of the bikeshedding :-)

comment:206 in reply to: ↑ 205 Changed 7 years ago by kcrisman

Replying to vbraun:

Enough of the bikeshedding :-)

True dat. And we still have to add in the new notebook to Sage 5.0, so presumably this will get lots of testing "in real-life situations" from users who haven't yet tried this out much (like me, though it did work well on my infamous machine, which now has no keyboard...). Presumably there are still another few betas to go. This is going to be one amazing release!

comment:207 follow-up: Changed 7 years ago by SimonKing

I have some questions though: How could we make it possible to build the GCC with graphite or with lto-support?

It should be possible to declare that one wants to build the GCC with these additional features - in that case, the necessary optional packages (the cloog_ppl package that I created respectively a libelf package, that probably needs to be created) would be downloaded and installed before building the gcc spkg.

Should that be done by yet another environment variable? I think there are too many already.

But perhaps it could be given as a build target to make. I think the following would be an acceptable way of usage:

  1. tar -xf sage-x.y.z.tar, cd sage-x.y.z/
  2. make cloog_ppl libelf gcc (installs CLooG-PPL, libelf and gcc with their dependencies, downloading optional packages as required; the install script of the gcc spkg would be modified such that graphite and lto support would be automatically added if cloog-ppl or libelf are present)
  3. make, to build the rest of Sage.

Generally, I suggest that make foo_bar toto should install the latest version of the foo_bar and toto spkgs in the given order including dependencies (optional or standard), whereas make without arguments would install all of Sage. Similarly, adding parallel to the make targets should be the same what is now done by SAGE_PARALLEL_SPKG_BUILD. To me, having make targets (automatically derived from spkg names) seems nicer than to have hundreds of environment variables.

How should we proceed with these ideas (provided that you don't say that they're nonsense)? Should there be a new ticket for a gcc with an install script that automatically picks up cloog_ppl and libelf? Or should this be done here, suspending the positive review?

comment:208 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from positive_review to needs_work

comment:209 Changed 7 years ago by jdemeyer

  • Status changed from needs_work to needs_review

Further testing revealed that SAGE_FORTRAN and SAGE_FORTRAN_LIB break stuff. 12369_no_sage_fortran.patch ensures that SAGE_FORTRAN and SAGE_FORTRAN_LIB are ignored when building GCC.

Needs review.

comment:210 Changed 7 years ago by kcrisman

  • Description modified (diff)

comment:211 in reply to: ↑ 207 Changed 7 years ago by jdemeyer

Replying to SimonKing:

How should we proceed with these ideas (provided that you don't say that they're nonsense)? Should there be a new ticket for a gcc with an install script that automatically picks up cloog_ppl and libelf? Or should this be done here, suspending the positive review?

Certainly not here, and I much prefer a ./configure script to make targets:

$ ./configure --with-optional-packages=cloog_ppl,libelf --enable-install-gcc
$ make

I guess having such a ./configure script is a long-term plan for Sage (I think #11073, #10492, #12479, #12602, #12102, #12631 are steps towards that goal).

comment:212 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:213 follow-up: Changed 7 years ago by jhpalmieri

The SAGE_FORTRAN fix doesn't work as is. On the skynet machine cleo, gfortran is some old version of fortran (in /usr/bin), so when you override the setting in /usr/local/skynet_bash_profile, which is

  SAGE_FORTRAN=/usr/local/gcc-4.6.2/ia64-Linux-rhel/bin/gfortran
  export SAGE_FORTRAN

then the prereq script says that there is a version mismatch between gcc and fortran, and quits. Maybe the prereq script shouldn't do this compatibility check if Sage is building gcc?

comment:214 Changed 7 years ago by jhpalmieri

  • Status changed from needs_review to needs_work

comment:215 in reply to: ↑ 213 Changed 7 years ago by jdemeyer

Replying to jhpalmieri:

Maybe the prereq script shouldn't do this compatibility check if Sage is building gcc?

Ideally, things would be the other way around: prereq should become a real configure script and decide whether or not to build GCC. But for now, we should go for a quick fix (I really don't want to overhaul prereq now and make GCC depend on that).

comment:216 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638, #12714, #12647 to #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638, #12714, #12647, #12112

See #12112.

comment:217 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638, #12714, #12647, #12112 to #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638, #12714, #12647, #12739, #12112

comment:218 Changed 7 years ago by SimonKing

Here is another experience:

I patched the gcc spkg as follows:

  • spkg-install

    diff --git a/spkg-install b/spkg-install
    
    a b  
    8282../src/configure \
    8383    --prefix="$SAGE_LOCAL" \
    8484    --with-local-prefix="$SAGE_LOCAL" \
     85    --with-cloog="$SAGE_LOCAL" --with-ppl="$SAGE_LOCAL" \
     86    --enable-lto \
    8587    --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" --with-mpc="$SAGE_LOCAL" \
    8688    --with-system-zlib \
    8789    --disable-multilib \

Then, I added the modified gcc spkg to sage-5.0.beta9, replaced the ppl package by the one from #12672, added the cloog-ppl package from #12666, replaced the glpk package by the one from #12703, added a new libelf spkg and of course also added the mpc spkg. I didn't bother to modify the spkg/deps file, but instead built the gcc spkg manually after installing the dependencies for graphite and lto support.

I have tried to build Sage with C(XX)FLAGS="-O3 -march=native -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing -flto" and LDFLAGS=-flto. Singular did not build (see #12738) and R did not build (see #12741), so that I removed -flto for these two spkgs.

In principle, it worked. sage -testall resulted in

All tests passed!
Total time for all tests: 6515.9 seconds

However, that is significantly slower than the 4951.8 seconds that sage -testall would take on this machine without optimization flags and with gcc 4.6.2.

Note that Leif pointed out that the failures when building Singular and R might come from a slightly too old ld version on my system. But don't be afraid: I do not intend to create a binutils spkg.

comment:219 follow-ups: Changed 7 years ago by jhpalmieri

Simon: what happens if you turn off debugging? I think a lot of Sage packages are built with debugging on by default. Will that conflict with the optimizations at all?

comment:220 in reply to: ↑ 219 Changed 7 years ago by SimonKing

Replying to jhpalmieri:

Simon: what happens if you turn off debugging? I think a lot of Sage packages are built with debugging on by default.

How can I change it externally if Sage packages internally switch debugging on?

Will that conflict with the optimizations at all?

Sorry, that would be far beyond my knowledge. Currently, I am exploring: People mentioned various compiler and linker optimization flags, and I simply wanted to test them, without having an idea what effects are really to be expected.

comment:221 Changed 7 years ago by jhpalmieri

Some packages (I don't know how many or which ones) will respect export SAGE_DEBUG=no.

comment:222 in reply to: ↑ 219 Changed 7 years ago by jdemeyer

Replying to jhpalmieri:

Will that conflict with the optimizations at all?

It shouldn't. Adding debugging information will make your executables a lot larger, which can cause a slow-down. But apart from that, the code should not run slower.

comment:223 Changed 7 years ago by jhpalmieri

By the way, the version of polybori in the testing release doesn't match the version in plain sage-5.0.beta10. The included version fails to compile for me on OpenSolaris and Solaris, while the old version seems to be okay.

comment:224 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:225 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:226 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:227 Changed 7 years ago by jhpalmieri

With the most recent testing release (beta12 vs. beta10), there is a new doctest failure on OS X Lion:

sage -t  --long -force_lib devel/sage/sage/misc/cachefunc.pyx
The doctested process was killed by signal 11

Any ideas what might be causing this? The file being tested hasn't been changed recently as far as I can tell.

comment:228 Changed 7 years ago by jhpalmieri

Should we add #12785 as a prerequisite?

comment:229 Changed 7 years ago by jdemeyer

Could somebody here please review #12782: it would allow building Sage with only a C compiler (and not a C++ compiler). Currently, MPIR is built with a C++ interface which obviously requires a C++ compiler. #12782 changes MPIR such that the C++ interface is not built while building GCC (i.e. when the environment variable SAGE_BUILD_TOOLCHAIN is set to "yes"). Since MPIR is built again after GCC, the C++ interface is built then.

comment:230 Changed 7 years ago by jdemeyer

New testing release sage-5.0.beta12-gcc at the usual location.

comment:231 follow-up: Changed 7 years ago by jhpalmieri

I'm not sure I understand the change related to #3537. If we're upgrading, then sage-env should already have been run (say, by the old sage-sage script), in which case (if it's from an old enough version) RM might be set to rm, which will break things. So would it be better to do

if [ "$RM" = "rm" ]; then
    unset RM
fi

Or maybe

if [ "$RM" = "rm" ]; then
    export RM="rm -f"
fi

comment:232 in reply to: ↑ 231 Changed 7 years ago by jdemeyer

Replying to jhpalmieri:

I'm not sure I understand the change related to #3537. If we're upgrading, then sage-env should already have been run (say, by the old sage-sage script), in which case (if it's from an old enough version) RM might be set to rm, which will break things.

Hmm, you might be right.

comment:233 follow-up: Changed 7 years ago by jdemeyer

Changed that code to

# If spkg/bin/sage-env doesn't exist, we are surely upgrading (sage-env
# was moved to spkg/bin in sage-5.0).  Manually set SAGE_UPGRADING=yes,
# as old versions of sage-upgrade didn't do this.
# Also avoid RM being set to "rm":
#   http://trac.sagemath.org/sage_trac/ticket/3537
if [ ! -f "$SAGE_ROOT/spkg/bin/sage-env" ]; then
    SAGE_UPGRADING=yes
    if [ "$RM" = rm ]; then
        unset RM
    fi
fi

This should remove the need to do "unset RM" in various spkg-install files. This change is unrelated to the ticket, but I put it here because I was patching spkg/install anyway and I didn't want to complicate ticket dependencies.

comment:234 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638, #12714, #12647, #12739, #12112 to #12479, #12602, #12608, #12609, #12647, #10492, #12367, #12368, #12405, #12570, #12574, #12423, #12425, #12456, #12363, #12223, #12515, #12519, #12548, #12562, #12629, #12638, #12714, #12647, #12739, #12112, #12631

Changed 7 years ago by jdemeyer

comment:235 Changed 7 years ago by jdemeyer

  • Status changed from needs_work to needs_review

Rebased 12369_gcc_root.patch to apply on top of #12631. Since this is a simple rebase, I don't think it needs a new review (this change is not yet in the testing release). The GCC spkg has not changed since the last positive_review.

I have tested the testing release on the buildbot and there are no problems. I have not yet properly tested upgrading, but will do that soon.

What needs review: 12369_gcc_root_part2.patch

Changed 7 years ago by jdemeyer

comment:236 Changed 7 years ago by jdemeyer

Tested upgrading from sage-4.5, sage-4.7 and sage-4.8. Everything fine, hg status is clean in the root repositories.

As far as I can tell, there should be no further obstructions to releasing this. So please review the last patch: 12369_gcc_root_part2.patch.

comment:237 Changed 7 years ago by was

  • Status changed from needs_review to positive_review

I read through 12369_gcc_root_part2.patch and it looks great to me (excellent comments!).

comment:238 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.0.beta13
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:239 in reply to: ↑ 109 Changed 7 years ago by leif

Replying to jdemeyer:

Replying to SimonKing:

Or perhaps better exempt the sage* spkgs from re-installation, since my Sage library is patched?

I honestly have no idea what would happen if you re-install the Sage library over a patched version.

That's a noop. Mercurial notices that your current Sage library is based on the one you're trying to reinstall, so won't destroy your patched one. ("Reinstallation" of the Sage library [spkg] almost always happens if you use env SAGE_UPGRADING=yes make after updating/reinstalling some standard spkg.)

comment:240 in reply to: ↑ 233 Changed 7 years ago by leif

Replying to jdemeyer:

Changed that code to

# If spkg/bin/sage-env doesn't exist, we are surely upgrading (sage-env
# was moved to spkg/bin in sage-5.0).  Manually set SAGE_UPGRADING=yes,
# as old versions of sage-upgrade didn't do this.
# Also avoid RM being set to "rm":
#   http://trac.sagemath.org/sage_trac/ticket/3537
if [ ! -f "$SAGE_ROOT/spkg/bin/sage-env" ]; then
    SAGE_UPGRADING=yes
    if [ "$RM" = rm ]; then
        unset RM
    fi
fi

This should remove the need to do "unset RM" in various spkg-install files. This change is unrelated to the ticket, but I put it here because I was patching spkg/install anyway and I didn't want to complicate ticket dependencies.

I'd just unset RM (or actually check whether $RM non-existent-file raises an error), since RM could be defined to rm -i, rm -v, /bin/rm or whatever, and not necessarily intentionally set by the user.

comment:241 Changed 7 years ago by jdemeyer

The new code is

if [ "$RM" = rm ]; then
    export RM="rm -f"
fi

comment:242 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:243 Changed 7 years ago by leif

I get GCC build errors on Ubuntu 12.04 (still beta).

Apart from that Ubuntu's [broken] GCC 4.6.3 is unable to build MPFR with -O3 (which I already reported; bug fix is on the way), the GCC from the spkg doesn't find crti.o. After creating a couple of links in /usr/lib/, the third attempt fails in compiling graphite.c, because of a missing ppl_c.h. (Apparently GCC found some parts of CLooG, or mistakenly assumed it was installed; I didn't have it installed, at least not below /usr/.)

More to come, either here or on sage-release.

comment:244 Changed 7 years ago by jdemeyer

  • Description modified (diff)
Note: See TracTickets for help on using tickets.