Building gcc 7.2.0 fails on CentOS / Red Hat / Fedora Linux systems because of crti.o

[mkoeppe]

On various Linux systemswith old compilers, building gcc 7.2.0 fails because crti.o is not found.

Reported for Red Hat 6.10 at ​https://groups.google.com/d/msg/sage-devel/ze5SzgXZAgk/tai9F4flBQAJ

Also occurs on a different system: CentOS Linux release 7.6.1810 (Core): There is only /lib64/crti.o (same as /usr/lib64/crti.o) and the gcc build does not find it and fails as well. Fix for this system:

export LIBRARY_PATH="/lib64"

before building sage.

[stumpc5]

This is the same issue as I have -- I first also had the "not found" error, which turned into the one I reported after the sysadmin tried to provide that file (I don't know what he actually did).

[mkoeppe]

Replying to stumpc5:

This is the same issue as I have -- I first also had the "not found" error, which turned into the one I reported after the sysadmin tried to provide that file (I don't know what he actually did).

Thanks for the clarification.

[mkoeppe]

This appears to be a frequent problem. See for example ​https://stackoverflow.com/questions/91576/crti-o-file-missing and ​https://github.com/Homebrew/linuxbrew-core/issues/7172

On the CentOS system mentioned in the ticket's description, the system's gcc is actually configured correctly: gcc -print-search-dirs has a long list of paths that includes /lib64 (= /usr/lib64 on this system).

The folks at Linuxbrew do the following in ​https://github.com/Homebrew/linuxbrew-core/blob/master/Formula/gcc@7.rb:

        # Set the search path for glibc libraries and objects, using the system's glibc
        # Fix the error: ld: cannot find crti.o: No such file or directory
        ENV.prepend_path "LIBRARY_PATH", Pathname.new(Utils.popen_read(ENV.cc, "-print-file-name=crti.o")).parent

On the CentOS system I can confirm that gcc -print-file-name=crti.o gives the correct file name. So this may be a sound approach.

[mkoeppe]

New commits:

​e8c0e64

build-gcc: Find crti.o via ${CC} -print-file-name

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​169fdf2

Add to end of LIBRARY_PATH, not front

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cdf180d

LIBRARY_PATH is needed in spkg-install too

[stumpc5]

Thank your for working on this! Unfortunately, your fix does not yet solve it for me:

[stumpcl2@dlp762 sage]$ CRTI=$(${CC:-gcc} -print-file-name=crti.o 2>/dev/null || true)
[stumpcl2@dlp762 sage]$ echo $CRTI
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/crti.o
[stumpcl2@dlp762 sage]$ if [ -n "$CRTI" ]; then
>  CRTI_DIR=$(dirname -- "$CRTI")
>  if [ -n "$CRTI_DIR" -a "$CRTI_DIR" != "." ]; then
>  export LIBRARY_PATH="$LIBRARY_PATH:$CRTI_DIR"
>  fi
> fi
[stumpcl2@dlp762 sage]$ echo $CRTI_DIR
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64
[stumpcl2@dlp762 sage]$ echo $LIBRARY_PATH
/lib64:/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64

while building from scratch yields the error

[gcc-7.2.0] /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/../lib/crti.o: could not read symbols: File in wrong format

So it leaves lib64 again and goes back to lib to look for crti.o.

[mkoeppe]

I'd suggest to first check with your sysadmin what it is that he did to try to fix the original problem and to undo that.

[stumpc5]

I'd suggest to first check with your sysadmin what it is that he did to try to fix the original problem and to undo that.

Well, he installed gcc-7.2.0 systemwide (and then also gcc-7.3.0). If I use either of these (making it available using ini gcc), I get the errors I originally reported on sage-devel (gfan-0.6.2.p1, ppl-1.2.p1). If I do not use that gcc, but the older gcc-4.4.7, the not-found-error turned into the given could-not-read-symbols-error.

[mkoeppe]

OK, thanks for the clarification

[mkoeppe]

Needs more work also for one of my systems, where it succeeds in building and installing gcc, but that gcc does not work without setting the LIBRARY_PATH variable during its runtime!

[mkoeppe]

Replying to mkoeppe:

OK, thanks for the clarification

Could you try the following:

file /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/crt*.o

[mkoeppe]

Replying to mkoeppe:

Needs more work also for one of my systems, where it succeeds in building and installing gcc, but that gcc does not work without setting the LIBRARY_PATH variable during its runtime!

Possible solution, again from Linuxbrew:

      gcc = bin/"gcc-7"
      libgcc = Pathname.new(Utils.popen_read(gcc, "-print-libgcc-file-name")).parent
      raise "command failed: #{gcc} -print-libgcc-file-name" if $CHILD_STATUS.exitstatus.nonzero?

      glibc = Formula["glibc"]
      glibc_installed = glibc.any_version_installed?

      # Symlink crt1.o and friends where gcc can find it.
      if glibc_installed
        crtdir = glibc.opt_lib
      else
        crtdir = Pathname.new(Utils.popen_read("/usr/bin/cc", "-print-file-name=crti.o")).parent
      end
      ln_sf Dir[crtdir/"*crt?.o"], libgcc

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f57d00c	Find SAGE_CRTI_DIR at configure time.
​73be6cf	Symlink crt1.o and friends where gcc can find it

[stumpc5]

Based on this 73be6cf, the error becomes

[gcc-7.2.0] /usr/lib/../lib/crti.o: could not read symbols: File in wrong format

with

[stumpcl2@dlp762 sage]$ file /usr/lib/../lib/crti.o
/usr/lib/../lib/crti.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped

[mkoeppe]

But what does file /usr/lib64/crt*.o give?

[stumpc5]

[stumpcl2@dlp762 sage]$ file /usr/lib64/crt*.o
/usr/lib64/crt1.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), for GNU/Linux 2.6.18, not stripped
/usr/lib64/crti.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
/usr/lib64/crtn.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), stripped

[mkoeppe]

Could you attach the latest .log file?

[mkoeppe]

The current version works on the CentOS Linux system where I first encountered this problem.

[stumpc5]

Replying to mkoeppe:

Could you attach the latest .log file?

posted on sage-devel

[mkoeppe]

Looks like configure was not regenerated. When you get a chance, please run ./bootstrap (this needs autotools installed...) If it still fails, please post both config.log and the gcc-7.2.0.log

[stumpc5]

This now also works for me compiling gcc-7.2.0, many thanks for fixing this! (Unfortunately, the next error occurred right after on Red Hat 6.10.

[mkoeppe]

Let's please get this into the next beta so that this can be tested by people without having to regenerate the configure scripts.

[stumpc5]

Yes, I agree. But do you see a way to auto-generate

# Location of system crti.o, in case we build our own gcc
export SAGE_CRTI_DIR="/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64"

which I tweaked by hand?

[mkoeppe]

Replying to stumpc5:

Yes, I agree. But do you see a way to auto-generate

# Location of system crti.o, in case we build our own gcc
export SAGE_CRTI_DIR="/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64"

which I tweaked by hand?

The scripts already do all of that. Something is broken in your setup that stops your copy of sage from regenerating the autotools scripts correctly. For example, in the last set of logs you posted there is no trace of the crti test at configure time. After make distclean your copy of sage likely downloaded the configure scripts from the latest beta, instead of regenerating them.

[dimpase]

could it be that the machine either has no autotools installed, or they are outdated/broken?

[mkoeppe]

Replying to dimpase:

could it be that the machine either has no autotools installed, or they are outdated/broken?

Yes, that's my guess too.

In any case, I have tested this ticket on one of my machines that had the same problem.

[embray]

Regarding the spkg-postinst, could you please add a complementary spkg-postrm which removes the symlink, if it exists?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d8e9adf

Remove symlinks in postrm

[mkoeppe]

New commits:

​d8e9adf

Remove symlinks in postrm

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cd9a0e8

Fixup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9e3da9b

Fix up

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a04ea6b

Fix up

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0558eef

Fix up

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4f08015

Define SAGE_SPKG_SCRIPTS in build scripts

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​18300da

Fix postrm additions

[mkoeppe]

Replying to embray:

Regarding the spkg-postinst, could you please add a complementary spkg-postrm which removes the symlink, if it exists?

Took me a while to find my way through this little maze of scripts. Seems to work now, the crt symlinks are now removed by the postrm script. Please take a look.

I notice that after make gcc-clean, an empty chain of directories gcc/x86_64-pc-linux-gnu/7.2.0/plugin/include/ada remains under $SAGE_LOCAL/lib. I think it's unrelated to this ticket.

[mkoeppe]

Needs review.

[embray]

I'm confused. What you used to have as spkg-postinst you renamed to spkg-postrm, which is not what I intended.

It also now reads:

-spkg-postinst
\ No newline at end of file
+# -*- mode: shell-script -*-
+
+# Force re-installation of gmp, mpir, mpfr and mpc with the GCC that we
+# just built. This is needed in particular because gmp/mpir was first
+# built without C++ support.

This script is run after you uninstall a package, so the comment about "the GCC that we just built" can't be accurate (it was accurate for when this was a postinst script.

I will have to take more time to actually understand what you're trying to do here. As it is I believe we are spending too much technical debt on trying to maintain gcc builds for old platforms...

[embray]

Also I wonder if we could base this on top of #21707? (Not to add additional hurdles to this ticket, as I want to move forward on #21707 ASAP as the number of build-related variables in sage-env-config is starting to get out of hand :)

[embray]

Went ahead and rebased on current develop so it could incorporate all the additional variables we've since added.

I didn't change the name of the script yet; I would still prefer to shorten it but if anyone disagrees we can leave it as-is.

---

New commits:

​a34a9f1	Move sage-dist-helpers from src/bin to build/bin
​ff07121	Split out build/bin/sage-build-env-config from sage-env-config

[embray]

Ack! Wrong ticket, sorry.

---

Last 10 new commits:

​cdf180d	LIBRARY_PATH is needed in spkg-install too
​f57d00c	Find SAGE_CRTI_DIR at configure time.
​73be6cf	Symlink crt1.o and friends where gcc can find it
​d8e9adf	Remove symlinks in postrm
​cd9a0e8	Fixup
​9e3da9b	Fix up
​a04ea6b	Fix up
​0558eef	Fix up
​4f08015	Define SAGE_SPKG_SCRIPTS in build scripts
​18300da	Fix postrm additions

[vbraun]

On fedora 30 (gcc 9) I'm also getting

[gcc-7.2.0] /home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/var/tmp/sage/build/gcc-7.2.0/gcc-build/./gcc/xgcc -B/home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/var/tmp/sage/build/gcc-7.2.0/gcc-build/./gcc/ -B/home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/x86_64-pc-linux-gnu/bin/ -B/home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/x86_64-pc-linux-gnu/lib/ -isystem /home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/x86_64-pc-linux-gnu/include -isystem /home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/x86_64-pc-linux-gnu/sys-include    -O2  -g -O2 -DIN_GCC    -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition  -isystem ./include   -fpic -mlong-double-80 -DUSE_ELF_SYMVER -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector  -shared -nodefaultlibs -Wl,--soname=libgcc_s.so.1 -Wl,--version-script=libgcc.map -o ./libgcc_s.so.1.tmp -g -O2 -B./ _muldi3_s.o _negdi2_s.o _lshrdi3_s.o _ashldi3_s.o _ashrdi3_s.o _cmpdi2_s.o _ucmpdi2_s.o _clear_cache_s.o _trampoline_s.o __main_s.o _absvsi2_s.o _absvdi2_s.o _addvsi3_s.o _addvdi3_s.o _subvsi3_s.o _subvdi3_s.o _mulvsi3_s.o _mulvdi3_s.o _negvsi2_s.o _negvdi2_s.o _ctors_s.o _ffssi2_s.o _ffsdi2_s.o _clz_s.o _clzsi2_s.o _clzdi2_s.o _ctzsi2_s.o _ctzdi2_s.o _popcount_tab_s.o _popcountsi2_s.o _popcountdi2_s.o _paritysi2_s.o _paritydi2_s.o _powisf2_s.o _powidf2_s.o _powixf2_s.o _mulhc3_s.o _mulsc3_s.o _muldc3_s.o _mulxc3_s.o _divhc3_s.o _divsc3_s.o _divdc3_s.o _divxc3_s.o _bswapsi2_s.o _bswapdi2_s.o _clrsbsi2_s.o _clrsbdi2_s.o _fixunssfsi_s.o _fixunsdfsi_s.o _fixunsxfsi_s.o _fixsfdi_s.o _fixdfdi_s.o _fixxfdi_s.o _fixunssfdi_s.o _fixunsdfdi_s.o _fixunsxfdi_s.o _floatdisf_s.o _floatdidf_s.o _floatdixf_s.o _floatundisf_s.o _floatundidf_s.o _floatundixf_s.o _divdi3_s.o _moddi3_s.o _divmoddi4_s.o _udivdi3_s.o _umoddi3_s.o _udivmoddi4_s.o _udiv_w_sdiv_s.o cpuinfo_s.o sfp-exceptions_s.o addtf3_s.o divtf3_s.o multf3_s.o negtf2_s.o subtf3_s.o unordtf2_s.o fixtfsi_s.o fixunstfsi_s.o floatsitf_s.o floatunsitf_s.o fixtfdi_s.o fixunstfdi_s.o floatditf_s.o floatunditf_s.o fixtfti_s.o fixunstfti_s.o floattitf_s.o floatuntitf_s.o extendsftf2_s.o extenddftf2_s.o extendxftf2_s.o trunctfsf2_s.o trunctfdf2_s.o trunctfxf2_s.o getf2_s.o letf2_s.o eqtf2_s.o _divtc3_s.o _multc3_s.o _powitf2_s.o enable-execute-stack_s.o unwind-dw2_s.o unwind-dw2-fde-dip_s.o unwind-sjlj_s.o unwind-c_s.o emutls_s.o libgcc.a -lc && rm -f ./libgcc_s.so && if [ -f ./libgcc_s.so.1 ]; then mv -f ./libgcc_s.so.1 ./libgcc_s.so.1.backup; else true; fi && mv ./libgcc_s.so.1.tmp ./libgcc_s.so.1 && (echo "/* GNU ld script"; echo "   Use the shared library, but some functions are only in"; echo "   the static library.  */"; echo "GROUP ( libgcc_s.so.1 -lgcc )" ) > ./libgcc_s.so
[gcc-7.2.0] /usr/bin/ld: cannot find crti.o: No such file or directory
[gcc-7.2.0] collect2: error: ld returned 1 exit status
[gcc-7.2.0] make[9]: *** [Makefile:982: libgcc_s.so] Error 1

For the record, its /lib64/crti.o

[release@zen Sage]$ cc -print-file-name=crti.o
/usr/lib/gcc/x86_64-redhat-linux/9/../../../../lib64/crti.o

[mkoeppe]

Replying to vbraun:

On fedora 30 (gcc 9) I'm also getting

[gcc-7.2.0] /home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/var/tmp/sage/build/gcc-7.2.0/gcc-build/./gcc/xgcc -B/home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/var/tmp/sage/build/gcc-7.2.0/gcc-build/./gcc/ -B/home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/x86_64-pc-linux-gnu/bin/ -B/home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/x86_64-pc-linux-gnu/lib/ -isystem /home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/x86_64-pc-linux-gnu/include -isystem /home/buildbot-sage/slave/binary_pkg/build/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas39/local/x86_64-pc-linux-gnu/sys-include    -O2  -g -O2 -DIN_GCC    -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition  -isystem ./include   -fpic -mlong-double-80 -DUSE_ELF_SYMVER -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector  -shared -nodefaultlibs -Wl,--soname=libgcc_s.so.1 -Wl,--version-script=libgcc.map -o ./libgcc_s.so.1.tmp -g -O2 -B./ _muldi3_s.o _negdi2_s.o _lshrdi3_s.o _ashldi3_s.o _ashrdi3_s.o _cmpdi2_s.o _ucmpdi2_s.o _clear_cache_s.o _trampoline_s.o __main_s.o _absvsi2_s.o _absvdi2_s.o _addvsi3_s.o _addvdi3_s.o _subvsi3_s.o _subvdi3_s.o _mulvsi3_s.o _mulvdi3_s.o _negvsi2_s.o _negvdi2_s.o _ctors_s.o _ffssi2_s.o _ffsdi2_s.o _clz_s.o _clzsi2_s.o _clzdi2_s.o _ctzsi2_s.o _ctzdi2_s.o _popcount_tab_s.o _popcountsi2_s.o _popcountdi2_s.o _paritysi2_s.o _paritydi2_s.o _powisf2_s.o _powidf2_s.o _powixf2_s.o _mulhc3_s.o _mulsc3_s.o _muldc3_s.o _mulxc3_s.o _divhc3_s.o _divsc3_s.o _divdc3_s.o _divxc3_s.o _bswapsi2_s.o _bswapdi2_s.o _clrsbsi2_s.o _clrsbdi2_s.o _fixunssfsi_s.o _fixunsdfsi_s.o _fixunsxfsi_s.o _fixsfdi_s.o _fixdfdi_s.o _fixxfdi_s.o _fixunssfdi_s.o _fixunsdfdi_s.o _fixunsxfdi_s.o _floatdisf_s.o _floatdidf_s.o _floatdixf_s.o _floatundisf_s.o _floatundidf_s.o _floatundixf_s.o _divdi3_s.o _moddi3_s.o _divmoddi4_s.o _udivdi3_s.o _umoddi3_s.o _udivmoddi4_s.o _udiv_w_sdiv_s.o cpuinfo_s.o sfp-exceptions_s.o addtf3_s.o divtf3_s.o multf3_s.o negtf2_s.o subtf3_s.o unordtf2_s.o fixtfsi_s.o fixunstfsi_s.o floatsitf_s.o floatunsitf_s.o fixtfdi_s.o fixunstfdi_s.o floatditf_s.o floatunditf_s.o fixtfti_s.o fixunstfti_s.o floattitf_s.o floatuntitf_s.o extendsftf2_s.o extenddftf2_s.o extendxftf2_s.o trunctfsf2_s.o trunctfdf2_s.o trunctfxf2_s.o getf2_s.o letf2_s.o eqtf2_s.o _divtc3_s.o _multc3_s.o _powitf2_s.o enable-execute-stack_s.o unwind-dw2_s.o unwind-dw2-fde-dip_s.o unwind-sjlj_s.o unwind-c_s.o emutls_s.o libgcc.a -lc && rm -f ./libgcc_s.so && if [ -f ./libgcc_s.so.1 ]; then mv -f ./libgcc_s.so.1 ./libgcc_s.so.1.backup; else true; fi && mv ./libgcc_s.so.1.tmp ./libgcc_s.so.1 && (echo "/* GNU ld script"; echo "   Use the shared library, but some functions are only in"; echo "   the static library.  */"; echo "GROUP ( libgcc_s.so.1 -lgcc )" ) > ./libgcc_s.so
[gcc-7.2.0] /usr/bin/ld: cannot find crti.o: No such file or directory
[gcc-7.2.0] collect2: error: ld returned 1 exit status
[gcc-7.2.0] make[9]: *** [Makefile:982: libgcc_s.so] Error 1

For the record, its /lib64/crti.o

[release@zen Sage]$ cc -print-file-name=crti.o
/usr/lib/gcc/x86_64-redhat-linux/9/../../../../lib64/crti.o

Has CRTI_DIR been set correctly in src/bin/sage-env-config on this system?

[mkoeppe]

Replying to embray:

I'm confused. What you used to have as spkg-postinst you renamed to spkg-postrm, which is not what I intended.

Before this ticket, spkg-postrm was a symlink to spkg-postinst. In both spkg-postrm and spkg-postinst, the same actions (trigger uninstalls, reconfigure) are taken.

I added additional things to the front of spkg-postinst.

(The diff does not show this very well.)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8ed2677

build/pkgs/gcc/spkg-postrm: Update comments

[mkoeppe]

Replying to embray:

[...] spkg-postrm [...] is run after you uninstall a package, so the comment about "the GCC that we just built" can't be accurate

I have updated the explanation in spkg-postrm. (Actually, in the spkg-postrm situation, I don't know why --keep-files would be used; but it's beyond the scope of this ticket to change it.)

[embray]

Got it--this is starting to make sense to me now. The part where you actually modify the the postrm script on the fly is interesting: That was not an explicitly supported use case, so it makes me a little nervous (since it means now we do have to support that), but if it works it works.

If for some reason we need to change things such that it would not be easy to support this usage, I believe a different solution could be found.

[embray]

Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).

[vbraun]

I would just unconditionally delete *crt?.o in the postrm script, more predictable and fewer moving parts. But either way is imho an acceptable solution, unless Eric objects I'm going to go ahead with this ticket....

[mkoeppe]

@stumpc5 Merged in 8.9.beta4, please test