Update GMP-ECM to version 7.0

[zimmerma]

GMP-ECM 7.0.4 is available at ​https://gforge.inria.fr/frs/download.php/file/36224/ecm-7.0.4.tar.gz.

---

(This was referring to version 7.0:)

Some issues were reported for the Debian package:

​https://buildd.debian.org/status/package.php?p=gmp-ecm, in particular on 32-bit PowerPC and FreeBSD. Is there any such machine on the Sage build farm?

---

Upgrade GMP-ECM to 7.0.4

• I haven't repackaged the upstream tarball (.tar.gz) [yet]. Not sure whether we could/should also delete some stuff not needed for Sage (cf. SPKG.txt); at least there's no spkg-src script.

• I haven't touched spkg-install yet. We may address potential issues with SAGE_FAT_BINARY (mentioned both there and in SPKG.txt), as well as known problems with -march=native -O3 on bdver[1-4] (AMD FX-???? Bulldozer/Piledriver, AMD Opteron 6[23]xx, etc.) where we should add -mno-xop because of bugs in (AFAIK) every GCC version that supports these processors, silently leading to broken code ("just" testsuite failures IIRC). Also, we currently copy Sage's config.guess (because it was newer at some point), which presumably is no longer necessary (but shouldn't hurt either, provided we keep Sage's up-to-date and GMP-ECM won't rely on its own).

• The doctests in sage/libs/libecm.pyx may need some changes, especially with respect to verbose mode tests. (But I'll presumably work on them elsewhere anyway, independently of this upgrade.)

• We definitely also have to fix the pexpect interface to ecm in src/sage/interfaces/ecm.py, as the command line interface has changed in GMP-ECM 7.x.

[leif]

Cool!

[leif]

Replying to zimmerma:

Some issues were reported for the Debian package: ​https://buildd.debian.org/status/package.php?p=gmp-ecm, in particular on 32-bit PowerPC and FreeBSD. Is there any such machine on the Sage build farm?

I fail to see failures there by now (the version listed there also is 7.0.3+ds-1 though).

And no, AFAICT we don't have any PowerPC (neither 32-bit nor 64-bit) nor any FreeBSD buildbot.

I guess François regularly tests on Linux PowerPC64 though, and I think Jeroen still has access to some PowerPC running MacOS X 10.5 (we might no longer support; AFAIK 32-bit OS, not sure, but 64-bit CPU).

[leif]

"Upgraded" to version 7.0.3 (of July 4th, currently the latest available).

(Strange that this version and 7.0.2 apparently haven't been tagged in the svn repo.)

[leif]

According to Paul's comment, we can drop the only patch (from #19233) we currently have in Sage.

[leif]

Should we repackage the tarball with bzip2?

[leif]

Replying to leif:

Should we repackage the tarball with bzip2?

900K	ecm-7.0.3.tar.bz2
1.1M	ecm-7.0.3.tar.gz

[leif]

Note that the patch is preliminary (cf. the commit message / ticket description), but I'm setting it to "needs review" as there's something that can already be tested.

---

New commits:

​c0f7846

GMP-ECM: #20385: Upgrade package to 7.0.3 (preliminary patch, see below)

[leif]

Replying to leif's ticket description:

• I haven't touched spkg-install yet. We may address [...] known problems with -march=native -O3 on bdver[1-4] (AMD FX-???? Bulldozer/Piledriver, AMD Opteron 6[23]xx, etc.) where we should add -mno-xop because of bugs in (AFAIK) every GCC version that supports these processors, silently leading to broken code ("just" testsuite failures IIRC).

This is still true for 7.0.3 (and Ubuntu's GCC 4.6.3 at least), while what used to be make check is now make longcheck:

...
GMP-ECM 7.0.3 [configured with GMP 5.1.1, --enable-asm-redc] [P+1]
Input number is 2277189375098448170118558775447117254551111605543304035536750762506158547102293199086726265869065639109 (103 digits)
Using B1=1000000, B2=0, polynomial x^1, x0=3
Step 1 took 584ms
GMP-ECM 7.0.3 [configured with GMP 5.1.1, --enable-asm-redc] [P+1]
############### ERROR ###############
Expected return code 14 but got 0
make: *** [longcheck] Error 1

I think we should do make check && make longcheck in spkg-check then.

[leif]

Ooops, on the other hand make check now fails with -march=native -mno-xop -O3 -g (while it passed without -mno-xop?!?!):

FAIL: test.pp1
PASS: test.pm1
PASS: test.ecm
============================================================================
Testsuite summary for ecm 7.0.3
============================================================================
# TOTAL: 3
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to ecm-discuss@lists.gforge.inria.fr
============================================================================
make[3]: *** [test-suite.log] Error 1
make[3]: Leaving directory `/home/leif/build/ecm-7.0.3-build.bulldozer-gcc-4.6.3-default'
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory `/home/leif/build/ecm-7.0.3-build.bulldozer-gcc-4.6.3-default'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/home/leif/build/ecm-7.0.3-build.bulldozer-gcc-4.6.3-default'
make: *** [check-recursive] Error 1

test-suite.log:

...
GMP-ECM 7.0.3 [configured with GMP 5.1.1, --enable-asm-redc] [P+1]
Input number is 2277189375098448170118558775447117254551111605543304035536750762506158547102293199086726265869065639109 (103 digits)
Using B1=2337233, B2=173055082, polynomial x^1, x0=3
Step 1 took 1368ms
Step 2 took 104ms
********** Factor found in step 2: 4190453151940208656715582382315221647
Found prime factor of 37 digits: 4190453151940208656715582382315221647
Prime cofactor 543423179434447039008165356160798838947285203071935410761431031147 has 66 digits
GMP-ECM 7.0.3 [configured with GMP 5.1.1, --enable-asm-redc] [P+1]
Save file test.pp1.save already exists, will not overwrite
############### ERROR ###############
Expected return code 0 but got 1
FAIL test.pp1 (exit status: 1)

(make longcheck fails in the same way as it apparently also performs the same test(s).)

[leif]

Hmmm, or maybe not, and make clean was just not enough.

After make distclean, make check again passes with -march=native -mtune=native -mno-xop -O3 -g in gmp.h (but now also --enable-assert).

But in make longcheck, I'm now getting a different, apparently unrelated error:

...
GMP-ECM 7.0.3 [configured with GMP 5.1.1, --enable-asm-redc, --enable-assert] [P+1]
Input number is 18446744073709551337 (20 digits)
Using B1=70823, B2=1320588, polynomial x^1, x0=2
Step 1 took 16ms
********** Factor found in step 1: 18446744073709551337
Found input number N
../../src/ecm-7.0.3/test.pp1: 151: ../../src/ecm-7.0.3/test.pp1: cannot open ./c155: No such file
############### ERROR ###############
Expected return code 0 but got 2
make: *** [longcheck] Error 1

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​64eb69e

GMP-ECM: #20385: (Also) run 'make longcheck' in spkg-check

[leif]

It seems make longcheck needs some upstream work, as I'm also getting (ignored!) "meta-errors" (syntax errors etc.) with a clean, fresh build on another (non-AMD) machine.

[leif]

Replying to leif:

It seems make longcheck needs some upstream work, as I'm also getting (ignored!) "meta-errors" (syntax errors etc.) with a clean, fresh build on another (non-AMD) machine.

Apparently also building out-of-tree is broken (more precisely, running make longcheck in an out-of-tree build, which definitely fails).

[fbissey]

Looks like I wasn't on cc even so Paul explicitly added me when the ticket was created. I missed all your action from the last few hours. My gentoo ebuild passes "make check" but then I don't do fancy flags. Will try "make longcheck". Before the release Paul opened a conversation on the last ecm upgrade ticket. Apart from the doctest there is interface change to deal with ​https://trac.sagemath.org/ticket/14151#comment:46

[leif]

Replying to fbissey:

Looks like I wasn't on cc even so Paul explicitly added me when the ticket was created. I missed all your action from the last few hours.

Hmmm, perhaps recheck your trac notification preferences as you already commented on this ticket, too (so you should by default receive notifications even when not explicitly cc'ed).

My gentoo ebuild passes "make check" but then I don't do fancy flags. Will try "make longcheck". Before the release Paul opened a conversation on the last ecm upgrade ticket. Apart from the doctest there is interface change to deal with ​https://trac.sagemath.org/ticket/14151#comment:46

And although I participated in that old ticket, I somehow missed all that discussion from a couple of months ago...

I have to admit I totally forgot Sage's pexpect interface to ecm.

Is the execstack issue meanwhile resolved upstream?

[fbissey]

Replying to leif:

Replying to fbissey:

Is the execstack issue meanwhile resolved upstream?

Nope. I have added myself to cc because I din't get your last message either.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0affd29

GMP-ECM: #20385: Restore note on work-around in SPKG.txt that accidentally got dropped in the 6.4.4 package.

[leif]

Hope you get this now, as the cc field was still empty...

[leif]

Added command line interface change to be addressed.

[leif]

Replying to fbissey:

Replying to leif:

Is the execstack issue meanwhile resolved upstream?

Nope.

Should we add -Wl,-z,noexecstack in spkg-install until that's fixed upstream?

[fbissey]

Replying to leif:

Replying to fbissey:

Replying to leif:

Is the execstack issue meanwhile resolved upstream?

Nope.

Should we add -Wl,-z,noexecstack in spkg-install until that's fixed upstream?

Yes. Upstream is open to a patch. Problem is the files are generated so you have to carefully fix the generator :(

Still no notification, I'll have to inspect the spam filter tomorrow but I suspect it is not even making it there. May have to change the email address for my notification. And go to the dashboard to see if I missed updates on other tickets.

[fbissey]

My goodness! notification this time.

[leif]

So after N attempts, I finally built GMP-ECM in-place two times (separate folders) on a Bulldozer (Opteron 6276), once with -march=native -mtune=native -O3 -g, and once with -march=native -mtune=native -mno-xop -O3 -g.

In both cases, make check and make longcheck succeeded now.

(This presumably means the relevant code in GMP-ECM 6.x has changed enough to no longer expose the bugs in GCC when mixing XOP and AVX instructions at higher optimization levels. That's not too surprising, as even minor changes made the issue vanish when I tried (hard!) to create a testcase for bugzilla.)

[leif]

Replying to fbissey:

Problem is the files are generated so you have to carefully fix the generator :(

? LDFLAGS? Perhaps CCASFLAGS?

Actually, upstream should add a configure check and add the appropriate flags when supported.

Do we have to care about anything but Linux (x86/x86_64/ppc/ppc64/SPARC...)?

Anything but GNU ld? (Yes, on Solaris, the Sun linker may get used, but Solaris' not Linux.)

[fbissey]

LDFLAGS, linux x86(_64) only I think. As for ld I still suffer AIX boxes on ppc64 but sage won't build over there anyway (building python packages need hand holding for starter).

[leif]

GMP (6.1.1) has this:

dnl Checks whether the stack can be marked nonexecutable by passing an option
dnl to the C-compiler when acting on .s files. Appends that option to ASMFLAGS.
dnl This macro is adapted from one found in GLIBC-2.3.5.
dnl FIXME: This test looks broken. It tests that a file with .note.GNU-stack...
dnl can be compiled/assembled with -Wa,--noexecstack.  It does not determine
dnl if that command-line option has any effect on general asm code.
AC_DEFUN([CL_AS_NOEXECSTACK],[
dnl AC_REQUIRE([AC_PROG_CC]) GMP uses something else
AC_CACHE_CHECK([whether assembler supports --noexecstack option],
cl_cv_as_noexecstack, [dnl
  cat > conftest.c <<EOF
void foo() {}
EOF
  if AC_TRY_COMMAND([${CC} $CFLAGS $CPPFLAGS
                     -S -o conftest.s conftest.c >/dev/null]) \
     && grep .note.GNU-stack conftest.s >/dev/null \
     && AC_TRY_COMMAND([${CC} $CFLAGS $CPPFLAGS -Wa,--noexecstack
                       -c -o conftest.o conftest.s >/dev/null])
  then
    cl_cv_as_noexecstack=yes
  else
    cl_cv_as_noexecstack=no
  fi
  rm -f conftest*])
  if test "$cl_cv_as_noexecstack" = yes; then
    ASMFLAGS="$ASMFLAGS -Wa,--noexecstack"
  fi
  AC_SUBST(ASMFLAGS)
])

Note that --noexecstack gets eventually passed to the assembler, not -z noexecstack to the linker.

[fbissey]

Ha, when I was looking at 7.0rc with Paul I only had gmp-6.1.0 and I still had a message about the executable stack from portage if I didn't add -Wl,-z,noexecstack to LDFLAGS. I moved to 6.1.1 on 21st of June. And indeed I don't need to add it anymore, it is added automatically:

libtool: link: x86_64-pc-linux-gnu-gcc -shared  -fPIC -DPIC  .libs/libecm_la-ecm.o .libs/libecm_la-ecm2.o .libs/libecm_la-pm1.o .libs/libecm_la-pp1.o .libs/libecm_la-getprime_r.o .libs/libecm_la-listz.o .libs/libecm_la-lucas.o .libs/libecm_la-stage2.o .libs/libecm_la-mpmod.o .libs/libecm_la-mul_lo.o .libs/libecm_la-polyeval.o .libs/libecm_la-median.o .libs/libecm_la-schoen_strass.o .libs/libecm_la-ks-multiply.o .libs/libecm_la-rho.o .libs/libecm_la-bestd.o .libs/libecm_la-auxlib.o .libs/libecm_la-random.o .libs/libecm_la-factor.o .libs/libecm_la-sp.o .libs/libecm_la-spv.o .libs/libecm_la-spm.o .libs/libecm_la-mpzspm.o .libs/libecm_la-mpzspv.o .libs/libecm_la-ntt_gfp.o .libs/libecm_la-ecm_ntt.o .libs/libecm_la-pm1fs2.o .libs/libecm_la-sets_long.o .libs/libecm_la-auxarith.o .libs/libecm_la-batch.o .libs/libecm_la-parametrizations.o .libs/libecm_la-cudawrapper.o aprtcle/.libs/libecm_la-mpz_aprcl.o  -Wl,--whole-archive ./x86_64/.libs/libmulredc.a -Wl,--no-whole-archive  -lgmp -lrt -lm  -g -O1 -march=native -ggdb -g -Wl,-znoexecstack -Wl,-O1 -Wl,--as-needed -O1 -march=native -ggdb   -Wl,-soname -Wl,libecm.so.1 -o .libs/libecm.so.1.0.0

[leif]

I was just about to note that GMP-ECM consistently uses CCAS (which defaults to CC) and CCASFLAGS (which default to CFLAGS)...

[fbissey]

In any case sage's default is mpir so we should pass it in the default case. Passing it with gmp-6.1.1 doesn't really hurt.

[leif]

Oh, GMP-ECM just harcodes it into Makefile.am:

libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g -Wl,-znoexecstack

(Executables are not linked with that option, but link to the above library, and presumably don't contain any other asm modules.)

And some linkers require a space IIRC...

[leif]

Replying to fbissey:

In any case sage's default is mpir so we should pass it in the default case. Passing it with gmp-6.1.1 doesn't really hurt.

? The option doesn't originate from GMP (see my last comment).

[fbissey]

collision. I posted before you. I guess it must have steamed from me communicating my concern to Paul. It definitely wasn't there in 7.0rc and I don't think it was in 7.0 either. One less thing to worry about.

[leif]

Replying to fbissey:

collision. I posted before you. I guess it must have steamed from me communicating my concern to Paul. It definitely wasn't there in 7.0rc and I don't think it was in 7.0 either. One less thing to worry about.

Well, until some linker bails out on that flag... Or is libtool smart enough to remove it in case it isn't supported?

[fbissey]

Replying to leif:

Replying to fbissey:

collision. I posted before you. I guess it must have steamed from me communicating my concern to Paul. It definitely wasn't there in 7.0rc and I don't think it was in 7.0 either. One less thing to worry about.

Well, until some linker bails out on that flag... Or is libtool smart enough to remove it in case it isn't supported?

Point. Most linkers discard unrecognised option without side effect, the problem is when it can be interpreted in a completely different way by the linker (yes I have a "funny" example of that - it involves AIX's linker). It would be better if the flag was tested for in configure.

[leif]

Did you already give it a try on MacOS X (recently)?

[fbissey]

No, and it will have to wait until morning now because it is my bed time. And I don't think upstream does regular check on OS X.

[leif]

What's weird is that despite

  --enable-shared[=PKGS]  build shared libraries [default=no]
  --enable-static[=PKGS]  build static libraries [default=yes]

and

...
checking whether the gcc-6.1 linker (ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... no
checking whether to build static libraries... yes
...

still a shared library gets built.

[leif]

Ok, at least within Sage, only a static library gets installed.

[fbissey]

Replying to leif:

Ok, at least within Sage, only a static library gets installed.

I personally don't think it is a good idea. Yes that's supposed to be better performance but if your going to mix it python module it will probably have to be compiled with pic which will remove part of your performance gain. Unless you have specialised hardware (embedded, BlueGene?) the benefits are marginal in my opinion.

[leif]

Replying to fbissey:

Replying to leif:

Ok, at least within Sage, only a static library gets installed.

[...] if your going to mix it python module it will probably have to be compiled with pic

We of course enforce that in spkg-install.

which will remove part of your performance gain.

Not necessarily, depends on the processor. You need relative addressing, but not indirect function calls, and most symbols will get resolved at .so build time, not when loading the shared library.

Unless you have specialised hardware (embedded, BlueGene?) the benefits are marginal in my opinion.

Hmmm, but having a static library included in the Python extension module (which of course is a shared library) prevents us from further headaches, too, IMHO. :-)

[fbissey]

OS X doesn't like this thing much

/bin/sh ./libtool  --tag=CC   --mode=link gcc  -g -march=native -g -O3  -fPIC  -version-info 1:0:0 -g -Wl,-znoexecstack  -L/Users/fbissey/build/sage-7.2.beta5/local/lib -Wl,-rpath,/Users/fbissey/build/sage-7.2.beta5/local/lib   -o libecm.la -rpath /Users/fbissey/build/sage-7.2.beta5/local/lib libecm_la-ecm.lo libecm_la-ecm2.lo libecm_la-pm1.lo libecm_la-pp1.lo libecm_la-getprime_r.lo libecm_la-listz.lo libecm_la-lucas.lo libecm_la-stage2.lo libecm_la-mpmod.lo libecm_la-mul_lo.lo libecm_la-polyeval.lo libecm_la-median.lo libecm_la-schoen_strass.lo libecm_la-ks-multiply.lo libecm_la-rho.lo libecm_la-bestd.lo libecm_la-auxlib.lo libecm_la-random.lo libecm_la-factor.lo libecm_la-sp.lo libecm_la-spv.lo libecm_la-spm.lo libecm_la-mpzspm.lo libecm_la-mpzspv.lo libecm_la-ntt_gfp.lo libecm_la-ecm_ntt.lo libecm_la-pm1fs2.lo libecm_la-sets_long.lo libecm_la-auxarith.lo libecm_la-batch.lo libecm_la-parametrizations.lo libecm_la-cudawrapper.lo aprtcle/libecm_la-mpz_aprcl.lo   ./x86_64/libmulredc.la  /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a -lm -lm -lm -lm -lm  

*** Warning: Linking the shared library libecm.la against the
*** static library /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a is not portable!
libtool: link: (cd .libs/libecm.lax/libmulredc.a && ar x "/Users/fbissey/build/sage-7.2.beta5/local/var/tmp/sage/build/ecm-7.0.3/src/./x86_64/.libs/libmulredc.a")
libtool: link: ar cru .libs/libecm.a /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a  libecm_la-ecm.o libecm_la-ecm2.o libecm_la-pm1.o libecm_la-pp1.o libecm_la-getprime_r.o libecm_la-listz.o libecm_la-lucas.o libecm_la-stage2.o libecm_la-mpmod.o libecm_la-mul_lo.o libecm_la-polyeval.o libecm_la-median.o libecm_la-schoen_strass.o libecm_la-ks-multiply.o libecm_la-rho.o libecm_la-bestd.o libecm_la-auxlib.o libecm_la-random.o libecm_la-factor.o libecm_la-sp.o libecm_la-spv.o libecm_la-spm.o libecm_la-mpzspm.o libecm_la-mpzspv.o libecm_la-ntt_gfp.o libecm_la-ecm_ntt.o libecm_la-pm1fs2.o libecm_la-sets_long.o libecm_la-auxarith.o libecm_la-batch.o libecm_la-parametrizations.o libecm_la-cudawrapper.o aprtcle/libecm_la-mpz_aprcl.o  .libs/libecm.lax/libmulredc.a/mulredc1.o .libs/libecm.lax/libmulredc.a/mulredc10.o .libs/libecm.lax/libmulredc.a/mulredc11.o .libs/libecm.lax/libmulredc.a/mulredc12.o .libs/libecm.lax/libmulredc.a/mulredc13.o .libs/libecm.lax/libmulredc.a/mulredc14.o .libs/libecm.lax/libmulredc.a/mulredc15.o .libs/libecm.lax/libmulredc.a/mulredc16.o .libs/libecm.lax/libmulredc.a/mulredc17.o .libs/libecm.lax/libmulredc.a/mulredc18.o .libs/libecm.lax/libmulredc.a/mulredc19.o .libs/libecm.lax/libmulredc.a/mulredc1_10.o .libs/libecm.lax/libmulredc.a/mulredc1_11.o .libs/libecm.lax/libmulredc.a/mulredc1_12.o .libs/libecm.lax/libmulredc.a/mulredc1_13.o .libs/libecm.lax/libmulredc.a/mulredc1_14.o .libs/libecm.lax/libmulredc.a/mulredc1_15.o .libs/libecm.lax/libmulredc.a/mulredc1_16.o .libs/libecm.lax/libmulredc.a/mulredc1_17.o .libs/libecm.lax/libmulredc.a/mulredc1_18.o .libs/libecm.lax/libmulredc.a/mulredc1_19.o .libs/libecm.lax/libmulredc.a/mulredc1_2.o .libs/libecm.lax/libmulredc.a/mulredc1_20.o .libs/libecm.lax/libmulredc.a/mulredc1_3.o .libs/libecm.lax/libmulredc.a/mulredc1_4.o .libs/libecm.lax/libmulredc.a/mulredc1_5.o .libs/libecm.lax/libmulredc.a/mulredc1_6.o .libs/libecm.lax/libmulredc.a/mulredc1_7.o .libs/libecm.lax/libmulredc.a/mulredc1_8.o .libs/libecm.lax/libmulredc.a/mulredc1_9.o .libs/libecm.lax/libmulredc.a/mulredc2.o .libs/libecm.lax/libmulredc.a/mulredc20.o .libs/libecm.lax/libmulredc.a/mulredc3.o .libs/libecm.lax/libmulredc.a/mulredc4.o .libs/libecm.lax/libmulredc.a/mulredc5.o .libs/libecm.lax/libmulredc.a/mulredc6.o .libs/libecm.lax/libmulredc.a/mulredc7.o .libs/libecm.lax/libmulredc.a/mulredc8.o .libs/libecm.lax/libmulredc.a/mulredc9.o 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libecm.a(libecm_la-cudawrapper.o) has no symbols
libtool: link: ranlib .libs/libecm.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libecm.a(libecm_la-cudawrapper.o) has no symbols
libtool: link: rm -fr .libs/libecm.lax
libtool: link: ( cd ".libs" && rm -f "libecm.la" && ln -s "../libecm.la" "libecm.la" )
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-auxi.o -MD -MP -MF .deps/ecm-auxi.Tpo -c -o ecm-auxi.o `test -f 'auxi.c' || echo './'`auxi.c
mv -f .deps/ecm-auxi.Tpo .deps/ecm-auxi.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-b1_ainc.o -MD -MP -MF .deps/ecm-b1_ainc.Tpo -c -o ecm-b1_ainc.o `test -f 'b1_ainc.c' || echo './'`b1_ainc.c
mv -f .deps/ecm-b1_ainc.Tpo .deps/ecm-b1_ainc.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-candi.o -MD -MP -MF .deps/ecm-candi.Tpo -c -o ecm-candi.o `test -f 'candi.c' || echo './'`candi.c
mv -f .deps/ecm-candi.Tpo .deps/ecm-candi.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-eval.o -MD -MP -MF .deps/ecm-eval.Tpo -c -o ecm-eval.o `test -f 'eval.c' || echo './'`eval.c
mv -f .deps/ecm-eval.Tpo .deps/ecm-eval.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-main.o -MD -MP -MF .deps/ecm-main.Tpo -c -o ecm-main.o `test -f 'main.c' || echo './'`main.c
mv -f .deps/ecm-main.Tpo .deps/ecm-main.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-resume.o -MD -MP -MF .deps/ecm-resume.Tpo -c -o ecm-resume.o `test -f 'resume.c' || echo './'`resume.c
mv -f .deps/ecm-resume.Tpo .deps/ecm-resume.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-getprime_r.o -MD -MP -MF .deps/ecm-getprime_r.Tpo -c -o ecm-getprime_r.o `test -f 'getprime_r.c' || echo './'`getprime_r.c
mv -f .deps/ecm-getprime_r.Tpo .deps/ecm-getprime_r.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT aprtcle/ecm-mpz_aprcl.o -MD -MP -MF aprtcle/.deps/ecm-mpz_aprcl.Tpo -c -o aprtcle/ecm-mpz_aprcl.o `test -f 'aprtcle/mpz_aprcl.c' || echo './'`aprtcle/mpz_aprcl.c
mv -f aprtcle/.deps/ecm-mpz_aprcl.Tpo aprtcle/.deps/ecm-mpz_aprcl.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-memusage.o -MD -MP -MF .deps/ecm-memusage.Tpo -c -o ecm-memusage.o `test -f 'memusage.c' || echo './'`memusage.c
mv -f .deps/ecm-memusage.Tpo .deps/ecm-memusage.Po
/bin/sh ./libtool  --tag=CC   --mode=link gcc  -g -march=native -g -O3  -fPIC  -L/Users/fbissey/build/sage-7.2.beta5/local/lib -Wl,-rpath,/Users/fbissey/build/sage-7.2.beta5/local/lib   -o ecm ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o libecm.la /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a -lm -lm -lm -lm -lm  
libtool: link: gcc -g -march=native -g -O3 -fPIC -Wl,-rpath -Wl,/Users/fbissey/build/sage-7.2.beta5/local/lib -o ecm ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o  -L/Users/fbissey/build/sage-7.2.beta5/local/lib ./.libs/libecm.a /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a -lm
ld: warning: ignoring file ./.libs/libecm.a, file was built for archive which is not the architecture being linked (x86_64): ./.libs/libecm.a
Undefined symbols for architecture x86_64:
  "___ecm_cputime", referenced from:
      _main in ecm-main.o
  "_ecm_clear", referenced from:
      _main in ecm-main.o
  "_ecm_factor", referenced from:
      _main in ecm-main.o
  "_ecm_init", referenced from:
      _main in ecm-main.o
  "_ecm_version", referenced from:
      _main in ecm-main.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [ecm] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Error building GMP-ECM.

It is not obvious that znoexecstack is at fault but I cannot rule it out.

I am otherwise in the camp that says dynamic linking is actually the lesser trouble :)

[leif]

P.S.: Actually libgmp.a should get linked into the Python extension module as well, but that's currently not the case.

[fbissey]

Interesting znoexecstack is not a problem building static - it is probably just ignored but build shared with ECM_CONFIGURE="--enable-shared --disable-static

libtool: link: gcc -dynamiclib -Wl,-undefined -Wl,dynamic_lookup -o .libs/libecm.1.dylib  .libs/libecm_la-ecm.o .libs/libecm_la-ecm2.o .libs/libecm_la-pm1.o .libs/libecm_la-pp1.o .libs/libecm_la-getprime_r.o .libs/libecm_la-listz.o .libs/libecm_la-lucas.o .libs/libecm_la-stage2.o .libs/libecm_la-mpmod.o .libs/libecm_la-mul_lo.o .libs/libecm_la-polyeval.o .libs/libecm_la-median.o .libs/libecm_la-schoen_strass.o .libs/libecm_la-ks-multiply.o .libs/libecm_la-rho.o .libs/libecm_la-bestd.o .libs/libecm_la-auxlib.o .libs/libecm_la-random.o .libs/libecm_la-factor.o .libs/libecm_la-sp.o .libs/libecm_la-spv.o .libs/libecm_la-spm.o .libs/libecm_la-mpzspm.o .libs/libecm_la-mpzspv.o .libs/libecm_la-ntt_gfp.o .libs/libecm_la-ecm_ntt.o .libs/libecm_la-pm1fs2.o .libs/libecm_la-sets_long.o .libs/libecm_la-auxarith.o .libs/libecm_la-batch.o .libs/libecm_la-parametrizations.o .libs/libecm_la-cudawrapper.o aprtcle/.libs/libecm_la-mpz_aprcl.o   -Wl,-force_load,./x86_64/.libs/libmulredc.a  -L/Users/fbissey/build/sage-7.2.beta5/local/lib -lgmp -lm  -g -march=native -g -O3 -g -Wl,-znoexecstack -Wl,-rpath -Wl,/Users/fbissey/build/sage-7.2.beta5/local/lib   -install_name  /Users/fbissey/build/sage-7.2.beta5/local/lib/libecm.1.dylib -compatibility_version 2 -current_version 2.0 -Wl,-single_module
ld: unknown option: -znoexecstack
collect2: error: ld returned 1 exit status

And that definitely kills the linking. So we will need to remove it from upstream sources.

Yes once you link static stuff you need to provide all the bits since static libraries are dumb objects.

[leif]

Presumably unrelated, but 7.2.beta5?

What "architecture" in Apple speak do the object files in libecm.a have?

[fbissey]

Replying to leif:

Presumably unrelated, but 7.2.beta5?

That's just the tarball I initially downloaded, but I pulled the latest develop and then your branch. I fully rebuild not so long ago.

What "architecture" in Apple speak do the object files in libecm.a have?

Good question, I'll see if I can figure it out.

[leif]

Replying to fbissey:

Interesting znoexecstack is not a problem building static - it is probably just ignored

It's not used at all when building static stuff, or more precisely, only when building libecm.la.

but build shared with ECM_CONFIGURE="--enable-shared --disable-static

ld: unknown option: -znoexecstack
collect2: error: ld returned 1 exit status

And that definitely kills the linking.

I told you so... ;-)

So we will need to remove it from upstream sources.

Well, patch a comma into it. (It's in Makefile.am and hence also in what's built from it.)

[leif]

Replying to leif:

Replying to fbissey:

ld: unknown option: -znoexecstack
collect2: error: ld returned 1 exit status

Well, patch a comma into it. (It's in Makefile.am and hence also in what's built from it.)

That is, in the source tarball only one instance in the top-level Makefile.in.

I wouldn't even use patch for that... ;-)

[leif]

But I'm really in favour of submitting something upstream that in configure adds -Wa,--noexecstack to CCASFLAGS when supported (or maybe -Wl,-z,noexecstack to some *LDFLAGS).

[fbissey]

Replying to leif:

Replying to leif:

Replying to fbissey:

ld: unknown option: -znoexecstack
collect2: error: ld returned 1 exit status

Well, patch a comma into it. (It's in Makefile.am and hence also in what's built from it.)

That is, in the source tarball only one instance in the top-level Makefile.in.

I wouldn't even use patch for that... ;-)

May be you wouldn't. But in terms of sage, I would do a spkg-src. You can only be that dirty in a big collaborative project of that kind for so long. That kind of stuff just doesn't scale and hurt you in the long term.

In my opinion the whole spkg-install is overdone. Just let the build system do its job unless you want to produce a binary.

+1 for a report upstream.

[leif]

It's just always a headache to patch autotools files, and spkg-src is IMHO not a good idea unless you pull some svn version (as opposed to a released tarball), or really remove loads of unnecessary files in order to keep the tarball small.

[leif]

(Sage's "build system" doesn't yet support upstream tarball URLs though, and the "design" of for example checksums.ini is ... -- can't say that here.)

[fbissey]

No comment. I am the sage-on-gentoo maintainer - anything I say on the matter is kind of political.

Back to our little problem on OS X. It looks like it is something done in spkg-install, going to a sage shell and the build location I just do

make distclean
./configure # by default only build static
make

is successful and

(sage-sh) fbissey@Mirage:src$ make check
Making check in x86_64
make[1]: Nothing to be done for `check'.
make  ecm \
	  test.pp1 test.pm1 test.ecm 
make[2]: `ecm' is up to date.
make[2]: Nothing to be done for `test.pp1'.
make[2]: Nothing to be done for `test.pm1'.
make[2]: Nothing to be done for `test.ecm'.
make  check-TESTS
PASS: test.pp1
PASS: test.pm1
PASS: test.ecm
============================================================================
Testsuite summary for ecm 7.0.3
============================================================================
# TOTAL: 3
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

so it is definitely the configuration/environment from spkg-install, the configuration picked up sage's gcc as the compiler and sage's mpir as gmp without a hitch and assembly code has been used.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6741ee0

GMP-ECM: #20385: Add temporary patch for '-Wl,znoexecstack' [sic] (Makefile.{am,in})

[leif]

ecm-7.0.3/Makefile.am

@@ -50 +50 @@
 # www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html
 # If any interfaces have been added, removed, or changed since the last
 # update, increment current, and set revision to 0.
-libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g -Wl,-znoexecstack
+libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g -Wl,-z,noexecstack
 libecm_la_LIBADD = $(MULREDCLIBRARY)
 if WANT_GPU 
   libecm_la_SOURCES += cudakernel.cu

ecm-7.0.3/Makefile.in

@@ -715 +715 @@
 # If any interfaces have been added, removed, or changed since the last
 # update, increment current, and set revision to 0.
 libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g \
-        -Wl,-znoexecstack $(am__append_5)
+        -Wl,-z,noexecstack $(am__append_5)
 libecm_la_LIBADD = $(MULREDCLIBRARY) $(am__append_4) $(GMPLIB)
 @WANT_GPU_TRUE@ecm_LDFLAGS = $(CUDARPATH)
 @WITH_GWNUM_TRUE@ecm_LDFLAGS = $(AM_LDFLAGS) -Wl,gwdata.ld

---

New commits:

​6741ee0

GMP-ECM: #20385: Add temporary patch for '-Wl,znoexecstack' [sic] (Makefile.{am,in})

[fbissey]

With that I have

ld: unknown option: -z

now when enabling shared libraries on OS X. At least that's consistent with the man page.

[leif]

Replying to fbissey:

Back to our little problem on OS X. It looks like it is something done in spkg-install, going to a sage shell and the build location I just do

make distclean
./configure # by default only build static
make

is successful [...] so it is definitely the configuration/environment from spkg-install, the configuration picked up sage's gcc as the compiler and sage's mpir as gmp without a hitch and assembly code has been used.

Well, post the part of the log where spkg-install prints all the relevant environment variables and configure options passed.

[leif]

Replying to fbissey:

With that I have

ld: unknown option: -z

now when enabling shared libraries on OS X. At least that's consistent with the man page.

What does gcc -c -Wa,--noexecstack foo.c give? (foo.c at your like, e.g. void foo() {}.)

(And probably also gcc -v -o foo ... without -c.)

[leif]

Replying to leif:

(And probably also gcc -v -o foo ... without -c.)

Ahem, forget that.

[fbissey]

clang: error: unsupported option '--noexecstack'

Not in the manual either.

[leif]

And/or attach the log of the last failed build here.

[fbissey]

ecm log - bare spkg-install run

[fbissey]

config.log matching ecm.log

[fbissey]

Done. So just plain ./sage -p ecm logs. No funky things in ECM_CONFIGURE.

[leif]

• Attachment ecm.log​ added

ecm log - bare spkg-install run

Interestingly, ld there didn't bail out on -z noexecstack, for whatever reason. (Did it at all get invoked?)

Otherwise I'm stuck at libtool weirdness and what Apple's ld considers "not x86_64 architecture".

Also, while we have

checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no

only -fPIC is used (which we add in spkg-install).

And libtool of course^TM keeps building .o and .lo files, and both foo.a and foo.la.

[fbissey]

Found the bit that cause problem --with-gmp=${SAGE_LOCAL} without it configure tells me

configure: Configuration:
configure: Build for host type x86_64-apple-darwin15.6.0
configure: CC=gcc, CFLAGS=-g -O2
configure: Linking GMP with -lgmp

and the build succeeds. With it

configure: Configuration:
configure: Build for host type x86_64-apple-darwin15.6.0
configure: CC=gcc, CFLAGS=-g -O2
configure: Linking GMP with /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a

and the build fails....

[fbissey]

Rebuilding develop branch from scratch to make sure mpir is as sane as it can be.

[leif]

Replying to fbissey:

Found the bit that cause problem --with-gmp=${SAGE_LOCAL} without it configure tells me

configure: Configuration:
configure: Build for host type x86_64-apple-darwin15.6.0
configure: CC=gcc, CFLAGS=-g -O2
configure: Linking GMP with -lgmp

I thought of that, but you told me "the configuration picked up sage's gcc as the compiler and sage's mpir as gmp without a hitch" and I was wondering whether that's due to Sage GCC's configuration.

But before my internet connection went down the drain 8/ I was about to post:

---

And then I'd need a log from your successful "manual" build (make distclean && ./configure && make) from within a Sage subshell to compare with.

OTOH the only (major) difference -- regarding the environment when configure is invoked -- should be -fPIC in CFLAGS, as ECM picks up MPIR's when CFLAGS are empty / not set (which is the case in a Sage subshell unless you've set them yourself).

Not sure how libtool reacts to that, or whether there's anything in configure that's influenced by that.

---

Now clearly the difference seems to (also) be that you don't have a static libgmp installed system-wide.

[fbissey]

I don't even have a dynamic one.

The environment passes -L${SAGE_LOCAL}/lib and -Wl,-rpath=${SAGE_LOCAL}/lib, further the configure script identifies gmp as being in fact mpir in both case. So the -lgmp picked can only be from ${SAGE_LOCAL}/lib.

Even further when outside of the sage shell configure bails out because it cannot find any gmp. So the only gmp available is in sage's install.

On another front I separately tried to add -fPIC to CFLAGS for configuration and that wasn't a problem. Just adding -fPIC lead to a successful compilation.

libgmp.a or the process of using libgmp.a wholesale is causing trouble. Using libgmp.a that way tries to fold the gmp archive inside libecm.a (so you don't need to link with libgmp.a as it is already inside), apparently it causes trouble on OS X. Possibly something in mpir rather than ecm in my opinion.

[leif]

Replying to fbissey:

libgmp.a or the process of using libgmp.a wholesale is causing trouble. Using libgmp.a that way tries to fold the gmp archive inside libecm.a (so you don't need to link with libgmp.a as it is already inside), apparently it causes trouble on OS X. Possibly something in mpir rather than ecm in my opinion.

It shouldn't. You cannot "fold" a static library into another, you can include object files from one into another, or when you build a shared library, include object files into the former (as needed -- symbols get resolved and the object files from which they originate get included, all ending up in the same single text segment etc.).

Wondering whether we build MPIR's/GMP's static library also with PIC. (Btw., ECM also has --with-pic.)

My guess is rather that it's ECM's fault.

[leif]

Replying to leif:

Wondering whether we build MPIR's/GMP's static library also with PIC.

Looks like it (MPIR, not explicitly we) does (-fPIC -DPIC).

[leif]

Note that in

libtool: link: gcc -g -march=native -g -O3 -fPIC
    -Wl,-rpath -Wl,/Users/fbissey/build/sage-7.2.beta5/local/lib 
    -o ecm
    ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o
    ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o
    -L/Users/fbissey/build/sage-7.2.beta5/local/lib ./.libs/libecm.a
    /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a -lm
ld: warning: ignoring file ./.libs/libecm.a, file was built for archive which is not the architecture being linked (x86_64): ./.libs/libecm.a

the linker only complains about libecm.a, but not libgmp.a.

[leif]

But you're right, ECM actually adds libgmp.a to libecm.a:

libtool: link: ar cru .libs/libecm.a /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a  libecm_la-ecm.o libecm_la-ecm2.o libecm_la-pm1.o libecm_la-pp1.o libecm_la-getprime_r.o libecm_la-listz.o libecm_la-lucas.o libecm_la-stage2.o libecm_la-mpmod.o libecm_la-mul_lo.o libecm_la-polyeval.o libecm_la-median.o libecm_la-schoen_strass.o libecm_la-ks-multiply.o libecm_la-rho.o libecm_la-bestd.o libecm_la-auxlib.o libecm_la-random.o libecm_la-factor.o libecm_la-sp.o libecm_la-spv.o libecm_la-spm.o libecm_la-mpzspm.o libecm_la-mpzspv.o libecm_la-ntt_gfp.o libecm_la-ecm_ntt.o libecm_la-pm1fs2.o libecm_la-sets_long.o libecm_la-auxarith.o libecm_la-batch.o libecm_la-parametrizations.o libecm_la-cudawrapper.o aprtcle/libecm_la-mpz_aprcl.o  .libs/libecm.lax/libmulredc.a/mulredc1.o .libs/libecm.lax/libmulredc.a/mulredc10.o .libs/libecm.lax/libmulredc.a/mulredc11.o .libs/libecm.lax/libmulredc.a/mulredc12.o .libs/libecm.lax/libmulredc.a/mulredc13.o .libs/libecm.lax/libmulredc.a/mulredc14.o .libs/libecm.lax/libmulredc.a/mulredc15.o .libs/libecm.lax/libmulredc.a/mulredc16.o .libs/libecm.lax/libmulredc.a/mulredc17.o .libs/libecm.lax/libmulredc.a/mulredc18.o .libs/libecm.lax/libmulredc.a/mulredc19.o .libs/libecm.lax/libmulredc.a/mulredc1_10.o .libs/libecm.lax/libmulredc.a/mulredc1_11.o .libs/libecm.lax/libmulredc.a/mulredc1_12.o .libs/libecm.lax/libmulredc.a/mulredc1_13.o .libs/libecm.lax/libmulredc.a/mulredc1_14.o .libs/libecm.lax/libmulredc.a/mulredc1_15.o .libs/libecm.lax/libmulredc.a/mulredc1_16.o .libs/libecm.lax/libmulredc.a/mulredc1_17.o .libs/libecm.lax/libmulredc.a/mulredc1_18.o .libs/libecm.lax/libmulredc.a/mulredc1_19.o .libs/libecm.lax/libmulredc.a/mulredc1_2.o .libs/libecm.lax/libmulredc.a/mulredc1_20.o .libs/libecm.lax/libmulredc.a/mulredc1_3.o .libs/libecm.lax/libmulredc.a/mulredc1_4.o .libs/libecm.lax/libmulredc.a/mulredc1_5.o .libs/libecm.lax/libmulredc.a/mulredc1_6.o .libs/libecm.lax/libmulredc.a/mulredc1_7.o .libs/libecm.lax/libmulredc.a/mulredc1_8.o .libs/libecm.lax/libmulredc.a/mulredc1_9.o .libs/libecm.lax/libmulredc.a/mulredc2.o .libs/libecm.lax/libmulredc.a/mulredc20.o .libs/libecm.lax/libmulredc.a/mulredc3.o .libs/libecm.lax/libmulredc.a/mulredc4.o .libs/libecm.lax/libmulredc.a/mulredc5.o .libs/libecm.lax/libmulredc.a/mulredc6.o .libs/libecm.lax/libmulredc.a/mulredc7.o .libs/libecm.lax/libmulredc.a/mulredc8.o .libs/libecm.lax/libmulredc.a/mulredc9.o 

and that's presumably the entry Apple's ld complains about (and instead of ignoring the entry, it drops the whole archive, libecm.a).

[leif]

I guess this is the culprit (in Makefile.am, line 61):

libecm_la_LIBADD += $(GMPLIB)

(In Makefile.in, on line 719.)

New tentative patch:

ecm-7.0.3/Makefile.am

@@ -58 +58 @@
   libecm_la_LDFLAGS += $(CUDALDFLAGS)
   ecm_LDFLAGS = $(CUDARPATH)
 endif
-libecm_la_LIBADD += $(GMPLIB)
+# libecm_la_LIBADD += $(GMPLIB)
 
 bin_PROGRAMS = ecm
 noinst_PROGRAMS = tune ecmfactor bench_mulredc aprcl

ecm-7.0.3/Makefile.in

@@ -716 +716 @@
 # update, increment current, and set revision to 0.
 libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g \
         -Wl,-znoexecstack $(am__append_5)
-libecm_la_LIBADD = $(MULREDCLIBRARY) $(am__append_4) $(GMPLIB)
+libecm_la_LIBADD = $(MULREDCLIBRARY) $(am__append_4)
 @WANT_GPU_TRUE@ecm_LDFLAGS = $(CUDARPATH)
 @WITH_GWNUM_TRUE@ecm_LDFLAGS = $(AM_LDFLAGS) -Wl,gwdata.ld
 # Most binaries want to link libecm.la, and the ones which don't will

Note that this new patch gets applied before the other one, so I had to change the latter as well since they unfortunately overlap. (But the first I added will get dropped later anyway, if Darwin's linker doesn't understand -z ... and also errors out on it.)

[leif]

I'll push the changes in a couple of minutes...

[fbissey]

Yes, removing that bit looks right. After all it is already added to LDFLAGS by configure. For the record I successfully installed ecm-7.0.3 on OS X with the following change

build/pkgs/ecm/spkg-install

@@ -237 +237 @@
 fi
 echo "  --prefix=\"$SAGE_LOCAL\""
 echo "  --libdir=\"$SAGE_LOCAL/lib\""
-echo "  --with-gmp=\"$SAGE_LOCAL\""
 for opt in $ECM_CONFIGURE; do
     echo "  $opt"
 done
@@ -249 +248 @@
 echo
 
 ./configure --prefix="$SAGE_LOCAL" --libdir="$SAGE_LOCAL/lib" \
-            --with-gmp="$SAGE_LOCAL" \
             $ECM_CONFIGURE
 if [ $? -ne 0 ]; then
     echo >&2 "Error configuring GMP-ECM."

Just doing that removes libgmp.a from the ar call

libtool: link: (cd .libs/libecm.lax/libmulredc.a && ar x "/Users/fbissey/build/sage-7.2.beta5/local/var/tmp/sage/build/ecm-7.0.3/src/./x86_64/.libs/libmulredc.a")
libtool: link: ar cru .libs/libecm.a  libecm_la-ecm.o libecm_la-ecm2.o libecm_la-pm1.o libecm_la-pp1.o libecm_la-getprime_r.o libecm_la-listz.o libecm_la-lucas.o libecm_la-stage2.o libecm_la-mpmod.o libecm_la-mul_lo.o libecm_la-polyeval.o libecm_la-median.o libecm_la-schoen_strass.o libecm_la-ks-multiply.o libecm_la-rho.o libecm_la-bestd.o libecm_la-auxlib.o libecm_la-random.o libecm_la-factor.o libecm_la-sp.o libecm_la-spv.o libecm_la-spm.o libecm_la-mpzspm.o libecm_la-mpzspv.o libecm_la-ntt_gfp.o libecm_la-ecm_ntt.o libecm_la-pm1fs2.o libecm_la-sets_long.o libecm_la-auxarith.o libecm_la-batch.o libecm_la-parametrizations.o libecm_la-cudawrapper.o aprtcle/libecm_la-mpz_aprcl.o  .libs/libecm.lax/libmulredc.a/mulredc1.o .libs/libecm.lax/libmulredc.a/mulredc10.o .libs/libecm.lax/libmulredc.a/mulredc11.o .libs/libecm.lax/libmulredc.a/mulredc12.o .libs/libecm.lax/libmulredc.a/mulredc13.o .libs/libecm.lax/libmulredc.a/mulredc14.o .libs/libecm.lax/libmulredc.a/mulredc15.o .libs/libecm.lax/libmulredc.a/mulredc16.o .libs/libecm.lax/libmulredc.a/mulredc17.o .libs/libecm.lax/libmulredc.a/mulredc18.o .libs/libecm.lax/libmulredc.a/mulredc19.o .libs/libecm.lax/libmulredc.a/mulredc1_10.o .libs/libecm.lax/libmulredc.a/mulredc1_11.o .libs/libecm.lax/libmulredc.a/mulredc1_12.o .libs/libecm.lax/libmulredc.a/mulredc1_13.o .libs/libecm.lax/libmulredc.a/mulredc1_14.o .libs/libecm.lax/libmulredc.a/mulredc1_15.o .libs/libecm.lax/libmulredc.a/mulredc1_16.o .libs/libecm.lax/libmulredc.a/mulredc1_17.o .libs/libecm.lax/libmulredc.a/mulredc1_18.o .libs/libecm.lax/libmulredc.a/mulredc1_19.o .libs/libecm.lax/libmulredc.a/mulredc1_2.o .libs/libecm.lax/libmulredc.a/mulredc1_20.o .libs/libecm.lax/libmulredc.a/mulredc1_3.o .libs/libecm.lax/libmulredc.a/mulredc1_4.o .libs/libecm.lax/libmulredc.a/mulredc1_5.o .libs/libecm.lax/libmulredc.a/mulredc1_6.o .libs/libecm.lax/libmulredc.a/mulredc1_7.o .libs/libecm.lax/libmulredc.a/mulredc1_8.o .libs/libecm.lax/libmulredc.a/mulredc1_9.o .libs/libecm.lax/libmulredc.a/mulredc2.o .libs/libecm.lax/libmulredc.a/mulredc20.o .libs/libecm.lax/libmulredc.a/mulredc3.o .libs/libecm.lax/libmulredc.a/mulredc4.o .libs/libecm.lax/libmulredc.a/mulredc5.o .libs/libecm.lax/libmulredc.a/mulredc6.o .libs/libecm.lax/libmulredc.a/mulredc7.o .libs/libecm.lax/libmulredc.a/mulredc8.o .libs/libecm.lax/libmulredc.a/mulredc9.o 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libecm.a(libecm_la-cudawrapper.o) has no symbols
libtool: link: ranlib .libs/libecm.a

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e115141

GMP-ECM: #20385: Add another patch avoiding nested archives and rebase the previous on it.

[leif]

Pushed the new patch(es), and thereby deleted my comment I hadn't posted yet -- no idea what I had written... >8( Thanks, HTTPS!

... I seem to remember that I had speculated that ..._LIBADD = $(GMPLIB) would presumably have worked if $(GMPLIB) was libgmp.la, as libtool indeed unpacks $(MULREDCLIBRARY) and adds its contents to the libecm archive.

---

Still have to see what harm my patch does to a shared libecm; it mayget underlinked now.

[fbissey]

Replying to leif:

Pushed the new patch(es), and thereby deleted my comment I hadn't posted yet -- no idea what I had written... >8( Thanks, HTTPS!

... I seem to remember that I had speculated that ..._LIBADD = $(GMPLIB) would presumably have worked if $(GMPLIB) was libgmp.la, as libtool indeed unpacks $(MULREDCLIBRARY) and adds its contents to the libecm archive.

---

Still have to see what harm my patch does to a shared libecm; it mayget underlinked now.

I wouldn't think so. Using libgmp.la means that it has to be present on the system. Relying on an installed, external to your project, .la file is extremely fragile. In any case I don't appear to have any lib{gmp,mpir}.la installed on OS X so it would fail - miserably.

[leif]

Replying to leif:

Still have to see what harm my patch does to a shared libecm; it may get underlinked now.

Exactly that happens.

[leif]

libecm_la_LDFLAGS += $(GMPLIB) to avoid underlinking?

[leif]

Replying to leif:

libecm_la_LDFLAGS += $(GMPLIB) to avoid underlinking?

Rather libecm_la_LDADD += $(GMPLIB) I think.

[leif]

Replying to leif:

Replying to leif:

Still have to see what harm my patch does to a shared libecm; it may get underlinked now.

Exactly that happens.

But AFAICS that wouldn't be a regression w.r.t. GMP-ECM 6.4.4 at least...

[leif]

Replying to leif:

Replying to leif:

libecm_la_LDFLAGS += $(GMPLIB) to avoid underlinking?

Rather libecm_la_LDADD += $(GMPLIB) I think.

François, could you test this (adding the latter to Makefile.am)?

I don't have automake 1.15 installed...

[fbissey]

Just looked at ecm's configure.ac. I am not impressed but I guess this is geared towards static linking. However the proper way tp link statically is not add $path_to/libname.a, if you have gcc you pass -static.

I can test adding that stuff, give me a few minutes (i may suffer from interruptions from a 5 year old that needs to go to bed).

[leif]

Replying to fbissey:

Just looked at ecm's configure.ac. I am not impressed but I guess this is geared towards static linking. However the proper way to link statically is not add $path_to/libname.a, if you have gcc you pass -static.

But that apparently doesn't work on Darwin, as in your log we had:

checking if gcc static flag -static works... no

Not sure what libtool magic offers.

Passing foo.a is certainly safer and more portable.

[fbissey]

One to bed. libtool as an -all-static mode for linking program that use only static libraries, not sure how it applies on OS X. But you cannot do that with libraries. You don't link static libraries, you just just ar+ranlib or equivalent. Folding one library in another is probably frowned on by libtool.

[fbissey]

Replying to leif:

Replying to leif:

Replying to leif:

libecm_la_LDFLAGS += $(GMPLIB) to avoid underlinking?

Rather libecm_la_LDADD += $(GMPLIB) I think.

François, could you test this (adding the latter to Makefile.am)?

I don't have automake 1.15 installed...

Didn't really like it

***** automake *****
***** PWD: /scratch2/portage/sci-mathematics/gmp-ecm-7.0.3-r1/work/ecm-7.0.3
***** automake --add-missing --copy --force-missing

configure.ac:161: installing './compile'
configure.ac:8: installing './missing'
Makefile.am:61: error: libecm_la_LDADD must be set with '=' before using '+='
Makefile.am:61: error: use 'libecm_la_LIBADD', not 'libecm_la_LDADD'
Makefile.am: installing './depcomp'
Makefile.am:61: warning: variable 'libecm_la_LDADD' is defined but no program or
Makefile.am:61: library has 'libecm_la' as canonical name (possible typo)

Using = instead of += doesn't really help the second message

***** automake *****
***** PWD: /scratch2/portage/sci-mathematics/gmp-ecm-7.0.3-r1/work/ecm-7.0.3
***** automake --add-missing --copy --force-missing

configure.ac:161: installing './compile'
configure.ac:8: installing './missing'
Makefile.am:61: error: use 'libecm_la_LIBADD', not 'libecm_la_LDADD'
Makefile.am: installing './depcomp'
Makefile.am:61: warning: variable 'libecm_la_LDADD' is defined but no program or
Makefile.am:61: library has 'libecm_la' as canonical name (possible typo)

[leif]

While I've now installed automake 1.15, it removes

runstatedir = @runstatedir@

from Makefile.in (without me having made any changes to Makefile.am)...

[fbissey]

That's the bit I don't like in configure.ac:

GMPLDFLAGS=""
if test -d "$with_gmp_lib"; then
  GMPLDFLAGS="-L$with_gmp_lib"
fi
GMPLIB="-lgmp"
if test "x$enable_shared" != xyes; then
  if test -r "$with_gmp_lib/libgmp.a"; then
    GMPLIB="$with_gmp_lib/libgmp.a"  <==================================
    dnl Don't need -L flag since we give full path to libgmp.a
    GMPLDFLAGS=""
  fi
fi
AC_SUBST([GMPLIB])
LDFLAGS="$LDFLAGS $GMPLDFLAGS"

[leif]

Replying to fbissey:

You don't link static libraries, you just just ar+ranlib or equivalent. Folding one library in another is probably frowned on by libtool.

No, that's exactly what it does with $(MULREDCLIBRARY) (aka libmulredc.[l]a), and it doesn't nest the archive, but "flattens" both libraries / archives into one.

(And there's also ld -r ..., but merging all object files into one is even more crude and will not always work.)

[leif]

Replying to fbissey:

That's the bit I don't like in configure.ac:

GMPLDFLAGS=""
if test -d "$with_gmp_lib"; then
  GMPLDFLAGS="-L$with_gmp_lib"
fi
GMPLIB="-lgmp"
if test "x$enable_shared" != xyes; then
  if test -r "$with_gmp_lib/libgmp.a"; then
    GMPLIB="$with_gmp_lib/libgmp.a"  <==================================
    dnl Don't need -L flag since we give full path to libgmp.a
    GMPLDFLAGS=""
  fi
fi
AC_SUBST([GMPLIB])
LDFLAGS="$LDFLAGS $GMPLDFLAGS"

Well add

  if test -r "$with_gmp_lib/libgmp.la"; then
    GMPLIB="$with_gmp_lib/libgmp.la" # That would have worked on Darwin as well.
  elif test -r "$with_gmp_lib/libgmp.a"; then
    GMPLIB="$with_gmp_lib/libgmp.a"
  ...

The meaning of --enable-shared isn't always clear.

Of course, you want a shared library (also) built, but that doesn't necessarily imply you want it itself to be dynamically linked (to libraries it depends on, no matter whether also built by the package or externally supplied).

I'd add an option --prefer-static[-gmp], say, in addition.

[leif]

P.S.: IMHO the real bug in what you pointed out is that ECM allows to build static and dynamic libraries at the same time, so it should (if present) define and use both GMPLIB (static) and GMPDYLIB (say, shared).

[fbissey]

I spent some time with the libtool mailing list and done some experiments. If you want to have a full static binary (program only), you will have to add -all-static to ${prog}_LDFLAGS. If you miss one of dependent library in static form: boom

/bin/sh ./libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc  -g -O1 -march=native -pipe -ggdb -all-static -Wl,-O1 -Wl,--as-needed -O1 -march=native -pipe -ggdb  -o ecm ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o libecm.la -lgmp -lrt -lm -lm -lm -lm -lm  
/bin/sh ./libtool  --tag=CC   --mode=link x86_64-pc-linux-gnu-gcc  -lpthread -O1 -march=native -pipe -ggdb  -Wl,-O1 -Wl,--as-needed -O1 -march=native -pipe -ggdb  -o ecmfactor ecmfactor-ecmfactor.o libecm.la -lgmp -lrt -lm -lm -lm -lm -lm  
libtool: link: x86_64-pc-linux-gnu-gcc -O1 -march=native -pipe -ggdb -Wl,-O1 -O1 -march=native -pipe -ggdb -o ecmfactor ecmfactor-ecmfactor.o  -lpthread -Wl,--as-needed ./.libs/libecm.a -lgmp -lrt -lm
libtool: link: x86_64-pc-linux-gnu-gcc -g -O1 -march=native -pipe -ggdb -static -Wl,-O1 -O1 -march=native -pipe -ggdb -o ecm ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o  -Wl,--as-needed ./.libs/libecm.a -lgmp -lrt -lm
/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lgmp
collect2: error: ld returned 1 exit status

In a normal scenario it is ok to build both static and dynamic at the same time. The dynamic library can be linked to the dependent libraries and the static one usually doesn't incorporate the dependent libraries (whatever you want to call the process).

The --enable-{shared,static} are concerning the building of libraries only. It doesn't say anything about programs which will be linked the default way unless you take steps.

You are right there is problem here when you try to build both static and dynamic libraries because the static one can use a deifferent recipe from normal if a static gmp is also available. At that point we have a break down.

The intent is probably to really have a mega-archive libecm.a including libgmp.a - while it is ok to incorporate internal libraies it is doubtful to do it with external ones. The best, sane, option I can offer is to compile the ecm executable all static if libgmp.a is available.

[leif]

I think this is (almost) what we want:

ecm-7.0.3/Makefile.am

@@ -59 +59 @@
   ecm_LDFLAGS = $(CUDARPATH)
 endif
 # libecm_la_LIBADD += $(GMPLIB)
+libecm_la_LDFLAGS += $(GMPLIB)
 
 bin_PROGRAMS = ecm
 noinst_PROGRAMS = tune ecmfactor bench_mulredc aprcl

ecm-7.0.3/Makefile.in

@@ -714 +712 @@
 # www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html
 # If any interfaces have been added, removed, or changed since the last
 # update, increment current, and set revision to 0.
+# libecm_la_LIBADD += $(GMPLIB)
 libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g \
-        -Wl,-z,noexecstack $(am__append_5)
+        -Wl,-z,noexecstack $(am__append_5) $(GMPLIB)
 libecm_la_LIBADD = $(MULREDCLIBRARY) $(am__append_4)
 @WANT_GPU_TRUE@ecm_LDFLAGS = $(CUDARPATH)
 @WITH_GWNUM_TRUE@ecm_LDFLAGS = $(AM_LDFLAGS) -Wl,gwdata.ld

(Without modifying configure.ac [yet] at least.)

[fbissey]

In Makefile.am you want to change line 52 which already has a definition for libecm_la_LDFLAGS.

[leif]

We should really ask upstream what they want(ed) or intend(ed), for example it IMHO makes sense to build statically linked programs (or incorporate libgmp.a) provided a static libgmp is available, unless --disable-static was given.

Building "everything" dynamically linked just because --enable-shared was (also) given IMHO doesn't make sense here.

(And I'd actually incorporate libgmp.a into libecm.so unless specified otherwise, which would require a new configure option.)

[leif]

Replying to fbissey:

In Makefile.am you want to change line 52 which already has a definition for libecm_la_LDFLAGS.

Oooops, forgot a plus, but for whatever reason, it worked for me (with version info etc.):

/bin/bash ./libtool  --tag=CC   --mode=link gcc-6.1  -g -march=native -g -O3   -version-info 1:0:0 -g -Wl,-z,noexecstack  -lgmp -L/data/Sage/release/devel/sage-7.3.rc0-gcc-6.1.0/local/lib -Wl,-rpath,/data/Sage/release/devel/sage-7.3.rc0-gcc-6.1.0/local/lib  -L/data/Sage/release/devel/sage-7.3.rc0-gcc-6.1.0/local/lib -o libecm.la -rpath /data/Sage/release/devel/sage-7.3.rc0-gcc-6.1.0/local/lib libecm_la-ecm.lo libecm_la-ecm2.lo libecm_la-pm1.lo libecm_la-pp1.lo libecm_la-getprime_r.lo libecm_la-listz.lo libecm_la-lucas.lo libecm_la-stage2.lo libecm_la-mpmod.lo libecm_la-mul_lo.lo libecm_la-polyeval.lo libecm_la-median.lo libecm_la-schoen_strass.lo libecm_la-ks-multiply.lo libecm_la-rho.lo libecm_la-bestd.lo libecm_la-auxlib.lo libecm_la-random.lo libecm_la-factor.lo libecm_la-sp.lo libecm_la-spv.lo libecm_la-spm.lo libecm_la-mpzspm.lo libecm_la-mpzspv.lo libecm_la-ntt_gfp.lo libecm_la-ecm_ntt.lo libecm_la-pm1fs2.lo libecm_la-sets_long.lo libecm_la-auxarith.lo libecm_la-batch.lo libecm_la-parametrizations.lo libecm_la-cudawrapper.lo aprtcle/libecm_la-mpz_aprcl.lo   ./x86_64/libmulredc.la  -lrt -lm -lm -lm -lm -lm
libtool: link: gcc-6.1 -shared  -fPIC -DPIC  .libs/libecm_la-ecm.o .libs/libecm_la-ecm2.o .libs/libecm_la-pm1.o .libs/libecm_la-pp1.o .libs/libecm_la-getprime_r.o .libs/libecm_la-listz.o .libs/libecm_la-lucas.o .libs/libecm_la-stage2.o .libs/libecm_la-mpmod.o .libs/libecm_la-mul_lo.o .libs/libecm_la-polyeval.o .libs/libecm_la-median.o .libs/libecm_la-schoen_strass.o .libs/libecm_la-ks-multiply.o .libs/libecm_la-rho.o .libs/libecm_la-bestd.o .libs/libecm_la-auxlib.o .libs/libecm_la-random.o .libs/libecm_la-factor.o .libs/libecm_la-sp.o .libs/libecm_la-spv.o .libs/libecm_la-spm.o .libs/libecm_la-mpzspm.o .libs/libecm_la-mpzspv.o .libs/libecm_la-ntt_gfp.o .libs/libecm_la-ecm_ntt.o .libs/libecm_la-pm1fs2.o .libs/libecm_la-sets_long.o .libs/libecm_la-auxarith.o .libs/libecm_la-batch.o .libs/libecm_la-parametrizations.o .libs/libecm_la-cudawrapper.o aprtcle/.libs/libecm_la-mpz_aprcl.o  -Wl,--whole-archive ./x86_64/.libs/libmulredc.a -Wl,--no-whole-archive  -lgmp -L/data/Sage/release/devel/sage-7.3.rc0-gcc-6.1.0/local/lib -lrt -lm  -g -march=native -g -O3 -g -Wl,-z -Wl,noexecstack -Wl,-rpath -Wl,/data/Sage/release/devel/sage-7.3.rc0-gcc-6.1.0/local/lib   -Wl,-soname -Wl,libecm.so.1 -o .libs/libecm.so.1.0.0
libtool: link: (cd ".libs" && rm -f "libecm.so.1" && ln -s "libecm.so.1.0.0" "libecm.so.1")
libtool: link: (cd ".libs" && rm -f "libecm.so" && ln -s "libecm.so.1.0.0" "libecm.so")
libtool: link: (cd .libs/libecm.lax/libmulredc.a && ar x "/tmp/sage-build-tmp/ecm-7.0.3/src/./x86_64/.libs/libmulredc.a")
libtool: link: ar cru .libs/libecm.a  libecm_la-ecm.o libecm_la-ecm2.o libecm_la-pm1.o libecm_la-pp1.o libecm_la-getprime_r.o libecm_la-listz.o libecm_la-lucas.o libecm_la-stage2.o libecm_la-mpmod.o libecm_la-mul_lo.o libecm_la-polyeval.o libecm_la-median.o libecm_la-schoen_strass.o libecm_la-ks-multiply.o libecm_la-rho.o libecm_la-bestd.o libecm_la-auxlib.o libecm_la-random.o libecm_la-factor.o libecm_la-sp.o libecm_la-spv.o libecm_la-spm.o libecm_la-mpzspm.o libecm_la-mpzspv.o libecm_la-ntt_gfp.o libecm_la-ecm_ntt.o libecm_la-pm1fs2.o libecm_la-sets_long.o libecm_la-auxarith.o libecm_la-batch.o libecm_la-parametrizations.o libecm_la-cudawrapper.o aprtcle/libecm_la-mpz_aprcl.o  .libs/libecm.lax/libmulredc.a/mulredc1.o .libs/libecm.lax/libmulredc.a/mulredc10.o .libs/libecm.lax/libmulredc.a/mulredc11.o .libs/libecm.lax/libmulredc.a/mulredc12.o .libs/libecm.lax/libmulredc.a/mulredc13.o .libs/libecm.lax/libmulredc.a/mulredc14.o .libs/libecm.lax/libmulredc.a/mulredc15.o .libs/libecm.lax/libmulredc.a/mulredc16.o .libs/libecm.lax/libmulredc.a/mulredc17.o .libs/libecm.lax/libmulredc.a/mulredc18.o .libs/libecm.lax/libmulredc.a/mulredc19.o .libs/libecm.lax/libmulredc.a/mulredc1_10.o .libs/libecm.lax/libmulredc.a/mulredc1_11.o .libs/libecm.lax/libmulredc.a/mulredc1_12.o .libs/libecm.lax/libmulredc.a/mulredc1_13.o .libs/libecm.lax/libmulredc.a/mulredc1_14.o .libs/libecm.lax/libmulredc.a/mulredc1_15.o .libs/libecm.lax/libmulredc.a/mulredc1_16.o .libs/libecm.lax/libmulredc.a/mulredc1_17.o .libs/libecm.lax/libmulredc.a/mulredc1_18.o .libs/libecm.lax/libmulredc.a/mulredc1_19.o .libs/libecm.lax/libmulredc.a/mulredc1_2.o .libs/libecm.lax/libmulredc.a/mulredc1_20.o .libs/libecm.lax/libmulredc.a/mulredc1_3.o .libs/libecm.lax/libmulredc.a/mulredc1_4.o .libs/libecm.lax/libmulredc.a/mulredc1_5.o .libs/libecm.lax/libmulredc.a/mulredc1_6.o .libs/libecm.lax/libmulredc.a/mulredc1_7.o .libs/libecm.lax/libmulredc.a/mulredc1_8.o .libs/libecm.lax/libmulredc.a/mulredc1_9.o .libs/libecm.lax/libmulredc.a/mulredc2.o .libs/libecm.lax/libmulredc.a/mulredc20.o .libs/libecm.lax/libmulredc.a/mulredc3.o .libs/libecm.lax/libmulredc.a/mulredc4.o .libs/libecm.lax/libmulredc.a/mulredc5.o .libs/libecm.lax/libmulredc.a/mulredc6.o .libs/libecm.lax/libmulredc.a/mulredc7.o .libs/libecm.lax/libmulredc.a/mulredc8.o .libs/libecm.lax/libmulredc.a/mulredc9.o
libtool: link: ranlib .libs/libecm.a
libtool: link: rm -fr .libs/libecm.lax
libtool: link: ( cd ".libs" && rm -f "libecm.la" && ln -s "../libecm.la" "libecm.la" )

(env ECM_CONFIGURE="--enable-shared" ./sage -f ecm)

[leif]

Replying to leif:

Replying to fbissey:

In Makefile.am you want to change line 52 which already has a definition for libecm_la_LDFLAGS.

Oooops, forgot a plus, but for whatever reason, it worked for me (with version info etc.).

No, after taking a closer look, I did not forget the plus. So nothing mysterious going on.

[fbissey]

But _I_ missed it :) real time for bed for me.

[leif]

Dreaming of libtool I guess... ;-)

[leif]

Oh-oh, it seems the last patch brings back ar cru libecm.a libgmp.a ... when building only a static library...

[fbissey]

I think we are majorly bike shedding.

Things to do: 1) -z noexecstack is a gnu ld option. Doesn't work on OS X. Change this and get it upstream.

2) if you provide enough info to configure to get the path to libgmp.a (not just detecting that you can link to gmp) and you build a static libecm then the build system will try to include libgmp.a in libecm.a.

• this fails on OS X
• if you build both a static and dynamic library interesting things will probably happen to the dynamic library, especially if libgmp.a has not been compiled with pic flags.

8 this is a recipe to squeeze as much performance as possible by getting an all static linking - it is possible to get the ecm binary compiled all static in some more portable ways (I think).

Number (2) is occurring in sage because we pass the install path to gmp/mpir when it is not needed to get gmp detected. This is my comment:76.

(1) has to be talked about with upstream (2) can easily build by adopting comment:76 in spkg-install.

[leif]

Yep.

For 1) we can just patch out -Wl,znoexecstack here and add it (or -Wa,--noexecstack) back in spkg-install when appropriate (i.e., supported; presumably just Linux or probably [recent] GAS).

And submit something upstream later.

To 2 ), dropping --with-gmp should IMHO only be a temporary work-around.

[leif]

Replying to leif:

To 2), dropping --with-gmp should IMHO only be a temporary work-around.

Is this sufficient or do we (still) have to patch out (the apparently plain wrong, with a static library at least)

libecm_la_LIBADD += $(GMPLIB)

?

I'll be back in a couple of hours...

[fbissey]

Replying to leif:

Replying to leif:

To 2), dropping --with-gmp should IMHO only be a temporary work-around.

Is this sufficient or do we (still) have to patch out (the apparently plain wrong, with a static library at least)

libecm_la_LIBADD += $(GMPLIB)

?

I'll be back in a couple of hours...

It is sufficient. I don't think Makefile.am is wrong. It is just feed something toxic in a corner case which upstream goes to great length to create.

Generally speaking OS X doesn't like static libraries much (still doable but some things are made difficult or more limited). So it probably should be avoided there.

[leif]

[Post to wrong ticket deleted.]

[leif]

Ouch, wrong ticket... :-)

[leif]

Just for the record, #21223 adds a Cygwin patch to the 6.4.4 package.

[leif]

Replying to leif:

Just for the record, #21223 adds a Cygwin patch to the 6.4.4 package.

... and we need to (trivially) backport that to 7.0.3 at least.

(Patches configure[.{in,ac}], single-line change each.)

[zimmerma]

I'm back from vacation, and I see about 100 messages in this ticket, woowww!!!! The 7.0.2 and 7.0.3 releases now each have a tag. Please tell me if you have still issues with 7.0.3, and some patches to be applied upstream.

Paul

[fbissey]

Replying to zimmerma:

I'm back from vacation, and I see about 100 messages in this ticket, woowww!!!! The 7.0.2 and 7.0.3 releases now each have a tag. Please tell me if you have still issues with 7.0.3, and some patches to be applied upstream.

Paul

I guess there are two main things we'll have to deal with:

• -Wl,znoexecstack only works with gnu ld. It causes OS X ld to choke.
• building against libgmp.a is also not working on OS X. That can be filtered out in configure.ac with an extra test in the logic to select pathto/libgmp.a

That's on top of my head.

[zimmerma]

Apparently also building out-of-tree is broken (more precisely, running make longcheck in an out-of-tree build, which definitely fails).

should be fixed in revision 2974.

Paul

[zimmerma]

should be fixed in revision 2974.

sorry, it should read 2975.

Paul

[zimmerma]

for the -Wl,znoexecstack issue, I've added a configure test in revision 2978. Please can you check it works on OS X now?

For the libgmp.a issue, please can you provide a patch?

Paul

[jpflori]

Posting back the issue I reported on #21223:

• on cygwin64, the assebmly for redc uses the GOT which is not supported on Windows when WANT_ASSERT is set and it is by default (passing --disable-assert circumvents the problem).

I'll now try different combinations of --enable-asm-redc and --enable-sse2 on Cygwin32 and Cygwin64 and report.

[jpflori]

(Do we need a so complicated spkg-install???)

[jpflori]

On my side, make check passes on a x86_64 machine with Cygwin32 passing --enable-sse2 --enable-asm-redc.

Same under Cygwin64 (modulo rebasing issues and passing --disable-assert).

[fbissey]

Replying to jpflori:

(Do we need a so complicated spkg-install???)

I personally don't think so... Too much hand holding on flags in my opinion.

[fbissey]

Replying to zimmerma:

for the -Wl,znoexecstack issue, I've added a configure test in revision 2978. Please can you check it works on OS X now?

I'll give it go when I have time (hopefully in the next few hours).

For the libgmp.a issue, please can you provide a patch?

Yes, I'll definitely work on that.

Francois

[fbissey]

On revision 2978 on OS X inside sage

configure: Memory debugging disabled

Now building GMP-ECM...
make  all-recursive
Making all in x86_64
m4 -I../ -DOPERATION_mulredc1 `test -f mulredc1.asm || echo './'`mulredc1.asm >mulredc1.s
/bin/sh ../libtool    --mode=compile gcc  -march=native -g -O3  -fPIC -c -o mulredc1.lo mulredc1.s
libtool: compile:  gcc -march=native -g -O3 -fPIC -c mulredc1.s -o mulredc1.o
mulredc1.s:59:2: error: unsupported symbol modifier in branch relocation
        call    abort@plt
        ^
make[2]: *** [mulredc1.lo] Error 1
rm mulredc1.s
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Error building GMP-ECM.

Ouch...

[jpflori]

Pass --disable-assert :p

[fbissey]

Thanks, that worked. Will we have to pass it for anything other than linux? Or should it be magically set?

In any case the main test was whether --noexecstack was properly disabled on OS X and it is.

[jpflori]

Replying to fbissey:

Thanks, that worked. Will we have to pass it for anything other than linux? Or should it be magically set?

I would say it's a bug. The assembly should be modified because:

• GOT is not supported on Windows (where everything is alway PIC),
• the call syntax is broken on OS X.

Using --disable-assert is just a workaround as it excludes the harming code. Note that we could still live without asserts on Cygwin and OS X.

In any case the main test was whether --noexecstack was properly disabled on OS X and it is.

Good!

[zimmerma]

about the issue from comment 121 (GOT not supported on Windows), I can't help on that. Please can someone provide a patch?

Since the noexecstack issue is now solved in the development version, it only remains to solve the libgmp.a issue, for which Francois is working on a patch.

Paul

[jpflori]

Replying to zimmerma:

about the issue from comment 121 (GOT not supported on Windows), I can't help on that. Please can someone provide a patch?

Note that the assembly line from 121 gives issues on OS X not Windows:

call abort@plt

The problematic line on Windows from #21223 is the previous one which uses:

$_GLOBAL_OFFSET_TABLE

The workaround I suggested is to use --disable-assert as both these lines are surrounded by a (m4-equivalent) of #ifdef HAVE_ASSERT.

A proper fix would be to:

• find equivalent assembly on OS X and Windows to keep asserts support,
• or always disable this code path on OS X and Windows.

[jpflori]

This comes from: ​https://gforge.inria.fr/scm/viewvc.php/ecm/trunk/x86_64/mulredc1.asm?r1=1460&r2=1526 and yes it is not portable!

In fact this only appeared here because in configure.ac (​https://gforge.inria.fr/scm/viewvc.php/ecm/trunk/configure.ac?view=markup#l51):

dnl Assertions are enabled by default for beta/rc releases.

[zimmerma]

thank you Jean-Pierre. If we go back to revision 1460, would it solve the OS X and Windows issues?

[jpflori]

Not completely sure if you mean go back to r1460 for OS X and Windows only or for all Oses.

Whatsoever, with no testing and realknowledge of what I'm talking about, I'd say that indeed using:

• using the r1460 code on Windows should do the trick: there the code is always PIC and hopefully no dark magic is needed to call abort;
• using the r1460 code on OS X I have no idea but it cannot be worse than now: binaries are not ELF, so no PLT, not sure if the simple call would always work in both shared and static libraries.

As far as Linux is concerned :

• either always use the r1526 code which according to the commit message always works,
• or use it only in PIC code and use the r1460 in non-PIC code.

[zimmerma]

I was meaning go back to r1460 in all cases. But I now propose to simply remove this WANT_ASSERT stuff in the assembly code. Anyway if something goes wrong, it will be detected by the assertions in the C code.

Paul

[jpflori]

Sure that's the more portable and easiest to maintain solution.

(And after a few more seconds searching about OS X, it might be the case that the toolchain there does the plt or equivalent black magic by itself.)

[zimmerma]

the WANT_ASSERT stuff has been removed in upstream revision 2981. This should solve the OS X and Windows issues.

Paul

[fbissey]

Replying to zimmerma:

about the issue from comment 121 (GOT not supported on Windows), I can't help on that. Please can someone provide a patch?

Since the noexecstack issue is now solved in the development version, it only remains to solve the libgmp.a issue, for which Francois is working on a patch.

Paul

Sent your way in a private email.

Francois

[zimmerma]

the patch for the libgmp.a issue on OS X is integrated in upstream revision 2982. Are all known issues solved now?

Paul

PS: I'll probably make a 7.0.4 patch release since a bug was reported in mpres_pow().

[zimmerma]

a new release candidate is available at

​https://members.loria.fr/PZimmermann/ecm-7.0.4-rc1.tar.gz

Please test and report success or failure.

Paul

[fbissey]

I still need --disable-assert on OS X otherwise I get

/bin/sh ./libtool  --tag=CC   --mode=link gcc  -g -march=native -g -O3  -fPIC  -L/Users/fbissey/build/sage-7.2.beta5/local/lib -Wl,-rpath,/Users/fbissey/build/sage-7.2.beta5/local/lib  -L/Users/fbissey/build/sage-7.2.beta5/local/lib -o ecm ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o libecm.la -lgmp -lm -lm -lm -lm -lm  
libtool: link: gcc -g -march=native -g -O3 -fPIC -Wl,-rpath -Wl,/Users/fbissey/build/sage-7.2.beta5/local/lib -o ecm ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o  -L/Users/fbissey/build/sage-7.2.beta5/local/lib ./.libs/libecm.a -lgmp -lm
Undefined symbols for architecture x86_64:
  "_fft_adjust_limbs", referenced from:
      ___ecm_mpres_mul_z_to_z in libecm.a(libecm_la-mpmod.o)
      ___ecm_mpres_mul in libecm.a(libecm_la-mpmod.o)
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [ecm] Error 1

ecm test suite is currently running but I am not expecting any failures.

[zimmerma]

Francois, I guess you are compiling GMP-ECM with MPIR, not with GMP. Since there is currently no public FFT interface for GMP, and the internal functions differ between GMP and MPIR, I decided to only support the GMP side. Thus unless someone volunteers to maintain GMP-ECM up to date with changes in MPIR (apparently fft_adjust_limbs now disappeared), MPIR will not be supported any more.

Paul

[fbissey]

Indeed I was compiling from inside a sagemath install with mpir. Which of course is the default in this case. Your comments make me wonder about what is happening on the linux side. Anyone???

[jpflori]

As long as this only fails with --enable-assert we could live with that failure. From a quick discussion with Bill Hart, I don't see anything on the MPIR side to converge back to the GMP interface in the near future.

I'll test on Linux when I find some time this week.

[jpflori]

I get the same failure on Linux with --enable-assert:

tune-mpmod.o: In function `__ecm_mpres_mul_z_to_z':
/home/jpflori/trunk/mpmod.c:1594: undefined reference to `fft_adjust_limbs'
tune-mpmod.o: In function `__ecm_mpres_mul':
/home/jpflori/trunk/mpmod.c:1419: undefined reference to `fft_adjust_limbs'
collect2: error: ld returned 1 exit status

No issue with --disable-assert.

[jpflori]

Is there really any more work needed here? Wait for a new upstream dot release ? Add --disable-assert to our configuration?

[fbissey]

I guess we should check if 7.0.4 is out and use --disable-assert.

[zimmerma]

7.0.4 is now out.

Paul

[fbissey]

OK, I'll try to put the branch in shape in the next 48hours.

[jpflori]

Or 48days :)

[fbissey]

I have a branch sitting there and for some reasons I don't remember I didn't push. I'll have to examine what it is. It is not finished in any case as changes in ecm interfaces are not yet taken into account in sage. It is just the package right now.

[isuruf]

Here's a patch that fixes sage interface, ​https://git.archlinux.org/svntogit/community.git/tree/trunk/ecm-7.patch?h=packages/sagemath

[fbissey]

I should compare it to the debian one. I was hoping that a debian or arch dev would finish the ticket so they can be properly credited but obviously someone else will have to.

[isuruf]

debian patch is from arch. You can give credit by setting the git author to the author of the patch, ​https://git.archlinux.org/svntogit/community.git/patch/trunk/ecm-7.patch?id=e6b360003ee0d12fcc7c2a7e3eaa60bdb570325f

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​eaee6e0

Fix libecm doctests

[jpflori]

@jeroen: more work needed?

[jdemeyer]

This has not been addressed yet:

Replying to zimmerma:

• We definitely also have to fix the pexpect interface to ecm in src/sage/interfaces/ecm.py, as the command line interface has changed in GMP-ECM 7.x.

[jdemeyer]

I don't plan to work more on this for now.

[fbissey]

Adding the patch from arch linux (and credit the author properly) and let's try to finish this.

---

New commits:

​dea25cd	Merge branch 'develop' into gmp-ecm-7.x
​dde736b	Add interface patch for emc 7 from the Arch linux port. The Arch commit is credited to Antonio Rojas (arojas@archlinux.org)

[jpflori]

Thanks I was working on this but it will be faster to just review this patch.

[vbraun]

Testsuite fails:

[ecm-7.0.4.p0] ********** Factor found in step 2: 4190453151940208656715582382315221647
[ecm-7.0.4.p0] Found prime factor of 37 digits: 4190453151940208656715582382315221647
[ecm-7.0.4.p0] Prime cofactor 543423179434447039008165356160798838947285203071935410761431031147 has 66 digits
[ecm-7.0.4.p0] GMP-ECM 7.0.4 [configured with MPIR 2.7.2, --enable-asm-redc] [P+1]
[ecm-7.0.4.p0] Save file test.pp1.save already exists, will not overwrite
[ecm-7.0.4.p0] ############### ERROR ###############
[ecm-7.0.4.p0] Expected return code 0 but got 1
[ecm-7.0.4.p0] Makefile:2521: recipe for target 'longcheck' failed
[ecm-7.0.4.p0] make[3]: *** [longcheck] Error 1
[ecm-7.0.4.p0] PASS: test.pm1
[ecm-7.0.4.p0] PASS: test.pp1
[ecm-7.0.4.p0] PASS: test.ecm
[ecm-7.0.4.p0] ============================================================================
[ecm-7.0.4.p0] Testsuite summary for ecm 7.0.4
[ecm-7.0.4.p0] ============================================================================
[ecm-7.0.4.p0] # TOTAL: 3
[ecm-7.0.4.p0] # PASS:  3
[ecm-7.0.4.p0] # SKIP:  0
[ecm-7.0.4.p0] # XFAIL: 0
[ecm-7.0.4.p0] # FAIL:  0
[ecm-7.0.4.p0] # XPASS: 0
[ecm-7.0.4.p0] # ERROR: 0
[ecm-7.0.4.p0] ============================================================================
[ecm-7.0.4.p0] make[6]: Leaving directory '/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/ecm-7.0.4.p0/src'
[ecm-7.0.4.p0] make[5]: Leaving directory '/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/ecm-7.0.4.p0/src'
[ecm-7.0.4.p0] make[4]: Leaving directory '/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/ecm-7.0.4.p0/src'
[ecm-7.0.4.p0] make[3]: Leaving directory '/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/ecm-7.0.4.p0/src'
[ecm-7.0.4.p0] Error: GMP-ECM's test suite failed.

• ​http://build.sagedev.org/#/builders/82/builds/8/steps/5/logs/stdio
• ​http://build.sagedev.org/#/builders/38/builds/6/steps/5/logs/stdio

[fbissey]

Looks like a race condition in the test suite.

[fbissey]

Yes, from Makefile.am

longcheck: ecm$(EXEEXT)
        $(srcdir)/test.pp1 "$(VALGRIND) ./ecm$(EXEEXT)"
        $(srcdir)/test.pp1 "$(VALGRIND) ./ecm$(EXEEXT) -no-ntt"
        $(srcdir)/test.pp1 "$(VALGRIND) ./ecm$(EXEEXT) -modmuln"
        $(srcdir)/test.pp1 "$(VALGRIND) ./ecm$(EXEEXT) -redc"
        $(srcdir)/test.pp1 "$(VALGRIND) ./ecm$(EXEEXT) -mpzmod"
        $(srcdir)/test.pm1 "$(VALGRIND) ./ecm$(EXEEXT)"
        $(srcdir)/test.pm1 "$(VALGRIND) ./ecm$(EXEEXT) -no-ntt"
        $(srcdir)/test.pm1 "$(VALGRIND) ./ecm$(EXEEXT) -modmuln"
        $(srcdir)/test.pm1 "$(VALGRIND) ./ecm$(EXEEXT) -redc"
        $(srcdir)/test.pm1 "$(VALGRIND) ./ecm$(EXEEXT) -mpzmod"
        $(srcdir)/test.ecm "$(VALGRIND) ./ecm$(EXEEXT)"
        $(srcdir)/test.ecm "$(VALGRIND) ./ecm$(EXEEXT) -no-ntt"
        $(srcdir)/test.ecm "$(VALGRIND) ./ecm$(EXEEXT) -modmuln"
        $(srcdir)/test.ecm "$(VALGRIND) ./ecm$(EXEEXT) -redc"
        $(srcdir)/test.ecm "$(VALGRIND) ./ecm$(EXEEXT) -mpzmod"
        $(srcdir)/test.ecm "$(VALGRIND) ./ecm$(EXEEXT) -treefile tree"
        $(srcdir)/testlong.pp1 "$(VALGRIND) ./ecm$(EXEEXT)"
        $(srcdir)/testlong.pm1 "$(VALGRIND) ./ecm$(EXEEXT)"
        $(srcdir)/testlong.ecm "$(VALGRIND) ./ecm$(EXEEXT)"

and on both bots the test are probably with -j8. Each instance of test.pp1 can write test.pp1.save. There could be similar problem with other tests. The only solution, I think, is to run the longcheck serially.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​946c580

making the longcheck tests run serially

[fbissey]

Putting back to positive review to send back to the bots.

[zimmerma]

for the record, I have fixed upstream (svn revision 3022) so that we can now run make check in parallel. You could also apply the corresponding patch.

[fbissey]

I see, you made all those little checkpointing files unique rather than re-using the same one all the time. Well this ticket is closed but we could have a follow up. And hopefully it won't languish for as much time as this one.

[vbraun]

Followup at #23013

[jdemeyer]

Since this hasn't been released yet, why not just unmerge this?

[vbraun]

No particular reason...