spkg-configure.m4 for gmp and mpir

[dimpase]

This should be easy, and it's a potential source of library conflicts, as gmp is often installed...

note that GMP is pulled in via ./configure --with-mp=gmp (as well as by specifying SAGE_MP_LIBRARY?)

Configure: ​http://users.ox.ac.uk/~coml0531/sage/configure-316.tar.gz

[dimpase]

here we check that GMP is new enough, by verifying it has a new function. The rest is just copypaste.

To actually build with GMP, one needs

./configure --with-mp=gmp

(configure does the test regardless)

---

New commits:

​1ed38e1

spkg-configure for GMP

[dimpase]

I suppose due to a special treatment of GMP/MPIR, despite configure reporting GMP as not being installed, it still gets picked up for installation from source. (I tested this on a clean build...)

[embray]

Yes, I've already put some thought into how to handle this case, though I don't have a solid answer. The spkg-configure.m4 seems fine, but we can't just leave it at that.

The tricky thing is that with MPIR it can be compiled with or without gmp-compatibility mode. On Sage we use --with-gmpcompat so if we link to a gmp we're actually getting the one installed by MPIR. But some systems might have MPIR built without gmp-compatiblity, and yet no separate libgmp.so. So in that case we'd actually want to use libmpir (if such a system exists I'm not sure).

So if we run ./configure with no --with-mp= flag the question is what to do? Do we just try to detect any "libgmp" and use that if found? Do look for a libmpir, ever? If the user passes --with-mp=mpir or --with-mp=gmp should that imply that it should just build the Sage SPKG, or do we explicitly go searching for, say, a libmpir in the former case?

[embray]

Deciding which BLAS to use from the system raises similar questions.

[embray]

The more I think about it, the more I'm inclined toward --with-mp=mpir should mean that the MP library must by MPIR by explicit request of the user, and if not found on the system then we should build the SPKG.

If --with-mp= is not otherwise specified then we can take whatever MP lib was found on the system.

We should also add an spkg-configure.m4 for MPIR. We can impose an ordering (e.g. first check for MPIR, and then for plain GMP) by putting AC_REQURE(SAGE_SPKG_CONFIGURE_MPIR) at the beginning of the spkg-configure.m4 for gmp. I did something similar for gfortran/gcc and seems to work.

[embray]

It should also be noted that we have also in the past patched other SPKGS (see e.g. #21188) to always link with -lgmp and #include <gmp.h> (and never use -lmpir and #include <mpir.h> respectively).

I'm not 100% convinced that that's necessary, though certainly one or the other should be used. So it's probably best to always ensure that there is at least a working libgmp and a gmp.h. The question then is whether or not it's possible to determine if that libgmp happened to be provided by MPIR.

Update: It seems that a libgmp.so provided by MPIR does contain a __mpir_version symbol, so that's something we can look for.

[embray]

It seems MPIR is not in fact packaged for Debian (not sure about other distros): ​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=708391 I had just assumed it would be a question of using either MPIR or GMP (in other words, they conflict with each other, and installing either one or the other is possible, as is the case in Sage).

[dimpase]

Replying to embray:

Yes, I've already put some thought into how to handle this case, though I don't have a solid answer. The spkg-configure.m4 seems fine, but we can't just leave it at that.

The tricky thing is that with MPIR it can be compiled with or without gmp-compatibility mode. On Sage we use --with-gmpcompat so if we link to a gmp we're actually getting the one installed by MPIR. But some systems might have MPIR built without gmp-compatiblity, and yet no separate libgmp.so. So in that case we'd actually want to use libmpir (if such a system exists I'm not sure).

FreeBSD provides GMP and MPIR, not in GMP-compatibility mode (I looked at the corresponding makefile) ​https://www.freshports.org/math/mpir ​https://www.freshports.org/math/gmp

both are in /usr/local/ there.

I would not be worried about a hypothetical situation of many libgmp's on the system (in fact, MPFR has a very long configure script dealing with detecting all sorts of info about GMP (or a GMP-compatible thing)

[embray]

I guess the main question is what to do with the --with-mp= flag in Sage's ./configure script. I don't think it necessarily needs to continue working exactly the same way, but whatever it does it should be sensible. The way I see there are a couple possible ways to interpret it:

One way is that there are really three possible values:

• detect
• gmp
• mpir

where the default value is "detect". In the "detect" case we just try to find the best working libgmp on the system and use it if found. If not found the next fallback would be to install the mpir SPKG (the current default). If passed "mpir" or "gmp" explicitly we don't bother detecting and just install the specifically requested SPKG.

Another possible interpretation is something more like:

• gmp
• mpir

where now default value "gmp" is similar to the first case: just try to detect any libgmp on the system to use, and failing that install an SPKG for some libgmp (where we might still use MPIR, but that fact is an implementation detail), whereas "mpir" means "detect MPIR specifically", which is to say find a libgmp that also, at a minimum, contains the __mpir_version flag. Perhaps, for some reason, a user might specifically want to use MPIR e.g. for performance reasons. If not found then we build the MPIR SPKG and use that.

Perhaps there are some other ways of interpreting/re-interpreting this flag, but those are a couple I have in mind.

[embray]

I think I lean more toward the latter case. Most people who want to build Sage do not care what GMP implementation they use, so I think it's good enough to use the one they have, if any, and if not pick one for them. Whereas if someone is explicitly passing --with-mp=mpir then the really want MPIR specifically, and we should not give them something that isn't it.

OTOH I'd also be curious in what cases MPIR gives better performance, and if we're doing users a disservice by not at least strongly recommending that they use it. I think this is more a question for the mailing list though. The details of what to do at configure-time can be malleable.

[dimpase]

Perhaps we could leave the mpir-based toolchain aside for a moment, after all mpir is an optional package, so it is a bug if it is pulled in once GMP is chosen, and concentrate on the rest of the surrounding number theory libs: mpc, mpfr, gf2x, ntl, pari.

So first we have to sort out why GMP is pulled in despite being recognised by configure as not to be installed, no?

---

Note that this spurious GMP installation isn't even triggered before some of spkgs dependent on GMP are installed, e.g. MPFR happily installs long before it's started on GMP. I get

$ ldd local/lib/libmpfr.so
	linux-vdso.so.1 (0x00007ffcc02e9000)
	libgmp.so.10 => /usr/lib64/libgmp.so.10 (0x00007f9345368000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f9345199000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9345696000)

but the beginning of MPFR install log says, wrongly,

  --with-gmp="/mnt/opt/Sage/sage-dev/local"

This is of course as expected, as it's hardwired in its spkg-install, as sdh_configure --with-gmp="$SAGE_LOCAL"...

It appears to me that instead the main ./configure should set up something like SAGE_GMP to be either SAGE_LOCAL or a location computed by GMP's spkg-configure.m4, and then this SAGE_GMP must be used by MPFR...

[dimpase]

And MPFR's only part of trouble:

$ grep "gmp=..SAGE_LOCAL." */spkg-install  
arb/spkg-install:    --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL"
deformation/spkg-install:            --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" \
ecm/spkg-install:sdh_configure --with-gmp="$SAGE_LOCAL" $ECM_CONFIGURE
flint/spkg-install:    --with-gmp="$SAGE_LOCAL" \
gap/spkg-install:sdh_configure --with-gmp="$SAGE_LOCAL"
gcc/spkg-install:    --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" --with-mpc="$SAGE_LOCAL" \
gdb/spkg-install:    --with-gmp="$SAGE_LOCAL"
gfortran/spkg-install:    --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" --with-mpc="$SAGE_LOCAL" \
givaro/spkg-install:sdh_configure --with-gmp="$SAGE_LOCAL" --enable-shared $GIVARO_CONFIGURE
glpk/spkg-install:#       GMP/MPIR and zlib, so we should just use `--with-gmp="$SAGE_LOCAL"` and
mpc/spkg-install:sdh_configure --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" $EXTRA
mpfi/spkg-install:sdh_configure --with-mpfr="$SAGE_LOCAL" --with-gmp="$SAGE_LOCAL"
mpfrcx/spkg-install:sdh_configure --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" \
mpfr/spkg-install:        ./configure --with-gmp="$SAGE_LOCAL" $SAGE_CONF_OPTS $MPFR_CONFIGURE) &>/dev/null;
mpfr/spkg-install:    sdh_configure --with-gmp="$SAGE_LOCAL" \
pari/spkg-install:    --with-readline="$SAGE_LOCAL" --with-gmp="$SAGE_LOCAL" \
polylib/spkg-install:sdh_configure --with-libgmp=${SAGE_LOCAL}/lib
singular/spkg-install:                  --with-gmp="$SAGE_LOCAL" \
topcom/spkg-install:    --with-gmp="$SAGE_LOCAL" \

So passing SAGE_LOCAL left and right for everything is something to work on.

[dimpase]

I don't think I understand how the .dummy target for skipping "dummy" packages in build/make/Makefile works, in particular, why it's just a single $(INST)/.dummy file for all of them.

Or perhaps it's because gmp spkg is known as well as $(MP_LIBRARY), and it's not found on the list of "dummy" packages, which duly includes gmp...

[dimpase]

I've added spkg-configures to the whole "toolchain" bunch (see attachment) - the one for MPIR is basically just a plug, and will now try running the build to see if GMP install still surfaces...

[dimpase]

no, this does not help - and while I didn't have patience for the build to come to a halt, I still saw GMP getting built. (and none of MP*'s)

[embray]

The thing to do with all of these is that configure needs to output some config file based on its results which contains the correct prefixes for all these libraries (and likely others to come).

For this it might be useful to use the existing sage-env-config file, since I believe it gets sourced when building each package. It would probably be useful to have there since having the correct paths to libraries may also be useful at runtime (e.g. when running cython(), I think). To this end it would also be nice to have a sort of sage.env_config Python module that is also generated, so these values can easily be loaded by the Python library without sage-env being sourced, but this is not necessary for the SPKG build process.

Then every --with-gmp=$SAGE_LOCAL" should be replaced with --with-gmp=$SAGE_GMP_PREFIX", or something like that (the value of which is still ofc $SAGE_LOCAL when using a GMP installed from an SPKG).

What I'm less sure about is how exactly to determine the value for $SAGE_GMP_PREFIX. Macros like AC_SEARCH_LIBS only tests that linking with -l<libname> works; it has no clue about how the system searches library paths in that case. I think the best way to handle this is not unlike the code I wrote in #27230, unfortunately.

[dimpase]

I think the .dummy magic breaks for $(MP_LIBRARY). To prove this, I did

build/make/deps

@@ -77 +77 @@
 # See #14168 and #14232.
 #
 # Note: This list consists of only the *runtime* dependencies of the toolchain.
-TOOLCHAIN_DEPS = zlib $(MP_LIBRARY) mpfr mpc
+TOOLCHAIN_DEPS = zlib gmp mpfr mpc
 TOOLCHAIN_DEP_INSTS = \
        $(foreach pkgname,$(TOOLCHAIN_DEPS),$(inst_$(pkgname)))
 
@@ -156 +156 @@
                $(inst_mpc) \
                $(inst_mpfi) \
                $(inst_mpfr) \
-               $(MP_LIBRARY) \
                $(inst_ntl) \
                $(inst_numpy) \
                $(inst_pari) \

(probably only the 2nd change, removal from sagelib: deps matters, not the 1st change.)

and, viola, no GMP building pops up. Needless to say, "dummies" of the form $(inst_...) do not get built.

[embray]

Replying to dimpase:

I think the .dummy magic breaks for $(MP_LIBRARY). To prove this, I did

Sorry, I didn't realize you were being stymied by this. Yes, to me there was no mystery here, which is in part why I was focused on how best to handle the existing --with-mp flag. I should have been more clear about that.

[embray]

Replying to embray:

What I'm less sure about is how exactly to determine the value for $SAGE_GMP_PREFIX. Macros like AC_SEARCH_LIBS only tests that linking with -l<libname> works; it has no clue about how the system searches library paths in that case. I think the best way to handle this is not unlike the code I wrote in #27230, unfortunately.

Another possibility of course is to just leave it empty and not pass --with-gmp flags at all, in this case they will be forced to just pick up whatever GMP they find on the default paths as well, which usually will be the same one.

[dimpase]

Pragmatically, I'd say that--with-mp=foo ought to always install foo from Sage's source, unless foo=detect. This is your first possibility from comment 9. (And leaving it out would result in either picking up system's "GMP", be it real GMP or MPIR in compatibility mode.)

This is reasonably flexible. Experts who need special versions of GMP/MPIR may just update Sage's packages and use them, or install them system-wide.

Most of all, it would not need fiddling with recongising "hidden" MPIR in a system's libgmp, something that would otherwise cater for non-existing systems, i.e. pure over-design for nothing and nobody...

[embray]

Replying to dimpase:

Pragmatically, I'd say that--with-mp=foo ought to always install foo from Sage's source, unless foo=detect. This is your first possibility from comment 9. (And leaving it out would result in either picking up system's "GMP", be it real GMP or MPIR in compatibility mode.)

This is reasonably flexible. Experts who need special versions of GMP/MPIR may just update Sage's packages and use them, or install them system-wide.

Most of all, it would not need fiddling with recongising "hidden" MPIR in a system's libgmp, something that would otherwise cater for non-existing systems, i.e. pure over-design for nothing and nobody...

Okay, let's go with that then, for now. That would be good because it is also the most "backwards compatible" with the current behavior.

As for the --with-gmp= question, I keep thinking about it and I think this is the best thing to do:

• in the gmp/spkg-configure.m4 add support for our own --with-gmp= flag. Most people won't need this, but it does provide some desirable flexibility. This would just set the GMP prefix explicitly (and be saved to $SAGE_GMP_PREFIX) to be passed on to other packages that look for a GMP

• when passed an explicit --with-gmp flag that should also imply that we want to use the system package, and if it can't be found this should be an error

• if a usable system GMP is detected (with or without --with-gmp) just go with it; if --with-gmp was not specified then we set $SAGE_GMP_PREFIX to blank, and in that case explicitly do not pass --with-gmp= to other packages: let them just pick up the system GMP (note: I checked, and passing --with-gmp= with a blank argument is buggy in at least a few of these packages, so better not to use the flag at all in that case)

• if a usable system GMP is not detected we just build the SPKG as normal, with whatever option was selected by --with-mp, with MPIR by default and set SAGE_GMP_PREFIX=$SAGE_LOCAL.

[embray]

I'm working on a prototype for the above scheme. Which is not to say I'm convinced is the best way, but it seems worth a try.

[dimpase]

OK, great. By the way, on Gentoo, with these extra spkg-configures in Attachment (all of them used to use system's libraries) , I let make ptests run, and all the tests passed.

I actually had one more, for gf2x

SAGE_SPKG_CONFIGURE([gf2x], [
    AC_CHECK_HEADER(gf2x.h, [], [sage_spkg_install_gf2x=yes])
dnl gf2x_mul_r appeared in version 1.2 of GF2X 
    AC_SEARCH_LIBS([gf2x_mul_r], [gf2x], [break], [sage_spkg_install_gf2x=yes])
])

I guess it should go on another ticket.

[embray]

Replying to embray:

• in the gmp/spkg-configure.m4 add support for our own --with-gmp= flag. Most people won't need this, but it does provide some desirable flexibility. This would just set the GMP prefix explicitly (and be saved to $SAGE_GMP_PREFIX) to be passed on to other packages that look for a GMP

• when passed an explicit --with-gmp flag that should also imply that we want to use the system package, and if it can't be found this should be an error

It turns out doing this right is likely more trouble than it's worth, so I'm going to say "YAGNI" to this, until and unless proven otherwise. It turns out even most of the packages that have a --with-gmp flag in their own configures don't really implement it well. As long as Sage is using the system GMP, this flag shouldn't really be needed anyways for compiling other packages against the system GMP. So better to just leave it off entirely, at least when using the system package.

[embray]

I updated a bunch of spkg-installs to not use --with-gmp when using the system package. The ./configure happily detected my system's GMP 6.1.2 (the most recent). Doing a full rebuild and make ptestlong against that to test.

[embray]

Getting weird segfaults and other errors from cypari when building with my system's GMP. Not sure if there's any relation though, something else could be wrong, at this point, with this build...

[embray]

Okay, I don't know where I got the idea that this system has GMP 6.1.2. I appear to have gotten mixed up between machines. This machine has GMP 5.1.3. Nevertheless that does seem to be a problem because this spkg-configure.m4 allowed me to use it. But apparently PARI does not support this old a version.

I get

[pari-2.11.1.p0] linux-x86_64-gmp30818: error while loading shared libraries: libgmp.so.23: cannot open shared object file: No such file or directory
[pari-2.11.1.p0] ### Your version of GMP is too old for PARI. Please upgrade

I don't know why it's looking for some "libgmp.so.23" though. I don't have anything like that on my system. So this could be a bug in the PARI build system. I will see if I can convince PARI to use this GMP and if it will work. Nevertheless it's not encouraging.

[dimpase]

here is what Singular does: ​https://github.com/Singular/LELA/blob/master/macros/gmp-check.m4

[embray]

Okay, I think pari does not actually need this version of GMP (which is not to say some other package might not need it, but I want to see how far I can get).

What was messing me up was that I did not prevent mpir from being installed in this build, so it was partially picking up the MPIR in $SAGE_LOCAL but in an incomplete and broken manner. After applying part of my fix to prevent MPIR from being installed, PARI built okay.

[embray]

Final update on this before I really go to sleep: pynac uses mpq_cmp_z which is definitely only available as of GMP 6.1.0. pynac could probably manage to not use it if it's not available, but I don't think it's worth bothering with. I'm perfectly happy to just require GMP>=6.1.0 in that case, which is still rather old at this point.

The symbol to check for in this case is __gmpq_cmp_z. I confirmed in does not exist in GMP 6.0.x or below, and has the benefit of being a function that is actually used in one of the dependents of GMP.

[embray]

Here's my attempt at this:

Moved all the --with-mp logic into the spkg-configure.m4 for MPIR so that it is local to where it is used.

Also, most of the logic is in mpir/spkg-configure.m4 rather than gmp/spkg-configure.m4 just because MPIR is the implementation we install by default if it can't be used from the sytem. It could go either way, but this way seemed a bit more logical for that reason.

Then, the gmp/spkg-configure.m4 is mostly just a stub.

Finally, uses the symbol __gmpq_cmp_z to check for GMP>=6.1.0 (or whatever the equivalent MPIR is).

So to summarize, by default --with-mp=system and it tries to use the libgmp from the system. If that check fails it falls back to installing the mpir SPKG. If --with-mp=mpir it just doesn't even check the system and installs the mpir SPKG; similarly for --with-mp=gmp.

[embray]

Oops, typo on branch name.

---

New commits:

​1ed38e1	spkg-configure for GMP
​dd39b84	Move all the --with-mp logic into the spkg-configure.m4 for MPIR so that it

[embray]

I accidentally left off a bunch of changes related to the --with-gmp flags in other packages.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​356621a	correctly setup sagelib dependencies-trac #27254
​da291d8	spkg-configure for GMP
​f7698ad	Move all the --with-mp logic into the spkg-configure.m4 for MPIR so that it
​27b4692	Replace various --with-gmp=$SAGE_LOCAL flags in spkg-installs with a

[embray]

Also rebased on #27254.

[embray]

I just realized there is another problem with this, which didn't appear until I rebased. Now if SAGE_MP_LIBRARY=system things break because there is no target "system".

[dimpase]

the ECM bit looks fishy:

--- a/build/pkgs/ecm/spkg-install
+++ b/build/pkgs/ecm/spkg-install
@@ -210,7 +210,10 @@ else
 fi
 echo "  --prefix=\"$SAGE_LOCAL\""
 echo "  --libdir=\"$SAGE_LOCAL/lib\""
-echo "  --with-gmp=\"$SAGE_LOCAL\""
+if [ -n "$SAGE_GMP_CONFIGURE" ]; then
+    echo "  --with-gmp=\"$SAGE_LOCAL\""
+fi
+

[embray]

It is a typo. It should be $SAGE_CONFIGURE_GMP. It's not really important though, I was just retaining the existing behavior in that code.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​566bd21	Reworked this a bit more
​a4ab468	fix typo
​b979d7b	added a bit more explanation

[embray]

I think this is fixed up now, though the commits should be squashed before setting final positive_review.

[dimpase]

I see no chaining in mpir/mpfr macros - if I intentionally break mpir recognition I still get mpfr not being installed, but picked up from the system... ---if I just run ./configire without parameters.

(I don't know whether this breaks builds or not, but I guess we would like to see in such a case, if mpfr is not available, mpfr being built as well)

[dimpase]

With this branch, I've run into a problem with IML package, see #27272

[dimpase]

Replying to dimpase:

With this branch, I've run into a problem with IML package, see #27272

this is platform-specific - e.g. on Fedora 26 I don't get this problem.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1e48b30

correct logic for SAGE_GMP_PREFIX etc

[isuruf]

There are some sage dependencies using libgmpxx and if gmpxx was compiled with a different C++ standard library, you'll get link failures. It'll be good check for this here.

[isuruf]

Here's an example file you can use, ​https://github.com/symengine/symengine/blob/master/cmake/checkgmpxx.cpp. This check fails when gmpxx is compiled using g++ and the dependent library is compiled using clang++ and vice-versa on OSX.

[dimpase]

Replying to isuruf:

Here's an example file you can use, ​https://github.com/symengine/symengine/blob/master/cmake/checkgmpxx.cpp. This check fails when gmpxx is compiled using g++ and the dependent library is compiled using clang++ and vice-versa on OSX.

I suppose that there is no compatibility even across different versions of g++, and perhaps not even when the same compiler is used with different -std=.

[isuruf]

​https://stackoverflow.com/questions/46746878/is-it-safe-to-link-c17-c14-and-c11-objects should answer your general question. For gmpxx though, I don't think it matters a lot.

[dimpase]

Replying to isuruf:

​https://stackoverflow.com/questions/46746878/is-it-safe-to-link-c17-c14-and-c11-objects should answer your general question. For gmpxx though, I don't think it matters a lot.

Thanks. It seems that this link basically says that as far as linking issues are concerned, you might be unable to chase down all the corner cases if one used different versions of g++ (which is unsurprising to me...).

[dimpase]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​1e48b30

correct logic for SAGE_GMP_PREFIX etc

setting of SAGE_GMP_PREFIX does not quite work there, it's always ''.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​50c3482

remove iffy if, do setting SAGE_GMP_PREFIX in cases

[embray]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​50c3482

remove iffy if, do setting SAGE_GMP_PREFIX in cases

This is still wrong because in that case statement, in the system) case you can still wind up installing the Sage SPKG if any of the checks fail (hence setting sage_spkg_install_mpir=yes). That's why there is a separate if/then clause later.

I think the problem here is is with quoting. You can see it in the configure log when running the previous version (and also fixing the SAGE_MP_LIBRARY = mpir line):

./configure: line 8899: test: too many arguments

I believe it's possible for $sage_spkg_install_gmp to not be defined here, so we need the x quoting.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​b364988

correct logic for SAGE_GMP_PREFIX etc

[embray]

This should fix it I think. I also fixed the AC_MSG_RESULT calls to report the correct result when --with-mp=system but the system lib cannot be used (and hence we fall back on installing the MPIR SPKG).

I think what might be continuing to confuse you is that the case for --with-mp=system can still result in the MPIR SPKG being selected instead.

[embray]

Building a fresh build from this branch on Ubuntu 16.04 I also ran into the same problem Dima was telling me about with iml, reported also at #27272. I just wanted to confirm that that was the case, so now I can also review that ticket with some intelligence.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cb6f124

add the AX_ABSOLUTE_HEADER macro

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cf359b1

iml in particular is very picky about being given an absolute path to the

[embray]

Added a fix to the spkg-install for iml for it to use the system GMP as specified by SAGE_GMP_INCLUDE.

I am a bit iffy on the hack for setting SAGE_GMP_LIB and am open to other ideas. But I did confirm that this works when using the system GMP on Ubuntu 16.04. With the above two fixes I was able to build Sage just fine and make ptestlong passes.

[embray]

I think it will make more sense to keep this ticket as a dependency of #27272 (or close #27272) since this is now covering both.

[dimpase]

It's good to go, I think.

[dimpase]

Probably this one needs an updated configure package - unless all the bots nowadays run ./bootstrap (I don't recall the current state of this).

[vbraun]

Merge conflict

[dimpase]

it's gcc and gfortran spkg's that were touched in the latest beta. Erik, could you rebase these, it's certainly your code there...

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​c4e42e6	spkg-configure for GMP
​b070088	Move all the --with-mp logic into the spkg-configure.m4 for MPIR so that it
​3b6eebd	Replace various --with-gmp=$SAGE_LOCAL flags in spkg-installs with a
​b80bf72	Reworked this a bit more
​0ca3f56	fix typo
​06798ca	added a bit more explanation
​b592d77	correct logic for SAGE_GMP_PREFIX etc
​5057680	add the AX_ABSOLUTE_HEADER macro
​101537b	iml in particular is very picky about being given an absolute path to the

[dimpase]

ok, it was trivial to rebase. now testing before setting it to positive again.

[dimpase]

The present mpfrcx does not work with this branch, but it does with the new version. Thus the new dependence.

[dimpase]

$SAGE_LOCAL/bin/sage-env-config must get updated after, say, sage -i mpir (or any other dummy package that sets anything in the build environment). Currently this does not happen. This is a bug that leads to sage -i mpir breaking stuff.

See #27373 to deal with this. (Hopefully everything that is on this ticket can stay as it is, so I don't flip it to "needs work")

[dimpase]

We need to decide how to move forward here. I'm fine with a no-fix for #27373, i.e. "re-run configure with the parameters you need, do not use ./sage -i for un-dummifying packages".

[embray]

I guess I don't really see how #27373 is relevant here. Or at least, I don't see that it's an issue particular to this ticket or that should be a dependency of it.

[jdemeyer]

Merge conflict

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​862ca6a

Merge remote-tracking branch 'trac/develop' into HEAD

[dimpase]

the only manual part of the merge was here:

build/pkgs/arb/spkg-install

@@ -4 +4 @@
 # be removed in arb >= 2.8 when it is released
 export EXTRA_SHARED_FLAGS=$LDFLAGS
 
-./configure --disable-static --prefix="$SAGE_LOCAL" --with-flint="$SAGE_LOCAL" \
-    --with-gmp="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" || \
+./configure --disable-static --prefix="$SAGE_LOCAL" $SAGE_CONFIGURE_GMP \
+            --with-flint="$SAGE_LOCAL" --with-mpfr="$SAGE_LOCAL" || \
     sdh_die "Error configuring arb."
 
 sdh_make verbose

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​7e2b100	Remove broken SAGE_BUILD_TOOLCHAIN support
​03dc987	Merged trac #27215 in

[dimpase]

I wonder whether for gcc and gfortran --with-gmp="$SAGE_LOCAL" is left there on purpose, or it's simply a typo/omission?

This might have "interesting" interactions with what is to come on #27373.

[dimpase]

Isn't the nesting of AC_CHECK_HEADER's and AC_SEARCH_LIBS needed here? Otherwise we may end up with system's GMP library, but without headers available.

[embray]

Replying to dimpase:

I wonder whether for gcc and gfortran --with-gmp="$SAGE_LOCAL" is left there on purpose, or it's simply a typo/omission?

This might have "interesting" interactions with what is to come on #27373.

I was just about to ask you about this. I'm working on rebasing this on top of current develop (mostly straightforward but not trivial). In the current upstream version of this branch, which you were the last to touch, gcc/gfortran was omitted in ​this commit. However, in my local copy of the branch it still includes them.

I was going to ask if that was intentional but now it's sounding like maybe it wasn't. So I'll put it back in for now.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​8c8285f	spkg-configure for GMP
​956e382	Move all the --with-mp logic into the spkg-configure.m4 for MPIR so that it
​687cdf0	Replace various --with-gmp=$SAGE_LOCAL flags in spkg-installs with a
​6de961d	Reworked this a bit more
​f76d6d5	fix typo
​5f90ff7	added a bit more explanation
​3dee97c	correct logic for SAGE_GMP_PREFIX etc
​0308142	add the AX_ABSOLUTE_HEADER macro
​eea0685	iml in particular is very picky about being given an absolute path to the

[embray]

Rebased on 8.8.beta0. Now to test again; this is all toward making progress on #27373...

[dimpase]

Once again, about nesting. Shouldn't e.g.

            AC_CHECK_HEADER(gmp.h, [], [sage_spkg_install_mpir=yes])
            AC_CHECK_HEADER(gmpxx.h, [], [sage_spkg_install_mpir=yes])
            dnl mpq_cmp_z appeared in GMP 6.1.0 and is used by pynac
            AC_SEARCH_LIBS([__gmpq_cmp_z], [gmp], [break],
                [sage_spkg_install_mpir=yes])

be written as

            AC_CHECK_HEADER(gmp.h, [
              AC_CHECK_HEADER(gmpxx.h, [
                dnl mpq_cmp_z appeared in GMP 6.1.0 and is used by pynac
                AC_SEARCH_LIBS([__gmpq_cmp_z], [gmp], [break],
                   [sage_spkg_install_mpir=yes])
              ], [sage_spkg_install_mpir=yes])
            ], [sage_spkg_install_mpir=yes])            

And, by the way, what's that break doing there? I gather it's to interrupt loops, but there is no loop here...

[embray]

I don't think the nesting is necessary. If finding the headers fails then it will set sage_spkg_install_mpir=yes; there's no advantage to then skipping the library check because it will still keep sage_spkg_install_mpir=yes reagardless. Nesting it just makes the code harder to read IMO.

And, by the way, what's that break doing there? I gather it's to interrupt loops, but there is no loop here...

Remember, most of these autoconf macros expand to shell code. So that's where the loop is, and you can see what a break would do there by looking at the expanded macro in the actual configure script.

However, I think it's superfluous in this case. It would break out of the loop if AC_SEARCH_LIBS finds that you can link the test program without explicitly passing -lgmp, but I don't think that would ever be the case (it is the case sometimes with other libraries, such as libm, which may be linked automatically). So I can just remove it if it's clearer (it's also harmless though).

[dimpase]

OK, it's fine then.

[embray]

Thanks--I still think it would be ok to remove #27373 as a dependency. Because although it is a known bug, it's not entirely new after all as I explained in comment:73:ticket:27373. In short, there are other cases where we should require re-running configure after installing new packages, but don't, resulting in an incorrect configuration.

It should still be fixed ASAP but I think for now it's good enough to say "don't do that".

[dimpase]

OK, let's get it merged then!

[vbraun]

Singular doesn't build for me...

[dimpase]

Logs please.

Do you have gmp installed in a weird place?

You might need to start from make distclean, to be on the safe side.

[dimpase]

and, needless to say, this needs a run of ./bootstrap.

what we might have forgotten is to bump up versions of gmp and mpir, to force re-building of appropriate packages, our poor man dependency resolution. I never understood why spkg-install and spkg-configure.m4 are not listed as package dependencies in automatically generated targets for packages.

[embray]

Replying to vbraun:

Singular doesn't build for me...

This that building from scratch, or what? Note that the configure script should be regenerated or lots of things won'g build properly due to the changes to sage-env-config.in.

[vbraun]

If thats the case then a new confball is needed on the branch...

[embray]

New configure tarball attached and linked in the description.

Nice number on that one =)

[dimpase]

This is a sub-optimal workflow, as ticket authors have little control over the moment the ticket comes up for merging, and thus it's not unlikely that the configure tarball would be out of sync, if there is more than one stream of tickets that touch configure.

And we do have quite a few of them on #27330.

[embray]

Indeed; I've complained about this as well. A better solution isn't immediately obvious to me other than to make sure that all of the buildbot workers have up-to-date autotools installed and run ./bootstrap as part of the normal build procedure. I think that should be done but it's not always trivial I guess (e.g. on OSX?).

In the meantime I'm happy to follow the workflow as it is; it could just be confusing for all the tickets attached to #27330. Maybe we can do something like bundle multiple of those tickets to be merged together once several of them have been otherwise positively reviewed individually.

[dimpase]

it is trivial to install autotools on OSX using Homebrew.

[vbraun]

The one and only OSX buildbot worker is for testing that Sage installs on a as-vanilla-as-possible OSX install...

[vbraun]

configure-314.tar.gz is for 8.8.beta2, as always you are supposed to bump the version if you change the file.

[dimpase]

I have no idea where this 314 can be found. If you update from source then upstream/configure-xyz.tar.gz might not correspond to anything at all, may not even exist.

E.g. my different sage builds, all based on 8.8.beta2, have different from 314 xyz.

I am tired of doing manually things that must have been automated 10 years ago. Why is that xyz there at all?! The build is run by make, which is intelligent enough to know if a file has been updated, you know...

[vbraun]

Actually, 8.8.beta2 has already configure-315.tar.gz. The version number is in build/pkgs/configure/package-version.txt, like all tarballs.

To make a new confball:

• merge in the latest beta (so you have the most up-to-date build/pkgs/configure/package-version.txt)
• run ./bootstrap -s -i
• Commit the new version and checksum, and upload the confball

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​a50fe85	spkg-configure for GMP
​8a16493	Move all the --with-mp logic into the spkg-configure.m4 for MPIR so that it
​299e367	Replace various --with-gmp=$SAGE_LOCAL flags in spkg-installs with a
​058d515	Reworked this a bit more
​7beadaa	fix typo
​b930874	added a bit more explanation
​6d2b496	correct logic for SAGE_GMP_PREFIX etc
​43be546	add the AX_ABSOLUTE_HEADER macro
​e343418	iml in particular is very picky about being given an absolute path to the
​e735b4b	configure version bump

[dimpase]

New commits:

​a50fe85	spkg-configure for GMP
​8a16493	Move all the --with-mp logic into the spkg-configure.m4 for MPIR so that it
​299e367	Replace various --with-gmp=$SAGE_LOCAL flags in spkg-installs with a
​058d515	Reworked this a bit more
​7beadaa	fix typo
​b930874	added a bit more explanation
​6d2b496	correct logic for SAGE_GMP_PREFIX etc
​43be546	add the AX_ABSOLUTE_HEADER macro
​e343418	iml in particular is very picky about being given an absolute path to the
​e735b4b	configure version bump

[dimpase]

rebased over 8.8.beta2, new configure tarball

[dimpase]

and now this fails on OSX 10.14 with external gmp, specifically, flint does not build:

          make build/interfaces/NTL-interface.lo; \
          clang++ -std=gnu++11 -fno-common -pedantic -Wall -O2 -funroll-loops -g -mpopcnt   -shared -install_name /Users/administrator/
py/sage/local/lib/libflint-13.5.2.dylib -compatibility_version 13.5 -current_version 13.5.2 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/Users
/administrator/py/sage/local/lib build/interfaces/NTL-interface.lo  build/printf.lo  build/fprintf.lo  build/sprintf.lo  build/scanf.lo
  build/fscanf.lo  build/sscanf.lo  build/clz_tab.lo  build/memory_manager.lo  build/version.lo  build/profiler.lo  build/thread_support.lo  build/ulong_extras.lo  build/long_extras.lo  build/perm.lo  build/fmpz.lo  build/fmpz_vec.lo  build/fmpz_poly.lo  build/fmpq_poly.lo  build/fmpz_mat.lo  build/fmpz_lll.lo  build/mpfr_vec.lo  build/mpfr_mat.lo  build/mpf_vec.lo  build/mpf_mat.lo  build/nmod_vec.lo  build/nmod_poly.lo  build/nmod_poly_factor.lo  build/arith.lo  build/mpn_extras.lo  build/nmod_mat.lo  build/fmpq.lo  build/fmpq_vec.lo  build/fmpq_mat.lo  build/padic.lo  build/fmpz_poly_q.lo  build/fmpz_poly_mat.lo  build/nmod_poly_mat.lo  build/fmpz_mod_poly.lo  build/fmpz_mod_poly_factor.lo  build/fmpz_factor.lo  build/fmpz_poly_factor.lo  build/fft.lo  build/qsieve.lo  build/double_extras.lo  build/d_vec.lo  build/d_mat.lo  build/padic_poly.lo  build/padic_mat.lo  build/qadic.lo  build/fq.lo  build/fq_vec.lo  build/fq_mat.lo  build/fq_poly.lo  build/fq_poly_factor.lo  build/fq_nmod.lo  build/fq_nmod_vec.lo  build/fq_nmod_mat.lo  build/fq_nmod_poly.lo  build/fq_nmod_poly_factor.lo  build/fq_zech.lo  build/fq_zech_vec.lo  build/fq_zech_mat.lo  build/fq_zech_poly.lo  build/fq_zech_poly_factor.lo  -o libflint-13.5.2.dylib -L/Users/administrator/py/sage/local/lib -Wl,-rpath,/Users/administrator/py/sage/local/lib  -L/Users/administrator/py/sage/local/var/tmp/sage/build/flint-2.5.2.p4/src -L/usr/local/lib -L/Users/administrator/py/sage/local/lib -L/Users/administrator/py/sage/local/lib -lmpfr -lgmp -lm -lntl -lpthread ; \
        fi
clang++ -std=gnu++11 -fPIC -fno-common -pedantic -Wall -O2 -funroll-loops -g -mpopcnt  -I/Users/administrator/py/sage/local/var/tmp/sage/build/flint-2.5.2.p4/src -I/usr/local/include -I/Users/administrator/py/sage/local/include -I/Users/administrator/py/sage/local/include -c interfaces/NTL-interface.cpp -o build/interfaces/NTL-interface.lo
ld: illegal thread local variable reference to regular symbol __ZN3NTL8ZZ_pInfoE for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [libflint-13.5.2.dylib] Error 1

note that mpfr and ntl, its dependencies build just fine with SAGE_CHECK=yes, so that's something in flint C++ interface that is broken...

[dimpase]

I suppose this is caused by the very recent (Apr 6) NTL update.

[dimpase]

No, it must be either lack of testing on OSX 10.14 (I tested on 10.13 before), or new clang in Xcode:

$ clang++ -v
Apple LLVM version 10.0.1 (clang-1001.0.46.3)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[dimpase]

Replying to vbraun:

The one and only OSX buildbot worker is for testing that Sage installs on a as-vanilla-as-possible OSX install...

Well, if you like I can install autotools on your OSX bot just from sources. (I recall doing that on OSX 10.12, it should still be possible on 10.14). This would be vanilla enough then, still...

[dimpase]

Replying to dimpase:

No, it must be either lack of testing on OSX 10.14 (I tested on 10.13 before), or new clang in Xcode:

$ clang++ -v
Apple LLVM version 10.0.1 (clang-1001.0.46.3)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

The problem seems to be caused by -std=gnu++11 option, see ​https://trac.sagemath.org/ticket/25532#comment:88

[dimpase]

facepalm.... That was just a library clash - I did have libntl installed in /usr/local, and this seems to be deadly on OSX...

[vbraun]

Replying to dimpase:

Well, if you like I can install autotools on your OSX bot just from sources. (I recall doing that on OSX 10.12, it should still be possible on 10.14).

Again, unless we make autotools a prerequisite we need a way to verify that Sage builds on OSX without autotools installed...

[dimpase]

Replying to vbraun:

Replying to dimpase:

Well, if you like I can install autotools on your OSX bot just from sources. (I recall doing that on OSX 10.12, it should still be possible on 10.14).

Again, unless we make autotools a prerequisite we need a way to verify that Sage builds on OSX without autotools installed...

Do we have Linux buildbots without autotools installed?

Anyhow, one can have autotools installed in a non-standard location only buildbot knows, so that it can run ./bootstrap, and then it's as if there are not present.

It's hard for me to understand the problem without knowing how exactly a release is made. (if be any chance you are in Berlin, you can show me - I'm here until 21st Apr).

[vbraun]

No matter how you slice it, either the buildbot has access to autotools OR it tests that you can build without autotools. The only safe way to avoid generating a confball by hand is to have some automatic system, e.g. whenever I submit a new buildbot job. That requires some work to avoid version clashes when I back out commits. Truth be told, changes to the configure in Sage are pretty rare so this issue does not come up very often.

I'm in Berlin, hit me up when you have some time. I'm located in the East (Elsenstrasse).

[embray]

Replying to dimpase:

facepalm.... That was just a library clash - I did have libntl installed in /usr/local, and this seems to be deadly on OSX...

I have had similar problems--not necessarily with NTL--but with other packages where having a copy of the development package already installed on my system created difficult to avoid clashes when trying to build some Sage SPKGs. See e.g. #22768. In that case it's more due to bugginess with Python's module build system, but I can envision similar problems in other build systems that make too many assumptions about your paths in some context.

[dimpase]

Replying to vbraun:

Truth be told, changes to the configure in Sage are pretty rare

I have a dozen tickets open that do change configure, see #27330. For the purpose of testing the configure spkg versioning, today I attached configure tarballs to 3 of them: #27258, #27265, #27186. No matter in which order they would be merged, I'd need to re-do these tarballs for all but one, you know. This is not exactly entertaining...

[jdemeyer]

Breakage: #27751