Opened 8 years ago

Closed 7 years ago

Last modified 7 years ago

#11635 closed defect (fixed)

Ensure that libraries link to the shared version of NTL by default

Reported by: kcrisman Owned by: tbd
Priority: minor Milestone: sage-5.6
Component: porting: Cygwin Keywords: cygwin ntl libtool spkg
Cc: mhansen, dimpase, jpflori, jdemeyer Merged in: sage-5.6.beta3
Authors: Jean-Pierre Flori Reviewers: Karl-Dieter Crisman
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

In #11547, we added copying of several needed dlls on Cygwin. However, in the meantime NTL had a fairly major update (#5731).

From version 5.5, NTL provides a new build system using libtool (although libtool itself is not provided!) to generate shared libraries. On Cygwin, libtool properly install the shared library as cygntl-?.dll in the bin directory and the import file as libntl.dll.a in the lib directory, together with a libtool file libntl.la and a static version of the library as libntl.a.

In particular, this ensures that the shared version is linked with by default because ld/gcc/g++ find the .dll.a file in the lib directory (which points to the dll file in the bin directory) before the .a file.

Moreover, only moving the previously built libntl.dll to libntl.dll.a was not a solution. Indeed, the built file hardcoded its own build-time name into itself (as a DT_SONAME on linux), so that libraries linked to it (before or after renaming, does not matter) would in fact search for a file with the build time name... and fail.

spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/ntl-5.5.2.p0.spkg

This spkg vastly simplifies the old spkg-install, uses the (slightly patched) upstream build system and includes its own version of libtool (or rather the tools needed to generate it on the fly).

Attachments (1)

ntl-5.5.2.p0.diff (67.6 KB) - added by jdemeyer 7 years ago.
Spkg diff, for review only.

Download all attachments as: .zip

Change History (74)

comment:1 Changed 8 years ago by kcrisman

  • Authors set to Karl-Dieter Crisman
  • Description modified (diff)
  • Status changed from new to needs_review

New spkg at http://sage.math.washington.edu/home/kcrisman/ntl-5.5.2.p0.spkg. No guarantees on whether this works, or even is necessary (!) but I strongly suspect both. You'll know it works if you do everything at #6743, including this, and then Sage has a segfault on startup.

In case there is any doubt that this is Cygwin-only:

    elif [ $UNAME = "CYGWIN" ]; then
        # Cygwin
        if [ "$CYGPKG" = "yes" ]; then
            if [ -f /lib/libgmp.dll.a ]; then
                ln -s /lib/libgmp.dll.a libgmp.a
            else
                # Added by William Stein on 2006-02-06 so that 
                # can build even if gmp is not built locally
                # i.e., even if using a system-wide gmp.
                # (though it has to be in /usr/lib, which is
                # where it is for cygwin).
                ln  -s /usr/lib/*gmp* .
                ln  -s libgmp.dll.a libgmp.a
            fi 
        else
   	    ln -s "$SAGE_LOCAL/lib/libgmp.a" .
        fi
	$MAKE libntl.dll
        if [ ! -f libntl.dll ]; then
            echo "Error creating ntl shared library."
            exit 1
        fi
	cp libntl.dll "$SAGE_LOCAL/lib/libntl.dll.a"
	cp libntl.dll "$SAGE_LOCAL/lib/libntl.dll"
	cp libntl.dll "$SAGE_LOCAL/bin/libntl.dll.a"
	cp libntl.dll "$SAGE_LOCAL/bin/libntl.dll"
        if [ ! -f "$SAGE_LOCAL/bin/libntl.dll.a" ]; then
            exit 1   # CRUCIAL that we have the dynamic link library
        fi

comment:2 Changed 8 years ago by jdemeyer

  • Cc jdemeyer removed

comment:3 Changed 8 years ago by kcrisman

  • Status changed from needs_review to needs_info

Interestingly, the most recent alpha I tested all the Cygwin stuff on a few weeks ago already had NTL 5.5.2!

$ ls spkg/installed
<snip>
ntl-5.5.2
<snip>

and it seemed to work fine (in the sense that it got to the Pari segfault when initializing).

That doesn't mean that later in the process things wouldn't happen, but at least for now this ticket can be lower priority, if it's needed at all, since we have to figure out how to get past that error first.

comment:4 Changed 8 years ago by kcrisman

  • Priority changed from major to minor

comment:5 Changed 8 years ago by kcrisman

This appears to be the same issue as #12104, which should probably be closed as a duplicate then.

comment:6 Changed 8 years ago by kcrisman

Since the 4.8.alpha3 spkg still is 5.5.2 with no patches, this spkg should still be valid to fix this.

comment:7 follow-up: Changed 8 years ago by kcrisman

  • Status changed from needs_info to needs_review

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

comment:8 in reply to: ↑ 7 ; follow-up: Changed 8 years ago by dimpase

  • Status changed from needs_review to needs_info

Replying to kcrisman:

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

Why does one need copies in local/bin ? And why does one need to copy rather than to create a symbolic link?

comment:9 in reply to: ↑ 8 ; follow-ups: Changed 8 years ago by kcrisman

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

Why does one need copies in local/bin ? And why does one need to copy rather than to create a symbolic link?

No idea! I just copied models from other spkgs. Feel free to only copy and/or make links where actually appropriate. I have never claimed to be anything close to an expert; I've only focused on getting it to work by following other spkgs I've seen.

I would love to have this "correct" so that we never have to worry about it again.

comment:10 in reply to: ↑ 9 Changed 8 years ago by kcrisman

Replying to kcrisman:

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

Why does one need copies in local/bin ? And why does one need to copy rather than to create a symbolic link?

No idea! I just copied models from other spkgs. Feel free to only copy and/or make links where actually appropriate. I have never claimed to be anything close to an expert; I've only focused on getting it to work by following other spkgs I've seen.

I would love to have this "correct" so that we never have to worry about it again.

This might even be part of the problem at #9167.

comment:11 in reply to: ↑ 9 ; follow-ups: Changed 8 years ago by kcrisman

Replying to kcrisman:

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

Why does one need copies in local/bin ? And why does one need to copy rather than to create a symbolic link?

Just for posterity (addressing the comments at #12104, perhaps?), here is the rationale for naming it dll.a, from #9050.

This is so that Cygwin will find the shared library before it finds the static library; otherwise, there are lots of segfaults in Sage under Cygwin

comment:12 in reply to: ↑ 11 Changed 8 years ago by dimpase

Replying to kcrisman:

Replying to kcrisman:

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

Why does one need copies in local/bin ? And why does one need to copy rather than to create a symbolic link?

Just for posterity (addressing the comments at #12104, perhaps?), here is the rationale for naming it dll.a, from #9050.

This is so that Cygwin will find the shared library before it finds the static library; otherwise, there are lots of segfaults in Sage under Cygwin

that means that only in the cases when both static and dynamic libraries are built, this is needed. IMHO there are not so many such cases among Sage spkg's.

comment:13 in reply to: ↑ 11 ; follow-up: Changed 8 years ago by dimpase

Replying to kcrisman:

Replying to kcrisman:

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

Why does one need copies in local/bin ? And why does one need to copy rather than to create a symbolic link?

Just for posterity (addressing the comments at #12104, perhaps?), here is the rationale for naming it dll.a, from #9050.

This is so that Cygwin will find the shared library before it finds the static library; otherwise, there are lots of segfaults in Sage under Cygwin

IMHO, this is, at best, a bad hack. In principle, *.dll.a files are import libraries for *.dll libaries, used by the linker to like against *.dll (i.e. dynamic libraries).

See Buiding/using DLLs

comment:14 in reply to: ↑ 13 Changed 8 years ago by kcrisman

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

Why does one need copies in local/bin ? And why does one need to copy rather than to create a symbolic link?

Just for posterity (addressing the comments at #12104, perhaps?), here is the rationale for naming it dll.a, from #9050.

This is so that Cygwin will find the shared library before it finds the static library; otherwise, there are lots of segfaults in Sage under Cygwin

IMHO, this is, at best, a bad hack. In principle, *.dll.a files are import libraries for *.dll libaries, used by the linker to like against *.dll (i.e. dynamic libraries).

Well, as I said in comment:9, I would more than happy to have someone who actually knows what they are talking about to fix this and post a different spkg :) especially if it ended up fixing #9167. Go for it!

comment:15 in reply to: ↑ 11 ; follow-up: Changed 8 years ago by dimpase

Replying to kcrisman:

Replying to kcrisman:

Oh, and no longer 'needs info', because #12104 shows that it's needed to start Sage.

Why does one need copies in local/bin ? And why does one need to copy rather than to create a symbolic link?

Just for posterity (addressing the comments at #12104, perhaps?), here is the rationale for naming it dll.a, from #9050.

This is so that Cygwin will find the shared library before it finds the static library; otherwise, there are lots of segfaults in Sage under Cygwin

It's probably OK to have "real" (i.e. dynamic libs, not misnamed dll.a or other files) .dlls in local/bin/, and bad to have them in local/lib/. At least that's what a Cygwin-aware libtool will be doing.

comment:16 in reply to: ↑ 15 Changed 8 years ago by kcrisman

It's probably OK to have "real" (i.e. dynamic libs, not misnamed dll.a or other files) .dlls in local/bin/, and bad to have them in local/lib/. At least that's what a Cygwin-aware libtool will be doing.

Can you be more specific (if only so I can make a good spkg)? Please check which of the following should be copied, which should be linked (with ln -s, or with other options?), and which are just horrible.

  • lib/libntl.dll
  • lib/libntl.dll.a
  • bin/libntl.dll
  • bin/libntl.dll.a

Recall that currently we also have

  • lib/libntl.a

comment:17 follow-up: Changed 8 years ago by kcrisman

Here's something else potentially relevant.

User 1@GC02635 /home/SageUser/sage-4.7.2
$ ./sage -t -verbose devel/sage/sage/calculus/functional.py
init.sage does not exist ... creating
sage -t -verbose "devel/sage/sage/calculus/functional.py"
Setting permissions of DOT_SAGE directory so only you can read and write it.
<snip LOTS of pretty much identical things>
1564601 [main] python 3544 fork: child 3776 - died waiting for dll loading, errno 11
1639139 [main] python 160 C:\cygwin\home\SageUser\sage-4.7.2\local\bin\python.exe: *** fatal error - unable to remap C:\cygwin\home\SageUser\sage-4.7.2\local\lib\libntl.dll to same address as parent: 0x19A40000 != 0x626C0000
Stack trace:
Frame     Function  Args
002093B8  6102796B  (002093B8, 00000000, 00000000, 00000000)
002096A8  6102796B  (6117EC60, 00008000, 00000000, 61180977)
0020A6D8  61004F1B  (611A7FAC, 6124A1B4, 19A40000, 626C0000)
End of stack trace
1642317 [main] python 3544 fork: child 160 - died waiting for dll loading, errno 11
Traceback (most recent call last):
  File "../.sage/tmp/functional_3772.py", line 6, in <module>
    from sage.all_cmdline import *;
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/all_cmdline.py", line 14, in <module>
    from sage.all import *
<snip>
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py", line 235, in <module>
    from sage.interfaces.all import (singular as singular_default,
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/interfaces/all.py", line 8, in <module>
    from gap import gap, gap_reset_workspace, gap_console, gap_version, is_GapElement, Gap
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/interfaces/gap.py", line 1188, in <module>
    gap_reset_workspace(verbose=False)
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/interfaces/gap.py", line 1179, in gap_reset_workspace
    g.save_workspace()
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/interfaces/gap.py", line 983, in save_workspace
    self.eval('SaveWorkspace("%s");'%WORKSPACE, allow_use_file=False)
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/interfaces/gap.py", line 374, in eval
    result = Expect.eval(self, input_line, **kwds)
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/interfaces/expect.py", line 1039, in eval
    for L in code.split('\n') if L != ''])
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/interfaces/gap.py", line 476, in _eval_line
    self._start()
  File "/home/SageUser/sage-4.7.2/local/lib/python/site-packages/sage/interfaces/gap.py", line 913, in _start
    raise RuntimeError, msg
RuntimeError: Unable to start gap because the command 'gap -r -b -p -T -o 3900m /home/SageUser/sage-4.7.2/data//extcode/gap/sage.g' failed.

         [11.4 s]

----------------------------------------------------------------------
The following tests failed:


        sage -t -verbose "devel/sage/sage/calculus/functional.py"
Total time for all tests: 11.7 seconds

comment:18 in reply to: ↑ 17 ; follow-up: Changed 8 years ago by dimpase

Replying to kcrisman:

Here's something else potentially relevant.

the usual Cygwin fork() crap, no?

comment:19 in reply to: ↑ 18 ; follow-up: Changed 8 years ago by kcrisman

Here's something else potentially relevant.

the usual Cygwin fork() crap, no?

I didn't think so, but apparently it was. Sorry for that last bit o' noise.

Any comments on which of the files above are "correct"?

comment:20 in reply to: ↑ 19 Changed 8 years ago by dimpase

Replying to kcrisman:

Here's something else potentially relevant.

the usual Cygwin fork() crap, no?

I didn't think so, but apparently it was. Sorry for that last bit o' noise.

Any comments on which of the files above are "correct"?

It would be good to find this out by looking at the working Python extensions in Sage --- what they do, how the files are built and named, etc. I think that also the information in Autobook is very relevant.

But I am afraid that we will also need to solve the fork() issue, which seems to mean eliminating the use of fork() in Python and Cython parts, at least. This is doable, but quite a lot of work... (unless an überhacker fixes fork() directly in Cygwin ––– probably needs a lot of very delicate reverse engineering of Windows kernel; the fact that is has not yet been done speaks volumes :-( )...

comment:21 Changed 7 years ago by kcrisman

  • Cc jpflori added
  • Status changed from needs_info to needs_review

This apparently is still needed, and someone who knows what they are talking about rediscovered it :)

comment:22 Changed 7 years ago by jpflori

I think the better solution would be to use the libtool build system the latest version of NTL propose.

Anyway, I found it quite strange that csage.dll wants a file called libntl.dll although only a file called libntl.dll.a was available at link time. Maybe there is a kind of DT_SONAME hidden in the NTL shared library.

comment:23 Changed 7 years ago by jpflori

There is indeed a linbntl.dll string within the shared library, so that may be it. Try strings libntl.dll | grep libntl.dll

Anyway, using libtool should solves our problem. At least I have previously built NTL properly with that on Cygwin and Linux. Let's hope it does not break on other systems.

comment:24 Changed 7 years ago by jpflori

Proof of concept spkg available at http://perso.telecom-paristech.fr/~flori/sage/ntl-5.5.2.p0.spkg

The regenerated patches need to be split. I've removed quite all biuld hacks to only use the NTL build system. This seems ok on Linux, I'll check Cygwin now.

comment:25 Changed 7 years ago by jpflori

The updated spkg depends on libtool (ok, this was somehow to be expected), which is not needed by any other part of Sage. Too bad.

comment:26 Changed 7 years ago by jpflori

  • Summary changed from Copy needed dll file for NTL on Cygwin to Ensure that libraries link to the shared version of NTL by default

comment:27 Changed 7 years ago by jpflori

And the spkg I posted needs to pass LDFLAGS=-no-undefined to NTL to be able to build a shared library on cygwin.

comment:28 follow-up: Changed 7 years ago by jpflori

The spkg at http://perso.telecom-paristech.fr/~flori/sage/ntl-5.5.2.p0.spkg is cleaner, functional, but still depends on libtool being system-wide installed.

comment:29 in reply to: ↑ 28 ; follow-up: Changed 7 years ago by dimpase

Replying to jpflori:

The spkg at http://perso.telecom-paristech.fr/~flori/sage/ntl-5.5.2.p0.spkg is cleaner, functional, but still depends on libtool being system-wide installed.

I am trying this spkg now. My build (with an older NTL) has finished, but I see exactly the same problem as reported on #12104.

comment:30 Changed 7 years ago by jpflori

It could be that some of your already build dlls are still linked with libntl.dll (which should still be inexistant if you did not copy it by hand earlier). You can check that by cygchecking some of them, sage/rings/... should be good culprits or sage/libs/ntl.... So you should rebuild all these broken dlls. Unfortunately I'm not sure yet how to easily automagically make them rebuild (as sage -b will not). I'm currently seeing if touching sage/libs/ntl/decl.pxi is enough to trigger that rebuild. Some spkg depending on NTL may be broken as well...

The easiest solution would be to rebuild from scratch...

comment:31 in reply to: ↑ 29 Changed 7 years ago by dimpase

Replying to dimpase:

Replying to jpflori:

The spkg at http://perso.telecom-paristech.fr/~flori/sage/ntl-5.5.2.p0.spkg is cleaner, functional, but still depends on libtool being system-wide installed.

I am trying this spkg now. My build (with an older NTL) has finished, but I see exactly the same problem as reported on #12104.

The spkg does not build. It ends up with rather weird message:

g++: /usr/lib/gcc/i686-pc-cygwin/4.3.4/crtbegin.o: No such file or directory
g++: /usr/lib/gcc/i686-pc-cygwin/4.3.4/crtend.o: No such file or directory
makefile:376: recipe for target `ntl.a' failed
make[1]: *** [ntl.a] Error 1
make[1]: Leaving directory `/usr/local/src/sage/sage-5.2/spkg/build/ntl-5.5.2.p0/src/src'
makefile:334: recipe for target `all' failed
make: *** [all] Error 2
Unable to tune and build NTL.

And indeed, my gcc is 4.5.3, and I have stuff in /usr/lib/gcc/i686-pc-cygwin/4.5.3/, but no /usr/lib/gcc/i686-pc-cygwin/4.3.4/

Please see the log here

comment:32 Changed 7 years ago by jpflori

This is because the postinstall script from the Cygwin gcc4-core package is broken. I've posted additional details on the CygwinPort page. You'll have to tweak the /usr/bin/libtool file yourself for now, replacing 4.3.4 by 4.5.3 (or reinstalling version 4.3.4 of gcc4 package).

comment:33 follow-up: Changed 7 years ago by jpflori

By the way, touching decl.pxi in sage/libs/ntl and then running "sage -b" seems to have fixed my install. At least Sage starts now without import errors.

comment:34 Changed 7 years ago by kcrisman

wrt to the libtool business - this doesn't become a Sage dependency though, right? Or could one just make it a Cygwin one (that's ok)? Sorry for being dense here.

comment:35 in reply to: ↑ 33 ; follow-ups: Changed 7 years ago by dimpase

Replying to jpflori:

By the way, touching decl.pxi in sage/libs/ntl and then running "sage -b" seems to have fixed my install. At least Sage starts now without import errors.

not for me - I still needed to copy SAGELOCAL/bin/libntl.dll.a to SAGELOCAL/bin/libntl.dll to get past import errors.

Then, the usual fork() problems on startup, and rebaseall does not help. I am afraid we're back to the root of the problem. I'll try to rebuild from scratch one more time, but this doesn't look good.

comment:36 in reply to: ↑ 35 ; follow-up: Changed 7 years ago by jpflori

Replying to dimpase:

Replying to jpflori:

By the way, touching decl.pxi in sage/libs/ntl and then running "sage -b" seems to have fixed my install. At least Sage starts now without import errors.

not for me - I still needed to copy SAGELOCAL/bin/libntl.dll.a to SAGELOCAL/bin/libntl.dll to get past import errors.

Then, the usual fork() problems on startup, and rebaseall does not help. I am afraid we're back to the root of the problem. I'll try to rebuild from scratch one more time, but this doesn't look good.

I also rebuild some spkg, at least flint, don't remember exactly which ones, could you check every dll? Writing such a script would be less time consuming than rebuilding Sage from scratch (and I did not have to rebase by the way). It would be really strange that some files still link to libntl.dll with a fresh instlal, as there is definitely no reference to such a file name with the new spkg, nor in other pieces of Sage.

Once again, I think that the problem before was just that there was some kind of DT_SONAME (which usually point to a "real" filename to use for a library at runtime) in the library buid by NTL. As the library was originally build as libntl.dll, this DT_SONAME was set to libntl.dll, so even though the library was rename to libntl.dll.a after installing and so was found before the static archive when we link to NTL, at runtime the pieces of Sage linked to it would look to libntl.dll.

With the new spkg libntl.dll never sees the light of day, so there is no reason something else thinks it has ever existed, or will.

And yes unfortunately, libtool is used on every platform, not only on Cygwin, the good point is that we use the upstream NTL build system rather than the one that was made only for Sage by Sage devs before it was more or less integrated with the help of the same people to NTL itself (although with a dependency on libtool). Nevertheless, libtool should only be 4 text files, so I'm looking for a way to integrate it to the spkg right now. Another solution is to continue with the Sage patched build system and add a proper linking line for Cygwin.

Last edited 7 years ago by jpflori (previous) (diff)

comment:37 Changed 7 years ago by jpflori

Could you run the following in your cygwin shell in a "sage -sh":

for d in `find . -name *.dll`; do echo $d; cygcheck $d|grep "not find"; done

and try to rebuild the libs whining about libntl.dll.

By the way, on my system, some dlls from R complain about libR.dll...

comment:38 Changed 7 years ago by jpflori

To actually 'find' the dlls of the Sage library, I had to 'cd' to local/lib/python/site-packages/sage/

comment:39 Changed 7 years ago by jpflori

And that showed me that I need to rebuild eclib which still looks fro libntl.dll :)

And that libs/lcalc/lcalc_Lfunction.dll looks for a libLfunction.so file.

comment:40 Changed 7 years ago by jpflori

Spkg updated to include libtool.

Ok on Linux, testing now on Cygwin.

comment:41 Changed 7 years ago by jpflori

In fact the included libtool does not work well on Cygwin. For some reason it decides to use gcc rather than g++ for the final linking and spits out a HUGE amount of undefined references to std::* stuffs.

comment:42 follow-up: Changed 7 years ago by jpflori

Ok that's just a matter of adding -lstdc++ to the LDLIBS.

comment:43 in reply to: ↑ 42 Changed 7 years ago by dimpase

Replying to jpflori:

Ok that's just a matter of adding -lstdc++ to the LDLIBS.

I'm doing the full rebuild. I even had to remove /etc/rebase* database and only then rebaseall, otherwise fork() errors kept popping up.

Please provide an updated ntl spkg.

Last edited 7 years ago by dimpase (previous) (diff)

comment:44 in reply to: ↑ 35 Changed 7 years ago by dimpase

Replying to dimpase:

Replying to jpflori:

By the way, touching decl.pxi in sage/libs/ntl and then running "sage -b" seems to have fixed my install. At least Sage starts now without import errors.

not for me - I still needed to copy SAGELOCAL/bin/libntl.dll.a to SAGELOCAL/bin/libntl.dll to get past import errors.

Then, the usual fork() problems on startup, and rebaseall does not help. I am afraid we're back to the root of the problem. I'll try to rebuild from scratch one more time, but this doesn't look good.

This might be due to the fact that I tried to follow your instruction to rebase everything in SAGEROOT rather than in SAGELOCAL, i.e. in SAGEROOT/local/, as I did before. In particular, there were dlls in devel/sage/build, which are not used, and this perhaps unnecessarily clogged up the address space.

comment:45 in reply to: ↑ 36 Changed 7 years ago by leif

Replying to jpflori:

Replying to dimpase:

Replying to jpflori:

By the way, touching decl.pxi in sage/libs/ntl and then running "sage -b" seems to have fixed my install.

I also rebuild some spkg, at least flint, don't remember exactly which ones, could you check every dll? Writing such a script would be less time consuming than rebuilding Sage from scratch (and I did not have to rebase by the way).

To rebuild all spkgs that depend on NTL you can do

$ touch spkg/installed/ntl-*  # Of course not needed if you just (re)built it
$ env SAGE_UPGRADING=yes make build

You may still have to run ./sage -ba-force afterwards; not sure whether module_list.py really covers all dependencies (on NTL in this case, but probably indirect dependencies as well).

Should still be faster than rebuilding all from scratch... ;-)

comment:46 Changed 7 years ago by jpflori

The one I just uploaded should be fine. It builds ok on Linux, and I am testing it right now on Cygwin.

comment:47 follow-up: Changed 7 years ago by jpflori

  • Authors changed from Karl-Dieter Crisman to Karl-Dieter Crisman, Jean-Pierre Flori
  • Description modified (diff)
  • Keywords cygwin ntl libtool spkg added

Ok, the latest spkg built correctly.

Anyone wants to try on Mac OS? or Solaris?

comment:48 in reply to: ↑ 47 Changed 7 years ago by dimpase

Replying to jpflori:

Ok, the latest spkg built correctly.

Anyone wants to try on Mac OS? or Solaris?

it works on MacOSX 10.6.8.

comment:49 in reply to: ↑ description Changed 7 years ago by jdemeyer

Replying to kcrisman:

and includes its own version of libtool (or rather the tools needed to generate it on the fly).

What does this mean?

comment:50 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to needs_work
  • Work issues set to HTTP 403

Anyway, the spkg file gives me a 403 Forbidden error when trying to download it.

comment:51 follow-up: Changed 7 years ago by jpflori

  • Status changed from needs_work to needs_review
  • Work issues HTTP 403 deleted

perso.telecom-paristech.fr seems down (at least for my homepage). The spkg is accessible at http://www.infres.enst.fr/~flori/sage/ntl-5.5.2.p0.spkg

And by does not ship libtool, I really mean does not ship the machinery necessary to generate libtoolize, that is ltmain.sh et al. which is kind of non sense but is the case.

comment:52 in reply to: ↑ 51 Changed 7 years ago by jdemeyer

Replying to jpflori:

And by does not ship libtool, I really mean does not ship the machinery necessary to generate libtoolize, that is ltmain.sh et al. which is kind of non sense but is the case.

I still don't quite understand what you mean, but I'll have a look some time...

comment:53 Changed 7 years ago by jpflori

I'll give it another try: Since version 5.5, NTL officially supports building a shared version of the library, but this require a system-wide installation of libtool(... be also aware that no other parts of autotools are used or required, there are only calls to "libtool -mode=link ..." in the Makefile, which is generated by a custom configure script): see http://www.shoup.net/ntl/doc/tour-unix.html#shared

The Sage spkg was crafted before that new build system, and build shared libraries by patching the NTL old build system and defining custom targets.

The problem is that on Cygwin this patched build system installs a shared libntl.dll and static libntl.a and the .a files gets picked up by default by the linker when -lntl is passed, whereas we would want the .dll file to be picked. A previous proposed workaround was to copy libntl.dll.a to libntl.dll.a (.dll.a corresponds to import files and are picked up before .a files by the linker) but that's more of a hack than a real fix.

So we have to solution:

  • continue with our patched build system and modify the cygwin target to build a proper import file and install it with the shared .dll and static .a files
  • use NTL libtool based build system, but that requires to be able to include the needed piece to generate libtool (not libtoolize), that is ltmain.sh, install-sh, config.guess and config.sub as mentioned at http://www.gnu.org/software/libtool/manual/html_node/Distributing.html#Distributing (or to make a system-wide libtool a Sage dependency, which looks to be a bad idea).

IIRC to use libtoolize to generate these 4 files would need some skeleton autotools project (i.e. a nearly empty configure.ac or Makefile.am and IIRC that's how I generated the four libtool files, just putting the strict minimal set of autotools macros in one of these two files).

comment:54 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:55 Changed 7 years ago by kcrisman

I don't know if I previously reported this, but this package works fine on Cygwin on XP, at least to the point that Sage starts and many doctests pass.

comment:56 Changed 7 years ago by kcrisman

JP, what would you say needs still to be reviewed for this to be ready?

comment:57 Changed 7 years ago by jpflori

I guess someone else than me has to decide if my solution is correct, that is:

  • use a as less patched upstream build system as possible, and include some libtool related files which this system needs,
  • rather than use a completely custom one which only use some pieces of the upstream one.

comment:58 Changed 7 years ago by kcrisman

I can't review that piece, sadly. I guess it would also be helpful to make sure this actually solves the Cygwin problem. I still have to get Sage to start, forking problems, sigh...

comment:59 Changed 7 years ago by kcrisman

Actually, now it does, magically. Somehow installing cvxopt and sage-location resetting paths did the job.

As to the rest: I like the removed files (Debian and non-patches) and most of the patch file changes, though I can't speak for mfile.patch. The changes in spkg-check seem right - why were those flags set anyway? Things should have already been compiled by that point. I would need some help in making sure that all the old stuff in spkg-install with respect to different soft links and so forth was no longer needed or handled by the ntl_install() function, but I assume it is.

comment:60 Changed 7 years ago by jpflori

There is no need for soft links and/or copying files and/or renaming them.

Using libtool, the upstream build system install the following files on Cygwin:

  • lib/libntl.la (libtool magical file)
  • lib/libntl.dll.a (import file for the dynamic library)
  • lib/libntl.a (static archive)
  • bin/cygntl.dll (dynamic library)

(some -<version> are surely missing here)

When using -lntl, ld will find first libntl.dll.a and in the end link to the dynamic library cygntl.dll. The only difference with linking directly to cygntl.dll (let's say by specifying its path rather than just -lntl) is that the linking will be a slower, but that's not really an issue, everything is already so slow on Cygwin, I don't think the difference is noticeable.

On other systems the difference in file installed are minimal and will surely go unnoticed and the shared library will still gets picked up first.

comment:61 Changed 7 years ago by kcrisman

This seems to be fine on Mac - relevant tests pass, if anything more quickly.

$ grep -r cygwin .
<a whole bunch of libtool files and a more or less irrelevant reference in src/>

I mean, if the Sage library passes tests with this, I'm inclined to say let's get this in. I still don't understand why upstream needs libtool to do its installing, but if that's a fairly standard solution in this kind of package then that should be okay too.

comment:62 follow-up: Changed 7 years ago by kcrisman

  • Reviewers set to Karl-Dieter Crisman

Okay, I can't give a 100% positive review because I know nothing about libtool, but using the upstream solution makes a heck of a lot more sense if we don't have to add any other dependencies and just a few files, and the pieces I do understand of it make sense.

I built 5.5.rc0 with this on sage.math from scratch, and used SAGE_UPGRADING=yes on my Mac with this spkg in 5.4.rc2, and no doctest failures except on sage/tests/startup.py on sage.math, where it consistently seems to take 2.1 seconds, not less than 2 - but this was also true of a 5.4.rc1 build which didn't have this spkg, so it seems like there is just a heavy load there or something. Coupled with its success on Cygwin, I am strongly inclined to give this positive review, modulo understanding libtool stuff.

Jeroen, do you get what JP is trying to do and can you give this positive review based on that part of the changes in the spkg?

comment:63 in reply to: ↑ 62 ; follow-up: Changed 7 years ago by kcrisman

  • Cc jdemeyer added

Jeroen, do you get what JP is trying to do and can you give this positive review based on that part of the changes in the spkg?

Ping. I almost think it would be better to just put this in and let the test farm find any problems; using upstream's tools seems logical.

comment:64 in reply to: ↑ 63 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to positive_review

Replying to kcrisman:

Jeroen, do you get what JP is trying to do and can you give this positive review based on that part of the changes in the spkg?

Ping. I almost think it would be better to just put this in and let the test farm find any problems

You mean the Sage buildbot, right? In that case, just put the ticket to positive review if you are convinced that testing is the only thing remaining to be done.

comment:65 Changed 7 years ago by jdemeyer

  • Status changed from positive_review to needs_work

Minor comment: you should quote $CUR in spkg-install:

cd "$CUR/src"

comment:66 Changed 7 years ago by jdemeyer

And libtool should not be part of the Mercurial history, since it's just upstream files (similar to src).

comment:67 Changed 7 years ago by jdemeyer

There is also a strange file src/src/#mfile# in the sources.

I am preparing a new spkg.

comment:68 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Changed 7 years ago by jdemeyer

Spkg diff, for review only.

comment:69 Changed 7 years ago by jpflori

Thanks for improving the spkg. The #mfile# is just some crap emacs must have produced.

comment:70 Changed 7 years ago by jpflori

  • Status changed from needs_review to positive_review

comment:71 Changed 7 years ago by kcrisman

Sweet, I'll try this out on XP again right now. Thanks

comment:72 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.6.beta3
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:73 Changed 7 years ago by kcrisman

  • Authors changed from Karl-Dieter Crisman, Jean-Pierre Flori to Jean-Pierre Flori

I think that my role as author was fairly limited in the final version, and presumably someone needs to have reviewed me in any case ;-)

Note: See TracTickets for help on using tickets.