Make the autotools spkg an optional spkg

[jdemeyer]

Instead of "brute force" building the autotools spkg, figure out the actual dependencies. Also add Texinfo. All this makes the spkg a lot more portable.

I believe this makes the package mature enough to be upgraded from "experimental" to "optional".

spkg: ​http://boxen.math.washington.edu/home/jdemeyer/spkg/autotools-20121217.spkg

autotools-20121217 (Jeroen Demeyer, 17 December 2012)

• Add proper dependency checking instead of bruce-force building.
• Add texinfo package
• Make the package portable
• Add patch m4-no-gets.patch for m4

[jpflori]

If I undersand correctly after just looking at the diff from the previous version, you now use spkg-write-makefile to generate Makefile.build which is later used to build autotools et al. and requires Sage with autotools, whence the impossibility to generate it on the fly. But, is there really no reason to track it in the Mercurial tree?

[jdemeyer]

Replying to jpflori:

If I undersand correctly after just looking at the diff from the previous version, you now use spkg-write-makefile to generate Makefile.build which is later used to build autotools et al. and requires Sage with autotools, whence the impossibility to generate it on the fly.

Exactly right. You need autoconf to properly determine which autoconf version is needed to parse a given configure.ac file. So paradoxically, the package needs itself to be installed in order to repackage it.

But, is there really no reason to track it in the Mercurial tree?

It's autogenerated, so I guess we shouldn't track it.

[jpflori]

Replying to jdemeyer:

But, is there really no reason to track it in the Mercurial tree?

It's autogenerated, so I guess we shouldn't track it.

I understand, just wanted to see if you could change your mind easily. I guess we could advocate both choices:

• track the file because it's shipped in the spkg and used by the build process and cannot be autogenerated on all systems,
• exclude it because it's automatically generated before shipping (as configure, Makefile et al files in autotools driven packages!)

I'll try to give the spkg a proper look by the end of the week (although everything looks fine after a quick look).

[jdemeyer]

Replying to jpflori:

I understand, just wanted to see if you could change your mind easily.

It's not that I refuse to track it, it's more that I don't see the reason for it.

• track the file because it's shipped in the spkg and used by the build process and cannot be autogenerated on all systems,

With this argument, we would need to track all of src/ also...

• exclude it because it's automatically generated before shipping (as configure, Makefile et al files in autotools driven packages!)

Exactly, other projects would not usually track configure in version control.

[jdemeyer]

Does it build on Cygwin by the way?

[jpflori]

Replying to jdemeyer:

Does it build on Cygwin by the way?

Not sure, but I guess so. As you ask I'll start a build tonight hoping it will be finished by tomorrow morning and that windows does not automatically restart because of updates. The problem is that for some reason configure scripts are awfully slow (maybe console output? I've check for BLODA, deactivated antivirus, ran in safe mode etc. but the behavior looked the same; as well, configuring MPIR is a real pain, especially since we run the configure script twice in the spkg, and quite strangely configuring GMP seems faster because less subdirs are treated, maybe some timestamps issues upstreams, or just that the build systems really diverged...).

I think the previous spkg would have built correctly, it's just it was already a few hours when I filed the bogus Trac ticket secretly hoping there was a spkg problem rather than having to wait for another bunch of hours, and then I went to sleep but Windows decided to restart after 4 hours because of updates (I was aware of that but forgot when I went to sleep and it was not sufficient to let the build finish).

[jpflori]

In fact it failed while building help2man with

config.status: creating Makefile
make[1] : on entre dans le répertoire « /home/jp/sage-5.2/spkg/build/autotools-20120930/src/help2man-1.40.11 »
perl help2man.PL
Extracting help2man (with variable substitutions)
make[1] : on quitte le répertoire « /home/jp/sage-5.2/spkg/build/autotools-20120930/src/help2man-1.40.11 »
make[1] : on entre dans le répertoire « /home/jp/sage-5.2/spkg/build/autotools-20120930/src/help2man-1.40.11 »
./mkinstalldirs /home/jp/sage-5.2/local/bin
./mkinstalldirs /home/jp/sage-5.2/local/lib/help2man
./mkinstalldirs /home/jp/sage-5.2/local/share/man/man1
./mkinstalldirs /home/jp/sage-5.2/local/share/info
/usr/bin/install -c help2man /home/jp/sage-5.2/local/bin
/usr/bin/install -c -m 644 $(perl -e 'print +(grep -f, map "$_/$ARGV[0]",  map +(length) ? $_ : ".", split ":", $ENV{VPATH} || ".")[0]' help2man.1) /home/jp/sage-5.2/local/share/man/man1
/usr/bin/install -c -m 644 $(perl -e 'print +(grep -f, map "$_/$ARGV[0]",  map +(length) ? $_ : ".", split ":", $ENV{VPATH} || ".")[0]' help2man.info) \
    /home/jp/sage-5.2/local/share/info/help2man.info
if test -f /home/jp/sage-5.2/local/share/info/dir; \
then \
    /home/jp/sage-5.2/local/bin/install-info --info-dir=/home/jp/sage-5.2/local/share/info \
        /home/jp/sage-5.2/local/share/info/help2man.info; \
fi
/usr/bin/bash: line 2: /home/jp/sage-5.2/local/bin/install-info: Permission denied
Makefile:51: recipe for target `install_base' failed
make[1]: *** [install_base] Error 126
make[1] : on quitte le répertoire « /home/jp/sage-5.2/spkg/build/autotools-20120930/src/help2man-1.40.11 »
Makefile:21: recipe for target `/home/jp/sage-5.2/local/bin/help2man' failed
make: *** [/home/jp/sage-5.2/local/bin/help2man] Error 2

I'm retesting the previous spkg to see what happened there.

[jpflori]

Strangely in the log from last night, there was no problem with help2man:

make[1] : on entre dans le répertoire « /home/jp/sage-5.2/spkg/build/autotools-\
20120810/src/help2man-1.40.11 »
perl help2man.PL
Extracting help2man (with variable substitutions)
./mkinstalldirs /home/jp/sage-5.2/local/bin
./mkinstalldirs /home/jp/sage-5.2/local/lib/help2man
mkdir -p -- /home/jp/sage-5.2/local/lib/help2man
./mkinstalldirs /home/jp/sage-5.2/local/share/man/man1
./mkinstalldirs /home/jp/sage-5.2/local/share/info
/usr/bin/install -c help2man /home/jp/sage-5.2/local/bin
/usr/bin/install -c -m 644 $(perl -e 'print +(grep -f, map "$_/$ARGV[0]",  map \
+(length) ? $_ : ".", split ":", $ENV{VPATH} || ".")[0]' help2man.1) /home/jp/s\
age-5.2/local/share/man/man1
/usr/bin/install -c -m 644 $(perl -e 'print +(grep -f, map "$_/$ARGV[0]",  map \
+(length) ? $_ : ".", split ":", $ENV{VPATH} || ".")[0]' help2man.info) \
    /home/jp/sage-5.2/local/share/info/help2man.info
if test -f /home/jp/sage-5.2/local/share/info/dir; \
then \
    /usr/bin/install-info --info-dir=/home/jp/sage-5.2/local/share/info \
        /home/jp/sage-5.2/local/share/info/help2man.info; \
fi
make[1] : on quitte le répertoire « /home/jp/sage-5.2/spkg/build/autotools-2012\
0810/src/help2man-1.40.11 »
( cd /home/jp/sage-5.2/spkg/build/autotools-20120810/src/autoconf && git archiv\
e --format=tar --prefix=autoconf-2.13.rc1/ autoconf-2-13-rc1 ) | tar xf -

Oh I see now, the /usr/bin/install-info used to be used, whereas some Sage's one is now. Is that related to the coreutils dependency drop?

[jdemeyer]

Did you build in parallel (i.e. with MAKE="make -jN")?

I don't know much about Cygwin, but does it use Unix-style permission bits? Could you do

ls -l /home/jp/sage-5.2/local/bin/install-info

It looks like that program was installed without execute permission.

[jdemeyer]

Replying to jpflori:

Oh I see now, the /usr/bin/install-info used to be used, whereas some Sage's one is now. Is that related to the coreutils dependency drop?

No, it's because I added Texinfo (which contains install-info) in the package. But somehow, it doesn't get installed correctly.

[jdemeyer]

Could you attach the complete log of the failed autotools build on Cygwin?

[jpflori]

Replying to jdemeyer:

Did you build in parallel (i.e. with MAKE="make -jN")?

No, I just launched the Cygwin shell and ran make.

I don't know much about Cygwin, but does it use Unix-style permission bits? Could you do

ls -l /home/jp/sage-5.2/local/bin/install-info

It looks like that program was installed without execute permission.

There are permission bits and the file installed is -rwx-r-x-r-x and belong to "jp None", much as all other exes, so should execute. Try to launch it from the command line fails as well with the same error.

By the way, the previous spkg build failed during the night because of memory exhaustion. I guess there is some memory leak (maybe because of some BLODA, but I'll have to find out which one, my previous investigation were unsuccessful), potenetially linked with the console slowness?

[jpflori]

Cygwin log.

[jpflori]

Replying to jpflori:

I don't know much about Cygwin, but does it use Unix-style permission bits? Could you do

ls -l /home/jp/sage-5.2/local/bin/install-info

It looks like that program was installed without execute permission.

There are permission bits and the file installed is -rwx-r-x-r-x and belong to "jp None", much as all other exes, so should execute. Try to launch it from the command line fails as well with the same error.

But doing so with admin rights is successful, so this must be a permission problem but hidden deeper.

[jpflori]

And indeed in the windows explorer, the file has a little shield on its icon (meaning it needs admin rights).

[jpflori]

But looking at the permission and security properties from the windows explorer does not reveal any difference with another file which can be executed (say infokey.exe just above in the directory).

[jdemeyer]

Hmm, it might simply be related to the name (i.e. Windows might assume that a program with "install" in the name needs admin rights). Could you rename/copy it to something more innocent like "place-info" and try to execute it?

[jdemeyer]

It might be the same problem as #11232.

[jpflori]

Replying to jdemeyer:

Hmm, it might simply be related to the name (i.e. Windows might assume that a program with "install" in the name needs admin rights). Could you rename/copy it to something more innocent like "place-info" and try to execute it?

In the meantime I went to the same conclusion and that's indeed the problem. Windows asks for admin privileges to execute any file whose name contains install. Rename the file and it will execute as usual.

The proper solution is to add a install-info.exe.manifest file together with the executable. In fact Cygwin distributed packages do so, and we might just copy it from them.

[jpflori]

Replying to jdemeyer:

It might be the same problem as #11232.

Indeed, so patch is filtered as well... Let's hope they do not filter sage in Windows 8.

[jpflori]

And about #11232, shouldn't we just create an appropriate manifest files (it's only a few line) rather than disable the build on Cygwin (from Windows 7)?

[jdemeyer]

Replying to jpflori:

The proper solution is to add a install-info.exe.manifest file together with the executable. In fact Cygwin distributed packages do so, and we might just copy it from them.

If you have the right file, I'd be happy to add it to the package.

[jpflori]

I've uploaded the files for install-info and patch. They can be found in upstream packages, found by example in the mirrors listed here: ​http://cygwin.com/mirrors.html Anyway, the structure is quite simple, the only variation within the files being the field "name", which anyway looks only informative (see he patch manifest file). The only important point seems that the manifest file should be named as the exe one (including .exe) + .manifest extension.

And such files seems to be needed on Seven (and later I guess), but not before, but they should not hurt on previous versions on Windows so could be installed all the time on Cygwin.

[jdemeyer]

Added install-info.exe.manifest to the package. Needs testing on Cygwin.

[jpflori]

I think a $ is missing in front of $UNAME. Adding let the manifest file be installed correctly on Cygwin. The build is currently going on.

[jpflori]

An failed at the same point... Maybe some timestamp issue, randomly touching install-info.exe[.manifest] made it really executable, I'm trying to make this determinist now.

[jpflori]

Don't ask me why, but touching install-info.exe seems to be the right solution...

[jdemeyer]

Okay, maybe I should not add the -p option to cp.

[jdemeyer]

I made a new version, could you try it?

[jpflori]

I just tested the previous spkg after removing the -p flag passed to cp (I assume this is what you did) and that was not sufficient.

Explicitely touching the exe file is still... Really strange.

[jdemeyer]

Maybe the manifest needs to be installed before the exe itself. Can you try the new version (same URL).

[jpflori]

It worked! install-info which was just installed after the manifest file can be normally executed. The build process is going on now, I'll report back when it passes the point it used to fail at. Unfortunately I fear the complete build will not finish because of the damn Cygwin memory leaks, but that has nothing to do with Sage.

[jpflori]

I think I spoke too fast. While trying to rebuild the spkg in safe mode, I encountered the same permission problem. I've just deleted the old files and relaunched the build to see if it fails again...

[jpflori]

Or not... Restarting under usual Windows, the file can be executed, so I guess the manifest file is not used under Safe, that's really a great news.

[jpflori]

... although the system-wide install-info is executable normally in safe mode ... ... what gets me even more confused.

[jpflori]

And if I remove the install-info.exe.manifest, the file becomes executable in safe mode. That's really terrible.

[jpflori]

I see the .manifest file which gets installed is not executable, but the system wide one is. Changing that let install-info.exe be executed in safe mode!

[jdemeyer]

Replying to jpflori:

I see the .manifest file which gets installed is not executable, but the system wide one is. Changing that let install-info.exe be executed in safe mode!

You mean that the manifest file itself should be executable? It doesn't make sense to me, but if it helps, I'll make that change.

[jpflori]

Replying to jdemeyer:

Replying to jpflori:

I see the .manifest file which gets installed is not executable, but the system wide one is. Changing that let install-info.exe be executed in safe mode!

You mean that the manifest file itself should be executable? It doesn't make sense to me, but if it helps, I'll make that change.

Indeed, it does not make sense to me either and does not seem to be required in "normal" mode, but to be in "safe" mode. My hope is that the BLODA which leads to memory exhaustion (I assume it is BLODA cause the memory don't get freed when I leave Cygwin) in "normal" mode is not running in "safe" mode and I'll be able to build the spkg completely there, just to be sure everything is fine.

[jdemeyer]

I made a new spkg, can you test it?

[jpflori]

This seems to fail building m4 on my Debian install:

...
gcc -std=gnu99  -I.     -g -O2 -MT clean-temp.o -MD -MP -MF .deps/clean-temp.Tpo -c -o clean-temp.o clean-temp.c
In file included from clean-temp.h:22:0,
                 from clean-temp.c:23:
./stdio.h:477:1: error: 'gets' undeclared here (not in a function)

Indeed it seems gets has been removed from recent Debian experimental glibc.

• ​http://lists.busybox.net/pipermail/buildroot/2012-July/055999.html
• ​http://sourceware.org/ml/libc-alpha/2012-06/msg00807.html
• ​http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commit;h=66712c23388e93e5c518ebc8515140fa0c807348

[jpflori]

It's quite funny, because it now fails on my recently reinstalled Cygwin on 64 bits Windows 7 after building m4 with no error message.

[jpflori]

The previous Cygwin problem might be related to tryin a parallel build, I've relaunched a sequential one to see what happens.

[jpflori]

My bad, I was using an old version of the spkg with the $UNAME problem, trying again with the correct one. So this has nothing to do with a parallel build.

[jdemeyer]

Diff for the autotools spkg. For reference / review only.

[jdemeyer]

I added a patch for the gets() issue, needs review.

[vbraun]

Looks good to me...

[schilly]

I just copied the spkg into the "optional" subdir on the server and mirrors and removed it from "experimental".

[jpflori]

Just a question, why not split out m4 and make it a standard spkg? And drop the corresponding dependency for Sage? We are shipping patch as a spkg, so why not m4?

[jdemeyer]

Replying to jpflori:

Just a question, why not split out m4 and make it a standard spkg?

I am totally in favour of this: ​https://groups.google.com/forum/?fromgroups#!msg/sage-devel/7LiuRMhO-B8/tlf2JWwfkfMJ

Note that a recent m4 is required to fix #11391.