Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#13557 closed enhancement (fixed)

Make the autotools spkg an optional spkg

Reported by: jdemeyer Owned by: tbd
Priority: major Milestone: sage-5.6
Component: packages: optional Keywords:
Cc: Merged in: sage-5.6.beta0
Authors: Jeroen Demeyer Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: recommended: #12707 Stopgaps:

Status badges

Description (last modified by jdemeyer)

Instead of "brute force" building the autotools spkg, figure out the actual dependencies. Also add Texinfo. All this makes the spkg a lot more portable.

I believe this makes the package mature enough to be upgraded from "experimental" to "optional".

spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/autotools-20121217.spkg

autotools-20121217 (Jeroen Demeyer, 17 December 2012)

  • Add proper dependency checking instead of bruce-force building.
  • Add texinfo package
  • Make the package portable
  • Add patch m4-no-gets.patch for m4

Attachments (4)

autotools-20120930.log (319.3 KB) - added by jpflori 9 years ago.
Cygwin log.
install-info.exe.manifest (575 bytes) - added by jpflori 9 years ago.
patch.exe.manifest (552 bytes) - added by jpflori 9 years ago.
autotools-20121217.diff (24.1 KB) - added by jdemeyer 9 years ago.
Diff for the autotools spkg. For reference / review only.

Download all attachments as: .zip

Change History (61)

comment:1 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:2 Changed 9 years ago by jdemeyer

  • Status changed from new to needs_review

comment:3 Changed 9 years ago by jdemeyer

  • Status changed from needs_review to needs_work
  • Summary changed from autotools spkg: add proper dependency checking to Improve the autotools spkg

comment:4 Changed 9 years ago by jdemeyer

  • Dependencies set to #12707 (sort of)

comment:5 Changed 9 years ago by jdemeyer

  • Dependencies changed from #12707 (sort of) to recommended: #12707
  • Description modified (diff)

comment:6 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:7 Changed 9 years ago by jdemeyer

  • Component changed from experimental package to optional packages
  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:8 follow-up: Changed 9 years ago by jpflori

If I undersand correctly after just looking at the diff from the previous version, you now use spkg-write-makefile to generate Makefile.build which is later used to build autotools et al. and requires Sage with autotools, whence the impossibility to generate it on the fly. But, is there really no reason to track it in the Mercurial tree?

comment:9 in reply to: ↑ 8 ; follow-up: Changed 9 years ago by jdemeyer

Replying to jpflori:

If I undersand correctly after just looking at the diff from the previous version, you now use spkg-write-makefile to generate Makefile.build which is later used to build autotools et al. and requires Sage with autotools, whence the impossibility to generate it on the fly.

Exactly right. You need autoconf to properly determine which autoconf version is needed to parse a given configure.ac file. So paradoxically, the package needs itself to be installed in order to repackage it.

But, is there really no reason to track it in the Mercurial tree?

It's autogenerated, so I guess we shouldn't track it.

comment:10 in reply to: ↑ 9 ; follow-up: Changed 9 years ago by jpflori

Replying to jdemeyer:

But, is there really no reason to track it in the Mercurial tree?

It's autogenerated, so I guess we shouldn't track it.

I understand, just wanted to see if you could change your mind easily. I guess we could advocate both choices:

  • track the file because it's shipped in the spkg and used by the build process and cannot be autogenerated on all systems,
  • exclude it because it's automatically generated before shipping (as configure, Makefile et al files in autotools driven packages!)

I'll try to give the spkg a proper look by the end of the week (although everything looks fine after a quick look).

comment:11 in reply to: ↑ 10 Changed 9 years ago by jdemeyer

Replying to jpflori:

I understand, just wanted to see if you could change your mind easily.

It's not that I refuse to track it, it's more that I don't see the reason for it.

  • track the file because it's shipped in the spkg and used by the build process and cannot be autogenerated on all systems,

With this argument, we would need to track all of src/ also...

  • exclude it because it's automatically generated before shipping (as configure, Makefile et al files in autotools driven packages!)

Exactly, other projects would not usually track configure in version control.

comment:12 follow-up: Changed 9 years ago by jdemeyer

Does it build on Cygwin by the way?

comment:13 in reply to: ↑ 12 Changed 9 years ago by jpflori

Replying to jdemeyer:

Does it build on Cygwin by the way?

Not sure, but I guess so. As you ask I'll start a build tonight hoping it will be finished by tomorrow morning and that windows does not automatically restart because of updates. The problem is that for some reason configure scripts are awfully slow (maybe console output? I've check for BLODA, deactivated antivirus, ran in safe mode etc. but the behavior looked the same; as well, configuring MPIR is a real pain, especially since we run the configure script twice in the spkg, and quite strangely configuring GMP seems faster because less subdirs are treated, maybe some timestamps issues upstreams, or just that the build systems really diverged...).

I think the previous spkg would have built correctly, it's just it was already a few hours when I filed the bogus Trac ticket secretly hoping there was a spkg problem rather than having to wait for another bunch of hours, and then I went to sleep but Windows decided to restart after 4 hours because of updates (I was aware of that but forgot when I went to sleep and it was not sufficient to let the build finish).

comment:14 follow-up: Changed 9 years ago by jpflori

In fact it failed while building help2man with

config.status: creating Makefile
make[1] : on entre dans le répertoire « /home/jp/sage-5.2/spkg/build/autotools-20120930/src/help2man-1.40.11 »
perl help2man.PL
Extracting help2man (with variable substitutions)
make[1] : on quitte le répertoire « /home/jp/sage-5.2/spkg/build/autotools-20120930/src/help2man-1.40.11 »
make[1] : on entre dans le répertoire « /home/jp/sage-5.2/spkg/build/autotools-20120930/src/help2man-1.40.11 »
./mkinstalldirs /home/jp/sage-5.2/local/bin
./mkinstalldirs /home/jp/sage-5.2/local/lib/help2man
./mkinstalldirs /home/jp/sage-5.2/local/share/man/man1
./mkinstalldirs /home/jp/sage-5.2/local/share/info
/usr/bin/install -c help2man /home/jp/sage-5.2/local/bin
/usr/bin/install -c -m 644 $(perl -e 'print +(grep -f, map "$_/$ARGV[0]",  map +(length) ? $_ : ".", split ":", $ENV{VPATH} || ".")[0]' help2man.1) /home/jp/sage-5.2/local/share/man/man1
/usr/bin/install -c -m 644 $(perl -e 'print +(grep -f, map "$_/$ARGV[0]",  map +(length) ? $_ : ".", split ":", $ENV{VPATH} || ".")[0]' help2man.info) \
    /home/jp/sage-5.2/local/share/info/help2man.info
if test -f /home/jp/sage-5.2/local/share/info/dir; \
then \
    /home/jp/sage-5.2/local/bin/install-info --info-dir=/home/jp/sage-5.2/local/share/info \
        /home/jp/sage-5.2/local/share/info/help2man.info; \
fi
/usr/bin/bash: line 2: /home/jp/sage-5.2/local/bin/install-info: Permission denied
Makefile:51: recipe for target `install_base' failed
make[1]: *** [install_base] Error 126
make[1] : on quitte le répertoire « /home/jp/sage-5.2/spkg/build/autotools-20120930/src/help2man-1.40.11 »
Makefile:21: recipe for target `/home/jp/sage-5.2/local/bin/help2man' failed
make: *** [/home/jp/sage-5.2/local/bin/help2man] Error 2

I'm retesting the previous spkg to see what happened there.

comment:15 follow-up: Changed 9 years ago by jpflori

Strangely in the log from last night, there was no problem with help2man:

make[1] : on entre dans le répertoire « /home/jp/sage-5.2/spkg/build/autotools-\
20120810/src/help2man-1.40.11 »
perl help2man.PL
Extracting help2man (with variable substitutions)
./mkinstalldirs /home/jp/sage-5.2/local/bin
./mkinstalldirs /home/jp/sage-5.2/local/lib/help2man
mkdir -p -- /home/jp/sage-5.2/local/lib/help2man
./mkinstalldirs /home/jp/sage-5.2/local/share/man/man1
./mkinstalldirs /home/jp/sage-5.2/local/share/info
/usr/bin/install -c help2man /home/jp/sage-5.2/local/bin
/usr/bin/install -c -m 644 $(perl -e 'print +(grep -f, map "$_/$ARGV[0]",  map \
+(length) ? $_ : ".", split ":", $ENV{VPATH} || ".")[0]' help2man.1) /home/jp/s\
age-5.2/local/share/man/man1
/usr/bin/install -c -m 644 $(perl -e 'print +(grep -f, map "$_/$ARGV[0]",  map \
+(length) ? $_ : ".", split ":", $ENV{VPATH} || ".")[0]' help2man.info) \
    /home/jp/sage-5.2/local/share/info/help2man.info
if test -f /home/jp/sage-5.2/local/share/info/dir; \
then \
    /usr/bin/install-info --info-dir=/home/jp/sage-5.2/local/share/info \
        /home/jp/sage-5.2/local/share/info/help2man.info; \
fi
make[1] : on quitte le répertoire « /home/jp/sage-5.2/spkg/build/autotools-2012\
0810/src/help2man-1.40.11 »
( cd /home/jp/sage-5.2/spkg/build/autotools-20120810/src/autoconf && git archiv\
e --format=tar --prefix=autoconf-2.13.rc1/ autoconf-2-13-rc1 ) | tar xf -

Oh I see now, the /usr/bin/install-info used to be used, whereas some Sage's one is now. Is that related to the coreutils dependency drop?

comment:16 in reply to: ↑ 14 ; follow-up: Changed 9 years ago by jdemeyer

Did you build in parallel (i.e. with MAKE="make -jN")?

I don't know much about Cygwin, but does it use Unix-style permission bits? Could you do

ls -l /home/jp/sage-5.2/local/bin/install-info

It looks like that program was installed without execute permission.

comment:17 in reply to: ↑ 15 Changed 9 years ago by jdemeyer

Replying to jpflori:

Oh I see now, the /usr/bin/install-info used to be used, whereas some Sage's one is now. Is that related to the coreutils dependency drop?

No, it's because I added Texinfo (which contains install-info) in the package. But somehow, it doesn't get installed correctly.

comment:18 Changed 9 years ago by jdemeyer

Could you attach the complete log of the failed autotools build on Cygwin?

comment:19 in reply to: ↑ 16 ; follow-up: Changed 9 years ago by jpflori

Replying to jdemeyer:

Did you build in parallel (i.e. with MAKE="make -jN")?

No, I just launched the Cygwin shell and ran make.

I don't know much about Cygwin, but does it use Unix-style permission bits? Could you do

ls -l /home/jp/sage-5.2/local/bin/install-info

It looks like that program was installed without execute permission.

There are permission bits and the file installed is -rwx-r-x-r-x and belong to "jp None", much as all other exes, so should execute. Try to launch it from the command line fails as well with the same error.

By the way, the previous spkg build failed during the night because of memory exhaustion. I guess there is some memory leak (maybe because of some BLODA, but I'll have to find out which one, my previous investigation were unsuccessful), potenetially linked with the console slowness?

Changed 9 years ago by jpflori

Cygwin log.

comment:20 in reply to: ↑ 19 Changed 9 years ago by jpflori

Replying to jpflori:

I don't know much about Cygwin, but does it use Unix-style permission bits? Could you do

ls -l /home/jp/sage-5.2/local/bin/install-info

It looks like that program was installed without execute permission.

There are permission bits and the file installed is -rwx-r-x-r-x and belong to "jp None", much as all other exes, so should execute. Try to launch it from the command line fails as well with the same error.

But doing so with admin rights is successful, so this must be a permission problem but hidden deeper.

comment:21 Changed 9 years ago by jpflori

And indeed in the windows explorer, the file has a little shield on its icon (meaning it needs admin rights).

comment:22 Changed 9 years ago by jpflori

But looking at the permission and security properties from the windows explorer does not reveal any difference with another file which can be executed (say infokey.exe just above in the directory).

comment:23 follow-up: Changed 9 years ago by jdemeyer

Hmm, it might simply be related to the name (i.e. Windows might assume that a program with "install" in the name needs admin rights). Could you rename/copy it to something more innocent like "place-info" and try to execute it?

comment:24 follow-up: Changed 9 years ago by jdemeyer

It might be the same problem as #11232.

comment:25 in reply to: ↑ 23 ; follow-up: Changed 9 years ago by jpflori

Replying to jdemeyer:

Hmm, it might simply be related to the name (i.e. Windows might assume that a program with "install" in the name needs admin rights). Could you rename/copy it to something more innocent like "place-info" and try to execute it?

In the meantime I went to the same conclusion and that's indeed the problem. Windows asks for admin privileges to execute any file whose name contains install. Rename the file and it will execute as usual.

The proper solution is to add a install-info.exe.manifest file together with the executable. In fact Cygwin distributed packages do so, and we might just copy it from them.

comment:26 in reply to: ↑ 24 Changed 9 years ago by jpflori

Replying to jdemeyer:

It might be the same problem as #11232.

Indeed, so patch is filtered as well... Let's hope they do not filter sage in Windows 8.

comment:27 Changed 9 years ago by jpflori

And about #11232, shouldn't we just create an appropriate manifest files (it's only a few line) rather than disable the build on Cygwin (from Windows 7)?

comment:28 in reply to: ↑ 25 Changed 9 years ago by jdemeyer

Replying to jpflori:

The proper solution is to add a install-info.exe.manifest file together with the executable. In fact Cygwin distributed packages do so, and we might just copy it from them.

If you have the right file, I'd be happy to add it to the package.

Changed 9 years ago by jpflori

Changed 9 years ago by jpflori

comment:29 Changed 9 years ago by jpflori

I've uploaded the files for install-info and patch. They can be found in upstream packages, found by example in the mirrors listed here: http://cygwin.com/mirrors.html Anyway, the structure is quite simple, the only variation within the files being the field "name", which anyway looks only informative (see he patch manifest file). The only important point seems that the manifest file should be named as the exe one (including .exe) + .manifest extension.

And such files seems to be needed on Seven (and later I guess), but not before, but they should not hurt on previous versions on Windows so could be installed all the time on Cygwin.

comment:30 Changed 9 years ago by jdemeyer

Added install-info.exe.manifest to the package. Needs testing on Cygwin.

comment:31 Changed 9 years ago by jpflori

I think a $ is missing in front of $UNAME. Adding let the manifest file be installed correctly on Cygwin. The build is currently going on.

comment:32 Changed 9 years ago by jpflori

An failed at the same point... Maybe some timestamp issue, randomly touching install-info.exe[.manifest] made it really executable, I'm trying to make this determinist now.

comment:33 Changed 9 years ago by jpflori

Don't ask me why, but touching install-info.exe seems to be the right solution...

comment:34 Changed 9 years ago by jdemeyer

Okay, maybe I should not add the -p option to cp.

comment:35 Changed 9 years ago by jdemeyer

I made a new version, could you try it?

comment:36 Changed 9 years ago by jpflori

I just tested the previous spkg after removing the -p flag passed to cp (I assume this is what you did) and that was not sufficient.

Explicitely touching the exe file is still... Really strange.

comment:37 Changed 9 years ago by jdemeyer

Maybe the manifest needs to be installed before the exe itself. Can you try the new version (same URL).

comment:38 Changed 9 years ago by jpflori

It worked! install-info which was just installed after the manifest file can be normally executed. The build process is going on now, I'll report back when it passes the point it used to fail at. Unfortunately I fear the complete build will not finish because of the damn Cygwin memory leaks, but that has nothing to do with Sage.

comment:39 Changed 9 years ago by jpflori

I think I spoke too fast. While trying to rebuild the spkg in safe mode, I encountered the same permission problem. I've just deleted the old files and relaunched the build to see if it fails again...

comment:40 Changed 9 years ago by jpflori

Or not... Restarting under usual Windows, the file can be executed, so I guess the manifest file is not used under Safe, that's really a great news.

comment:41 Changed 9 years ago by jpflori

... although the system-wide install-info is executable normally in safe mode ... ... what gets me even more confused.

comment:42 Changed 9 years ago by jpflori

And if I remove the install-info.exe.manifest, the file becomes executable in safe mode. That's really terrible.

comment:43 follow-up: Changed 9 years ago by jpflori

I see the .manifest file which gets installed is not executable, but the system wide one is. Changing that let install-info.exe be executed in safe mode!

comment:44 in reply to: ↑ 43 ; follow-up: Changed 9 years ago by jdemeyer

Replying to jpflori:

I see the .manifest file which gets installed is not executable, but the system wide one is. Changing that let install-info.exe be executed in safe mode!

You mean that the manifest file itself should be executable? It doesn't make sense to me, but if it helps, I'll make that change.

comment:45 in reply to: ↑ 44 Changed 9 years ago by jpflori

Replying to jdemeyer:

Replying to jpflori:

I see the .manifest file which gets installed is not executable, but the system wide one is. Changing that let install-info.exe be executed in safe mode!

You mean that the manifest file itself should be executable? It doesn't make sense to me, but if it helps, I'll make that change.

Indeed, it does not make sense to me either and does not seem to be required in "normal" mode, but to be in "safe" mode. My hope is that the BLODA which leads to memory exhaustion (I assume it is BLODA cause the memory don't get freed when I leave Cygwin) in "normal" mode is not running in "safe" mode and I'll be able to build the spkg completely there, just to be sure everything is fine.

comment:46 Changed 9 years ago by jdemeyer

I made a new spkg, can you test it?

comment:47 Changed 9 years ago by jpflori

  • Status changed from needs_review to needs_work

This seems to fail building m4 on my Debian install:

...
gcc -std=gnu99  -I.     -g -O2 -MT clean-temp.o -MD -MP -MF .deps/clean-temp.Tpo -c -o clean-temp.o clean-temp.c
In file included from clean-temp.h:22:0,
                 from clean-temp.c:23:
./stdio.h:477:1: error: 'gets' undeclared here (not in a function)

Indeed it seems gets has been removed from recent Debian experimental glibc.

comment:48 Changed 9 years ago by jpflori

It's quite funny, because it now fails on my recently reinstalled Cygwin on 64 bits Windows 7 after building m4 with no error message.

comment:49 Changed 9 years ago by jpflori

The previous Cygwin problem might be related to tryin a parallel build, I've relaunched a sequential one to see what happens.

comment:50 Changed 9 years ago by jpflori

My bad, I was using an old version of the spkg with the $UNAME problem, trying again with the correct one. So this has nothing to do with a parallel build.

comment:51 Changed 9 years ago by jdemeyer

  • Description modified (diff)
  • Milestone changed from sage-5.5 to sage-5.6
  • Status changed from needs_work to needs_review

Changed 9 years ago by jdemeyer

Diff for the autotools spkg. For reference / review only.

comment:52 Changed 9 years ago by jdemeyer

I added a patch for the gets() issue, needs review.

comment:53 Changed 9 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review
  • Summary changed from Improve the autotools spkg to Make the autotools spkg an optional spkg

Looks good to me...

comment:54 Changed 9 years ago by schilly

I just copied the spkg into the "optional" subdir on the server and mirrors and removed it from "experimental".

comment:55 Changed 9 years ago by jdemeyer

  • Merged in set to sage-5.6.beta0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:56 follow-up: Changed 9 years ago by jpflori

Just a question, why not split out m4 and make it a standard spkg? And drop the corresponding dependency for Sage? We are shipping patch as a spkg, so why not m4?

comment:57 in reply to: ↑ 56 Changed 9 years ago by jdemeyer

Replying to jpflori:

Just a question, why not split out m4 and make it a standard spkg?

I am totally in favour of this: https://groups.google.com/forum/?fromgroups#!msg/sage-devel/7LiuRMhO-B8/tlf2JWwfkfMJ

Note that a recent m4 is required to fix #11391.

Last edited 9 years ago by jdemeyer (previous) (diff)
Note: See TracTickets for help on using tickets.