broken tar on cygwin (Windows 7), cddlib and networkx fail to install

[Dima Pasechnik]

on Windows 7 tar does not unpack cddlib spkg correctly, symbolic links get broken. See ​http://groups.google.com/group/sage-windows/browse_thread/thread/d4965bfbc83a9bf8 and ​http://permalink.gmane.org/gmane.os.cygwin/127433

A remedy is to replace them with files, i.e. use tar with -h option to recreate the spkg---on a machine where tar works, certainly. Works for me. Something along the following lines:

/* on a Unix, with good tar: */
$ tar xjf foo.spkg
$ tar chjf foo.spkg foo/
$ tar xjf foo.spkg
$ sage -spkg foo
$ scp foo.spkg win7cygwinhost:/sageroot/spkg/standard/

This is a list of all spkgs containing symbolic links:

1. cddlib
2. iml
3. libm4ri
4. networkx
5. r

But it seems only cddlib and networkx are affected.

---

New spkgs available at ​http://boxen.math.washington.edu/home/jdemeyer/spkg/cddlib-094f.p10.spkg and ​networkx-1.2.p2.spkg

[David Kirkby]

Using

tar -format=posix

might be more reliable. Sun's tar command can't extract the archives created in Sage.

[Dima Pasechnik]

Replying to drkirkby:

Using

tar -format=posix

might be more reliable. Sun's tar command can't extract the archives created in Sage.

It's a different story here. Actually, it is a GNU tar ported to create some kind of Windows symlinks. It does not untar files it itself creates correctly...

[Dima Pasechnik]

Replying to dimpase:

on Windows 7 tar does not unpack cddlib spkg correctly, symbolic links get broken. See ​http://groups.google.com/group/sage-windows/browse_thread/thread/d4965bfbc83a9bf8#

hopefully, this will be patched in Cygwin, as proposed by Dan Grayson: ​http://cygwin.com/ml/cygwin/2011-04/msg00385.html

[Karl-Dieter Crisman]

I can confirm this problem on Windows 7 only. Testing the fix now, though I'm not sure exactly how this would get incorporated into Sage...

[Karl-Dieter Crisman]

I can confirm that this enables building of cddlib on Windows 7.

While waiting to try for other things, I have a question. The process is

1. Untar
2. Tar with links followed
3. Untar again
4. Use sage -pkg to create the new spkg

So does that mean that the untarred one the second time does have the followed links, not just symbolic ones? I guess I am curious as to why steps 3 and 4 are needed, though I assume it is in order to get the correct spkg checking.

[Dima Pasechnik]

Replying to kcrisman:

I can confirm that this enables building of cddlib on Windows 7.

While waiting to try for other things, I have a question. The process is

1. Untar
2. Tar with links followed
3. Untar again
4. Use sage -pkg to create the new spkg

So does that mean that the untarred one the second time does have the followed links, not just symbolic ones? I guess I am curious as to why steps 3 and 4 are needed, though I assume it is in order to get the correct spkg checking.

right, 1,2 and 3 replace links by file copies, and 4 makes a kosher spkg. This must be done on a system with working tar, not on Win7, obviously...

In fact, hopefully Cygwin will have a patched tar soon, patch supplied by Dan Grayson (yes, the K-theorist!), and this won't be needed.

[Karl-Dieter Crisman]

Yes, I saw that on your thread :)

But until that time (which would also fix some of our other problems, iirc) we might as well have this option.

I wonder if most spkgs just don't have symlinks? It would seem weird to me, in a source package, but maybe it's normal.

Any ideas on how to make this 'portable', or would we just have a special Cygwin-only version of the spkg?

[Karl-Dieter Crisman]

On another note:

Just as a data point, I did not have trouble with this issue on networkx. Perhaps there are fewer symlinks, so there is some chance greater than epsilon that that spkg will unpack correctly.

[Karl-Dieter Crisman]

I made an spkg for cddlib for myself at ​this link, but I don't know whether this is something that could be merged... because, for one thing, I didn't update the instructions, and every time this is remade this problem would happen again. One way to fix this would be to change the symlinks in this package to copies "by hand" and put that in the spkg notes...

[Karl-Dieter Crisman]

I've still never seen this on networkx.

I think that it would be enough to fix this issue to do the spkg this special way ONCE and then from then on just make sure there are build instructions to this effect in the spkg. My spkg doesn't have any changes like that, so this is not ready for review.

[Karl-Dieter Crisman]

Okay, I now have a p9 version of cddlib which just changes the instructions to indicate this needs to be done (and of course has had it done!).

New spkg available at ​cddlib-094f.p9.spkg. The only change is in SPKG.txt and in having done the tar -h above.

Also cc:ing the spkg maintainers!

[Dima Pasechnik]

Replying to kcrisman:

I've still never seen this on networkx.

I think that it would be enough to fix this issue to do the spkg this special way ONCE and then from then on just make sure there are build instructions to this effect in the spkg. My spkg doesn't have any changes like that, so this is not ready for review.

Cygwin developers finally managed to reproduce the problem: ​http://permalink.gmane.org/gmane.os.cygwin/127433

it boils down to a bug/feature introduced in 1.7.9, and not really related to tar itself; so apparently it will be fixed soon.

[Karl-Dieter Crisman]

User "RegB" was able to use this successfully for this on Vista - see ​this sage-windows thread.

This still needs review on Windows 7, of course, since that is where the problem is.

[Karl-Dieter Crisman]

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

[Dima Pasechnik]

Replying to kcrisman:

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

You mean, your cdd package? Yes, it works. I guess if you can test the networkx one that I made we can give this ticket positive review.

[Karl-Dieter Crisman]

Replying to dimpase:

Replying to kcrisman:

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

You mean, your cdd package? Yes, it works. I guess if you can test the networkx one that I made we can give this ticket positive review.

Okay, tomorrow morning I'm going to be able to start a Win 7 build (at least I assume so), so I should be able to deal with this. Have to have something to do during the Putnam other than grade exams :)

[Karl-Dieter Crisman]

After other problems discussed on other tickets, I did pass this point on two different builds on Win 7. Goes in fine on OSX, tests in graphs directory pass. Adding patches to hg was a good idea too. Assuming there are no other changes to the spkg, should be good to go.

[Jeroen Demeyer]

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

[Karl-Dieter Crisman]

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

My sense is that Dima wouldn't have made the networkx package still if it was working properly. I didn't even try without it, though, and am having other problems with my build currently, so I'm not going to be able to in the immediate future.

[Dima Pasechnik]

Replying to jdemeyer:

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

with a stable version, Cygwin 1.7.9-1, less than a month ago, they were still present, and 1.7.10 is still far from being released.

[Karl-Dieter Crisman]

So I think we can go back to positive review?

[Jeroen Demeyer]

cddlib needs to be rebased to #12131.

[Karl-Dieter Crisman]

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!'' - it seems like it would have been reasonable to merge this first.

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there, and to do Dima's little trick to it, and make that a p10. And by then I'm sure something else will happen to networkx and we'll have to change that one.

Sorry, just had to get that out. I understand that OpenSUSE is more important than Cygwin for now. Though hopefully if we ever get Cygwin going, we'll have several orders of magnitude more downloads of it than any Linux other than Ubuntu...

[Karl-Dieter Crisman]

Let's try that again. It wasn't all supposed to be italicized. I forgot about the magical properties of the exclamation point.

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!! - it seems like it would have been reasonable to merge this first. It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there, and to do Dima's little trick to it, and make that a p10. And by then I'm sure something else will happen to networkx and we'll have to change that one.

Sorry, just had to get that out. I understand that OpenSUSE is more important than Cygwin for now. Though hopefully if we ever get Cygwin going, we'll have several orders of magnitude more downloads of it than any Linux other than Ubuntu...

[Karl-Dieter Crisman]

Forget it...

[Karl-Dieter Crisman]

Unless...

[Jeroen Demeyer]

Replying to kcrisman:

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!'' - it seems like it would have been reasonable to merge this first.

Well, maybe you are right but this is just the way things worked out. I don't have a sophisticated bookkeeping of which tickets change which packages, it just happened that I merged #12131 first. Besides, making the change required for this ticket is quite simple, so it can't be a big problem.

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there

Why is this specific to cddlib? Any of the 29 packages merged in #12131 could be broken in Cygwin.

And by then I'm sure something else will happen to networkx and we'll have to change that one.

I can assure you I won't let this happen.

[Karl-Dieter Crisman]

Well, maybe you are right but this is just the way things worked out. I don't have a sophisticated bookkeeping of which tickets change which packages, it just happened that I merged #12131 first. Besides, making the change required for this ticket is quite simple, so it can't be a big problem.

Well, it's still time-consuming for me. I know that sounds lame, but it's a fact :(

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there

Why is this specific to cddlib? Any of the 29 packages merged in #12131 could be broken in Cygwin.

Haha! Good point... maybe I need to start another build of the "proto" alpha5 tonight.

Except that mpir doesn't work because of the yasm business (if not other things) - #12115. And going all the way up to 2.5 looks like a big deal, one I don't want to do without the experts who actually use those big integers on a daily basis.

Still can't figure out how to remove the italics. I know someone was hoping to upgrade Trac at some point, but it's not really that important.

[Volker Braun]

Easy, I'll rebase it.

[Volker Braun]

I'll switch it back to positive review since its just a trivial rebase.

[Karl-Dieter Crisman]

I'll switch it back to positive review since its just a trivial rebase.

I'd feel most comfortable actually trying this out on a Win 7 box (where the problem is) but I won't have access to one for quite some time now, afaik. But this is the part I was most worried about - I don't quite understand why this change makes the difference (Dima does, of course) and so I was hoping to test it. I guess if Volker vouches that he carried out the "special instructions" we can be reasonably confident. :) Thanks.

I find the date order of the SPKG.txt info amusing as well.

[Dima Pasechnik]

Replying to kcrisman:

I'll switch it back to positive review since its just a trivial rebase.

I'd feel most comfortable actually trying this out on a Win 7 box (where the problem is) but I won't have access to one for quite some time now, afaik. But this is the part I was most worried about - I don't quite understand why this change makes the difference (Dima does, of course) and so I was hoping to test it. I guess if Volker vouches that he carried out the "special instructions" we can be reasonably confident. :) Thanks.

I might be able to try this on Win 7 on Monday - provided that VPN will let me login to NTU from Tohoku, etc...

[Volker Braun]

For the record, I just forward-ported the one-line change from #12131 to this spkg, no special treatment necessary ;-)

italic be gone.

[Karl-Dieter Crisman]

For the record, I just forward-ported the one-line change from #12131 to this spkg, no special treatment necessary ;-)

And you also did the "special instructions" with tar -h?

italic be gone.

Nice to see there are things we can't fix, keeps us humble :)

[Volker Braun]

tar -h just dereferences symlinks. You only need to dereference them once. There were no symlinks in the spkg before I added the change from #12131, and there are none now.

[Jeroen Demeyer]

The new cddlib here breaks a doctest on OS X:

sage -t -long  -force_lib devel/sage/sage/schemes/generic/toric_divisor.py
**********************************************************************
File "/Users/buildbot/build/sage/bsd-1/bsd_full/build/sage-4.8.alpha5/devel/sage-main/sage/schemes/generic/toric_divisor.py", line 1590:
    sage: supp.Vrepresentation()
Expected:
    [A vertex at (-1, 1), A vertex at (0, 2), A vertex at (0, -1), A vertex at (3, -1)]
Got:
    [A vertex at (-1, 1), A vertex at (0, 2), A vertex at (3, -1), A vertex at (0, -1)]
**********************************************************************

[Volker Braun]

Thats the old problem of cdd relying on the libc rng which produces different random numbers on different platforms. I patched that earlier to be the same on all platforms, but the source file is symlinked into another directory. Now that we dereferenced the symlinks, we need to patch both places. I've updated the spkg to do so.

I've verified that the doctest is fixed on bsd.math.

[Jeroen Demeyer]

Looks good to me. I made an updated spkg, with only the permissions of src changed.