Opened 11 years ago

Closed 11 years ago

#11245 closed defect (fixed)

broken tar on cygwin (Windows 7), cddlib and networkx fail to install

Reported by: Dima Pasechnik Owned by: tbd
Priority: major Milestone: sage-4.8
Component: porting: Cygwin Keywords:
Cc: Volker Braun, mhampton Merged in: sage-4.8.alpha6
Authors: Karl-Dieter Crisman, Dmitrii Pasechnik Reviewers: Reg Burgess, Dmitrii Pasechnik, Karl-Dieter Crisman
Report Upstream: Fixed upstream, but not in a stable release. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Jeroen Demeyer)

on Windows 7 tar does not unpack cddlib spkg correctly, symbolic links get broken. See http://groups.google.com/group/sage-windows/browse_thread/thread/d4965bfbc83a9bf8 and http://permalink.gmane.org/gmane.os.cygwin/127433

A remedy is to replace them with files, i.e. use tar with -h option to recreate the spkg---on a machine where tar works, certainly. Works for me. Something along the following lines:

/* on a Unix, with good tar: */
$ tar xjf foo.spkg
$ tar chjf foo.spkg foo/
$ tar xjf foo.spkg
$ sage -spkg foo
$ scp foo.spkg win7cygwinhost:/sageroot/spkg/standard/

This is a list of all spkgs containing symbolic links:

  1. cddlib
  2. iml
  3. libm4ri
  4. networkx
  5. r

But it seems only cddlib and networkx are affected.


New spkgs available at http://boxen.math.washington.edu/home/jdemeyer/spkg/cddlib-094f.p10.spkg and networkx-1.2.p2.spkg

Change History (45)

comment:1 Changed 11 years ago by Dima Pasechnik

Description: modified (diff)

comment:2 Changed 11 years ago by David Kirkby

Using

tar -format=posix

might be more reliable. Sun's tar command can't extract the archives created in Sage.

comment:3 in reply to:  2 Changed 11 years ago by Dima Pasechnik

Replying to drkirkby:

Using

tar -format=posix

might be more reliable. Sun's tar command can't extract the archives created in Sage.

It's a different story here. Actually, it is a GNU tar ported to create some kind of Windows symlinks. It does not untar files it itself creates correctly...

comment:4 Changed 11 years ago by Dima Pasechnik

Description: modified (diff)

comment:5 in reply to:  description Changed 11 years ago by Dima Pasechnik

Replying to dimpase:

on Windows 7 tar does not unpack cddlib spkg correctly, symbolic links get broken. See http://groups.google.com/group/sage-windows/browse_thread/thread/d4965bfbc83a9bf8#

hopefully, this will be patched in Cygwin, as proposed by Dan Grayson: http://cygwin.com/ml/cygwin/2011-04/msg00385.html

comment:6 Changed 11 years ago by Karl-Dieter Crisman

I can confirm this problem on Windows 7 only. Testing the fix now, though I'm not sure exactly how this would get incorporated into Sage...

comment:7 Changed 11 years ago by Karl-Dieter Crisman

I can confirm that this enables building of cddlib on Windows 7.

While waiting to try for other things, I have a question. The process is

  1. Untar
  2. Tar with links followed
  3. Untar again
  4. Use sage -pkg to create the new spkg

So does that mean that the untarred one the second time does have the followed links, not just symbolic ones? I guess I am curious as to why steps 3 and 4 are needed, though I assume it is in order to get the correct spkg checking.

comment:8 in reply to:  7 Changed 11 years ago by Dima Pasechnik

Replying to kcrisman:

I can confirm that this enables building of cddlib on Windows 7.

While waiting to try for other things, I have a question. The process is

  1. Untar
  2. Tar with links followed
  3. Untar again
  4. Use sage -pkg to create the new spkg

So does that mean that the untarred one the second time does have the followed links, not just symbolic ones? I guess I am curious as to why steps 3 and 4 are needed, though I assume it is in order to get the correct spkg checking.

right, 1,2 and 3 replace links by file copies, and 4 makes a kosher spkg. This must be done on a system with working tar, not on Win7, obviously...

In fact, hopefully Cygwin will have a patched tar soon, patch supplied by Dan Grayson (yes, the K-theorist!), and this won't be needed.

comment:9 Changed 11 years ago by Karl-Dieter Crisman

Yes, I saw that on your thread :)

But until that time (which would also fix some of our other problems, iirc) we might as well have this option.

I wonder if most spkgs just don't have symlinks? It would seem weird to me, in a source package, but maybe it's normal.

Any ideas on how to make this 'portable', or would we just have a special Cygwin-only version of the spkg?

comment:10 Changed 11 years ago by Karl-Dieter Crisman

On another note:

Just as a data point, I did not have trouble with this issue on networkx. Perhaps there are fewer symlinks, so there is some chance greater than epsilon that that spkg will unpack correctly.

comment:11 Changed 11 years ago by Karl-Dieter Crisman

I made an spkg for cddlib for myself at this link, but I don't know whether this is something that could be merged... because, for one thing, I didn't update the instructions, and every time this is remade this problem would happen again. One way to fix this would be to change the symlinks in this package to copies "by hand" and put that in the spkg notes...

comment:12 Changed 11 years ago by Karl-Dieter Crisman

I've still never seen this on networkx.

I think that it would be enough to fix this issue to do the spkg this special way ONCE and then from then on just make sure there are build instructions to this effect in the spkg. My spkg doesn't have any changes like that, so this is not ready for review.

comment:13 Changed 11 years ago by Karl-Dieter Crisman

Authors: Karl-Dieter Crisman, Dima Pasechnik
Cc: Volker Braun mhampton added
Description: modified (diff)
Status: newneeds_review

Okay, I now have a p9 version of cddlib which just changes the instructions to indicate this needs to be done (and of course has had it done!).

New spkg available at cddlib-094f.p9.spkg. The only change is in SPKG.txt and in having done the tar -h above.

Also cc:ing the spkg maintainers!

comment:14 in reply to:  12 Changed 11 years ago by Dima Pasechnik

Description: modified (diff)
Report Upstream: N/AReported upstream. Developers acknowledge bug.

Replying to kcrisman:

I've still never seen this on networkx.

I think that it would be enough to fix this issue to do the spkg this special way ONCE and then from then on just make sure there are build instructions to this effect in the spkg. My spkg doesn't have any changes like that, so this is not ready for review.

Cygwin developers finally managed to reproduce the problem: http://permalink.gmane.org/gmane.os.cygwin/127433

it boils down to a bug/feature introduced in 1.7.9, and not really related to tar itself; so apparently it will be fixed soon.

comment:15 Changed 11 years ago by Karl-Dieter Crisman

Milestone: sage-4.7.1sage-4.7.2
Report Upstream: Reported upstream. Developers acknowledge bug.Fixed upstream, but not in a stable release.
Reviewers: Reg Burgess

User "RegB" was able to use this successfully for this on Vista - see this sage-windows thread.

This still needs review on Windows 7, of course, since that is where the problem is.

comment:16 Changed 11 years ago by Dima Pasechnik

Description: modified (diff)

comment:17 Changed 11 years ago by Dima Pasechnik

Summary: broken tar on cygwin (Windows 7), cddlib fails to installbroken tar on cygwin (Windows 7), cddlib and networkx fail to install

comment:18 Changed 11 years ago by Karl-Dieter Crisman

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

comment:19 in reply to:  18 ; Changed 11 years ago by Dima Pasechnik

Replying to kcrisman:

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

You mean, your cdd package? Yes, it works. I guess if you can test the networkx one that I made we can give this ticket positive review.

comment:20 in reply to:  19 Changed 11 years ago by Karl-Dieter Crisman

Replying to dimpase:

Replying to kcrisman:

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

You mean, your cdd package? Yes, it works. I guess if you can test the networkx one that I made we can give this ticket positive review.

Okay, tomorrow morning I'm going to be able to start a Win 7 build (at least I assume so), so I should be able to deal with this. Have to have something to do during the Putnam other than grade exams :)

comment:21 Changed 11 years ago by Karl-Dieter Crisman

Authors: Karl-Dieter Crisman, Dima PasechnikKarl-Dieter Crisman, Dmitrii Pasechnik
Reviewers: Reg BurgessReg Burgess, Dmitrii Pasechnik, Karl-Dieter Crisman
Status: needs_reviewpositive_review

After other problems discussed on other tickets, I did pass this point on two different builds on Win 7. Goes in fine on OSX, tests in graphs directory pass. Adding patches to hg was a good idea too. Assuming there are no other changes to the spkg, should be good to go.

comment:22 Changed 11 years ago by Jeroen Demeyer

Status: positive_reviewneeds_info

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

comment:23 in reply to:  22 Changed 11 years ago by Karl-Dieter Crisman

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

My sense is that Dima wouldn't have made the networkx package still if it was working properly. I didn't even try without it, though, and am having other problems with my build currently, so I'm not going to be able to in the immediate future.

comment:24 Changed 11 years ago by Jeroen Demeyer

Description: modified (diff)

comment:25 in reply to:  22 Changed 11 years ago by Dima Pasechnik

Status: needs_infoneeds_review

Replying to jdemeyer:

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

with a stable version, Cygwin 1.7.9-1, less than a month ago, they were still present, and 1.7.10 is still far from being released.

comment:26 Changed 11 years ago by Karl-Dieter Crisman

Status: needs_reviewpositive_review

So I think we can go back to positive review?

comment:27 Changed 11 years ago by Jeroen Demeyer

Status: positive_reviewneeds_work
Work issues: rebase cddlib to #12131

cddlib needs to be rebased to #12131.

comment:28 in reply to:  27 ; Changed 11 years ago by Karl-Dieter Crisman

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!'' - it seems like it would have been reasonable to merge this first.

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there, and to do Dima's little trick to it, and make that a p10. And by then I'm sure something else will happen to networkx and we'll have to change that one.

Sorry, just had to get that out. I understand that OpenSUSE is more important than Cygwin for now. Though hopefully if we ever get Cygwin going, we'll have several orders of magnitude more downloads of it than any Linux other than Ubuntu...

comment:29 in reply to:  28 Changed 11 years ago by Karl-Dieter Crisman

Let's try that again. It wasn't all supposed to be italicized. I forgot about the magical properties of the exclamation point.

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!! - it seems like it would have been reasonable to merge this first. It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there, and to do Dima's little trick to it, and make that a p10. And by then I'm sure something else will happen to networkx and we'll have to change that one.

Sorry, just had to get that out. I understand that OpenSUSE is more important than Cygwin for now. Though hopefully if we ever get Cygwin going, we'll have several orders of magnitude more downloads of it than any Linux other than Ubuntu...

comment:30 Changed 11 years ago by Karl-Dieter Crisman

Forget it...

comment:31 Changed 11 years ago by Karl-Dieter Crisman

Unless...

comment:32 Changed 11 years ago by Jeroen Demeyer

Replying to kcrisman:

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!'' - it seems like it would have been reasonable to merge this first.

Well, maybe you are right but this is just the way things worked out. I don't have a sophisticated bookkeeping of which tickets change which packages, it just happened that I merged #12131 first. Besides, making the change required for this ticket is quite simple, so it can't be a big problem.

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there

Why is this specific to cddlib? Any of the 29 packages merged in #12131 could be broken in Cygwin.

And by then I'm sure something else will happen to networkx and we'll have to change that one.

I can assure you I won't let this happen.

comment:33 in reply to:  32 Changed 11 years ago by Karl-Dieter Crisman

Well, maybe you are right but this is just the way things worked out. I don't have a sophisticated bookkeeping of which tickets change which packages, it just happened that I merged #12131 first. Besides, making the change required for this ticket is quite simple, so it can't be a big problem.

Well, it's still time-consuming for me. I know that sounds lame, but it's a fact :(

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there

Why is this specific to cddlib? Any of the 29 packages merged in #12131 could be broken in Cygwin.

Haha! Good point... maybe I need to start another build of the "proto" alpha5 tonight.

Except that mpir doesn't work because of the yasm business (if not other things) - #12115. And going all the way up to 2.5 looks like a big deal, one I don't want to do without the experts who actually use those big integers on a daily basis.

Still can't figure out how to remove the italics. I know someone was hoping to upgrade Trac at some point, but it's not really that important.

comment:34 Changed 11 years ago by Volker Braun

Easy, I'll rebase it.

comment:35 Changed 11 years ago by Volker Braun

Description: modified (diff)
Status: needs_workpositive_review

I'll switch it back to positive review since its just a trivial rebase.

comment:36 in reply to:  35 ; Changed 11 years ago by Karl-Dieter Crisman

I'll switch it back to positive review since its just a trivial rebase.

I'd feel most comfortable actually trying this out on a Win 7 box (where the problem is) but I won't have access to one for quite some time now, afaik. But this is the part I was most worried about - I don't quite understand why this change makes the difference (Dima does, of course) and so I was hoping to test it. I guess if Volker vouches that he carried out the "special instructions" we can be reasonably confident. :) Thanks.

I find the date order of the SPKG.txt info amusing as well.

comment:37 in reply to:  36 Changed 11 years ago by Dima Pasechnik

Replying to kcrisman:

I'll switch it back to positive review since its just a trivial rebase.

I'd feel most comfortable actually trying this out on a Win 7 box (where the problem is) but I won't have access to one for quite some time now, afaik. But this is the part I was most worried about - I don't quite understand why this change makes the difference (Dima does, of course) and so I was hoping to test it. I guess if Volker vouches that he carried out the "special instructions" we can be reasonably confident. :) Thanks.

I might be able to try this on Win 7 on Monday - provided that VPN will let me login to NTU from Tohoku, etc...

comment:38 Changed 11 years ago by Volker Braun

For the record, I just forward-ported the one-line change from #12131 to this spkg, no special treatment necessary ;-)

italic be gone.

comment:39 in reply to:  38 Changed 11 years ago by Karl-Dieter Crisman

For the record, I just forward-ported the one-line change from #12131 to this spkg, no special treatment necessary ;-)

And you also did the "special instructions" with tar -h?

italic be gone.

Nice to see there are things we can't fix, keeps us humble :)

comment:40 Changed 11 years ago by Volker Braun

tar -h just dereferences symlinks. You only need to dereference them once. There were no symlinks in the spkg before I added the change from #12131, and there are none now.

comment:41 Changed 11 years ago by Jeroen Demeyer

Work issues: rebase cddlib to #12131

comment:42 Changed 11 years ago by Jeroen Demeyer

Status: positive_reviewneeds_work

The new cddlib here breaks a doctest on OS X:

sage -t -long  -force_lib devel/sage/sage/schemes/generic/toric_divisor.py
**********************************************************************
File "/Users/buildbot/build/sage/bsd-1/bsd_full/build/sage-4.8.alpha5/devel/sage-main/sage/schemes/generic/toric_divisor.py", line 1590:
    sage: supp.Vrepresentation()
Expected:
    [A vertex at (-1, 1), A vertex at (0, 2), A vertex at (0, -1), A vertex at (3, -1)]
Got:
    [A vertex at (-1, 1), A vertex at (0, 2), A vertex at (3, -1), A vertex at (0, -1)]
**********************************************************************

comment:43 Changed 11 years ago by Volker Braun

Status: needs_workneeds_review

Thats the old problem of cdd relying on the libc rng which produces different random numbers on different platforms. I patched that earlier to be the same on all platforms, but the source file is symlinked into another directory. Now that we dereferenced the symlinks, we need to patch both places. I've updated the spkg to do so.

I've verified that the doctest is fixed on bsd.math.

comment:44 Changed 11 years ago by Jeroen Demeyer

Description: modified (diff)
Status: needs_reviewpositive_review

Looks good to me. I made an updated spkg, with only the permissions of src changed.

comment:45 Changed 11 years ago by Jeroen Demeyer

Merged in: sage-4.8.alpha6
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.