Opened 10 years ago

Closed 9 years ago

#11245 closed defect (fixed)

broken tar on cygwin (Windows 7), cddlib and networkx fail to install

Reported by: dimpase Owned by: tbd
Priority: major Milestone: sage-4.8
Component: porting: Cygwin Keywords:
Cc: vbraun, mhampton Merged in: sage-4.8.alpha6
Authors: Karl-Dieter Crisman, Dmitrii Pasechnik Reviewers: Reg Burgess, Dmitrii Pasechnik, Karl-Dieter Crisman
Report Upstream: Fixed upstream, but not in a stable release. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by jdemeyer)

on Windows 7 tar does not unpack cddlib spkg correctly, symbolic links get broken. See http://groups.google.com/group/sage-windows/browse_thread/thread/d4965bfbc83a9bf8 and http://permalink.gmane.org/gmane.os.cygwin/127433

A remedy is to replace them with files, i.e. use tar with -h option to recreate the spkg---on a machine where tar works, certainly. Works for me. Something along the following lines:

/* on a Unix, with good tar: */
$ tar xjf foo.spkg
$ tar chjf foo.spkg foo/
$ tar xjf foo.spkg
$ sage -spkg foo
$ scp foo.spkg win7cygwinhost:/sageroot/spkg/standard/

This is a list of all spkgs containing symbolic links:

  1. cddlib
  2. iml
  3. libm4ri
  4. networkx
  5. r

But it seems only cddlib and networkx are affected.


New spkgs available at http://boxen.math.washington.edu/home/jdemeyer/spkg/cddlib-094f.p10.spkg and networkx-1.2.p2.spkg

Change History (45)

comment:1 Changed 10 years ago by dimpase

  • Description modified (diff)

comment:2 follow-up: Changed 10 years ago by drkirkby

Using

tar -format=posix

might be more reliable. Sun's tar command can't extract the archives created in Sage.

comment:3 in reply to: ↑ 2 Changed 10 years ago by dimpase

Replying to drkirkby:

Using

tar -format=posix

might be more reliable. Sun's tar command can't extract the archives created in Sage.

It's a different story here. Actually, it is a GNU tar ported to create some kind of Windows symlinks. It does not untar files it itself creates correctly...

comment:4 Changed 10 years ago by dimpase

  • Description modified (diff)

comment:5 in reply to: ↑ description Changed 10 years ago by dimpase

Replying to dimpase:

on Windows 7 tar does not unpack cddlib spkg correctly, symbolic links get broken. See http://groups.google.com/group/sage-windows/browse_thread/thread/d4965bfbc83a9bf8#

hopefully, this will be patched in Cygwin, as proposed by Dan Grayson: http://cygwin.com/ml/cygwin/2011-04/msg00385.html

comment:6 Changed 10 years ago by kcrisman

I can confirm this problem on Windows 7 only. Testing the fix now, though I'm not sure exactly how this would get incorporated into Sage...

comment:7 follow-up: Changed 10 years ago by kcrisman

I can confirm that this enables building of cddlib on Windows 7.

While waiting to try for other things, I have a question. The process is

  1. Untar
  2. Tar with links followed
  3. Untar again
  4. Use sage -pkg to create the new spkg

So does that mean that the untarred one the second time does have the followed links, not just symbolic ones? I guess I am curious as to why steps 3 and 4 are needed, though I assume it is in order to get the correct spkg checking.

comment:8 in reply to: ↑ 7 Changed 10 years ago by dimpase

Replying to kcrisman:

I can confirm that this enables building of cddlib on Windows 7.

While waiting to try for other things, I have a question. The process is

  1. Untar
  2. Tar with links followed
  3. Untar again
  4. Use sage -pkg to create the new spkg

So does that mean that the untarred one the second time does have the followed links, not just symbolic ones? I guess I am curious as to why steps 3 and 4 are needed, though I assume it is in order to get the correct spkg checking.

right, 1,2 and 3 replace links by file copies, and 4 makes a kosher spkg. This must be done on a system with working tar, not on Win7, obviously...

In fact, hopefully Cygwin will have a patched tar soon, patch supplied by Dan Grayson (yes, the K-theorist!), and this won't be needed.

comment:9 Changed 10 years ago by kcrisman

Yes, I saw that on your thread :)

But until that time (which would also fix some of our other problems, iirc) we might as well have this option.

I wonder if most spkgs just don't have symlinks? It would seem weird to me, in a source package, but maybe it's normal.

Any ideas on how to make this 'portable', or would we just have a special Cygwin-only version of the spkg?

comment:10 Changed 10 years ago by kcrisman

On another note:

Just as a data point, I did not have trouble with this issue on networkx. Perhaps there are fewer symlinks, so there is some chance greater than epsilon that that spkg will unpack correctly.

comment:11 Changed 10 years ago by kcrisman

I made an spkg for cddlib for myself at this link, but I don't know whether this is something that could be merged... because, for one thing, I didn't update the instructions, and every time this is remade this problem would happen again. One way to fix this would be to change the symlinks in this package to copies "by hand" and put that in the spkg notes...

comment:12 follow-up: Changed 10 years ago by kcrisman

I've still never seen this on networkx.

I think that it would be enough to fix this issue to do the spkg this special way ONCE and then from then on just make sure there are build instructions to this effect in the spkg. My spkg doesn't have any changes like that, so this is not ready for review.

comment:13 Changed 10 years ago by kcrisman

  • Authors set to Karl-Dieter Crisman, Dima Pasechnik
  • Cc vbraun mhampton added
  • Description modified (diff)
  • Status changed from new to needs_review

Okay, I now have a p9 version of cddlib which just changes the instructions to indicate this needs to be done (and of course has had it done!).

New spkg available at cddlib-094f.p9.spkg. The only change is in SPKG.txt and in having done the tar -h above.

Also cc:ing the spkg maintainers!

comment:14 in reply to: ↑ 12 Changed 10 years ago by dimpase

  • Description modified (diff)
  • Report Upstream changed from N/A to Reported upstream. Developers acknowledge bug.

Replying to kcrisman:

I've still never seen this on networkx.

I think that it would be enough to fix this issue to do the spkg this special way ONCE and then from then on just make sure there are build instructions to this effect in the spkg. My spkg doesn't have any changes like that, so this is not ready for review.

Cygwin developers finally managed to reproduce the problem: http://permalink.gmane.org/gmane.os.cygwin/127433

it boils down to a bug/feature introduced in 1.7.9, and not really related to tar itself; so apparently it will be fixed soon.

comment:15 Changed 10 years ago by kcrisman

  • Milestone changed from sage-4.7.1 to sage-4.7.2
  • Report Upstream changed from Reported upstream. Developers acknowledge bug. to Fixed upstream, but not in a stable release.
  • Reviewers set to Reg Burgess

User "RegB" was able to use this successfully for this on Vista - see this sage-windows thread.

This still needs review on Windows 7, of course, since that is where the problem is.

comment:16 Changed 9 years ago by dimpase

  • Description modified (diff)

comment:17 Changed 9 years ago by dimpase

  • Summary changed from broken tar on cygwin (Windows 7), cddlib fails to install to broken tar on cygwin (Windows 7), cddlib and networkx fail to install

comment:18 follow-up: Changed 9 years ago by kcrisman

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

comment:19 in reply to: ↑ 18 ; follow-up: Changed 9 years ago by dimpase

Replying to kcrisman:

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

You mean, your cdd package? Yes, it works. I guess if you can test the networkx one that I made we can give this ticket positive review.

comment:20 in reply to: ↑ 19 Changed 9 years ago by kcrisman

Replying to dimpase:

Replying to kcrisman:

So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.

You mean, your cdd package? Yes, it works. I guess if you can test the networkx one that I made we can give this ticket positive review.

Okay, tomorrow morning I'm going to be able to start a Win 7 build (at least I assume so), so I should be able to deal with this. Have to have something to do during the Putnam other than grade exams :)

comment:21 Changed 9 years ago by kcrisman

  • Authors changed from Karl-Dieter Crisman, Dima Pasechnik to Karl-Dieter Crisman, Dmitrii Pasechnik
  • Reviewers changed from Reg Burgess to Reg Burgess, Dmitrii Pasechnik, Karl-Dieter Crisman
  • Status changed from needs_review to positive_review

After other problems discussed on other tickets, I did pass this point on two different builds on Win 7. Goes in fine on OSX, tests in graphs directory pass. Adding patches to hg was a good idea too. Assuming there are no other changes to the spkg, should be good to go.

comment:22 follow-ups: Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_info

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

comment:23 in reply to: ↑ 22 Changed 9 years ago by kcrisman

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

My sense is that Dima wouldn't have made the networkx package still if it was working properly. I didn't even try without it, though, and am having other problems with my build currently, so I'm not going to be able to in the immediate future.

comment:24 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:25 in reply to: ↑ 22 Changed 9 years ago by dimpase

  • Status changed from needs_info to needs_review

Replying to jdemeyer:

It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...

with a stable version, Cygwin 1.7.9-1, less than a month ago, they were still present, and 1.7.10 is still far from being released.

comment:26 Changed 9 years ago by kcrisman

  • Status changed from needs_review to positive_review

So I think we can go back to positive review?

comment:27 follow-up: Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_work
  • Work issues set to rebase cddlib to #12131

cddlib needs to be rebased to #12131.

comment:28 in reply to: ↑ 27 ; follow-up: Changed 9 years ago by kcrisman

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!'' - it seems like it would have been reasonable to merge this first.

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there, and to do Dima's little trick to it, and make that a p10. And by then I'm sure something else will happen to networkx and we'll have to change that one.

Sorry, just had to get that out. I understand that OpenSUSE is more important than Cygwin for now. Though hopefully if we ever get Cygwin going, we'll have several orders of magnitude more downloads of it than any Linux other than Ubuntu...

comment:29 in reply to: ↑ 28 Changed 9 years ago by kcrisman

Let's try that again. It wasn't all supposed to be italicized. I forgot about the magical properties of the exclamation point.

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!! - it seems like it would have been reasonable to merge this first. It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there, and to do Dima's little trick to it, and make that a p10. And by then I'm sure something else will happen to networkx and we'll have to change that one.

Sorry, just had to get that out. I understand that OpenSUSE is more important than Cygwin for now. Though hopefully if we ever get Cygwin going, we'll have several orders of magnitude more downloads of it than any Linux other than Ubuntu...

comment:30 Changed 9 years ago by kcrisman

Forget it...

comment:31 Changed 9 years ago by kcrisman

Unless...

comment:32 follow-up: Changed 9 years ago by jdemeyer

Replying to kcrisman:

cddlib needs to be rebased to #12131.

I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!'' - it seems like it would have been reasonable to merge this first.

Well, maybe you are right but this is just the way things worked out. I don't have a sophisticated bookkeeping of which tickets change which packages, it just happened that I merged #12131 first. Besides, making the change required for this ticket is quite simple, so it can't be a big problem.

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there

Why is this specific to cddlib? Any of the 29 packages merged in #12131 could be broken in Cygwin.

And by then I'm sure something else will happen to networkx and we'll have to change that one.

I can assure you I won't let this happen.

comment:33 in reply to: ↑ 32 Changed 9 years ago by kcrisman

Well, maybe you are right but this is just the way things worked out. I don't have a sophisticated bookkeeping of which tickets change which packages, it just happened that I merged #12131 first. Besides, making the change required for this ticket is quite simple, so it can't be a big problem.

Well, it's still time-consuming for me. I know that sounds lame, but it's a fact :(

It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there

Why is this specific to cddlib? Any of the 29 packages merged in #12131 could be broken in Cygwin.

Haha! Good point... maybe I need to start another build of the "proto" alpha5 tonight.

Except that mpir doesn't work because of the yasm business (if not other things) - #12115. And going all the way up to 2.5 looks like a big deal, one I don't want to do without the experts who actually use those big integers on a daily basis.

Still can't figure out how to remove the italics. I know someone was hoping to upgrade Trac at some point, but it's not really that important.

comment:34 Changed 9 years ago by vbraun

Easy, I'll rebase it.

comment:35 follow-up: Changed 9 years ago by vbraun

  • Description modified (diff)
  • Status changed from needs_work to positive_review

I'll switch it back to positive review since its just a trivial rebase.

comment:36 in reply to: ↑ 35 ; follow-up: Changed 9 years ago by kcrisman

I'll switch it back to positive review since its just a trivial rebase.

I'd feel most comfortable actually trying this out on a Win 7 box (where the problem is) but I won't have access to one for quite some time now, afaik. But this is the part I was most worried about - I don't quite understand why this change makes the difference (Dima does, of course) and so I was hoping to test it. I guess if Volker vouches that he carried out the "special instructions" we can be reasonably confident. :) Thanks.

I find the date order of the SPKG.txt info amusing as well.

comment:37 in reply to: ↑ 36 Changed 9 years ago by dimpase

Replying to kcrisman:

I'll switch it back to positive review since its just a trivial rebase.

I'd feel most comfortable actually trying this out on a Win 7 box (where the problem is) but I won't have access to one for quite some time now, afaik. But this is the part I was most worried about - I don't quite understand why this change makes the difference (Dima does, of course) and so I was hoping to test it. I guess if Volker vouches that he carried out the "special instructions" we can be reasonably confident. :) Thanks.

I might be able to try this on Win 7 on Monday - provided that VPN will let me login to NTU from Tohoku, etc...

comment:38 follow-up: Changed 9 years ago by vbraun

For the record, I just forward-ported the one-line change from #12131 to this spkg, no special treatment necessary ;-)

italic be gone.

comment:39 in reply to: ↑ 38 Changed 9 years ago by kcrisman

For the record, I just forward-ported the one-line change from #12131 to this spkg, no special treatment necessary ;-)

And you also did the "special instructions" with tar -h?

italic be gone.

Nice to see there are things we can't fix, keeps us humble :)

comment:40 Changed 9 years ago by vbraun

tar -h just dereferences symlinks. You only need to dereference them once. There were no symlinks in the spkg before I added the change from #12131, and there are none now.

comment:41 Changed 9 years ago by jdemeyer

  • Work issues rebase cddlib to #12131 deleted

comment:42 Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_work

The new cddlib here breaks a doctest on OS X:

sage -t -long  -force_lib devel/sage/sage/schemes/generic/toric_divisor.py
**********************************************************************
File "/Users/buildbot/build/sage/bsd-1/bsd_full/build/sage-4.8.alpha5/devel/sage-main/sage/schemes/generic/toric_divisor.py", line 1590:
    sage: supp.Vrepresentation()
Expected:
    [A vertex at (-1, 1), A vertex at (0, 2), A vertex at (0, -1), A vertex at (3, -1)]
Got:
    [A vertex at (-1, 1), A vertex at (0, 2), A vertex at (3, -1), A vertex at (0, -1)]
**********************************************************************

comment:43 Changed 9 years ago by vbraun

  • Status changed from needs_work to needs_review

Thats the old problem of cdd relying on the libc rng which produces different random numbers on different platforms. I patched that earlier to be the same on all platforms, but the source file is symlinked into another directory. Now that we dereferenced the symlinks, we need to patch both places. I've updated the spkg to do so.

I've verified that the doctest is fixed on bsd.math.

comment:44 Changed 9 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_review to positive_review

Looks good to me. I made an updated spkg, with only the permissions of src changed.

comment:45 Changed 9 years ago by jdemeyer

  • Merged in set to sage-4.8.alpha6
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.