Opened 10 years ago
Closed 9 years ago
#11245 closed defect (fixed)
broken tar on cygwin (Windows 7), cddlib and networkx fail to install
Reported by: | dimpase | Owned by: | tbd |
---|---|---|---|
Priority: | major | Milestone: | sage-4.8 |
Component: | porting: Cygwin | Keywords: | |
Cc: | vbraun, mhampton | Merged in: | sage-4.8.alpha6 |
Authors: | Karl-Dieter Crisman, Dmitrii Pasechnik | Reviewers: | Reg Burgess, Dmitrii Pasechnik, Karl-Dieter Crisman |
Report Upstream: | Fixed upstream, but not in a stable release. | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
on Windows 7 tar does not unpack cddlib spkg correctly, symbolic links get broken. See http://groups.google.com/group/sage-windows/browse_thread/thread/d4965bfbc83a9bf8 and http://permalink.gmane.org/gmane.os.cygwin/127433
A remedy is to replace them with files, i.e. use tar with -h option to recreate the spkg---on a machine where tar works, certainly. Works for me. Something along the following lines:
/* on a Unix, with good tar: */ $ tar xjf foo.spkg $ tar chjf foo.spkg foo/ $ tar xjf foo.spkg $ sage -spkg foo $ scp foo.spkg win7cygwinhost:/sageroot/spkg/standard/
This is a list of all spkgs containing symbolic links:
- cddlib
- iml
- libm4ri
- networkx
- r
But it seems only cddlib and networkx are affected.
New spkgs available at http://boxen.math.washington.edu/home/jdemeyer/spkg/cddlib-094f.p10.spkg and networkx-1.2.p2.spkg
Change History (45)
comment:1 Changed 10 years ago by
- Description modified (diff)
comment:2 follow-up: ↓ 3 Changed 10 years ago by
comment:3 in reply to: ↑ 2 Changed 10 years ago by
Replying to drkirkby:
Using
tar -format=posixmight be more reliable. Sun's tar command can't extract the archives created in Sage.
It's a different story here. Actually, it is a GNU tar ported to create some kind of Windows symlinks. It does not untar files it itself creates correctly...
comment:4 Changed 10 years ago by
- Description modified (diff)
comment:5 in reply to: ↑ description Changed 10 years ago by
Replying to dimpase:
on Windows 7 tar does not unpack cddlib spkg correctly, symbolic links get broken. See http://groups.google.com/group/sage-windows/browse_thread/thread/d4965bfbc83a9bf8#
hopefully, this will be patched in Cygwin, as proposed by Dan Grayson: http://cygwin.com/ml/cygwin/2011-04/msg00385.html
comment:6 Changed 10 years ago by
I can confirm this problem on Windows 7 only. Testing the fix now, though I'm not sure exactly how this would get incorporated into Sage...
comment:7 follow-up: ↓ 8 Changed 10 years ago by
I can confirm that this enables building of cddlib on Windows 7.
While waiting to try for other things, I have a question. The process is
- Untar
- Tar with links followed
- Untar again
- Use
sage -pkg
to create the new spkg
So does that mean that the untarred one the second time does have the followed links, not just symbolic ones? I guess I am curious as to why steps 3 and 4 are needed, though I assume it is in order to get the correct spkg checking.
comment:8 in reply to: ↑ 7 Changed 10 years ago by
Replying to kcrisman:
I can confirm that this enables building of cddlib on Windows 7.
While waiting to try for other things, I have a question. The process is
- Untar
- Tar with links followed
- Untar again
- Use
sage -pkg
to create the new spkgSo does that mean that the untarred one the second time does have the followed links, not just symbolic ones? I guess I am curious as to why steps 3 and 4 are needed, though I assume it is in order to get the correct spkg checking.
right, 1,2 and 3 replace links by file copies, and 4 makes a kosher spkg. This must be done on a system with working tar, not on Win7, obviously...
In fact, hopefully Cygwin will have a patched tar soon, patch supplied by Dan Grayson (yes, the K-theorist!), and this won't be needed.
comment:9 Changed 10 years ago by
Yes, I saw that on your thread :)
But until that time (which would also fix some of our other problems, iirc) we might as well have this option.
I wonder if most spkgs just don't have symlinks? It would seem weird to me, in a source package, but maybe it's normal.
Any ideas on how to make this 'portable', or would we just have a special Cygwin-only version of the spkg?
comment:10 Changed 10 years ago by
On another note:
Just as a data point, I did not have trouble with this issue on networkx. Perhaps there are fewer symlinks, so there is some chance greater than epsilon that that spkg will unpack correctly.
comment:11 Changed 10 years ago by
I made an spkg for cddlib for myself at this link, but I don't know whether this is something that could be merged... because, for one thing, I didn't update the instructions, and every time this is remade this problem would happen again. One way to fix this would be to change the symlinks in this package to copies "by hand" and put that in the spkg notes...
comment:12 follow-up: ↓ 14 Changed 10 years ago by
I've still never seen this on networkx.
I think that it would be enough to fix this issue to do the spkg this special way ONCE and then from then on just make sure there are build instructions to this effect in the spkg. My spkg doesn't have any changes like that, so this is not ready for review.
comment:13 Changed 10 years ago by
- Cc vbraun mhampton added
- Description modified (diff)
- Status changed from new to needs_review
Okay, I now have a p9 version of cddlib which just changes the instructions to indicate this needs to be done (and of course has had it done!).
New spkg available at cddlib-094f.p9.spkg. The only change is in SPKG.txt and in having done the tar -h above.
Also cc:ing the spkg maintainers!
comment:14 in reply to: ↑ 12 Changed 10 years ago by
- Description modified (diff)
- Report Upstream changed from N/A to Reported upstream. Developers acknowledge bug.
Replying to kcrisman:
I've still never seen this on networkx.
I think that it would be enough to fix this issue to do the spkg this special way ONCE and then from then on just make sure there are build instructions to this effect in the spkg. My spkg doesn't have any changes like that, so this is not ready for review.
Cygwin developers finally managed to reproduce the problem: http://permalink.gmane.org/gmane.os.cygwin/127433
it boils down to a bug/feature introduced in 1.7.9, and not really related to tar itself; so apparently it will be fixed soon.
comment:15 Changed 10 years ago by
- Milestone changed from sage-4.7.1 to sage-4.7.2
- Report Upstream changed from Reported upstream. Developers acknowledge bug. to Fixed upstream, but not in a stable release.
- Reviewers set to Reg Burgess
User "RegB" was able to use this successfully for this on Vista - see this sage-windows thread.
This still needs review on Windows 7, of course, since that is where the problem is.
comment:16 Changed 9 years ago by
- Description modified (diff)
comment:17 Changed 9 years ago by
- Summary changed from broken tar on cygwin (Windows 7), cddlib fails to install to broken tar on cygwin (Windows 7), cddlib and networkx fail to install
comment:18 follow-up: ↓ 19 Changed 9 years ago by
So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.
comment:19 in reply to: ↑ 18 ; follow-up: ↓ 20 Changed 9 years ago by
Replying to kcrisman:
So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.
You mean, your cdd package? Yes, it works. I guess if you can test the networkx one that I made we can give this ticket positive review.
comment:20 in reply to: ↑ 19 Changed 9 years ago by
Replying to dimpase:
Replying to kcrisman:
So does the cddlib package work? Then I (or someone else) would just have to test the networkx one and make sure it untars correctly.
You mean, your cdd package? Yes, it works. I guess if you can test the networkx one that I made we can give this ticket positive review.
Okay, tomorrow morning I'm going to be able to start a Win 7 build (at least I assume so), so I should be able to deal with this. Have to have something to do during the Putnam other than grade exams :)
comment:21 Changed 9 years ago by
- Reviewers changed from Reg Burgess to Reg Burgess, Dmitrii Pasechnik, Karl-Dieter Crisman
- Status changed from needs_review to positive_review
After other problems discussed on other tickets, I did pass this point on two different builds on Win 7. Goes in fine on OSX, tests in graphs directory pass. Adding patches to hg was a good idea too. Assuming there are no other changes to the spkg, should be good to go.
comment:22 follow-ups: ↓ 23 ↓ 25 Changed 9 years ago by
- Status changed from positive_review to needs_info
It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...
comment:23 in reply to: ↑ 22 Changed 9 years ago by
It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...
My sense is that Dima wouldn't have made the networkx package still if it was working properly. I didn't even try without it, though, and am having other problems with my build currently, so I'm not going to be able to in the immediate future.
comment:24 Changed 9 years ago by
- Description modified (diff)
comment:25 in reply to: ↑ 22 Changed 9 years ago by
- Status changed from needs_info to needs_review
Replying to jdemeyer:
It seems the problem is fixed with more recent versions of Cygwin, so I see no reason to merge this...
with a stable version, Cygwin 1.7.9-1, less than a month ago, they were still present, and 1.7.10 is still far from being released.
comment:26 Changed 9 years ago by
- Status changed from needs_review to positive_review
So I think we can go back to positive review?
comment:27 follow-up: ↓ 28 Changed 9 years ago by
- Status changed from positive_review to needs_work
- Work issues set to rebase cddlib to #12131
cddlib needs to be rebased to #12131.
comment:28 in reply to: ↑ 27 ; follow-up: ↓ 29 Changed 9 years ago by
cddlib needs to be rebased to #12131.
I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!'' - it seems like it would have been reasonable to merge this first.
It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there, and to do Dima's little trick to it, and make that a p10. And by then I'm sure something else will happen to networkx and we'll have to change that one.
Sorry, just had to get that out. I understand that OpenSUSE is more important than Cygwin for now. Though hopefully if we ever get Cygwin going, we'll have several orders of magnitude more downloads of it than any Linux other than Ubuntu...
comment:29 in reply to: ↑ 28 Changed 9 years ago by
Let's try that again. It wasn't all supposed to be italicized. I forgot about the magical properties of the exclamation point.
cddlib needs to be rebased to #12131.
I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!! - it seems like it would have been reasonable to merge this first. It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there, and to do Dima's little trick to it, and make that a p10. And by then I'm sure something else will happen to networkx and we'll have to change that one.
Sorry, just had to get that out. I understand that OpenSUSE is more important than Cygwin for now. Though hopefully if we ever get Cygwin going, we'll have several orders of magnitude more downloads of it than any Linux other than Ubuntu...
comment:30 Changed 9 years ago by
Forget it...
comment:31 Changed 9 years ago by
Unless...
comment:32 follow-up: ↓ 33 Changed 9 years ago by
Replying to kcrisman:
cddlib needs to be rebased to #12131.
I understand the prerogative to merge a massive spkg bomb by the release manager. But considering that this has had positive review for 10 days - since before #12131 even existed!!'' - it seems like it would have been reasonable to merge this first.
Well, maybe you are right but this is just the way things worked out. I don't have a sophisticated bookkeeping of which tickets change which packages, it just happened that I merged #12131 first. Besides, making the change required for this ticket is quite simple, so it can't be a big problem.
It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there
Why is this specific to cddlib
? Any of the 29 packages merged in #12131 could be broken in Cygwin.
And by then I'm sure something else will happen to networkx and we'll have to change that one.
I can assure you I won't let this happen.
comment:33 in reply to: ↑ 32 Changed 9 years ago by
Well, maybe you are right but this is just the way things worked out. I don't have a sophisticated bookkeeping of which tickets change which packages, it just happened that I merged #12131 first. Besides, making the change required for this ticket is quite simple, so it can't be a big problem.
Well, it's still time-consuming for me. I know that sounds lame, but it's a fact :(
It's just not that easy to test that things still work on Cygwin, and now I will have to wait until I (or someone else) gets access to make sure this new spkg from #12131 still works there
Why is this specific to
cddlib
? Any of the 29 packages merged in #12131 could be broken in Cygwin.
Haha! Good point... maybe I need to start another build of the "proto" alpha5 tonight.
Except that mpir doesn't work because of the yasm business (if not other things) - #12115. And going all the way up to 2.5 looks like a big deal, one I don't want to do without the experts who actually use those big integers on a daily basis.
Still can't figure out how to remove the italics. I know someone was hoping to upgrade Trac at some point, but it's not really that important.
comment:34 Changed 9 years ago by
Easy, I'll rebase it.
comment:35 follow-up: ↓ 36 Changed 9 years ago by
- Description modified (diff)
- Status changed from needs_work to positive_review
I'll switch it back to positive review since its just a trivial rebase.
comment:36 in reply to: ↑ 35 ; follow-up: ↓ 37 Changed 9 years ago by
I'll switch it back to positive review since its just a trivial rebase.
I'd feel most comfortable actually trying this out on a Win 7 box (where the problem is) but I won't have access to one for quite some time now, afaik. But this is the part I was most worried about - I don't quite understand why this change makes the difference (Dima does, of course) and so I was hoping to test it. I guess if Volker vouches that he carried out the "special instructions" we can be reasonably confident. :) Thanks.
I find the date order of the SPKG.txt info amusing as well.
comment:37 in reply to: ↑ 36 Changed 9 years ago by
Replying to kcrisman:
I'll switch it back to positive review since its just a trivial rebase.
I'd feel most comfortable actually trying this out on a Win 7 box (where the problem is) but I won't have access to one for quite some time now, afaik. But this is the part I was most worried about - I don't quite understand why this change makes the difference (Dima does, of course) and so I was hoping to test it. I guess if Volker vouches that he carried out the "special instructions" we can be reasonably confident. :) Thanks.
I might be able to try this on Win 7 on Monday - provided that VPN will let me login to NTU from Tohoku, etc...
comment:38 follow-up: ↓ 39 Changed 9 years ago by
For the record, I just forward-ported the one-line change from #12131 to this spkg, no special treatment necessary ;-)
italic be gone.
comment:39 in reply to: ↑ 38 Changed 9 years ago by
For the record, I just forward-ported the one-line change from #12131 to this spkg, no special treatment necessary ;-)
And you also did the "special instructions" with tar -h?
italic be gone.
Nice to see there are things we can't fix, keeps us humble :)
comment:40 Changed 9 years ago by
tar -h just dereferences symlinks. You only need to dereference them once. There were no symlinks in the spkg before I added the change from #12131, and there are none now.
comment:41 Changed 9 years ago by
- Work issues rebase cddlib to #12131 deleted
comment:42 Changed 9 years ago by
- Status changed from positive_review to needs_work
The new cddlib here breaks a doctest on OS X:
sage -t -long -force_lib devel/sage/sage/schemes/generic/toric_divisor.py ********************************************************************** File "/Users/buildbot/build/sage/bsd-1/bsd_full/build/sage-4.8.alpha5/devel/sage-main/sage/schemes/generic/toric_divisor.py", line 1590: sage: supp.Vrepresentation() Expected: [A vertex at (-1, 1), A vertex at (0, 2), A vertex at (0, -1), A vertex at (3, -1)] Got: [A vertex at (-1, 1), A vertex at (0, 2), A vertex at (3, -1), A vertex at (0, -1)] **********************************************************************
comment:43 Changed 9 years ago by
- Status changed from needs_work to needs_review
Thats the old problem of cdd relying on the libc rng which produces different random numbers on different platforms. I patched that earlier to be the same on all platforms, but the source file is symlinked into another directory. Now that we dereferenced the symlinks, we need to patch both places. I've updated the spkg to do so.
I've verified that the doctest is fixed on bsd.math.
comment:44 Changed 9 years ago by
- Description modified (diff)
- Status changed from needs_review to positive_review
Looks good to me. I made an updated spkg, with only the permissions of src
changed.
comment:45 Changed 9 years ago by
- Merged in set to sage-4.8.alpha6
- Resolution set to fixed
- Status changed from positive_review to closed
Using
might be more reliable. Sun's tar command can't extract the archives created in Sage.