#12057 closed enhancement (fixed)
Upgrade R (r-project.org)
Reported by: | vbraun | Owned by: | amhou |
---|---|---|---|
Priority: | critical | Milestone: | sage-4.8 |
Component: | packages: standard | Keywords: | R spkg r-project graphics PNG X11 |
Cc: | leif, jdemeyer, jason | Merged in: | sage-4.8.alpha4 |
Authors: | Volker Braun | Reviewers: | Karl-Dieter Crisman, Jeroen Demeyer |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
The current version of R (r-project) disables graphics if no X11 is available even though r.capabilities()
states that PNG is enabled. But you can't generate PNGs, which is why R graphics is broken in the Sage virtual machine. This is fixed in the newest upstream version, of which I made a new spkg.
There is no other change except for the source code and a rediffed patches/R.sh.in
On an unrelated note, it would be nice if tickets talking about R would contain "r-project" on trac since it is otherwise hopeless to search for.
Apply:
The upgraded spkg also fixes #12065.
Attachments (1)
Change History (43)
comment:1 Changed 9 years ago by
- Description modified (diff)
- Status changed from new to needs_review
- Type changed from PLEASE CHANGE to defect
comment:2 Changed 9 years ago by
comment:3 Changed 9 years ago by
- Cc leif added
Hmm, I don't mind this in theory. I'd want to test it out a little. Can you point to one of the R announcements where they end up fixing this? They have very detailed and thorough announcements, which I usually cannot make it all the way through when searching for this sort of thing.
Also, does this mean we don't need libpango or cairo or all that stuff? Or would we still need that?
Some relevant things.
- Re: #8868 - One concern is whether it always makes graphics work - OS X, Linux, Solaris. Otherwise I wouldn't want to re-enable the doctests, which are currently optional. If they work, then this ticket should enable them.
- Great idea on the r-project. I always put R things that are not about statistics per se in the "packages" component. I'm updating others now.
- Leif has put a lot of work in separating rpy from R at #9906. I'm not sure what should have priority, though, given that we are fairly behind the times and reviewing #9906 is a boatload of work.
Here are a bunch of other relevant tickets which perhaps (?) should be addressed here, or at least mentioned.
comment:4 Changed 9 years ago by
- Cc jdemeyer added
You still need cairo+pango+fontconfig+fonts; R doesn't include fonts or a font renderer. But it works now in the virtual machine without X installed. I don't have a reference for the release announcement where it was fixed.
Given that we get relatively frequent requests for R graphics in the virtual machine I would prefer it if we could get this ticket reviewed asap. Really all there is to do is regression testing, and the easiest way to do so is to run it on the build bot.
comment:5 in reply to: ↑ description Changed 9 years ago by
- Keywords spkg PNG X11 added
- Type changed from defect to enhancement
Replying to vbraun:
On an unrelated note, it would be nice if tickets talking about R would contain "r-project" on trac since it is otherwise hopeless to search for.
You can also search trac for "R" with Google.
I intentionally did not (yet) update R or RPy on #9906 or #9668. #11970 is rather unrelated to R itself; it fixes the readline spkg. #10967 could in principle be closed as invalid, as the R script in $SAGE_LOCAL/bin/
is not intended to be run directly; one should instead use sage --R ...
(although adding a sanity check to the script doesn't hurt, which I did or intended to do on #9668, which depends on #9906...).
comment:6 follow-up: ↓ 7 Changed 9 years ago by
- Priority changed from major to critical
Can we get the updated R spkg into Sage before fixing the laundry list of unrelated issues? None of the tickets leif mentioned is close to resolution. This ticket is ready to be reviewed and included right now, and I think a large group of users will benefit from being able to use graphics in the virtual machine.
comment:7 in reply to: ↑ 6 Changed 9 years ago by
Replying to vbraun:
Can we get the updated R spkg into Sage before fixing the laundry list of unrelated issues? None of the tickets leif mentioned is close to resolution. This ticket is ready to be reviewed and included right now, and I think a large group of users will benefit from being able to use graphics in the virtual machine.
I think that given Leif's lukewarmness on this and Volker's adamance, along with my own natural inclination, let's go for it. My guess is that we may not even need to backport everything for the others... ?
Testing on OSX 10.4 PPC now :)
Did you notice whether there were any needed patches in a -t -long
situation?
comment:8 Changed 9 years ago by
- Description modified (diff)
Patch attached for some small changes in sage/interfaces/r.py
. Highlights are that R now returns the class as ordered list and that it doesn't accept the somewhat spurious 'source(file=file(...))' construct that we feed it.
comment:9 follow-up: ↓ 11 Changed 9 years ago by
Questions:
- Won't
return 'file=file("%s",open="r")\nsource(file)'%filename
cause problems if someone wants to use thefile
command again, or is that local in scope? - Also, might as well add to reference manual if it's now all formatted correctly? Should just be one line adding it to the toc or something...
- Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?
Once I'm done testing it installs I'll be more formal in looking at this. What a great thing to do, thank you much!
comment:10 follow-up: ↓ 12 Changed 9 years ago by
- Status changed from needs_review to needs_work
On sage.math, I get:
make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' Makefile:111: *** multiple target patterns. Stop. make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' make[5]: *** [R] Error 2 make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' make[4]: *** [make.X11] Error 2 make[4]: *** Waiting for unfinished jobs....
comment:11 in reply to: ↑ 9 ; follow-up: ↓ 15 Changed 9 years ago by
Replying to kcrisman:
- Won't [...] cause problems if someone wants to use the
file
command again, or is that local in scope?
Functions can't be lvalues in R, so its fine though maybe not beautiful:
> file=123 > file [1] 123 > file('test.txt') description class mode text opened can read "test.txt" "file" "r" "text" "closed" "yes" can write "yes" > file [1] 123
- Also, might as well add to reference manual if it's now all formatted correctly? Should just be one line adding it to the toc or something...
I only patched up the sphinx breakage where I fixed things. There are still 2kloc to go, but I didn't want to break each and every patch for sage/interfaces/r.py
on trac...
- Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?
You just have to recompile the new R in my virtual machine and it'll basically work. You need to install fonts, though, otherwise all text in the plots will be emtpy squares.
comment:12 in reply to: ↑ 10 ; follow-up: ↓ 13 Changed 9 years ago by
Replying to jdemeyer:
On sage.math, I get:
make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' Makefile:111: *** multiple target patterns. Stop. make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' make[5]: *** [R] Error 2 make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' make[4]: *** [make.X11] Error 2 make[4]: *** Waiting for unfinished jobs....
I just built on sage.math and it worked for me (20x parallel). I'm using sage-4.8.alpha2, though. Did you merge anything in alpha3 that might cause this?
comment:13 in reply to: ↑ 12 Changed 9 years ago by
Replying to vbraun:
Replying to jdemeyer:
On sage.math, I get:
make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' Makefile:111: *** multiple target patterns. Stop. make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' make[5]: *** [R] Error 2 make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11' make[4]: *** [make.X11] Error 2 make[4]: *** Waiting for unfinished jobs....I just built on sage.math and it worked for me (20x parallel). I'm using sage-4.8.alpha2, though.
Well, it could be some strange error happening only sporadically. Or, an error which happens only if Sage is built from scratch (as opposed to installing the spkg in an existing Sage build). I will try again later.
Did you merge anything in alpha3 that might cause this?
I don't think so.
comment:14 Changed 9 years ago by
For what it's worth, I took sage-4.8.alpha2, replaced the R spkg with this one, and merged the Sage library patch. Then I built from scratch (on sage.math) and all tests passed. I'm also trying on OS X 10.6.8: the build has succeeded, and doctests are still going.
comment:15 in reply to: ↑ 11 Changed 9 years ago by
That answered most of my questions.
- Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?
You just have to recompile the new R in my virtual machine and it'll basically work. You need to install fonts, though, otherwise all text in the plots will be emtpy squares.
Ok, I'll try that.
Other comments:
- I have no problems on OS X 10.4 PPC with this or the patch. Passes relevant long optional tests. That's good!
- Has anyone tested this on Solaris? Just because sometimes there are config options which change there which have caused problems in the past.
- I would suggest that perhaps this should then remove some of the bizarre stuff in spkg-install that checks for the X headers, etc. and only for this reason. Though I suppose that could still be in another ticket, since that is for X support, and apparently the pngs still appear? So maybe forget this...
- This typo probably should be fixed in SPKG.txt, though:
=== r-2.14.0.p0 (Volker Braun, 19th November 2011) === * #12057: upgrade to new upstram version
comment:16 follow-up: ↓ 17 Changed 9 years ago by
- Status changed from needs_work to needs_review
- I tested it on mark.skynet (sparc solaris) and it works fine
- I don't want to rewrite the spkg-install right now. This should be done when rpy is separated out.
- Fixed the typo.
Jeroen, did you figure out which problem you encountered on sage.math? It seems to be working for everyone else...
comment:17 in reply to: ↑ 16 Changed 9 years ago by
Replying to vbraun:
- I tested it on mark.skynet (sparc solaris) and it works fine
- I don't want to rewrite the spkg-install right now. This should be done when rpy is separated out.
- Fixed the typo.
Jeroen, did you figure out which problem you encountered on sage.math? It seems to be working for everyone else...
I managed to compile R on the second attempt. So it certainly sometimes works. Could be a race condition.
comment:18 Changed 9 years ago by
Did anyone install this with SAGE_CHECK=yes
? On PPC OSX 10.4 I get everything ok until "running regression tests" in which reg-tests-1a
apparently gets an error. I have not had time to check this on modern Mac or on Linux (sage.math) yet, just saw this when I walked in the office this morning.
comment:19 Changed 9 years ago by
For me (Linux x86_64), the reg_tests_1a
passes. The testsuite errors out a little bit later on in
running tests of Internet and socket functions expect some differences make[3]: Entering directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests' running code in 'internet.R' ...make[3]: *** [internet.Rout] Error 1 make[3]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests' make[2]: [test-Internet] Error 2 (ignored) make[2]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests' make[1]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests' Now cleaning up tmp files. Making Sage/Python scripts relocatable... Making script relocatable Finished installing r-2.14.0.p0.spkg
probably because I'm behind a firewall. But it seems like they anticipated that and the error is caught in the R testsuite, so it does not lead to a test failure.
comment:20 Changed 9 years ago by
- Description modified (diff)
comment:21 Changed 9 years ago by
Please delete the following from spkg-install
(needed for #11073):
# sage -f "$RPY_VER".spkg # EVIL # Sanity check, should never happen: if [ ! -x "$SAGE_LOCAL/bin/sage-spkg" ]; then echo "Error: Fatal dependency error:" echo " $SAGE_LOCAL/bin/sage-spkg" echp "does not (yet) exist (or isn't executable):" ls -l "$SAGE_LOCAL/bin/sage-spkg" echo "Please report this to one of the Google groups" echo " sage-devel or sage-release." echo "Error installing Rpy, which is part of the R spkg." exit 1 fi
(note also the typo "echp")
If sage-spkg
really doesn't exist, then we will detect that failure when doing
"$SAGE_ROOT/spkg/pipestatus" \ "sage-spkg -f '$RPY_PKG' 2>&1" \ "tee -a '$SAGE_ROOT/spkg/logs/'$RPY_VER.log"
below.
comment:22 Changed 9 years ago by
Done!
comment:23 Changed 9 years ago by
- Description modified (diff)
comment:24 follow-up: ↓ 25 Changed 9 years ago by
- Reviewers set to Karl-Dieter Crisman, Jeroen Demeyer
Jeroen, what else needs to be done for positive review?
comment:25 in reply to: ↑ 24 Changed 9 years ago by
Replying to kcrisman:
Jeroen, what else needs to be done for positive review?
I don't know, I haven't actually reviewed the ticket. I just tried to install it on various configurations.
comment:26 Changed 9 years ago by
- Cc jason added
comment:27 follow-up: ↓ 28 Changed 9 years ago by
- Component changed from statistics to packages
- Status changed from needs_review to positive_review
I haven't checked that it fixes the problems mentioned in the ticket, but apart from that: looks good. The spkg could do with some cleaning up (also, who's idea was it to put a spkg inside a spkg???), but I don't want to force this. So, positive_review.
comment:28 in reply to: ↑ 27 Changed 9 years ago by
Replying to jdemeyer:
I haven't checked that it fixes the problems mentioned in the ticket, but apart from that: looks good. The spkg could do with some cleaning up (also, who's idea was it to put a spkg inside a spkg???), but I don't want to force this. So, positive_review.
And there are plenty of other tickets open about those things in any case :)
comment:29 Changed 9 years ago by
- Merged in set to sage-4.8.alpha4
- Resolution set to fixed
- Status changed from positive_review to closed
comment:30 Changed 9 years ago by
With this new version, this doctest is problematic (line 352):
sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris
On my Mac OS X 10.6.8 box, doctesting this starts up X Windows. This is new: it didn't do this in Sage 4.8.alpha3.
- Any ideas why?
- I don't like this behavior: doctesting shouldn't start up external applications. What should be done about it?
comment:31 follow-up: ↓ 32 Changed 9 years ago by
I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")
? If yes, does
Sys.unsetenv("DISPLAY")
change anything?
comment:32 in reply to: ↑ 31 ; follow-up: ↓ 34 Changed 9 years ago by
Replying to vbraun:
I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in
capabilities("png")
?
Yes. It also reports TRUE
for this.
If yes, does
Sys.unsetenv("DISPLAY")
change anything?
Yes, X11 doesn't start, and it reports "False" for png:
> Sys.unsetenv("DISPLAY") > capabilities("png") png FALSE > capabilities("aqua") aqua TRUE
(I don't know why there are leading spaces in the returned string " png".)
comment:33 Changed 9 years ago by
Does R just start up X11 or does it also show anything in it? Can you strace the R execution and see where it executes X?
comment:34 in reply to: ↑ 32 Changed 9 years ago by
Replying to jhpalmieri:
Replying to vbraun:
I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in
capabilities("png")
?Yes. It also reports
TRUE
for this.
Wow, that is really unexpected. Also weird, because I tested the entire library overnight on my local 4.8.alpha4 (for another ticket) and nothing happened like that, or I would have noticed this morning.
new-host-2:sage-4.8.alpha4 karl-dietercrisman$ ./sage -t devel/sage/sage/interfaces/r.py sage -t "devel/sage/sage/interfaces/r.py" [13.6 s] ---------------------------------------------------------------------- All tests passed! Total time for all tests: 13.6 seconds
Of course, I also get
> capabilities("png") png FALSE
Your problem is that someone Sage got compiled with png support! It's not supposed to on Mac, we try to configure it to use (only) Aqua. Hence the doctest being a little unusual:
sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris sage: t = r.eval('capabilities("aqua")') # Should be on all supported Mac versions sage: "TRUE" in s+t # optional -- rgraphics True
My guess is you have something set in your preferences or elsewhere that caused R to pick up a system version of something. In general, getting R to compile on Mac with png support is interesting and hackish - I can't find the page I'm looking for right now, but it's referenced somewhere on one of the previous "get graphics in R in Sage to work" Trac tickets.
In other words, while not ideal, I wouldn't call this something to unmerge this new spkg for.
comment:35 follow-up: ↓ 37 Changed 9 years ago by
vbraun:
Does R just start up X11 or does it also show anything in it? Can you strace the R execution and see where it executes X?
X11 just starts up. It doesn't seem to have any windows running. I don't have strace installed on this computer, and I wouldn't know how to use it if I did ;)
By the way, if I log into an OS X box remotely, I don't have this problem. It must correctly detect that I can't run X11, and it also says that I don't have 'png' capabilities.
kcrisman: Do you have X11 installed on your Mac? Maybe if it's not installed, it doesn't go through this check and returns "FALSE" for png capabilities.
I remember seeing something about compiling R with png support on http://ask.sagemath.org/, but I don't remember beyond that. I don't think I have a particularly unusual setup, but who knows what R looks at? In the install log for R, I see this:
Interfaces supported: X11, aqua, tcltk External libraries: readline, ICU Additional capabilities: PNG, JPEG, NLS
Anyway, I'm not suggesting unmerging the spkg, but fixing the doctest somehow (in a followup ticket, if that's best). Can we do the equivalent in Sage of
Sys.unsetenv("DISPLAY") capabilities("png") capabilities("aqua")
without breaking things on Linux or Solaris?
comment:36 Changed 9 years ago by
r.eval('Sys.unsetenv("DISPLAY")')
when starting up the R pexpect interface?
In any case, I prefer that you make a new ticket and move the discussion there.
comment:37 in reply to: ↑ 35 Changed 9 years ago by
kcrisman: Do you have X11 installed on your Mac? Maybe if it's not installed, it doesn't go through this check and returns "FALSE" for png capabilities.
Yes, I do.
Anyway, I'm not suggesting unmerging the spkg, but fixing the doctest somehow (in a followup ticket, if that's best). Can we
Be sure to cc: me and Jason on that, then.
comment:38 Changed 9 years ago by
Followup ticket is #12172
comment:39 follow-up: ↓ 40 Changed 9 years ago by
On OS X 10.6.8 I cannot install this spkg in sage-4.8.alpha3 (I build the alpha3 from source so the old spkg does install).
Doing
md:sage-4.8.alpha3 maarten$ ./sage -f ../sage-4.8.alpha4/spkg/standard/r-2.14.0.p0.spkg
Hangs as described on: http://pastebin.com/7K7qLMJh
The guilty command is: /Users/maarten/Downloads//sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/bin/exec/R --no-restore --slave --args nextArg--pkglocknextArg--data-compress=xznextArg--byte-compilenextArg-lnextArg../../../librarynextArgMASS.tgz
comment:40 in reply to: ↑ 39 Changed 9 years ago by
Replying to mderickx:
On OS X 10.6.8 I cannot install this spkg in sage-4.8.alpha3 (I build the alpha3 from source so the old spkg does install).
Questions:
- Do you have any unusual environment variables set?
- Do you have any R env. variables, or another R installation separate from this one?
- Is there anything unusual you have on your machine (I am currently on the exact same OS version, w/out problems) regarding compiling?
Usually we've found that R problems are related to it picking up system versions or variables.
I'd recommend we open a new ticket for this, though, since "most" OS X 10.6 systems seem to be just fine.
comment:41 Changed 9 years ago by
On the new ticket it would also be useful to attach dtrace to the hanging R process to see what it is doing.
comment:42 Changed 9 years ago by
Ok, I created #12201 . I try to answer your questions there.
This ticket obsoletes #8868