Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

#12057 closed enhancement (fixed)

Upgrade R (r-project.org)

Reported by: vbraun Owned by: amhou
Priority: critical Milestone: sage-4.8
Component: packages: standard Keywords: R spkg r-project graphics PNG X11
Cc: leif, jdemeyer, jason Merged in: sage-4.8.alpha4
Authors: Volker Braun Reviewers: Karl-Dieter Crisman, Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

The current version of R (r-project) disables graphics if no X11 is available even though r.capabilities() states that PNG is enabled. But you can't generate PNGs, which is why R graphics is broken in the Sage virtual machine. This is fixed in the newest upstream version, of which I made a new spkg.

There is no other change except for the source code and a rediffed patches/R.sh.in

On an unrelated note, it would be nice if tickets talking about R would contain "r-project" on trac since it is otherwise hopeless to search for.

Apply:

  1. spkg http://www.stp.dias.ie/~vbraun/Sage/spkg/r-2.14.0.p0.spkg
  2. trac_12057_fix_doctests.patch

The upgraded spkg also fixes #12065.

Attachments (1)

trac_12057_fix_doctests.patch (7.6 KB) - added by vbraun 8 years ago.
Minor docstring fix.

Download all attachments as: .zip

Change History (43)

comment:1 Changed 8 years ago by vbraun

  • Description modified (diff)
  • Status changed from new to needs_review
  • Type changed from PLEASE CHANGE to defect

comment:2 Changed 8 years ago by vbraun

This ticket obsoletes #8868

comment:3 Changed 8 years ago by kcrisman

  • Cc leif added

Hmm, I don't mind this in theory. I'd want to test it out a little. Can you point to one of the R announcements where they end up fixing this? They have very detailed and thorough announcements, which I usually cannot make it all the way through when searching for this sort of thing.

Also, does this mean we don't need libpango or cairo or all that stuff? Or would we still need that?

Some relevant things.

  • Re: #8868 - One concern is whether it always makes graphics work - OS X, Linux, Solaris. Otherwise I wouldn't want to re-enable the doctests, which are currently optional. If they work, then this ticket should enable them.
  • Great idea on the r-project. I always put R things that are not about statistics per se in the "packages" component. I'm updating others now.
  • Leif has put a lot of work in separating rpy from R at #9906. I'm not sure what should have priority, though, given that we are fairly behind the times and reviewing #9906 is a boatload of work.

Here are a bunch of other relevant tickets which perhaps (?) should be addressed here, or at least mentioned.

  • #7865 R on 64-bit Solaris C and Fortran mixing
  • #7664 R and Latex
  • #9668 hardcoding paths in R binary
  • #11970 readline issues on Oneric Ocelot or something
  • #10967 more issues with the script
  • #11249 R graphics occasionally living in weird places

comment:4 Changed 8 years ago by vbraun

  • Cc jdemeyer added

You still need cairo+pango+fontconfig+fonts; R doesn't include fonts or a font renderer. But it works now in the virtual machine without X installed. I don't have a reference for the release announcement where it was fixed.

Given that we get relatively frequent requests for R graphics in the virtual machine I would prefer it if we could get this ticket reviewed asap. Really all there is to do is regression testing, and the easiest way to do so is to run it on the build bot.

comment:5 in reply to: ↑ description Changed 8 years ago by leif

  • Keywords spkg PNG X11 added
  • Type changed from defect to enhancement

Replying to vbraun:

On an unrelated note, it would be nice if tickets talking about R would contain "r-project" on trac since it is otherwise hopeless to search for.

You can also search trac for "R" with Google.


I intentionally did not (yet) update R or RPy on #9906 or #9668. #11970 is rather unrelated to R itself; it fixes the readline spkg. #10967 could in principle be closed as invalid, as the R script in $SAGE_LOCAL/bin/ is not intended to be run directly; one should instead use sage --R ... (although adding a sanity check to the script doesn't hurt, which I did or intended to do on #9668, which depends on #9906...).

comment:6 follow-up: Changed 8 years ago by vbraun

  • Priority changed from major to critical

Can we get the updated R spkg into Sage before fixing the laundry list of unrelated issues? None of the tickets leif mentioned is close to resolution. This ticket is ready to be reviewed and included right now, and I think a large group of users will benefit from being able to use graphics in the virtual machine.

comment:7 in reply to: ↑ 6 Changed 8 years ago by kcrisman

Replying to vbraun:

Can we get the updated R spkg into Sage before fixing the laundry list of unrelated issues? None of the tickets leif mentioned is close to resolution. This ticket is ready to be reviewed and included right now, and I think a large group of users will benefit from being able to use graphics in the virtual machine.

I think that given Leif's lukewarmness on this and Volker's adamance, along with my own natural inclination, let's go for it. My guess is that we may not even need to backport everything for the others... ?

Testing on OSX 10.4 PPC now :)

Did you notice whether there were any needed patches in a -t -long situation?

comment:8 Changed 8 years ago by vbraun

  • Description modified (diff)

Patch attached for some small changes in sage/interfaces/r.py. Highlights are that R now returns the class as ordered list and that it doesn't accept the somewhat spurious 'source(file=file(...))' construct that we feed it.

comment:9 follow-up: Changed 8 years ago by kcrisman

Questions:

  • Won't
    return 'file=file("%s",open="r")\nsource(file)'%filename 
    
    cause problems if someone wants to use the file command again, or is that local in scope?
  • Also, might as well add to reference manual if it's now all formatted correctly? Should just be one line adding it to the toc or something...
  • Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?

Once I'm done testing it installs I'll be more formal in looking at this. What a great thing to do, thank you much!

comment:10 follow-up: Changed 8 years ago by jdemeyer

  • Status changed from needs_review to needs_work

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

comment:11 in reply to: ↑ 9 ; follow-up: Changed 8 years ago by vbraun

Replying to kcrisman:

  • Won't [...] cause problems if someone wants to use the file command again, or is that local in scope?

Functions can't be lvalues in R, so its fine though maybe not beautiful:

> file=123
> file
[1] 123
> file('test.txt')
description       class        mode        text      opened    can read 
 "test.txt"      "file"         "r"      "text"    "closed"       "yes" 
  can write 
      "yes" 
> file
[1] 123
  • Also, might as well add to reference manual if it's now all formatted correctly? Should just be one line adding it to the toc or something...

I only patched up the sphinx breakage where I fixed things. There are still 2kloc to go, but I didn't want to break each and every patch for sage/interfaces/r.py on trac...

  • Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?

You just have to recompile the new R in my virtual machine and it'll basically work. You need to install fonts, though, otherwise all text in the plots will be emtpy squares.

Changed 8 years ago by vbraun

Minor docstring fix.

comment:12 in reply to: ↑ 10 ; follow-up: Changed 8 years ago by vbraun

Replying to jdemeyer:

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

I just built on sage.math and it worked for me (20x parallel). I'm using sage-4.8.alpha2, though. Did you merge anything in alpha3 that might cause this?

comment:13 in reply to: ↑ 12 Changed 8 years ago by jdemeyer

Replying to vbraun:

Replying to jdemeyer:

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

I just built on sage.math and it worked for me (20x parallel). I'm using sage-4.8.alpha2, though.

Well, it could be some strange error happening only sporadically. Or, an error which happens only if Sage is built from scratch (as opposed to installing the spkg in an existing Sage build). I will try again later.

Did you merge anything in alpha3 that might cause this?

I don't think so.

comment:14 Changed 8 years ago by jhpalmieri

For what it's worth, I took sage-4.8.alpha2, replaced the R spkg with this one, and merged the Sage library patch. Then I built from scratch (on sage.math) and all tests passed. I'm also trying on OS X 10.6.8: the build has succeeded, and doctests are still going.

comment:15 in reply to: ↑ 11 Changed 8 years ago by kcrisman

That answered most of my questions.

  • Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?

You just have to recompile the new R in my virtual machine and it'll basically work. You need to install fonts, though, otherwise all text in the plots will be emtpy squares.

Ok, I'll try that.

Other comments:

  • I have no problems on OS X 10.4 PPC with this or the patch. Passes relevant long optional tests. That's good!
  • Has anyone tested this on Solaris? Just because sometimes there are config options which change there which have caused problems in the past.
  • I would suggest that perhaps this should then remove some of the bizarre stuff in spkg-install that checks for the X headers, etc. and only for this reason. Though I suppose that could still be in another ticket, since that is for X support, and apparently the pngs still appear? So maybe forget this...
  • This typo probably should be fixed in SPKG.txt, though:
    === r-2.14.0.p0 (Volker Braun, 19th November 2011) ===
     * #12057: upgrade to new upstram version
    

comment:16 follow-up: Changed 8 years ago by vbraun

  • Status changed from needs_work to needs_review
  • I tested it on mark.skynet (sparc solaris) and it works fine
  • I don't want to rewrite the spkg-install right now. This should be done when rpy is separated out.
  • Fixed the typo.

Jeroen, did you figure out which problem you encountered on sage.math? It seems to be working for everyone else...

comment:17 in reply to: ↑ 16 Changed 8 years ago by jdemeyer

Replying to vbraun:

  • I tested it on mark.skynet (sparc solaris) and it works fine
  • I don't want to rewrite the spkg-install right now. This should be done when rpy is separated out.
  • Fixed the typo.

Jeroen, did you figure out which problem you encountered on sage.math? It seems to be working for everyone else...

I managed to compile R on the second attempt. So it certainly sometimes works. Could be a race condition.

comment:18 Changed 8 years ago by kcrisman

Did anyone install this with SAGE_CHECK=yes? On PPC OSX 10.4 I get everything ok until "running regression tests" in which reg-tests-1a apparently gets an error. I have not had time to check this on modern Mac or on Linux (sage.math) yet, just saw this when I walked in the office this morning.

comment:19 Changed 8 years ago by vbraun

For me (Linux x86_64), the reg_tests_1a passes. The testsuite errors out a little bit later on in

running tests of Internet and socket functions
  expect some differences
make[3]: Entering directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests'
running code in 'internet.R' ...make[3]: *** [internet.Rout] Error 1
make[3]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests'
make[2]: [test-Internet] Error 2 (ignored)
make[2]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests'
make[1]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests'
Now cleaning up tmp files.
Making Sage/Python scripts relocatable...
Making script relocatable
Finished installing r-2.14.0.p0.spkg

probably because I'm behind a firewall. But it seems like they anticipated that and the error is caught in the R testsuite, so it does not lead to a test failure.

comment:20 Changed 8 years ago by vbraun

  • Description modified (diff)

comment:21 Changed 8 years ago by jdemeyer

Please delete the following from spkg-install (needed for #11073):

# sage -f "$RPY_VER".spkg # EVIL

# Sanity check, should never happen:
if [ ! -x "$SAGE_LOCAL/bin/sage-spkg" ]; then
    echo "Error: Fatal dependency error:"
    echo "    $SAGE_LOCAL/bin/sage-spkg"
    echp "does not (yet) exist (or isn't executable):"
    ls -l "$SAGE_LOCAL/bin/sage-spkg"
    echo "Please report this to one of the Google groups"
    echo "    sage-devel or sage-release."
    echo "Error installing Rpy, which is part of the R spkg."
    exit 1
fi

(note also the typo "echp")

If sage-spkg really doesn't exist, then we will detect that failure when doing

"$SAGE_ROOT/spkg/pipestatus" \
    "sage-spkg -f '$RPY_PKG' 2>&1" \
    "tee -a '$SAGE_ROOT/spkg/logs/'$RPY_VER.log"

below.

comment:22 Changed 8 years ago by vbraun

Done!

comment:23 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:24 follow-up: Changed 8 years ago by kcrisman

  • Reviewers set to Karl-Dieter Crisman, Jeroen Demeyer

Jeroen, what else needs to be done for positive review?

comment:25 in reply to: ↑ 24 Changed 8 years ago by jdemeyer

Replying to kcrisman:

Jeroen, what else needs to be done for positive review?

I don't know, I haven't actually reviewed the ticket. I just tried to install it on various configurations.

comment:26 Changed 8 years ago by jason

  • Cc jason added

comment:27 follow-up: Changed 8 years ago by jdemeyer

  • Component changed from statistics to packages
  • Status changed from needs_review to positive_review

I haven't checked that it fixes the problems mentioned in the ticket, but apart from that: looks good. The spkg could do with some cleaning up (also, who's idea was it to put a spkg inside a spkg???), but I don't want to force this. So, positive_review.

comment:28 in reply to: ↑ 27 Changed 8 years ago by kcrisman

Replying to jdemeyer:

I haven't checked that it fixes the problems mentioned in the ticket, but apart from that: looks good. The spkg could do with some cleaning up (also, who's idea was it to put a spkg inside a spkg???), but I don't want to force this. So, positive_review.

And there are plenty of other tickets open about those things in any case :)

comment:29 Changed 8 years ago by jdemeyer

  • Merged in set to sage-4.8.alpha4
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:30 Changed 8 years ago by jhpalmieri

With this new version, this doctest is problematic (line 352):

    sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris

On my Mac OS X 10.6.8 box, doctesting this starts up X Windows. This is new: it didn't do this in Sage 4.8.alpha3.

  • Any ideas why?
  • I don't like this behavior: doctesting shouldn't start up external applications. What should be done about it?

comment:31 follow-up: Changed 8 years ago by vbraun

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")? If yes, does

Sys.unsetenv("DISPLAY")

change anything?

comment:32 in reply to: ↑ 31 ; follow-up: Changed 8 years ago by jhpalmieri

Replying to vbraun:

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")?

Yes. It also reports TRUE for this.

If yes, does

Sys.unsetenv("DISPLAY")

change anything?

Yes, X11 doesn't start, and it reports "False" for png:

> Sys.unsetenv("DISPLAY") 
> capabilities("png")
  png 
FALSE 
> capabilities("aqua")
aqua 
TRUE 

(I don't know why there are leading spaces in the returned string " png".)

comment:33 Changed 8 years ago by vbraun

Does R just start up X11 or does it also show anything in it? Can you strace the R execution and see where it executes X?

comment:34 in reply to: ↑ 32 Changed 8 years ago by kcrisman

Replying to jhpalmieri:

Replying to vbraun:

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")?

Yes. It also reports TRUE for this.

Wow, that is really unexpected. Also weird, because I tested the entire library overnight on my local 4.8.alpha4 (for another ticket) and nothing happened like that, or I would have noticed this morning.

new-host-2:sage-4.8.alpha4 karl-dietercrisman$ ./sage -t devel/sage/sage/interfaces/r.py 
sage -t  "devel/sage/sage/interfaces/r.py"                  
	 [13.6 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 13.6 seconds

Of course, I also get

> capabilities("png")
  png 
FALSE 

Your problem is that someone Sage got compiled with png support! It's not supposed to on Mac, we try to configure it to use (only) Aqua. Hence the doctest being a little unusual:

            sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris
            sage: t = r.eval('capabilities("aqua")') # Should be on all supported Mac versions
            sage: "TRUE" in s+t                      # optional -- rgraphics
            True

My guess is you have something set in your preferences or elsewhere that caused R to pick up a system version of something. In general, getting R to compile on Mac with png support is interesting and hackish - I can't find the page I'm looking for right now, but it's referenced somewhere on one of the previous "get graphics in R in Sage to work" Trac tickets.

In other words, while not ideal, I wouldn't call this something to unmerge this new spkg for.

comment:35 follow-up: Changed 8 years ago by jhpalmieri

vbraun:

Does R just start up X11 or does it also show anything in it? Can you strace the R execution and see where it executes X?

X11 just starts up. It doesn't seem to have any windows running. I don't have strace installed on this computer, and I wouldn't know how to use it if I did ;)

By the way, if I log into an OS X box remotely, I don't have this problem. It must correctly detect that I can't run X11, and it also says that I don't have 'png' capabilities.

kcrisman: Do you have X11 installed on your Mac? Maybe if it's not installed, it doesn't go through this check and returns "FALSE" for png capabilities.

I remember seeing something about compiling R with png support on http://ask.sagemath.org/, but I don't remember beyond that. I don't think I have a particularly unusual setup, but who knows what R looks at? In the install log for R, I see this:

  Interfaces supported:      X11, aqua, tcltk
  External libraries:        readline, ICU
  Additional capabilities:   PNG, JPEG, NLS

Anyway, I'm not suggesting unmerging the spkg, but fixing the doctest somehow (in a followup ticket, if that's best). Can we do the equivalent in Sage of

Sys.unsetenv("DISPLAY") 
capabilities("png")
capabilities("aqua")

without breaking things on Linux or Solaris?

comment:36 Changed 8 years ago by jdemeyer

r.eval('Sys.unsetenv("DISPLAY")')

when starting up the R pexpect interface?

In any case, I prefer that you make a new ticket and move the discussion there.

comment:37 in reply to: ↑ 35 Changed 8 years ago by kcrisman

kcrisman: Do you have X11 installed on your Mac? Maybe if it's not installed, it doesn't go through this check and returns "FALSE" for png capabilities.

Yes, I do.

Anyway, I'm not suggesting unmerging the spkg, but fixing the doctest somehow (in a followup ticket, if that's best). Can we

Be sure to cc: me and Jason on that, then.

comment:38 Changed 8 years ago by vbraun

Followup ticket is #12172

comment:39 follow-up: Changed 8 years ago by mderickx

On OS X 10.6.8 I cannot install this spkg in sage-4.8.alpha3 (I build the alpha3 from source so the old spkg does install).

Doing

md:sage-4.8.alpha3 maarten$ ./sage -f ../sage-4.8.alpha4/spkg/standard/r-2.14.0.p0.spkg

Hangs as described on: http://pastebin.com/7K7qLMJh

The guilty command is: /Users/maarten/Downloads//sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/bin/exec/R --no-restore --slave --args nextArg--pkglocknextArg--data-compress=xznextArg--byte-compilenextArg-lnextArg../../../librarynextArgMASS.tgz

comment:40 in reply to: ↑ 39 Changed 8 years ago by kcrisman

Replying to mderickx:

On OS X 10.6.8 I cannot install this spkg in sage-4.8.alpha3 (I build the alpha3 from source so the old spkg does install).

Questions:

  • Do you have any unusual environment variables set?
  • Do you have any R env. variables, or another R installation separate from this one?
  • Is there anything unusual you have on your machine (I am currently on the exact same OS version, w/out problems) regarding compiling?

Usually we've found that R problems are related to it picking up system versions or variables.

I'd recommend we open a new ticket for this, though, since "most" OS X 10.6 systems seem to be just fine.

comment:41 Changed 8 years ago by vbraun

On the new ticket it would also be useful to attach dtrace to the hanging R process to see what it is doing.

comment:42 Changed 8 years ago by mderickx

Ok, I created #12201 . I try to answer your questions there.

Note: See TracTickets for help on using tickets.