Upgrade R (r-project.org)

[vbraun]

The current version of R (r-project) disables graphics if no X11 is available even though r.capabilities() states that PNG is enabled. But you can't generate PNGs, which is why R graphics is broken in the Sage virtual machine. This is fixed in the newest upstream version, of which I made a new spkg.

There is no other change except for the source code and a rediffed patches/R.sh.in

On an unrelated note, it would be nice if tickets talking about R would contain "r-project" on trac since it is otherwise hopeless to search for.

Apply:

1. spkg ​http://www.stp.dias.ie/~vbraun/Sage/spkg/r-2.14.0.p0.spkg
2. trac_12057_fix_doctests.patch​

The upgraded spkg also fixes #12065.

[vbraun]

This ticket obsoletes #8868

[kcrisman]

Hmm, I don't mind this in theory. I'd want to test it out a little. Can you point to one of the R announcements where they end up fixing this? They have very detailed and thorough announcements, which I usually cannot make it all the way through when searching for this sort of thing.

Also, does this mean we don't need libpango or cairo or all that stuff? Or would we still need that?

Some relevant things.

• Re: #8868 - One concern is whether it always makes graphics work - OS X, Linux, Solaris. Otherwise I wouldn't want to re-enable the doctests, which are currently optional. If they work, then this ticket should enable them.
• Great idea on the r-project. I always put R things that are not about statistics per se in the "packages" component. I'm updating others now.
• Leif has put a lot of work in separating rpy from R at #9906. I'm not sure what should have priority, though, given that we are fairly behind the times and reviewing #9906 is a boatload of work.

Here are a bunch of other relevant tickets which perhaps (?) should be addressed here, or at least mentioned.

• #7865 R on 64-bit Solaris C and Fortran mixing
• #7664 R and Latex
• #9668 hardcoding paths in R binary
• #11970 readline issues on Oneric Ocelot or something
• #10967 more issues with the script
• #11249 R graphics occasionally living in weird places

[vbraun]

You still need cairo+pango+fontconfig+fonts; R doesn't include fonts or a font renderer. But it works now in the virtual machine without X installed. I don't have a reference for the release announcement where it was fixed.

Given that we get relatively frequent requests for R graphics in the virtual machine I would prefer it if we could get this ticket reviewed asap. Really all there is to do is regression testing, and the easiest way to do so is to run it on the build bot.

[leif]

Replying to vbraun:

On an unrelated note, it would be nice if tickets talking about R would contain "r-project" on trac since it is otherwise hopeless to search for.

You can also ​search trac for "R" with Google.

I intentionally did not (yet) update R or RPy on #9906 or #9668. #11970 is rather unrelated to R itself; it fixes the readline spkg. #10967 could in principle be closed as invalid, as the R script in $SAGE_LOCAL/bin/ is not intended to be run directly; one should instead use sage --R ... (although adding a sanity check to the script doesn't hurt, which I did or intended to do on #9668, which depends on #9906...).

[vbraun]

Can we get the updated R spkg into Sage before fixing the laundry list of unrelated issues? None of the tickets leif mentioned is close to resolution. This ticket is ready to be reviewed and included right now, and I think a large group of users will benefit from being able to use graphics in the virtual machine.

[kcrisman]

Replying to vbraun:

Can we get the updated R spkg into Sage before fixing the laundry list of unrelated issues? None of the tickets leif mentioned is close to resolution. This ticket is ready to be reviewed and included right now, and I think a large group of users will benefit from being able to use graphics in the virtual machine.

I think that given Leif's lukewarmness on this and Volker's adamance, along with my own natural inclination, let's go for it. My guess is that we may not even need to backport everything for the others... ?

Testing on OSX 10.4 PPC now :)

Did you notice whether there were any needed patches in a -t -long situation?

[vbraun]

Patch attached for some small changes in sage/interfaces/r.py. Highlights are that R now returns the class as ordered list and that it doesn't accept the somewhat spurious 'source(file=file(...))' construct that we feed it.

[kcrisman]

Questions:

• Won't

  return 'file=file("%s",open="r")\nsource(file)'%filename 
  

  cause problems if someone wants to use the file command again, or is that local in scope?
• Also, might as well add to reference manual if it's now all formatted correctly? Should just be one line adding it to the toc or something...
• Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?

Once I'm done testing it installs I'll be more formal in looking at this. What a great thing to do, thank you much!

[jdemeyer]

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

[vbraun]

Replying to kcrisman:

• Won't [...] cause problems if someone wants to use the file command again, or is that local in scope?

Functions can't be lvalues in R, so its fine though maybe not beautiful:

> file=123
> file
[1] 123
> file('test.txt')
description       class        mode        text      opened    can read 
 "test.txt"      "file"         "r"      "text"    "closed"       "yes" 
  can write 
      "yes" 
> file
[1] 123

• Also, might as well add to reference manual if it's now all formatted correctly? Should just be one line adding it to the toc or something...

I only patched up the sphinx breakage where I fixed things. There are still 2kloc to go, but I didn't want to break each and every patch for sage/interfaces/r.py on trac...

• Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?

You just have to recompile the new R in my virtual machine and it'll basically work. You need to install fonts, though, otherwise all text in the plots will be emtpy squares.

[vbraun]

Minor docstring fix.

[vbraun]

Replying to jdemeyer:

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

I just built on sage.math and it worked for me (20x parallel). I'm using sage-4.8.alpha2, though. Did you merge anything in alpha3 that might cause this?

[jdemeyer]

Replying to vbraun:

Replying to jdemeyer:

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

I just built on sage.math and it worked for me (20x parallel). I'm using sage-4.8.alpha2, though.

Well, it could be some strange error happening only sporadically. Or, an error which happens only if Sage is built from scratch (as opposed to installing the spkg in an existing Sage build). I will try again later.

Did you merge anything in alpha3 that might cause this?

I don't think so.

[jhpalmieri]

For what it's worth, I took sage-4.8.alpha2, replaced the R spkg with this one, and merged the Sage library patch. Then I built from scratch (on sage.math) and all tests passed. I'm also trying on OS X 10.6.8: the build has succeeded, and doctests are still going.

[kcrisman]

That answered most of my questions.

• Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?

You just have to recompile the new R in my virtual machine and it'll basically work. You need to install fonts, though, otherwise all text in the plots will be emtpy squares.

Ok, I'll try that.

Other comments:

• I have no problems on OS X 10.4 PPC with this or the patch. Passes relevant long optional tests. That's good!
• Has anyone tested this on Solaris? Just because sometimes there are config options which change there which have caused problems in the past.
• I would suggest that perhaps this should then remove some of the bizarre stuff in spkg-install that checks for the X headers, etc. and only for this reason. Though I suppose that could still be in another ticket, since that is for X support, and apparently the pngs still appear? So maybe forget this...
• This typo probably should be fixed in SPKG.txt, though:

  === r-2.14.0.p0 (Volker Braun, 19th November 2011) ===
   * #12057: upgrade to new upstram version
  

[vbraun]

• I tested it on mark.skynet (sparc solaris) and it works fine
• I don't want to rewrite the spkg-install right now. This should be done when rpy is separated out.
• Fixed the typo.

Jeroen, did you figure out which problem you encountered on sage.math? It seems to be working for everyone else...

[jdemeyer]

Replying to vbraun:

• I tested it on mark.skynet (sparc solaris) and it works fine
• I don't want to rewrite the spkg-install right now. This should be done when rpy is separated out.
• Fixed the typo.

Jeroen, did you figure out which problem you encountered on sage.math? It seems to be working for everyone else...

I managed to compile R on the second attempt. So it certainly sometimes works. Could be a race condition.

[kcrisman]

Did anyone install this with SAGE_CHECK=yes? On PPC OSX 10.4 I get everything ok until "running regression tests" in which reg-tests-1a apparently gets an error. I have not had time to check this on modern Mac or on Linux (sage.math) yet, just saw this when I walked in the office this morning.

[vbraun]

For me (Linux x86_64), the reg_tests_1a passes. The testsuite errors out a little bit later on in

running tests of Internet and socket functions
  expect some differences
make[3]: Entering directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests'
running code in 'internet.R' ...make[3]: *** [internet.Rout] Error 1
make[3]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests'
make[2]: [test-Internet] Error 2 (ignored)
make[2]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests'
make[1]: Leaving directory `/home/vbraun/opt/sage-4.8.alpha2/spkg/build/r-2.14.0.p0/src/tests'
Now cleaning up tmp files.
Making Sage/Python scripts relocatable...
Making script relocatable
Finished installing r-2.14.0.p0.spkg

probably because I'm behind a firewall. But it seems like they anticipated that and the error is caught in the R testsuite, so it does not lead to a test failure.

[jdemeyer]

Please delete the following from spkg-install (needed for #11073):

# sage -f "$RPY_VER".spkg # EVIL

# Sanity check, should never happen:
if [ ! -x "$SAGE_LOCAL/bin/sage-spkg" ]; then
    echo "Error: Fatal dependency error:"
    echo "    $SAGE_LOCAL/bin/sage-spkg"
    echp "does not (yet) exist (or isn't executable):"
    ls -l "$SAGE_LOCAL/bin/sage-spkg"
    echo "Please report this to one of the Google groups"
    echo "    sage-devel or sage-release."
    echo "Error installing Rpy, which is part of the R spkg."
    exit 1
fi

(note also the typo "echp")

If sage-spkg really doesn't exist, then we will detect that failure when doing

"$SAGE_ROOT/spkg/pipestatus" \
    "sage-spkg -f '$RPY_PKG' 2>&1" \
    "tee -a '$SAGE_ROOT/spkg/logs/'$RPY_VER.log"

below.

[vbraun]

Done!

[kcrisman]

Jeroen, what else needs to be done for positive review?

[jdemeyer]

Replying to kcrisman:

Jeroen, what else needs to be done for positive review?

I don't know, I haven't actually reviewed the ticket. I just tried to install it on various configurations.

[jdemeyer]

I haven't checked that it fixes the problems mentioned in the ticket, but apart from that: looks good. The spkg could do with some cleaning up (also, who's idea was it to put a spkg inside a spkg???), but I don't want to force this. So, positive_review.

[kcrisman]

Replying to jdemeyer:

I haven't checked that it fixes the problems mentioned in the ticket, but apart from that: looks good. The spkg could do with some cleaning up (also, who's idea was it to put a spkg inside a spkg???), but I don't want to force this. So, positive_review.

And there are plenty of other tickets open about those things in any case :)

[jhpalmieri]

With this new version, this doctest is problematic (line 352):

    sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris

On my Mac OS X 10.6.8 box, doctesting this starts up X Windows. This is new: it didn't do this in Sage 4.8.alpha3.

• Any ideas why?
• I don't like this behavior: doctesting shouldn't start up external applications. What should be done about it?

[vbraun]

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")? If yes, does

Sys.unsetenv("DISPLAY")

change anything?

[jhpalmieri]

Replying to vbraun:

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")?

Yes. It also reports TRUE for this.

If yes, does

Sys.unsetenv("DISPLAY")

change anything?

Yes, X11 doesn't start, and it reports "False" for png:

> Sys.unsetenv("DISPLAY") 
> capabilities("png")
  png 
FALSE 
> capabilities("aqua")
aqua 
TRUE 

(I don't know why there are leading spaces in the returned string " png".)

[vbraun]

Does R just start up X11 or does it also show anything in it? Can you strace the R execution and see where it executes X?

[kcrisman]

Replying to jhpalmieri:

Replying to vbraun:

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")?

Yes. It also reports TRUE for this.

Wow, that is really unexpected. Also weird, because I tested the entire library overnight on my local 4.8.alpha4 (for another ticket) and nothing happened like that, or I would have noticed this morning.

new-host-2:sage-4.8.alpha4 karl-dietercrisman$ ./sage -t devel/sage/sage/interfaces/r.py 
sage -t  "devel/sage/sage/interfaces/r.py"                  
	 [13.6 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 13.6 seconds

Of course, I also get

> capabilities("png")
  png 
FALSE 

Your problem is that someone Sage got compiled with png support! It's not supposed to on Mac, we try to configure it to use (only) Aqua. Hence the doctest being a little unusual:

            sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris
            sage: t = r.eval('capabilities("aqua")') # Should be on all supported Mac versions
            sage: "TRUE" in s+t                      # optional -- rgraphics
            True

My guess is you have something set in your preferences or elsewhere that caused R to pick up a system version of something. In general, getting R to compile on Mac with png support is interesting and hackish - I can't find the page I'm looking for right now, but it's referenced somewhere on one of the previous "get graphics in R in Sage to work" Trac tickets.

In other words, while not ideal, I wouldn't call this something to unmerge this new spkg for.

[jhpalmieri]

vbraun:

Does R just start up X11 or does it also show anything in it? Can you strace the R execution and see where it executes X?

X11 just starts up. It doesn't seem to have any windows running. I don't have strace installed on this computer, and I wouldn't know how to use it if I did ;)

By the way, if I log into an OS X box remotely, I don't have this problem. It must correctly detect that I can't run X11, and it also says that I don't have 'png' capabilities.

kcrisman: Do you have X11 installed on your Mac? Maybe if it's not installed, it doesn't go through this check and returns "FALSE" for png capabilities.

I remember seeing something about compiling R with png support on ​http://ask.sagemath.org/, but I don't remember beyond that. I don't think I have a particularly unusual setup, but who knows what R looks at? In the install log for R, I see this:

  Interfaces supported:      X11, aqua, tcltk
  External libraries:        readline, ICU
  Additional capabilities:   PNG, JPEG, NLS

Anyway, I'm not suggesting unmerging the spkg, but fixing the doctest somehow (in a followup ticket, if that's best). Can we do the equivalent in Sage of

Sys.unsetenv("DISPLAY") 
capabilities("png")
capabilities("aqua")

without breaking things on Linux or Solaris?

[jdemeyer]

r.eval('Sys.unsetenv("DISPLAY")')

when starting up the R pexpect interface?

In any case, I prefer that you make a new ticket and move the discussion there.

[kcrisman]

kcrisman: Do you have X11 installed on your Mac? Maybe if it's not installed, it doesn't go through this check and returns "FALSE" for png capabilities.

Yes, I do.

Anyway, I'm not suggesting unmerging the spkg, but fixing the doctest somehow (in a followup ticket, if that's best). Can we

Be sure to cc: me and Jason on that, then.

[vbraun]

Followup ticket is #12172

[mderickx]

On OS X 10.6.8 I cannot install this spkg in sage-4.8.alpha3 (I build the alpha3 from source so the old spkg does install).

Doing

md:sage-4.8.alpha3 maarten$ ./sage -f ../sage-4.8.alpha4/spkg/standard/r-2.14.0.p0.spkg

Hangs as described on: ​http://pastebin.com/7K7qLMJh

The guilty command is: /Users/maarten/Downloads//sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/bin/exec/R --no-restore --slave --args nextArg--pkglocknextArg--data-compress=xznextArg--byte-compilenextArg-lnextArg../../../librarynextArgMASS.tgz

[kcrisman]

Replying to mderickx:

On OS X 10.6.8 I cannot install this spkg in sage-4.8.alpha3 (I build the alpha3 from source so the old spkg does install).

Questions:

• Do you have any unusual environment variables set?
• Do you have any R env. variables, or another R installation separate from this one?
• Is there anything unusual you have on your machine (I am currently on the exact same OS version, w/out problems) regarding compiling?

Usually we've found that R problems are related to it picking up system versions or variables.

I'd recommend we open a new ticket for this, though, since "most" OS X 10.6 systems seem to be just fine.

[vbraun]

On the new ticket it would also be useful to attach dtrace to the hanging R process to see what it is doing.

[mderickx]

Ok, I created #12201 . I try to answer your questions there.