Upgrade R to version 3.0.2

[Emmanuel Charpentier]

New spkg: ​http://boxen.math.washington.edu/home/jpflori/spkg/r-3.0.2.p0.spkg

r-3.0.2.p0 (Jean-Pierre Flori, 30 October 2013)

• #14706: update to 3.0.2.
• Remove sqrtl.patch which is now upstream.
• Move old install on OS X.
• Fix visibility of symbols in built-in libintl.

r-3.0.1.p1 (Jean-Pierre Flori, 10 June 2013)

• #14706: Use --disable-long-double on Cygwin rather than patching.
• Add sqrtl.patch from upstream to make --disable-long-double work.

r-3.0.1.p0 (Emmanuel Charpentier, 8 June 2013)

• #14706: Drop-in replacement of upstream sources.

---

Apply: trac_14706-version.patch​

[Leif Leonhardy]

For the record: Removing -L$SAGE_LOCAL/lib from LDFLAGS (#13443) recently broke building R on CentOS, cf. ​http://trac.sagemath.org/sage_trac/ticket/13443#comment:12 and sage-devel.

[Jeroen Demeyer]

Replying to charpent:

This ticket is a place where I plan to send new updates to the R version used in Sage (R is, in principle, updated every six months). So I intend to open, close, reopen... as long as newer versions do not entail any substantial modification beyond dropping new R source in place and fixing the expected version numbers in the r.py doctest. Any further problems should be reported in specific tickets. As a new R version arrives (and Real Life (TM) allowing), I'll start from the last accepted patch for this spkg...

This is routine work (i. e. even I can do it:-), but using an up-to-date R interpreter is a sine qua non for reporting problems to R core, so it probably has to be done for Sage users to get answers from R core.

I honestly don't understand anything of this ticket description. What is this ticket about???

[Leif Leonhardy]

Replying to jdemeyer:

I honestly don't understand anything of this ticket description. What is this ticket about???

Atm upgrading R to 3.0.1 it seems...

[Emmanuel Charpentier]

• Status changed from new to needs_review

1) New spkg : available on Google drive (​https://docs.google.com/file/d/0B1gfn4_V_wm3clctVFFhUnlINVU/edit?usp=sharing)

2) What this ticket is about :

a) Having an up-to-date R is a sine qua non to get answers from R Core.

b) R in Sage is rarely up to date (more talented Sage developers work on more important issues).

c) Therefore, R-in-Sage users have trouble communicating with R Core

d) I am able to create "drop-in replacements" of the R spkg, thus giving R-in-Sage users an up-to-date R, thus allowing them to get answers from R Core...

e) Since this is routine work that few people seem to tackle, and since it is in my limited ability range, I'll try to do it after R upstream upgrades.

Is that clearer ?

[Jeroen Demeyer]

Replying to charpent:

Is that clearer ?

Why not just say "Upgrade R to version 3.0.1" like leif said?

For the spkg, I really need a URL to the spkg, not some link to some website where I have to click on something...

Also: fill in your real name as Author.

[Leif Leonhardy]

Replying to jdemeyer:

For the spkg, I really need a URL to the spkg, not some link to some website where I have to click on something...

There's btw. ​http://spkg-upload.googlecode.com...

[Leif Leonhardy]

... and feel free to add yourself to the ​list of developers on the Sage wiki.

[Emmanuel Charpentier]

Replying to jdemeyer:

Replying to charpent :

Is that clearer ?

Why not just say "Upgrade R to version 3.0.1" like leif said?

Because I intend to re-open/re-close it as upstream spits out new versions... I see no point in cluttering the ticket stream with identical repeats of the same trivial point...

For the spkg, I really need a URL to the spkg, not some link to some website where I have to click on something...

Upload in progress as I write this (I hadn't yet credentials for spkg-uploads)

Also: fill in your real name as Author.

If you say so...

[Emmanuel Charpentier]

Spkg uploaded to spkg-uploads :

​https://code.google.com/p/spkg-upload/downloads/detail?name=r-3.0.1-p0.spkg&can=2&q=

[Jeroen Demeyer]

Replying to charpent:

Because I intend to re-open/re-close it as upstream spits out new versions...

That's not allowed. A closed ticket also serves as a reference for when a particular package was merged in a particular Sage version, and the comments show any issues which appeared. If different upgrades would happen on one ticket, that would be a mess.

Standard practice is to create a new ticket for every upgrade. However, as long as the new package is not yet merged into Sage, you can change the ticket. Imagine that R 3.0.2 comes out today, you can change this ticket to upgrade to R 3.0.2 (since this ticket isn't merged yet).

[Leif Leonhardy]

Replying to charpent:

Spkg uploaded to spkg-uploads :

​https://code.google.com/p/spkg-upload/downloads/detail?name=r-3.0.1-p0.spkg&can=2&q=

Could you please rename the file to r-3.0.1.p0.spkg?

(IIRC, renaming isn't possible there, so that would mean: rename your local file, (re)upload it, and delete the old one there.)

[Leif Leonhardy]

Replying to leif:

Replying to charpent:

Spkg uploaded to spkg-uploads :

​https://code.google.com/p/spkg-upload/downloads/detail?name=r-3.0.1-p0.spkg&can=2&q=

Could you please rename the file to r-3.0.1.p0.spkg?

(IIRC, renaming isn't possible there, so that would mean: rename your local file, (re)upload it, and delete the old one there.)

Oh, forgot:

You'd of course have to repackage the spkg with the proper folder name first.

[Leif Leonhardy]

... and change the changelog entry in SPKG.txt accordingly.

Remove SPKG.txt~, and commit your changes...

Haven't yet investigated why, but the new spkg is also much larger than the previous one (27 MB vs. 20 MB).

EDIT: 27 MB that is, more than +33%

[Leif Leonhardy]

Replying to leif:

Haven't yet investigated why, but the new spkg is also much larger than the previous one (27 MB vs. 20 MB).

There's a lot of redundant stuff in different src directories:

$ diff -qr --report r-3.0.1-p0/src/src/ r-3.0.1-p0/src/R-3.0.1/src/ | grep identical | wc -l
1502

[Leif Leonhardy]

I.e., r-3.0.1-p0/src/R-3.0.1/src/ is a proper subset of r-3.0.1-p0/src/src/.

[Leif Leonhardy]

Replying to leif:

I.e., r-3.0.1-p0/src/R-3.0.1/src/ is a proper subset of r-3.0.1-p0/src/src/.

Yep, src/ is not vanilla:

$ diff -qr upstream/R-3.0.1/ r-3.0.1-p0/src/
Only in r-3.0.1-p0/src/: R-3.0.1

[Leif Leonhardy]

Fixed, untested spkg for testing is here:

​http://boxen.math.washington.edu/home/leif/Sage/spkgs/r-3.0.1.p0.spkg

27M	r-3.0.1-p0.spkg
22M	r-3.0.1.p0.spkg

(Haven't committed the changes to SPKG.txt.)

[Leif Leonhardy]

P.S.: The SPKG.txt changelog entry should also contain the ticket number, which I haven't added (yet).

[Emmanuel Charpentier]

Replying to leif:

... and change the changelog entry in SPKG.txt accordingly.

Remove SPKG.txt~, and commit your changes...

Wups ! Done (and uploaded)... I oversaw the relevant part of the developer's guide (had to upload Mercurial : this can't be done from Sage AFAICT).

Haven't yet investigated why, but the new spkg is also much larger than the previous one (27 MB vs. 20 MB).

EDIT: 27 MB that is, more than +33%

No explanation : R-3.0.1 is a tag larger than R-2.15.2 but by about 2 MB. Your guess is about as good as mine...

[Emmanuel Charpentier]

Replying to leif: comments 19 to 23

Leif, you're too fast for me ! your changes might hve been overwrittent by mine, or vice-versa...

Replying to leif:

Haven't yet investigated why, but the new spkg is also much larger than the previous one (27 MB vs. 20 MB).

There's a lot of redundant stuff in different src directories:

$ diff -qr --report r-3.0.1-p0/src/src/ r-3.0.1-p0/src/R-3.0.1/src/ | grep identical | wc -l
1502

I must have had a thinko (mental typo)...

OK : shall I go back to this file, or will you take over ? We can't both try to do the same...

[Leif Leonhardy]

Replying to charpent:

OK : shall I go back to this file, or will you take over ? We can't both try to do the same...

Well, did you make any further changes besides

SPKG.txt

@@ -45 +45 @@
 
 == Changelog ==
 
+=== r-3.0.1.p0 (Emmanuel Charpentier, 8 June 2013) ===
+ * Drop-in replacement of upstream sources.
+
 === r-2.15.2.p2 (John H. Palmieri, 17 March 2013) ===
  * #9668: fix hardcoding of paths in R by defining R_SHARE_DIR,
    R_INCLUDE_DIR, and R_DOC_DIR relative to R_HOME_DIR, or rather, by

(and removing the temporary editor file, restoring vanilla upstream in src/)?

I can add the ticket number to the changelog and commit the changes, then we have a clean spkg ready for review / testing.

[Leif Leonhardy]

Replying to leif:

I can add the ticket number to the changelog and commit the changes, then we have a clean spkg ready for review / testing.

Ok, did so.

changeset:   63:4e5b35a04f0f
tag:         tip
user:        Leif Leonhardy <not.really@online.de>
date:        Sun Jun 09 10:08:09 2013 -0700
files:       .hgtags
description:
Added tag r-3.0.1.p0 for changeset 1965338a1840


changeset:   62:1965338a1840
tag:         r-3.0.1.p0
user:        Emmanuel Charpentier <emm.charpentier@free.fr>
date:        Sun Jun 09 10:07:32 2013 -0700
files:       SPKG.txt
description:
#14706 (r-3.0.1.p0): Upgrade to new upstream version 3.0.1

=== r-3.0.1.p0 (Emmanuel Charpentier, 8 June 2013) ===
 * #14706: Drop-in replacement of upstream sources.


changeset:   61:a9186d48ab63
user:        Jeroen Demeyer <jdemeyer@cage.ugent.be>
date:        Thu Apr 25 13:00:29 2013 +0000
files:       .hgtags
description:
Added tag r-2.15.2.p2 for changeset 485622df5b13

[Leif Leonhardy]

Diff between the previous spkg and the 3.0.1.p0. For reference / review only.

[Leif Leonhardy]

Hmmm, trac apparently doesn't like # in attachment filenames very much; it's IMHO a bad idea to use such anyway.

[Leif Leonhardy]

P.P.S.:

We should IMHO still address the readline / LDFLAGS issue (unless R meanwhile supports --with-readline=<prefix>); that should be done in a .p1 based on the .p0.

[Emmanuel Charpentier]

Replying to leif:

P.P.S.:

We should IMHO still address the readline / LDFLAGS issue (unless R meanwhile supports --with-readline=<prefix>); that should be done in a .p1 based on the .p0.

Hmmm... I do not know CentOS, but it sounds more like a CentOS configuration problem. Diferent assumptions about pre-loaded libraries ? I'm afraid that my knowledge of the relevant documents is a bit insufficient for this hunt...

PS : I'll delete my (incorrect) spkg from spkg-uploads now.

[Leif Leonhardy]

Replying to leif:

... unless R meanwhile supports --with-readline=<prefix>

It doesn't:

## Readline.

# Check whether --with-readline was given.
if test "${with_readline+set}" = set; then :
  withval=$with_readline; if test "${withval}" = no; then
  use_readline=no
else
  use_readline=yes
fi

else
  use_readline=yes
fi

[Leif Leonhardy]

Replying to leif:

We should IMHO still address the readline / LDFLAGS issue

See ​http://groups.google.com/group/sage-devel/browse_thread/thread/5803a1cc05fe8a6e/e927d932b5d650e1#e927d932b5d650e1 ff. for the CentOS error report.

[Leif Leonhardy]

New spkg builds for me (and passes its test suite with some "noise") on Linux x86_64, GCC 4.8.0, Sage 5.10.rc1.

Doctests in sage/interfaces/r.py also pass with the attached patch.

[Leif Leonhardy]

Replying to leif:

Replying to leif:

We should IMHO still address the readline / LDFLAGS issue

See ​http://groups.google.com/group/sage-devel/browse_thread/thread/5803a1cc05fe8a6e/e927d932b5d650e1#e927d932b5d650e1 ff. for the CentOS error report.

Ok, this is IMHO rather unrelated to R, or more precisely, a general problem, caused by appending $SAGE_LOCAL/lib to LIBRARY_PATH instead of prepending, so I'll open a separate ticket for that.

[Leif Leonhardy]

Replying to leif:

Replying to leif:

Replying to leif:

We should IMHO still address the readline / LDFLAGS issue

See ​http://groups.google.com/group/sage-devel/browse_thread/thread/5803a1cc05fe8a6e/e927d932b5d650e1#e927d932b5d650e1 ff. for the CentOS error report.

Ok, this is IMHO rather unrelated to R, or more precisely, a general problem, caused by appending $SAGE_LOCAL/lib to LIBRARY_PATH instead of prepending.

Or maybe not (just). Dropping -L${SAGE_LOCAL}/lib and abusing LIBRARY_PATH simply doesn't work, since GCC may add system folders to the library search path before those specified in LIBRARY_PATH.

It's rather luck that the error so far showed up on CentOS only; others appear to have either no or compatible readline versions installed system-wide.

[Jean-Pierre Flori]

I think there is now a --disable-long-double option to pass to configure which should be used on Cygwin rather than our patch which at some point will surely miss some parts of R code.

[Leif Leonhardy]

Replying to jpflori:

I think there is now a --disable-long-double option to pass to configure which should be used on Cygwin rather than our patch which at some point will surely miss some parts of R code.

Can you create a .p1 and test that on Cygwin?

(Interestingly, the patches still apply, modulo offsets for the patch to configure, which IMHO aren't worth fixing, since that just bloats Mercurial history).

And if so, also change --with-readline=... to --with-readline=yes, with a remark that only yes/no is supported.

[P.S.: Just noticed the Cygwin patch is indeed very unlikely to no longer apply... B) ]

[Leif Leonhardy]

Replying to leif:

Replying to jpflori:

I think there is now a --disable-long-double option to pass to configure which should be used on Cygwin rather than our patch which at some point will surely miss some parts of R code.

Can you create a .p1 and test that on Cygwin?

(I mean I could do that as well, but that doesn't make much sense, since I cannot test it.)

[Leif Leonhardy]

P.P.S.: Are long double functions a general problem on Cygwin? For R, we currently just rename logl(), so using --disable-long-double may have other impacts I don't know...

[Jean-Pierre Flori]

Replying to leif:

P.P.S.: Are long double functions a general problem on Cygwin?

Yes, Cygwin libm has no long double functions.

For R, we currently just rename logl(), so using --disable-long-double may have other impacts I don't know...

That's the point, we want to disable all uses of long double in R, I cannot say that the only place it is currently used in 3.0.1 is with this logl() call, nor if that will be the case in future version of R. So using --disable-long-double is safer. Or we should reenable Cephes on Cygwin to provide long double functions but I cannot say if that will be easy or not.

[Leif Leonhardy]

Replying to jpflori:

Replying to leif:

For R, we currently just rename logl(), so using --disable-long-double may have other impacts I don't know...

That's the point, we want to disable all uses of long double in R

That was my question (whether doing so was intended; the basic operations on long doubles are provided by GCC / libgcc) ... ;-)

Are you going to create a new spkg?

[Jean-Pierre Flori]

I'll try tonight.

[Jean-Pierre Flori]

Spkg at:

• ​http://boxen.math.washington.edu/home/jpflori/spkg/r-3.0.1.p1.spkg

but it doesn't build because it seems an sqrtl function is still used.

[Jean-Pierre Flori]

New spkg currently uploading with a new patch to fix --disable-long-double.

[Jean-Pierre Flori]

The patch does not work, I'll try to fix it tomorrow.

[Jean-Pierre Flori]

According to ​http://developer.r-project.org/R_svnlog_2013, there is:

r62832 | ripley | 2013-05-29 11:26:12 -0400 (Wed, 29 May 2013) | 1 line
Changed paths:
   M /trunk/src/library/stats/src/cov.c

avoid sqrtl (PR#15326)

Just have to find a way to get the commit without mirroring the R svn.

[Jean-Pierre Flori]

Upstream bug report here: ​https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=15326

[Jean-Pierre Flori]

Ok, new spkg with upstream fix. No Cygwin install on my hand right now, I'll test it tonight, but it really seems ok.

Not sure the license info is correct.

[Jeroen Demeyer]

Spkg diff, for review only.

[Jeroen Demeyer]

Committed spkg changes.

[Jean-Pierre Flori]

Thanks, I have some trouble correctly running hg addremove...

[Karl-Dieter Crisman]

Just want to approve of the long double fix, I think that is the right thing for now. I'm excited that this might get in.

[Jeroen Demeyer]

Okay, let's go for it.

[Karl-Dieter Crisman]

Note: that doesn't mean I tested it (on Cygwin or anywhere else), just that if JP et al. did test it, then I can confirm that this is the 'right' way to deal with this in our Cygwin experience.

[Leif Leonhardy]

Replying to jpflori:

Not sure the license info is correct.

AFAICS, R is (meanwhile) GNU GPLv2 or GPLv3 (but not v2+; some parts are LGPL'ed).

(I don't see any related change to SPKG.txt, which stated "GPL 2 (2.15.2 release)".)

[Leif Leonhardy]

Replying to leif:

Replying to jpflori:

Not sure the license info is correct.

AFAICS, R is (meanwhile) GNU GPLv2 or GPLv3 (but not v2+; some parts are LGPL'ed).

CHANGES IN R VERSION 2.13.1:

  ...

  LICENCE:

    o No parts of R are now licensed solely under GPL-2.  The licences
      for packages rpart and survival have been changed, which means
      that the licence terms for R as distributed are GPL-2 | GPL-3.

(from NEWS)

[Jean-Pierre Flori]

I can confirm it now compiles on Cygwin (32 bits).

[Jeroen Demeyer]

OK, made this change to the spkg:

SPKG.txt

@@ -11 +11 @@
 (taken from http://www.r-project.org/)
 
 == License ==
- * GPL 2 (2.15.2 release)
+ * GPL v2 or GPL v3
 
 == SPKG Maintainers ==
 

[Jeroen Demeyer]

There are some problems with upgrading on OS X:

gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c text.c -o text.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c init.c -o init.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c Rmd5.c -o Rmd5.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c md5.c -o md5.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c signals.c -o signals.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c install.c -o install.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c getfmts.c -o getfmts.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c http.c -o http.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c gramLatex.c -o gramLatex.o
gcc -std=gnu99 -I../../../../include -DNDEBUG -I../../../include -I../../../../src/include -DHAVE_CONFIG_H -I../../../../src/main     -fPIC  -g -O2   -c gramRd.c -o gramRd.o
gcc -std=gnu99 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -o tools.so text.o init.o Rmd5.o md5.o signals.o install.o getfmts.o http.o gramLatex.o gramRd.o -L../../../../lib -lR -dylib_file libRblas.dylib:../../../../lib/libRblas.dylib -Wl,-framework -Wl,CoreFoundation
make[8]: `Makedeps' is up to date.
mkdir ../../../../library/tools/libs
dyld: Library not loaded: libR.dylib
  Referenced from: /Users/dehayebuildbot/build/sage/dehaye/dehaye_upgrade_5.4.1/build/sage-5.11.beta1/spkg/build/r-3.0.1.p1/src/bin/exec/R
  Reason: Incompatible library version: R requires version 3.0.0 or later, but libR.dylib provides version 2.14.0
/bin/sh: line 1: 83314 Done                    echo "tools:::.install_package_description('.', '"../../../library/tools"')"
     83315 Trace/BPT trap: 5       | R_DEFAULT_PACKAGES=NULL ../../../bin/R --vanilla --slave > /dev/null
make[5]: *** [all] Error 133
make[4]: *** [R] Error 1
make[3]: *** [R] Error 1
make[2]: *** [R] Error 1
Error building R.

[Jeroen Demeyer]

The solution seems to be to remove local/lib/R before rebuilding R.

[John Palmieri]

Replying to jdemeyer:

The solution seems to be to remove local/lib/R before rebuilding R.

Maybe rename it instead, then rebuild R. If the build is successful, delete the old version, and if the build fails, restore the old one?

[Leif Leonhardy]

Replying to jhpalmieri:

Replying to jdemeyer:

The solution seems to be to remove local/lib/R before rebuilding R.

Maybe rename it instead, then rebuild R. If the build is successful, delete the old version, and if the build fails, restore the old one?

Or mess with DYLD_LIBRARY_PATH... (in spkg-install, during the build of R, that is)

[Leif Leonhardy]

Replying to leif:

Replying to jhpalmieri:

Replying to jdemeyer:

The solution seems to be to remove local/lib/R before rebuilding R.

Maybe rename it instead, then rebuild R. If the build is successful, delete the old version, and if the build fails, restore the old one?

Or mess with DYLD_LIBRARY_PATH... (in spkg-install, during the build of R, that is)

Instead of prepending some path of the R build tree, perhaps better simply remove $SAGE_LOCAL/lib/R from [DY]LD_LIBRARY_PATH...

[Jeroen Demeyer]

Replying to jhpalmieri:

Maybe rename it instead, then rebuild R. If the build is successful, delete the old version, and if the build fails, restore the old one?

I would vote for the simple solution of simply removing local/lib/R. Why make it more complicated than that?

[Leif Leonhardy]

Replying to jdemeyer:

Replying to jhpalmieri:

Maybe rename it instead, then rebuild R. If the build is successful, delete the old version, and if the build fails, restore the old one?

I would vote for the simple solution of simply removing local/lib/R. Why make it more complicated than that?

Well, temporarily renaming isn't much harder (you only have to make sure the previous state gets restored in case installation fails, and, less important, to delete the old installation in case of success).

Deleting the old installation before building the new version has succeeded is ... -- bad.

[Leif Leonhardy]

P.S.: But I'd prefer to simply remove the old R lib dir from [DY]LD_LIBRARY_PATH -- it doesn't make any sense to have it there when (a new) R (version) gets built, other than potentially causing problems.

[Leif Leonhardy]

Replying to leif:

P.S.: But I'd prefer to simply remove the old R lib dir from [DY]LD_LIBRARY_PATH -- it doesn't make any sense to have it there when (a new) R (version) gets built, other than potentially causing problems.

P.P.S.: I would even guess that's an upstream bug; R at least used to mess around with LD_LIBRARY_PATH -- presumably also to avoid such problems (which don't happen on Linux); they probably just missed DYLD_LIBRARY_PATH on Darwin...

[Jean-Pierre Flori]

I'm not sure its an upstream bug after a quick look at R sources, maybe just us messing around with DYLD_LIBRARY_PATH.

[Emmanuel Charpentier]

Messing around with the dynamic library mechanism does not seem to be such a great idea.  The 3.0.1.p1 patch gives (on Debian amd64 testing) an executable which is unable to install some packages using external libraries ( among them tcl/tk, curl, sqlite3 and more important, cpp libraries). I had oddles of trouble trying to install packages depending on RcppArmadillo?. Reverting to the 3.0.1.p0 patch solved this problem.

I'd be sorely tempted to change the status to "needs work" ... if I knew how to do this.

I'd like to recommend :

1. to keep the (simple) fix to get the 3.0.1 sources (which, AFAIU, compile fine on Unices and Mac OS X) and leave the dynamic library gizmoes alone, and
2. to open *another* ticket for the dynamic library issues, which, if I understand the history, aim to compile this cleanly on cygwin.

HTH,

Emmanuel Charpentier

[Jean-Pierre Flori]

Replying to charpent:

Messing around with the dynamic library mechanism does not seem to be such a great idea.  The 3.0.1.p1 patch gives (on Debian amd64 testing) an executable which is unable to install some packages using external libraries ( among them tcl/tk, curl, sqlite3 and more important, cpp libraries). I had oddles of trouble trying to install packages depending on RcppArmadillo?. Reverting to the 3.0.1.p0 patch solved this problem.

I'd be sorely tempted to change the status to "needs work" ... if I knew how to do this.

I'd like to recommend :

1. to keep the (simple) fix to get the 3.0.1 sources (which, AFAIU, compile fine on Unices and Mac OS X) and leave the dynamic library gizmoes alone, and
2. to open *another* ticket for the dynamic library issues, which, if I understand the history, aim to compile this cleanly on cygwin.

HTH,

Emmanuel Charpentier

That's really strange.

The difference between the p0 and the p1 has nothing to do with *LD_LIBRARY_PATH. It merely triggers an upstream compilation option on Cygwin only. Not on any other OS (unless uname on that os contains CYGWIN which would be strange outside of Cygwin). Just have a look at r-3.0.1.p1.diff​...

Nor has the diff between the p0 and the previous 2.* R version.

The *LD_LIBRARY_PATH comes from $SAGE_ROOT/spkg/bin/sage-env (or $SAGE_LOCAL, too lazy to check) or R itself, not from the spkg scripts. If you're not convinced, just grep them.

[Karl-Dieter Crisman]

This will (trivially only) conflict with #13686.

[Volker Braun]

Currently R doesn't build on sage.math, dying with

../../lib/libR.so: undefined reference to `rl_sort_completion_matches'

Can we clean this ticket up?

Just a few comments from cursory reading this ticket:

• Delete anything that can interfere with installation at the beginning of spkg-install (yes one could try to catch errors, but we don't have a framework for testing that so it won't ever work)

• try to mess with library paths as little as possible.

• Cygwin issues can go to a different ticket.

[Jean-Pierre Flori]

What was done in this ticket was really trivial. Basically:

• drop in replacement of the upstram sources,
• a trivial fix for the Cygwin fixes (basically using the now upstream provided option rather than our custon patch + add an upstream patch not in the current stable version to fix the new upstream option; this has really no reason to affect any other systems than Cygwin unless it makes your computer proner to attracting gamma rays) so I don't see any reason not to keep them in.

So if R is now broken, its surely because of some hell with *LDLIBRARY_PATH either on Sage sideor on R side itself (as R seems to be as bad as Sage as far as hacking LDLIBRARY_PATH is concerned).

I agree that cleaning up the spkg-install would be a good idea as well.

Also note that R 3.0.2 was released a few days ago, so maybe the patch fixing the disable_long-double option is now included.

[Volker Braun]

Just to clarify, both the R in sage-5.12.rc1 as well as the one on this ticket fail to compile on sage.math.

[Jean-Pierre Flori]

I guess the readline problem is:

• ​http://lists.gnu.org/archive/html/bug-readline/2013-07/msg00017.html

[Jean-Pierre Flori]

There is a new readline release beta:

• ​http://permalink.gmane.org/gmane.comp.gnu.readline/406

I guess its worth giving it a try...

[Volker Braun]

Its not a bug in readline (that would be huge and immediatly fail). The R-project guys are just confused about the readelf output on their mailinglist (hint: --wide won't truncate output). The issue is here:

gcc -std=gnu99 -I../../src/extra/zlib -I../../src/extra/bzip2 -I../../src/extra/pcre -I../../src/extra  -I../../src/extra/xz/api -I. -I../../src/include -I../../src/include  -DHAVE_CONFIG_H   -fopenmp -fpic  -g -O2   -c Rmain.c -o Rmain.o
gcc -std=gnu99 -Wl,--export-dynamic -fopenmp   -o R.bin Rmain.o  -L../../lib -lR -lRblas
../../lib/libR.so: undefined reference to `rl_sort_completion_matches'

There is no -lreadline, so it shouldn't work.

[Jean-Pierre Flori]

Yup, that sounds more sensible :)

[Jean-Pierre Flori]

I've updated a trivially updated spkg, could you give it a shot? And post or link the failing log?

It doesn't seem the failing line you mentioned changed. (Note that the line I get is slightly different, I don't get the -lRblas.) Before that line at least, on my install, when libR.so is built, -lreadline is correctly passed, so maybe this changed for 3.0.2.

At this point, I changed nothing to deal with a potentially conflicting old version of R 2.x as mentioned earlier on this ticket.

[Jean-Pierre Flori]

Groumpf, I just gave 5.13.beta1 (I know its not stabilized...) on sage.math and it succesfully built the shipped R 2.x spkg. Strange.

[Jean-Pierre Flori]

No problem with 3.0.2 spkg on sage.math for me (on top of 5.13.beta1, Sage's GCC was built)... Strangely I don't get the -lRblas flag.

Volker: Could you post your logs?

[Jean-Pierre Flori]

Same behavior for me on sage.math on top of 5.12.rc1 (with Sage's GCC built) with the provided 2.x spkg and the new 3.0.2 one. Still no -lRblas and no error. So I cannot do much about Volker's problem without further info...

It would also be nice if Emmanuel could confirm the error on OS X is still there.

[Volker Braun]

Sorry, should have been more clear: I'm trying to compile the current git master. It still fails with 3.0.2.p0.

Build log is here: ​http://boxen.math.washington.edu/home/vbraun/logs/r-3.0.2.p0.log

[Jean-Pierre Flori]

I tried the master branch from ​https://github.com/sagemath/sage.git and had no problems with r-2.x and 3.x...

Which repo/branch did you use?

[Jean-Pierre Flori]

Here is a difference in the log of your and my r-3.0.2 builds

428,432c431,432
< checking for dgemm_ in -L/home/jpflori/build/sage.git/local/lib -lf77blas -latlas... yes
< checking whether double complex BLAS can be used... yes
< checking whether the BLAS is complete... yes
< checking for dpstrf_... no
< checking for dpstrf_ in -L/home/jpflori/build/sage.git/local/lib -llapack -lcblas... yes
---
> checking for dgemm_ in -L/home/vbraun/Sage/git/local/lib -lf77blas -latlas... yes
> checking whether double complex BLAS can be used... no

I guess this is at the root of the "-lRblas" you get and I don't. What version of ATLAS or BLAS did you use? Not sure it will then lead us to the readline problem.

[Jean-Pierre Flori]

Ok, it seems from your log at ​http://boxen.math.washington.edu/home/vbraun/Sage/git/logs/pkgs/atlas-3.10.1.p5.log that you used the system ATLAS. I'll do the same.

[Jean-Pierre Flori]

Comparing the previously succesful libR.so (with Sage's ATLAS) and the new one (with system-wide ATLAS), the former is using:

libreadline.so.6 => /home/jpflori/build/sage.git/local/lib/libreadline.so.6 (0x00007fe83965c000)

whereas the latter is using

libreadline.so.5 => /lib/libreadline.so.5 (0x00007f9a46b59000)

[Jean-Pierre Flori]

In the working case, when libR.so is built the gcc command includes:

-L/home/jpflori/build/sage.git/local/lib -lf77blas -latlas -lgfortran -lm -lquadmath   -lreadline  -lrt -ldl -lm

whereas in the other case it includes:

-L../../lib -lRblas -lgfortran -lm -lquadmath   -lreadline  -lrt -ldl -lm

The "-L" stuff is here because it is passed through --with-blas to configure, and I guess gets dropped and replace in the latter case because R does not like the systm wide ATLAS.

Now what's strange is that we pass --with-readline="$SAGE_LOCAL" to configure but it does not seem to care about the location we give it.

[Jean-Pierre Flori]

Indeed, R configure does not give a * about what we give to --with-readline (unless its no).

So we should explicitely pass $SAGE_LOCAL through LDFLAGS I guess.

[Jean-Pierre Flori]

It seems enough indeed.

[Jean-Pierre Flori]

As far as OS X is concerned, could some of you check if reverting ​http://trac.sagemath.org/attachment/ticket/1939/Sage-2.10.1.rc0-fix-rpy-import-on-OSX.patch helps in some way?

[Jean-Pierre Flori]

I guess this is only a problem on OS X because of

R_LD_LIBRARY_PATH_save=${R_LD_LIBRARY_PATH}
R_LD_LIBRARY_PATH=
case "${host_os}" in
  darwin*)
    ## Darwin provides a full path in the ID of each library such 
    ## that the linker can add library's path to the binary at link time.
    ## This allows the dyld to find libraries even without xx_LIBRARY_PATH.
    ## No paths should be added to R_LD_LIBRARY_PATH (which in turn
    ## changes DYLD_LIBRARY_PATH), because they override the system
    ## look-up sequence. Such automatic override has proven to break things
    ## like system frameworks (e.g. ImageIO or OpenGL framework).
    ;;
  *)
    for arg in ${LDFLAGS}; do
      case "${arg}" in
        -L*)
	  lib=`echo ${arg} | sed "s/^-L//"`
	  R_SH_VAR_ADD(R_LD_LIBRARY_PATH, [${lib}], [${PATH_SEPARATOR}])
	  ;;
      esac
    done
    ;;
esac

and this could be fishy as well because it look somehow contradictory with the above

case "${host_os}" in
  darwin*)
    ## In order to allow the R build to be relocatable, we strip paths
    ## from all shlibs and rely on DYLD_LIBRARY_PATH. Unfortunately
    ## Darwin linker ignores it at build-time and doesn't use -L to
    ## resolve dylib dependencies, so libRblas will not be found unless
    ## we tell ld where it lives. I don't know of any more elegant solution :/
    if test "x${use_blas_shlib}" = xyes; then
      LIBR="${LIBR} -dylib_file libRblas.dylib:\$(R_HOME)/lib\$(R_ARCH)/libRblas.dylib"
    fi
  ;;
esac
AC_SUBST(LIBR)

in configure.ac.

[Jean-Pierre Flori]

I guess the real problem on OS X is that as shown above, R does not modify DYLD_LIBRARY_PATH, but rather DYLD_FALLBACK_LIBRARY_PATH through etc/ldpaths in R.sh, whereas Sage modifies directly DYLD_LIBRARY_PATH. See

## We already added -L's from LDFLAGS (except on Darwin): 
## seem to be doing it again
for arg in ${LDFLAGS} ${FLIBS} ${BLAS_LIBS} ${LAPACK_LIBS} ${X_LIBS} \
           ${TCLTK_LIBS}; do
  case "${arg}" in
    -L*)
      lib=`echo ${arg} | sed "s/^-L//"`
      r_want_lib=true
      ## don't add anything for Darwin
      case "${host_os}" in darwin*) r_want_lib=false ;; esac
      ## Do not add non-existent directories.
      test -d "${lib}" || r_want_lib=false
      if test x"${r_want_lib}" = xtrue; then
        ## Canonicalize (/usr/lib/gcc-lib/i486-linux/3.3.4/../../..).
        lib=`cd "${lib}" && ${GETWD}`
        ## Do not add something twice, or default paths.
        r_save_IFS="${IFS}"; IFS="${PATH_SEPARATOR}"
        for dir in ${R_LD_LIBRARY_PATH}${IFS}${r_ld_library_defaults}; do
          if test x"${dir}" = x"${lib}"; then
            r_want_lib=false
            break
          fi
        done
        IFS="${r_save_IFS}"
        if test x"${r_want_lib}" = xtrue; then
          R_SH_VAR_ADD(R_LD_LIBRARY_PATH, [${lib}], [${PATH_SEPARATOR}])
        fi
      fi
      ;;
  esac
done
fi

at the end of configure.ac.

So when R is not installed, at installation time, when R is run, no libR.dylib is found in DYLD_LIBRARY_PATH, but it is in the FALLBACK variant. But when R is already installed, then libR.dylib is found in DYLD_LIBRARY_PATH and that's potentially a wrong version...

[Jean-Pierre Flori]

Updated spkg to deal with the OS X issue. Use spkg at same address.

Apply:

• trac_14706-version.patch​

[Jean-Pierre Flori]

Note I've only tested this on Ubuntu, no access to OS X (now and later as well).

[John Palmieri]

This builds and all tests pass on OS X 10.8.5 (Mountain Lion).

[John Palmieri]

(This failed to build on OS X 10.9 (Mavericks), but the current R spkg fails, too. So this should not be an obstacle to a positive review for this ticket.)

[Jean-Pierre Flori]

I guess you actually tried to update from R 2.x to R 3.x and not a build from scratch with the spkg replaced?

So you might give a positive review as the OS X issue was the only thing which made this go from positive_review back to needs_work?

[John Palmieri]

I don't remember what I did before, probably upgraded an existing installation. Earlier today I started with a new Sage tarball, put in the new R spkg and applied the patch. All tests passed again.

[John Palmieri]

Just now I successfully took an existing Sage build and updated just the R spkg. That went well, and all tests passed. I found it a little odd that the rpy spkg wasn't rebuilt (it tried to install it and exited, saying Package rpy2-2.0.8.p0 is already installed). I hope that the rpy spkg just relies on information like where R has been installed, not specific version information.

[Jean-Pierre Flori]

Replying to jhpalmieri:

Just now I successfully took an existing Sage build and updated just the R spkg. That went well, and all tests passed. I found it a little odd that the rpy spkg wasn't rebuilt (it tried to install it and exited, saying Package rpy2-2.0.8.p0 is already installed). I would definitely say this is not an issue with the R spkg, but with Sage build system.

[Jeroen Demeyer]

This fails to install on Solaris SPARC:

Loading required package: Matrix
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/home/buildbot/build/sage/mark-1/mark_full/build/sage-5.13.beta3/spkg/build/r-3.0.2.p0/src/library/Matrix/libs/Matrix.so':
  ld.so.1: R: fatal: relocation error: file /home/buildbot/build/sage/mark-1/mark_full/build/sage-5.13.beta3/spkg/build/r-3.0.2.p0/src/library/Matrix/libs/Matrix.so: symbol libintl_dngettext: referenced symbol not found
Error : require(Matrix) is not TRUE
ERROR: installing package indices failed
* removing '/home/buildbot/build/sage/mark-1/mark_full/build/sage-5.13.beta3/spkg/build/r-3.0.2.p0/src/library/Matrix'
make[5]: *** [Matrix.ts] Error 1

Full log: ​http://build.sagemath.org/sage/builders/Skynet%20mark%20%28SunOS%205.10-32%29/builds/361/steps/shell_5/logs/r

[Jean-Pierre Flori]

Looks like what's reported here:

• ​http://osol.wikidot.com/logentry:software

Another madness of Solaris shipping a bunch of different versions of the same libs.

[Jean-Pierre Flori]

Spkg diff, for review only.

[Jean-Pierre Flori]

Hopefully, I fixed the problem. The Matrix module wants to use the gettext dngettext function. On usual system this is provided by the system libc or the system libintl. On Solaris the system libintl does not provide it so R decides to build its own libintl (in src/extra/intl) but with symbols hidden by default (-fvisibility-hidden) and only publicize a few of them, but not dngettext. So I treated that last function as the few others one already visible.

spkg at same address. diff updated.

[Jean-Pierre Flori]

• Non-standard location readline problem reported as ​https://bugs.r-project.org/bugzilla/show_bug.cgi?id=15556
• Solaris gettext problem as ​https://bugs.r-project.org/bugzilla/show_bug.cgi?id=15555

[Jeroen Demeyer]

Nitpick about the library patch: the second r.version() should be r_version(). By the way, I wouldn't mind deprecating r_version() since I really don't see the point. It would be consistent with the deprecation of gap_version().

[Jean-Pierre Flori]

Patch updated.

Sure, I have nothing agains deprecating r_version which seems not so useful. But I won't have time to do it right now and this is not really the scope of this ticket, so two good reasons to do that in a follow-up ticket.

[Karl-Dieter Crisman]

See the discussion at #13686 as well (and perhaps elsewhere?) on the R version function issue.

[Marc Mezzarobba]

Shouldn't r-3.0.2 be available from ​http://www.sagemath.org/packages/upstream/r/r-3.0.2.tar so that the build system of the git master branch can download it automatically?

[Jean-Pierre Flori]

I've no idea who is in charge of maintaining packages/upstream, nor if it's automatically updated when Mercurial releases are merged into the git repo, nor how we should upload future tarballs when we're in git-only mode (in particular note that some tarball are not upstream one's, maybe all dependencies should now have spkg-src which automate the creation of the tarballs --- vanilla or not).

[Marc Mezzarobba]

Ok, thanks for your reply. I also added a comment on the issue at ​http://trac.sagemath.org/ticket/14480#comment:51, so hopefully someone who knows where to find the right version can upload it!

[Jean-Pierre Flori]

I've posted on sage-git where it might receive more attention:

• ​https://groups.google.com/d/msg/sage-git/wyM7OSZcw2E/gbVNVq0KaKIJ

[Jean-Pierre Flori]

Replying to jpflori:

• Non-standard location readline problem reported as ​https://bugs.r-project.org/bugzilla/show_bug.cgi?id=15556

Upstream answered: you're the first one to report that in 15 years, unless you provide a patch we don't care.

• Solaris gettext problem as ​https://bugs.r-project.org/bugzilla/show_bug.cgi?id=15555

This has been fixed in R trunk.

[Dima Pasechnik]

log on an ARM system

[Dima Pasechnik]

I am trying to build Sage 6.0 on a ARM system I just got access to, and it fails with a strange error. See the corr. attachment for the full log.

begin installing recommended package lattice
* installing *source* package 'lattice' ...
** package 'lattice' successfully unpacked and MD5 sums checked
** libs
make[6]: Entering directory `/tmp/Rtmpu7TiBm/R.INSTALL74b515accca6/lattice/src'
gcc -std=gnu99 -I/home/dimpase/sage/sage/local/var/tmp/sage/build/r-3.0.2.p0/src
/include -DNDEBUG      -fpic  -g -O2   -c init.c -o init.o
gcc -std=gnu99 -I/home/dimpase/sage/sage/local/var/tmp/sage/build/r-3.0.2.p0/src
/include -DNDEBUG      -fpic  -g -O2   -c threeDplot.c -o threeDplot.o
gcc -std=gnu99 -shared -L/home/dimpase/sage/sage/local/lib/ -o latticeSHLIB_EXT 
init.o threeDplot.o SHLIB_LIBADD -L/home/dimpase/sage/sage/local/var/tmp/sage/bu
ild/r-3.0.2.p0/src/lib -lR
gcc: error: SHLIB_LIBADD: No such file or directory
make[6]: *** [latticeSHLIB_EXT] Error 1
make[6]: Leaving directory `/tmp/Rtmpu7TiBm/R.INSTALL74b515accca6/lattice/src'
ERROR: compilation failed for package 'lattice'
* removing '/home/dimpase/sage/sage/local/var/tmp/sage/build/r-3.0.2.p0/src/library/lattice'
make[5]: *** [lattice.ts] Error 1

[Volker Braun]

See ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679180 for an analysis and patch of the SHLIB_LIBADD issue

[Dima Pasechnik]

Replying to vbraun:

See ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679180 for an analysis and patch of the SHLIB_LIBADD issue

Anyone familiar with R building system around? I tried this, and got weird stuff:

** package 'lattice' successfully unpacked and MD5 sums checked
** libs
make[6]: Entering directory `/tmp/RtmpYoKYEk/R.INSTALL57a933daa9d6/lattice/src'
gcc -std=gnu99 -I/home/dimpase/sage/sage/local/var/tmp/sage/build/r-3.0.2.p0/src/include -DNDEBUG 
     -fpic  -g -O2   -c init.c -o init.o
gcc -std=gnu99 -I/home/dimpase/sage/sage/local/var/tmp/sage/build/r-3.0.2.p0/src/include -DNDEBUG 
     -fpic  -g -O2   -c threeDplot.c -o threeDplot.o
gcc -std=gnu99 -shared -L/home/dimpase/sage/sage/local/lib/ -o latticeSHLIB_EXT init.o threeDplot.
o -L/home/dimpase/sage/sage/local/var/tmp/sage/build/r-3.0.2.p0/src/lib -lR
gcc -std=gnu99 -shared -L/home/dimpase/sage/sage/local/lib/ -o .so init.o threeDplot.o -L/home/dim
pase/sage/sage/local/var/tmp/sage/build/r-3.0.2.p0/src/lib -lR
make[6]: Leaving directory `/tmp/RtmpYoKYEk/R.INSTALL57a933daa9d6/lattice/src'

note that linker is called twice, once with -o latticeSHLIB_EXT, and then with -o .so !? wtf... I can only speculate that the generation of SHLIB_EXT is still broken and injects unprintable chars into the string, which leads to the linking command executed twice.

[Dima Pasechnik]

Replying to dimpase:

Replying to vbraun:

See ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679180 for an analysis and patch of the SHLIB_LIBADD issue

Anyone familiar with R building system around? I tried this, and got weird stuff:

it turns out that SHLIB_EXT is set up in more than 1 place in install.R. So this workaround has to be done for all these places to work right.

I've opened a new ticket with the patch. See #15762.

[Emmanuel Charpentier]

Dear co-authors,

Please have a look at Trac#16694, whoch might be germane to your work and solution.

Emmanuel Charpentier