Opened 6 years ago

Closed 6 years ago

#20190 closed enhancement (fixed)

Upgrade R to 3.2.4

Reported by: charpent Owned by:
Priority: minor Milestone: sage-7.2
Component: packages: standard Keywords: r-project
Cc: leif, jdemeyer, vbraun Merged in:
Authors: Emmanuel Charpentier Reviewers: Travis Scrimshaw, Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: 6b46d9d (Commits, GitHub, GitLab) Commit: 6b46d9d61e1480e46ab7f562ed0ed5a7639f5ef7
Dependencies: Stopgaps:

Status badges

Description (last modified by charpent)

It's this time of the year again...

Original tarball is here : https://cran.r-project.org/src/base/R-3/R-3.2.4.tar.gz.

"Revised" tarball (long story, see below) here.

Attachments (4)

r-3.2.4.p0.log (55.8 KB) - added by tscrim 6 years ago.
r-3.2.4.p0-EC.log (252.4 KB) - added by charpent 6 years ago.
Compilation log on my (charpent's) machine
r-3.2.4-revised.p0.log (3.3 KB) - added by charpent 6 years ago.
(Unsuccessfull) compilation log of the "revised" package
r-3.2.4-revised.p0.v2 (266.6 KB) - added by charpent 6 years ago.
Log of a successfull compilation without testing.

Download all attachments as: .zip

Change History (43)

comment:1 Changed 6 years ago by charpent

  • Component changed from PLEASE CHANGE to packages: standard

comment:2 Changed 6 years ago by charpent

  • Branch set to u/charpent/upgrade_r_to_3_2_4

comment:3 Changed 6 years ago by charpent

  • Authors set to Emmanuel Charpentier
  • Commit set to 385c51d8fc95817eec10780fa40723d9f02865e0
  • Status changed from new to needs_review

Passes ptestlong with no error nor glitch ==> needs_review

Changed 6 years ago by tscrim

comment:4 Changed 6 years ago by tscrim

  • Status changed from needs_review to needs_work

I could not build it on Ubuntu 14.04 LTS by applying this on top of 7.1.rc0 and running make build. Log is attached.

My guess is it is related to the fuzz from applying the patches.

comment:5 Changed 6 years ago by tscrim

  • Cc leif jdemeyer vbraun added

From building old versions of R, those logs say the command should be rm -f liblzma.a (which it what I was thinking it should be). I suspect it has to do with the dont_lose_libz_libbz2_in_configure.patch, but I'm not sure what to look for. Do one of you know what needs to be done (and do you also get a build failure)?

comment:6 Changed 6 years ago by charpent

Works for me on Debian (three instances of testing/amd64 w. 6, 7 and 16 Gb RAM : 7 Gb seems the bare acceptable minimum to install/run rstan under R (The 6 Gb machine starts to swap/trash on this one...)

I'll have a look tonight (maybe tomorrow), but on a smallish machine (Ubuntu 15.10/amd64) not well suited to debuging sage (4 GB RAM...).

HTH,

comment:7 Changed 6 years ago by tscrim

FYI - I just did a make distclean && make build (with SAGE_NUM_THREADS=8 and MAKE='make -j8') and I got the same problem.

Changed 6 years ago by charpent

Compilation log on my (charpent's) machine

comment:8 Changed 6 years ago by charpent

Compilation on my machine(s) does not follow the same path as on yours. My compilation never reaches the point where yours goes.

I do not (yet) understand qwhy. Different installed software ?

HTH,

comment:9 Changed 6 years ago by tscrim

I believe it has to do with (lib)lzma being present on the system (FYI, I have a Core i7). In particular, your log has

checking for lzma_version_number in -llzma... yes

whereas my in no. So it probably has to do with dont_lose_libz_libbz2_in_configure.patch and changes in the configure script for R (there was similar fuzz when building R-3.2.3.p1 on my machine as well).

Although I have no idea what to look for on how to fix this, but perhaps this will allow you to reproduce my build error?

comment:10 Changed 6 years ago by tscrim

Were you able to reproduce the build failure?

comment:11 follow-up: Changed 6 years ago by charpent

Eureka ! (maybe...)

Sorry for the delay : my schedule is a little bit busy nowadays...

I found this :

charpent@SAP5057241:~/Temporaire/DemeRde$ diff -ur R-3.2.3/src/extra/xz R-3.2.4/src/extra/xz
diff -ur R-3.2.3/src/extra/xz/Makefile.in R-3.2.4/src/extra/xz/Makefile.in
--- R-3.2.3/src/extra/xz/Makefile.in	2010-11-09 00:05:05.000000000 +0100
+++ R-3.2.4/src/extra/xz/Makefile.in	2015-12-11 00:15:12.000000000 +0100
@@ -58,8 +58,8 @@
 	touch stamp
 
 liblzma.a: $(liblzma_a_OBJECTS)
-	rm -f $@
-	$(AR) cr $@ $(liblzma_a_OBJECTS)
+	$rm -f $@
+	$(AR) -cr $@ $(liblzma_a_OBJECTS)
 	$(RANLIB) $@
 
 mostlyclean: clean

I'll ask around in the R mailing list to know what was intended by this (highly suspicious) change.

Do you want an (interim) patch to revert it ?

comment:12 in reply to: ↑ 11 ; follow-up: Changed 6 years ago by charpent

Replying to myself :

This is indeed an upstream bug. See the r-devel thread starting at this posting : it's ultimately a Makefile typo.

However, this thread says that :

Notice that in 3.3.x, the included xz & al. will disappear.

This might mean that the prerequisites for R may change at 3.3.0 time (which was announced to take place on 2016-04-11 last time I looked...).

Furthermore, r-devel also had this announcement (that I missed) :

The 3.2.4 release had two annoyances which we would rather not have in an "ultra-stable" release, designed to hang around for the duration of the 3.3 series. One was a relatively minor Makefile issue affecting system using R's bundled lzma library. The other, rather more serious, affected printing and formatting of POSIXlt objects, which would unpredictably get the Daylight Savings Time wrong.

A new tarball fixing this issue in 3.2.4 (+ another relevant to Posix time printing) has been released as http://cran.r-project.org/src/base/R-3/R-3.2.4-revised.tar.gz.

So it seems to me that the best "interim" solution is :

  • to get the "revised" tarball from CRAN
  • to document its renaming in spkg-src

and to revise the whole situation at R 3.3.0 release.

Unless I hear contrary opinions before Sun April 3, 2016, I'll do that (and also fix the patches fuzz, which is not incorrect but unaesthetic...).

Your opinions/advice urgently required...

comment:13 in reply to: ↑ 12 ; follow-up: Changed 6 years ago by jdemeyer

Replying to charpent:

  • to document its renaming in spkg-src

Which renaming?

comment:14 in reply to: ↑ 13 Changed 6 years ago by charpent

Replying to jdemeyer:

Replying to charpent:

  • to document its renaming in spkg-src

Which renaming?

R-3.2.4-revised.tar.gz ==> R-3.2.4.tar.gz

comment:15 follow-up: Changed 6 years ago by jdemeyer

-1 to renaming it. I know that 3.2.4-revised is a really silly version number, but that's what upstream chose, so Sage should just use that.

By the way, I wonder why upstream didn't just name it version 3.2.5.

comment:16 in reply to: ↑ 15 Changed 6 years ago by charpent

Replying to jdemeyer:

-1 to renaming it. I know that 3.2.4-revised is a really silly version number, but that's what upstream chose, so Sage should just use that.

Okay (unless I hear other opinions by Sunday). But that's really silly...

By the way, I wonder why upstream didn't just name it version 3.2.5.

De gustibus coloribusque...

comment:17 Changed 6 years ago by tscrim

  • Milestone changed from sage-7.1 to sage-7.2

My 2 cents are with Jeroen, let us try and keep it as close to upstream as we can.

comment:18 follow-up: Changed 6 years ago by charpent

I'm wondering how to do this : upstream has adopted a curious name for this release ; furthermore, it has done it inconsistently :

  • the tarball is named R-3.2.4-revised, but
  • the source tree is named R-revised.

Sage's build system chokes on this ; short of renaming the tarball and/or rebuilding it with a sensible source tree name, I don't see how to get Sage's building system accept this.

We can either :

  • rename/rebuild this tarball consistently via spkg-src, or
  • skip this one-week wonder entierely (last time I looked, R 3.3.0 was scheduled in Apr 11, 2016).

Your thoughts ?

comment:19 in reply to: ↑ 18 ; follow-up: Changed 6 years ago by jdemeyer

Replying to charpent:

Sage's build system chokes on this

Can you be more specific?

comment:20 in reply to: ↑ 19 ; follow-up: Changed 6 years ago by charpent

Replying to jdemeyer:

Replying to charpent:

Sage's build system chokes on this

Can you be more specific?

At the start of the package build log, I see that the script tries to mv to the source tree, but fails to stat it (the script nevertheless continues and fails other steps such as patching and MAKEing it.

comment:21 in reply to: ↑ 20 ; follow-up: Changed 6 years ago by jdemeyer

Replying to charpent:

At the start of the package build log, I see that the script tries to mv to the source tree, but fails to stat it

Full log please.

comment:22 in reply to: ↑ 21 Changed 6 years ago by charpent

Replying to jdemeyer:

Full log please.

This will have to wait : I'm not on my development machine, and have ti reproduce it on a (much slower) work machine...

Changed 6 years ago by charpent

(Unsuccessfull) compilation log of the "revised" package

comment:23 Changed 6 years ago by charpent

  • Description modified (diff)

As requested, log added.

HTH,

comment:24 follow-up: Changed 6 years ago by jdemeyer

Easy fix: replace the cd src in spkg-install by the correct directory name.

comment:25 in reply to: ↑ 24 ; follow-up: Changed 6 years ago by charpent

Replying to jdemeyer:

Easy fix: replace the cd src in spkg-install by the correct directory name.

Nope : src is a subdirectory of the source tree. The point is that the source tree name is nowhere to be seen in spkg-install : the current Sage building architecture is supposed to create a build directory for that.

It currently fails to do this in a useable way (see the top of the log).

comment:26 in reply to: ↑ 25 ; follow-up: Changed 6 years ago by jdemeyer

Replying to charpent:

Replying to jdemeyer:

Easy fix: replace the cd src in spkg-install by the correct directory name.

Nope : src is a subdirectory of the source tree. The point is that the source tree name is nowhere to be seen in spkg-install : the current Sage building architecture is supposed to create a build directory for that.

Nobody is forcing the directory to be called src. Like I said: just replace the cd src in spkg-install by the correct directory name and it will work.

comment:27 follow-up: Changed 6 years ago by tscrim

How much work do you think it will be to upgrade to 3.3.0? If you think it will take some work/time, then we should push this in. Otherwise, I am okay with waiting another week or so.

comment:28 in reply to: ↑ 26 ; follow-up: Changed 6 years ago by charpent

Replying to jdemeyer:

Replying to charpent:

Replying to jdemeyer:

Easy fix: replace the cd src in spkg-install by the correct directory name.

Nope : src is a subdirectory of the source tree. The point is that the source tree name is nowhere to be seen in spkg-install : the current Sage building architecture is supposed to create a build directory for that.

Nobody is forcing the directory to be called src. Like I said: just replace the cd src in spkg-install by the correct directory name and it will work.

Nope : the error at the top of the log happens before the start of spkg-install. spkg-install cannot intercept it.

And, again, the src names designates the directory src of the source tree (which should be called R-3.2.4-revised and not R-revised), and not the source tree itself.

I tried :

  • changing cd src to cd R-revised/src
  • adding (at the top of spkg-src) ln -s R-revised R-3.2.4-revised

to no avail.

The only way out I'm able to see would be to document in spkg-src :

  • untar R-3.2.4-revised.tar.gz
  • in the resulting tree, mv R-revised R-3.2.4-revised
  • retar the resulting tree

Do you really want that ?

comment:29 in reply to: ↑ 27 Changed 6 years ago by charpent

Replying to tscrim:

How much work do you think it will be to upgrade to 3.3.0? If you think it will take some work/time, then we should push this in. Otherwise, I am okay with waiting another week or so.

I do not know the Sage build system well enough to understand what is really needed to fix it. Short of fixing the (inconsistent) source tree, I see no way.

Since R.3.3.0 release (which may entail significant work) is less tan one week away, I'm a bit reluctant to produce a patch that will become irrelevant so soon...

comment:30 in reply to: ↑ 28 Changed 6 years ago by jdemeyer

Replying to charpent:

  • changing cd src to cd R-revised/src

You should change cd src to cd R-revised (or whatever the name is of the top-level directory in the R tarball).

comment:31 Changed 6 years ago by git

  • Commit changed from 385c51d8fc95817eec10780fa40723d9f02865e0 to feb037ac1e653436043ea1e76400fc052eda1119

Branch pushed to git repo; I updated commit sha1. New commits:

feb037aUpgrade R to 3.2.4-revised

comment:32 Changed 6 years ago by charpent

  • Priority changed from major to minor
  • Status changed from needs_work to needs_review

I do not (yet)understand how this works, but it worked.

Passes ptestlong with no error nor glitches ==> needs_review

comment:33 Changed 6 years ago by tscrim

  • Reviewers set to Travis Scrimshaw
  • Status changed from needs_review to positive_review

Ditto. Thanks.

comment:34 follow-up: Changed 6 years ago by vbraun

  • Status changed from positive_review to needs_work
Successfully installed r-3.2.4-revised.p0
Running the test suite for r-3.2.4-revised.p0...
./spkg-check: line 8: cd: src: No such file or directory
make[3]: Entering directory '/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/r-3.2.4-revised.p0'
make[3]: *** No rule to make target 'check'.
make[3]: Leaving directory '/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/r-3.2.4-revised.p0'
Error while running the R testsuite ... exiting

Changed 6 years ago by charpent

Log of a successfull compilation without testing.

comment:35 Changed 6 years ago by git

  • Commit changed from feb037ac1e653436043ea1e76400fc052eda1119 to 6b46d9d61e1480e46ab7f562ed0ed5a7639f5ef7

Branch pushed to git repo; I updated commit sha1. New commits:

6b46d9d(Semi-)fixed(?) test suite of R-3.2.4-revised release.

comment:36 in reply to: ↑ 34 Changed 6 years ago by charpent

  • Status changed from needs_work to needs_review

Replying to vbraun:

Damn ! I forgot even the existence of this...

Successfully installed r-3.2.4-revised.p0
Running the test suite for r-3.2.4-revised.p0...
./spkg-check: line 8: cd: src: No such file or directory
make[3]: Entering directory '/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/r-3.2.4-revised.p0'
make[3]: *** No rule to make target 'check'.
make[3]: Leaving directory '/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/r-3.2.4-revised.p0'
Error while running the R testsuite ... exiting

Huh ! I think that the fix is as trivial (and as mysterious) as the preceding one.

I tried it, but I can't seemingly test it. Whereas in my "develop" (7.2beta2 IIRC) setup, testing (of the current R-3.2.3) goes well, it does not start in this branch, which is based on 7.1rc0 :

After SAGE_CHECK_PACKAGES="yes" ./sage -f -c r, the build system rebuilds the packages, and says it has been built successfully. It does NOT attempt to test anything, nor does give an error (see uploaded log).

Nevertheless, I pushed the resulting patch (since it lo longer raises an error in the test suite...) and positioned needs_review. Feel free to send me to hell if you think that this cover-up is not acceptable.

comment:37 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to positive_review

It's running the test suite for me, not sure what you did wrong...

comment:38 Changed 6 years ago by jdemeyer

  • Reviewers changed from Travis Scrimshaw to Travis Scrimshaw, Jeroen Demeyer

comment:39 Changed 6 years ago by vbraun

  • Branch changed from u/charpent/upgrade_r_to_3_2_4 to 6b46d9d61e1480e46ab7f562ed0ed5a7639f5ef7
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.