Opened 8 years ago

Closed 8 years ago

#16299 closed enhancement (fixed)

upgrade numpy/scipy to 1.8.1 and 0.14

Reported by: fbissey Owned by:
Priority: major Milestone: sage-6.3
Component: packages: standard Keywords:
Cc: Merged in:
Authors: François Bissey Reviewers: Nathan Dunfield
Report Upstream: N/A Work issues:
Branch: 5a16124 (Commits, GitHub, GitLab) Commit: 5a161248dd34be17b790e79a00414354abf0311f
Dependencies: Stopgaps:

Status badges

Description (last modified by fbissey)

Upgrade numpy/scipy to latest versions. scipy 0.14 has finally ditched oldnumeric which means it won't generate warnings all over the place. A few doctests will have to be updated.

Package sources:

Change History (18)

comment:1 Changed 8 years ago by fbissey

  • Description modified (diff)

comment:2 Changed 8 years ago by fbissey

  • Branch set to numpy181

comment:3 Changed 8 years ago by fbissey

  • Created changed from 05/07/14 01:42:37 to 05/07/14 01:42:37
  • Description modified (diff)
  • Modified changed from 05/08/14 11:05:48 to 05/08/14 11:05:48

comment:4 Changed 8 years ago by fbissey

  • Branch changed from numpy181 to u/fbissey/numpy181
  • Commit set to 62c44eee3c74b71ce7a08357d68c8fcb707ed0ae

OK so it

comment:5 Changed 8 years ago by fbissey

  • Status changed from new to needs_review

Woopsie. Anyway I have updated the checksums and SPKG.txt for numpy and scipy. No changes where necessary to the packages themselves as far as I could tell. I fixed three doctests:

  • src/sage/functions/exp_integral.py: the last digit of a test changed, I put "..." since it is probably a little unstable.
  • src/sage/numerical/optimize.py: also a numerical precision thing. In this case we ask for a value within 0.0001 tolerance but tests the answer to 16 decimal places. No surprise when we have changes behind the 5th place.
  • src/sage/modules/vector_double_dense.pyx: a doctest warning has changed code.

comment:6 Changed 8 years ago by fbissey

  • Authors set to François Bissey

comment:7 Changed 8 years ago by dunfield

  • Status changed from needs_review to positive_review

I tried out this branch and the new packages with Sage 6.2 on two systems: OX Mavericks and RHEL 6.3 (gcc 4.4). Both numpy and scipy installed cleanly and the three modified doctests passed on both systems. Other changes in the patch look fine, so setting status to "positive review".

comment:8 Changed 8 years ago by dunfield

  • Reviewers set to Nathan Dunfield

comment:9 Changed 8 years ago by vbraun

  • Status changed from positive_review to needs_work

Breaks doctests on OSX (10.6 and 10.9):

sage -t --long src/sage/tests/french_book/calculus_doctest.py
**********************************************************************
File "src/sage/tests/french_book/calculus_doctest.py", line 216, in sage.tests.french_book.calculus_doctest
Failed example:
    find_root(expr, 0.1, pi)
Expected:
    2.0943951023931957
Got:
    2.0943951023931953

comment:10 Changed 8 years ago by fbissey

OK, will amend that. Any other failures I missed?

comment:11 Changed 8 years ago by git

  • Commit changed from 62c44eee3c74b71ce7a08357d68c8fcb707ed0ae to 5a161248dd34be17b790e79a00414354abf0311f

Branch pushed to git repo; I updated commit sha1. New commits:

5a16124Fix doctest failing on OS X on the last digit after upgarding numpy/scipy to 1.8.1/0.14.0

comment:12 Changed 8 years ago by fbissey

  • Status changed from needs_work to needs_review

OK fixed the doctest and pushed. Ready for another round of review.

comment:13 Changed 8 years ago by vbraun

I don't see any reason why this should give a different result on OSX. Do you?

comment:14 Changed 8 years ago by fbissey

Not sure. We are talking about a difference in the 4E-16 which is in the relative tolerance of the function and well within its own tolerance.

def find_root(f, a, b, xtol=10e-13, rtol=4.5e-16, maxiter=100, full_output=False):

But I would have been more ready to accept a difference based on CPU arch rather than OS on similar CPU.

Looking at scipy the underlying c code hasn't changed since 2008 but higher up tolerance checking has been enforced in 0.14rc2 (scipy/optimize/zeros.py) that's the only thing I can see having an impact.

comment:15 follow-up: Changed 8 years ago by dunfield

  • Status changed from needs_review to positive_review

I did a little testing, and definitely the issue is a result of the scipy upgrade and not the numpy one. It also shows up on OS X version 10.8 as well.

My experience is that the last digit in numerics like this can be remarkably sensitive to the OS/CPU/compiler version/phase of the moon. I once had two Ubuntu 12.04 virtual machines running on the same physical hardware where the last digit differed like this; one had a 32bit kernel and the other a 64bit one, which you wouldn't think would make a difference...

I think this sort of thing is to be expected, and it certainly seems harmless enough in this instance. So I've changed the status back to "positive review".

comment:16 in reply to: ↑ 15 Changed 8 years ago by fbissey

Replying to dunfield:

I did a little testing, and definitely the issue is a result of the scipy upgrade and not the numpy one. It also shows up on OS X version 10.8 as well.

My experience is that the last digit in numerics like this can be remarkably sensitive to the OS/CPU/compiler version/phase of the moon. I once had two Ubuntu 12.04 virtual machines running on the same physical hardware where the last digit differed like this; one had a 32bit kernel and the other a 64bit one, which you wouldn't think would make a difference...

Oh, yes I would. But then I have seen tickets where that happens in the last 6 years.

I think this sort of thing is to be expected, and it certainly seems harmless enough in this instance. So I've changed the status back to "positive review".

It is technically harmless it is however a curiosity because the change is sudden. If I had time I would check other versions of scipy to narrow down when the change happened and try to understand why.

comment:17 Changed 8 years ago by fbissey

In the end I spent the time checking https://github.com/scipy/scipy/commit/8edf03864b7c3e43e652ac154f98e96ee4913687 and https://github.com/scipy/scipy/commit/0ee24117ad63399b7a3353269de4e916fe409ad7 are responsible. I didn't try to pin it on one or the other. So it looks like we approch the root slightly differently between OS X and linux. My guess is that without the enforcement of tolerance OS X would go through one (or) more iteration(s) and end up with the same value as Linux.

comment:18 Changed 8 years ago by vbraun

  • Branch changed from u/fbissey/numpy181 to 5a161248dd34be17b790e79a00414354abf0311f
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.