Opened 9 years ago

Closed 9 years ago

#10887 closed task (fixed)

Upgrade scipy to 0.9

Reported by: jason Owned by: tbd
Priority: major Milestone: sage-4.7
Component: packages: standard Keywords:
Cc: kcrisman, rbeezer, fbissey Merged in: sage-4.7.alpha2
Authors: François Bissey Reviewers: Rob Beezer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

Scipy 0.9 was released a few days ago. We should upgrade. There was at least one person that was having problems with the scipy in Sage, and the fix is in 0.9.

This should preferably be merged along with numpy-1.5.1 (#10792).

Support for SAGE_SPKG_INSTALL_DOCS will be added later in a different ticket.

New spkg: http://spkg-upload.googlecode.com/files/scipy-0.9.spkg

Apply:

Attachments (1)

trac_10887-fix_matrix_double_dense_doctest.patch (882 bytes) - added by fbissey 9 years ago.
fix matrix_double_dense doctest

Download all attachments as: .zip

Change History (24)

comment:1 Changed 9 years ago by fbissey

  • Cc fbissey added

comment:2 Changed 9 years ago by fbissey

I will check if it breaks things shortly in sage-on-gentoo. I can make a spkg later, would be nice to have it alongside numpy-1.5.1 (although the later may suffer some delay).

comment:3 Changed 9 years ago by fbissey

I am still going through the test suite but I should give a few pieces of info I collected so far. It takes a _lot_ longer to compile. Not sure what landed but it must have been big. The jump is from 23mn to 1h05mn on this slow box. So far there has been two tests failures related to the upgrade, one in matrix_double_dense.py, a warning about complex conversion to real like in numpy-1.5.1. Possibly something has been switched to numpy internally but it will have to be digged out. The other is in sage/numerical/test.py some business with arpack which may turn out to be gentoo specific as we carry some arpack patches. While the test has not finished it seems like the amount of work needed will be minimal. I am planning to make a spkg in the next few days unless someone beats me to it and run another set of tests to check the arpack stuff.

comment:4 Changed 9 years ago by fbissey

After looking more carefully

sage -t -force_lib "devel/sage/sage/matrix/matrix_double_dense.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/matrix/matrix_double_dense.pyx", line 1527:
    sage: A.exp(algorithm='eig')
Expected:
    doctest:94: ComplexWarning: Casting complex values to real discards the imaginary part
    [51.9689561987  74.736564567]
    [112.104846851 164.073803049]
Got:
    [51.9689561987  74.736564567]
    [112.104846851 164.073803049]

The warning actually disappeared - easy!

comment:5 Changed 9 years ago by fbissey

sage -t  -long -force_lib "devel/sage/sage/numerical/test.py"
sage -t -long -force_lib "devel/sage/sage/numerical/test.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/numerical/test.py", line 12:
    sage: from scipy.sparse.linalg.eigen.arpack import speigs
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_0[10]>", line 1, in <module>
        from scipy.sparse.linalg.eigen.arpack import speigs###line 12:
    sage: from scipy.sparse.linalg.eigen.arpack import speigs
    ImportError: cannot import name speigs
**********************************************************************
File "/usr/share/sage/devel/sage/sage/numerical/test.py", line 27:
    sage: e,v=speigs.ARPACK_eigs(A.matvec,500,6,which='SM')
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_0[19]>", line 1, in <module>
        e,v=speigs.ARPACK_eigs(A.matvec,Integer(500),Integer(6),which='SM')###line 27:
    sage: e,v=speigs.ARPACK_eigs(A.matvec,500,6,which='SM')
    NameError: name 'speigs' is not defined
**********************************************************************
File "/usr/share/sage/devel/sage/sage/numerical/test.py", line 28:
    sage: e[0]*float(501/pi)**2
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_0[20]>", line 1, in <module>
        e[Integer(0)]*float(Integer(501)/pi)**Integer(2)###line 28:
    sage: e[0]*float(501/pi)**2
    TypeError: 'sage.symbolic.constants_c.E' object does not support indexing
**********************************************************************

From scipy-0.9 release notes http://docs.scipy.org/doc/scipy-0.9.0/reference/release.0.9.0.html this is probably a real problem as the arpack interface changed.

comment:6 follow-up: Changed 9 years ago by kcrisman

Interestingly, it now includes Qhull for triangulations. This is an optional Sage package, but maybe after this upgrade we wouldn't need it as that any more? (Of course, version rot could be a problem.)

comment:7 in reply to: ↑ 6 Changed 9 years ago by fbissey

Replying to kcrisman:

Interestingly, it now includes Qhull for triangulations. This is an optional Sage package, but maybe after this upgrade we wouldn't need it as that any more? (Of course, version rot could be a problem.)

I have looked a little bit more in depth at that. I noticed that in Gentoo scipy-0.9.0 now pulls qhull as a dependency. If there a version of libqhull.{so,a} on the system it will be used rather than the internally built static library. Said static library is not installed by scipy just rolled up in their created module.

In summary this version of scipy includes libqhull functionality (as far as scipy uses it at least) but doesn't provide the qhull library or any of the executables built by qhull normally.

Depending on what you want to do I think that means you don't want to remove qhull from the optional spkg unless you want to force access to its functionality through the scipy interface.

For info it also bundles superLU from experimental in the same way.

comment:8 Changed 9 years ago by fbissey

OK so my big doctest failure will turn out to be Gentoo specific because we don't have scipy_sandbox and relied on arpack functionality included in scipy. Arpack is also bundled in scipy so presumably we could switch to using

from scipy.sparse.linalg.eigen.arpack import arpack

instead of

from arpack import speigs

there are a few other things to clean afterwards but nothing serious. The functionality has been shipped in scipy for a while. The other part of scipy_sandbox (delaunay) can be provided by matplotlib so we could drop scipy_sandbox altogether but that would be another ticket.

I think this ticket should be merged at the same time or after numpy-1.5.1 (#10792) otherwise scipy will need artificial bumping when numpy is merged.

I should get on with that spkg, I had forgotten that I didn't make one.

comment:9 follow-up: Changed 9 years ago by fbissey

Here is the spkg http://spkg-upload.googlecode.com/files/scipy-0.9.spkg I actually have no idea on how to build the doc. Which is why I didn't add support for SAGE_SPKG_INSTALL_DOCS.

After testing I can confirm the issue with matrix_double_dense.pyx. numerical/test.py pass without problems as expected. So what needs to be done:

  • add support for installing the docs
  • patch matrix_double_dense.pyx

comment:10 in reply to: ↑ 9 Changed 9 years ago by kcrisman

  • Status changed from new to needs_work

After testing I can confirm the issue with matrix_double_dense.pyx. numerical/test.py pass without problems as expected. So what needs to be done:

  • add support for installing the docs

I would say that if this is nontrivial, it is better to get the new numpy and scipy in before next week's Sage Days, where many Numpy devels will be around. We can always add another patch and ticket for this later.

  • patch matrix_double_dense.pyx

Putting as 'needs work' for this.

comment:11 Changed 9 years ago by jason

+1 to getting the ticket in without delay, even if it doesn't have the doc-building thing.

Changed 9 years ago by fbissey

fix matrix_double_dense doctest

comment:12 Changed 9 years ago by fbissey

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Ok, I am attaching a patch and an updated description. It would be cool to have #10936 on board as well.

comment:13 follow-up: Changed 9 years ago by rbeezer

  • Authors set to François Bissey
  • Reviewers set to Rob beezer
  • Status changed from needs_review to positive_review

Patch to Sage library looks fine and applies to 4.7.alpha1.

New spkg installs and builds.

Combination of the two passes all tests with "make ptestlong".

This is the first time I've reviewed an spkg upgrade. Is there more to be done? If so, feel free to flip this back to "needs work."

François - thanks for putting this together. I'll go look at #10936 now, but I think the NumPy upgrade is beyond me right now.

comment:14 Changed 9 years ago by rbeezer

  • Reviewers changed from Rob beezer to Rob Beezer

comment:15 in reply to: ↑ 13 ; follow-ups: Changed 9 years ago by kcrisman

This is the first time I've reviewed an spkg upgrade. Is there more to be done? If so, feel free to flip this back to "needs work."

In theory, one should check if the spkg is properly formed, by unpacking it, reading any changes, and making sure that when one sage -pkgs it back up there are no red flags (e.g., unchecked changes).

François - thanks for putting this together. I'll go look at #10936 now, but I think the NumPy upgrade is beyond me right now.

It would be helpful to know if all these upgrades play well together, though, so that would helpful for finishing this ticket off.

comment:16 in reply to: ↑ 15 Changed 9 years ago by fbissey

Replying to kcrisman:

It would be helpful to know if all these upgrades play well together, though, so that would helpful for finishing this ticket off.

As far as I know they do. They are somewhat orthogonal and could be upgraded separately without trouble. But you want to have scipy rebuilt after a numpy upgrade for "safety".

They certainly play well together on sage-on-gentoo, I now have many copies of "vanilla" sage all over the place with various patches and upgrades. I am not sure I have one with both but it wouldn't surprise me.

comment:17 in reply to: ↑ 15 ; follow-up: Changed 9 years ago by kcrisman

Replying to kcrisman:

This is the first time I've reviewed an spkg upgrade. Is there more to be done? If so, feel free to flip this back to "needs work."

In theory, one should check if the spkg is properly formed, by unpacking it, reading any changes, and making sure that when one sage -pkgs it back up there are no red flags (e.g., unchecked changes).

Which is all fine.

comment:18 in reply to: ↑ 17 Changed 9 years ago by rbeezer

Replying to kcrisman:

Which is all fine.

Thanks, KDC. I figured maybe I should peek inside, but was unsure just what to look at. Next time.

I'm testing #10936 right now. I'll put this on top of it and see how that goes. but I'm going to stay away from the numpy ticket. Too many cooks already on that one. ;-)

comment:19 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:20 Changed 9 years ago by kcrisman

@jdemeyer - I've seen you make that change on a few patches. I'm pretty sure that Trac picks up the link and makes it active anyway (without the brackets); is there something additional that adding the brackets does? (Such as making patchbot actually read the Description? That would be convenient.)

comment:21 follow-up: Changed 9 years ago by jdemeyer

I guess you're right, but the brackets make the word "attachment:" disappear from the link. Compare:

comment:22 in reply to: ↑ 21 Changed 9 years ago by kcrisman

Replying to jdemeyer:

I guess you're right, but the brackets make the word "attachment:" disappear from the link. Compare:

Understood. Brackets it is!

comment:23 Changed 9 years ago by jdemeyer

  • Merged in set to sage-4.7.alpha2
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.