Opened 8 years ago
Closed 8 years ago
#10887 closed task (fixed)
Upgrade scipy to 0.9
Reported by: | jason | Owned by: | tbd |
---|---|---|---|
Priority: | major | Milestone: | sage-4.7 |
Component: | packages: standard | Keywords: | |
Cc: | kcrisman, rbeezer, fbissey | Merged in: | sage-4.7.alpha2 |
Authors: | François Bissey | Reviewers: | Rob Beezer |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
Scipy 0.9 was released a few days ago. We should upgrade. There was at least one person that was having problems with the scipy in Sage, and the fix is in 0.9.
This should preferably be merged along with numpy-1.5.1 (#10792).
Support for SAGE_SPKG_INSTALL_DOCS will be added later in a different ticket.
New spkg: http://spkg-upload.googlecode.com/files/scipy-0.9.spkg
Apply:
Attachments (1)
Change History (24)
comment:1 Changed 8 years ago by
- Cc fbissey added
comment:2 Changed 8 years ago by
comment:3 Changed 8 years ago by
I am still going through the test suite but I should give a few pieces of info I collected so far. It takes a _lot_ longer to compile. Not sure what landed but it must have been big. The jump is from 23mn to 1h05mn on this slow box. So far there has been two tests failures related to the upgrade, one in matrix_double_dense.py, a warning about complex conversion to real like in numpy-1.5.1. Possibly something has been switched to numpy internally but it will have to be digged out. The other is in sage/numerical/test.py some business with arpack which may turn out to be gentoo specific as we carry some arpack patches. While the test has not finished it seems like the amount of work needed will be minimal. I am planning to make a spkg in the next few days unless someone beats me to it and run another set of tests to check the arpack stuff.
comment:4 Changed 8 years ago by
After looking more carefully
sage -t -force_lib "devel/sage/sage/matrix/matrix_double_dense.pyx" ********************************************************************** File "/usr/share/sage/devel/sage/sage/matrix/matrix_double_dense.pyx", line 1527: sage: A.exp(algorithm='eig') Expected: doctest:94: ComplexWarning: Casting complex values to real discards the imaginary part [51.9689561987 74.736564567] [112.104846851 164.073803049] Got: [51.9689561987 74.736564567] [112.104846851 164.073803049]
The warning actually disappeared - easy!
comment:5 Changed 8 years ago by
sage -t -long -force_lib "devel/sage/sage/numerical/test.py" sage -t -long -force_lib "devel/sage/sage/numerical/test.py" ********************************************************************** File "/usr/share/sage/devel/sage/sage/numerical/test.py", line 12: sage: from scipy.sparse.linalg.eigen.arpack import speigs Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_0[10]>", line 1, in <module> from scipy.sparse.linalg.eigen.arpack import speigs###line 12: sage: from scipy.sparse.linalg.eigen.arpack import speigs ImportError: cannot import name speigs ********************************************************************** File "/usr/share/sage/devel/sage/sage/numerical/test.py", line 27: sage: e,v=speigs.ARPACK_eigs(A.matvec,500,6,which='SM') Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_0[19]>", line 1, in <module> e,v=speigs.ARPACK_eigs(A.matvec,Integer(500),Integer(6),which='SM')###line 27: sage: e,v=speigs.ARPACK_eigs(A.matvec,500,6,which='SM') NameError: name 'speigs' is not defined ********************************************************************** File "/usr/share/sage/devel/sage/sage/numerical/test.py", line 28: sage: e[0]*float(501/pi)**2 Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_0[20]>", line 1, in <module> e[Integer(0)]*float(Integer(501)/pi)**Integer(2)###line 28: sage: e[0]*float(501/pi)**2 TypeError: 'sage.symbolic.constants_c.E' object does not support indexing **********************************************************************
From scipy-0.9 release notes http://docs.scipy.org/doc/scipy-0.9.0/reference/release.0.9.0.html this is probably a real problem as the arpack interface changed.
comment:6 follow-up: ↓ 7 Changed 8 years ago by
Interestingly, it now includes Qhull for triangulations. This is an optional Sage package, but maybe after this upgrade we wouldn't need it as that any more? (Of course, version rot could be a problem.)
comment:7 in reply to: ↑ 6 Changed 8 years ago by
Replying to kcrisman:
Interestingly, it now includes Qhull for triangulations. This is an optional Sage package, but maybe after this upgrade we wouldn't need it as that any more? (Of course, version rot could be a problem.)
I have looked a little bit more in depth at that. I noticed that in Gentoo scipy-0.9.0 now pulls qhull as a dependency. If there a version of libqhull.{so,a} on the system it will be used rather than the internally built static library. Said static library is not installed by scipy just rolled up in their created module.
In summary this version of scipy includes libqhull functionality (as far as scipy uses it at least) but doesn't provide the qhull library or any of the executables built by qhull normally.
Depending on what you want to do I think that means you don't want to remove qhull from the optional spkg unless you want to force access to its functionality through the scipy interface.
For info it also bundles superLU from experimental in the same way.
comment:8 Changed 8 years ago by
OK so my big doctest failure will turn out to be Gentoo specific because we don't have scipy_sandbox and relied on arpack functionality included in scipy. Arpack is also bundled in scipy so presumably we could switch to using
from scipy.sparse.linalg.eigen.arpack import arpack
instead of
from arpack import speigs
there are a few other things to clean afterwards but nothing serious. The functionality has been shipped in scipy for a while. The other part of scipy_sandbox (delaunay) can be provided by matplotlib so we could drop scipy_sandbox altogether but that would be another ticket.
I think this ticket should be merged at the same time or after numpy-1.5.1 (#10792) otherwise scipy will need artificial bumping when numpy is merged.
I should get on with that spkg, I had forgotten that I didn't make one.
comment:9 follow-up: ↓ 10 Changed 8 years ago by
Here is the spkg http://spkg-upload.googlecode.com/files/scipy-0.9.spkg I actually have no idea on how to build the doc. Which is why I didn't add support for SAGE_SPKG_INSTALL_DOCS.
After testing I can confirm the issue with matrix_double_dense.pyx. numerical/test.py pass without problems as expected. So what needs to be done:
- add support for installing the docs
- patch matrix_double_dense.pyx
comment:10 in reply to: ↑ 9 Changed 8 years ago by
- Status changed from new to needs_work
After testing I can confirm the issue with matrix_double_dense.pyx. numerical/test.py pass without problems as expected. So what needs to be done:
- add support for installing the docs
I would say that if this is nontrivial, it is better to get the new numpy and scipy in before next week's Sage Days, where many Numpy devels will be around. We can always add another patch and ticket for this later.
- patch matrix_double_dense.pyx
Putting as 'needs work' for this.
comment:11 Changed 8 years ago by
+1 to getting the ticket in without delay, even if it doesn't have the doc-building thing.
comment:12 Changed 8 years ago by
- Description modified (diff)
- Status changed from needs_work to needs_review
Ok, I am attaching a patch and an updated description. It would be cool to have #10936 on board as well.
comment:13 follow-up: ↓ 15 Changed 8 years ago by
- Reviewers set to Rob beezer
- Status changed from needs_review to positive_review
Patch to Sage library looks fine and applies to 4.7.alpha1.
New spkg installs and builds.
Combination of the two passes all tests with "make ptestlong".
This is the first time I've reviewed an spkg upgrade. Is there more to be done? If so, feel free to flip this back to "needs work."
François - thanks for putting this together. I'll go look at #10936 now, but I think the NumPy
upgrade is beyond me right now.
comment:14 Changed 8 years ago by
- Reviewers changed from Rob beezer to Rob Beezer
comment:15 in reply to: ↑ 13 ; follow-ups: ↓ 16 ↓ 17 Changed 8 years ago by
This is the first time I've reviewed an spkg upgrade. Is there more to be done? If so, feel free to flip this back to "needs work."
In theory, one should check if the spkg is properly formed, by unpacking it, reading any changes, and making sure that when one sage -pkg
s it back up there are no red flags (e.g., unchecked changes).
François - thanks for putting this together. I'll go look at #10936 now, but I think the
NumPy
upgrade is beyond me right now.
It would be helpful to know if all these upgrades play well together, though, so that would helpful for finishing this ticket off.
comment:16 in reply to: ↑ 15 Changed 8 years ago by
Replying to kcrisman:
It would be helpful to know if all these upgrades play well together, though, so that would helpful for finishing this ticket off.
As far as I know they do. They are somewhat orthogonal and could be upgraded separately without trouble. But you want to have scipy rebuilt after a numpy upgrade for "safety".
They certainly play well together on sage-on-gentoo, I now have many copies of "vanilla" sage all over the place with various patches and upgrades. I am not sure I have one with both but it wouldn't surprise me.
comment:17 in reply to: ↑ 15 ; follow-up: ↓ 18 Changed 8 years ago by
Replying to kcrisman:
This is the first time I've reviewed an spkg upgrade. Is there more to be done? If so, feel free to flip this back to "needs work."
In theory, one should check if the spkg is properly formed, by unpacking it, reading any changes, and making sure that when one
sage -pkg
s it back up there are no red flags (e.g., unchecked changes).
Which is all fine.
comment:18 in reply to: ↑ 17 Changed 8 years ago by
Replying to kcrisman:
Which is all fine.
Thanks, KDC. I figured maybe I should peek inside, but was unsure just what to look at. Next time.
I'm testing #10936 right now. I'll put this on top of it and see how that goes. but I'm going to stay away from the numpy ticket. Too many cooks already on that one. ;-)
comment:19 Changed 8 years ago by
- Description modified (diff)
comment:20 Changed 8 years ago by
@jdemeyer - I've seen you make that change on a few patches. I'm pretty sure that Trac picks up the link and makes it active anyway (without the brackets); is there something additional that adding the brackets does? (Such as making patchbot actually read the Description? That would be convenient.)
comment:21 follow-up: ↓ 22 Changed 8 years ago by
I guess you're right, but the brackets make the word "attachment:" disappear from the link. Compare:
comment:22 in reply to: ↑ 21 Changed 8 years ago by
Replying to jdemeyer:
I guess you're right, but the brackets make the word "attachment:" disappear from the link. Compare:
Understood. Brackets it is!
comment:23 Changed 8 years ago by
- Merged in set to sage-4.7.alpha2
- Resolution set to fixed
- Status changed from positive_review to closed
I will check if it breaks things shortly in sage-on-gentoo. I can make a spkg later, would be nice to have it alongside numpy-1.5.1 (although the later may suffer some delay).