Opened 4 years ago
Closed 4 years ago
#17643 closed enhancement (fixed)
Update scipy to 0.16.1
Reported by:  jpflori  Owned by:  

Priority:  major  Milestone:  sage6.10 
Component:  packages: standard  Keywords:  
Cc:  Merged in:  
Authors:  François Bissey  Reviewers:  Volker Braun 
Report Upstream:  N/A  Work issues:  
Branch:  cc6accc (Commits)  Commit:  cc6accce6a2b0622296f6c2c9f3f21395dfcc046 
Dependencies:  Stopgaps: 
Description (last modified by )
Upgrade to scipy 0.16.1 this is intended to be done at the same time as numpy 1.10.1 and matplotlib 1.5.0 as noted in the ticket dependencies.
Change History (36)
comment:1 Changed 4 years ago by
comment:2 Changed 4 years ago by
scipy 0.16.0 is out, one doctest broken:
sage t long usr/share/doc/sage/en/faq/faqusage.rst Running doctests with ID 2015080511070820037f0b. Using optional=optional,sage Doctesting 1 file. sage t long warnlong 64.8 usr/share/doc/sage/en/faq/faqusage.rst ********************************************************************** File "usr/share/doc/sage/en/faq/faqusage.rst", line 351, in faqusage Failed example: stats.ttest_ind(list([1,2,3,4,5]),list([2,3,4,5,.6])) Expected: (array(0.07675295564533369), 0.94070490247380478) Got: Ttest_indResult(statistic=array(0.07675295564533369), pvalue=0.94070490247380478) ********************************************************************** 1 item had failures: 1 of 36 in faqusage [24 tests, 1 failure, 0.68 s]  sage t long warnlong 64.8 usr/share/doc/sage/en/faq/faqusage.rst # 1 doctest failed 
this is on top of 6.9.beta0.
comment:3 Changed 4 years ago by
 Description modified (diff)
 Milestone changed from sage6.5 to sage6.10
 Summary changed from Update scipy to 0.15 to Update scipy to 0.16.1
comment:4 Changed 4 years ago by
 Branch set to u/fbissey/scipy_0.16.1
 Commit set to f3ff13922552d775a054cec05af925351370de44
New commits:
05c84c1  upgrade to numpy 1.10.1, updating the package including a few cherrypicked patch from master

de8f740  Merge branch 'develop' into numpy1.10

ebdf899  Fix to get rid of the problem with result_type. Pending upstream approval or suggestion of an alternative.

d12591f  Upgarde matplotlib to 1.5.0

f3ff139  Fix the doctest warning by adding it to the expected out. I am expecting the warning to originate from matplotlib and to be

comment:5 Changed 4 years ago by
 Dependencies set to #17642 #19556
comment:6 Changed 4 years ago by
 Description modified (diff)
comment:7 Changed 4 years ago by
 Commit changed from f3ff13922552d775a054cec05af925351370de44 to 31669f2db8ae1ae6484fb5654ab049caa418a15b
Branch pushed to git repo; I updated commit sha1. New commits:
31669f2  Correct further warnings

comment:8 Changed 4 years ago by
 Commit changed from 31669f2db8ae1ae6484fb5654ab049caa418a15b to b21be98e75eb044998b061e5f8da47ddf6a810a3
Branch pushed to git repo; I updated commit sha1. New commits:
b21be98  Removing line numbers from doctest

comment:9 Changed 4 years ago by
New branch compatible with the new push at #19556 coming soon.
comment:10 Changed 4 years ago by
 Branch changed from u/fbissey/scipy_0.16.1 to u/fbissey/scipy0.16.1
 Commit changed from b21be98e75eb044998b061e5f8da47ddf6a810a3 to 6d59bad6f1aec57f7d0f4cb7b057a990d8ce3dcf
New commits:
7177516  Upgrade matplotlib to 1.5.0

7ce23d0  Fix deprecation warnings in doctests.

c7ac0a7  Updated numpy patch according to feedback from upstream

52a45f4  Merge branch 'numpy1.10' into mpl1.5.0

a733692  remove test for a limitation that doesn't exist anymore.

6d59bad  Upgrade to scipy 0.16.1 and fix affected doctest

comment:11 Changed 4 years ago by
 Status changed from new to needs_review
comment:12 Changed 4 years ago by
Just a confirmation that with the three tickets applied all tests pass (debian 64bits).
Could you replace
if np.array_equal(vert1, tpath.vertices) and np.array_equal(code1, tpath.codes): return True else: return False
with
return np.array_equal(vert1, tpath.vertices) and np.array_equal(code1, tpath.codes)
Or there is a reason to let it as you did?
comment:13 followup: ↓ 15 Changed 4 years ago by
No reason I didn't do that. I was so focused in finding a replacement for that particular line (it took some effort for me) that I didn't realise the code could be streamlined. I will make the change your suggest in a few hours. Would you otherwise be happy to review positively the tickets?
comment:14 Changed 4 years ago by
 Commit changed from 6d59bad6f1aec57f7d0f4cb7b057a990d8ce3dcf to 8b383de997ace1bc57564e4297c82b4b0a85a040
comment:15 in reply to: ↑ 13 Changed 4 years ago by
Replying to fbissey:
I will make the change your suggest in a few hours. Would you otherwise be happy to review positively the tickets?
All done.
comment:16 Changed 4 years ago by
 Reviewers set to Volker Braun
 Status changed from needs_review to positive_review
comment:17 Changed 4 years ago by
 Status changed from positive_review to needs_work
Fails on arm
gfortran: error: unrecognized command line option 'msse2' error: Command "gfortran Wall g ffixedform fnosecondunderscore fPIC O2 march=x8664 DMS_WIN64 mtune=generic msse2 I/home/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/numpy/core/include c c scipy/fftpack/src/dfftpack/dfftf1.f o build/temp.linuxarmv7l2.7/scipy/fftpack/src/dfftpack/dfftf1.o" failed with exit status 1
comment:18 Changed 4 years ago by
 Dependencies changed from #17642 #19556 to #19556
comment:19 Changed 4 years ago by
 Dependencies #19556 deleted
comment:20 Changed 4 years ago by
Serious fail from distutils
customize Gnu95FCompiler customize Gnu95FCompiler using build_clib building 'dfftpack' library compiling Fortran sources Fortran f77 compiler: gfortran Wall g ffixedform fnosecondunderscore fPIC O2 march=x8664 DMS_WIN64 mtune=generic msse2 Fortran f90 compiler: gfortran Wall g fnosecondunderscore fPIC O2 march=x8664 DMS_WIN64 mtune=generic msse2 Fortran fix compiler: gfortran Wall g ffixedform fnosecondunderscore Wall g fnosecondunderscore fPIC O2 march=x8664 DMS_WIN64 mtune=generic msse2
arm being mistaken for windows64?
comment:21 Changed 4 years ago by
Looking around distros that provide arm
support I am wondering how it was building on arm before. I think none of fedora, debian or gentoo have trouble with arm but they all pass config_fc noarch
to setup.py
which was done at some point in sage but has been removed a while ago.
I note from https://pkgs.fedoraproject.org/cgit/numpy.git/tree/numpy.spec
* Wed Oct 14 2015 Thomas Spura <tomspur@fedoraproject.org>  1:1.10.13  Remove fortran flags or arm would build with march=x8664
I am guessing from this that the bug is recent but because everyone but sage
passes noarch
it did go unnoticed in scipy
. I'll add it and put this back to need_review.
comment:22 Changed 4 years ago by
 Commit changed from 8b383de997ace1bc57564e4297c82b4b0a85a040 to d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0
Branch pushed to git repo; I updated commit sha1. New commits:
d40c0e9  Adding "config_fc noarch" to setup.py call so we don't get miscompilation on ARM.

comment:23 Changed 4 years ago by
 Status changed from needs_work to needs_review
comment:24 Changed 4 years ago by
 Status changed from needs_review to positive_review
Can you report it somewhere upstream?
comment:25 Changed 4 years ago by
I'll look into it. I am guessing it is technically a bug in numpy
's distutils rather than scipy
but I have no idea how to fix it or how it happens.
comment:26 Changed 4 years ago by
It looks like someone already acted in upstream numpy
, I guess an option is to try to backport the commit in the numpy
we ship.
https://github.com/numpy/numpy/commit/a08437e12d6b6796f4b0beda49bce9adc37db63d
comment:27 followup: ↓ 28 Changed 4 years ago by
I think I might have removed the no_arch
stuff when I began to clean up the spkg mess.
Backporting the fix looks a good idea for a further ticket.
Or we just wait for a numpy version bump.
comment:28 in reply to: ↑ 27 Changed 4 years ago by
Replying to jpflori:
Backporting the fix looks a good idea for a further ticket. Or we just wait for a numpy version bump.
The danger with those two scenario is that the removal will be forgotten. However considering that everyone else has it with some variations (Gentoo has noopt
and fedora has fcompiler=gunf95
[or something]) I guess it doesn't hurt to keep it.
comment:29 Changed 4 years ago by
 Branch changed from u/fbissey/scipy0.16.1 to d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0
 Resolution set to fixed
 Status changed from positive_review to closed
comment:30 Changed 4 years ago by
 Commit d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0 deleted
I must say I spent a bit of my night worrying that it was the wrong fix. Good that it really works.
comment:31 Changed 4 years ago by
 Branch changed from d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0 to u/fbissey/scipy0.16.1
 Commit set to d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0
 Resolution fixed deleted
 Status changed from closed to new
Argh still doesn't work... http://build.sagedev.org/release/builders/%20%20slow%20Oxford%20ARM%20%28Ubuntu%2012.10%29%20incremental/builds/330/steps/compile/logs/scipy
Last 10 new commits:
ebdf899  Fix to get rid of the problem with result_type. Pending upstream approval or suggestion of an alternative.

7177516  Upgrade matplotlib to 1.5.0

7ce23d0  Fix deprecation warnings in doctests.

c7ac0a7  Updated numpy patch according to feedback from upstream

52a45f4  Merge branch 'numpy1.10' into mpl1.5.0

a733692  remove test for a limitation that doesn't exist anymore.

6d59bad  Upgrade to scipy 0.16.1 and fix affected doctest

04997f6  Simplify flow in arrow.py

8b383de  Merge branch 'mpl1.5.0' into scipy0.16.1

d40c0e9  Adding "config_fc noarch" to setup.py call so we don't get miscompilation on ARM.

comment:32 Changed 4 years ago by
OK so I was right to worry it was the wrong fix. I'll remove the last commit on the scipy branch and add the upstream commit in numpy
. I guess if I had gone for noopt
instead of noarch
it would have worked.
comment:33 Changed 4 years ago by
 Commit changed from d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0 to 93a8c3d8da5af43e14d2cbb63b3df77da9aa0c06
Branch pushed to git repo; I updated commit sha1. New commits:
93a8c3d  Revert "Adding "config_fc noarch" to setup.py call so we don't get miscompilation on ARM."

comment:34 Changed 4 years ago by
 Commit changed from 93a8c3d8da5af43e14d2cbb63b3df77da9aa0c06 to cc6accce6a2b0622296f6c2c9f3f21395dfcc046
comment:35 Changed 4 years ago by
 Status changed from new to needs_review
comment:36 Changed 4 years ago by
 Branch changed from u/fbissey/scipy0.16.1 to cc6accce6a2b0622296f6c2c9f3f21395dfcc046
 Resolution set to fixed
 Status changed from needs_review to closed
Didn't know it was out. Not in the Gentoo tree yet so I have no idea so far.