Opened 4 years ago

Closed 3 years ago

#17643 closed enhancement (fixed)

Update scipy to 0.16.1

Reported by: jpflori Owned by:
Priority: major Milestone: sage-6.10
Component: packages: standard Keywords:
Cc: Merged in:
Authors: François Bissey Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: cc6accc (Commits) Commit: cc6accce6a2b0622296f6c2c9f3f21395dfcc046
Dependencies: Stopgaps:

Description (last modified by fbissey)

Upgrade to scipy 0.16.1 this is intended to be done at the same time as numpy 1.10.1 and matplotlib 1.5.0 as noted in the ticket dependencies.

http://www.lmona.de/files/sage/scipy-0.16.1.tar.gz

Change History (36)

comment:1 Changed 4 years ago by fbissey

Didn't know it was out. Not in the Gentoo tree yet so I have no idea so far.

comment:2 Changed 4 years ago by fbissey

scipy 0.16.0 is out, one doctest broken:

sage -t --long usr/share/doc/sage/en/faq/faq-usage.rst
Running doctests with ID 2015-08-05-11-07-08-20037f0b.
Using --optional=optional,sage
Doctesting 1 file.
sage -t --long --warn-long 64.8 usr/share/doc/sage/en/faq/faq-usage.rst
**********************************************************************
File "usr/share/doc/sage/en/faq/faq-usage.rst", line 351, in faq-usage
Failed example:
    stats.ttest_ind(list([1,2,3,4,5]),list([2,3,4,5,.6]))
Expected:
    (array(0.07675295564533369), 0.94070490247380478)
Got:
    Ttest_indResult(statistic=array(0.07675295564533369), pvalue=0.94070490247380478)
**********************************************************************
1 item had failures:
   1 of  36 in faq-usage
    [24 tests, 1 failure, 0.68 s]
----------------------------------------------------------------------
sage -t --long --warn-long 64.8 usr/share/doc/sage/en/faq/faq-usage.rst  # 1 doctest failed
----------------------------------------------------------------------

this is on top of 6.9.beta0.

comment:3 Changed 3 years ago by fbissey

  • Description modified (diff)
  • Milestone changed from sage-6.5 to sage-6.10
  • Summary changed from Update scipy to 0.15 to Update scipy to 0.16.1

comment:4 Changed 3 years ago by fbissey

  • Authors set to François Bissey
  • Branch set to u/fbissey/scipy_0.16.1
  • Commit set to f3ff13922552d775a054cec05af925351370de44

New commits:

05c84c1upgrade to numpy 1.10.1, updating the package including a few cherrypicked patch from master
de8f740Merge branch 'develop' into numpy-1.10
ebdf899Fix to get rid of the problem with result_type. Pending upstream approval or suggestion of an alternative.
d12591fUpgarde matplotlib to 1.5.0
f3ff139Fix the doctest warning by adding it to the expected out. I am expecting the warning to originate from matplotlib and to be

comment:5 Changed 3 years ago by fbissey

  • Dependencies set to #17642 #19556

comment:6 Changed 3 years ago by fbissey

  • Description modified (diff)

comment:7 Changed 3 years ago by git

  • Commit changed from f3ff13922552d775a054cec05af925351370de44 to 31669f2db8ae1ae6484fb5654ab049caa418a15b

Branch pushed to git repo; I updated commit sha1. New commits:

31669f2Correct further warnings

comment:8 Changed 3 years ago by git

  • Commit changed from 31669f2db8ae1ae6484fb5654ab049caa418a15b to b21be98e75eb044998b061e5f8da47ddf6a810a3

Branch pushed to git repo; I updated commit sha1. New commits:

b21be98Removing line numbers from doctest

comment:9 Changed 3 years ago by fbissey

New branch compatible with the new push at #19556 coming soon.

comment:10 Changed 3 years ago by fbissey

  • Branch changed from u/fbissey/scipy_0.16.1 to u/fbissey/scipy-0.16.1
  • Commit changed from b21be98e75eb044998b061e5f8da47ddf6a810a3 to 6d59bad6f1aec57f7d0f4cb7b057a990d8ce3dcf

New commits:

7177516Upgrade matplotlib to 1.5.0
7ce23d0Fix deprecation warnings in doctests.
c7ac0a7Updated numpy patch according to feedback from upstream
52a45f4Merge branch 'numpy-1.10' into mpl-1.5.0
a733692remove test for a limitation that doesn't exist anymore.
6d59badUpgrade to scipy 0.16.1 and fix affected doctest

comment:11 Changed 3 years ago by fbissey

  • Status changed from new to needs_review

comment:12 Changed 3 years ago by vdelecroix

Just a confirmation that with the three tickets applied all tests pass (debian 64bits).

Could you replace

if np.array_equal(vert1, tpath.vertices) and np.array_equal(code1, tpath.codes):
     return True
else:
     return False

with

return np.array_equal(vert1, tpath.vertices) and np.array_equal(code1, tpath.codes)

Or there is a reason to let it as you did?

comment:13 follow-up: Changed 3 years ago by fbissey

No reason I didn't do that. I was so focused in finding a replacement for that particular line (it took some effort for me) that I didn't realise the code could be streamlined. I will make the change your suggest in a few hours. Would you otherwise be happy to review positively the tickets?

comment:14 Changed 3 years ago by git

  • Commit changed from 6d59bad6f1aec57f7d0f4cb7b057a990d8ce3dcf to 8b383de997ace1bc57564e4297c82b4b0a85a040

Branch pushed to git repo; I updated commit sha1. New commits:

04997f6Simplify flow in arrow.py
8b383deMerge branch 'mpl-1.5.0' into scipy0.16.1

comment:15 in reply to: ↑ 13 Changed 3 years ago by fbissey

Replying to fbissey:

I will make the change your suggest in a few hours. Would you otherwise be happy to review positively the tickets?

All done.

comment:16 Changed 3 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

comment:17 Changed 3 years ago by vbraun

  • Status changed from positive_review to needs_work

Fails on arm

gfortran: error: unrecognized command line option '-msse2'
error: Command "gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O2 -march=x86-64 -DMS_WIN64 -mtune=generic -msse2 -I/home/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/include -c -c scipy/fftpack/src/dfftpack/dfftf1.f -o build/temp.linux-armv7l-2.7/scipy/fftpack/src/dfftpack/dfftf1.o" failed with exit status 1

http://build.sagedev.org/release/builders/%20%20slow%20Oxford%20ARM%20%28Ubuntu%2012.10%29%20incremental/builds/327/steps/compile/logs/scipy

comment:18 Changed 3 years ago by vbraun

  • Dependencies changed from #17642 #19556 to #19556

comment:19 Changed 3 years ago by vbraun

  • Dependencies #19556 deleted

comment:20 Changed 3 years ago by fbissey

Serious fail from distutils

customize Gnu95FCompiler
customize Gnu95FCompiler using build_clib
building 'dfftpack' library
compiling Fortran sources
Fortran f77 compiler: gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O2 -march=x86-64 -DMS_WIN64 -mtune=generic -msse2
Fortran f90 compiler: gfortran -Wall -g -fno-second-underscore -fPIC -O2 -march=x86-64 -DMS_WIN64 -mtune=generic -msse2
Fortran fix compiler: gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O2 -march=x86-64 -DMS_WIN64 -mtune=generic -msse2

arm being mistaken for windows64?

comment:21 Changed 3 years ago by fbissey

Looking around distros that provide arm support I am wondering how it was building on arm before. I think none of fedora, debian or gentoo have trouble with arm but they all pass config_fc --noarch to setup.py which was done at some point in sage but has been removed a while ago.

I note from https://pkgs.fedoraproject.org/cgit/numpy.git/tree/numpy.spec

* Wed Oct 14 2015 Thomas Spura <tomspur@fedoraproject.org> - 1:1.10.1-3
- Remove fortran flags or arm would build with -march=x86-64

I am guessing from this that the bug is recent but because everyone but sage passes --noarch it did go unnoticed in scipy. I'll add it and put this back to need_review.

comment:22 Changed 3 years ago by git

  • Commit changed from 8b383de997ace1bc57564e4297c82b4b0a85a040 to d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0

Branch pushed to git repo; I updated commit sha1. New commits:

d40c0e9Adding "config_fc --noarch" to setup.py call so we don't get miscompilation on ARM.

comment:23 Changed 3 years ago by fbissey

  • Status changed from needs_work to needs_review

comment:24 Changed 3 years ago by vbraun

  • Status changed from needs_review to positive_review

Can you report it somewhere upstream?

comment:25 Changed 3 years ago by fbissey

I'll look into it. I am guessing it is technically a bug in numpy's distutils rather than scipy but I have no idea how to fix it or how it happens.

comment:26 Changed 3 years ago by fbissey

It looks like someone already acted in upstream numpy, I guess an option is to try to backport the commit in the numpy we ship. https://github.com/numpy/numpy/commit/a08437e12d6b6796f4b0beda49bce9adc37db63d

comment:27 follow-up: Changed 3 years ago by jpflori

I think I might have removed the no_arch stuff when I began to clean up the spkg mess. Backporting the fix looks a good idea for a further ticket. Or we just wait for a numpy version bump.

comment:28 in reply to: ↑ 27 Changed 3 years ago by fbissey

Replying to jpflori:

Backporting the fix looks a good idea for a further ticket. Or we just wait for a numpy version bump.

The danger with those two scenario is that the removal will be forgotten. However considering that everyone else has it with some variations (Gentoo has noopt and fedora has fcompiler=gunf95 [or something]) I guess it doesn't hurt to keep it.

comment:29 Changed 3 years ago by vbraun

  • Branch changed from u/fbissey/scipy-0.16.1 to d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:30 Changed 3 years ago by fbissey

  • Commit d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0 deleted

I must say I spent a bit of my night worrying that it was the wrong fix. Good that it really works.

comment:31 Changed 3 years ago by vbraun

  • Branch changed from d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0 to u/fbissey/scipy-0.16.1
  • Commit set to d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0
  • Resolution fixed deleted
  • Status changed from closed to new

Argh still doesn't work... http://build.sagedev.org/release/builders/%20%20slow%20Oxford%20ARM%20%28Ubuntu%2012.10%29%20incremental/builds/330/steps/compile/logs/scipy


Last 10 new commits:

ebdf899Fix to get rid of the problem with result_type. Pending upstream approval or suggestion of an alternative.
7177516Upgrade matplotlib to 1.5.0
7ce23d0Fix deprecation warnings in doctests.
c7ac0a7Updated numpy patch according to feedback from upstream
52a45f4Merge branch 'numpy-1.10' into mpl-1.5.0
a733692remove test for a limitation that doesn't exist anymore.
6d59badUpgrade to scipy 0.16.1 and fix affected doctest
04997f6Simplify flow in arrow.py
8b383deMerge branch 'mpl-1.5.0' into scipy0.16.1
d40c0e9Adding "config_fc --noarch" to setup.py call so we don't get miscompilation on ARM.

comment:32 Changed 3 years ago by fbissey

OK so I was right to worry it was the wrong fix. I'll remove the last commit on the scipy branch and add the upstream commit in numpy. I guess if I had gone for noopt instead of noarch it would have worked.

comment:33 Changed 3 years ago by git

  • Commit changed from d40c0e9a97ca9d2a0a05b7596ab84d368ea97bc0 to 93a8c3d8da5af43e14d2cbb63b3df77da9aa0c06

Branch pushed to git repo; I updated commit sha1. New commits:

93a8c3dRevert "Adding "config_fc --noarch" to setup.py call so we don't get miscompilation on ARM."

comment:34 Changed 3 years ago by git

  • Commit changed from 93a8c3d8da5af43e14d2cbb63b3df77da9aa0c06 to cc6accce6a2b0622296f6c2c9f3f21395dfcc046

Branch pushed to git repo; I updated commit sha1. New commits:

9c6e94cAdd upstream patch to reverse inclusion of mingwin64 support. It cause ARM to be taken for win64!
414fccfMerge branch 'numpy-1.10' into mpl-1.5.0
cc6acccMerge branch 'mpl-1.5.0' into scipy0.16.1

comment:35 Changed 3 years ago by fbissey

  • Status changed from new to needs_review

comment:36 Changed 3 years ago by vbraun

  • Branch changed from u/fbissey/scipy-0.16.1 to cc6accce6a2b0622296f6c2c9f3f21395dfcc046
  • Resolution set to fixed
  • Status changed from needs_review to closed
Note: See TracTickets for help on using tickets.