Opened 9 years ago
Last modified 7 years ago
#11334 closed task
Update numpy to 1.7.0 — at Version 88
Reported by: | jason | Owned by: | tbd |
---|---|---|---|
Priority: | major | Milestone: | sage-5.10 |
Component: | packages: standard | Keywords: | sd40.5 |
Cc: | fbissey, kini, Snark | Merged in: | |
Authors: | Reviewers: | ||
Report Upstream: | Fixed upstream, in a later stable release. | Work issues: | |
Branch: | Commit: | ||
Dependencies: | #12415, #13992 | Stopgaps: |
Description (last modified by )
Numpy-1.7.0 will be available soon and will fix the coercion problem that prevented us to go to numpy-1.6.x.
New spkg: http://spkg-upload.googlecode.com/files/numpy-1.7.0.rc2.spkg
Apply:
Change History (92)
comment:1 Changed 9 years ago by
- Cc fbissey added
comment:2 Changed 9 years ago by
Thx. it looks like numpy has been bumped to 1.6 in Gentoo so I can do some quick compatibility tests. It looks like a flag for gfortran has been added in the ebuild to help with the compiler autodetection madness it may be worth having it sage side too.
Do you know if a scipy release is to follow, if not it would be best to revbump scipy to make sure it is rebuilt after numpy.
Other matter there has been noise about lapack in numpy do we want to disable it, make it optional or keep the status quo?
comment:3 Changed 9 years ago by
I don't remember seeing anything on the scipy lists about an imminent scipy release.
comment:4 Changed 9 years ago by
just in:
sage -t -long -force_lib "devel/sage/doc/en/numerical_sage/numpy.rst" ********************************************************************** File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 97: sage: 2.5*l Expected: array([ 0. , 2.5, 5. , 7.5, 10. , 12.5, 15. , 17.5, 20. , 22.5]) Got: array([0.000000000000000, 2.50000000000000, 5.00000000000000, 7.50000000000000, 10.0000000000000, 12.5000000000000, 15.0000000000000, 17.5000000000000, 20.0000000000000, 22.5000000000000], dtype=object) ********************************************************************** File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 289: sage: xx Expected: array([[ 0. , 0.25, 0.5 , 0.75, 1. ], [ 0. , 0.25, 0.5 , 0.75, 1. ], [ 0. , 0.25, 0.5 , 0.75, 1. ], [ 0. , 0.25, 0.5 , 0.75, 1. ], [ 0. , 0.25, 0.5 , 0.75, 1. ]]) Got: array([[0.000000000000000, 0.250000000000000, 0.500000000000000, 0.750000000000000, 1.00000000000000], [0.000000000000000, 0.250000000000000, 0.500000000000000, 0.750000000000000, 1.00000000000000], [0.000000000000000, 0.250000000000000, 0.500000000000000, 0.750000000000000, 1.00000000000000], [0.000000000000000, 0.250000000000000, 0.500000000000000, 0.750000000000000, 1.00000000000000], [0.000000000000000, 0.250000000000000, 0.500000000000000, 0.750000000000000, 1.00000000000000]], dtype=object) ********************************************************************** File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 295: sage: yy Expected: array([[ 0. , 0. , 0. , 0. , 0. ], [ 0.25, 0.25, 0.25, 0.25, 0.25], [ 0.5 , 0.5 , 0.5 , 0.5 , 0.5 ], [ 0.75, 0.75, 0.75, 0.75, 0.75], [ 1. , 1. , 1. , 1. , 1. ]]) Got: array([[0.000000000000000, 0.000000000000000, 0.000000000000000, 0.000000000000000, 0.000000000000000], [0.250000000000000, 0.250000000000000, 0.250000000000000, 0.250000000000000, 0.250000000000000], [0.500000000000000, 0.500000000000000, 0.500000000000000, 0.500000000000000, 0.500000000000000], [0.750000000000000, 0.750000000000000, 0.750000000000000, 0.750000000000000, 0.750000000000000], [1.00000000000000, 1.00000000000000, 1.00000000000000, 1.00000000000000, 1.00000000000000]], dtype=object) ********************************************************************** File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 301: sage: f(xx,yy) Expected: array([[ 0. , 0.0625, 0.25 , 0.5625, 1. ], [ 0.0625, 0.125 , 0.3125, 0.625 , 1.0625], [ 0.25 , 0.3125, 0.5 , 0.8125, 1.25 ], [ 0.5625, 0.625 , 0.8125, 1.125 , 1.5625], [ 1. , 1.0625, 1.25 , 1.5625, 2. ]]) Got: array([[0.000000000000000, 0.0625000000000000, 0.250000000000000, 0.562500000000000, 1.00000000000000], [0.0625000000000000, 0.125000000000000, 0.312500000000000, 0.625000000000000, 1.06250000000000], [0.250000000000000, 0.312500000000000, 0.500000000000000, 0.812500000000000, 1.25000000000000], [0.562500000000000, 0.625000000000000, 0.812500000000000, 1.12500000000000, 1.56250000000000], [1.00000000000000, 1.06250000000000, 1.25000000000000, 1.56250000000000, 2.00000000000000]], dtype=object) ********************************************************************** 2 items had failures: 1 of 55 in __main__.example_0 3 of 13 in __main__.example_1 ***Test Failed*** 4 failures.
there may be more
comment:5 Changed 9 years ago by
- Description modified (diff)
I have cut a new spkg for 1.6.0, I decided to go for plain update for now.
comment:6 Changed 9 years ago by
seems my home install has problems. Probably stuff messed around between python 2.6 and 2.7 because I am working on #9958. Another install didn't report any problems. So I will check on a vanilla sage but I think we won't need to do any patching.
comment:7 Changed 9 years ago by
For some reason the other install didn't upgrade that's why I didn't see anything on the second machine. So right now it looks like we have the following to correct:
sage -t -long -force_lib devel/sage/doc/en/numerical_sage/numpy.rst # 4 doctests failed sage -t -long -force_lib devel/sage/sage/symbolic/function.pyx # 1 doctests failed sage -t -long -force_lib devel/sage/sage/plot/plot_field.py # 1 doctests failed sage -t -long -force_lib devel/sage/sage/plot/matrix_plot.py # 1 doctests failed sage -t -long -force_lib devel/sage/sage/matrix/matrix1.pyx # 1 doctests failed sage -t -long -force_lib devel/sage/sage/numerical/optimize.py # 1 doctests failed sage -t -long -force_lib devel/sage/sage/rings/integer.pyx # 1 doctests failed sage -t -long -force_lib devel/sage/sage/functions/hyperbolic.py # 2 doctests failed sage -t -long -force_lib devel/sage/sage/functions/trig.py # 3 doctests failed sage -t -long -force_lib devel/sage/sage/functions/other.py # 2 doctests failed
The first one is already in the ticket, then we have
sage -t -long -force_lib "devel/sage/sage/symbolic/function.pyx" ********************************************************************** File "/usr/share/sage/devel/sage/sage/symbolic/function.pyx", line 596: sage: csc(a) Expected: array([ inf, 1.18839511, 1.09975017, 7.0861674 , -1.32134871]) Got: doctest:270: RuntimeWarning: divide by zero encountered in divide array([ inf, 1.18839511, 1.09975017, 7.0861674 , -1.32134871]) **********************************************************************
sage -t -long -force_lib "devel/sage/sage/plot/plot_field.py" ********************************************************************** File "/usr/share/sage/devel/sage/sage/plot/plot_field.py", line 216: sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2)).show(aspect_ratio=1) Expected nothing Got: doctest:622: RuntimeWarning: divide by zero encountered in divide doctest:623: RuntimeWarning: invalid value encountered in multiply doctest:624: RuntimeWarning: invalid value encountered in multiply **********************************************************************
sage -t -long -force_lib "devel/sage/sage/plot/matrix_plot.py" ********************************************************************** File "/usr/share/sage/devel/sage/sage/plot/matrix_plot.py", line 383: sage: matrix_plot([[sin(x), cos(x)], [1, 0]]) Expected: Traceback (most recent call last): ... ValueError: can not convert entries to floating point numbers Got: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_8[36]>", line 1, in <module> matrix_plot([[sin(x), cos(x)], [Integer(1), Integer(0)]])###line 383: sage: matrix_plot([[sin(x), cos(x)], [1, 0]]) File "/usr/lib/python2.7/site-packages/sage/misc/decorators.py", line 381, in wrapper return func(*args, **kwds) File "/usr/lib/python2.7/site-packages/sage/misc/decorators.py", line 432, in wrapper return func(*args, **options) File "/usr/lib/python2.7/site-packages/sage/plot/matrix_plot.py", line 423, in matrix_plot raise TypeError, "mat must be a Matrix or a two dimensional array" TypeError: mat must be a Matrix or a two dimensional array **********************************************************************
sage -t -long -force_lib "devel/sage/sage/matrix/matrix1.pyx" ********************************************************************** File "/usr/share/sage/devel/sage/sage/matrix/matrix1.pyx", line 468: sage: sorted(numpy.typecodes.items()) Expected: [('All', '?bhilqpBHILQPfdgFDGSUVO'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')] Got: [('All', '?bhilqpBHILQPefdgFDGSUVOMm'), ('AllFloat', 'efdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'efdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')] **********************************************************************
sage -t -long -force_lib "devel/sage/sage/numerical/optimize.py" ********************************************************************** File "/usr/share/sage/devel/sage/sage/numerical/optimize.py", line 162: sage: find_minimum_on_interval(f, 1, 5, tol=1e-3) Expected: (-3.28837136189098..., 3.42575079030572...) Got: (-3.28837136189, 3.4257507903057229)
sage -t -long -force_lib "devel/sage/sage/rings/integer.pyx" ********************************************************************** File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4738: sage: numpy.array(2**40).dtype Expected: dtype('int64') Got: dtype('object') **********************************************************************
sage -t -long -force_lib "devel/sage/sage/functions/hyperbolic.py" ********************************************************************** File "/usr/share/sage/devel/sage/sage/functions/hyperbolic.py", line 609: sage: arcsech(a) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_23[4]>", line 1, in <module> arcsech(a)###line 609: sage: arcsech(a) File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794) File "/usr/lib/python2.7/site-packages/sage/functions/hyperbolic.py", line 612, in _eval_numpy_ return arccosh(1.0 / x) ZeroDivisionError: float division by zero ********************************************************************** File "/usr/share/sage/devel/sage/sage/functions/hyperbolic.py", line 658: sage: arccsch(a) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_26[4]>", line 1, in <module> arccsch(a)###line 658: sage: arccsch(a) File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794) File "/usr/lib/python2.7/site-packages/sage/functions/hyperbolic.py", line 661, in _eval_numpy_ return arcsinh(1.0 / x) ZeroDivisionError: float division by zero **********************************************************************
sage -t -long -force_lib "devel/sage/sage/functions/trig.py" ********************************************************************** File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 827: sage: atan2(a, b) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_34[16]>", line 1, in <module> atan2(a, b)###line 827: sage: atan2(a, b) File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475) File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794) File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568) NotImplementedError: The Function arctan2 does not support numpy arrays as arguments ********************************************************************** File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 830: sage: atan2(1,a) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_34[17]>", line 1, in <module> atan2(Integer(1),a)###line 830: sage: atan2(1,a) File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475) File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794) File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568) NotImplementedError: The Function arctan2 does not support numpy arrays as arguments ********************************************************************** File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 833: sage: atan2(a, 1) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_34[18]>", line 1, in <module> atan2(a, Integer(1))###line 833: sage: atan2(a, 1) File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475) File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794) File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568) NotImplementedError: The Function arctan2 does not support numpy arrays as arguments **********************************************************************
sage -t -long -force_lib "devel/sage/sage/functions/other.py" ********************************************************************** File "/usr/share/sage/devel/sage/sage/functions/other.py", line 211: sage: ceil(a) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_5[20]>", line 1, in <module> ceil(a)###line 211: sage: ceil(a) File "/usr/lib/python2.7/site-packages/sage/functions/other.py", line 251, in __call__ return numpy.ceil(x) AttributeError: ceil ********************************************************************** File "/usr/share/sage/devel/sage/sage/functions/other.py", line 366: sage: floor(a) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_9[15]>", line 1, in <module> floor(a)###line 366: sage: floor(a) File "/usr/lib/python2.7/site-packages/sage/functions/other.py", line 405, in __call__ return numpy.floor(x) AttributeError: floor **********************************************************************
The last three are particularly concerning, could you check what's up with these in numpy?
comment:8 Changed 9 years ago by
This seems to have installed fine on OS X 10.4 PPC. I'm running doctests now (which will take a LONG time) but I did already see the matrix/matrix1.pyx one while testing something else.
comment:9 Changed 9 years ago by
That's good that it at least install. On gentoo there are interactions problem between numpy and scipy to get cblas/blas/lapack right in scipy. I should check if the resulting scipy in vanilla sage suffers from the same problem. http://bugs.gentoo.org/show_bug.cgi?id=371099
The last few failing tests are show stoppers as far as I am concerned.
comment:10 Changed 9 years ago by
Okay, I got pretty much the same failures on a long run. One weird thing... did we change find_minimum_on_interval
away from SciPy recently? This is the full output I get on the (otherwise same as your failure) test for that.
sage: find_minimum_on_interval(f, 1, 5, tol=1e-3) Expected: (-3.28837136189098..., 3.42575079030572...) Got: (-3.28837136189, 3.4257507903057229) GLPK Simplex Optimizer, v4.44 6 rows, 3 columns, 8 non-zeros Preprocessing... 2 rows, 2 columns, 4 non-zeros Scaling... A: min|aij| = 2.400e+01 max|aij| = 5.000e+01 ratio = 2.083e+00 GM: min|aij| = 8.128e-01 max|aij| = 1.230e+00 ratio = 1.514e+00 EQ: min|aij| = 6.606e-01 max|aij| = 1.000e+00 ratio = 1.514e+00 Constructing initial basis... Size of triangular part = 2 * 0: obj = -5.100000000e+01 infeas = 0.000e+00 (0) * 1: obj = -5.225000000e+01 infeas = 0.000e+00 (0) OPTIMAL SOLUTION FOUND
comment:11 Changed 9 years ago by
1.6.1 is now available. I'll have a look sometimes this week.
comment:12 Changed 9 years ago by
Looks like more of the same but maybe a little bit more. There are a few other things that may interfere on my setup and I had to solve that bit: http://bugs.gentoo.org/show_bug.cgi?id=371099 before doing the trial.
comment:13 Changed 9 years ago by
Once I had cleared a few blas problem that I had while solving the gentoo bug. I get exactly the same results. I also get the output from Karl in optimize. This is a verbose output of glpk which get called at some point. You only get to see the such verbose output if there was a failure in the test but not necessarily the test for which glpk is called funnily enough.
comment:14 Changed 9 years ago by
Should we try to move forward on this again? I will have a go at trying it with 4.8.alpha2. Any ideas about the current failures?
comment:15 Changed 9 years ago by
Ok this is still broken in 4.8.alpha2. Some food for thought
sage ---------------------------------------------------------------------- | Sage Version 4.8.alpha2, Release Date: 2011-11-19 | | Type notebook() for the GUI, and license() for information. | ---------------------------------------------------------------------- ********************************************************************** * * * Warning: this is a prerelease version, and it may be unstable. * * * ********************************************************************** sage: import numpy sage: a = numpy.linspace(0,2,6) sage: a array([0.0, 0.4, 0.8, 1.2, 1.6, 2], dtype=object) sage: ceil(a) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /home/fbissey/<ipython console> in <module>() /usr/lib64/python2.7/site-packages/sage/functions/other.py in __call__(self, x, maximum_bits) 250 elif type(x).__module__ == 'numpy': 251 import numpy --> 252 return numpy.ceil(x) 253 254 x_original = x AttributeError: ceil
with numpy-1.5.1 we just have
sage: a array([0.0, 0.4, 0.8, 1.2, 1.6, 2])
I suspect that for this one (functions/other.py) and functions/trig.py the pynac interface may need to be touched. All stuff that takes an numpy array and return a numpy array may have trouble.
comment:16 Changed 9 years ago by
Just to add a bit of info. Same python and ipython
ipython Python 2.7.2 (default, Oct 28 2011, 11:36:16) Type "copyright", "credits" or "license" for more information. IPython 0.10.2 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]: import numpy In [2]: a = numpy.linspace(0,2,6) In [3]: a Out[3]: array([ 0. , 0.4, 0.8, 1.2, 1.6, 2. ]) In [4]: ceil(a) --------------------------------------------------------------------------- NameError Traceback (most recent call last) /home/fbissey/<ipython console> in <module>() NameError: name 'ceil' is not defined In [5]: numpy.ceil(a) Out[5]: array([ 0., 1., 1., 2., 2., 2.])
So it works in ipython but we don't have the dtype=object attached to the array. Anyone knows what that means actually?
comment:17 Changed 9 years ago by
numpy-1.5.1
sage: numpy.dtype(a[1]) dtype('float64')
numpy-1.6.1
sage: numpy.dtype(a[1]) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/fbissey/<ipython console> in <module>() TypeError: data type not understood
comment:18 Changed 9 years ago by
sage: type(a[0]) <type 'float'> sage: type(a[0:5]) <type 'numpy.ndarray'> sage: type(a[2]) <type 'float'> sage: type(a[3]) <type 'float'> sage: type(a[4]) <type 'float'> sage: type(a[5]) <type 'sage.rings.integer.Integer'>
For some reason numpy.linspace return an array with sage objects rather than just floats. I am guessing it may be deeper than that. Numpy probably uses sage types rather than converting to numpy types. The types of a[0] to a[4] is strange, is there a sage type for floats? I mean I tried numpy.linspace(0,2.0,6) and I got <type 'sage.rings.real_mpfr.RealNumber?'> for a[0] to a[4] and <type 'sage.rings.real_mpfr.RealLiteral?'> for a[5].
Doing a cast numpy.array(a,dtype=float) returns something that will work.
comment:19 Changed 8 years ago by
- Description modified (diff)
- Summary changed from Update numpy to 1.6 to Update numpy to 1.6.1
comment:20 Changed 8 years ago by
I'll be starting to make patches to correct the doctests and document the new behavior in numpy 1.6.x. Some documentation may need updating.
comment:21 Changed 8 years ago by
Actually I am a bit confused by one of the tests:
``plot_slope_field`` takes a function of two variables xvar and yvar (for instance, if the variables are `x` and `y`, take `f(x,y)`), and at representative points `(x_i,y_i)` between xmin, xmax, and ymin, ymax respectively, plots a line with slope `f(x_i,y_i)` (see below). ``plot_slope_field(f, (xvar, xmin, xmax), (yvar, ymin, ymax))`` EXAMPLES: A logistic function modeling population growth:: sage: x,y = var('x y') sage: capacity = 3 # thousand sage: growth_rate = 0.7 # population increases by 70% per unit of time sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2))
That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be.
comment:22 Changed 8 years ago by
I am guessing it should have been
growth_rate*(1-x/capacity)*y
y being the time and x the population in thousands. That satisfy a simple dimensional analysis.
comment:23 Changed 8 years ago by
No, wasn't it right before? That's the standard logistic population model, where y is the population. See http://en.wikipedia.org/wiki/Logistic_function#In_ecology:_modeling_population_growth, for example.
comment:24 Changed 8 years ago by
sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2)) }}}} > That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in This should be ok. The point is that the slope dy/dx is a function of y only; this can be completely symbolically solved using separation of variables to get the [http://en.wikipedia.org/wiki/Logistic_function#In_ecology:_modeling_population_growth logistic function]. > this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be. If this is the division by zero stuff, it's the whole business with the arrowheads having zero length. See #11208. I think Jason has a fix for this, actually, but I don't recall the whole story of what happened once it was reported to mpl.
comment:25 Changed 8 years ago by
Sorry for that noise. Jason is right.
sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2))
That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in
This should be ok. The point is that the slope dy/dx is a function of y only; this can be completely symbolically solved using separation of variables to get the logistic function.
this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be.
If this is the division by zero stuff, it's the whole business with the arrowheads having zero length. See #11208. I think Jason has a fix for this, actually, but I don't recall the whole story of what happened once it was reported to mpl.
comment:26 Changed 8 years ago by
OK I was interpreting the formula another way because I wasn't sure what it is actually computing. Seems like #11208 numpy, mpl and scipy all go hands in hands so if something start happening with one the others may get it too.
Overall the change of behavior of numpy around type conversion is quite annoying.
comment:27 Changed 8 years ago by
It strikes me that the problem may be more an issue of the coercion to numpy types not working properly with numpy-1.6. I will look into that.
comment:28 Changed 8 years ago by
Any ideas why stuff like this is happening with numpy-1.6.x
sage: numpy.arange(10.0) array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) sage: numpy.linspace(0.0,9.0,10) array([0.000000000000000, 1.00000000000000, 2.00000000000000, 3.00000000000000, 4.00000000000000, 5.00000000000000, 6.00000000000000, 7.00000000000000, 8.00000000000000, 9.00000000000000], dtype=object)
comment:29 follow-up: ↓ 33 Changed 8 years ago by
It looks like it didn't automatically convert from RR to float, like it should. That's what these two lines should do in sage/rings/real_mpfr.pyx
cdef object numpy_double_interface = {'typestr': '=f8'} cdef object numpy_object_interface = {'typestr': '|O'}
and this property in that same file:
property __array_interface__: def __get__(self): """ Used for NumPy conversion. EXAMPLES:: sage: import numpy sage: numpy.arange(10.0) array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) sage: numpy.array([1.0, 1.1, 1.2]).dtype dtype('float64') sage: numpy.array([1.000000000000000000000000000000000000]).dtype dtype('object') """ if (<RealField_class>self._parent).__prec <= 57: # max size of repr(float) return numpy_double_interface else: return numpy_object_interface
So maybe numpy doesn't do something with __array_interface__
anymore in linspace?
Can you try numpy.linspace(0.0,9.0,float(10))
, numpy.linspace(0.0,float(9),10)
, numpy.linspace(float(0),9.0,10)
and various other combinations of float vs. Sage types in the arguments to narrow down what is triggering it?
comment:30 Changed 8 years ago by
I was writing something else:
And more subtly:
sage: a=numpy.linspace(0,9,10) sage: a array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9], dtype=object) sage: type(a[0]) <type 'float'> sage: type(a[9]) <type 'sage.rings.integer.Integer'>
So linspace and arange deal with sage input very differently. There should be some coercion but it is not working properly.
now for what you are asking:
sage: numpy.linspace(0.0,9.0,float(10)) array([0.000000000000000, 1.00000000000000, 2.00000000000000, 3.00000000000000, 4.00000000000000, 5.00000000000000, 6.00000000000000, 7.00000000000000, 8.00000000000000, 9.00000000000000], dtype=object) sage: numpy.linspace(0.0,float(9),10) array([0.000000000000000, 1.00000000000000, 2.00000000000000, 3.00000000000000, 4.00000000000000, 5.00000000000000, 6.00000000000000, 7.00000000000000, 8.00000000000000, 9.0], dtype=object) sage: numpy.linspace(float(0),9.0,10) array([0.000000000000000, 1.00000000000000, 2.00000000000000, 3.00000000000000, 4.00000000000000, 5.00000000000000, 6.00000000000000, 7.00000000000000, 8.00000000000000, 9.00000000000000], dtype=object)
I know about sage/rings/real_mpfr.pyx and the others that's how I got on thinking of coercion not working.
comment:31 Changed 8 years ago by
Weirder! The code for linspace uses arange inside. No change in the code between the 1.5.1 and 1.6.1 but the juicy important bit is as follow
step = (stop-start)/float(num) y = _nx.arange(0, num) * step + start
step will become float (unless start or stop is complex or something). But the type of step and start will influence the final array.
comment:32 Changed 8 years ago by
Anyone wants to ask on a numpy mailing list if it is a sign of bug in numpy or something that is clearly under-documented?
comment:33 in reply to: ↑ 29 Changed 8 years ago by
Replying to jason:
So maybe numpy doesn't do something with
__array_interface__
anymore in linspace?
I've been tinkering here and this is exactly what is happening. I deleted all the
property __array_interface__:
blocks that were added by 5081-numpy-types.patch and 6506-numpy-types.patch with no apparent effect on what numpy.linspace() returns with numpy-1.6.1 installed. Of course, the deleted blocks do affect what numpy.arange() and numpy.array() return:
sage: numpy.arange(10.0) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object) sage: numpy.array([1.0, 1.1, 1.2]).dtype dtype('object')
Also the example failure
from scipy import stats stats.uniform(0,15).ppf([0.5,0.7])
from ticket #5081 now again fails with numpy-1.6.x (s-o-g result):
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) /storage/strogdon/gentoo/usr/local/portage/sage-on-gentoo/<ipython console> in <module>() /storage/strogdon/gentoo/usr/lib/python2.7/site-packages/scipy/stats/distributions.pyc in ppf(self, q) 449 450 def ppf(self, q): --> 451 return self.dist.ppf(q, *self.args, **self.kwds) 452 453 def isf(self, q): /storage/strogdon/gentoo/usr/lib/python2.7/site-packages/scipy/stats/distributions.pyc in ppf(self, q, *args, **kwds) 1514 goodargs = argsreduce(cond, *((q,)+args+(scale,loc))) 1515 scale, loc, goodargs = goodargs[-2], goodargs[-1], goodargs[:-2] -> 1516 place(output,cond,self._ppf(*goodargs)*scale + loc) 1517 if output.ndim == 0: 1518 return output[()] /storage/strogdon/gentoo/usr/lib/python2.7/site-packages/numpy/lib/function_base.pyc in place(arr, mask, vals) 1333 1334 """ -> 1335 return _insert(arr, mask, vals) 1336 1337 def _nanop(op, fill, a, axis=None): TypeError: array cannot be safely cast to required type
Starting with numpy-1.6.0 scalar upcasting rules were changed http://docs.scipy.org/doc/numpy/reference/ufuncs.html and I'm wondering if this change is interfering with how sage/cython is handling the python __array_interface__ extension, at least relative to numpy.linspace(), and perhaps other numpy functions that are pure python.
comment:34 Changed 8 years ago by
That's the best clue we had in ages Steve. I am not sure when I will be able to dig it further as we have an extra long week end here (Tuesday is a university holyday).
comment:35 Changed 8 years ago by
Here is a minimal example of the problem:
sage: import numpy sage: numpy.array(0.5) array(0.500000000000000, dtype=object) sage: numpy.array(0.5r) array(0.5)
where as before it was
sage: import numpy sage: numpy.array(0.5) array(0.5) sage: numpy.array(0.5r) array(0.5)
comment:36 Changed 8 years ago by
Also, we should incorporate #7831
comment:37 Changed 8 years ago by
Sure Mike do you have any idea on how to fix the coercion problem? The way it's going we'll update directly to 1.7.
comment:38 Changed 8 years ago by
Looking into this it seems that the issue is that in numpy/core/src/multiarray/ctors.c
, PyArray_FromAny
no longer calls _array_find_type
which I believe was responsible for accessing the __array_interface__
attribute on our objects. I'm sending a message to numpy-discussion now.
comment:39 follow-up: ↓ 48 Changed 8 years ago by
comment:40 Changed 8 years ago by
- Keywords sd40.5 added
comment:41 Changed 8 years ago by
Thanks Mike, hopefully we'll get a meaningful answer.
comment:42 Changed 8 years ago by
Looks like 1.6.2 was released last week. I guess I'll update my spkg.
comment:43 Changed 8 years ago by
Looks like I found the commit where the change has been made https://github.com/numpy/numpy/commit/2635398db3f26529ce2aaea4028a8118844f3c48
If you look carefully the call to _array_find_type is still there but it has changed location. In 1.5.1 there was a sequence if(xx==NULL) then call _array_find_type else do some other stuff. In 1.6.1 the structure if(xx!=NULL) then do the other stuff else call _array_find_type.
I will try to write a patch to see if this the cause of our observed behaviour.
comment:44 Changed 8 years ago by
I wrote a small patch to just ignore any code other than _array_find_type in ctors.c no luck so I don't think that's the real source of the problem at all. I thought the presence of a requested_dtype passed to this function could have been the source of the problem but my patch excludes that possibility entirely. So we have to go back to looking.
comment:45 Changed 8 years ago by
Problem still present with numpy-1.6.2
comment:46 Changed 8 years ago by
- Description modified (diff)
New spkg for 1.6.2 rebased on 1.5.1.p1 as well.
comment:47 Changed 8 years ago by
- Summary changed from Update numpy to 1.6.1 to Update numpy to 1.6.2
Mike, I think if you want to ask a question that may have a chance to attract attention on numpy-discussion you should ask: Why do arange and linspace return different dtype object on similar inputs of the same type?
You can then expand on the fact that arange which is completely coded in C respect our request from array interface but linspace which call arange and then do something in python with the result doesn't respect it.
comment:48 in reply to: ↑ 39 Changed 8 years ago by
Replying to mhansen:
See http://mail.scipy.org/pipermail/numpy-discussion/2012-May/062560.html
I believe Mike is correct in that the commit referenced at the above numpy-discussion is the source of the "regression" in being unable to access the Sage-assigned __array_interface__ attribute. I cloned the numpy.git repository, set the master to the commit mentioned my Mike, and from that commit generated a numpy spkg which I installed. Sage manifested the undesirable "dtype=object" when tested. Now if I set the master to the commit just prior to the above mentioned commit and generated a new numpy spkg, then when Sage was tested I got the numpy-1.5.1 behavior. I'm just not sure what's going inside numpy?
comment:49 Changed 8 years ago by
OK so this is indeed the guilty commit. I tried something very simple to see if it was the small bit of code around _array_find_dtype and it was obviously insufficient to uncover the problem. It is quite a big commit,. I am thinking that considering we see an inconsistent behavior that we should just open a bug with numpy.
comment:50 Changed 8 years ago by
- Report Upstream changed from N/A to Reported upstream. No feedback yet.
Thanks Mike for the post on numpy's github tracker https://github.com/numpy/numpy/issues/291 I am not sure how they distribute the load between the github tracker and their trac server http://projects.scipy.org/numpy/report. I just hope they don't ignore it.
comment:51 Changed 8 years ago by
- Report Upstream changed from Reported upstream. No feedback yet. to Reported upstream. Developers acknowledge bug.
It looks like numpy may revert to the old behavior in 1.7.0. There is some traction upstream for this to happen.
comment:52 Changed 8 years ago by
- Report Upstream changed from Reported upstream. Developers acknowledge bug. to Fixed upstream, in a later stable release.
Looks like 1.7.0 will have a fix: https://github.com/numpy/numpy/issues/291. 1.7.0 should be out very soon now.
comment:53 Changed 8 years ago by
I know and I am waiting for that.
comment:54 Changed 8 years ago by
I have been toying with numpy 1.7.0b2 in sage-on-gentoo. I have added a small patch I discussed with Steve Trogdon that I will attach. In the meantime I have gobbles of the following:
sage -t -long -force_lib "devel/sage-main/sage/functions/log.py" ********************************************************************** File "/usr/share/sage/devel/sage-main/sage/functions/log.py", line 630: sage: lambert_w(RDF(1)) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_12[4]>", line 1, in <module> lambert_w(RDF(Integer(1)))###line 630: sage: lambert_w(RDF(1)) File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 560, in __call__ return BuiltinFunction.__call__(self, 0, args[0], **kwds) File "function.pyx", line 437, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:5006) File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 607, in _eval_ return self._evalf_(n, z, parent=sage_structure_coerce_parent(z)) File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 636, in _evalf_ return scipy.special.lambertw(z, n) File "lambertw.pyx", line 342, in scipy.special.lambertw.lambertw (scipy/special/lambertw.c:1438) TypeError: expected a readable buffer object ********************************************************************** 1 items had failures: 1 of 6 in __main__.example_12 ***Test Failed*** 1 failures. For whitespace errors, see the file /home/fbissey/.sage/tmp/log_1583.py [2.6 s]
Anything using matplotlib spams you to death with these. Without the patch we get the following instead:
sage -t -long -force_lib "devel/sage-main/sage/functions/log.py" ********************************************************************** File "/usr/share/sage/devel/sage-main/sage/functions/log.py", line 630: sage: lambert_w(RDF(1)) Exception raised: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_12[4]>", line 1, in <module> lambert_w(RDF(Integer(1)))###line 630: sage: lambert_w(RDF(1)) File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 560, in __call__ return BuiltinFunction.__call__(self, 0, args[0], **kwds) File "function.pyx", line 437, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:5006) File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 607, in _eval_ return self._evalf_(n, z, parent=sage_structure_coerce_parent(z)) File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 636, in _evalf_ return scipy.special.lambertw(z, n) File "lambertw.pyx", line 342, in scipy.special.lambertw.lambertw (scipy/special/lambertw.c:1438) ValueError: Missing __array_interface__ shape ********************************************************************** 1 items had failures: 1 of 6 in __main__.example_12 ***Test Failed*** 1 failures. For whitespace errors, see the file /home/fbissey/.sage/tmp/log_15682.py [2.6 s]
It seems that the patch introduce some expectation on the type that are not identified at the present time.
comment:55 Changed 8 years ago by
- Description modified (diff)
Here is a spkg so other people can enjoy. numpy 1.7 also has a new API. By default you can use the old one but we may want to move on to the new one eventually.
comment:56 Changed 8 years ago by
Steve pointed out to me that 1.7.0b2 didn't contain 2 little patches that we need and that will hopefully be in 1.7.0 final. The spkg is updated and there are way fewer doctest failures (matplotlib 1.2.0 will be a major source of doctest noise when we upgrade). So the current failures: 1) cosmetics
sage -t -long -force_lib "devel/sage-main/sage/matrix/matrix1.pyx" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/matrix/matrix1.pyx", line 494: sage: sorted(numpy.typecodes.items()) Expected: [('All', '?bhilqpBHILQPfdgFDGSUVO'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')] Got: [('All', '?bhilqpBHILQPefdgFDGSUVOMm'), ('AllFloat', 'efdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'efdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')] ********************************************************************** 1 items had failures:
2) lots of divide by 0 warnings - including our old friend riemann.pyx encountered in a previous upgrade.
sage -t -long -force_lib "devel/sage-main/sage/calculus/riemann.pyx" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 111: sage: m = Riemann_Map([f], [fprime], 0) # long time (4 sec) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 595: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 647: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 712: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 801: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 867: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 163: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 214: sage: isinstance(Riemann_Map([f], [fprime], 0)._repr_(), str) # long time Expected: True Got: doctest:1: RuntimeWarning: divide by zero encountered in divide True ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 228: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 334: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 404: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 447: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 515: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 543: sage: m = Riemann_Map([f], [fprime], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide **********************************************************************
sage -t -long -force_lib "devel/sage-main/sage/calculus/interpolators.pyx" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/interpolators.pyx", line 52: sage: m = Riemann_Map([lambda x: ps.value(real(x))], [lambda x: ps.derivative(real(x))],0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/interpolators.pyx", line 183: sage: m = Riemann_Map([lambda x: cs.value(real(x))], [lambda x: cs.derivative(real(x))], 0) Expected nothing Got: doctest:1: RuntimeWarning: divide by zero encountered in divide **********************************************************************
sage -t -long -force_lib "devel/sage-main/sage/symbolic/function.pyx" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/symbolic/function.pyx", line 627: sage: csc(a) Expected: array([ inf, 1.18839511, 1.09975017, 7.0861674 , -1.32134871]) Got: doctest:270: RuntimeWarning: divide by zero encountered in divide array([ inf, 1.18839511, 1.09975017, 7.0861674 , -1.32134871]) **********************************************************************
sage -t -long -force_lib "devel/sage-main/sage/functions/hyperbolic.py" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/functions/hyperbolic.py", line 610: sage: arcsech(a) Expected: array([ inf, 1.3169579, 0. ]) Got: doctest:613: RuntimeWarning: divide by zero encountered in divide array([ inf, 1.3169579, 0. ]) ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/functions/hyperbolic.py", line 659: sage: arccsch(a) Expected: array([ inf, 1.44363548, 0.88137359]) Got: doctest:662: RuntimeWarning: divide by zero encountered in divide array([ inf, 1.44363548, 0.88137359]) **********************************************************************
sage -t -long -force_lib "devel/sage-main/sage/modules/vector_double_dense.pyx" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/modules/vector_double_dense.pyx", line 712: sage: w.norm(p=-1.6) Expected: 0.0 Got: doctest:1992: RuntimeWarning: divide by zero encountered in power 0.0 **********************************************************************
3) odd bits that needs looking at:
sage -t -long -force_lib "devel/sage-main/sage/rings/real_mpfr.pyx" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/rings/real_mpfr.pyx", line 1270: sage: numpy.array([1.000000000000000000000000000000000000]).dtype Expected: dtype('object') Got: dtype('O') **********************************************************************
and similarly in rings/integer.pyx, rings/complex_number.pyx and rings/rational.pyx.
4) ????
sage -t -long -force_lib "devel/sage-main/sage/plot/matrix_plot.py" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/plot/matrix_plot.py", line 438: sage: matrix_plot([[sin(x), cos(x)], [1, 0]]) Expected: Traceback (most recent call last): ... ValueError: can not convert entries to floating point numbers Got: Traceback (most recent call last): File "/usr/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_8[40]>", line 1, in <module> matrix_plot([[sin(x), cos(x)], [Integer(1), Integer(0)]])###line 438: sage: matrix_plot([[sin(x), cos(x)], [1, 0]]) File "/usr/lib64/python2.7/site-packages/sage/misc/decorators.py", line 456, in wrapper return func(*args, **kwds) File "/usr/lib64/python2.7/site-packages/sage/misc/decorators.py", line 456, in wrapper return func(*args, **kwds) File "/usr/lib64/python2.7/site-packages/sage/misc/decorators.py", line 534, in wrapper return func(*args, **options) File "/usr/lib64/python2.7/site-packages/sage/plot/matrix_plot.py", line 478, in matrix_plot raise TypeError, "mat must be a Matrix or a two dimensional array" TypeError: mat must be a Matrix or a two dimensional array **********************************************************************
and a last one which I am not sure is numpy or sage-on-gentoo or a combination:
sage -t -long -force_lib "devel/sage-main/sage/matrix/matrix_double_dense.pyx" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/matrix/matrix_double_dense.pyx", line 4033: sage: A.exp(order=2) Expected: [51.8888631634 74.6198348038] [111.929752206 163.818615369] Got: [51.9689561987 74.736564567] [112.104846851 164.073803049] ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/matrix/matrix_double_dense.pyx", line 4052: sage: A.exp(order=2) Expected: [-19.6130852955 + 12.5327938535*I 3.81156364812 + 28.891438232*I] [-32.3827876895 + 21.9087393169*I 2.29565402142 + 44.915581543*I] Got: [-19.6146029538 + 12.5177438468*I 3.79496364496 + 28.8837993066*I] [-32.3835809809 + 21.8842359579*I 2.26963300409 + 44.9013248277*I] **********************************************************************
comment:57 Changed 8 years ago by
I forgot that one which is another that puzzle me
sage -t -long -force_lib "devel/sage-main/sage/plot/plot3d/implicit_surface.pyx" ********************************************************************** File "/usr/share/sage/devel/sage-main/sage/plot/plot3d/implicit_surface.pyx", line 563: sage: cube_marcher.y_vertices.any() or cube_marcher.z_vertices.any() # This shouldn't affect the Y or Z vertices. Expected: False Got nothing ********************************************************************** File "/usr/share/sage/devel/sage-main/sage/plot/plot3d/implicit_surface.pyx", line 458: sage: cube_marcher.x_vertices.any() # This shouldn't affect the X vertices. Expected: False Got nothing **********************************************************************
And the matrix_double_dense.pyx filures seem to have disappeared after my last rebuild.
comment:58 Changed 8 years ago by
- Description modified (diff)
- Summary changed from Update numpy to 1.6.2 to Update numpy to 1.7.0
Steve pointed out that the array_interface-shape.patch patch is not necessary as the old numpy 1.5.1 behavior as been restored in full. It doesn't hurt either way but we don't need to add something we don't need.
comment:59 Changed 8 years ago by
After pulling the latest code from 1.7.x maintenance branch I can now see that the patch is still necessary. The pull that Steve had mentioned to me happened on the master not on the 1.7.x maintenance branch so it is not in the current 1.7 branch on github.
comment:60 Changed 8 years ago by
Sorry about the last message after more tests my conclusions are:
- the current tip of the 1.7.x maintenance branch as well as 1.7.0b2 need two extra patches (included in the current spkg)
- the array_interface-shape.patch patch is not necessary after all.
During my early tests I was expecting the two numpy patch to have been merged in the 1.7.x maintenance branch and that lead to confusion.
comment:61 Changed 8 years ago by
Does the rst patch above need to be updated to not talk about the 1.6 bug of converting Sage types?
comment:62 Changed 8 years ago by
At this stage there are no patch to add to sage the rst patch was something I started before I realized that we should do something else. I will put any patch I create for sage in the summary. There are a at least a couple of things we can fix now. The big question in my mind is: what do we do with all those divide by 0 warnings? Last time we had the author of riemann.pyx fix things.
comment:63 Changed 8 years ago by
I'll try taking a look at it. I don't have the most recent Sage yet, but I guess it's not a huge problem right now since numpy 1.7 hasn't been released yet either.
Changed 8 years ago by
tentative fix for matrix_plot.py - not sure it is the right thing to do as we have a change of behavior rather than output.
comment:64 Changed 8 years ago by
- Description modified (diff)
OK for you to have a look in the meantime here are the easy patches. Not completely sure it is the right thing to do in matrix_plot.py. While we are expecting an error we were expecting a different kind of error before.
comment:65 Changed 8 years ago by
I'm unable to get the extra patches included in numpy-1.7.0b2.spkg to apply. The file to be patched in the header of each patch is missing a leading "/" for
patch -p1 <"$patch"
to work. One possibility is to change "-p1" to "-p0" in spkg-install.
comment:66 Changed 8 years ago by
Thanks for testing I forgot to do that and I tested my first version of the spkg but not the second. I will fix this shortly.
comment:67 Changed 8 years ago by
spkg updated in the same location. I just edited the patch as we want the patching command to be uniform with the already existing cygwin patch which is still needed in my opinion.
comment:68 Changed 8 years ago by
Here is some background on the failing test
sage -t -long -force_lib "devel/sage-main/sage/plot/matrix_plot.py"
that's mentioned above above. The failure in Sage of
sage: matrix_plot([[sin(x), cos(x)], [1, 0]])
seems to be associated with the following sequence
sage: import numpy as np sage: np.asarray([[sin(x), cos(x)], [1, 0]], dtype = float)
With numpy-1.5.1 I get
sage: np.asarray([[sin(x), cos(x)], [1, 0]], dtype = float) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /storage/sage/sage-5.5.rc0/<ipython console> in <module>() /storage/sage/sage-5.5.rc0/local/lib/python2.7/site-packages/numpy/core/numeric.py in asarray(a, dtype, order) 282 283 """ --> 284 return array(a, dtype, copy=False, order=order) 285 286 def asanyarray(a, dtype=None, order=None): ValueError: setting an array element with a sequence.
and with numpy-1.7.0b2
sage: np.asarray([[sin(x), cos(x)], [1, 0]], dtype = float) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /storage/sage/sage-5.5.rc0/<ipython console> in <module>() /storage/sage/sage-5.5.rc0/local/lib/python2.7/site-packages/numpy/core/numeric.pyc in asarray(a, dtype, order) 318 319 """ --> 320 return array(a, dtype, copy=False, order=order) 321 322 def asanyarray(a, dtype=None, order=None): /storage/sage/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/symbolic/expression.so in sage.symbolic.expression.Expression.__float__ (sage/symbolic/expression.cpp:7023)() TypeError: unable to simplify to float approximation
So it would appear that the ValueError? when using numpy-1.5.1 is thrown my numpy but the TypeError? when using numpy-1.7.0b2 is thrown by Sage.
comment:69 Changed 8 years ago by
Ondřej just pulled a request with all we need in the 1.7 maintainance branch. Expect new spkg/ebuild early next week based on a git pull.
comment:70 Changed 8 years ago by
There is a now a rc1. I will update when I can.
comment:71 Changed 7 years ago by
- Description modified (diff)
spkg updated. I will soon run test against a vanilla 5.6.rc0 (as opposed to sage-on-gentoo).
comment:72 Changed 7 years ago by
Tests run here is the list of failing doctest
sage -t -long devel/sage-main/sage/calculus/interpolators.pyx # 2 doctests failed sage -t -long devel/sage-main/sage/misc/inline_fortran.py # 4 doctests failed sage -t -long devel/sage-main/sage/calculus/riemann.pyx # 14 doctests failed sage -t -long devel/sage-main/sage/symbolic/function.pyx # 1 doctests failed sage -t -long devel/sage-main/sage/plot/plot_field.py # 1 doctests failed sage -t -long devel/sage-main/sage/plot/plot3d/implicit_surface.pyx # 2 doctests failed sage -t -long devel/sage-main/sage/plot/matrix_plot.py # 1 doctests failed sage -t -long devel/sage-main/sage/matrix/matrix1.pyx # 1 doctests failed sage -t -long devel/sage-main/sage/modules/vector_double_dense.pyx # 1 doctests failed sage -t -long devel/sage-main/sage/rings/complex_number.pyx # 1 doctests failed sage -t -long devel/sage-main/sage/rings/real_mpfr.pyx # 1 doctests failed sage -t -long devel/sage-main/sage/rings/rational.pyx # 1 doctests failed sage -t -long devel/sage-main/sage/rings/integer.pyx # 1 doctests failed sage -t -long devel/sage-main/sage/functions/hyperbolic.py # 2 doctests failed
There are a couple of new things I believe.
sage -t -long "devel/sage-main/sage/misc/inline_fortran.py" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/misc/inline_fortran.py", line 29: sage: fortran(_example) Exception raised: Traceback (most recent call last): File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[4]>", line 1, in <module> fortran(_example)###line 29: sage: fortran(_example) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/sage/misc/inline_fortran.py", line 21, in __call__ return self.eval(*args, **kwds) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/sage/misc/inline_fortran.py", line 84, in eval f2py.compile(x, name, extra_args = extra_args, source_fn=fortran_file) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/numpy/f2py/__init__.py", line 40, in compile s,o = exec_command(c) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/numpy/distutils/exec_command.py", line 197, in exec_command if _with_python and (0 or sys.stdout.fileno()==-1): AttributeError: _SpoofOut instance has no attribute 'fileno' ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/misc/inline_fortran.py", line 32: sage: fib(n,int(10)) Exception raised: Traceback (most recent call last): File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[7]>", line 1, in <module> fib(n,int(Integer(10)))###line 32: sage: fib(n,int(10)) NameError: name 'fib' is not defined ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/misc/inline_fortran.py", line 33: sage: n Expected: array([ 0., 1., 1., 2., 3., 5., 8., 13., 21., 34.]) Got: array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/misc/inline_fortran.py", line 39: sage: fortran.eval("SYNTAX ERROR !@#$") Expected: Traceback (most recent call last): ... RuntimeError: failed to compile Fortran code:... Got: Traceback (most recent call last): File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[10]>", line 1, in <module> fortran.eval("SYNTAX ERROR !@#$")###line 39: sage: fortran.eval("SYNTAX ERROR !@#$") File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/sage/misc/inline_fortran.py", line 84, in eval f2py.compile(x, name, extra_args = extra_args, source_fn=fortran_file) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/numpy/f2py/__init__.py", line 40, in compile s,o = exec_command(c) File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/numpy/distutils/exec_command.py", line 197, in exec_command if _with_python and (0 or sys.stdout.fileno()==-1): AttributeError: _SpoofOut instance has no attribute 'fileno'
which also happens in 5.5 but I overlooked previously (because in my sage-on-gentoo this test always failed in parallel). This is therefore not caused by #13887. This is new I believe
sage -t -long "devel/sage-main/sage/plot/plot_field.py" ********************************************************************** File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/plot/plot_field.py", line 49: sage: r.yvec_array Expected: masked_array(data = [0.0 0.707106781187 0.707106781187 0.894427191], mask = [False False False False], fill_value = 1e+20) Got: masked_array(data = [0.0 0.7071067811865475 0.7071067811865475 0.8944271909999159], mask = [False False False False], fill_value = 1e+20) <BLANKLINE>
Numerical noise and a blank line..
comment:73 Changed 7 years ago by
I just edited my last comment to repair some mismatching between sage version (I ran the tests on files from sage 5.6.rc0 with sage 5.5...)
comment:74 Changed 7 years ago by
Breakage of inline fortran reported upstream at https://github.com/numpy/numpy/issues/2915 as a recent commit look suspicious.
comment:75 Changed 7 years ago by
It's more than suspicious. Reverting one line of this patch in the installed file enables the doctest to complete without errors.
comment:76 Changed 7 years ago by
- Dependencies set to #12415
Numpy upstream pointed me to #12415 which does solve the problem. So depending on it.
comment:77 Changed 7 years ago by
- Dependencies changed from #12415 to #12415, #13992
comment:78 Changed 7 years ago by
Will do. It's not like we are any closer to final on that one.
comment:79 Changed 7 years ago by
Just a word of encouragement to those on this ticket - we've had a number of requests for pandas in Sage lately, but apparently this would be a prereq for easy_install-ing that, so we'd need it.
comment:80 Changed 7 years ago by
Thanks Karl-dieter. Actually if you could provide help with the implicit_surface.py test (cube marcher) or get someone who know the code on that case we possibly could wrap it. We would may be wait for an official 1.7.0 but we'd be ready.
comment:81 follow-up: ↓ 83 Changed 7 years ago by
I reviewed the marching cubes code when it went in. I think here is the problem. With the previous version of numpy:
import numpy numpy.array([[None,None],[None,None]]).any()
returns False, but now returns None.
comment:82 Changed 7 years ago by
I posted to numpy-discussion about it: http://mail.scipy.org/pipermail/numpy-discussion/2013-February/065377.html
comment:83 in reply to: ↑ 81 Changed 7 years ago by
Replying to jason:
I reviewed the marching cubes code when it went in. I think here is the problem. With the previous version of numpy:
import numpy numpy.array([[None,None],[None,None]]).any()returns False, but now returns None.
Yes that correlate with some experiments I have with the marching cube code but I hadn't isolated the problem that deeply. I'll look at the discussion but I am not sure that is a wrong behavior.
comment:84 Changed 7 years ago by
I can see the behavior choice going either way. The question for the numpy folks was if the change was intentional. If not, the follow-up question is whether it should be changed to be backwards compatible.
At any rate, I think fixing the doctests to return nothing is fine.
comment:85 Changed 7 years ago by
Before I do that can I ask about the purpose of test? The comment suggest that the calls have no adverse effects but this is not tested afterwards. Because I do not see the point of the test I would just remove it. Any comment on that?
Once this is sorted, we'll have to see how things combine with #12415. From what I understand from it and some other threads, some "divide by 0 warnings" will be with us forever. We probably should check when they appear but they cannot be hidden anymore (I think evanandel was playing hide and seek with them in riemann.pyx).
comment:86 Changed 7 years ago by
This references the failure in the test
sage -t -long -force_lib "devel/sage-main/sage/plot/matrix_plot.py"
I have some concerns with the following block of code in matrix_plot.py
try: if sparse: xy_data_array = mat else: xy_data_array = np.asarray(mat, dtype = float) except TypeError: raise TypeError, "mat must be a Matrix or a two dimensional array" except ValueError: raise ValueError, "can not convert entries to floating point numbers" if len(xy_data_array.shape) < 2: raise TypeError, "mat must be a Matrix or a two dimensional array"
The subject failure occurs when the object "mat" is not sparse. In which case, the errors are thrown by
np.asarray(mat, dtype = float)
I have not been able to pass any object "mat" to np.asarray() that returns a TypeError
or ValueError
that remotely hints of the cause as being due to
"mat must be a Matrix or a two dimensional array"
I question whether this information can be returned by np.asarray() when used from Sage. I could be wrong though. The referenced failure does return a TypeError
See above. But this error reflects the inability to coerce the symbolic [sin(x), cos(x)] expressions to floats. It is possible to generate a ValueError
when the object "mat" contains complex entries:
sage: import numpy sage: numpy.asarray([[1+1j,2+3j],[1,1]], dtype=float) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-5-2bd7229efabd> in <module>() ----> 1 numpy.asarray([[Integer(1)+ComplexNumber(0, '1'),Integer(2)+ComplexNumber(0, '3')],[Integer(1),Integer(1)]], dtype=float) /storage/sage/sage-5.7.beta3/local/lib/python2.7/site-packages/numpy/core/numeric.pyc in asarray(a, dtype, order) 318 319 """ --> 320 return array(a, dtype, copy=False, order=order) 321 322 def asanyarray(a, dtype=None, order=None): ValueError: setting an array element with a sequence.
This is they type of error that's returned for the referenced test when using numpy-1.5.1. Sage interprets this error as
"can not convert entries to floating point numbers"
This seems to be a correct interpretation, at least from the python side with the numpy of this ticket:
>>> import numpy >>> numpy.asarray([[1+1j,2+3j],[1,1]], dtype=float) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/storage/sage/sage-5.7.beta3/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 320, in asarray return array(a, dtype, copy=False, order=order) TypeError: can't convert complex to float
although here there is a TypeError
?
So it seems to me that regardless of whether there is a TypeError
or ValueError
the root cause is inability to coerce to a float. The question of whether "mat" is a Matrix or appropriate array is determined by the lines in the code that parse the shape attribute of an object that has been successfully converted to a ndarray. But this is probably minor compared to #12415. Perhaps Mike (mhansen) has some input?
comment:87 Changed 7 years ago by
That would be good to have a little bit more on this although it is indeed not as big as #12415.
In other news 1.7rc2 has been released less than an hour ago. I am trying to get it up as quickly as possible with the changes Jeroen wants.
CCing fbissey since he's been extremely helpful in previous numpy upgrades.