Opened 9 years ago

Last modified 7 years ago

#11334 closed task

Update numpy to 1.6.1 — at Version 46

Reported by: jason Owned by: tbd
Priority: major Milestone: sage-5.10
Component: packages: standard Keywords: sd40.5
Cc: fbissey, kini, Snark Merged in:
Authors: Reviewers:
Report Upstream: Fixed upstream, in a later stable release. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Change History (47)

comment:1 Changed 9 years ago by jason

  • Cc fbissey added

CCing fbissey since he's been extremely helpful in previous numpy upgrades.

comment:2 Changed 9 years ago by fbissey

Thx. it looks like numpy has been bumped to 1.6 in Gentoo so I can do some quick compatibility tests. It looks like a flag for gfortran has been added in the ebuild to help with the compiler autodetection madness it may be worth having it sage side too.

Do you know if a scipy release is to follow, if not it would be best to revbump scipy to make sure it is rebuilt after numpy.

Other matter there has been noise about lapack in numpy do we want to disable it, make it optional or keep the status quo?

comment:3 Changed 9 years ago by jason

I don't remember seeing anything on the scipy lists about an imminent scipy release.

comment:4 Changed 9 years ago by fbissey

just in:

sage -t  -long -force_lib "devel/sage/doc/en/numerical_sage/numpy.rst"
**********************************************************************
File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 97:
    sage: 2.5*l
Expected:
    array([  0. ,   2.5,   5. ,   7.5,  10. ,  12.5,  15. ,  17.5,  20. ,  22.5])
Got:
    array([0.000000000000000, 2.50000000000000, 5.00000000000000,
           7.50000000000000, 10.0000000000000, 12.5000000000000,
           15.0000000000000, 17.5000000000000, 20.0000000000000,
           22.5000000000000], dtype=object)
**********************************************************************
File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 289:
    sage: xx
Expected:
    array([[ 0.  ,  0.25,  0.5 ,  0.75,  1.  ],
           [ 0.  ,  0.25,  0.5 ,  0.75,  1.  ],
           [ 0.  ,  0.25,  0.5 ,  0.75,  1.  ],
           [ 0.  ,  0.25,  0.5 ,  0.75,  1.  ],
           [ 0.  ,  0.25,  0.5 ,  0.75,  1.  ]])
Got:
    array([[0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000],
           [0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000],
           [0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000],
           [0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000],
           [0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000]], dtype=object)
**********************************************************************
File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 295:
    sage: yy
Expected:
    array([[ 0.  ,  0.  ,  0.  ,  0.  ,  0.  ],
           [ 0.25,  0.25,  0.25,  0.25,  0.25],
           [ 0.5 ,  0.5 ,  0.5 ,  0.5 ,  0.5 ],
           [ 0.75,  0.75,  0.75,  0.75,  0.75],
           [ 1.  ,  1.  ,  1.  ,  1.  ,  1.  ]])
Got:
    array([[0.000000000000000, 0.000000000000000, 0.000000000000000,
            0.000000000000000, 0.000000000000000],
           [0.250000000000000, 0.250000000000000, 0.250000000000000,
            0.250000000000000, 0.250000000000000],
           [0.500000000000000, 0.500000000000000, 0.500000000000000,
            0.500000000000000, 0.500000000000000],
           [0.750000000000000, 0.750000000000000, 0.750000000000000,
            0.750000000000000, 0.750000000000000],
           [1.00000000000000, 1.00000000000000, 1.00000000000000,
            1.00000000000000, 1.00000000000000]], dtype=object)
**********************************************************************
File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 301:
    sage: f(xx,yy)
Expected:
    array([[ 0.    ,  0.0625,  0.25  ,  0.5625,  1.    ],
           [ 0.0625,  0.125 ,  0.3125,  0.625 ,  1.0625],
           [ 0.25  ,  0.3125,  0.5   ,  0.8125,  1.25  ],
           [ 0.5625,  0.625 ,  0.8125,  1.125 ,  1.5625],
           [ 1.    ,  1.0625,  1.25  ,  1.5625,  2.    ]])
Got:
    array([[0.000000000000000, 0.0625000000000000, 0.250000000000000,
            0.562500000000000, 1.00000000000000],
           [0.0625000000000000, 0.125000000000000, 0.312500000000000,
            0.625000000000000, 1.06250000000000],
           [0.250000000000000, 0.312500000000000, 0.500000000000000,
            0.812500000000000, 1.25000000000000],
           [0.562500000000000, 0.625000000000000, 0.812500000000000,
            1.12500000000000, 1.56250000000000],
           [1.00000000000000, 1.06250000000000, 1.25000000000000,
            1.56250000000000, 2.00000000000000]], dtype=object)
**********************************************************************
2 items had failures:
   1 of  55 in __main__.example_0
   3 of  13 in __main__.example_1
***Test Failed*** 4 failures.

there may be more

comment:5 Changed 9 years ago by fbissey

  • Description modified (diff)

I have cut a new spkg for 1.6.0, I decided to go for plain update for now.

comment:6 Changed 9 years ago by fbissey

seems my home install has problems. Probably stuff messed around between python 2.6 and 2.7 because I am working on #9958. Another install didn't report any problems. So I will check on a vanilla sage but I think we won't need to do any patching.

comment:7 Changed 9 years ago by fbissey

For some reason the other install didn't upgrade that's why I didn't see anything on the second machine. So right now it looks like we have the following to correct:

        sage -t  -long -force_lib devel/sage/doc/en/numerical_sage/numpy.rst #
 4 doctests failed
         sage -t  -long -force_lib devel/sage/sage/symbolic/function.pyx # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/plot/plot_field.py # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/plot/matrix_plot.py # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/matrix/matrix1.pyx # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/numerical/optimize.py # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/rings/integer.pyx # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/functions/hyperbolic.py # 2
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/functions/trig.py # 3
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/functions/other.py # 2
 doctests failed

The first one is already in the ticket, then we have

sage -t -long -force_lib "devel/sage/sage/symbolic/function.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/symbolic/function.pyx", line 596:
    sage: csc(a)
Expected:
    array([        inf,  1.18839511,  1.09975017,  7.0861674 , -1.32134871])
Got:
    doctest:270: RuntimeWarning: divide by zero encountered in divide
    array([        inf,  1.18839511,  1.09975017,  7.0861674 , -1.32134871])
**********************************************************************
sage -t -long -force_lib "devel/sage/sage/plot/plot_field.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/plot/plot_field.py", line 216:
    sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2)).show(aspect_ratio=1)
Expected nothing
Got:
    doctest:622: RuntimeWarning: divide by zero encountered in divide
    doctest:623: RuntimeWarning: invalid value encountered in multiply
    doctest:624: RuntimeWarning: invalid value encountered in multiply
**********************************************************************
sage -t -long -force_lib "devel/sage/sage/plot/matrix_plot.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/plot/matrix_plot.py", line 383:
    sage: matrix_plot([[sin(x), cos(x)], [1, 0]])
Expected:
    Traceback (most recent call last):
    ...
    ValueError: can not convert entries to floating point numbers
Got:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_8[36]>", line 1, in <module>
        matrix_plot([[sin(x), cos(x)], [Integer(1), Integer(0)]])###line 383:
    sage: matrix_plot([[sin(x), cos(x)], [1, 0]])
      File "/usr/lib/python2.7/site-packages/sage/misc/decorators.py", line 381, in wrapper
        return func(*args, **kwds)
      File "/usr/lib/python2.7/site-packages/sage/misc/decorators.py", line 432, in wrapper
        return func(*args, **options)
      File "/usr/lib/python2.7/site-packages/sage/plot/matrix_plot.py", line 423, in matrix_plot
        raise TypeError, "mat must be a Matrix or a two dimensional array"
    TypeError: mat must be a Matrix or a two dimensional array
**********************************************************************
sage -t -long -force_lib "devel/sage/sage/matrix/matrix1.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/matrix/matrix1.pyx", line 468:
    sage: sorted(numpy.typecodes.items())
Expected:
    [('All', '?bhilqpBHILQPfdgFDGSUVO'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
Got:
    [('All', '?bhilqpBHILQPefdgFDGSUVOMm'), ('AllFloat', 'efdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'efdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
**********************************************************************
sage -t -long -force_lib "devel/sage/sage/numerical/optimize.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/numerical/optimize.py", line 162:
    sage: find_minimum_on_interval(f, 1, 5, tol=1e-3)
Expected:
    (-3.28837136189098..., 3.42575079030572...)
Got:
    (-3.28837136189, 3.4257507903057229)
sage -t -long -force_lib "devel/sage/sage/rings/integer.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4738:
    sage: numpy.array(2**40).dtype
Expected:
    dtype('int64')
Got:
    dtype('object')
**********************************************************************
sage -t -long -force_lib "devel/sage/sage/functions/hyperbolic.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/hyperbolic.py", line 609:
    sage: arcsech(a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_23[4]>", line 1, in <module>
        arcsech(a)###line 609:
    sage: arcsech(a)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "/usr/lib/python2.7/site-packages/sage/functions/hyperbolic.py", line 612, in _eval_numpy_
        return arccosh(1.0 / x)
    ZeroDivisionError: float division by zero
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/hyperbolic.py", line 658:
    sage: arccsch(a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_26[4]>", line 1, in <module>
        arccsch(a)###line 658:
    sage: arccsch(a)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "/usr/lib/python2.7/site-packages/sage/functions/hyperbolic.py", line 661, in _eval_numpy_
        return arcsinh(1.0 / x)
    ZeroDivisionError: float division by zero
**********************************************************************
sage -t -long -force_lib "devel/sage/sage/functions/trig.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 827:
    sage: atan2(a, b)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_34[16]>", line 1, in <module>
        atan2(a, b)###line 827:
    sage: atan2(a, b)
      File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568)
    NotImplementedError: The Function arctan2 does not support numpy arrays as arguments
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 830:
    sage: atan2(1,a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_34[17]>", line 1, in <module>
        atan2(Integer(1),a)###line 830:
    sage: atan2(1,a)
      File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568)
    NotImplementedError: The Function arctan2 does not support numpy arrays as arguments
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 833:
    sage: atan2(a, 1)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_34[18]>", line 1, in <module>
        atan2(a, Integer(1))###line 833:
    sage: atan2(a, 1)
      File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568)
    NotImplementedError: The Function arctan2 does not support numpy arrays as arguments
**********************************************************************
sage -t -long -force_lib "devel/sage/sage/functions/other.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/other.py", line 211:
    sage: ceil(a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_5[20]>", line 1, in <module>
        ceil(a)###line 211:
    sage: ceil(a)
      File "/usr/lib/python2.7/site-packages/sage/functions/other.py", line 251, in __call__
        return numpy.ceil(x)
    AttributeError: ceil
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/other.py", line 366:
    sage: floor(a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_9[15]>", line 1, in <module>
        floor(a)###line 366:
    sage: floor(a)
      File "/usr/lib/python2.7/site-packages/sage/functions/other.py", line 405, in __call__
        return numpy.floor(x)
    AttributeError: floor
**********************************************************************

The last three are particularly concerning, could you check what's up with these in numpy?

comment:8 Changed 9 years ago by kcrisman

This seems to have installed fine on OS X 10.4 PPC. I'm running doctests now (which will take a LONG time) but I did already see the matrix/matrix1.pyx one while testing something else.

comment:9 Changed 9 years ago by fbissey

That's good that it at least install. On gentoo there are interactions problem between numpy and scipy to get cblas/blas/lapack right in scipy. I should check if the resulting scipy in vanilla sage suffers from the same problem. http://bugs.gentoo.org/show_bug.cgi?id=371099

The last few failing tests are show stoppers as far as I am concerned.

comment:10 Changed 9 years ago by kcrisman

Okay, I got pretty much the same failures on a long run. One weird thing... did we change find_minimum_on_interval away from SciPy recently? This is the full output I get on the (otherwise same as your failure) test for that.

    sage: find_minimum_on_interval(f, 1, 5, tol=1e-3)
Expected:
    (-3.28837136189098..., 3.42575079030572...)
Got:
    (-3.28837136189, 3.4257507903057229)
GLPK Simplex Optimizer, v4.44
6 rows, 3 columns, 8 non-zeros
Preprocessing...
2 rows, 2 columns, 4 non-zeros
Scaling...
 A: min|aij| =  2.400e+01  max|aij| =  5.000e+01  ratio =  2.083e+00
GM: min|aij| =  8.128e-01  max|aij| =  1.230e+00  ratio =  1.514e+00
EQ: min|aij| =  6.606e-01  max|aij| =  1.000e+00  ratio =  1.514e+00
Constructing initial basis...
Size of triangular part = 2
*     0: obj =  -5.100000000e+01  infeas =  0.000e+00 (0)
*     1: obj =  -5.225000000e+01  infeas =  0.000e+00 (0)
OPTIMAL SOLUTION FOUND

comment:11 Changed 9 years ago by fbissey

1.6.1 is now available. I'll have a look sometimes this week.

comment:12 Changed 9 years ago by fbissey

Looks like more of the same but maybe a little bit more. There are a few other things that may interfere on my setup and I had to solve that bit: http://bugs.gentoo.org/show_bug.cgi?id=371099 before doing the trial.

comment:13 Changed 9 years ago by fbissey

Once I had cleared a few blas problem that I had while solving the gentoo bug. I get exactly the same results. I also get the output from Karl in optimize. This is a verbose output of glpk which get called at some point. You only get to see the such verbose output if there was a failure in the test but not necessarily the test for which glpk is called funnily enough.

comment:14 Changed 9 years ago by fbissey

Should we try to move forward on this again? I will have a go at trying it with 4.8.alpha2. Any ideas about the current failures?

comment:15 Changed 9 years ago by fbissey

Ok this is still broken in 4.8.alpha2. Some food for thought

sage
----------------------------------------------------------------------
| Sage Version 4.8.alpha2, Release Date: 2011-11-19                  |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
sage: import numpy
sage: a = numpy.linspace(0,2,6)
sage: a
array([0.0, 0.4, 0.8, 1.2, 1.6, 2], dtype=object)
sage: ceil(a)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/home/fbissey/<ipython console> in <module>()

/usr/lib64/python2.7/site-packages/sage/functions/other.py in __call__(self, x, maximum_bits)
    250             elif type(x).__module__ == 'numpy':
    251                 import numpy
--> 252                 return numpy.ceil(x)
    253 
    254         x_original = x

AttributeError: ceil

with numpy-1.5.1 we just have

sage: a
array([0.0, 0.4, 0.8, 1.2, 1.6, 2])

I suspect that for this one (functions/other.py) and functions/trig.py the pynac interface may need to be touched. All stuff that takes an numpy array and return a numpy array may have trouble.

comment:16 Changed 9 years ago by fbissey

Just to add a bit of info. Same python and ipython

ipython
Python 2.7.2 (default, Oct 28 2011, 11:36:16) 
Type "copyright", "credits" or "license" for more information.

IPython 0.10.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import numpy

In [2]: a = numpy.linspace(0,2,6)

In [3]: a
Out[3]: array([ 0. ,  0.4,  0.8,  1.2,  1.6,  2. ])

In [4]: ceil(a)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

/home/fbissey/<ipython console> in <module>()

NameError: name 'ceil' is not defined

In [5]: numpy.ceil(a)
Out[5]: array([ 0.,  1.,  1.,  2.,  2.,  2.])

So it works in ipython but we don't have the dtype=object attached to the array. Anyone knows what that means actually?

comment:17 Changed 9 years ago by fbissey

numpy-1.5.1

sage: numpy.dtype(a[1])
dtype('float64')

numpy-1.6.1

sage: numpy.dtype(a[1])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/home/fbissey/<ipython console> in <module>()

TypeError: data type not understood

comment:18 Changed 9 years ago by fbissey

sage: type(a[0])
<type 'float'>
sage: type(a[0:5])
<type 'numpy.ndarray'>
sage: type(a[2])
<type 'float'>
sage: type(a[3])
<type 'float'>
sage: type(a[4])
<type 'float'>
sage: type(a[5])
<type 'sage.rings.integer.Integer'>

For some reason numpy.linspace return an array with sage objects rather than just floats. I am guessing it may be deeper than that. Numpy probably uses sage types rather than converting to numpy types. The types of a[0] to a[4] is strange, is there a sage type for floats? I mean I tried numpy.linspace(0,2.0,6) and I got <type 'sage.rings.real_mpfr.RealNumber?'> for a[0] to a[4] and <type 'sage.rings.real_mpfr.RealLiteral?'> for a[5].

Doing a cast numpy.array(a,dtype=float) returns something that will work.

comment:19 Changed 8 years ago by fbissey

  • Description modified (diff)
  • Summary changed from Update numpy to 1.6 to Update numpy to 1.6.1

comment:20 Changed 8 years ago by fbissey

I'll be starting to make patches to correct the doctests and document the new behavior in numpy 1.6.x. Some documentation may need updating.

comment:21 Changed 8 years ago by fbissey

Actually I am a bit confused by one of the tests:

    ``plot_slope_field`` takes a function of two variables xvar and yvar
    (for instance, if the variables are `x` and `y`, take `f(x,y)`), and at
    representative points `(x_i,y_i)` between xmin, xmax, and ymin, ymax
    respectively, plots a line with slope `f(x_i,y_i)` (see below).

    ``plot_slope_field(f, (xvar, xmin, xmax), (yvar, ymin, ymax))``

    EXAMPLES: 

    A logistic function modeling population growth::

        sage: x,y = var('x y')
        sage: capacity = 3 # thousand
        sage: growth_rate = 0.7 # population increases by 70% per unit of time
        sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2))

That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be.

comment:22 Changed 8 years ago by fbissey

I am guessing it should have been

growth_rate*(1-x/capacity)*y

y being the time and x the population in thousands. That satisfy a simple dimensional analysis.

comment:23 Changed 8 years ago by jason

No, wasn't it right before? That's the standard logistic population model, where y is the population. See http://en.wikipedia.org/wiki/Logistic_function#In_ecology:_modeling_population_growth, for example.

comment:24 Changed 8 years ago by kcrisman

sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2))
}}}}
> That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in

This should be ok. The point is that the slope dy/dx is a function of y only; this can be completely symbolically solved using separation of variables to get the [http://en.wikipedia.org/wiki/Logistic_function#In_ecology:_modeling_population_growth logistic function].

> this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be.

If this is the division by zero stuff, it's the whole business with the arrowheads having zero length. See #11208. I think Jason has a fix for this, actually, but I don't recall the whole story of what happened once it was reported to mpl.

comment:25 Changed 8 years ago by kcrisman

Sorry for that noise. Jason is right.

sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2))

That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in

This should be ok. The point is that the slope dy/dx is a function of y only; this can be completely symbolically solved using separation of variables to get the logistic function.

this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be.

If this is the division by zero stuff, it's the whole business with the arrowheads having zero length. See #11208. I think Jason has a fix for this, actually, but I don't recall the whole story of what happened once it was reported to mpl.

comment:26 Changed 8 years ago by fbissey

OK I was interpreting the formula another way because I wasn't sure what it is actually computing. Seems like #11208 numpy, mpl and scipy all go hands in hands so if something start happening with one the others may get it too.

Overall the change of behavior of numpy around type conversion is quite annoying.

Changed 8 years ago by fbissey

first draft at updating the documentation

comment:27 Changed 8 years ago by fbissey

It strikes me that the problem may be more an issue of the coercion to numpy types not working properly with numpy-1.6. I will look into that.

comment:28 Changed 8 years ago by fbissey

Any ideas why stuff like this is happening with numpy-1.6.x

sage: numpy.arange(10.0)
array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])
sage: numpy.linspace(0.0,9.0,10)
array([0.000000000000000, 1.00000000000000, 2.00000000000000,
       3.00000000000000, 4.00000000000000, 5.00000000000000,
       6.00000000000000, 7.00000000000000, 8.00000000000000,
       9.00000000000000], dtype=object)

comment:29 follow-up: Changed 8 years ago by jason

It looks like it didn't automatically convert from RR to float, like it should. That's what these two lines should do in sage/rings/real_mpfr.pyx

cdef object numpy_double_interface = {'typestr': '=f8'}
cdef object numpy_object_interface = {'typestr': '|O'}

and this property in that same file:

    property __array_interface__:
        def __get__(self):
            """
            Used for NumPy conversion.
            
            EXAMPLES::

                sage: import numpy
                sage: numpy.arange(10.0)
                array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])
                sage: numpy.array([1.0, 1.1, 1.2]).dtype
                dtype('float64')
                sage: numpy.array([1.000000000000000000000000000000000000]).dtype
                dtype('object')
            """
            if (<RealField_class>self._parent).__prec <= 57: # max size of repr(float)
                return numpy_double_interface
            else:
                return numpy_object_interface

So maybe numpy doesn't do something with __array_interface__ anymore in linspace?

Can you try numpy.linspace(0.0,9.0,float(10)), numpy.linspace(0.0,float(9),10), numpy.linspace(float(0),9.0,10) and various other combinations of float vs. Sage types in the arguments to narrow down what is triggering it?

comment:30 Changed 8 years ago by fbissey

I was writing something else:

And more subtly:

sage: a=numpy.linspace(0,9,10)
sage: a
array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9], dtype=object)
sage: type(a[0])
<type 'float'>
sage: type(a[9])
<type 'sage.rings.integer.Integer'>

So linspace and arange deal with sage input very differently. There should be some coercion but it is not working properly.

now for what you are asking:

sage: numpy.linspace(0.0,9.0,float(10))
array([0.000000000000000, 1.00000000000000, 2.00000000000000,
       3.00000000000000, 4.00000000000000, 5.00000000000000,
       6.00000000000000, 7.00000000000000, 8.00000000000000,
       9.00000000000000], dtype=object)
sage: numpy.linspace(0.0,float(9),10)
array([0.000000000000000, 1.00000000000000, 2.00000000000000,
       3.00000000000000, 4.00000000000000, 5.00000000000000,
       6.00000000000000, 7.00000000000000, 8.00000000000000, 9.0], dtype=object)
sage: numpy.linspace(float(0),9.0,10)
array([0.000000000000000, 1.00000000000000, 2.00000000000000,
       3.00000000000000, 4.00000000000000, 5.00000000000000,
       6.00000000000000, 7.00000000000000, 8.00000000000000,
       9.00000000000000], dtype=object)

I know about sage/rings/real_mpfr.pyx and the others that's how I got on thinking of coercion not working.

comment:31 Changed 8 years ago by fbissey

Weirder! The code for linspace uses arange inside. No change in the code between the 1.5.1 and 1.6.1 but the juicy important bit is as follow

        step = (stop-start)/float(num)
        y = _nx.arange(0, num) * step + start

step will become float (unless start or stop is complex or something). But the type of step and start will influence the final array.

comment:32 Changed 8 years ago by fbissey

Anyone wants to ask on a numpy mailing list if it is a sign of bug in numpy or something that is clearly under-documented?

comment:33 in reply to: ↑ 29 Changed 8 years ago by strogdon

Replying to jason:

So maybe numpy doesn't do something with __array_interface__ anymore in linspace?

I've been tinkering here and this is exactly what is happening. I deleted all the

property __array_interface__:

blocks that were added by 5081-numpy-types.patch and 6506-numpy-types.patch with no apparent effect on what numpy.linspace() returns with numpy-1.6.1 installed. Of course, the deleted blocks do affect what numpy.arange() and numpy.array() return:

sage: numpy.arange(10.0)
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object)
sage: numpy.array([1.0, 1.1, 1.2]).dtype
dtype('object')

Also the example failure

from scipy import stats
stats.uniform(0,15).ppf([0.5,0.7])

from ticket #5081 now again fails with numpy-1.6.x (s-o-g result):

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/storage/strogdon/gentoo/usr/local/portage/sage-on-gentoo/<ipython console> in <module>()

/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/scipy/stats/distributions.pyc in ppf(self, q)
    449 
    450     def ppf(self, q):
--> 451         return self.dist.ppf(q, *self.args, **self.kwds)
    452 
    453     def isf(self, q):

/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/scipy/stats/distributions.pyc in ppf(self, q, *args, **kwds)
   1514             goodargs = argsreduce(cond, *((q,)+args+(scale,loc)))
   1515             scale, loc, goodargs = goodargs[-2], goodargs[-1], goodargs[:-2]
-> 1516             place(output,cond,self._ppf(*goodargs)*scale + loc)
   1517         if output.ndim == 0:
   1518             return output[()]

/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/numpy/lib/function_base.pyc in place(arr, mask, vals)
   1333 
   1334     """
-> 1335     return _insert(arr, mask, vals)
   1336 
   1337 def _nanop(op, fill, a, axis=None):

TypeError: array cannot be safely cast to required type

Starting with numpy-1.6.0 scalar upcasting rules were changed http://docs.scipy.org/doc/numpy/reference/ufuncs.html and I'm wondering if this change is interfering with how sage/cython is handling the python __array_interface__ extension, at least relative to numpy.linspace(), and perhaps other numpy functions that are pure python.

comment:34 Changed 8 years ago by fbissey

That's the best clue we had in ages Steve. I am not sure when I will be able to dig it further as we have an extra long week end here (Tuesday is a university holyday).

comment:35 Changed 8 years ago by mhansen

Here is a minimal example of the problem:

sage: import numpy
sage: numpy.array(0.5)
array(0.500000000000000, dtype=object)
sage: numpy.array(0.5r)
array(0.5)

where as before it was

sage: import numpy
sage: numpy.array(0.5)
array(0.5)
sage: numpy.array(0.5r)
array(0.5)

comment:36 Changed 8 years ago by mhansen

Also, we should incorporate #7831

comment:37 Changed 8 years ago by fbissey

Sure Mike do you have any idea on how to fix the coercion problem? The way it's going we'll update directly to 1.7.

comment:38 Changed 8 years ago by mhansen

Looking into this it seems that the issue is that in numpy/core/src/multiarray/ctors.c, PyArray_FromAny no longer calls _array_find_type which I believe was responsible for accessing the __array_interface__ attribute on our objects. I'm sending a message to numpy-discussion now.

comment:40 Changed 8 years ago by mhansen

  • Keywords sd40.5 added

comment:41 Changed 8 years ago by fbissey

Thanks Mike, hopefully we'll get a meaningful answer.

comment:42 Changed 8 years ago by fbissey

Looks like 1.6.2 was released last week. I guess I'll update my spkg.

comment:43 Changed 8 years ago by fbissey

Looks like I found the commit where the change has been made https://github.com/numpy/numpy/commit/2635398db3f26529ce2aaea4028a8118844f3c48

If you look carefully the call to _array_find_type is still there but it has changed location. In 1.5.1 there was a sequence if(xx==NULL) then call _array_find_type else do some other stuff. In 1.6.1 the structure if(xx!=NULL) then do the other stuff else call _array_find_type.

I will try to write a patch to see if this the cause of our observed behaviour.

comment:44 Changed 8 years ago by fbissey

I wrote a small patch to just ignore any code other than _array_find_type in ctors.c no luck so I don't think that's the real source of the problem at all. I thought the presence of a requested_dtype passed to this function could have been the source of the problem but my patch excludes that possibility entirely. So we have to go back to looking.

comment:45 Changed 8 years ago by fbissey

Problem still present with numpy-1.6.2

comment:46 Changed 8 years ago by fbissey

  • Description modified (diff)

New spkg for 1.6.2 rebased on 1.5.1.p1 as well.

Note: See TracTickets for help on using tickets.