Update numpy to 1.7.0

[jason]

Numpy-1.7.0 is now available and fixes the coercion problem that prevented us to go to numpy-1.6.x.

New spkg: ​http://spkg-upload.googlecode.com/files/numpy-1.7.0.spkg

Apply:

• trac_11334-dtype_object.patch​
• trac_11334-matrix_plot.patch​
• trac_11334-typecodes.patch​
• trac_11334-suppressdivide.patch​
• trac_11334-divide-v2.patch​
• trac_11334-cubemarcher.patch​
• trac_11334-numericalnoise.patch​
• 11334_noise2.patch​

[jason]

CCing fbissey since he's been extremely helpful in previous numpy upgrades.

[fbissey]

Thx. it looks like numpy has been bumped to 1.6 in Gentoo so I can do some quick compatibility tests. It looks like a flag for gfortran has been added in the ebuild to help with the compiler autodetection madness it may be worth having it sage side too.

Do you know if a scipy release is to follow, if not it would be best to revbump scipy to make sure it is rebuilt after numpy.

Other matter there has been noise about lapack in numpy do we want to disable it, make it optional or keep the status quo?

[jason]

I don't remember seeing anything on the scipy lists about an imminent scipy release.

[fbissey]

just in:

sage -t  -long -force_lib "devel/sage/doc/en/numerical_sage/numpy.rst"
**********************************************************************
File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 97:
    sage: 2.5*l
Expected:
    array([  0. ,   2.5,   5. ,   7.5,  10. ,  12.5,  15. ,  17.5,  20. ,  22.5])
Got:
    array([0.000000000000000, 2.50000000000000, 5.00000000000000,
           7.50000000000000, 10.0000000000000, 12.5000000000000,
           15.0000000000000, 17.5000000000000, 20.0000000000000,
           22.5000000000000], dtype=object)
**********************************************************************
File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 289:
    sage: xx
Expected:
    array([[ 0.  ,  0.25,  0.5 ,  0.75,  1.  ],
           [ 0.  ,  0.25,  0.5 ,  0.75,  1.  ],
           [ 0.  ,  0.25,  0.5 ,  0.75,  1.  ],
           [ 0.  ,  0.25,  0.5 ,  0.75,  1.  ],
           [ 0.  ,  0.25,  0.5 ,  0.75,  1.  ]])
Got:
    array([[0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000],
           [0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000],
           [0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000],
           [0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000],
           [0.000000000000000, 0.250000000000000, 0.500000000000000,
            0.750000000000000, 1.00000000000000]], dtype=object)
**********************************************************************
File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 295:
    sage: yy
Expected:
    array([[ 0.  ,  0.  ,  0.  ,  0.  ,  0.  ],
           [ 0.25,  0.25,  0.25,  0.25,  0.25],
           [ 0.5 ,  0.5 ,  0.5 ,  0.5 ,  0.5 ],
           [ 0.75,  0.75,  0.75,  0.75,  0.75],
           [ 1.  ,  1.  ,  1.  ,  1.  ,  1.  ]])
Got:
    array([[0.000000000000000, 0.000000000000000, 0.000000000000000,
            0.000000000000000, 0.000000000000000],
           [0.250000000000000, 0.250000000000000, 0.250000000000000,
            0.250000000000000, 0.250000000000000],
           [0.500000000000000, 0.500000000000000, 0.500000000000000,
            0.500000000000000, 0.500000000000000],
           [0.750000000000000, 0.750000000000000, 0.750000000000000,
            0.750000000000000, 0.750000000000000],
           [1.00000000000000, 1.00000000000000, 1.00000000000000,
            1.00000000000000, 1.00000000000000]], dtype=object)
**********************************************************************
File "/usr/share/sage/devel/sage/doc/en/numerical_sage/numpy.rst", line 301:
    sage: f(xx,yy)
Expected:
    array([[ 0.    ,  0.0625,  0.25  ,  0.5625,  1.    ],
           [ 0.0625,  0.125 ,  0.3125,  0.625 ,  1.0625],
           [ 0.25  ,  0.3125,  0.5   ,  0.8125,  1.25  ],
           [ 0.5625,  0.625 ,  0.8125,  1.125 ,  1.5625],
           [ 1.    ,  1.0625,  1.25  ,  1.5625,  2.    ]])
Got:
    array([[0.000000000000000, 0.0625000000000000, 0.250000000000000,
            0.562500000000000, 1.00000000000000],
           [0.0625000000000000, 0.125000000000000, 0.312500000000000,
            0.625000000000000, 1.06250000000000],
           [0.250000000000000, 0.312500000000000, 0.500000000000000,
            0.812500000000000, 1.25000000000000],
           [0.562500000000000, 0.625000000000000, 0.812500000000000,
            1.12500000000000, 1.56250000000000],
           [1.00000000000000, 1.06250000000000, 1.25000000000000,
            1.56250000000000, 2.00000000000000]], dtype=object)
**********************************************************************
2 items had failures:
   1 of  55 in __main__.example_0
   3 of  13 in __main__.example_1
***Test Failed*** 4 failures.

there may be more

[fbissey]

I have cut a new spkg for 1.6.0, I decided to go for plain update for now.

[fbissey]

seems my home install has problems. Probably stuff messed around between python 2.6 and 2.7 because I am working on #9958. Another install didn't report any problems. So I will check on a vanilla sage but I think we won't need to do any patching.

[fbissey]

For some reason the other install didn't upgrade that's why I didn't see anything on the second machine. So right now it looks like we have the following to correct:

        sage -t  -long -force_lib devel/sage/doc/en/numerical_sage/numpy.rst #
 4 doctests failed
         sage -t  -long -force_lib devel/sage/sage/symbolic/function.pyx # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/plot/plot_field.py # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/plot/matrix_plot.py # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/matrix/matrix1.pyx # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/numerical/optimize.py # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/rings/integer.pyx # 1
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/functions/hyperbolic.py # 2
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/functions/trig.py # 3
 doctests failed
         sage -t  -long -force_lib devel/sage/sage/functions/other.py # 2
 doctests failed

The first one is already in the ticket, then we have

sage -t -long -force_lib "devel/sage/sage/symbolic/function.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/symbolic/function.pyx", line 596:
    sage: csc(a)
Expected:
    array([        inf,  1.18839511,  1.09975017,  7.0861674 , -1.32134871])
Got:
    doctest:270: RuntimeWarning: divide by zero encountered in divide
    array([        inf,  1.18839511,  1.09975017,  7.0861674 , -1.32134871])
**********************************************************************

sage -t -long -force_lib "devel/sage/sage/plot/plot_field.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/plot/plot_field.py", line 216:
    sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2)).show(aspect_ratio=1)
Expected nothing
Got:
    doctest:622: RuntimeWarning: divide by zero encountered in divide
    doctest:623: RuntimeWarning: invalid value encountered in multiply
    doctest:624: RuntimeWarning: invalid value encountered in multiply
**********************************************************************

sage -t -long -force_lib "devel/sage/sage/plot/matrix_plot.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/plot/matrix_plot.py", line 383:
    sage: matrix_plot([[sin(x), cos(x)], [1, 0]])
Expected:
    Traceback (most recent call last):
    ...
    ValueError: can not convert entries to floating point numbers
Got:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_8[36]>", line 1, in <module>
        matrix_plot([[sin(x), cos(x)], [Integer(1), Integer(0)]])###line 383:
    sage: matrix_plot([[sin(x), cos(x)], [1, 0]])
      File "/usr/lib/python2.7/site-packages/sage/misc/decorators.py", line 381, in wrapper
        return func(*args, **kwds)
      File "/usr/lib/python2.7/site-packages/sage/misc/decorators.py", line 432, in wrapper
        return func(*args, **options)
      File "/usr/lib/python2.7/site-packages/sage/plot/matrix_plot.py", line 423, in matrix_plot
        raise TypeError, "mat must be a Matrix or a two dimensional array"
    TypeError: mat must be a Matrix or a two dimensional array
**********************************************************************

sage -t -long -force_lib "devel/sage/sage/matrix/matrix1.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/matrix/matrix1.pyx", line 468:
    sage: sorted(numpy.typecodes.items())
Expected:
    [('All', '?bhilqpBHILQPfdgFDGSUVO'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
Got:
    [('All', '?bhilqpBHILQPefdgFDGSUVOMm'), ('AllFloat', 'efdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'efdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
**********************************************************************

sage -t -long -force_lib "devel/sage/sage/numerical/optimize.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/numerical/optimize.py", line 162:
    sage: find_minimum_on_interval(f, 1, 5, tol=1e-3)
Expected:
    (-3.28837136189098..., 3.42575079030572...)
Got:
    (-3.28837136189, 3.4257507903057229)

sage -t -long -force_lib "devel/sage/sage/rings/integer.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4738:
    sage: numpy.array(2**40).dtype
Expected:
    dtype('int64')
Got:
    dtype('object')
**********************************************************************

sage -t -long -force_lib "devel/sage/sage/functions/hyperbolic.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/hyperbolic.py", line 609:
    sage: arcsech(a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_23[4]>", line 1, in <module>
        arcsech(a)###line 609:
    sage: arcsech(a)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "/usr/lib/python2.7/site-packages/sage/functions/hyperbolic.py", line 612, in _eval_numpy_
        return arccosh(1.0 / x)
    ZeroDivisionError: float division by zero
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/hyperbolic.py", line 658:
    sage: arccsch(a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_26[4]>", line 1, in <module>
        arccsch(a)###line 658:
    sage: arccsch(a)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "/usr/lib/python2.7/site-packages/sage/functions/hyperbolic.py", line 661, in _eval_numpy_
        return arcsinh(1.0 / x)
    ZeroDivisionError: float division by zero
**********************************************************************

sage -t -long -force_lib "devel/sage/sage/functions/trig.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 827:
    sage: atan2(a, b)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_34[16]>", line 1, in <module>
        atan2(a, b)###line 827:
    sage: atan2(a, b)
      File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568)
    NotImplementedError: The Function arctan2 does not support numpy arrays as arguments
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 830:
    sage: atan2(1,a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_34[17]>", line 1, in <module>
        atan2(Integer(1),a)###line 830:
    sage: atan2(1,a)
      File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568)
    NotImplementedError: The Function arctan2 does not support numpy arrays as arguments
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/trig.py", line 833:
    sage: atan2(a, 1)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_34[18]>", line 1, in <module>
        atan2(a, Integer(1))###line 833:
    sage: atan2(a, 1)
      File "function.pyx", line 718, in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6475)
      File "function.pyx", line 357, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:3794)
      File "function.pyx", line 604, in sage.symbolic.function.Function._eval_numpy_ (sage/symbolic/function.cpp:5568)
    NotImplementedError: The Function arctan2 does not support numpy arrays as arguments
**********************************************************************

sage -t -long -force_lib "devel/sage/sage/functions/other.py"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/other.py", line 211:
    sage: ceil(a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_5[20]>", line 1, in <module>
        ceil(a)###line 211:
    sage: ceil(a)
      File "/usr/lib/python2.7/site-packages/sage/functions/other.py", line 251, in __call__
        return numpy.ceil(x)
    AttributeError: ceil
**********************************************************************
File "/usr/share/sage/devel/sage/sage/functions/other.py", line 366:
    sage: floor(a)
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_9[15]>", line 1, in <module>
        floor(a)###line 366:
    sage: floor(a)
      File "/usr/lib/python2.7/site-packages/sage/functions/other.py", line 405, in __call__
        return numpy.floor(x)
    AttributeError: floor
**********************************************************************

The last three are particularly concerning, could you check what's up with these in numpy?

[kcrisman]

This seems to have installed fine on OS X 10.4 PPC. I'm running doctests now (which will take a LONG time) but I did already see the matrix/matrix1.pyx one while testing something else.

[fbissey]

That's good that it at least install. On gentoo there are interactions problem between numpy and scipy to get cblas/blas/lapack right in scipy. I should check if the resulting scipy in vanilla sage suffers from the same problem. ​http://bugs.gentoo.org/show_bug.cgi?id=371099

The last few failing tests are show stoppers as far as I am concerned.

[kcrisman]

Okay, I got pretty much the same failures on a long run. One weird thing... did we change find_minimum_on_interval away from SciPy recently? This is the full output I get on the (otherwise same as your failure) test for that.

    sage: find_minimum_on_interval(f, 1, 5, tol=1e-3)
Expected:
    (-3.28837136189098..., 3.42575079030572...)
Got:
    (-3.28837136189, 3.4257507903057229)
GLPK Simplex Optimizer, v4.44
6 rows, 3 columns, 8 non-zeros
Preprocessing...
2 rows, 2 columns, 4 non-zeros
Scaling...
 A: min|aij| =  2.400e+01  max|aij| =  5.000e+01  ratio =  2.083e+00
GM: min|aij| =  8.128e-01  max|aij| =  1.230e+00  ratio =  1.514e+00
EQ: min|aij| =  6.606e-01  max|aij| =  1.000e+00  ratio =  1.514e+00
Constructing initial basis...
Size of triangular part = 2
*     0: obj =  -5.100000000e+01  infeas =  0.000e+00 (0)
*     1: obj =  -5.225000000e+01  infeas =  0.000e+00 (0)
OPTIMAL SOLUTION FOUND

[fbissey]

1.6.1 is now available. I'll have a look sometimes this week.

[fbissey]

Looks like more of the same but maybe a little bit more. There are a few other things that may interfere on my setup and I had to solve that bit: ​http://bugs.gentoo.org/show_bug.cgi?id=371099 before doing the trial.

[fbissey]

Once I had cleared a few blas problem that I had while solving the gentoo bug. I get exactly the same results. I also get the output from Karl in optimize. This is a verbose output of glpk which get called at some point. You only get to see the such verbose output if there was a failure in the test but not necessarily the test for which glpk is called funnily enough.

[fbissey]

Should we try to move forward on this again? I will have a go at trying it with 4.8.alpha2. Any ideas about the current failures?

[fbissey]

Ok this is still broken in 4.8.alpha2. Some food for thought

sage
----------------------------------------------------------------------
| Sage Version 4.8.alpha2, Release Date: 2011-11-19                  |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
sage: import numpy
sage: a = numpy.linspace(0,2,6)
sage: a
array([0.0, 0.4, 0.8, 1.2, 1.6, 2], dtype=object)
sage: ceil(a)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/home/fbissey/<ipython console> in <module>()

/usr/lib64/python2.7/site-packages/sage/functions/other.py in __call__(self, x, maximum_bits)
    250             elif type(x).__module__ == 'numpy':
    251                 import numpy
--> 252                 return numpy.ceil(x)
    253 
    254         x_original = x

AttributeError: ceil

with numpy-1.5.1 we just have

sage: a
array([0.0, 0.4, 0.8, 1.2, 1.6, 2])

I suspect that for this one (functions/other.py) and functions/trig.py the pynac interface may need to be touched. All stuff that takes an numpy array and return a numpy array may have trouble.

[fbissey]

Just to add a bit of info. Same python and ipython

ipython
Python 2.7.2 (default, Oct 28 2011, 11:36:16) 
Type "copyright", "credits" or "license" for more information.

IPython 0.10.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import numpy

In [2]: a = numpy.linspace(0,2,6)

In [3]: a
Out[3]: array([ 0. ,  0.4,  0.8,  1.2,  1.6,  2. ])

In [4]: ceil(a)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

/home/fbissey/<ipython console> in <module>()

NameError: name 'ceil' is not defined

In [5]: numpy.ceil(a)
Out[5]: array([ 0.,  1.,  1.,  2.,  2.,  2.])

So it works in ipython but we don't have the dtype=object attached to the array. Anyone knows what that means actually?

[fbissey]

numpy-1.5.1

sage: numpy.dtype(a[1])
dtype('float64')

numpy-1.6.1

sage: numpy.dtype(a[1])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/home/fbissey/<ipython console> in <module>()

TypeError: data type not understood

[fbissey]

sage: type(a[0])
<type 'float'>
sage: type(a[0:5])
<type 'numpy.ndarray'>
sage: type(a[2])
<type 'float'>
sage: type(a[3])
<type 'float'>
sage: type(a[4])
<type 'float'>
sage: type(a[5])
<type 'sage.rings.integer.Integer'>

For some reason numpy.linspace return an array with sage objects rather than just floats. I am guessing it may be deeper than that. Numpy probably uses sage types rather than converting to numpy types. The types of a[0] to a[4] is strange, is there a sage type for floats? I mean I tried numpy.linspace(0,2.0,6) and I got <type 'sage.rings.real_mpfr.RealNumber?'> for a[0] to a[4] and <type 'sage.rings.real_mpfr.RealLiteral?'> for a[5].

Doing a cast numpy.array(a,dtype=float) returns something that will work.

[fbissey]

I'll be starting to make patches to correct the doctests and document the new behavior in numpy 1.6.x. Some documentation may need updating.

[fbissey]

Actually I am a bit confused by one of the tests:

    ``plot_slope_field`` takes a function of two variables xvar and yvar
    (for instance, if the variables are `x` and `y`, take `f(x,y)`), and at
    representative points `(x_i,y_i)` between xmin, xmax, and ymin, ymax
    respectively, plots a line with slope `f(x_i,y_i)` (see below).

    ``plot_slope_field(f, (xvar, xmin, xmax), (yvar, ymin, ymax))``

    EXAMPLES: 

    A logistic function modeling population growth::

        sage: x,y = var('x y')
        sage: capacity = 3 # thousand
        sage: growth_rate = 0.7 # population increases by 70% per unit of time
        sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2))

That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be.

[fbissey]

I am guessing it should have been

growth_rate*(1-x/capacity)*y

y being the time and x the population in thousands. That satisfy a simple dimensional analysis.

[jason]

No, wasn't it right before? That's the standard logistic population model, where y is the population. See ​http://en.wikipedia.org/wiki/Logistic_function#In_ecology:_modeling_population_growth, for example.

[kcrisman]

sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2))
}}}}
> That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in

This should be ok. The point is that the slope dy/dx is a function of y only; this can be completely symbolically solved using separation of variables to get the [http://en.wikipedia.org/wiki/Logistic_function#In_ecology:_modeling_population_growth logistic function].

> this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be.

If this is the division by zero stuff, it's the whole business with the arrowheads having zero length. See #11208. I think Jason has a fix for this, actually, but I don't recall the whole story of what happened once it was reported to mpl.

[kcrisman]

Sorry for that noise. Jason is right.

sage: plot_slope_field(growth_rate*(1-y/capacity)*y, (x,0,5), (y,0,capacity*2))

That function is independent of x, the next example after this depend on both x and y. I am suspecting the messages we get in

This should be ok. The point is that the slope dy/dx is a function of y only; this can be completely symbolically solved using separation of variables to get the ​logistic function.

this particular case is quite justified and it is surprising we didn't get any before. I am not sure what the correct formula should be.

If this is the division by zero stuff, it's the whole business with the arrowheads having zero length. See #11208. I think Jason has a fix for this, actually, but I don't recall the whole story of what happened once it was reported to mpl.

[fbissey]

OK I was interpreting the formula another way because I wasn't sure what it is actually computing. Seems like #11208 numpy, mpl and scipy all go hands in hands so if something start happening with one the others may get it too.

Overall the change of behavior of numpy around type conversion is quite annoying.

[fbissey]

first draft at updating the documentation

[fbissey]

It strikes me that the problem may be more an issue of the coercion to numpy types not working properly with numpy-1.6. I will look into that.

[fbissey]

Any ideas why stuff like this is happening with numpy-1.6.x

sage: numpy.arange(10.0)
array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])
sage: numpy.linspace(0.0,9.0,10)
array([0.000000000000000, 1.00000000000000, 2.00000000000000,
       3.00000000000000, 4.00000000000000, 5.00000000000000,
       6.00000000000000, 7.00000000000000, 8.00000000000000,
       9.00000000000000], dtype=object)

[jason]

It looks like it didn't automatically convert from RR to float, like it should. That's what these two lines should do in sage/rings/real_mpfr.pyx

cdef object numpy_double_interface = {'typestr': '=f8'}
cdef object numpy_object_interface = {'typestr': '|O'}

and this property in that same file:

    property __array_interface__:
        def __get__(self):
            """
            Used for NumPy conversion.
            
            EXAMPLES::

                sage: import numpy
                sage: numpy.arange(10.0)
                array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])
                sage: numpy.array([1.0, 1.1, 1.2]).dtype
                dtype('float64')
                sage: numpy.array([1.000000000000000000000000000000000000]).dtype
                dtype('object')
            """
            if (<RealField_class>self._parent).__prec <= 57: # max size of repr(float)
                return numpy_double_interface
            else:
                return numpy_object_interface

So maybe numpy doesn't do something with __array_interface__ anymore in linspace?

Can you try numpy.linspace(0.0,9.0,float(10)), numpy.linspace(0.0,float(9),10), numpy.linspace(float(0),9.0,10) and various other combinations of float vs. Sage types in the arguments to narrow down what is triggering it?

[fbissey]

I was writing something else:

And more subtly:

sage: a=numpy.linspace(0,9,10)
sage: a
array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9], dtype=object)
sage: type(a[0])
<type 'float'>
sage: type(a[9])
<type 'sage.rings.integer.Integer'>

So linspace and arange deal with sage input very differently. There should be some coercion but it is not working properly.

now for what you are asking:

sage: numpy.linspace(0.0,9.0,float(10))
array([0.000000000000000, 1.00000000000000, 2.00000000000000,
       3.00000000000000, 4.00000000000000, 5.00000000000000,
       6.00000000000000, 7.00000000000000, 8.00000000000000,
       9.00000000000000], dtype=object)
sage: numpy.linspace(0.0,float(9),10)
array([0.000000000000000, 1.00000000000000, 2.00000000000000,
       3.00000000000000, 4.00000000000000, 5.00000000000000,
       6.00000000000000, 7.00000000000000, 8.00000000000000, 9.0], dtype=object)
sage: numpy.linspace(float(0),9.0,10)
array([0.000000000000000, 1.00000000000000, 2.00000000000000,
       3.00000000000000, 4.00000000000000, 5.00000000000000,
       6.00000000000000, 7.00000000000000, 8.00000000000000,
       9.00000000000000], dtype=object)

I know about sage/rings/real_mpfr.pyx and the others that's how I got on thinking of coercion not working.

[fbissey]

Weirder! The code for linspace uses arange inside. No change in the code between the 1.5.1 and 1.6.1 but the juicy important bit is as follow

        step = (stop-start)/float(num)
        y = _nx.arange(0, num) * step + start

step will become float (unless start or stop is complex or something). But the type of step and start will influence the final array.

[fbissey]

Anyone wants to ask on a numpy mailing list if it is a sign of bug in numpy or something that is clearly under-documented?

[strogdon]

Replying to jason:

So maybe numpy doesn't do something with __array_interface__ anymore in linspace?

I've been tinkering here and this is exactly what is happening. I deleted all the

property __array_interface__:

blocks that were added by ​5081-numpy-types.patch and ​6506-numpy-types.patch with no apparent effect on what numpy.linspace() returns with numpy-1.6.1 installed. Of course, the deleted blocks do affect what numpy.arange() and numpy.array() return:

sage: numpy.arange(10.0)
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object)
sage: numpy.array([1.0, 1.1, 1.2]).dtype
dtype('object')

Also the example failure

from scipy import stats
stats.uniform(0,15).ppf([0.5,0.7])

from ticket #5081 now again fails with numpy-1.6.x (s-o-g result):

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/storage/strogdon/gentoo/usr/local/portage/sage-on-gentoo/<ipython console> in <module>()

/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/scipy/stats/distributions.pyc in ppf(self, q)
    449 
    450     def ppf(self, q):
--> 451         return self.dist.ppf(q, *self.args, **self.kwds)
    452 
    453     def isf(self, q):

/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/scipy/stats/distributions.pyc in ppf(self, q, *args, **kwds)
   1514             goodargs = argsreduce(cond, *((q,)+args+(scale,loc)))
   1515             scale, loc, goodargs = goodargs[-2], goodargs[-1], goodargs[:-2]
-> 1516             place(output,cond,self._ppf(*goodargs)*scale + loc)
   1517         if output.ndim == 0:
   1518             return output[()]

/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/numpy/lib/function_base.pyc in place(arr, mask, vals)
   1333 
   1334     """
-> 1335     return _insert(arr, mask, vals)
   1336 
   1337 def _nanop(op, fill, a, axis=None):

TypeError: array cannot be safely cast to required type

Starting with numpy-1.6.0 scalar upcasting rules were changed ​http://docs.scipy.org/doc/numpy/reference/ufuncs.html and I'm wondering if this change is interfering with how sage/cython is handling the python __array_interface__ extension, at least relative to numpy.linspace(), and perhaps other numpy functions that are pure python.

[fbissey]

That's the best clue we had in ages Steve. I am not sure when I will be able to dig it further as we have an extra long week end here (Tuesday is a university holyday).

[mhansen]

Here is a minimal example of the problem:

sage: import numpy
sage: numpy.array(0.5)
array(0.500000000000000, dtype=object)
sage: numpy.array(0.5r)
array(0.5)

where as before it was

sage: import numpy
sage: numpy.array(0.5)
array(0.5)
sage: numpy.array(0.5r)
array(0.5)

[mhansen]

Also, we should incorporate #7831

[fbissey]

Sure Mike do you have any idea on how to fix the coercion problem? The way it's going we'll update directly to 1.7.

[mhansen]

Looking into this it seems that the issue is that in numpy/core/src/multiarray/ctors.c, PyArray_FromAny no longer calls _array_find_type which I believe was responsible for accessing the __array_interface__ attribute on our objects. I'm sending a message to numpy-discussion now.

[mhansen]

See ​http://mail.scipy.org/pipermail/numpy-discussion/2012-May/062560.html

[fbissey]

Thanks Mike, hopefully we'll get a meaningful answer.

[fbissey]

Looks like 1.6.2 was released last week. I guess I'll update my spkg.

[fbissey]

Looks like I found the commit where the change has been made ​https://github.com/numpy/numpy/commit/2635398db3f26529ce2aaea4028a8118844f3c48

If you look carefully the call to _array_find_type is still there but it has changed location. In 1.5.1 there was a sequence if(xx==NULL) then call _array_find_type else do some other stuff. In 1.6.1 the structure if(xx!=NULL) then do the other stuff else call _array_find_type.

I will try to write a patch to see if this the cause of our observed behaviour.

[fbissey]

I wrote a small patch to just ignore any code other than _array_find_type in ctors.c no luck so I don't think that's the real source of the problem at all. I thought the presence of a requested_dtype passed to this function could have been the source of the problem but my patch excludes that possibility entirely. So we have to go back to looking.

[fbissey]

Problem still present with numpy-1.6.2

[fbissey]

New spkg for 1.6.2 rebased on 1.5.1.p1 as well.

[fbissey]

Mike, I think if you want to ask a question that may have a chance to attract attention on numpy-discussion you should ask: Why do arange and linspace return different dtype object on similar inputs of the same type?

You can then expand on the fact that arange which is completely coded in C respect our request from array interface but linspace which call arange and then do something in python with the result doesn't respect it.

[strogdon]

Replying to mhansen:

See ​http://mail.scipy.org/pipermail/numpy-discussion/2012-May/062560.html

I believe Mike is correct in that the commit referenced at the above numpy-discussion is the source of the "regression" in being unable to access the Sage-assigned __array_interface__ attribute. I cloned the numpy.git repository, set the master to the commit mentioned my Mike, and from that commit generated a numpy spkg which I installed. Sage manifested the undesirable "dtype=object" when tested. Now if I set the master to the commit just prior to the above mentioned commit and generated a new numpy spkg, then when Sage was tested I got the numpy-1.5.1 behavior. I'm just not sure what's going inside numpy?

[fbissey]

OK so this is indeed the guilty commit. I tried something very simple to see if it was the small bit of code around _array_find_dtype and it was obviously insufficient to uncover the problem. It is quite a big commit,. I am thinking that considering we see an inconsistent behavior that we should just open a bug with numpy.

[fbissey]

Thanks Mike for the post on numpy's github tracker ​https://github.com/numpy/numpy/issues/291 I am not sure how they distribute the load between the github tracker and their trac server ​http://projects.scipy.org/numpy/report. I just hope they don't ignore it.

[fbissey]

It looks like numpy may revert to the old behavior in 1.7.0. There is some traction upstream for this to happen.

[jason]

Looks like 1.7.0 will have a fix: ​https://github.com/numpy/numpy/issues/291. 1.7.0 should be out very soon now.

[fbissey]

I know and I am waiting for that.

[fbissey]

I have been toying with numpy 1.7.0b2 in sage-on-gentoo. I have added a small patch I discussed with Steve Trogdon that I will attach. In the meantime I have gobbles of the following:

sage -t -long -force_lib "devel/sage-main/sage/functions/log.py"
**********************************************************************
File "/usr/share/sage/devel/sage-main/sage/functions/log.py", line 630:
    sage: lambert_w(RDF(1))
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_12[4]>", line 1, in <module>
        lambert_w(RDF(Integer(1)))###line 630:
    sage: lambert_w(RDF(1))
      File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 560, in __call__
        return BuiltinFunction.__call__(self, 0, args[0], **kwds)
      File "function.pyx", line 437, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:5006)
      File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 607, in _eval_
        return self._evalf_(n, z, parent=sage_structure_coerce_parent(z))
      File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 636, in _evalf_
        return scipy.special.lambertw(z, n)
      File "lambertw.pyx", line 342, in scipy.special.lambertw.lambertw (scipy/special/lambertw.c:1438)
    TypeError: expected a readable buffer object
**********************************************************************
1 items had failures:
   1 of   6 in __main__.example_12
***Test Failed*** 1 failures.
For whitespace errors, see the file /home/fbissey/.sage/tmp/log_1583.py
         [2.6 s]

Anything using matplotlib spams you to death with these. Without the patch we get the following instead:

sage -t -long -force_lib "devel/sage-main/sage/functions/log.py"
**********************************************************************
File "/usr/share/sage/devel/sage-main/sage/functions/log.py", line 630:
    sage: lambert_w(RDF(1))
Exception raised:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_12[4]>", line 1, in <module>
        lambert_w(RDF(Integer(1)))###line 630:
    sage: lambert_w(RDF(1))
      File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 560, in __call__
        return BuiltinFunction.__call__(self, 0, args[0], **kwds)
      File "function.pyx", line 437, in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:5006)
      File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 607, in _eval_
        return self._evalf_(n, z, parent=sage_structure_coerce_parent(z))
      File "/usr/lib64/python2.7/site-packages/sage/functions/log.py", line 636, in _evalf_
        return scipy.special.lambertw(z, n)
      File "lambertw.pyx", line 342, in scipy.special.lambertw.lambertw (scipy/special/lambertw.c:1438)
    ValueError: Missing __array_interface__ shape
**********************************************************************
1 items had failures:
   1 of   6 in __main__.example_12
***Test Failed*** 1 failures.
For whitespace errors, see the file /home/fbissey/.sage/tmp/log_15682.py
         [2.6 s]

It seems that the patch introduce some expectation on the type that are not identified at the present time.

[fbissey]

add array_shape to coercion rule

[fbissey]

Here is a spkg so other people can enjoy. numpy 1.7 also has a new API. By default you can use the old one but we may want to move on to the new one eventually.

[fbissey]

Steve pointed out to me that 1.7.0b2 didn't contain 2 little patches that we need and that will hopefully be in 1.7.0 final. The spkg is updated and there are way fewer doctest failures (matplotlib 1.2.0 will be a major source of doctest noise when we upgrade). So the current failures: 1) cosmetics

sage -t -long -force_lib "devel/sage-main/sage/matrix/matrix1.pyx"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/matrix/matrix1.pyx", line 494:
    sage: sorted(numpy.typecodes.items())
Expected:
    [('All', '?bhilqpBHILQPfdgFDGSUVO'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
Got:
    [('All', '?bhilqpBHILQPefdgFDGSUVOMm'), ('AllFloat', 'efdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'efdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
**********************************************************************
1 items had failures:

2) lots of divide by 0 warnings - including our old friend riemann.pyx encountered in a previous upgrade.

sage -t -long -force_lib "devel/sage-main/sage/calculus/riemann.pyx"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 111:
    sage: m = Riemann_Map([f], [fprime], 0)  # long time (4 sec)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 595:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 647:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 712:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 801:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 867:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 163:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 214:
    sage: isinstance(Riemann_Map([f], [fprime], 0)._repr_(), str)  # long time
Expected:
    True
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
    True
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 228:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 334:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 404:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 447:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 515:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/riemann.pyx", line 543:
    sage: m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************

sage -t -long -force_lib "devel/sage-main/sage/calculus/interpolators.pyx"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/interpolators.pyx", line 52:
    sage: m = Riemann_Map([lambda x: ps.value(real(x))], [lambda x: ps.derivative(real(x))],0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/calculus/interpolators.pyx", line 183:
    sage: m = Riemann_Map([lambda x: cs.value(real(x))], [lambda x: cs.derivative(real(x))], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************

sage -t -long -force_lib "devel/sage-main/sage/symbolic/function.pyx"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/symbolic/function.pyx", line 627:
    sage: csc(a)
Expected:
    array([        inf,  1.18839511,  1.09975017,  7.0861674 , -1.32134871])
Got:
    doctest:270: RuntimeWarning: divide by zero encountered in divide
    array([        inf,  1.18839511,  1.09975017,  7.0861674 , -1.32134871])
**********************************************************************

sage -t -long -force_lib "devel/sage-main/sage/functions/hyperbolic.py"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/functions/hyperbolic.py", line 610:
    sage: arcsech(a)
Expected:
    array([       inf,  1.3169579,  0.       ])
Got:
    doctest:613: RuntimeWarning: divide by zero encountered in divide
    array([       inf,  1.3169579,  0.       ])
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/functions/hyperbolic.py", line 659:
    sage: arccsch(a)
Expected:
    array([        inf,  1.44363548,  0.88137359])
Got:
    doctest:662: RuntimeWarning: divide by zero encountered in divide
    array([        inf,  1.44363548,  0.88137359])
**********************************************************************

sage -t -long -force_lib "devel/sage-main/sage/modules/vector_double_dense.pyx"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/modules/vector_double_dense.pyx", line 712:
    sage: w.norm(p=-1.6)
Expected:
    0.0
Got:
    doctest:1992: RuntimeWarning: divide by zero encountered in power
    0.0
**********************************************************************

3) odd bits that needs looking at:

sage -t -long -force_lib "devel/sage-main/sage/rings/real_mpfr.pyx"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/rings/real_mpfr.pyx", line 1270:
    sage: numpy.array([1.000000000000000000000000000000000000]).dtype
Expected:
    dtype('object')
Got:
    dtype('O')
**********************************************************************

and similarly in rings/integer.pyx, rings/complex_number.pyx and rings/rational.pyx.

4) ????

sage -t -long -force_lib "devel/sage-main/sage/plot/matrix_plot.py"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/plot/matrix_plot.py", line 438:
    sage: matrix_plot([[sin(x), cos(x)], [1, 0]])
Expected:
    Traceback (most recent call last):
    ...
    ValueError: can not convert entries to floating point numbers
Got:
    Traceback (most recent call last):
      File "/usr/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_8[40]>", line 1, in <module>
        matrix_plot([[sin(x), cos(x)], [Integer(1), Integer(0)]])###line 438:
    sage: matrix_plot([[sin(x), cos(x)], [1, 0]])
      File "/usr/lib64/python2.7/site-packages/sage/misc/decorators.py", line 456, in wrapper
        return func(*args, **kwds)
      File "/usr/lib64/python2.7/site-packages/sage/misc/decorators.py", line 456, in wrapper
        return func(*args, **kwds)
      File "/usr/lib64/python2.7/site-packages/sage/misc/decorators.py", line 534, in wrapper
        return func(*args, **options)
      File "/usr/lib64/python2.7/site-packages/sage/plot/matrix_plot.py", line 478, in matrix_plot
        raise TypeError, "mat must be a Matrix or a two dimensional array"
    TypeError: mat must be a Matrix or a two dimensional array
**********************************************************************

and a last one which I am not sure is numpy or sage-on-gentoo or a combination:

sage -t -long -force_lib "devel/sage-main/sage/matrix/matrix_double_dense.pyx"
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/matrix/matrix_double_dense.pyx", line 4033:
    sage: A.exp(order=2)
Expected:
    [51.8888631634 74.6198348038]
    [111.929752206 163.818615369]
Got:
    [51.9689561987  74.736564567]
    [112.104846851 164.073803049]
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.4.rc0/devel/sage-main/sage/matrix/matrix_double_dense.pyx", line 4052:
    sage: A.exp(order=2)
Expected:
    [-19.6130852955 + 12.5327938535*I   3.81156364812 + 28.891438232*I]
    [-32.3827876895 + 21.9087393169*I   2.29565402142 + 44.915581543*I]
Got:
    [-19.6146029538 + 12.5177438468*I  3.79496364496 + 28.8837993066*I]
    [-32.3835809809 + 21.8842359579*I  2.26963300409 + 44.9013248277*I]
**********************************************************************

[fbissey]

I forgot that one which is another that puzzle me

sage -t -long -force_lib "devel/sage-main/sage/plot/plot3d/implicit_surface.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage-main/sage/plot/plot3d/implicit_surface.pyx", line 563:
    sage: cube_marcher.y_vertices.any() or cube_marcher.z_vertices.any() # This shouldn't affect the Y or Z vertices.
Expected:
    False
Got nothing
**********************************************************************
File "/usr/share/sage/devel/sage-main/sage/plot/plot3d/implicit_surface.pyx", line 458:
    sage: cube_marcher.x_vertices.any() # This shouldn't affect the X vertices.
Expected:
    False
Got nothing
**********************************************************************

And the matrix_double_dense.pyx filures seem to have disappeared after my last rebuild.

[fbissey]

Steve pointed out that the array_interface-shape.patch patch is not necessary as the old numpy 1.5.1 behavior as been restored in full. It doesn't hurt either way but we don't need to add something we don't need.

[fbissey]

After pulling the latest code from 1.7.x maintenance branch I can now see that the patch is still necessary. The pull that Steve had mentioned to me happened on the master not on the 1.7.x maintenance branch so it is not in the current 1.7 branch on github.

[fbissey]

Sorry about the last message after more tests my conclusions are:

• the current tip of the 1.7.x maintenance branch as well as 1.7.0b2 need two extra patches (included in the current spkg)
• the array_interface-shape.patch patch is not necessary after all.

During my early tests I was expecting the two numpy patch to have been merged in the 1.7.x maintenance branch and that lead to confusion.

[jason]

Does the rst patch above need to be updated to not talk about the 1.6 bug of converting Sage types?

[fbissey]

At this stage there are no patch to add to sage the rst patch was something I started before I realized that we should do something else. I will put any patch I create for sage in the summary. There are a at least a couple of things we can fix now. The big question in my mind is: what do we do with all those divide by 0 warnings? Last time we had the author of riemann.pyx fix things.

[jason]

I'll try taking a look at it. I don't have the most recent Sage yet, but I guess it's not a huge problem right now since numpy 1.7 hasn't been released yet either.

[fbissey]

tentative fix for matrix_plot.py - not sure it is the right thing to do as we have a change of behavior rather than output.

[fbissey]

new typecodes

[fbissey]

OK for you to have a look in the meantime here are the easy patches. Not completely sure it is the right thing to do in matrix_plot.py. While we are expecting an error we were expecting a different kind of error before.

[strogdon]

I'm unable to get the extra patches included in numpy-1.7.0b2.spkg to apply. The file to be patched in the header of each patch is missing a leading "/" for

patch -p1 <"$patch"

to work. One possibility is to change "-p1" to "-p0" in spkg-install.

[fbissey]

Thanks for testing I forgot to do that and I tested my first version of the spkg but not the second. I will fix this shortly.

[fbissey]

spkg updated in the same location. I just edited the patch as we want the patching command to be uniform with the already existing cygwin patch which is still needed in my opinion.

[strogdon]

Here is some background on the failing test

sage -t -long -force_lib "devel/sage-main/sage/plot/matrix_plot.py"

that's mentioned above ​above. The failure in Sage of

sage: matrix_plot([[sin(x), cos(x)], [1, 0]])

seems to be associated with the following sequence

sage: import numpy as np
sage: np.asarray([[sin(x), cos(x)], [1, 0]], dtype = float)

With numpy-1.5.1 I get

sage:  np.asarray([[sin(x), cos(x)], [1, 0]], dtype = float)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/storage/sage/sage-5.5.rc0/<ipython console> in <module>()

/storage/sage/sage-5.5.rc0/local/lib/python2.7/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    282 
    283     """
--> 284     return array(a, dtype, copy=False, order=order)
    285 
    286 def asanyarray(a, dtype=None, order=None):

ValueError: setting an array element with a sequence.

and with numpy-1.7.0b2

sage: np.asarray([[sin(x), cos(x)], [1, 0]], dtype = float)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/storage/sage/sage-5.5.rc0/<ipython console> in <module>()

/storage/sage/sage-5.5.rc0/local/lib/python2.7/site-packages/numpy/core/numeric.pyc in asarray(a, dtype, order)
    318 
    319     """
--> 320     return array(a, dtype, copy=False, order=order)
    321 
    322 def asanyarray(a, dtype=None, order=None):

/storage/sage/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/symbolic/expression.so in sage.symbolic.expression.Expression.__float__ (sage/symbolic/expression.cpp:7023)()

TypeError: unable to simplify to float approximation

So it would appear that the ValueError? when using numpy-1.5.1 is thrown my numpy but the TypeError? when using numpy-1.7.0b2 is thrown by Sage.

[fbissey]

Ondřej just pulled a request with all we need in the 1.7 maintainance branch. Expect new spkg/ebuild early next week based on a git pull.

[fbissey]

There is a now a rc1. I will update when I can.

[fbissey]

spkg updated. I will soon run test against a vanilla 5.6.rc0 (as opposed to sage-on-gentoo).

[fbissey]

Tests run here is the list of failing doctest

        sage -t -long devel/sage-main/sage/calculus/interpolators.pyx # 2 doctests failed
        sage -t -long devel/sage-main/sage/misc/inline_fortran.py # 4 doctests failed
        sage -t -long devel/sage-main/sage/calculus/riemann.pyx # 14 doctests failed
        sage -t -long devel/sage-main/sage/symbolic/function.pyx # 1 doctests failed
        sage -t -long devel/sage-main/sage/plot/plot_field.py # 1 doctests failed
        sage -t -long devel/sage-main/sage/plot/plot3d/implicit_surface.pyx # 2 doctests failed
        sage -t -long devel/sage-main/sage/plot/matrix_plot.py # 1 doctests failed
        sage -t -long devel/sage-main/sage/matrix/matrix1.pyx # 1 doctests failed
        sage -t -long devel/sage-main/sage/modules/vector_double_dense.pyx # 1 doctests failed
        sage -t -long devel/sage-main/sage/rings/complex_number.pyx # 1 doctests failed
        sage -t -long devel/sage-main/sage/rings/real_mpfr.pyx # 1 doctests failed
        sage -t -long devel/sage-main/sage/rings/rational.pyx # 1 doctests failed
        sage -t -long devel/sage-main/sage/rings/integer.pyx # 1 doctests failed
        sage -t -long devel/sage-main/sage/functions/hyperbolic.py # 2 doctests failed

There are a couple of new things I believe.

sage -t -long "devel/sage-main/sage/misc/inline_fortran.py" 
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/misc/inline_fortran.py", line 29:
    sage: fortran(_example)
Exception raised:
    Traceback (most recent call last):
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[4]>", line 1, in <module>
        fortran(_example)###line 29:
    sage: fortran(_example)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/sage/misc/inline_fortran.py", line 21, in __call__
        return self.eval(*args, **kwds)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/sage/misc/inline_fortran.py", line 84, in eval
        f2py.compile(x, name, extra_args = extra_args, source_fn=fortran_file)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/numpy/f2py/__init__.py", line 40, in compile
        s,o = exec_command(c)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/numpy/distutils/exec_command.py", line 197, in exec_command
        if _with_python and (0 or sys.stdout.fileno()==-1):
    AttributeError: _SpoofOut instance has no attribute 'fileno'
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/misc/inline_fortran.py", line 32:
    sage: fib(n,int(10))
Exception raised:
    Traceback (most recent call last):
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[7]>", line 1, in <module>
        fib(n,int(Integer(10)))###line 32:
    sage: fib(n,int(10))
    NameError: name 'fib' is not defined
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/misc/inline_fortran.py", line 33:
    sage: n
Expected:
    array([  0.,   1.,   1.,   2.,   3.,   5.,   8.,  13.,  21.,  34.])
Got:
    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/misc/inline_fortran.py", line 39:
    sage: fortran.eval("SYNTAX ERROR !@#$")
Expected:
    Traceback (most recent call last):
    ...
    RuntimeError: failed to compile Fortran code:...
Got:
    Traceback (most recent call last):
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[10]>", line 1, in <module>
        fortran.eval("SYNTAX ERROR !@#$")###line 39:
    sage: fortran.eval("SYNTAX ERROR !@#$")
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/sage/misc/inline_fortran.py", line 84, in eval
        f2py.compile(x, name, extra_args = extra_args, source_fn=fortran_file)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/numpy/f2py/__init__.py", line 40, in compile
        s,o = exec_command(c)
      File "/home/work/fbissey/sandbox/sage-5.6.rc0/local/lib/python/site-packages/numpy/distutils/exec_command.py", line 197, in exec_command
        if _with_python and (0 or sys.stdout.fileno()==-1):
    AttributeError: _SpoofOut instance has no attribute 'fileno'

which also happens in 5.5 but I overlooked previously (because in my sage-on-gentoo this test always failed in parallel). This is therefore not caused by #13887. This is new I believe

sage -t -long "devel/sage-main/sage/plot/plot_field.py"     
**********************************************************************
File "/home/work/fbissey/sandbox/sage-5.6.rc0/devel/sage-main/sage/plot/plot_field.py", line 49:
    sage: r.yvec_array
Expected:
    masked_array(data = [0.0 0.707106781187 0.707106781187 0.894427191],
                 mask = [False False False False],
           fill_value = 1e+20)
Got:
    masked_array(data = [0.0 0.7071067811865475 0.7071067811865475 0.8944271909999159],
                 mask = [False False False False],
           fill_value = 1e+20)
    <BLANKLINE>

Numerical noise and a blank line..

[fbissey]

I just edited my last comment to repair some mismatching between sage version (I ran the tests on files from sage 5.6.rc0 with sage 5.5...)

[fbissey]

Breakage of inline fortran reported upstream at ​https://github.com/numpy/numpy/issues/2915 as a recent commit look suspicious.

[fbissey]

It's more than suspicious. Reverting one line of this patch in the installed file enables the doctest to complete without errors.

[fbissey]

Numpy upstream pointed me to #12415 which does solve the problem. So depending on it.

[jdemeyer]

Please remove all reference to sage_fortran (#13349 and #13992), i.e. remove the following lines in spkg-install:

export FC="${SAGE_LOCAL}/bin/sage_fortran"
export F77="${SAGE_LOCAL}/bin/sage_fortran"
export F90="${SAGE_LOCAL}/bin/sage_fortran"
export F95="${SAGE_LOCAL}/bin/sage_fortran"

[fbissey]

Will do. It's not like we are any closer to final on that one.

[kcrisman]

Just a word of encouragement to those on this ticket - we've had a number of requests for ​pandas in Sage lately, but apparently this would be a prereq for easy_install-ing that, so we'd need it.

[fbissey]

Thanks Karl-dieter. Actually if you could provide help with the implicit_surface.py test (cube marcher) or get someone who know the code on that case we possibly could wrap it. We would may be wait for an official 1.7.0 but we'd be ready.

[jason]

I reviewed the marching cubes code when it went in. I think here is the problem. With the previous version of numpy:

import numpy
numpy.array([[None,None],[None,None]]).any()

returns False, but now returns None.

[jason]

I posted to numpy-discussion about it: ​http://mail.scipy.org/pipermail/numpy-discussion/2013-February/065377.html

[fbissey]

Replying to jason:

I reviewed the marching cubes code when it went in. I think here is the problem. With the previous version of numpy:

import numpy
numpy.array([[None,None],[None,None]]).any()

returns False, but now returns None.

Yes that correlate with some experiments I have with the marching cube code but I hadn't isolated the problem that deeply. I'll look at the discussion but I am not sure that is a wrong behavior.

[jason]

I can see the behavior choice going either way. The question for the numpy folks was if the change was intentional. If not, the follow-up question is whether it should be changed to be backwards compatible.

At any rate, I think fixing the doctests to return nothing is fine.

[fbissey]

Before I do that can I ask about the purpose of test? The comment suggest that the calls have no adverse effects but this is not tested afterwards. Because I do not see the point of the test I would just remove it. Any comment on that?

Once this is sorted, we'll have to see how things combine with #12415. From what I understand from it and some other threads, some "divide by 0 warnings" will be with us forever. We probably should check when they appear but they cannot be hidden anymore (I think evanandel was playing hide and seek with them in riemann.pyx).

[strogdon]

This references the failure in the test

sage -t -long -force_lib "devel/sage-main/sage/plot/matrix_plot.py"

I have some concerns with the following block of code in matrix_plot.py

    try:
        if sparse:
            xy_data_array = mat
        else:
            xy_data_array = np.asarray(mat, dtype = float)
    except TypeError:
        raise TypeError, "mat must be a Matrix or a two dimensional array"
    except ValueError:
        raise ValueError, "can not convert entries to floating point numbers"

    if len(xy_data_array.shape) < 2:
        raise TypeError, "mat must be a Matrix or a two dimensional array"

The subject failure occurs when the object "mat" is not sparse. In which case, the errors are thrown by

np.asarray(mat, dtype = float)

I have not been able to pass any object "mat" to np.asarray() that returns a TypeError or ValueError that remotely hints of the cause as being due to

"mat must be a Matrix or a two dimensional array"

I question whether this information can be returned by np.asarray() when used from Sage. I could be wrong though. The referenced failure does return a TypeError ​See above. But this error reflects the inability to coerce the symbolic [sin(x), cos(x)] expressions to floats. It is possible to generate a ValueError when the object "mat" contains complex entries:

sage: import numpy
sage: numpy.asarray([[1+1j,2+3j],[1,1]], dtype=float)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-2bd7229efabd> in <module>()
----> 1 numpy.asarray([[Integer(1)+ComplexNumber(0, '1'),Integer(2)+ComplexNumber(0, '3')],[Integer(1),Integer(1)]], dtype=float)

/storage/sage/sage-5.7.beta3/local/lib/python2.7/site-packages/numpy/core/numeric.pyc in asarray(a, dtype, order)
    318 
    319     """
--> 320     return array(a, dtype, copy=False, order=order)
    321 
    322 def asanyarray(a, dtype=None, order=None):

ValueError: setting an array element with a sequence.

This is they type of error that's returned for the referenced test when using numpy-1.5.1. Sage interprets this error as

"can not convert entries to floating point numbers"

This seems to be a correct interpretation, at least from the python side with the numpy of this ticket:

>>> import numpy
>>> numpy.asarray([[1+1j,2+3j],[1,1]], dtype=float)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/storage/sage/sage-5.7.beta3/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 320, in asarray
    return array(a, dtype, copy=False, order=order)
TypeError: can't convert complex to float

although here there is a TypeError?

So it seems to me that regardless of whether there is a TypeError or ValueError the root cause is inability to coerce to a float. The question of whether "mat" is a Matrix or appropriate array is determined by the lines in the code that parse the shape attribute of an object that has been successfully converted to a ndarray. But this is probably minor compared to #12415. Perhaps Mike (mhansen) has some input?

[fbissey]

That would be good to have a little bit more on this although it is indeed not as big as #12415.

In other news 1.7rc2 has been released less than an hour ago. I am trying to get it up as quickly as possible with the changes Jeroen wants.

[fbissey]

spkg updated to rc2 in summary.

[jdemeyer]

When removing the Fortran stuff, you might need to add -fPIC as in #14022.

[fbissey]

Replying to jdemeyer:

When removing the Fortran stuff, you might need to add -fPIC as in #14022.

Funilly enough I thought about it :) and it is in. There is a very good chance that rc2 will be identical to the final release.

[fbissey]

ok build and tested on top of 5.7.beta3 (but without #12415) and there are no new doctests failures. sage-on-gentoo and an older vanilla build got me a couple of latex output slightly different latex output. Very strange but probably steeming from something else.

[fbissey]

OK I completely missed the release which happened 4 days after rc2. New spkg will be available soon.

[fbissey]

spkg updated to numpy 1.7.0 final which is identical to 1.7.0.rc2. We need patches for the divide by zero warnings and the marching cubes test.

[fbissey]

correct dtype output for objects

[fbissey]

rebased one patch against 5.9.beta0, more patches to follow.

[fbissey]

add divide by 0 warnings

[fbissey]

fix cube_marcher doctests

[fbissey]

fix numerical noise

[fbissey]

More patches. Up to review.

[jdemeyer]

I assume this is caused by Numpy:

sage -t devel/sage/sage/calculus/riemann.pyx
**********************************************************************
File "devel/sage/sage/calculus/riemann.pyx", line 120, in sage.calculus.riemann.Riemann_Map
Failed example:
    m = Riemann_Map([f, hf], [fprime, hfprime], 0.5 + 0.5*I)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************

[fbissey]

It smells like one. There were 14 warnings and I patched 14 warnings in that file and that wasn't one of them. Where the heck does it come from?

[fbissey]

I checked I don't have this one against 5.9.beta2 so it must be triggered by something else. I am OK to add a small patch for it but would very much like to know what ticket triggered it.

[jdemeyer]

This was with a testing version of sage-5.9.beta4, so there were a bunch of tickets added.

[fbissey]

Not really sure what would trigger something like that from that lot of tickets, my suspicions go with #13031, a change in the way the extension is built. I am assuming you are releasing 5.9.beta3 fairly soon so at least I'll be able to see if it appears on top of it.

[strogdon]

Replying to jdemeyer:

I assume this is caused by Numpy:

sage -t devel/sage/sage/calculus/riemann.pyx
**********************************************************************
File "devel/sage/sage/calculus/riemann.pyx", line 120, in sage.calculus.riemann.Riemann_Map
Failed example:
    m = Riemann_Map([f, hf], [fprime, hfprime], 0.5 + 0.5*I)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************

I get this failure here too and I'm testing sage-5.9.beta1 and I don't get the failure associated with the patch for the test starting at line 109

sage: f(t) = e^(I*t)
sage: fprime(t) = I*e^(I*t)
sage: m = Riemann_Map([f], [fprime], 0)  # long time (4 sec)

[fbissey]

I find the whole thing very curious. If you look at the divide patch you'll notice that all the warning generated by Riemann_Map are of the kind Riemann_Map([f], [fprime], 0) - it may be possible for other calls to generate a divide by zero warning but the ending "0" argument in Riemann_Map will automatically give you one. We could check with Eviatar but I am fairly sure that it is a case were you will have one, no matter what. So you both see one that doesn't have a "0", that's possible but that you, Steve, don't see one at line 109 rings alarm bells. Would it be the machine on which you experience ​https://github.com/cschwan/sage-on-gentoo/issues/208 by any chance?

[strogdon]

Replying to fbissey:

Steve, don't see one at line 109 rings alarm bells. Would it be the machine on which you experience ​https://github.com/cschwan/sage-on-gentoo/issues/208 by any chance?

No, this is another machine. Vanilla Sage was built in prefix on this machine. And even on the machine associate with ​https://github.com/cschwan/sage-on-gentoo/issues/208, vanilla Sage does not exhibit the odd behavior of that issue. I get with the prefix-built Sage

$ ./sage -t devel/sage/sage/calculus/riemann.pyx
Running doctests with ID 2013-04-03-16-48-12-0f9b93ae.
Doctesting 1 file.
sage -t devel/sage/sage/calculus/riemann.pyx
**********************************************************************
File "devel/sage/sage/calculus/riemann.pyx", line 119, in sage.calculus.riemann.Riemann_Map
Failed example:
    m = Riemann_Map([f, hf], [fprime, hfprime], 0.5 + 0.5*I)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "devel/sage/sage/calculus/riemann.pyx", line 163, in sage.calculus.riemann.Riemann_Map.__init__
Failed example:
    m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************
File "devel/sage/sage/calculus/riemann.pyx", line 228, in sage.calculus.riemann.Riemann_Map._generate_theta_array
Failed example:
    m = Riemann_Map([f], [fprime], 0)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************

There is no failure associated with the test starting at line 109. However on a fresh start of Sage I get for the test starting at line 109:

sage: f(t) = e^(I*t)                                          
sage: fprime(t) = I*e^(I*t)                                   
sage: m = Riemann_Map([f], [fprime], 0)  # long time (4 sec)  
/storage/sage/sage-5.9.beta1/local/bin/sage-ipython:1: RuntimeWarning: divide by zero encountered in divide
  #!/usr/bin/env python

but if I run the commands again without restarting Sage I get no RuntimeWarning. Is this to be expected? Is something being cached somewhere? And oddly with your patch ​trac_11334-divide.patch I only get

$ ./sage -t devel/sage/sage/calculus/riemann.pyx                                                                                                                                                                                                                                Running doctests with ID 2013-04-03-17-14-01-9d1e8739.
Doctesting 1 file.
sage -t devel/sage/sage/calculus/riemann.pyx
**********************************************************************
File "devel/sage/sage/calculus/riemann.pyx", line 120, in sage.calculus.riemann.Riemann_Map
Failed example:
    m = Riemann_Map([f, hf], [fprime, hfprime], 0.5 + 0.5*I)
Expected nothing
Got:
    doctest:1: RuntimeWarning: divide by zero encountered in divide
**********************************************************************

[fbissey]

OK, I think not getting the warning again is expected but I am not sure about the details of why. After my patch the test from line 119 becomes line 120 so that bit is consistent.

What happens if you run the example of line 119/120 in sage? With or without the patch?

[strogdon]

Replying to fbissey:

OK, I think not getting the warning again is expected but I am not sure about the details of why. After my patch the test from line 119 becomes line 120 so that bit is consistent.

What happens if you run the example of line 119/120 in sage? With or without the patch?

It fails, at least on the first try it does

sage: f(t) = e^(I*t)
sage: fprime(t) = I*e^(I*t)
sage: hf(t) = 0.5*e^(-I*t)
sage: hfprime(t) = 0.5*-I*e^(-I*t)
sage: m = Riemann_Map([f, hf], [fprime, hfprime], 0.5 + 0.5*I)
/storage/sage/sage-5.9.beta1/local/bin/sage-ipython:1: RuntimeWarning: divide by zero encountered in divide
  #!/usr/bin/env python

[fbissey]

OK I am building 5.9.beta2 from scratch on power7 and will try that here as well to see for myself. But 5.9.beta2 s-o-g doesn't suffer from this here.

[fbissey]

Oh wait in sage-on-gentoo I get it when I run the example manually but not when I run the doctest, this is getting weirder.

[fbissey]

OK,

----------------------------------------------------------------------
| Sage Version 5.9.beta2, Release Date: 2013-03-28                   |
| Type "notebook()" for the browser-based notebook interface.        |
| Type "help()" for help.                                            |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
sage: f(t) = e^(I*t)
sage: fprime(t) = I*e^(I*t)
sage: hf(t) = 0.5*e^(-I*t)
sage: hfprime(t) = 0.5*-I*e^(-I*t)
sage: m = Riemann_Map([f, hf], [fprime, hfprime], 0.5 + 0.5*I)
/usr/bin/sage-ipython:1: RuntimeWarning: divide by zero encountered in divide
  #!/usr/bin/env python

but

----------------------------------------------------------------------
| Sage Version 5.9.beta2, Release Date: 2013-03-28                   |
| Type "notebook()" for the browser-based notebook interface.        |
| Type "help()" for help.                                            |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
sage: f(t) = e^(I*t)
sage: fprime(t) = I*e^(I*t)
sage: m = Riemann_Map([f], [fprime], 0)
/usr/bin/sage-ipython:1: RuntimeWarning: divide by zero encountered in divide
  #!/usr/bin/env python
sage:  hf(t) = 0.5*e^(-I*t)
sage: hfprime(t) = 0.5*-I*e^(-I*t)
sage: m = Riemann_Map([f, hf], [fprime, hfprime], 0.5 + 0.5*I)

so in some conditions numpy warnings are not reset between these two adjacent tests and some other it is. I was going to ask if they could have been run out of order but I think the ticket for that is not happening anytime soon. But that would have explained why you see a failure in 119 but not 109.

[fbissey]

I get it in my build of 5.9.beta2 on power7, I cannot understand how the thing happens.

[strogdon]

I've now tested 5.9.beta3 on two different machines and I get the following numpy-related warnings which I also saw with 5.9.beta2.

./sage -t --long devel/sage/sage/matrix/matrix2.pyx devel/sage/sage/symbolic/expression.pyx
Running doctests with ID 2013-04-04-12-42-24-9b2fa346.
Doctesting 2 files.
sage -t --long devel/sage/sage/matrix/matrix2.pyx
**********************************************************************
File "devel/sage/sage/matrix/matrix2.pyx", line 11476, in sage.matrix.matrix2.Matrix.diagonal.plot
Failed example:
    A.plot()
Expected nothing
Got:
    doctest:146: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
    doctest:146: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility
    doctest:3: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
    doctest:3: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility
    doctest:148: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
    doctest:148: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility
    doctest:150: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
    doctest:150: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility
    <BLANKLINE>
**********************************************************************
1 item had failures:
   1 of   7 in sage.matrix.matrix2.Matrix.diagonal.plot
    [2000 tests, 1 failure, 38.68 s]
sage -t --long devel/sage/sage/symbolic/expression.pyx
**********************************************************************
File "devel/sage/sage/symbolic/expression.pyx", line 8747, in sage.symbolic.expression.Expression.find_root
Failed example:
    f.find_root(-2, 3)
Expected:
    1.0
Got:
    doctest:146: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
    doctest:146: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility
    doctest:3: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
    doctest:3: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility
    doctest:148: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
    doctest:148: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility
    doctest:150: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
    doctest:150: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility
    1.0
**********************************************************************
1 item had failures:
   1 of  29 in sage.symbolic.expression.Expression.find_root
    [2041 tests, 1 failure, 36.88 s]
----------------------------------------------------------------------
sage -t --long devel/sage/sage/matrix/matrix2.pyx  # 1 doctest failed
sage -t --long devel/sage/sage/symbolic/expression.pyx  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 77.1 seconds
    cpu time: 72.3 seconds
    cumulative wall time: 75.6 seconds

What is really odd is that the seemingly offending lines when run from the Sage prompt give _no_ warnings. And I don't see where in the .pyx files RuntimeWarnings are explicitly turned on or off. It is as if warnings are only being turned on during doctesting or perhaps something else is up.

[fbissey]

Did install sage and then upgrade numpy? I think it means that some components depending on numpy haven't been rebuilt. I have done my builds by replacing the numpy spkg in spkg/standard and then building from scratch to avoid that kind of problems.

[strogdon]

Yes. I see. That would explain it. Normally I do an install from scratch but didn't here. I'll try again.

[fbissey]

suppress divide warnings in riemann.pyx

[strogdon]

OK, so the above RuntimeWarnings are bogus and don't appear when things are built correctly (from scratch). Curiously,

sage -t --long devel/sage/sage/calculus/riemann.pyx

does not fail when testing using 'make ptestlong', but does yield the 'divide by zero' warning when run individually.

[fbissey]

Yes very curious things like that happens. I am preparing a new set of patch with a different strategy to deal with riemann.pyx. I already uploaded one but I need to make a new patch for other divide warnings not coming from riemann.pyx - people willingly wanting to see 1/0 results.

[fbissey]

fix non riemann.pyx divivde by zero warnings

[fbissey]

OK new patch set. If that doesn't work for you we have a serious problem. Back to need review.

[strogdon]

It's odd that one has to resort to such "trickery" to ignore warnings. However, it is not without warrant since the ignore mechanism was already operative at a certain level in riemann.pyx. This works for me here - all "divide by zero" warnings have been removed from output and everything survives make ptestlong. Positive review.

[fbissey]

I suspect the strange behavior we have seen may be a symptom of something deeper on the way the doctesting framework deals with warnings. Anyway it should be sorted by that last couple of patch for this case.

[jdemeyer]

11334_noise2.patch​ (numerical noise on OS X 10.8) needs review.

[fbissey]

When I first saw that I was wondering if it was a combination with python 2.7.4 but I haven't conducted any test on OS X 10.8. The patch looks fine to me and is compatible with the rest. I am taking it as a reviewer patch and putting this back to positive review.