Coercion for numpy types

[vdelecroix]

Plenty of bugs have been reported for the interaction with numpy:

• #8426: polynomial * constant does not work if constant is a numpy type
• #8949: symbolic functions dont work with numpy.int32
• #9769: symbolic function do not work with numpy.int64 arguments
• #13386: comparison of Sage integer with Numpy integer
• #15695: Coercion problems between numpy and sage floats
• #17758: Intervals and numpy floats do not compare correctly
• #17865: get rid of _native_coercion_ranks_inv and _native_coercion_ranks

We solve them all by defining coercions of numpy integers to ZZ, numpy floating to RDF and numpy complex floating to CDF. Also, coercion between numerical types (in sage.structure.coerce) are now done directly via the addition. In particular, all of this used to fail

sage: import numpy

sage: f(t) = t^2
sage: f(numpy.int32('1'))
1

sage: sin(numpy.int32(10))
sin(10)

sage: 123 == numpy.int32(123)
True

sage: 1j + numpy.float(2)
2.00000000000000 + 1.00000000000000*I
sage: parent(_)
Complex Field with 53 bits of precision

sage: RIF(1) <= numpy.float64(2.0)
True 

Some of them depend on modifying some internal in numpy and will not be solved here:

• #8824: Make it so that numpy datatypes are integrated into the coercion model

[vdelecroix]

New commits:

​f739d53

Trac 18076: be nicer with numpy

[vdelecroix]

Not so bad... got three problematic files

sage -t --long src/sage/plot/plot.py  # 2 doctests failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/doc/en/thematic_tutorials/numerical_sage/numpy.rst  # 1 doctest failed

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1a76757

Trac 18076: fix coercions of numpy arrays

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​b0c8375

Trac 18076: fix coercions of numpy arrays

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​1d2148e

Trac 18076: fix coercions of numpy arrays

[chapoton]

3 failing doctests, see patchbot report

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​f187d02	Trac 18076: be nicer with numpy
​fd5601e	Trac 18076: fix coercions of numpy arrays
​dbe635f	Trac 18076: fix some doctests

[vdelecroix]

Replying to chapoton:

3 failing doctests, see patchbot report

Is that all?

[chapoton]

still one failing doctest, see patchbot report.

Désolé de ne pas faire de commentaires plus constructifs, vraiment.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6b51a65

Trac 18076: fix some doctests

[vdelecroix]

Replying to chapoton:

still one failing doctest, see patchbot report.

Désolé de ne pas faire de commentaires plus constructifs, vraiment.

Done!

[slabbe]

There is a incomplete trac number XYZ:

+ This used to fail (see :trac:`XYZ`)::

[jdemeyer]

My intention is to make this depend on #18121.

[jdemeyer]

This looks wrong to me:

sage: numpy.int8('12') + 1/3
12.333333333333334

Why is the answer a float???

---

New commits:

​5532969	Use PyTypeObject from Cython
​fd2531d	Merge commit '5532969cc7c462d24788585030ab662cf05b024b' into t/18076/18076
​c3e4671	Introduce new function "is_numpy_type"

[vdelecroix]

Replying to jdemeyer:

This looks wrong to me:

sage: numpy.int8('12') + 1/3
12.333333333333334

Why is the answer a float???

This is because the addition is handled by numpy and not by the rational. Simlarly

sage: import numpy
sage: numpy.int64('1') + 1
2
sage: parent(_)
<type 'numpy.int64'>

There is a comment somewhere that this can be an upstream request.

[vdelecroix]

see also #8824

[jdemeyer]

Replying to vdelecroix:

Replying to jdemeyer:

This looks wrong to me:

sage: numpy.int8('12') + 1/3
12.333333333333334

Why is the answer a float???

This is because the addition is handled by numpy and not by the rational.

I see. It doesn't use the Sage coercion framework at all.

Could you clarify this in the docstring when you write this example?

[jdemeyer]

Could you please add doctests for all the tickets you're marking as duplicate of this one? For example, there is no doctest for #8949.

[vdelecroix]

New commits:

​9c7a6f5

Trac 18076: precision about numpy and Sage coercion

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8af0216

Trac 18076: document all corrected bugs

[jdemeyer]

Feel free to set this ticket back to needs_review (although somebody needs to review #18121 before this ticket can be formally reviewed).

[jdemeyer]

Various conflicts, rebasing...

[jdemeyer]

New commits:

​a10dcba	Use PyTypeObject from Cython
​77e38d2	Define 3-argument version Py_VISIT3 of Py_VISIT
​58c17fb	cimport PyWeakref_GetObject() from cpython.weakref
​d4c53ac	Coercion for numpy types

[vdelecroix]

This is fine for me. And all test pass.

Vincent

[jdemeyer]

Not so fast, I haven't actually reviewed all your changes.

[jdemeyer]

My comments (all easy to fix):

1. Conversion numpy.floating -> QQ should use RR directly, instead of numpy.floating -> RDF which then converts to RR.

2. The reference to #15965 is wrong.

3. In src/sage/rings/real_double.pyx, I would prefer to split up if S is ZZ or S is QQ or S is RLF or (isinstance(S, RealField_class) and S.prec() >= 53) in several cases, just like for CDF (in particular, if the precision is too small, we can return None immediately).

4. Python considers bool a subclass of int, so

   issubclass(py_type, int) or issubclass(py_type, long) or issubclass(py_type, bool)
   

can be replaced by

issubclass(py_type, int) or issubclass(py_type, long)

5. In doctests, please replace 1jr by the more natural I (unless you have a good reason to not do this)

6. In NumpyToSRMorphism.__init__, could you raise a TypeError if numpy_type doesn't satisfy any of the issubclass() checks (and add a doctest for this case). Also, the second if should better be an elif.

[jdemeyer]

I'm also wondering if this check is really needed:

if ty != type(y + x):
    raise RuntimeError("x + y and y + x have different types...")

First of all, I cannot find any case where this happens. Second, even if it does happen, would it really be a problem?

[vdelecroix]

Replying to jdemeyer:

I'm also wondering if this check is really needed:

if ty != type(y + x):
    raise RuntimeError("x + y and y + x have different types...")

First of all, I cannot find any case where this happens.

sage: numpy.int16(1) + 1.0
2.0
sage: type(_)
<type 'numpy.float64'>
sage: 1.0 + numpy.int16(1)
2.00000000000000
sage: type(_)
<type 'sage.rings.real_mpfr.RealNumber'>

But the canonical coercion is fine since we do have a coercion from numpy types to real numbers.

Second, even if it does happen, would it really be a problem?

I don't know. You would prefer that we get rid of it?

Vincent

[vdelecroix]

New commits:

​dc77040

Trac 18076: review commit

[jdemeyer]

Replying to vdelecroix:

I don't know. You would prefer that we get rid of it?

Given that I see no reason to have the check, I'd rather remove it.

I think you did your search-and-replace a bit too naive, this is a syntax error: numpy.complex128(-2+.I)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ccaf80c

Trac 18076: more fixes in structure.coerce

[vbraun]

On 32-bit:

sage -t --long src/sage/structure/coerce.pyx
**********************************************************************
File "src/sage/structure/coerce.pyx", line 334, in sage.structure.coerce.CoercionModel_cache_maps
Failed example:
    type(_)
Expected:
    <type 'numpy.int64'>
Got:
    <type 'numpy.int32'>
**********************************************************************
1 item had failures:
   1 of  22 in sage.structure.coerce.CoercionModel_cache_maps
    [272 tests, 1 failure, 0.41 s]

[jdemeyer]

New commits:

​8b0cf39

Fix doctest different on 32 vs. 64 bits

[vdelecroix]

It does not affect floating point?

sage: numpy.int8('12') + 1/3
12.333333333333334
sage: type(_)
<type 'numpy.float64'>

[vdelecroix]

Let's assume that it's right...