Opened 11 years ago

Closed 11 years ago

#5081 closed enhancement (fixed)

[with patch, positive review] Make numpy play nice with Sage types

Reported by: jason Owned by: was
Priority: major Milestone: sage-4.1.1
Component: linear algebra Keywords:
Cc: Merged in: Sage 4.1.1.rc0
Authors: Robert Bradshaw Reviewers: Jason Grout
Report Upstream: Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description

We should make numpy understand Sage float and complex types, at least the RDF and CDF types. See the following thread on the numpy list.

http://thread.gmane.org/gmane.comp.python.numeric.general/25251/focus=25273

Attachments (1)

5081-numpy-types.patch (4.8 KB) - added by robertwb 11 years ago.

Download all attachments as: .zip

Change History (18)

comment:1 Changed 11 years ago by jason

[17:22] <cwitty> One problem is that Python has no particular C-level support for __complex__(); you actually have to use generic code to call that method.

comment:2 Changed 11 years ago by fccoelho

an example of failure:

from scipy import stats
stats.uniform(0,15).ppf([0.5,0.7])

Traceback (most recent call last):

File "<stdin>", line 1, in <module> File "/home/Users/fcoelho/.sage/sage_notebook/worksheets/admin/0/code/19.py", line 7, in <module>

stats.uniform(_sage_const_0 ,_sage_const_15 ).ppf([_sage_const_0p5 ,_sage_const_0p7 ])

File "/usr/local/sage-3.4/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/", line 1, in <module>

File "/usr/local/sage-3.4/local/lib/python2.5/site-packages/scipy/stats/distributions.py", line 112, in ppf

return self.dist.ppf(q,*self.args,self.kwds)

File "/usr/local/sage-3.4/local/lib/python2.5/site-packages/scipy/stats/distributions.py", line 563, in ppf

place(output,cond,self._ppf(*goodargs)*scale + loc)

File "/usr/local/sage-3.4/local/lib/python2.5/site-packages/numpy/lib/function_base.py", line 1357, in place

return _insert(arr, mask, vals)

TypeError?: array cannot be safely cast to required type

comment:3 Changed 11 years ago by robertwb

  • Summary changed from Make numpy play nice with Sage types to [with patch, needs review] Make numpy play nice with Sage types

Solves the integer and real types. Complex types are still an issue which will need to be resolved by fixing NumPy?.

Changed 11 years ago by robertwb

comment:4 Changed 11 years ago by jason

Brilliant! I'm upgrading now so I can test this!

comment:5 Changed 11 years ago by jason

Is there a reason that you didn't use the same trick on ZZ that you used on RealField? (with regards to not needing to specify whether dtype=object or not)?

comment:6 Changed 11 years ago by jason

What does the '=' in the type descriptor mean? I couldn't find that documented here: http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

comment:7 Changed 11 years ago by jason

Ah, I found the answer to my last question here: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html#specifying-and-constructing-data-types

'=' means hardware-default endian behavior.

comment:8 Changed 11 years ago by jason

Things seem to work fine if I put the logic that the patch has in real_mpfr.pyx also in integer.pyx. Is there a reason the logic wasn't included dealing with switching from hardware types to object types?

comment:9 Changed 11 years ago by robertwb

http://bugs.python.org/issue1675423 may resolve the complex case.

As for not using the precision trick for ZZ, I was thinking that a single ring should map to a single dtype. A sticky issue is that once you switch from ints to objects, you have different overflow semantics.

I guess numpy folks are already used to this

>>> import numpy 
>>> numpy.array(2**20)
array(1048576)
>>> numpy.array(2**20)**2
0
>>> numpy.array(2**40)   
array(1099511627776L, dtype=int64)
>>> numpy.array(2**40)**2
0
>>> numpy.array(2**80)
array(1208925819614629174706176L, dtype=object)
>>> numpy.array(2**80)**2
1461501637330902918203684832716283019655932542976L

We should maybe have three cases--long (sys.maxint), int64, and object. Of course, the former would only be for 32-bit machines.

  • Robert

comment:10 Changed 11 years ago by was

  • Summary changed from [with patch, needs review] Make numpy play nice with Sage types to [with patch, positive review] Make numpy play nice with Sage types

This looks good, is well documented, and is a long time coming. I'm giving it a positive review.

Regarding:

We should maybe have three cases--long (sys.maxint), int64, and object. Of course, the former would only be for 32-bit machines.

Yes, that would be good. But I don't consider it a show stopper for this patch. It could easily be added on another ticket later.

comment:11 Changed 11 years ago by mvngu

  • Summary changed from [with patch, positive review] Make numpy play nice with Sage types to [with patch, needs work] Make numpy play nice with Sage types

I'm getting the following test failures:

sage -t -long devel/sage-exp/sage/matrix/matrix1.pyx
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/sage/matrix/matrix1.pyx", line 428:
    sage: a.numpy()
Expected:
    array([[0, 1, 2, 3],
           [4, 5, 6, 7],
           [8, 9, 10, 11]], dtype=object)
Got:
    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]])
**********************************************************************
1 items had failures:
   1 of   9 in __main__.example_13
***Test Failed*** 1 failures.
For whitespace errors, see the file /scratch/mvngu/release/sage-4.1.1/tmp/.doctest_matrix1.py
	 [3.5 s]

<SNIP>

sage -t -long devel/sage-exp/sage/rings/number_field/totallyreal_rel.py
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/sage/rings/number_field/totallyreal_rel.py", line 156:
    sage: T = sage.rings.number_field.totallyreal_rel.tr_data_rel(F, 2, 2000)
Exception raised:
    Traceback (most recent call last):
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_3[3]>", line 1, in <module>
        T = sage.rings.number_field.totallyreal_rel.tr_data_rel(F, Integer(2), Integer(2000))###line 156:
    sage: T = sage.rings.number_field.totallyreal_rel.tr_data_rel(F, 2, 2000)
      File "/scratch/mvngu/release/sage-4.1.1/local/lib/python/site-packages/sage/rings/number_field/totallyreal_rel.py", line 200, in __init__
        adj = pari(Q).qflll()[self.d]
      File "gen.pyx", line 9174, in sage.libs.pari.gen._pari_trap (sage/libs/pari/gen.c:44241)
    PariError: unexpected character (2)
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/sage/rings/number_field/totallyreal_rel.py", line 569:
    sage: enumerate_totallyreal_fields_rel(F, 2, 2000)
Exception raised:
    Traceback (most recent call last):
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_5[4]>", line 1, in <module>
        enumerate_totallyreal_fields_rel(F, Integer(2), Integer(2000))###line 569:
    sage: enumerate_totallyreal_fields_rel(F, 2, 2000)
      File "/scratch/mvngu/release/sage-4.1.1/local/lib/python/site-packages/sage/rings/number_field/totallyreal_rel.py", line 647, in enumerate_totallyreal_fields_rel
        T = tr_data_rel(F,m,B,a)
      File "/scratch/mvngu/release/sage-4.1.1/local/lib/python/site-packages/sage/rings/number_field/totallyreal_rel.py", line 200, in __init__
        adj = pari(Q).qflll()[self.d]
      File "gen.pyx", line 9174, in sage.libs.pari.gen._pari_trap (sage/libs/pari/gen.c:44241)
    PariError: unexpected character (2)
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/sage/rings/number_field/totallyreal_rel.py", line 578:
    sage: ls = enumerate_totallyreal_fields_rel(F, 2, 10^4)
Exception raised:
    Traceback (most recent call last):
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_5[6]>", line 1, in <module>
        ls = enumerate_totallyreal_fields_rel(F, Integer(2), Integer(10)**Integer(4))###line 578:
    sage: ls = enumerate_totallyreal_fields_rel(F, 2, 10^4)
      File "/scratch/mvngu/release/sage-4.1.1/local/lib/python/site-packages/sage/rings/number_field/totallyreal_rel.py", line 647, in enumerate_totallyreal_fields_rel
        T = tr_data_rel(F,m,B,a)
      File "/scratch/mvngu/release/sage-4.1.1/local/lib/python/site-packages/sage/rings/number_field/totallyreal_rel.py", line 200, in __init__
        adj = pari(Q).qflll()[self.d]
      File "gen.pyx", line 9174, in sage.libs.pari.gen._pari_trap (sage/libs/pari/gen.c:44241)
    PariError: unexpected character (2)
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/sage/rings/number_field/totallyreal_rel.py", line 579:
    sage: print "ignore this";  ls # random (the second factor is platform-dependent)
Exception raised:
    Traceback (most recent call last):
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_5[7]>", line 1, in <module>
        print "ignore this";  ls # random (the second factor is platform-dependent)###line 579:
    sage: print "ignore this";  ls # random (the second factor is platform-dependent)
    NameError: name 'ls' is not defined
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/sage/rings/number_field/totallyreal_rel.py", line 601:
    sage: [ f[0] for f in ls ]
Exception raised:
    Traceback (most recent call last):
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_5[8]>", line 1, in <module>
        [ f[Integer(0)] for f in ls ]###line 601:
    sage: [ f[0] for f in ls ]
    NameError: name 'ls' is not defined
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/sage/rings/number_field/totallyreal_rel.py", line 604:
    sage: [NumberField(ZZx(x[1]), 't').is_galois() for x in ls]
Exception raised:
    Traceback (most recent call last):
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/scratch/mvngu/release/sage-4.1.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_5[9]>", line 1, in <module>
        [NumberField(ZZx(x[Integer(1)]), 't').is_galois() for x in ls]###line 604:
    sage: [NumberField(ZZx(x[1]), 't').is_galois() for x in ls]
    NameError: name 'ls' is not defined
**********************************************************************
2 items had failures:
   1 of   4 in __main__.example_3
   5 of  12 in __main__.example_5
***Test Failed*** 6 failures.
For whitespace errors, see the file /scratch/mvngu/release/sage-4.1.1/tmp/.doctest_totallyreal_rel.py
	 [2.2 s]

<SNIP>

sage -t -long devel/sage-exp/doc/en/numerical_sage/numpy.rst
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/doc/en/numerical_sage/numpy.rst", line 19:
    sage: l
Expected:
    array([1, 2, 3], dtype=object)
Got:
    array([1, 2, 3])
**********************************************************************
File "/scratch/mvngu/release/sage-4.1.1/devel/sage-exp/doc/en/numerical_sage/numpy.rst", line 54:
    sage: l
Expected:
    array([1.00000000000000, 2.00000000000000, 3.00000000000000], dtype=object)
Got:
    array([ 1.,  2.,  3.])
**********************************************************************
1 items had failures:
   2 of  52 in __main__.example_0
***Test Failed*** 2 failures.
For whitespace errors, see the file /scratch/mvngu/release/sage-4.1.1/tmp/.doctest_numpy.py
	 [1.8 s]

<SNIP>

----------------------------------------------------------------------

The following tests failed:

	sage -t -long devel/sage-exp/sage/matrix/matrix1.pyx # 1 doctests failed
	sage -t -long devel/sage-exp/sage/rings/number_field/totallyreal_rel.py # 6 doctests failed
	sage -t -long devel/sage-exp/doc/en/numerical_sage/numpy.rst # 2 doctests failed
----------------------------------------------------------------------
Total time for all tests: 525.6 seconds

comment:12 Changed 11 years ago by mvngu

This ticket is related to #6497.

comment:13 Changed 11 years ago by jason

See #6506 for a patch which fixes most of these issues and discusses another.

comment:14 Changed 11 years ago by robertwb

  • Summary changed from [with patch, needs work] Make numpy play nice with Sage types to [with patch, needs review] Make numpy play nice with Sage types

All issues fixed in #6506, review of this patch should happen with review of the other.

comment:15 follow-up: Changed 11 years ago by jason

  • Authors set to Robert Bradshaw
  • Reviewers set to Jason Grout
  • Summary changed from [with patch, needs review] Make numpy play nice with Sage types to [with patch, positive review] Make numpy play nice with Sage types

Positive review, since #6506 is positive review and fixes all issues listed here.

mvngu: are you a reviewer as well?

comment:16 in reply to: ↑ 15 Changed 11 years ago by mvngu

Replying to jason:

mvngu: are you a reviewer as well?

No, not really. I often like to apply patches and doctest them to see them break. Seeing doctests break is more fun than seeing all tests passed :-)

comment:17 Changed 11 years ago by mvngu

  • Merged in set to Sage 4.1.1.rc0
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.