Sage: Ticket #11027: Schur matrix decomposition over RDF/CDF
https://trac.sagemath.org/ticket/11027
<p>
Wraps the <code>schur()</code> function from <code>SciPy</code>.
</p>
<p>
<strong>Apply:</strong>
</p>
<ol><li><a class="attachment" href="https://trac.sagemath.org/attachment/ticket/11027/trac_11027-schur-decomposition-v3.patch" title="Attachment 'trac_11027-schur-decomposition-v3.patch' in Ticket #11027">trac_11027-schur-decomposition-v3.patch</a><a class="trac-rawlink" href="https://trac.sagemath.org/raw-attachment/ticket/11027/trac_11027-schur-decomposition-v3.patch" title="Download"></a>
</li><li><a class="attachment" href="https://trac.sagemath.org/attachment/ticket/11027/trac_11027-schur-decomposition-doctest-v2.patch" title="Attachment 'trac_11027-schur-decomposition-doctest-v2.patch' in Ticket #11027">trac_11027-schur-decomposition-doctest-v2.patch</a><a class="trac-rawlink" href="https://trac.sagemath.org/raw-attachment/ticket/11027/trac_11027-schur-decomposition-doctest-v2.patch" title="Download"></a>
</li></ol>en-usSagehttps://trac.sagemath.org/chrome/site/logo_sagemath_trac.png
https://trac.sagemath.org/ticket/11027
Trac 1.1.6rbeezerFri, 25 Mar 2011 15:59:24 GMTattachment set
https://trac.sagemath.org/ticket/11027
https://trac.sagemath.org/ticket/11027
<ul>
<li><strong>attachment</strong>
set to <em>trac_11027-schur-decomposition.patch</em>
</li>
</ul>
TicketmraumSat, 26 Mar 2011 00:57:57 GMTstatus changed; reviewer set
https://trac.sagemath.org/ticket/11027#comment:1
https://trac.sagemath.org/ticket/11027#comment:1
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_work</em>
</li>
<li><strong>reviewer</strong>
set to <em>Martin Raum</em>
</li>
</ul>
<p>
Some discussion is necessary: As a doctest I get
</p>
<pre class="wiki">Expected:
[ 0.006 - 0.001*I 0.0425 + 0.1018*I -0.6446 + 0.2201*I 0.3981 + 0.6045*I]
[ 0.0875 - 0.005*I 0.1984 + 0.5537*I -0.5022 + 0.1944*I -0.3414 - 0.4896*I]
[ 0.3572 - 0.0159*I 0.2444 + 0.685*I 0.4499 - 0.1643*I 0.1983 + 0.2728*I]
[ 0.929 - 0.0377*I -0.111 - 0.317*I -0.1212 + 0.044*I -0.0466 - 0.0627*I]
Got:
[ 0.006 - 0.001*I 0.0283 + 0.1066*I -0.6784 + 0.0611*I 0.3686 + 0.6229*I]
[ 0.0875 - 0.005*I 0.1216 + 0.5755*I -0.534 + 0.0699*I -0.3175 - 0.5055*I]
[ 0.3572 - 0.0159*I 0.1494 + 0.7118*I 0.476 - 0.053*I 0.185 + 0.282*I]
[ 0.929 - 0.0377*I -0.067 - 0.3291*I -0.1282 + 0.0141*I -0.0435 - 0.0649*I]
**********************************************************************
File "/scratch/mraum/sagedev/devel/sage-dev/sage/matrix/matrix_double_dense.pyx", line 1382:
sage: T.round(4)
Expected:
[ 30.7332 + 4648.5418*I -2342.252 + 767.2915*I -477.0197 - 1460.6287*I -532.7363 + 318.6099*I]
[ 0 -0.1841 - 159.0576*I 151.3315 + 0.458*I -15.2369 - 67.0242*I]
[ 0 0 -0.5239 + 11.158*I 6.2073 - 1.1208*I]
[ 0 0 0 -0.0252 - 0.6422*I]
Got:
[ 30.7332 + 4648.5418*I -2424.5875 + 443.008*I -117.3203 - 1532.0639*I -547.4078 + 292.6858*I]
[ 0 -0.1841 - 159.0576*I 150.4761 + 16.0737*I -21.0591 - 65.4288*I]
[ 0 0 -0.5239 + 11.158*I 5.8811 - 2.2803*I]
[ 0 0 0 -0.0252 - 0.6422*I]
</pre><p>
Mathematica gives:
</p>
<pre class="wiki">In[1]:= m = m = Table[Table[0.0 + i + 4 * j, {i, 0, 3}], {j, 0, 3}] + Table[Table[(i + 4 * j)^3 * I, {i, 0, 3}], {j, 0, 3}]
Out[1]= {{0., 1. + 1. I, 2. + 8. I, 3. + 27. I},
> {4. + 64. I, 5. + 125. I, 6. + 216. I, 7. + 343. I},
> {8. + 512. I, 9. + 729. I, 10. + 1000. I, 11. + 1331. I},
> {12. + 1728. I, 13. + 2197. I, 14. + 2744. I, 15. + 3375. I}}
In[2]:= {matU, matT} = SchurDecomposition[m]
Out[2]= {{{0.0060036 - 0.000979619 I, -0.0373719 + 0.10381 I,
> -0.0166147 + 0.680909 I, 0.370437 + 0.621815 I},
> {0.0875304 - 0.00504818 I, -0.227102 + 0.542591 I,
> 0.0085438 + 0.538502 I, -0.318925 - 0.504541 I},
> {0.357191 - 0.0158906 I, -0.281727 + 0.670563 I,
> 0.0016419 - 0.478947 I, 0.185814 + 0.281457 I},
> {0.928988 - 0.0376762 I, 0.131963 - 0.308882 I,
> -0.000646133 + 0.128983 I, -0.0436872 - 0.064763 I}},
> {{30.7332 + 4648.54 I, -2246.55 - 1013.86 I, -1535.45 - 58.1104 I,
> -546.563 + 294.26 I}, {0. + 0. I, -0.184122 - 159.058 I,
> -56.6912 - 140.312 I, -54.6402 - 41.6996 I},
> {0. + 0. I, 0. + 0. I, -0.523917 + 11.158 I, 2.95198 + 5.57432 I},
> {0. + 0. I, 0. + 0. I, 0. + 0. I, -0.0251565 - 0.642223 I}}}
</pre><p>
That is three different results, probably depending on the machine you run it on.
</p>
TicketrbeezerSun, 27 Mar 2011 05:54:55 GMT
https://trac.sagemath.org/ticket/11027#comment:2
https://trac.sagemath.org/ticket/11027#comment:2
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:1" title="Comment 1">mraum</a>:
</p>
<blockquote class="citation">
<p>
Some discussion is necessary
</p>
</blockquote>
<p>
Definitely! That is disturbing. I don't know if <code>SciPy</code> has a random element, or it is that platform dependent.
</p>
<p>
This decomposition is not unique (mathematically). I cannot determine if <code>SciPy</code> is suppose to return the same result across platforms. I'll dig deeper.
</p>
<p>
In any event, your three examples all seem correct, in that the diagonals of T all have the eigenvalues of the matrix. Run in Sage, the doctests should have checked all the key properties (unitary, upper-triangular, decomposition).
</p>
<p>
Did you check Mathematica's result for unitary and decomposition?
</p>
<p>
Rob
</p>
TicketrbeezerWed, 30 Mar 2011 20:41:12 GMT
https://trac.sagemath.org/ticket/11027#comment:3
https://trac.sagemath.org/ticket/11027#comment:3
<div class="wiki-code"><div class="code"><pre>A <span class="o">=</span> matrix<span class="p">(</span>RDF<span class="p">,</span> <span class="p">[[</span><span class="o">-</span><span class="mf">7.</span><span class="p">,</span> <span class="mf">5.</span><span class="p">,</span> <span class="mf">11.</span><span class="p">,</span> <span class="o">-</span><span class="mf">4.</span><span class="p">,</span> <span class="mf">13.</span><span class="p">],</span>
<span class="p">[</span><span class="o">-</span><span class="mf">11.</span><span class="p">,</span> <span class="o">-</span><span class="mf">3.</span><span class="p">,</span> <span class="mf">11.</span><span class="p">,</span> <span class="mf">8.</span><span class="p">,</span> <span class="o">-</span><span class="mf">19.</span><span class="p">],</span>
<span class="p">[</span><span class="o">-</span><span class="mf">6.</span><span class="p">,</span> <span class="mf">3.</span><span class="p">,</span> <span class="o">-</span><span class="mf">5.</span><span class="p">,</span> <span class="mf">0.</span><span class="p">,</span> <span class="o">-</span><span class="mf">12.</span><span class="p">],</span>
<span class="p">[</span><span class="o">-</span><span class="mf">4.</span><span class="p">,</span> <span class="o">-</span><span class="mf">12.</span><span class="p">,</span> <span class="o">-</span><span class="mf">14.</span><span class="p">,</span> <span class="mf">8.</span><span class="p">,</span> <span class="o">-</span><span class="mf">8.</span><span class="p">],</span>
<span class="p">[</span><span class="mf">11.</span><span class="p">,</span> <span class="mf">0.</span><span class="p">,</span> <span class="mf">9.</span><span class="p">,</span> <span class="mf">6.</span><span class="p">,</span> <span class="mf">10.</span><span class="p">]])</span>
Q<span class="p">,</span> T <span class="o">=</span> A<span class="o">.</span>schur<span class="p">(</span>base_ring<span class="o">=</span>CDF<span class="p">)</span>
T<span class="o">.</span>round<span class="p">(</span><span class="mi">4</span><span class="p">)</span>
</pre></div></div><p>
Unknown hardware, etc<br />
<a class="ext-link" href="http://abel.ee.ucla.edu/cvxopt/userguide/lapack.html"><span class="icon"></span>http://abel.ee.ucla.edu/cvxopt/userguide/lapack.html</a>
</p>
<pre class="wiki">[ 5.67e+00+j1.69e+01 -2.13e+01+j2.85e+00 1.40e+00+j5.88e+00 -4.19e+00+j2.05e-01 3.19e+00-j1.01e+01]
[ 0.00e+00-j0.00e+00 5.67e+00-j1.69e+01 1.09e+01+j5.93e-01 -3.29e+00-j1.26e+00 -1.26e+01+j7.80e+00]
[ 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 1.27e+01+j3.43e-17 -6.83e+00+j2.18e+00 5.31e+00-j1.69e+00]
[ 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 -1.31e+01-j0.00e+00 -2.60e-01-j0.00e+00]
[ 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 -7.86e+00-j0.00e+00]
</pre><p>
sage-4.7.alpha2, sage.math
</p>
<pre class="wiki">[ 5.6668 - 16.9373*I -8.6458 + 7.8734*I -7.1561 + 14.5703*I 5.7811 + 2.688*I -8.5999 - 5.1071*I]
[ 0 5.6668 + 16.9373*I -10.0921 + 9.9632*I -1.5618 - 5.8776*I 7.4924 + 8.7084*I]
[ 0 0 12.6587 1.6512 + 1.049*I 11.1778 + 4.4387*I]
[ 0 0 0 -13.136 -0.1856 + 0.0353*I]
[ 0 0 0 0 -7.8563]
</pre><p>
sage4.7.alpha2, Intel i7-2600
</p>
<pre class="wiki">[ 5.6668 - 16.9373*I -8.6458 + 7.8734*I -7.1561 + 14.5703*I 5.7811 + 2.688*I -8.5999 - 5.1071*I]
[ 0 5.6668 + 16.9373*I -10.0921 + 9.9632*I -1.5618 - 5.8776*I 7.4924 + 8.7084*I]
[ 0 0 12.6587 1.6512 + 1.049*I 11.1778 + 4.4387*I]
[ 0 0 0 -13.136 -0.1856 + 0.0353*I]
[ 0 0 0 0 -7.8563]
</pre><p>
Mathematica 6.0, sage.math
</p>
<pre class="wiki">{
{5.66679 - 16.9373 I, -10.6919 + 4.73528 I, -7.15609 + 14.5703 I, 5.84829 + 2.53856 I, -8.59995 - 5.10712 I},
{0. + 0. I, 5.66679 + 16.9373 I, -6.42363 + 12.6433 I, -3.46984 - 4.99463 I, 9.86177 + 5.89218 I},
{0. + 0. I, 0. + 0. I, 12.6587 + 1.16017 10^-15 I, 1.67762 + 1.0062 I, 11.1778 + 4.4386 I},
{0. + 0. I, 0. + 0. I, 0. + 0. I, -13.136 + 5.90551 10^-16 I, -0.186487 + 0.0305276 I},
{0. + 0. I, 0. + 0. I, 0. + 0. I, 0. + 0. I, -7.85626 + 1.08747 10^-17 I}
}
</pre><p>
Results test out as correct (did not have a unitary matrix for the first example), but obvious variability. The failing doctest has a large condition number (~1000), but so does another test which is not failing. Notice that super-diagonal starts to agree at the right side.
</p>
<p>
It would be easy to fix doctests, by focusing on what the matrices should <em>do</em>, not on what they <em>are</em>. The eigenvalues could be stripped from the diagonal of the upper-triangular matrix to test as a list, since they should not vary.
</p>
<p>
This function seems too useful elsewhere (testing Hermitian, normal, etc) to abandon it. Thoughts on moving forward? We could
</p>
<ul><li> Abandon failing test.
</li></ul><ul><li> Fix tests to concentrate on properties and not on uniqueness.
</li></ul><ul><li> Solicit testing on sage-devel to see how variable this is across systems and processors.
</li></ul>
TicketrbeezerWed, 30 Mar 2011 20:50:25 GMTcc set
https://trac.sagemath.org/ticket/11027#comment:4
https://trac.sagemath.org/ticket/11027#comment:4
<ul>
<li><strong>cc</strong>
<em>jason</em> added
</li>
</ul>
TicketmraumWed, 30 Mar 2011 21:22:04 GMT
https://trac.sagemath.org/ticket/11027#comment:5
https://trac.sagemath.org/ticket/11027#comment:5
<p>
I wouldn't support abandoning the test. But of cause it tests something, that is not guaranteed by the (mathematical) definitions.
My suggestion is:
1) Emphasize in the documentation that the decomposition cannot be assumed to be unique. In particular, it can even vary on different machines.
2) Actually the crucial: U is unitary, A is upper tridiagonal, the eigenvalues are in "right" order.
3) One can also test the Frobenius norm. That's not very deep, but it is a further invariant for the nilpotent part of A.
</p>
<p>
Your thoughts?
</p>
TicketjasonWed, 30 Mar 2011 21:29:59 GMT
https://trac.sagemath.org/ticket/11027#comment:6
https://trac.sagemath.org/ticket/11027#comment:6
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:3" title="Comment 3">rbeezer</a>:
</p>
<blockquote class="citation">
<ul><li> Fix tests to concentrate on properties and not on uniqueness.
</li></ul></blockquote>
<p>
+1
</p>
<blockquote class="citation">
<ul><li> Solicit testing on sage-devel to see how variable this is across systems and processors.
</li></ul></blockquote>
<p>
Also, you might write to the scipy list and ask how much variability we should expect on different platforms.
</p>
TicketrbeezerWed, 30 Mar 2011 22:06:41 GMT
https://trac.sagemath.org/ticket/11027#comment:7
https://trac.sagemath.org/ticket/11027#comment:7
<p>
Hi Martin and Jason,
</p>
<p>
Thanks - those are all great suggestions. I'll probably get back to this tomorrow.
</p>
<p>
Rob
</p>
TicketrbeezerThu, 31 Mar 2011 19:01:01 GMTattachment set
https://trac.sagemath.org/ticket/11027
https://trac.sagemath.org/ticket/11027
<ul>
<li><strong>attachment</strong>
set to <em>trac_11027-schur-decomposition-v2.patch</em>
</li>
</ul>
TicketrbeezerThu, 31 Mar 2011 19:03:58 GMTstatus, description changed
https://trac.sagemath.org/ticket/11027#comment:8
https://trac.sagemath.org/ticket/11027#comment:8
<ul>
<li><strong>status</strong>
changed from <em>needs_work</em> to <em>needs_review</em>
</li>
<li><strong>description</strong>
modified (<a href="/ticket/11027?action=diff&version=8">diff</a>)
</li>
</ul>
<p>
v2 patch adjusts the tests to not rely on exact values in the unitary matrix, nor above the diagonal in the upper-triangular matrix. It adds a few more checks on the diagonal entries, which are the eigenvalues of the matrix. There are also a couple of new checks on the 2-norm of the similar matrices, one for the real case and one for the complex case.
</p>
<p>
This passes tests on my desktop (Ubuntu, i7-2600) and on sage.math.
</p>
<p>
Jason - would you mind testing this on a Mac before we turn it loose?
</p>
<p>
Rob
</p>
TicketmraumFri, 01 Apr 2011 01:08:27 GMTstatus changed; author set
https://trac.sagemath.org/ticket/11027#comment:9
https://trac.sagemath.org/ticket/11027#comment:9
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
<li><strong>author</strong>
set to <em>Rob Beezer</em>
</li>
</ul>
<p>
That's very good. And now all tests pass.
</p>
<p>
I just noticed, and include this as a comment: Shouldn't we think of Rstyfying the matrix_double_dense file? Currently all your nice documentation doesn't show up in the reference manual. In particular, the schur method, that is somewhat unique to this class, won't be advertised.
</p>
TicketrbeezerFri, 01 Apr 2011 01:22:47 GMT
https://trac.sagemath.org/ticket/11027#comment:10
https://trac.sagemath.org/ticket/11027#comment:10
<p>
Thanks, Martin! Yes, I went through this whole file once, but then started changing and adding whole routines and realized I was walking all over my formatting changes/updates. So I thought it would be best to wait just a little bit.
</p>
<p>
Once changes settle down I <em>will</em> go through and do some documentation (and minor code) clean-up and add this to the docs. It'll go up on <a class="closed ticket" href="https://trac.sagemath.org/ticket/8046" title="defect: Add matrix/matrix_double_dense.py to documentation (closed: fixed)">#8046</a> once it happens. I've been checking my new documentation in the notebook as I go, so it should all format properly.
</p>
<p>
(I may add an exact Schur method, just for instructional use.)
</p>
TicketjdemeyerTue, 05 Apr 2011 11:59:20 GMTstatus changed; resolution, merged set
https://trac.sagemath.org/ticket/11027#comment:11
https://trac.sagemath.org/ticket/11027#comment:11
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
<li><strong>merged</strong>
set to <em>sage-4.7.alpha4</em>
</li>
</ul>
TicketjhpalmieriMon, 11 Apr 2011 18:54:09 GMT
https://trac.sagemath.org/ticket/11027#comment:12
https://trac.sagemath.org/ticket/11027#comment:12
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:8" title="Comment 8">rbeezer</a>:
</p>
<blockquote class="citation">
<p>
Jason - would you mind testing this on a Mac before we turn it loose?
</p>
</blockquote>
<p>
It looks like this didn't happen, and I'm getting doctest failures. On a Mac (OS X 10.6.7, Intel):
</p>
<pre class="wiki">File "/Applications/sage_builds/clean/sage-4.7.alpha4/devel/sage-main/sage/matrix/matrix_double_dense.pyx", line 1538:
sage: T.round(4)
Expected:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 -0.0 -0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
Got:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 0.0 0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
</pre><p>
On t2.math.washington.edu (Solaris on sparc):
</p>
<pre class="wiki">
File "/scratch/palmieri/clean/sage-4.7.alpha4/devel/sage/sage/matrix/matrix_double_dense.pyx", line 1538:
sage: T.round(4)
Expected:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 -0.0 -0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
Got:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 0.0 -0.0]
[ 0.0 0.0 7.7664 -0.0]
[ 0.0 0.0 0.0 11.6542]
</pre>
TicketjdemeyerMon, 11 Apr 2011 19:13:19 GMTstatus changed; resolution, merged deleted
https://trac.sagemath.org/ticket/11027#comment:13
https://trac.sagemath.org/ticket/11027#comment:13
<ul>
<li><strong>status</strong>
changed from <em>closed</em> to <em>new</em>
</li>
<li><strong>resolution</strong>
<em>fixed</em> deleted
</li>
<li><strong>merged</strong>
<em>sage-4.7.alpha4</em> deleted
</li>
</ul>
<p>
Unmerging this in sage-4.7.alpha5...
</p>
TicketjdemeyerMon, 11 Apr 2011 19:13:27 GMTstatus changed
https://trac.sagemath.org/ticket/11027#comment:14
https://trac.sagemath.org/ticket/11027#comment:14
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_work</em>
</li>
</ul>
TicketdrkirkbyMon, 11 Apr 2011 19:29:55 GMT
https://trac.sagemath.org/ticket/11027#comment:15
https://trac.sagemath.org/ticket/11027#comment:15
<p>
On OpenSolaris 06/2009 (quad core Xeon):
</p>
<pre class="wiki">File "/export/home/drkirkby/sage-4.7.alpha4/devel/sage/sage/matrix/matrix_double_dense.pyx", line 1538:
sage: T.round(4)
Expected:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 -0.0 -0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
Got:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 0.0 -0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
**********************************************************************
</pre>
TicketrbeezerMon, 11 Apr 2011 19:49:03 GMT
https://trac.sagemath.org/ticket/11027#comment:16
https://trac.sagemath.org/ticket/11027#comment:16
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:12" title="Comment 12">jhpalmieri</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:8" title="Comment 8">rbeezer</a>:
</p>
<blockquote class="citation">
<p>
Jason - would you mind testing this on a Mac before we turn it loose?
</p>
</blockquote>
<p>
It looks like this didn't happen, and I'm getting doctest failures. On a Mac (OS X 10.6.7, Intel):
</p>
</blockquote>
<p>
Thanks, John and Dave.
</p>
<p>
I'm more concerned about <a class="closed ticket" href="https://trac.sagemath.org/ticket/10837" title="enhancement: Matrix and vector norms, condition number, over RDF/CDF (closed: fixed)">#10837</a>, so if you see those variable results, please let me know about that.
</p>
<p>
John - I have an account on Skynet, so I should do some testing there, I see. Do you have any pointers to etiquette/practices there? Do I build in my home directory? Should I "nice" things or limit the number of cores?
</p>
<p>
Rob
</p>
TicketjdemeyerMon, 11 Apr 2011 20:29:11 GMT
https://trac.sagemath.org/ticket/11027#comment:17
https://trac.sagemath.org/ticket/11027#comment:17
<p>
Some clarification of how this got into sage-4.7.alpha4: when I build a new Sage alpha release, it is tested on the buildbot which tests on many machines, including (most of) the Skynet machines. So this doctest problems did show up. However, in my original sage-4.7.alpha4, I also had <a class="closed ticket" href="https://trac.sagemath.org/ticket/10837" title="enhancement: Matrix and vector norms, condition number, over RDF/CDF (closed: fixed)">#10837</a> and I mistook those doctest failures to come from <a class="closed ticket" href="https://trac.sagemath.org/ticket/10837" title="enhancement: Matrix and vector norms, condition number, over RDF/CDF (closed: fixed)">#10837</a> instead. So I unmerged <a class="closed ticket" href="https://trac.sagemath.org/ticket/10837" title="enhancement: Matrix and vector norms, condition number, over RDF/CDF (closed: fixed)">#10837</a> and went ahead to release sage-4.7.alpha4 without testing again. Experience shows that testing again after unmerging a patch from a candidate alpha is usually not necessary and only slows down things. In this case, it would have caught the errors from <a class="closed ticket" href="https://trac.sagemath.org/ticket/11027" title="enhancement: Schur matrix decomposition over RDF/CDF (closed: fixed)">#11027</a>.
</p>
<p>
Long story short: even if you don't test yourself, a patch will not get in a final Sage release if it fails doctests on Skynet.
</p>
TicketrbeezerTue, 12 Apr 2011 03:47:38 GMT
https://trac.sagemath.org/ticket/11027#comment:18
https://trac.sagemath.org/ticket/11027#comment:18
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:17" title="Comment 17">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
Some clarification of how this got into sage-4.7.alpha4:
</p>
</blockquote>
<p>
Thanks, Jeroen. No problems here.
</p>
<p>
I'd misunderstood and thought <a class="closed ticket" href="https://trac.sagemath.org/ticket/10837" title="enhancement: Matrix and vector norms, condition number, over RDF/CDF (closed: fixed)">#10837</a> was in as well, so this helped clear that up for me.
</p>
<p>
I just need to get into the habit of testing on a wider variety of platforms if I'm going to tackle these numerical procedures. sage.math, of course, but I should add <code>SkyNet</code> to my repertoire also.
</p>
<p>
Sorry for the trouble - I should have realized these -0.0 entries were a bad sign.
</p>
<p>
As always, thanks for all your careful work as release manager.
</p>
<p>
Rob
</p>
TicketrbeezerTue, 12 Apr 2011 04:01:24 GMT
https://trac.sagemath.org/ticket/11027#comment:19
https://trac.sagemath.org/ticket/11027#comment:19
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:16" title="Comment 16">rbeezer</a>:
</p>
<blockquote class="citation">
<p>
John - I have an account on Skynet, so I should do some testing there, I see. Do you have any pointers to etiquette/practices there? Do I build in my home directory? Should I "nice" things or limit the number of cores?
</p>
</blockquote>
<p>
Ignore the request, John. I think I have found my way. Maybe.
</p>
<p>
Rob
</p>
TicketrbeezerTue, 12 Apr 2011 05:11:47 GMTstatus, description changed
https://trac.sagemath.org/ticket/11027#comment:20
https://trac.sagemath.org/ticket/11027#comment:20
<ul>
<li><strong>status</strong>
changed from <em>needs_work</em> to <em>needs_review</em>
</li>
<li><strong>description</strong>
modified (<a href="/ticket/11027?action=diff&version=20">diff</a>)
</li>
</ul>
<p>
Well, it looks like this was just carelessness on my part, and not any kind of platform-variability problem. On other doctests, which seem to pass for the various testers, I have code like
</p>
<pre class="wiki">sage: T = T.zero_at(1.0e-12).change_ring(RDF)
sage: T.round(6)
</pre><p>
The first line was missing on this one problem test - the doctest patch just adds in such a line. Likely a cut/paste when there should have been a copy/paste.
</p>
<p>
With the new patch, this passes tests on my 64-bit Ubuntu machine and on sage.math. I'm in the process of testing on <code>SkyNet/cleo</code>. I'm confident this fix should be OK, but "once burned, twice shy."
</p>
<p>
This is ready for review, but I'd like to have the cleo result before it gets flipped to positive review.
</p>
TicketrbeezerWed, 13 Apr 2011 00:31:15 GMT
https://trac.sagemath.org/ticket/11027#comment:21
https://trac.sagemath.org/ticket/11027#comment:21
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:20" title="Comment 20">rbeezer</a>:
</p>
<blockquote class="citation">
<p>
This is ready for review, but I'd like to have the cleo result before it gets flipped to positive review.
</p>
</blockquote>
<p>
I couldn't get Sage past building PARI on cleo, and this seems to be a problem with a few tickets floating around, and I didn't see a quick fix (ie one I could understand). So I'm going to abandon that and let the buildbots do any further checks. As mentioned, i'm pretty confident about this fixing the problem, across platforms.
</p>
<p>
It's a one-line patch, and this decomposition will be <em>really</em> useful for lots of checks (like Hermitian, unitary, etc), so this is ready to go for a review.
</p>
TicketjhpalmieriWed, 13 Apr 2011 01:16:40 GMTstatus changed
https://trac.sagemath.org/ticket/11027#comment:22
https://trac.sagemath.org/ticket/11027#comment:22
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>needs_work</em>
</li>
</ul>
<p>
Hi Rob,
</p>
<p>
I think it's going to have to be more than a one-line patch, maybe three or four lines: in addition to adding the call to <code>zero_at</code>, the matrix entries in the doctest which are currently <code>-0.0</code> need to be changed to <code>0.0</code>, don't they?
</p>
TicketrbeezerWed, 13 Apr 2011 06:06:43 GMT
https://trac.sagemath.org/ticket/11027#comment:23
https://trac.sagemath.org/ticket/11027#comment:23
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:22" title="Comment 22">jhpalmieri</a>:
</p>
<blockquote class="citation">
<p>
Hi Rob,
</p>
<p>
I think it's going to have to be more than a one-line patch, maybe three or four lines: in addition to adding the call to <code>zero_at</code>, the matrix entries in the doctest which are currently <code>-0.0</code> need to be changed to <code>0.0</code>, don't they?
</p>
</blockquote>
<p>
Aargh! Yes, of course. I think I was doctesting the wrong file since I've had my head into matrix2.pyx heavily the past few days.
</p>
<p>
I'll get it right in the morning. Thanks for the gentle correction.
</p>
<p>
Rob
</p>
TicketrbeezerWed, 13 Apr 2011 17:04:48 GMTattachment set
https://trac.sagemath.org/ticket/11027
https://trac.sagemath.org/ticket/11027
<ul>
<li><strong>attachment</strong>
set to <em>trac_11027-schur-decomposition-doctest.patch</em>
</li>
</ul>
TicketrbeezerWed, 13 Apr 2011 17:06:48 GMTstatus changed
https://trac.sagemath.org/ticket/11027#comment:24
https://trac.sagemath.org/ticket/11027#comment:24
<ul>
<li><strong>status</strong>
changed from <em>needs_work</em> to <em>needs_review</em>
</li>
</ul>
<p>
I've replaced the "doctest" patch with a new one. With two lines now.
</p>
TicketjhpalmieriWed, 13 Apr 2011 18:48:57 GMTstatus, reviewer changed
https://trac.sagemath.org/ticket/11027#comment:25
https://trac.sagemath.org/ticket/11027#comment:25
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
<li><strong>reviewer</strong>
changed from <em>Martin Raum</em> to <em>Martin Raum, John Palmieri</em>
</li>
</ul>
<p>
This now works on Solaris, OS X, sage.math.
</p>
TicketjdemeyerWed, 13 Apr 2011 18:51:56 GMTmilestone changed
https://trac.sagemath.org/ticket/11027#comment:26
https://trac.sagemath.org/ticket/11027#comment:26
<ul>
<li><strong>milestone</strong>
changed from <em>sage-4.7</em> to <em>sage-4.7.1</em>
</li>
</ul>
TicketrbeezerThu, 14 Apr 2011 04:24:47 GMT
https://trac.sagemath.org/ticket/11027#comment:27
https://trac.sagemath.org/ticket/11027#comment:27
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:25" title="Comment 25">jhpalmieri</a>:
</p>
<blockquote class="citation">
<p>
This now works on Solaris, OS X, sage.math.
</p>
</blockquote>
<p>
Hi John,
</p>
<p>
Thanks for sticking with this mess and the extra testing. It'll be a valuable one to have - I had not appreciated how useful this is.
</p>
<p>
I've been looking at making an exact version for instructional purposes as well.
</p>
<p>
Rob
</p>
TicketjdemeyerThu, 14 Apr 2011 07:20:02 GMTstatus changed; resolution, merged set
https://trac.sagemath.org/ticket/11027#comment:28
https://trac.sagemath.org/ticket/11027#comment:28
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
<li><strong>merged</strong>
set to <em>sage-4.7.1.alpha0</em>
</li>
</ul>
TicketrbeezerTue, 19 Apr 2011 00:10:58 GMT
https://trac.sagemath.org/ticket/11027#comment:29
https://trac.sagemath.org/ticket/11027#comment:29
<p>
Attempting to appease the buildbot's prerequisites for <a class="closed ticket" href="https://trac.sagemath.org/ticket/10848" title="enhancement: Checks for Hermitian matrices (closed: fixed)">#10848</a>.
</p>
<p>
Apply trac_11027-schur-decomposition-v2.patch, trac_11027-schur-decomposition-doctest.patch
</p>
TicketjdemeyerTue, 24 May 2011 10:21:09 GMTattachment set
https://trac.sagemath.org/ticket/11027
https://trac.sagemath.org/ticket/11027
<ul>
<li><strong>attachment</strong>
set to <em>trac_11027-schur-decomposition-v3.patch</em>
</li>
</ul>
<p>
Patch rebased to sage-4.7
</p>
TicketjdemeyerTue, 24 May 2011 10:21:27 GMTdescription changed
https://trac.sagemath.org/ticket/11027#comment:30
https://trac.sagemath.org/ticket/11027#comment:30
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/11027?action=diff&version=30">diff</a>)
</li>
</ul>
TicketjdemeyerThu, 26 May 2011 09:06:21 GMTstatus changed; resolution, merged deleted
https://trac.sagemath.org/ticket/11027#comment:31
https://trac.sagemath.org/ticket/11027#comment:31
<ul>
<li><strong>status</strong>
changed from <em>closed</em> to <em>new</em>
</li>
<li><strong>resolution</strong>
<em>fixed</em> deleted
</li>
<li><strong>merged</strong>
<em>sage-4.7.1.alpha0</em> deleted
</li>
</ul>
<p>
OS X 10.4 PPC returns eigenvalues in a different order:
</p>
<pre class="wiki">sage -t -long "devel/sage/sage/matrix/matrix_double_dense.pyx"
**********************************************************************
File "/Users/jdemeyer/sage-4.7.1.alpha0/devel/sage/sage/matrix/matrix_double_dense.pyx", line 1571:
sage: A.eigenvalues()
Expected:
[-0.789... + 2.336...*I, -0.789... - 2.336...*I, -5.710... + 8.382...*I, -5.710... - 8.382...*I]
Got:
[-5.71017354352 + 8.38264062467*I, -5.71017354352 - 8.38264062467*I, -0.789826456477 + 2.33693044103*I, -0.789826456477 - 2.33693044103*I]
**********************************************************************
</pre>
TicketjdemeyerThu, 26 May 2011 09:06:57 GMTstatus changed
https://trac.sagemath.org/ticket/11027#comment:32
https://trac.sagemath.org/ticket/11027#comment:32
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_review</em>
</li>
</ul>
TicketjdemeyerThu, 26 May 2011 09:07:08 GMTstatus changed
https://trac.sagemath.org/ticket/11027#comment:33
https://trac.sagemath.org/ticket/11027#comment:33
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>needs_work</em>
</li>
</ul>
TicketjdemeyerWed, 08 Jun 2011 12:12:41 GMT
https://trac.sagemath.org/ticket/11027#comment:34
https://trac.sagemath.org/ticket/11027#comment:34
<p>
*ping* any ideas?
</p>
TicketrbeezerFri, 10 Jun 2011 00:46:31 GMT
https://trac.sagemath.org/ticket/11027#comment:35
https://trac.sagemath.org/ticket/11027#comment:35
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:34" title="Comment 34">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
*ping* any ideas?
</p>
</blockquote>
<p>
Hi Jeroen. Thanks for the interest.
</p>
<p>
Ideas? Lots:
</p>
<p>
(a) Doctesting this numerical stuff is hard.
</p>
<p>
(b) We should not be relying on Apple's vector libraries.
</p>
<p>
(c) Organizing Sage Days 31 has me distracted.
</p>
<p>
But seriously,
</p>
<p>
(i) I've thinking about taking on eigenvalues and eigenvectors, generally. There is a lot of inconsistency floating around. Ordering eigenvalues would be one thing to tackle. Which is the real reason I have not moved on this one.
</p>
<p>
(ii) Doctest in question is about verifying the 2x2 blocks along the diagonal of the real Schur decomposition. It would probably be better to not force the eigenvalues of the 2x2 blocks to match the usual Sage output, and instead just sort the "plain" eigenvalues to match the sorted list from the 2x2 blocks. That would be an easy fix.
</p>
<p>
(iii) The ordering of eigenvalues under "old" OS X might need to be investigated separately and fixed up. On Linux and Sage 4.7 I get:
</p>
<pre class="wiki">sage: A = matrix(RDF, 4, 4, [[1, 0, -3, -1],
... [4, -16, -7, 0],
... [1, 21, 1, -2],
... [26, -1, -2, 1]])
sage: A.eigenvalues()
[-0.789826456477 + 2.33693044103*I, -0.789826456477 - 2.33693044103*I, -5.71017354352 + 8.38264062467*I, -5.71017354352 - 8.38264062467*I]
</pre><p>
which is different from what you are reporting on OS X 10.4 PPC. Probably happening at the <code>NumPy</code> level or further down.
</p>
<p>
I'll likely make the change mentioned above in a few days, perhaps while at Sage Days. On vacation at the moment.
</p>
<p>
Rob
</p>
TicketrbeezerFri, 10 Jun 2011 02:05:13 GMTattachment set
https://trac.sagemath.org/ticket/11027
https://trac.sagemath.org/ticket/11027
<ul>
<li><strong>attachment</strong>
set to <em>trac_11027-schur-decomposition-doctest-v2.patch</em>
</li>
</ul>
TicketrbeezerFri, 10 Jun 2011 02:08:15 GMTstatus, description changed
https://trac.sagemath.org/ticket/11027#comment:36
https://trac.sagemath.org/ticket/11027#comment:36
<ul>
<li><strong>status</strong>
changed from <em>needs_work</em> to <em>needs_review</em>
</li>
<li><strong>description</strong>
modified (<a href="/ticket/11027?action=diff&version=36">diff</a>)
</li>
</ul>
<p>
I've made the change mentioned in (ii) above, and incorporated it into the "doctest" patch, which is now version 2.
</p>
<p>
Doctest works properly under Linux and conveys the same information as before. It should be more platform-independent, so hopefully it'll perform OK on OSX PPC.
</p>
TicketjdemeyerFri, 10 Jun 2011 08:32:14 GMT
https://trac.sagemath.org/ticket/11027#comment:37
https://trac.sagemath.org/ticket/11027#comment:37
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:35" title="Comment 35">rbeezer</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:34" title="Comment 34">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
*ping* any ideas?
</p>
</blockquote>
<p>
Hi Jeroen. Thanks for the interest.
</p>
</blockquote>
<p>
To be honest, I am not particularly interested in this patch. I just think it would be a pity to lose this work. There are already too many tickets which are 90% finished, let this not become one of them.
</p>
TicketrbeezerSun, 12 Jun 2011 03:23:36 GMT
https://trac.sagemath.org/ticket/11027#comment:38
https://trac.sagemath.org/ticket/11027#comment:38
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:37" title="Comment 37">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
I just think it would be a pity to lose this work. There are already too many tickets which are 90% finished, let this not become one of them.
</p>
</blockquote>
<p>
That qualifies as interest. ;-) I may be slow, but rarely give up. Except for <a class="needs_work ticket" href="https://trac.sagemath.org/ticket/10765" title="task: Obliterate sage/matrix/misc.pyx (needs_work)">#10765</a>.
</p>
<p>
Patchbot gives this a green light. What it really needs is a test run on OS X 10.4 PPC. Anybody willing (and able)?
</p>
TicketrbeezerTue, 14 Jun 2011 03:25:38 GMTcc changed
https://trac.sagemath.org/ticket/11027#comment:39
https://trac.sagemath.org/ticket/11027#comment:39
<ul>
<li><strong>cc</strong>
<em>kcrisman</em> added
</li>
</ul>
TicketjdemeyerWed, 15 Jun 2011 15:30:02 GMT
https://trac.sagemath.org/ticket/11027#comment:40
https://trac.sagemath.org/ticket/11027#comment:40
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:38" title="Comment 38">rbeezer</a>:
</p>
<blockquote class="citation">
<p>
What it really needs is a test run on OS X 10.4 PPC. Anybody willing (and able)?
</p>
</blockquote>
<p>
It works on OS X 10.4 PPC.
</p>
TicketrbeezerWed, 15 Jun 2011 21:10:11 GMT
https://trac.sagemath.org/ticket/11027#comment:41
https://trac.sagemath.org/ticket/11027#comment:41
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:40" title="Comment 40">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
It works on OS X 10.4 PPC.
</p>
</blockquote>
<p>
Thanks very much, Jeroen!
</p>
<p>
I think this is all ready for somebody to give it a (quick) positive review. Hint, hint. ;-)
</p>
TicketjdemeyerWed, 15 Jun 2011 21:21:34 GMTstatus, reviewer changed
https://trac.sagemath.org/ticket/11027#comment:42
https://trac.sagemath.org/ticket/11027#comment:42
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
<li><strong>reviewer</strong>
changed from <em>Martin Raum, John Palmieri</em> to <em>Martin Raum, John Palmieri, Jeroen Demeyer</em>
</li>
</ul>
<p>
Okay, I have looked at <a class="attachment" href="https://trac.sagemath.org/attachment/ticket/11027/trac_11027-schur-decomposition-doctest-v2.patch" title="Attachment 'trac_11027-schur-decomposition-doctest-v2.patch' in Ticket #11027">trac_11027-schur-decomposition-doctest-v2.patch</a><a class="trac-rawlink" href="https://trac.sagemath.org/raw-attachment/ticket/11027/trac_11027-schur-decomposition-doctest-v2.patch" title="Download"></a> and it makes sense. In fact, the new patch is cleaner than the old one. Since the main patch is already positively reviewed, this gives... positive_review!
</p>
TicketjdemeyerThu, 16 Jun 2011 07:24:06 GMTstatus changed; resolution, merged set
https://trac.sagemath.org/ticket/11027#comment:43
https://trac.sagemath.org/ticket/11027#comment:43
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
<li><strong>merged</strong>
set to <em>sage-4.7.1.alpha3</em>
</li>
</ul>
TicketrbeezerThu, 16 Jun 2011 13:38:45 GMT
https://trac.sagemath.org/ticket/11027#comment:44
https://trac.sagemath.org/ticket/11027#comment:44
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11027#comment:42" title="Comment 42">jdemeyer</a>:
</p>
<blockquote class="citation">
<blockquote>
<p>
this gives... positive_review!
</p>
</blockquote>
</blockquote>
<p>
Jeroen - thanks for finishing this one off. (And for your interest!)
</p>
Ticket