Sage: Ticket #5930: switch from maxima to pynac for core symbolic manipulation system
https://trac.sagemath.org/ticket/5930
<p>
SEE <a class="ext-link" href="http://wiki.sagemath.org/symbolics/pynac_todo/push"><span class="icon"></span>http://wiki.sagemath.org/symbolics/pynac_todo/push</a>
</p>
<p>
If you want to try the code out do this:
</p>
<ol><li>Install sage-3.4.2.
</li></ol><ol start="2"><li>Build pynac (should take about 3-5 minutes):
<pre class="wiki">./sage -i http://sage.math.washington.edu/home/mhansen/symbolics/pynac-0.1.6-mh.spkg
</pre></li></ol><ol start="3"><li>Apply the two patches at <a class="ext-link" href="http://sage.math.washington.edu/home/mhansen/symbolics/"><span class="icon"></span>http://sage.math.washington.edu/home/mhansen/symbolics/</a>
<pre class="wiki">sage: hg_sage.apply('http://sage.math.washington.edu/home/mhansen/symbolics/symbolics_1.patch')
sage: hg_sage.apply('http://sage.math.washington.edu/home/mhansen/symbolics/symbolics_2.patch')
</pre></li></ol><p>
Just check in each with some random comment.
</p>
<ol start="4"><li>Delete all traces of constants.so:
<pre class="wiki">rm devel/sage/build/*/sage/symbolic/constants*
rm devel/sage/build/sage/symbolic/constants.so
</pre></li></ol><ol start="5"><li>Build with "sage -br", and it should work. Building will take a while, since the patches touch many files.
</li></ol>en-usSagehttps://trac.sagemath.org/chrome/site/logo_sagemath_trac.png
https://trac.sagemath.org/ticket/5930
Trac 1.1.6burcinTue, 05 May 2009 09:57:11 GMTcomponent changed; cc set
https://trac.sagemath.org/ticket/5930#comment:1
https://trac.sagemath.org/ticket/5930#comment:1
<ul>
<li><strong>cc</strong>
<em>burcin</em> added
</li>
<li><strong>component</strong>
changed from <em>algebra</em> to <em>symbolics</em>
</li>
</ul>
TicketwasThu, 07 May 2009 16:56:27 GMTdescription changed
https://trac.sagemath.org/ticket/5930#comment:2
https://trac.sagemath.org/ticket/5930#comment:2
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/5930?action=diff&version=2">diff</a>)
</li>
</ul>
<p>
<a class="missing wiki">Bug/Issue?</a>: Control-C doesn't work in some cases.
</p>
<pre class="wiki">----------------------------------------------------------------------
| Sage Version 3.4.2, Release Date: 2009-05-05 |
| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------
sage: var('x,y,z')
(x, y, z)
sage: time f = (x+y+z)^5
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
sage: g = f*(f+1)
sage: timeit('g.expand()')
5 loops, best of 3: 53.2 ms per loop
sage: %prun g.expand()
8890 function calls in 0.069 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.065 0.065 0.069 0.069 {method 'expand' of 'sage.symbolic.expression.Expression' objects}
8863 0.003 0.000 0.003 0.000 functional.py:393(imag)
1 0.000 0.000 0.069 0.069 <string>:1(<module>)
4 0.000 0.000 0.000 0.000 arith.py:1140(gcd)
4 0.000 0.000 0.000 0.000 {method 'lcm' of 'sage.structure.element.PrincipalIdealDomainElement' objects}
4 0.000 0.000 0.000 0.000 arith.py:1256(lcm)
4 0.000 0.000 0.000 0.000 {method 'gcd' of 'sage.rings.integer.Integer' objects}
8 0.000 0.000 0.000 0.000 {hasattr}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
sage: %prun v = [g.expand() for _ in range(1000)]
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored
</pre>
TicketwasThu, 07 May 2009 17:06:42 GMTdescription changed
https://trac.sagemath.org/ticket/5930#comment:3
https://trac.sagemath.org/ticket/5930#comment:3
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/5930?action=diff&version=3">diff</a>)
</li>
</ul>
TicketwasThu, 07 May 2009 17:33:56 GMT
https://trac.sagemath.org/ticket/5930#comment:4
https://trac.sagemath.org/ticket/5930#comment:4
<p>
There is a sequence of *serious* speed regressions that I think Burcin caused by changes to the pynac spkg:
</p>
<p>
IN SAGE-3.2 we get OK timings for this benchmarks. They aren't great, but I can live with them, since it is"only" 57 times slower than Singular:
</p>
<pre class="wiki">sage: var('x,y,z', ns=1); f = (x+y+z)^6;
sage: timeit('(f*(f+1)).expand()')
125 loops, best of 3: 3.25 ms per loop
</pre><p>
In Singular:
</p>
<pre class="wiki">sage: R.<x,y,z> = QQ[]
sage: timeit('g=(x+y+z)^6*((x+y+z)^6+1)')
625 loops, best of 3: 56.3 µs per loop
sage: 3250/56.3
57.7264653641208
</pre><p>
It's hard to tell, but Mathematica seems to take about 1.7ms, which is comparable to the above:
</p>
<pre class="wiki">sage: timeit("s=mathematica('Expand[(x+y+z)^6*((x+y+z)^6+1)]')")
125 loops, best of 3: 1.81 ms per loop
sage: timeit("s=mathematica('2+3')")
625 loops, best of 3: 125 µs per loop
</pre><p>
Maxima via Sage takes about 61 ms, since I guess (c)lisp is slow, etc.:
</p>
<pre class="wiki">sage: timeit("s=maxima('expand((x+y+z)^6*((x+y+z)^6+1))')")
5 loops, best of 3: 61.8 ms per loop
</pre><p>
First there was a MAJOR unacceptable speed regression going to sage-3.3 (this is probably all caused by the pynac spkg). The timing jumped all the way to 42ms, so now it's almost as bad as Maxima:
</p>
<pre class="wiki">sage: var('x,y,z', ns=1); f = (x+y+z)^6
(x, y, z)
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 42.1 ms per loop
</pre><p>
In fact, directly in sage-3.2 with old Maxima symbolics:
</p>
<pre class="wiki">sage: var('x,y,z', ns=0); f = (x+y+z)^6
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 106 ms per loop
</pre><p>
IN SAGE-3.4.2 with new symbolics:
</p>
<pre class="wiki">sage: var('x,y,z', ns=1); f = (x+y+z)^6
(x, y, z)
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 206 ms per loop
</pre><p>
This may be due to a change in the pynac layer, where it is calling out to Python for some reason, even though it shouldn't need to. 206ms is really unacceptable. It's much slower than Maxima itself, it's 63 times slower than Pynac *was* just a few months ago, and it's 367 times slower than Singular.
</p>
TicketwasThu, 07 May 2009 23:25:44 GMTdescription changed
https://trac.sagemath.org/ticket/5930#comment:5
https://trac.sagemath.org/ticket/5930#comment:5
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/5930?action=diff&version=5">diff</a>)
</li>
</ul>
TicketwasFri, 08 May 2009 00:44:39 GMTdescription changed
https://trac.sagemath.org/ticket/5930#comment:6
https://trac.sagemath.org/ticket/5930#comment:6
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/5930?action=diff&version=6">diff</a>)
</li>
</ul>
TicketwasMon, 18 May 2009 07:27:02 GMTdescription, milestone changed
https://trac.sagemath.org/ticket/5930#comment:7
https://trac.sagemath.org/ticket/5930#comment:7
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/5930?action=diff&version=7">diff</a>)
</li>
<li><strong>milestone</strong>
changed from <em>sage-4.0.1</em> to <em>sage-4.0</em>
</li>
</ul>
TicketmhansenTue, 19 May 2009 04:11:36 GMTstatus, owner, description, summary changed
https://trac.sagemath.org/ticket/5930#comment:8
https://trac.sagemath.org/ticket/5930#comment:8
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>assigned</em>
</li>
<li><strong>owner</strong>
changed from <em>tbd</em> to <em>mhansen</em>
</li>
<li><strong>description</strong>
modified (<a href="/ticket/5930?action=diff&version=8">diff</a>)
</li>
<li><strong>summary</strong>
changed from <em>switch from maxima to pynac for core symbolic manipulation system</em> to <em>[with patch, needs review] switch from maxima to pynac for core symbolic manipulation system</em>
</li>
</ul>
<p>
I've attached symbolics_final2.patch which applies on top of <a class="closed ticket" href="https://trac.sagemath.org/ticket/5777" title="defect: [with package and patches, positive review] update to pynac 0.1.6 (closed: fixed)">#5777</a>. This patch also requires the Pynac 0.1.7 spkg at <a class="ext-link" href="http://sage.math.washington.edu/home/mhansen/pynac-0.1.7.spkg"><span class="icon"></span>http://sage.math.washington.edu/home/mhansen/pynac-0.1.7.spkg</a>
</p>
TicketwasTue, 19 May 2009 20:16:51 GMTdescription changed
https://trac.sagemath.org/ticket/5930#comment:9
https://trac.sagemath.org/ticket/5930#comment:9
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/5930?action=diff&version=9">diff</a>)
</li>
</ul>
TicketwasTue, 19 May 2009 20:19:08 GMTdescription changed
https://trac.sagemath.org/ticket/5930#comment:10
https://trac.sagemath.org/ticket/5930#comment:10
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/5930?action=diff&version=10">diff</a>)
</li>
</ul>
TicketwasTue, 19 May 2009 20:21:25 GMT
https://trac.sagemath.org/ticket/5930#comment:11
https://trac.sagemath.org/ticket/5930#comment:11
<p>
After applying the first patch, the second fails. This is to clean 4.0.alpha0:
</p>
<pre class="wiki">sage: hg_sage.apply('http://trac.sagemath.org/sage_trac/attachment/ticket/5930/symbolics_final2.patch')
Attempting to load remote file: http://trac.sagemath.org/sage_trac/raw-attachment/ticket/5930/symbolics_final2.patch
Loading: [..................................................]
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg status
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg status
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg import "/Users/wstein/.sage/temp/teragon.local/1113/tmp_2.patch"
applying /Users/wstein/.sage/temp/teragon.local/1113/tmp_2.patch
patching file doc/en/constructions/calculus.rst
Hunk #5 FAILED at 141
1 out of 10 hunks FAILED -- saving rejects to file doc/en/constructions/calculus.rst.rej
patching file sage/calculus/calculus.py
Hunk #15 FAILED at 291
1 out of 25 hunks FAILED -- saving rejects to file sage/calculus/calculus.py.rej
patching file sage/quadratic_forms/quadratic_form__mass__Conway_Sloane_masses.py
Hunk #1 succeeded at 4 with fuzz 2 (offset 0 lines).
patching file sage/symbolic/function.pyx
Hunk #9 FAILED at 206
1 out of 25 hunks FAILED -- saving rejects to file sage/symbolic/function.pyx.rej
abort: patch failed to apply
</pre>
TicketmabshoffTue, 19 May 2009 20:23:15 GMT
https://trac.sagemath.org/ticket/5930#comment:12
https://trac.sagemath.org/ticket/5930#comment:12
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/5930#comment:11" title="Comment 11">was</a>:
</p>
<blockquote class="citation">
<p>
After applying the first patch, the second fails. This is to clean 4.0.alpha0:
</p>
</blockquote>
<p>
This patch set is on top of last night's 4.0.rc0 merge tree and will not work with 4.0.a0.
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketmabshoffTue, 19 May 2009 20:59:29 GMT
https://trac.sagemath.org/ticket/5930#comment:13
https://trac.sagemath.org/ticket/5930#comment:13
<p>
symbolics_final2.patch does not import:
</p>
<pre class="wiki">mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ hg import symbolics_final1.patch
applying symbolics_final1.patch
mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ hg import symbolics_final2.patch
applying symbolics_final2.patch
patching file sage/symbolic/function.pyx
Hunk #9 FAILED at 206
1 out of 25 hunks FAILED -- saving rejects to file sage/symbolic/function.pyx.rej
abort: patch failed to apply
mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ less sage/symbolic/function.pyx.rej
</pre><p>
I have not touched the file that sees rejects since Mike pulled it last night, so please fix this :)
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketmhansenTue, 19 May 2009 23:02:16 GMTattachment set
https://trac.sagemath.org/ticket/5930
https://trac.sagemath.org/ticket/5930
<ul>
<li><strong>attachment</strong>
set to <em>symbolics_final2.patch</em>
</li>
</ul>
TicketmabshoffTue, 19 May 2009 23:27:04 GMT
https://trac.sagemath.org/ticket/5930#comment:14
https://trac.sagemath.org/ticket/5930#comment:14
<p>
Ok, the latest patch applies, but lacks a commit message.
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketmabshoffWed, 20 May 2009 01:02:07 GMTsummary changed
https://trac.sagemath.org/ticket/5930#comment:15
https://trac.sagemath.org/ticket/5930#comment:15
<ul>
<li><strong>summary</strong>
changed from <em>[with patch, needs review] switch from maxima to pynac for core symbolic manipulation system</em> to <em>[with patch, needs work] switch from maxima to pynac for core symbolic manipulation system</em>
</li>
</ul>
<p>
The spkg needs work:
</p>
<pre class="wiki">configure: creating ./config.statusConfiguration of GiNaC 0.1.5 done. Now type "make".
/bin/sh ./config.status
config.status: creating Makefileconfig.status: creating pynac.spec
config.status: creating pynac.pcconfig.status: creating ginac/Makefile
config.status: creating ginac/version.h
config.status: creating config.hconfig.status: config.h is unchanged
config.status: executing depfiles commandscd . && /bin/sh /home/mabshoff/build-4.0.alpha0/sage-4.0.alpha0-cleo-system/spkg/build/pynac-0.1.7/src/missing --run autoheader
aclocal.m4:20: warning: this file was generated for autoconf 2.61.
You have another version of autoconf. It may work, but is not guaranteed to.If you have problems, you may need to regenerate the build system entirely.
To do so, use the procedure documented by the package, typically `autoreconf'.configure.ac:26: error: Autoconf version 2.60 or higher is required
aclocal.m4:7127: AM_INIT_AUTOMAKE is expanded from...
configure.ac:26: the top levelautom4te: /usr/bin/m4 failed with exit status: 63
autoheader: /usr/bin/autom4te failed with exit status: 63
</pre><p>
All this autocrap should never be run in the spkg. The version number as well as the name of the library is also wrong.
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketboothbyWed, 20 May 2009 04:50:08 GMT
https://trac.sagemath.org/ticket/5930#comment:16
https://trac.sagemath.org/ticket/5930#comment:16
<p>
I've been reading through the megabyte patch, and testing every function that's been touched. I'd estimate my coverage of the file to be about 20%, and I made sure to bounce around a bunch (that is, I didn't just read the top or bottom 20%). I didn't look at many corner cases, because I was trying to test as much as I could, and it often takes thought to come up with good corner cases. In the end, I found some things that Maxima crashed on when it tried to simplify, like
</p>
<pre class="wiki"> exp(sum([log(x^(1/n)) for n in range(1,1000)]))
</pre><p>
but other than that, I found the entire system to be stable and useful. In a lot of easy cases, it's *much* faster than before, which makes me happy.
</p>
TicketmabshoffWed, 20 May 2009 10:49:23 GMT
https://trac.sagemath.org/ticket/5930#comment:17
https://trac.sagemath.org/ticket/5930#comment:17
<p>
I am doing the following
</p>
<ul><li>install pynac-1.0.17
</li><li>pull in latest changes
</li><li>sage -ba
</li><li>install dsage
</li><li>update pickle jar
</li><li>run testlong
</li></ul><p>
on
</p>
<ul><li>Linux x86, x86-64 and Itanium
</li><li>some of the above with gcc 4.4
</li><li>Solaris Sparc (32 bit)
</li><li>OSX 10.4 (PPC)
</li></ul><p>
I am currently running <code>testlong</code>, so we should know more in the morning (local time)
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketmabshoffWed, 20 May 2009 13:45:59 GMT
https://trac.sagemath.org/ticket/5930#comment:18
https://trac.sagemath.org/ticket/5930#comment:18
<p>
Ok, no joy: On <a class="missing wiki">Solaris/Sparc?</a> this patchset causes a hang in maxima.py. Mike told me that William and him allegedly fixed the underlying issue (see the discussion about the semicolon at <a class="closed ticket" href="https://trac.sagemath.org/ticket/6054" title="defect: [with patch, positive review] fix display2d with ecl on OSX (closed: fixed)">#6054</a>), but it causes the hang. All Maxima related doctests pass on that machine (modulo two tiny numerical noise problems), so the patch on this ticket is to blame.
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketmabshoffWed, 20 May 2009 13:48:38 GMT
https://trac.sagemath.org/ticket/5930#comment:19
https://trac.sagemath.org/ticket/5930#comment:19
<p>
And the failing doctest is this one:
</p>
<pre class="wiki">Trying:
maxima('2+2')###line 751:_sage_ >>> maxima('2+2')
Expecting:
4
</pre><p>
Cheers,
</p>
<p>
Michael
</p>
TicketmabshoffWed, 20 May 2009 13:57:49 GMT
https://trac.sagemath.org/ticket/5930#comment:20
https://trac.sagemath.org/ticket/5930#comment:20
<p>
Hmm, this looks suspicious, but reverting it does not fix the problem:
</p>
<pre class="wiki">@@ -755,7 +755,7 @@
if self._expect is None: return
r = randrange(2147483647)
s = marker + str(r+1)
- cmd = ''';sconcat("%s",(%s+1));\n'''%(marker,r)
+ cmd = '''0;sconcat("%s",(%s+1));\n'''%(marker,r)
self._sendstr(cmd)
try:
self._expect_expr(timeout=0.5)
</pre><p>
Running <code>maxima('2+2')</code> in a loop does work, so I am not sure what the problem is yet.
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketmabshoffWed, 20 May 2009 14:03:52 GMT
https://trac.sagemath.org/ticket/5930#comment:21
https://trac.sagemath.org/ticket/5930#comment:21
<p>
Hmm, another thing:
</p>
<pre class="wiki"> cdef class Matrix_symbolic_dense(matrix_dense.Matrix_dense):
r"""
@@ -162,7 +161,7 @@
sage: cmp(m,m)
0
sage: cmp(m,3)
- -1
+ 1
"""
return self._richcmp(right, op)
</pre><p>
is cmp() in this case deterministic?
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketmabshoffThu, 21 May 2009 02:46:20 GMTsummary changed
https://trac.sagemath.org/ticket/5930#comment:22
https://trac.sagemath.org/ticket/5930#comment:22
<ul>
<li><strong>summary</strong>
changed from <em>[with patch, needs work] switch from maxima to pynac for core symbolic manipulation system</em> to <em>[with patch, positive review] switch from maxima to pynac for core symbolic manipulation system</em>
</li>
</ul>
<p>
Formal positive review by various other people not me. Followup should be directed to new tickets for 4.0 as long as it is open.
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketmabshoffThu, 21 May 2009 02:46:34 GMTstatus changed; resolution set
https://trac.sagemath.org/ticket/5930#comment:23
https://trac.sagemath.org/ticket/5930#comment:23
<ul>
<li><strong>status</strong>
changed from <em>assigned</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
</ul>
<p>
Merged in Sage 4.0.rc0.
</p>
<p>
Cheers,
</p>
<p>
Michael
</p>
TicketwasThu, 21 May 2009 09:53:03 GMTsummary changed
https://trac.sagemath.org/ticket/5930#comment:24
https://trac.sagemath.org/ticket/5930#comment:24
<ul>
<li><strong>summary</strong>
changed from <em>[with patch, positive review] switch from maxima to pynac for core symbolic manipulation system</em> to <em>[with patch, positive review] switch from maxima to pynac for core symbolic manipulation system</em>
</li>
</ul>
<p>
See <a class="closed ticket" href="https://trac.sagemath.org/ticket/6111" title="defect: [with patch; positive review] review symbolics in sage-4.0 (switch to ... (closed: fixed)">#6111</a> for refereeing of symbolics.
</p>
TicketjdemeyerWed, 07 Jun 2017 12:55:52 GMTupstream set
https://trac.sagemath.org/ticket/5930#comment:25
https://trac.sagemath.org/ticket/5930#comment:25
<ul>
<li><strong>upstream</strong>
set to <em>N/A</em>
</li>
</ul>
<p>
Does anybody happen to remember why there is this strange condition involving <code>inspect.ismethod</code>?
</p>
<div class="wiki-code"><div class="code"><pre> <span class="kn">import</span> <span class="nn">inspect</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span>_the_element<span class="p">,</span><span class="s">'_fast_callable_'</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> inspect<span class="o">.</span>ismethod<span class="p">(</span>_the_element<span class="o">.</span>_fast_callable_<span class="p">):</span>
<span class="c"># only warn if _the_element is not dynamic</span>
<span class="kn">from</span> <span class="nn">sage.misc.superseded</span> <span class="kn">import</span> deprecation
deprecation<span class="p">(</span><span class="mi">5930</span><span class="p">,</span> <span class="s">"Substitution using function-call syntax and unnamed arguments is deprecated and will be removed from a future release of Sage; you ca</span>
</pre></div></div>
TicketjdemeyerWed, 07 Jun 2017 13:00:42 GMT
https://trac.sagemath.org/ticket/5930#comment:26
https://trac.sagemath.org/ticket/5930#comment:26
<p>
Never mind, that condition was actually added in <a class="closed ticket" href="https://trac.sagemath.org/ticket/2516" title="enhancement: generalized hypergeometric functions should be implemented (closed: fixed)">#2516</a>.
</p>
Ticket