Opened 14 years ago

Closed 14 years ago

Last modified 6 years ago

#5930 closed enhancement (fixed)

[with patch, positive review] switch from maxima to pynac for core symbolic manipulation system

Reported by: William Stein Owned by: Mike Hansen
Priority: blocker Milestone: sage-4.0
Component: symbolics Keywords:
Cc: Burcin Erocal Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by William Stein)

SEE http://wiki.sagemath.org/symbolics/pynac_todo/push

If you want to try the code out do this:

  1. Apply symbolics_final1.patch from #5777
  1. Apply symbolics_final2.patch from this ticket.
  1. Build new pynac spkg:

http://sage.math.washington.edu/home/mhansen/pynac-0.1.7.spkg

Attachments (1)

symbolics_final2.patch (1.2 MB) - added by Mike Hansen 14 years ago.

Download all attachments as: .zip

Change History (27)

comment:1 Changed 14 years ago by Burcin Erocal

Cc: Burcin Erocal added
Component: algebrasymbolics

comment:2 Changed 14 years ago by William Stein

Description: modified (diff)

Bug/Issue?: Control-C doesn't work in some cases.

----------------------------------------------------------------------
| Sage Version 3.4.2, Release Date: 2009-05-05                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: var('x,y,z')
(x, y, z)
sage: time f = (x+y+z)^5
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
sage: g = f*(f+1)
sage: timeit('g.expand()')
5 loops, best of 3: 53.2 ms per loop
sage: %prun g.expand()
         8890 function calls in 0.069 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.065    0.065    0.069    0.069 {method 'expand' of 'sage.symbolic.expression.Expression' objects}
     8863    0.003    0.000    0.003    0.000 functional.py:393(imag)
        1    0.000    0.000    0.069    0.069 <string>:1(<module>)
        4    0.000    0.000    0.000    0.000 arith.py:1140(gcd)
        4    0.000    0.000    0.000    0.000 {method 'lcm' of 'sage.structure.element.PrincipalIdealDomainElement' objects}
        4    0.000    0.000    0.000    0.000 arith.py:1256(lcm)
        4    0.000    0.000    0.000    0.000 {method 'gcd' of 'sage.rings.integer.Integer' objects}
        8    0.000    0.000    0.000    0.000 {hasattr}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

sage: %prun v = [g.expand() for _ in range(1000)]
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored

comment:3 Changed 14 years ago by William Stein

Description: modified (diff)

comment:4 Changed 14 years ago by William Stein

There is a sequence of *serious* speed regressions that I think Burcin caused by changes to the pynac spkg:

IN SAGE-3.2 we get OK timings for this benchmarks. They aren't great, but I can live with them, since it is"only" 57 times slower than Singular:

sage: var('x,y,z', ns=1); f = (x+y+z)^6;
sage: timeit('(f*(f+1)).expand()')
125 loops, best of 3: 3.25 ms per loop

In Singular:

sage: R.<x,y,z> = QQ[]
sage: timeit('g=(x+y+z)^6*((x+y+z)^6+1)')
625 loops, best of 3: 56.3 µs per loop
sage: 3250/56.3
57.7264653641208

It's hard to tell, but Mathematica seems to take about 1.7ms, which is comparable to the above:

sage: timeit("s=mathematica('Expand[(x+y+z)^6*((x+y+z)^6+1)]')")
125 loops, best of 3: 1.81 ms per loop
sage: timeit("s=mathematica('2+3')")
625 loops, best of 3: 125 µs per loop

Maxima via Sage takes about 61 ms, since I guess (c)lisp is slow, etc.:

sage: timeit("s=maxima('expand((x+y+z)^6*((x+y+z)^6+1))')")
5 loops, best of 3: 61.8 ms per loop

First there was a MAJOR unacceptable speed regression going to sage-3.3 (this is probably all caused by the pynac spkg). The timing jumped all the way to 42ms, so now it's almost as bad as Maxima:

sage: var('x,y,z', ns=1); f = (x+y+z)^6
(x, y, z)
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 42.1 ms per loop

In fact, directly in sage-3.2 with old Maxima symbolics:

sage: var('x,y,z', ns=0); f = (x+y+z)^6
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 106 ms per loop

IN SAGE-3.4.2 with new symbolics:

sage: var('x,y,z', ns=1); f = (x+y+z)^6
(x, y, z)
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 206 ms per loop

This may be due to a change in the pynac layer, where it is calling out to Python for some reason, even though it shouldn't need to. 206ms is really unacceptable. It's much slower than Maxima itself, it's 63 times slower than Pynac *was* just a few months ago, and it's 367 times slower than Singular.

comment:5 Changed 14 years ago by William Stein

Description: modified (diff)

comment:6 Changed 14 years ago by William Stein

Description: modified (diff)

comment:7 Changed 14 years ago by William Stein

Description: modified (diff)
Milestone: sage-4.0.1sage-4.0

comment:8 Changed 14 years ago by Mike Hansen

Description: modified (diff)
Owner: changed from tbd to Mike Hansen
Status: newassigned
Summary: switch from maxima to pynac for core symbolic manipulation system[with patch, needs review] switch from maxima to pynac for core symbolic manipulation system

I've attached symbolics_final2.patch which applies on top of #5777. This patch also requires the Pynac 0.1.7 spkg at http://sage.math.washington.edu/home/mhansen/pynac-0.1.7.spkg

comment:9 Changed 14 years ago by William Stein

Description: modified (diff)

comment:10 Changed 14 years ago by William Stein

Description: modified (diff)

comment:11 Changed 14 years ago by William Stein

After applying the first patch, the second fails. This is to clean 4.0.alpha0:

sage: hg_sage.apply('http://trac.sagemath.org/sage_trac/attachment/ticket/5930/symbolics_final2.patch')
Attempting to load remote file: http://trac.sagemath.org/sage_trac/raw-attachment/ticket/5930/symbolics_final2.patch
Loading: [..................................................]
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg status
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg status
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg import   "/Users/wstein/.sage/temp/teragon.local/1113/tmp_2.patch"
applying /Users/wstein/.sage/temp/teragon.local/1113/tmp_2.patch
patching file doc/en/constructions/calculus.rst
Hunk #5 FAILED at 141
1 out of 10 hunks FAILED -- saving rejects to file doc/en/constructions/calculus.rst.rej
patching file sage/calculus/calculus.py
Hunk #15 FAILED at 291
1 out of 25 hunks FAILED -- saving rejects to file sage/calculus/calculus.py.rej
patching file sage/quadratic_forms/quadratic_form__mass__Conway_Sloane_masses.py
Hunk #1 succeeded at 4 with fuzz 2 (offset 0 lines).
patching file sage/symbolic/function.pyx
Hunk #9 FAILED at 206
1 out of 25 hunks FAILED -- saving rejects to file sage/symbolic/function.pyx.rej
abort: patch failed to apply

comment:12 in reply to:  11 Changed 14 years ago by Michael Abshoff

Replying to was:

After applying the first patch, the second fails. This is to clean 4.0.alpha0:

This patch set is on top of last night's 4.0.rc0 merge tree and will not work with 4.0.a0.

Cheers,

Michael

comment:13 Changed 14 years ago by Michael Abshoff

symbolics_final2.patch does not import:

mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ hg import symbolics_final1.patch
applying symbolics_final1.patch
mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ hg import symbolics_final2.patch
applying symbolics_final2.patch
patching file sage/symbolic/function.pyx
Hunk #9 FAILED at 206
1 out of 25 hunks FAILED -- saving rejects to file sage/symbolic/function.pyx.rej
abort: patch failed to apply
mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ less sage/symbolic/function.pyx.rej

I have not touched the file that sees rejects since Mike pulled it last night, so please fix this :)

Cheers,

Michael

Changed 14 years ago by Mike Hansen

Attachment: symbolics_final2.patch added

comment:14 Changed 14 years ago by Michael Abshoff

Ok, the latest patch applies, but lacks a commit message.

Cheers,

Michael

comment:15 Changed 14 years ago by Michael Abshoff

Summary: [with patch, needs review] switch from maxima to pynac for core symbolic manipulation system[with patch, needs work] switch from maxima to pynac for core symbolic manipulation system

The spkg needs work:

configure: creating ./config.statusConfiguration of GiNaC 0.1.5 done. Now type "make".
 /bin/sh ./config.status
config.status: creating Makefileconfig.status: creating pynac.spec
config.status: creating pynac.pcconfig.status: creating ginac/Makefile
config.status: creating ginac/version.h
config.status: creating config.hconfig.status: config.h is unchanged
config.status: executing depfiles commandscd . && /bin/sh /home/mabshoff/build-4.0.alpha0/sage-4.0.alpha0-cleo-system/spkg/build/pynac-0.1.7/src/missing --run autoheader
aclocal.m4:20: warning: this file was generated for autoconf 2.61.
You have another version of autoconf.  It may work, but is not guaranteed to.If you have problems, you may need to regenerate the build system entirely.
To do so, use the procedure documented by the package, typically `autoreconf'.configure.ac:26: error: Autoconf version 2.60 or higher is required
aclocal.m4:7127: AM_INIT_AUTOMAKE is expanded from...
configure.ac:26: the top levelautom4te: /usr/bin/m4 failed with exit status: 63
autoheader: /usr/bin/autom4te failed with exit status: 63

All this autocrap should never be run in the spkg. The version number as well as the name of the library is also wrong.

Cheers,

Michael

comment:16 Changed 14 years ago by Kelly Boothby

I've been reading through the megabyte patch, and testing every function that's been touched. I'd estimate my coverage of the file to be about 20%, and I made sure to bounce around a bunch (that is, I didn't just read the top or bottom 20%). I didn't look at many corner cases, because I was trying to test as much as I could, and it often takes thought to come up with good corner cases. In the end, I found some things that Maxima crashed on when it tried to simplify, like

 exp(sum([log(x^(1/n)) for n in range(1,1000)]))

but other than that, I found the entire system to be stable and useful. In a lot of easy cases, it's *much* faster than before, which makes me happy.

comment:17 Changed 14 years ago by Michael Abshoff

I am doing the following

  • install pynac-1.0.17
  • pull in latest changes
  • sage -ba
  • install dsage
  • update pickle jar
  • run testlong

on

  • Linux x86, x86-64 and Itanium
  • some of the above with gcc 4.4
  • Solaris Sparc (32 bit)
  • OSX 10.4 (PPC)

I am currently running testlong, so we should know more in the morning (local time)

Cheers,

Michael

comment:18 Changed 14 years ago by Michael Abshoff

Ok, no joy: On Solaris/Sparc? this patchset causes a hang in maxima.py. Mike told me that William and him allegedly fixed the underlying issue (see the discussion about the semicolon at #6054), but it causes the hang. All Maxima related doctests pass on that machine (modulo two tiny numerical noise problems), so the patch on this ticket is to blame.

Cheers,

Michael

comment:19 Changed 14 years ago by Michael Abshoff

And the failing doctest is this one:

Trying:
    maxima('2+2')###line 751:_sage_    >>> maxima('2+2')
Expecting:
    4

Cheers,

Michael

comment:20 Changed 14 years ago by Michael Abshoff

Hmm, this looks suspicious, but reverting it does not fix the problem:

@@ -755,7 +755,7 @@
         if self._expect is None: return
         r = randrange(2147483647)
         s = marker + str(r+1)
-        cmd = ''';sconcat("%s",(%s+1));\n'''%(marker,r)
+        cmd = '''0;sconcat("%s",(%s+1));\n'''%(marker,r)
         self._sendstr(cmd)
         try:
             self._expect_expr(timeout=0.5)

Running maxima('2+2') in a loop does work, so I am not sure what the problem is yet.

Cheers,

Michael

comment:21 Changed 14 years ago by Michael Abshoff

Hmm, another thing:

 cdef class Matrix_symbolic_dense(matrix_dense.Matrix_dense):
     r"""
@@ -162,7 +161,7 @@
             sage: cmp(m,m)
             0
             sage: cmp(m,3)
-            -1
+            1
         """
         return self._richcmp(right, op)
 

is cmp() in this case deterministic?

Cheers,

Michael

comment:22 Changed 14 years ago by Michael Abshoff

Summary: [with patch, needs work] switch from maxima to pynac for core symbolic manipulation system[with patch, positive review] switch from maxima to pynac for core symbolic manipulation system

Formal positive review by various other people not me. Followup should be directed to new tickets for 4.0 as long as it is open.

Cheers,

Michael

comment:23 Changed 14 years ago by Michael Abshoff

Resolution: fixed
Status: assignedclosed

Merged in Sage 4.0.rc0.

Cheers,

Michael

comment:24 Changed 14 years ago by William Stein

Summary: [with patch, positive review] switch from maxima to pynac for core symbolic manipulation system[with patch, positive review] switch from maxima to pynac for core symbolic manipulation system

See #6111 for refereeing of symbolics.

comment:25 Changed 6 years ago by Jeroen Demeyer

Report Upstream: N/A

Does anybody happen to remember why there is this strange condition involving inspect.ismethod?

            import inspect
            if not hasattr(_the_element,'_fast_callable_') or not inspect.ismethod(_the_element._fast_callable_):
                # only warn if _the_element is not dynamic
                from sage.misc.superseded import deprecation
                deprecation(5930, "Substitution using function-call syntax and unnamed arguments is deprecated and will be removed from a future release of Sage; you ca

comment:26 Changed 6 years ago by Jeroen Demeyer

Never mind, that condition was actually added in #2516.

Note: See TracTickets for help on using tickets.