Opened 10 years ago

Closed 10 years ago

Last modified 18 months ago

#5930 closed enhancement (fixed)

[with patch, positive review] switch from maxima to pynac for core symbolic manipulation system

Reported by: was Owned by: mhansen
Priority: blocker Milestone: sage-4.0
Component: symbolics Keywords:
Cc: burcin Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by was)

SEE http://wiki.sagemath.org/symbolics/pynac_todo/push

If you want to try the code out do this:

  1. Apply symbolics_final1.patch from #5777
  1. Apply symbolics_final2.patch from this ticket.
  1. Build new pynac spkg:

http://sage.math.washington.edu/home/mhansen/pynac-0.1.7.spkg

Attachments (1)

symbolics_final2.patch (1.2 MB) - added by mhansen 10 years ago.

Download all attachments as: .zip

Change History (27)

comment:1 Changed 10 years ago by burcin

  • Cc burcin added
  • Component changed from algebra to symbolics

comment:2 Changed 10 years ago by was

  • Description modified (diff)

Bug/Issue?: Control-C doesn't work in some cases.

----------------------------------------------------------------------
| Sage Version 3.4.2, Release Date: 2009-05-05                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: var('x,y,z')
(x, y, z)
sage: time f = (x+y+z)^5
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
sage: g = f*(f+1)
sage: timeit('g.expand()')
5 loops, best of 3: 53.2 ms per loop
sage: %prun g.expand()
         8890 function calls in 0.069 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.065    0.065    0.069    0.069 {method 'expand' of 'sage.symbolic.expression.Expression' objects}
     8863    0.003    0.000    0.003    0.000 functional.py:393(imag)
        1    0.000    0.000    0.069    0.069 <string>:1(<module>)
        4    0.000    0.000    0.000    0.000 arith.py:1140(gcd)
        4    0.000    0.000    0.000    0.000 {method 'lcm' of 'sage.structure.element.PrincipalIdealDomainElement' objects}
        4    0.000    0.000    0.000    0.000 arith.py:1256(lcm)
        4    0.000    0.000    0.000    0.000 {method 'gcd' of 'sage.rings.integer.Integer' objects}
        8    0.000    0.000    0.000    0.000 {hasattr}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

sage: %prun v = [g.expand() for _ in range(1000)]
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored
^CException exceptions.KeyboardInterrupt: KeyboardInterrupt() in 'sage.symbolic.pynac.py_is_real' ignored

comment:3 Changed 10 years ago by was

  • Description modified (diff)

comment:4 Changed 10 years ago by was

There is a sequence of *serious* speed regressions that I think Burcin caused by changes to the pynac spkg:

IN SAGE-3.2 we get OK timings for this benchmarks. They aren't great, but I can live with them, since it is"only" 57 times slower than Singular:

sage: var('x,y,z', ns=1); f = (x+y+z)^6;
sage: timeit('(f*(f+1)).expand()')
125 loops, best of 3: 3.25 ms per loop

In Singular:

sage: R.<x,y,z> = QQ[]
sage: timeit('g=(x+y+z)^6*((x+y+z)^6+1)')
625 loops, best of 3: 56.3 µs per loop
sage: 3250/56.3
57.7264653641208

It's hard to tell, but Mathematica seems to take about 1.7ms, which is comparable to the above:

sage: timeit("s=mathematica('Expand[(x+y+z)^6*((x+y+z)^6+1)]')")
125 loops, best of 3: 1.81 ms per loop
sage: timeit("s=mathematica('2+3')")
625 loops, best of 3: 125 µs per loop

Maxima via Sage takes about 61 ms, since I guess (c)lisp is slow, etc.:

sage: timeit("s=maxima('expand((x+y+z)^6*((x+y+z)^6+1))')")
5 loops, best of 3: 61.8 ms per loop

First there was a MAJOR unacceptable speed regression going to sage-3.3 (this is probably all caused by the pynac spkg). The timing jumped all the way to 42ms, so now it's almost as bad as Maxima:

sage: var('x,y,z', ns=1); f = (x+y+z)^6
(x, y, z)
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 42.1 ms per loop

In fact, directly in sage-3.2 with old Maxima symbolics:

sage: var('x,y,z', ns=0); f = (x+y+z)^6
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 106 ms per loop

IN SAGE-3.4.2 with new symbolics:

sage: var('x,y,z', ns=1); f = (x+y+z)^6
(x, y, z)
sage: timeit('(f*(f+1)).expand()')
5 loops, best of 3: 206 ms per loop

This may be due to a change in the pynac layer, where it is calling out to Python for some reason, even though it shouldn't need to. 206ms is really unacceptable. It's much slower than Maxima itself, it's 63 times slower than Pynac *was* just a few months ago, and it's 367 times slower than Singular.

comment:5 Changed 10 years ago by was

  • Description modified (diff)

comment:6 Changed 10 years ago by was

  • Description modified (diff)

comment:7 Changed 10 years ago by was

  • Description modified (diff)
  • Milestone changed from sage-4.0.1 to sage-4.0

comment:8 Changed 10 years ago by mhansen

  • Description modified (diff)
  • Owner changed from tbd to mhansen
  • Status changed from new to assigned
  • Summary changed from switch from maxima to pynac for core symbolic manipulation system to [with patch, needs review] switch from maxima to pynac for core symbolic manipulation system

I've attached symbolics_final2.patch which applies on top of #5777. This patch also requires the Pynac 0.1.7 spkg at http://sage.math.washington.edu/home/mhansen/pynac-0.1.7.spkg

comment:9 Changed 10 years ago by was

  • Description modified (diff)

comment:10 Changed 10 years ago by was

  • Description modified (diff)

comment:11 follow-up: Changed 10 years ago by was

After applying the first patch, the second fails. This is to clean 4.0.alpha0:

sage: hg_sage.apply('http://trac.sagemath.org/sage_trac/attachment/ticket/5930/symbolics_final2.patch')
Attempting to load remote file: http://trac.sagemath.org/sage_trac/raw-attachment/ticket/5930/symbolics_final2.patch
Loading: [..................................................]
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg status
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg status
cd "/Users/wstein/build/sage-4.0.alpha0/devel/sage" && hg import   "/Users/wstein/.sage/temp/teragon.local/1113/tmp_2.patch"
applying /Users/wstein/.sage/temp/teragon.local/1113/tmp_2.patch
patching file doc/en/constructions/calculus.rst
Hunk #5 FAILED at 141
1 out of 10 hunks FAILED -- saving rejects to file doc/en/constructions/calculus.rst.rej
patching file sage/calculus/calculus.py
Hunk #15 FAILED at 291
1 out of 25 hunks FAILED -- saving rejects to file sage/calculus/calculus.py.rej
patching file sage/quadratic_forms/quadratic_form__mass__Conway_Sloane_masses.py
Hunk #1 succeeded at 4 with fuzz 2 (offset 0 lines).
patching file sage/symbolic/function.pyx
Hunk #9 FAILED at 206
1 out of 25 hunks FAILED -- saving rejects to file sage/symbolic/function.pyx.rej
abort: patch failed to apply

comment:12 in reply to: ↑ 11 Changed 10 years ago by mabshoff

Replying to was:

After applying the first patch, the second fails. This is to clean 4.0.alpha0:

This patch set is on top of last night's 4.0.rc0 merge tree and will not work with 4.0.a0.

Cheers,

Michael

comment:13 Changed 10 years ago by mabshoff

symbolics_final2.patch does not import:

mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ hg import symbolics_final1.patch
applying symbolics_final1.patch
mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ hg import symbolics_final2.patch
applying symbolics_final2.patch
patching file sage/symbolic/function.pyx
Hunk #9 FAILED at 206
1 out of 25 hunks FAILED -- saving rejects to file sage/symbolic/function.pyx.rej
abort: patch failed to apply
mabshoff@sage:/scratch/mabshoff/sage-4.0.rc0/devel/sage$ less sage/symbolic/function.pyx.rej

I have not touched the file that sees rejects since Mike pulled it last night, so please fix this :)

Cheers,

Michael

Changed 10 years ago by mhansen

comment:14 Changed 10 years ago by mabshoff

Ok, the latest patch applies, but lacks a commit message.

Cheers,

Michael

comment:15 Changed 10 years ago by mabshoff

  • Summary changed from [with patch, needs review] switch from maxima to pynac for core symbolic manipulation system to [with patch, needs work] switch from maxima to pynac for core symbolic manipulation system

The spkg needs work:

configure: creating ./config.statusConfiguration of GiNaC 0.1.5 done. Now type "make".
 /bin/sh ./config.status
config.status: creating Makefileconfig.status: creating pynac.spec
config.status: creating pynac.pcconfig.status: creating ginac/Makefile
config.status: creating ginac/version.h
config.status: creating config.hconfig.status: config.h is unchanged
config.status: executing depfiles commandscd . && /bin/sh /home/mabshoff/build-4.0.alpha0/sage-4.0.alpha0-cleo-system/spkg/build/pynac-0.1.7/src/missing --run autoheader
aclocal.m4:20: warning: this file was generated for autoconf 2.61.
You have another version of autoconf.  It may work, but is not guaranteed to.If you have problems, you may need to regenerate the build system entirely.
To do so, use the procedure documented by the package, typically `autoreconf'.configure.ac:26: error: Autoconf version 2.60 or higher is required
aclocal.m4:7127: AM_INIT_AUTOMAKE is expanded from...
configure.ac:26: the top levelautom4te: /usr/bin/m4 failed with exit status: 63
autoheader: /usr/bin/autom4te failed with exit status: 63

All this autocrap should never be run in the spkg. The version number as well as the name of the library is also wrong.

Cheers,

Michael

comment:16 Changed 10 years ago by boothby

I've been reading through the megabyte patch, and testing every function that's been touched. I'd estimate my coverage of the file to be about 20%, and I made sure to bounce around a bunch (that is, I didn't just read the top or bottom 20%). I didn't look at many corner cases, because I was trying to test as much as I could, and it often takes thought to come up with good corner cases. In the end, I found some things that Maxima crashed on when it tried to simplify, like

 exp(sum([log(x^(1/n)) for n in range(1,1000)]))

but other than that, I found the entire system to be stable and useful. In a lot of easy cases, it's *much* faster than before, which makes me happy.

comment:17 Changed 10 years ago by mabshoff

I am doing the following

  • install pynac-1.0.17
  • pull in latest changes
  • sage -ba
  • install dsage
  • update pickle jar
  • run testlong

on

  • Linux x86, x86-64 and Itanium
  • some of the above with gcc 4.4
  • Solaris Sparc (32 bit)
  • OSX 10.4 (PPC)

I am currently running testlong, so we should know more in the morning (local time)

Cheers,

Michael

comment:18 Changed 10 years ago by mabshoff

Ok, no joy: On Solaris/Sparc? this patchset causes a hang in maxima.py. Mike told me that William and him allegedly fixed the underlying issue (see the discussion about the semicolon at #6054), but it causes the hang. All Maxima related doctests pass on that machine (modulo two tiny numerical noise problems), so the patch on this ticket is to blame.

Cheers,

Michael

comment:19 Changed 10 years ago by mabshoff

And the failing doctest is this one:

Trying:
    maxima('2+2')###line 751:_sage_    >>> maxima('2+2')
Expecting:
    4

Cheers,

Michael

comment:20 Changed 10 years ago by mabshoff

Hmm, this looks suspicious, but reverting it does not fix the problem:

@@ -755,7 +755,7 @@
         if self._expect is None: return
         r = randrange(2147483647)
         s = marker + str(r+1)
-        cmd = ''';sconcat("%s",(%s+1));\n'''%(marker,r)
+        cmd = '''0;sconcat("%s",(%s+1));\n'''%(marker,r)
         self._sendstr(cmd)
         try:
             self._expect_expr(timeout=0.5)

Running maxima('2+2') in a loop does work, so I am not sure what the problem is yet.

Cheers,

Michael

comment:21 Changed 10 years ago by mabshoff

Hmm, another thing:

 cdef class Matrix_symbolic_dense(matrix_dense.Matrix_dense):
     r"""
@@ -162,7 +161,7 @@
             sage: cmp(m,m)
             0
             sage: cmp(m,3)
-            -1
+            1
         """
         return self._richcmp(right, op)
 

is cmp() in this case deterministic?

Cheers,

Michael

comment:22 Changed 10 years ago by mabshoff

  • Summary changed from [with patch, needs work] switch from maxima to pynac for core symbolic manipulation system to [with patch, positive review] switch from maxima to pynac for core symbolic manipulation system

Formal positive review by various other people not me. Followup should be directed to new tickets for 4.0 as long as it is open.

Cheers,

Michael

comment:23 Changed 10 years ago by mabshoff

  • Resolution set to fixed
  • Status changed from assigned to closed

Merged in Sage 4.0.rc0.

Cheers,

Michael

comment:24 Changed 10 years ago by was

  • Summary changed from [with patch, positive review] switch from maxima to pynac for core symbolic manipulation system to [with patch, positive review] switch from maxima to pynac for core symbolic manipulation system

See #6111 for refereeing of symbolics.

comment:25 Changed 18 months ago by jdemeyer

  • Report Upstream set to N/A

Does anybody happen to remember why there is this strange condition involving inspect.ismethod?

            import inspect
            if not hasattr(_the_element,'_fast_callable_') or not inspect.ismethod(_the_element._fast_callable_):
                # only warn if _the_element is not dynamic
                from sage.misc.superseded import deprecation
                deprecation(5930, "Substitution using function-call syntax and unnamed arguments is deprecated and will be removed from a future release of Sage; you ca

comment:26 Changed 18 months ago by jdemeyer

Never mind, that condition was actually added in #2516.

Note: See TracTickets for help on using tickets.