Upgrade Maxima to 5.33.0

[dimpase]

This is a continuation of #13364, and aims at including more upstream bug fixes, which e.g. fix an ​issue reported on sage-support, which was reported as Maxima bug ​2535, and marked there as closed in post-5.29.1.

Fixes #11894, #13526, #13712, #14209, #14306, #15386 and possibly other Maxima-related bugs.

Ought to fix #13733 (but loading the abs_integrate package currently causes this to fail).

Ought to fix #8728 (but Maxima asks for more constraints which follow from the existing ones).

Does not fix #14591, #14677, #15033.

tarball: ​http://homepages.warwick.ac.uk/staff/P.Bruin/sage/maxima-5.33.0.tar.bz2

Note about the existing maxima_bug_2526.patch: ​the bug was marked "closed" in the ​Maxima bug tracking system, but the commit somehow disappeared from the Maxima repository and the bug has been reopened. Hence this patch is still needed to fix it.

[kcrisman]

So for instance going at least as far as the fix for #13733 would be good.

[zimmerma]

while we are at it, it seems strange to me that sage/interfaces/maxima_abstract.py contains a doctest for the maxima version used:

File "/localdisk/tmp/sage-5.5/devel/sage-12615b/sage/interfaces/maxima_abstract.py", line 422:
    sage: maxima.version()
Expected:
    '5.26.0'
Got:
    '5.29.1'

What really matters is the functionality provided, not the actual version? Does Sage check the actual version of each component?

Paul

[dimpase]

Replying to zimmerma:

while we are at it, it seems strange to me that sage/interfaces/maxima_abstract.py contains a doctest for the maxima version used:

File "/localdisk/tmp/sage-5.5/devel/sage-12615b/sage/interfaces/maxima_abstract.py", line 422:
    sage: maxima.version()
Expected:
    '5.26.0'
Got:
    '5.29.1'

seems that ​http://trac.sagemath.org/sage_trac/attachment/ticket/13364/doctests.2.patch was not installed into the Sage installation you work with.

What really matters is the functionality provided, not the actual version? Does Sage check the actual version of each component?

Sage provides versions at least for some other components, e.g. gap.version()

Sometimes it might be useful; AFAIK some optional packages check this.

Dima

Paul

[zimmerma]

Dima, I don't object about providing a function maxima_version(), but checking it returns a particular value.

Paul

[dimpase]

Replying to zimmerma:

Dima, I don't object about providing a function maxima_version(), but checking it returns a particular value.

well, one would like to test Sage functions... How else would you check that it works and returns something meaningful?

Paul

[zimmerma]

we could for example check that the version is of the form x.y.z, or is greater than or equal to a given number.

Paul

[zimmerma]

see #14306 (issue fixed in Maxima git)

Paul

[kcrisman]

Also note that Maxima is now at 5.30.0.

[fbissey]

Yes it is at 5.30.0. No check from sage-on-gentoo on whether it works so far because the maxima ecl library building is currently broken for that particular version - an adsf upgrade problem in gentoo.

I second checking the form of the version returned rather than the value. That would make my life easier when R, maxima are upgraded in s-o-g independently of sage itself (and without side effects at leat in doctests).

[jdemeyer]

I'll try making a new Maxima spkg just to see what happpens...

[jdemeyer]

This looks problematic:

sage -t devel/sage/sage/calculus/calculus.py
**********************************************************************
File "devel/sage/sage/calculus/calculus.py", line 145, in sage.calculus.calculus
Failed example:
    e^M
Exception raised:
    Traceback (most recent call last):
      File "/mazur/release/merger/sage-5.10.rc2/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 466, in _run
        self.execute(example, compiled, test.globs)
      File "/mazur/release/merger/sage-5.10.rc2/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 825, in execute
        exec compiled in globs
      File "<doctest sage.calculus.calculus[32]>", line 1, in <module>
        e**M
      File "constants_c.pyx", line 275, in sage.symbolic.constants_c.E.__pow__ (sage/symbolic/constants_c.cpp:2779)
      File "matrix_symbolic_dense.pyx", line 387, in sage.matrix.matrix_symbolic_dense.Matrix_symbolic_dense.exp (sage/matrix/matrix_symbolic_dense.c:3433)
      File "/mazur/release/merger/sage-5.10.rc2/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
        return cls(self, x, name=name)
      File "/mazur/release/merger/sage-5.10.rc2/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 626, in __init__
        raise TypeError, x
    TypeError: ECL says: Error executing code in Maxima: Unable to find the spectral representation
**********************************************************************

[jdemeyer]

Also

sage -t devel/sage/sage/interfaces/maxima.py  # Time out

due to a timeout of

sage: maxima('limit(x^a,x,0)')

This is fixable, it is because of a slightly different output format of assumptions.

[kcrisman]

That is awesome, thanks for checking this, Jeroen.

[kcrisman]

This is fixable, it is because of a slightly different output format of assumptions.

Jeroen, while it's true that the error is slightly different in format, I don't think this is the problem (or at least not the whole one). The other things are still passing doctests, and we never even get to the

           if i > 0:
                v = self._expect.before

branch, as one can verify by inserting print statements.

This looks problematic:

This is #2400 again, to some extent. Not sure why this doesn't work any more. In Maxima 5.29.1, this is fine.

Maxima 5.29.1 http://maxima.sourceforge.net
using Lisp ECL 12.12.1
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
(%i1) m: matrix([%i*%pi]);
(%o1)                             [ %i %pi ]
(%i4) matrixexp(m), keepfloat:true;
(%o4)                                 - 1
(%i5) matrixexp(m),keepfloat:false;
(%o5)                                 - 1

which is good. In 5.30, though, we have

(%i1) m: matrix([%i*%pi]);
(%o1)                             [ %i %pi ]
(%i6) matrixexp(m),keepfloat:true;

Unable to find the spectral representation
 -- an error. To debug this try: debugmode(true);
(%i7) matrixexp(m),keepfloat:false;

Unable to find the spectral representation
 -- an error. To debug this try: debugmode(true);

[kcrisman]

Note that the spkg changes are not committed.

[kcrisman]

Thanks to Barton Willis for this workaround.

This bug is due to circa January 2013 changes to matrix inversion. A workaround is to set ratmx to true:

Maxima branch_5_30_base_98_g29f9239_dirty http://maxima.sourceforge.net
using Lisp Clozure Common Lisp Version 1.9-r15764  (WindowsX8632)

(%i1) matrixexp(matrix([%i*%pi]));

Unable to find the spectral representation

(%i2) matrixexp(matrix([%i*%pi])), ratmx=true;
(%o2)                                 - 1

I've reported this ​here. I don't know if doing this ratmx thing universally is good, though:

 if FALSE will cause determinant and matrix addition, subtraction, and multiplication to be performed in the representation of the matrix elements and will cause the result of matrix inversion to be left in general representation. If it is TRUE, the 4 operations mentioned above will be performed in CRE form and the result of matrix inverse will be in CRE form. Note that this may cause the elements to be expanded (depending on the setting of RATFAC) which might not always be desired.

[kcrisman]

5.31 has been released.

[pbruin]

Maxima is now at version 5.33.0. The bug described in comments 15, 19 and 22 still exists.

[pbruin]

I found a patch that I think has the same effect as the workaround in comment:22. I don't have a sourceforge account, so here it is in case there are any Maxima experts here who want to check it and (if it is any good) maybe forward it to the Maxima bug tracking system:

share/linearalgebra/matrixexp.lisp

@@ -138 +138 @@
           (print `(ratvars = ,$ratvars gcd = '$gcd algebraic = ,$algebraic))
           (print `(ratfac = ,$ratfac))
           (merror "Unable to find the spectrum")))
-   
-    (setq res ($fullratsimp (ncpower (sub (mult z ($ident n)) mat) -1) z))
+
+    (setq res ($fullratsimp ($invert (sub (mult z ($ident n)) mat) '$crering) z))
     (setq m (length sp))
     (dotimes (i m)
       (setq zi (nth i sp))

I'll try to see if upgrading to 5.33.0 gives any further unexpected problems.

[kcrisman]

I found a patch that I think has the same effect as the workaround in comment:22. I don't have a sourceforge account, so here it is in case there are any Maxima experts here who want to check it and (if it is any good) maybe forward it to the Maxima bug tracking system:

I have added it to the ​bug report in question.

I'll try to see if upgrading to 5.33.0 gives any further unexpected problems.

Let's hope not!

[pbruin]

Thanks! No major problems so far; there is one integration doctest that does not finish, but Maxima itself returns the answer quickly, so the timeout has to occur in the interface.

[pbruin]

The problematic doctest is the following one from sage/misc/functional.py:

sage: f = exp(-x) * sinh(sqrt(x))
sage: t = integrate(f, x, 0, Infinity)

With an unpatched Sage 6.2 this does finish, but takes a very long time (2-3 minutes). The same situation occurs for

sage: maxima('integrate(exp(-x)*sinh(sqrt(x)), x, 0, inf);')

This is surprising given that the following takes less than a second in Maxima (both the one that comes with Sage and a standalone copy, and both versions 5.29.1 and 5.33.0):

(%i1) integrate(exp(-x)*sinh(sqrt(x)), x, 0, inf); 
(%o1) 
    1/4                                              1
  %e    (2 gamma_incomplete(1, 1) - gamma_incomplete(-, 1) - sqrt(%pi) - 2)
                                                     2
- -------------------------------------------------------------------------
                                      4
                                    1/4                  1
     1/4                          %e    gamma_incomplete(-, 1)
   %e    gamma_incomplete(1, 1)                          2
 + ---------------------------- - ----------------------------
                2                              4
     1/4               1/4
   %e    sqrt(%pi)   %e
 + --------------- - -----
          4            2

It would seem to be an interface problem that already exists in some form in Sage 6.2 and manifests itself more relentlessly with the Maxima 5.33.0 spkg I am making.

[dimpase]

Replying to pbruin:

The problematic doctest is the following one from sage/misc/functional.py:

sage: f = exp(-x) * sinh(sqrt(x))
sage: t = integrate(f, x, 0, Infinity)

With an unpatched Sage 6.2 this does finish, but takes a very long time (2-3 minutes). The same situation occurs for

sage: maxima('integrate(exp(-x)*sinh(sqrt(x)), x, 0, inf);')

This is surprising given that the following takes less than a second in Maxima (both the one that comes with Sage and a standalone copy, and both versions 5.29.1 and 5.33.0):

I am not at all convinced it is the interface problem. It is more likely the Maxima packages loaded by default in Sage library (see the corr. Sage source files) and not loaded in your test run. When I only load(abs_integrate) i Sage's 6.2 Maxima, or don't load anything at all, I get an almost instant answer. When I load the whole bunch from lines 142-4 of sage/interfaces/maxima_lib.py

init_code = ['display2d : false', 'domain : complex', 'keepfloat : true',
            'load(to_poly_solve)', 'load(simplify_sum)',
            'load(abs_integrate)']

it goes much slower.

[pbruin]

Thanks! In fact it seems to be the domain : complex setting that is causing the slowness.

[pbruin]

How about the minimal workaround of temporarily setting domain: real in this doctest?

[dimpase]

Replying to pbruin:

How about the minimal workaround of temporarily setting domain: real in this doctest?

sure, this is not the 1st time something like this is needed.

[pbruin]

Note about the patches: maxima_bug_2526.patch, which fixes ​http://sourceforge.net/p/maxima/bugs/2526/, is still necessary even though the bug is marked "closed" in the Maxima bug tracking system; the commit somehow didn't make it into the Maxima repository.

[mmezzarobba]

sage -t --long src/sage/interfaces/maxima_lib.py
**********************************************************************
File "src/sage/interfaces/maxima_lib.py", line 702, in sage.interfaces.maxima_lib.MaximaLib.sr_integral
Failed example:
    integrate(cos(x + abs(x)), x)
Expected:
    -1/4*(2*x - sin(2*x))*realpart(sgn(x)) + 1/2*x + 1/4*sin(2*x)
Got:
    -1/4*(2*x - sin(2*x))*real_part(sgn(x)) + 1/2*x + 1/4*sin(2*x)
**********************************************************************

This is with Volker's current develop branch on github (d409685), which includes a few dozens new tickets compared to 6.3.beta1.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a130eed

Trac 13973: realpart should be real_part

[pbruin]

Thanks, that must be a consequence of #16224.

[kcrisman]

Thanks, that must be a consequence of #16224.

So... did we have an incorrect translation of Maxima's realpart all along? Interesting.

[mmezzarobba]

Is there a reason for no longer using z... in the following test?

@@ -9231,7 +9231,7 @@ cdef class Expression(CommutativeRingElement):
             sage: from sage.calculus.calculus import maxima
             sage: sol = maxima(cos(x)==0).to_poly_solve(x)
             sage: sol.sage()
-            [[x == -1/2*pi + 2*pi*z...], [x == 1/2*pi + 2*pi*z...]]
+            [[x == 1/2*pi + pi*z82]]
 
         If a returned unsolved expression has a denominator, but the
         original one did not, this may also be true::

The same comment may apply to z[0-9]+ in

src/sage/interfaces/maxima_lib.py:            [[x == pi*z54]]
src/sage/symbolic/relation.py:        [[x == 1/4*pi + pi*z80, y == -1/4*pi - pi*z80]]

The bugs listed in the description indeed appear to be fixed with the update, except #13733, for which I get:

sage: integral(log(cot(x)-1),x,0,pi/4)
...
RuntimeError: ECL says: Error executing code in Maxima: PSLOG: internal error.

Otherwise looks good to me.

[pbruin]

Replying to mmezzarobba:

Is there a reason for no longer using z... in the following test?

Mostly readability (since the tests are also part of the documentation); with the ... it is not clear that it is just a "serial number" and not part of a larger expression that has been left out. Of course it is somewhat random, so it may change once in a while with Maxima upgrades or changes in other doctests in the same file. I personally think displaying a complete answer is worth the trouble of having to change it every now and then, but if you insist I can change all such examples to ....

The bugs listed in the description indeed appear to be fixed with the update, except #13733, for which I get:

sage: integral(log(cot(x)-1),x,0,pi/4)
...
RuntimeError: ECL says: Error executing code in Maxima: PSLOG: internal error.

You're right; I tested this inside Maxima (sage --maxima), but Sage loads additional packages. In this case the failure is caused by loading abs_integrate. Using 5.33.0:

$ sage --maxima
...
(%i1) display2d: false$

(%i2) integrate(log(cot(x)-1),x,0,%pi/4);

(%o2) (%i*li[2]((%i+1)/2)-%i*li[2](-(%i-1)/2))/2
 -(%i*(2*li[2](%i+1)-2*li[2](1-%i))+%pi*log(2))/4
(%i3) load(abs_integrate)$

(%i4) integrate(log(cot(x)-1),x,0,%pi/4);

log: encountered log(0).
log: encountered log(0).
log: encountered log(0).
log: encountered log(0).
log: encountered log(0).
log: encountered log(0).
log: encountered log(0).
log: encountered log(0).
log: encountered log(0).
PSLOG: internal error.
 -- an error. To debug this try: debugmode(true);

[mmezzarobba]

Replying to pbruin:

Replying to mmezzarobba:

Is there a reason for no longer using z... in the following test?

Mostly readability (since the tests are also part of the documentation); with the ... it is not clear that it is just a "serial number" and not part of a larger expression that has been left out. Of course it is somewhat random, so it may change once in a while with Maxima upgrades or changes in other doctests in the same file. I personally think displaying a complete answer is worth the trouble of having to change it every now and then, but if you insist I can change all such examples to ....

No, that's fine with me, I just wanted to make sure that it was intentional.

[kcrisman]

Note about the patches: maxima_bug_2526.patch, which fixes ​http://sourceforge.net/p/maxima/bugs/2526/, is still necessary even though the bug is marked "closed" in the Maxima bug tracking system; the commit somehow didn't make it into the Maxima repository.

Okay, this is ​finally in Maxima, so the next Maxima upgrade will get to remove this patch.

[kcrisman]

#14306 needs review.