integration with non symbolic bounds broken

[burcin]

​Reported on sage-support by Victor Miller:

sage: var('a'); function('f',a)
sage: g = f(a).integrate(a,0,a^2)
sage: g
integrate(f(a),0,a^2)

sage: g.derivative(a)

Error in lines 1-1
Traceback (most recent call last):
  File "/mnt/home/lQoU8m2s/.sagemathcloud/sage_server.py", line 498, in execute
    exec compile(block+'\n', '', 'single') in namespace, locals
  File "", line 1, in <module>
  File "expression.pyx", line 3006, in sage.symbolic.expression.Expression.derivative (sage/symbolic/expression.cpp:15855)
  File "derivative.pyx", line 216, in sage.misc.derivative.multi_derivative (sage/misc/derivative.c:2715)
  File "expression.pyx", line 3078, in sage.symbolic.expression.Expression._derivative (sage/symbolic/expression.cpp:16245)
  File "/usr/local/sage/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/symbolic/integration/integral.py", line 224, in _tderivative_
    - f.subs(x==a)*a.diff(diff_param)
  File "element.pyx", line 344, in sage.structure.element.Element.__getattr__ (sage/structure/element.c:3871)
  File "misc.pyx", line 257, in sage.structure.misc.getattr_from_other_class (sage/structure/misc.c:1696)
AttributeError: 'sage.rings.integer.Integer' object has no attribute 'diff'

sage: g1 = f(a).integrate(a)
sage: g1(0)
Error in lines 1-1
Traceback (most recent call last):
  File "/mnt/home/lQoU8m2s/.sagemathcloud/sage_server.py", line 498, in execute
    exec compile(block+'\n', '', 'single') in namespace, locals
  File "", line 1, in <module>
  File "expression.pyx", line 3973, in sage.symbolic.expression.Expression.__call__ (sage/symbolic/expression.cpp:19900)
  File "ring.pyx", line 685, in sage.symbolic.ring.SymbolicRing._call_element_ (sage/symbolic/ring.cpp:7672)
  File "expression.pyx", line 3824, in sage.symbolic.expression.Expression.substitute (sage/symbolic/expression.cpp:19177)
  File "/usr/local/sage/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/symbolic/integration/integral.py", line 75, in _eval_
    if len(x.variables()) == 1:
  File "element.pyx", line 344, in sage.structure.element.Element.__getattr__ (sage/structure/element.c:3871)
  File "misc.pyx", line 257, in sage.structure.misc.getattr_from_other_class (sage/structure/misc.c:1696)
AttributeError: 'sage.rings.integer.Integer' object has no attribute 'variables'

[charpent]

The problem is that (maxima's ?) integrate needs to assert that integration bounds are real: consider this (slightly generalized) example:

Maxima 5.30.0 http://maxima.sourceforge.net
using Lisp GNU Common Lisp (GCL) GCL 2.6.7 (a.k.a. GCL)
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
(%i1) display2d:false;

(%o1) false
(%i2) define(h(x), integrate(f(t), t, g1(x), g2(x)));

defint: lower limit of integration must be real; found g1(x)
 -- an error. To debug this try: debugmode(true);
(%i3) declare(g1, real, g2, real);

(%o3) done
(%i4) define(h(x), integrate(f(t), t, g1(x), g2(x)));

(%o4) h(x):='integrate(f(t),t,g1(x),g2(x))
(%i5) diff(h(x), x);

(%o5) f(g2(x))*'diff(g2(x),x,1)-f(g1(x))*'diff(g1(x),x,1)
(%i6) 

Maxima allows for declaring "after the fact" that g1 and g2 are sympbols for real values, and extends this declaration to the case where g1 and 2 are symbols for functions, which are now interpreted as real-valued functions.

As far as I know, there is no way to declare the domain of a function in Sage.

And, since the Sage-to-Maxima interface uses arbitrary (and volatile !) symbols, one cannot simply use "maxima('declare(g1, real, g2, real);')" to this effect...

sage: var("x,t")
(x, t)
sage: f=function("f",t)
sage: g1=function("g1",t)
sage: g2=function("g2",t)
sage: maxima("declare(g1,real,g2,real);")
done
sage: h(x)=integrate(f(t),t,g1(x),g2(x))
/usr/local/sage-5.10/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py:2721: DeprecationWarning: Substitution using function-call syntax and unnamed arguments is deprecated and will be removed from a future release of Sage; you can use named arguments instead, like EXPR(x=..., y=...)
See http://trac.sagemath.org/5930 for details.
  exec code_obj in self.user_global_ns, self.user_ns
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-7dd129794e9a> in <module>()
----> 1 __tmp__=var("x"); h = symbolic_expression(integrate(f(t),t,g1(x),g2(x))).function(x)

/usr/local/sage-5.10/local/lib/python2.7/site-packages/sage/misc/functional.pyc in integral(x, *args, **kwds)
    738     """
    739     if hasattr(x, 'integral'):
--> 740         return x.integral(*args, **kwds)
    741     else:
    742         from sage.symbolic.ring import SR

/usr/local/sage-5.10/local/lib/python2.7/site-packages/sage/symbolic/expression.so in sage.symbolic.expression.Expression.integral (sage/symbolic/expression.cpp:39592)()

/usr/local/sage-5.10/local/lib/python2.7/site-packages/sage/symbolic/integration/integral.pyc in integrate(expression, v, a, b, algorithm)
    686         return indefinite_integral(expression, v)
    687     else:
--> 688         return definite_integral(expression, v, a, b)
    689 
    690 integral= integrate

/usr/local/sage-5.10/local/lib/python2.7/site-packages/sage/symbolic/function.so in sage.symbolic.function.Function.__call__ (sage/symbolic/function.cpp:5114)()

/usr/local/sage-5.10/local/lib/python2.7/site-packages/sage/symbolic/integration/integral.pyc in _eval_(self, f, x, a, b)
    171         for integrator in self.integrators:
    172             try:
--> 173                 return integrator(*args)
    174             except NotImplementedError:
    175                 pass

/usr/local/sage-5.10/local/lib/python2.7/site-packages/sage/symbolic/integration/external.pyc in maxima_integrator(expression, v, a, b)
     19         result = maxima.sr_integral(expression,v)
     20     else:
---> 21         result = maxima.sr_integral(expression, v, a, b)
     22     return result._sage_()
     23 

/usr/local/sage-5.10/local/lib/python2.7/site-packages/sage/interfaces/maxima_lib.pyc in sr_integral(self, *args)
    745                 raise ValueError, "Computation failed since Maxima requested additional constraints; using the 'assume' command before integral evaluation *may* help (example of legal syntax is 'assume(" + s[4:k] +">0)', see `assume?` for more details)\n" + s
    746             else:
--> 747                 raise error
    748 
    749     def sr_sum(self,*args):

RuntimeError: ECL says: Error executing code in Maxima: defint: lower limit of integration must be real; found g1(x)

Maxima won't, however, accept to declare a function as real :

(%i6) declare(s(x), real);

declare: improper argument: s(x)
 -- an error. To debug this try: debugmode(true);

There is an *horrible* workaround : create *variables*, g1 and g2, assume them real, then create functions f, g1 and g2:

sage: var("x, t, g1, g2")
(x, t, g1, g2)
sage: assume(g1, "real", g2, "real")
sage: f=function("f",t)
sage: g1=function("g1",x)
sage: g2=function("g2",x)
sage: h(x)=integrate(f(t), t,g1(x), g2(x))
sage: h(x)
integrate(f(t), t, g1(x), g2(x))
sage: diff(h(x),x)
-f(g1(x))*D[0](g1)(x) + f(g2(x))*D[0](g2)(x)
sage: 

The core of the problem is, however, the inability to declare a function as real. Maxima's declaration of a *symbol's* domain "happens to work", but a cleaner solution is needed. I am afraid that it involves work both on Sage *and* Maxima.

HTH,

Emmanuel Charpentier

[nbruin]

Looking at the returned error, I do not get the impression this has anything to do with maxima or with checking whether something is real-valued. The error we're getting is that attributes like variables and derivative end up being looked up on sage integer objects rather than on SR elements. This must happen somewhere during the picking apart of the expression:

sage: g
integrate(f(a), a, 0, a^2)
sage: [type(o) for o in g.operands()]
[sage.symbolic.expression.Expression,
 sage.symbolic.expression.Expression,
 sage.symbolic.expression.Expression,
 sage.symbolic.expression.Expression]
sage: g.operands()[2].diff(a)
0

as you see, all quantities involved are symbolic expressions and this "symbolic constant 0" has no problem being differentiated.

sage: g.derivative(a)
AttributeError
sage: %debug
ipdb> up
> /usr/local/sage/5.7/local/lib/python2.7/site-packages/sage/symbolic/integration/integral.py(224)_tderivative_()
    223         return ans + f.subs(x==b)*b.diff(diff_param) \
--> 224                     - f.subs(x==a)*a.diff(diff_param)
    225 
ipdb> p [(c,type(c)) for c in [f,x,a,b,diff_param]]
[(f(a), <type 'sage.symbolic.expression.Expression'>), (a, <type 'sage.symbolic.expression.Expression'>), (0, <type 'sage.rings.integer.Integer'>), (a^2, <type 'sage.symbolic.expression.Expression'>), (a, <type 'sage.symbolic.expression.Expression'>)]

I suspect that this lower bound a=0 (here a is the local variable in _tderivative) is coming from the lower integration bound involved in the definition of g, and apparently this bound got stripped out of SR a little prematurely for this purpose. Probably replacing the code above with

        return ans + f.subs(x==b)*SR(b).diff(diff_param) \
                    - f.subs(x==a)*SR(a).diff(diff_param)

would solve the problem. However it might be worthwhile to look why a got stripped out of SR in the first place and whether that should simply be prevented at the spot (so that _tderivative gets called with symbolic a,b regardless of whether they happen to be integer constants)

[charpent]

I disagree. See below :

Replying to nbruin:

Looking at the returned error, I do not get the impression this has anything to do with maxima or with checking whether something is real-valued. The error we're getting is that attributes like variables and derivative end up being looked up on sage integer objects rather than on SR elements. This must happen somewhere during the picking apart of the expression:

sage: g
integrate(f(a), a, 0, a^2)
sage: [type(o) for o in g.operands()]
[sage.symbolic.expression.Expression,
 sage.symbolic.expression.Expression,
 sage.symbolic.expression.Expression,
 sage.symbolic.expression.Expression]
sage: g.operands()[2].diff(a)
0

as you see, all quantities involved are symbolic expressions and this "symbolic constant 0" has no problem being differentiated.

sage: g.derivative(a)
AttributeError
sage: %debug
ipdb> up
> /usr/local/sage/5.7/local/lib/python2.7/site-packages/sage/symbolic/integration/integral.py(224)_tderivative_()
    223         return ans + f.subs(x==b)*b.diff(diff_param) \
--> 224                     - f.subs(x==a)*a.diff(diff_param)
    225 
ipdb> p [(c,type(c)) for c in [f,x,a,b,diff_param]]
[(f(a), <type 'sage.symbolic.expression.Expression'>), (a, <type 'sage.symbolic.expression.Expression'>), (0, <type 'sage.rings.integer.Integer'>), (a^2, <type 'sage.symbolic.expression.Expression'>), (a, <type 'sage.symbolic.expression.Expression'>)]

I suspect that this lower bound a=0 (here a is the local variable in _tderivative) is coming from the lower integration bound involved in the definition of g, and apparently this bound got stripped out of SR a little prematurely for this purpose.

So ? If this was tre source of the error, it shouldn't happen with symbolic lower and upper bounds (see my slightly generalized example), but it does. Furrthermore, declaring the bounds as real allows for completing the task (bothh from maxima and from sage)...

Probably replacing the code above with

        return ans + f.subs(x==b)*SR(b).diff(diff_param) \
                    - f.subs(x==a)*SR(a).diff(diff_param)

would solve the problem. However it might be worthwhile to look why a got stripped out of SR in the first place and whether that should simply be prevented at the spot (so that _tderivative gets called with symbolic a,b regardless of whether they happen to be integer constants)

[nbruin]

Replying to charpent:

So ? If this was tre source of the error, it shouldn't happen with symbolic lower and upper bounds (see my slightly generalized example), but it does. Furrthermore, declaring the bounds as real allows for completing the task (bothh from maxima and from sage)...

It's the source of the errors reported in the ticket. The problems reported in the ticket arise without interaction with maxima.

You're diagnosing a different problem, that integration with certain symbolic bounds is also broken. Compare:

sage: var('x,t,a,b')
(x, t, a, b)
sage: function('f')
f
sage: function('g')
g
sage: integrate(f(x),x,a,b)
integrate(f(x), x, a, b)
sage: integrate(f(x),x,sin(t),b)
integrate(f(x), x, sin(t), b)
sage: integrate(f(x),x,sin(1+i*t),b)
RuntimeError: ECL says: Error executing code in Maxima: defint: lower limit of integration must be real; found sin(%i*t+1)
sage: integrate(f(x),x,g(t),b)
RuntimeError: ECL says: Error executing code in Maxima: defint: lower limit of integration must be real; found g(t)

Indeed, for definite integrals, maxima seems to want to know if the integration bounds are real. However, Maxima seems to assume quite happily by itself that a,b,sin(t) are real, but doesn't think sin(1+i*t) is real and for some reason refuses to assume anything about g(t).

Your workaround is interesting in that it shows that maxima's "assume" facility can be of some help to nudge maxima into the desired direction. It seems to me the problem you're diagnosing is best addressed by working mainly on maxima. As far as I've seen, sage is offering reasonable input to maxima.

[charpent]

Replying to nbruin:

Replying to charpent :

So ? If this was tre source of the error, it shouldn't happen with symbolic lower and upper bounds (see my slightly generalized example), but it does. Furrthermore, declaring the bounds as real allows for completing the task (bothh from maxima and from sage)...

It's the source of the errors reported in the ticket. The problems reported in the ticket arise without interaction with maxima.

You might be right. I tend to think too much like a physician and tend to attach all symptoms to the same cause (it is rare that a patient has symptoms caused by two different diseases with onset at the same time...).

You're diagnosing a different problem, that integration with certain symbolic bounds is also broken. Compare: sage: var('x,t,a,b') (x, t, a, b) sage: function('f') f sage: function('g') g sage: integrate(f(x),x,a,b) integrate(f(x), x, a, b) sage: integrate(f(x),x,sin(t),b) integrate(f(x), x, sin(t), b) sage: integrate(f(x),x,sin(1+i*t),b) RuntimeError : ECL says: Error executing code in Maxima: defint: lower limit of integration must be real; found sin(%i*t+1) sage: integrate(f(x),x,g(t),b) RuntimeError : ECL says: Error executing code in Maxima: defint: lower limit of integration must be real; found g(t)Indeed, for definite integrals, maxima seems to want to know if the integration bounds are real. However, Maxima seems to assume quite happily by itself that a,b,sin(t) are real, but doesn't think sin(1+i*t) is real and for some reason refuses to assume anything about g(t) . Your workaround is interesting in that it shows that maxima's "assume" facility can be of some help to nudge maxima into the desired direction. It seems to me the problem you're diagnosing is best addressed by working mainly on maxima. As far as I've seen, sage is offering reasonable input to maxima.

Hmmm... Maxima's "assume" also has serious limitations. Many questions Maxima may ask during a computation cannot be prevented by previous assumptions, since those cannot use expressions. For example, during an (unrelated) integration, maxima asked "Is (m/s) an integer ?". I checked that maxima does *not* allow for "assume(m/s, noninteger);".

Furthermore, sage noes not (currently) allows for interaction during such a computation. Instead, it spits out an error suggesting to use assume...

So we have *two* limitations : Maxima's assumption system, which is indeed maxima-specific, *AND* sage's non-use of interactions.

IIRC, Mathematica, when confronted with such a question, does not interact with user but tries to generate a list (a tree ?) of possible questions and gives a list of answers along a set of conditions. Of course, there is no guarantee that such a tree is finite...

Any thoughts ? I reported the gist of the problem to Maxima's mailing list. Should I file a bug abainst Maxima's assume ?

[nbruin]

Replying to charpent:

Hmmm... Maxima's "assume" also has serious limitations. Many questions Maxima may ask during a computation cannot be prevented by previous assumptions, since those cannot use expressions. For example, during an (unrelated) integration, maxima asked "Is (m/s) an integer ?". I checked that maxima does *not* allow for "assume(m/s, noninteger);".

It's known and widely acknowledged that maxima's assume facility is rather weak and not very well integrated (many functions that could benefit from assumptions do not look at them). If you search you'll find many threads about it.

There also have been various attempts in changing maxima's tendency to ask questions (but never in maxima proper, as far as I know. It's simply taken as a design decision there).

For sage's purposes, our only reasonable option is to treat a question as an error condition, because we simply cannot assume there is anyone to answer the question. That's at least what people have found up to now. If you have a viable prototype for handling questions otherwise, it can always be considered, of course.

Any thoughts ? I reported the gist of the problem to Maxima's mailing list. Should I file a bug abainst Maxima's assume ?

Mentioning the difference between the handling of

integrate(f(x),x,a,b)

and

integrate(f(x),x,a(t),b(t))

is probably worthwhile. For the rest, you're just pointing out well-known shortcomings.

[rws]

Replying to charpent:

The unnecessary interaction with the user is now treated in #16653. Now back to the originally reported error which can be presumably fixed in Sage.

[rws]

Replying to nbruin:

You're diagnosing a different problem, that integration with certain symbolic bounds is also broken. Compare:

sage: var('x,t,a,b')
(x, t, a, b)
sage: function('f')
f
sage: function('g')
g
sage: integrate(f(x),x,a,b)
integrate(f(x), x, a, b)
sage: integrate(f(x),x,sin(t),b)
integrate(f(x), x, sin(t), b)
sage: integrate(f(x),x,sin(1+i*t),b)
RuntimeError: ECL says: Error executing code in Maxima: defint: lower limit of integration must be real; found sin(%i*t+1)
sage: integrate(f(x),x,g(t),b)
RuntimeError: ECL says: Error executing code in Maxima: defint: lower limit of integration must be real; found g(t)

This no longer gives an error.