Opened 3 years ago
Closed 3 years ago
#28964 closed defect (fixed)
sympy: Problem with integration/differentiation of symbolic functions.
Reported by:  charpent  Owned by:  

Priority:  critical  Milestone:  sage9.1 
Component:  symbolics  Keywords:  integrate, integral, sympy 
Cc:  Merged in:  
Authors:  Markus Wageringel  Reviewers:  Vincent Delecroix 
Report Upstream:  N/A  Work issues:  
Branch:  57a8e83 (Commits, GitHub, GitLab)  Commit:  57a8e83d6e1f1b6a667b444817f1314effb0f95e 
Dependencies:  Stopgaps: 
Description (last modified by )
This ticket fixes several problems related to the conversion of derivatives to and from sympy:
 the derivative arguments were permuted when converting to sympy (comment:7)
 the conversion from Sage to sympy did not account for the case of variables with multiple occurences (comment:1)
 the first argument of the derivative was skipped when converting from sympy (comment:6)
 sympy returns arguments of type
sympy.core.containers.Tuple
which was not handled  Sage's derivative does not accept tuples, so the arguments must be flattened
 sympy's integer type is converted to Sage's rational type, but should be converted to Sage's integer type
(Original description below.)
Minimal case of a problem reported on ask.sagemath.org.
sage: reset() sage: var("x,t") (x, t) sage: f=function("f") ## Note : no formal arguments sage: diff(f(x,t),x) diff(f(x, t), x) sage: diff(f(x,t),x).integrate(x) f(x, t)
As expected. But :
sage: diff(f(x,t),x).integrate(t) f(x, t)
No, no, no. And no.
Slightly better:
sage: reset() sage: var("x,t") (x, t) sage: f=function("f")(x,t) ## Note : specified formal arguments. sage: diff(f(x,t),x) diff(f(t, x), x) sage: diff(f(x,t),x).integrate(x) f(t, x) sage: diff(f(x,t),x).integrate(t)  ValueError Traceback (most recent call last) /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/all_cmdline.py in <module>() > 1 diff(f(x,t),x).integrate(t) /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/symbolic/expression.pyx in sage.symbolic.expression.Expression.integral (build/cythonized/sage/symbolic/expression.cpp:64575)() 12389 R = ring.SR 12390 return R(integral(f, v, a, b, **kwds)) > 12391 return integral(self, *args, **kwds) 12392 12393 integrate = integral /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/symbolic/integration/integral.py in integrate(expression, v, a, b, algorithm, hold) 927 return integrator(expression, v, a, b) 928 if a is None: > 929 return indefinite_integral(expression, v, hold=hold) 930 else: 931 return definite_integral(expression, v, a, b, hold=hold) /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/symbolic/function.pyx in sage.symbolic.function.BuiltinFunction.__call__ (build/cythonized/sage/symbolic/function.cpp:12262)() 1136 res = self._evalf_try_(*args) 1137 if res is None: > 1138 res = super(BuiltinFunction, self).__call__( 1139 *args, coerce=coerce, hold=hold) 1140 /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/symbolic/function.pyx in sage.symbolic.function.Function.__call__ (build/cythonized/sage/symbolic/function.cpp:7039)() 600 (<Expression>args[0])._gobj, hold) 601 elif self._nargs == 2: > 602 res = g_function_eval2(self._serial, (<Expression>args[0])._gobj, 603 (<Expression>args[1])._gobj, hold) 604 elif self._nargs == 3: /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/symbolic/integration/integral.py in _eval_(self, f, x) 100 for integrator in self.integrators: 101 try: > 102 A = integrator(f, x) 103 except (NotImplementedError, TypeError, AttributeError): 104 pass /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/symbolic/integration/external.py in sympy_integrator(expression, v, a, b) 66 else: 67 result = sympy.integrate(ex, (v, a._sympy_(), b._sympy_())) > 68 return result._sage_() 69 70 def mma_free_integrator(expression, v, a=None, b=None): /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/interfaces/sympy.py in _sympysage_integral(self) 307 """ 308 from sage.misc.functional import integral > 309 f, limits = self.function._sage_(), list(self.limits) 310 for limit in limits: 311 if len(limit) == 1: /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/interfaces/sympy.py in _sympysage_derivative(self) 333 f = self.args[0]._sage_() 334 args = [[a._sage_() for a in arg] if isinstance(arg,tuple) else arg._sage_() for arg in self.args[2:]] > 335 return derivative(f, *args) 336 337 def _sympysage_order(self): /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/calculus/functional.py in derivative(f, *args, **kwds) 129 """ 130 try: > 131 return f.derivative(*args, **kwds) 132 except AttributeError: 133 pass /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/symbolic/expression.pyx in sage.symbolic.expression.Expression.derivative (build/cythonized/sage/symbolic/expression.cpp:25806)() 4186 ValueError: No differentiation variable specified. 4187 """ > 4188 return multi_derivative(self, args) 4189 4190 diff = differentiate = derivative /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/misc/derivative.pyx in sage.misc.derivative.multi_derivative (build/cythonized/sage/misc/derivative.c:3014)() 217 if not args: 218 # fast version where no arguments supplied > 219 return F._derivative() 220 221 for arg in derivative_parse(args): /usr/local/sageP32/local/lib/python3.7/sitepackages/sage/symbolic/expression.pyx in sage.symbolic.expression.Expression._derivative (build/cythonized/sage/symbolic/expression.cpp:26148)() 4248 return self.gradient() 4249 else: > 4250 raise ValueError("No differentiation variable specified.") 4251 if not isinstance(deg, (int, long, sage.rings.integer.Integer)) \ 4252 or deg < 1: ValueError: No differentiation variable specified.
No again. But at least, this is an explicit error, not a silently accepted wrong result.
The problem seems to point to handling formal arguments of undefined functions. I do not (yet) understand this code...
IMHO, at least the first case is critical (possibly blocker), because it silently returns a wrong result.
Note that the handling of defined functions seems to be correct. For example:
sage: sin(t*x).diff(x) t*cos(t*x) sage: sin(t*x).diff(x).integrate(x) sin(t*x) sage: sin(t*x).diff(x).integrate(t) (t*x*sin(t*x) + cos(t*x))/x^2
Change History (15)
comment:2 followup: 3 Changed 3 years ago by
Replying to nbruin:
[ Snip... ]
we can make that cause an error by fiddling with some more variables and then we see that the error traces back in something being called from
_sympysage_integral()
, so I think this error was introduced relatively recently, when sympy was added as an integrator by default.
According to a comment posted around Jan 7, 2020 07:00 by dsejas in the original ask.sagemath question, the problem might have been introduced between 8.8 and 8.9.
It should be possible to git bisect
this range to pinpoint the commit. I don't have the necessary resources on hand...
comment:3 Changed 3 years ago by
Replying to charpent:
we can make that cause an error by fiddling with some more variables and then we see that the error traces back in something being called from
_sympysage_integral()
, so I think this error was introduced relatively recently, when sympy was added as an integrator by default.According to a comment posted around Jan 7, 2020 07:00 by dsejas in the original ask.sagemath question, the problem might have been introduced between 8.8 and 8.9.
The culprit might be #27958.
comment:4 Changed 3 years ago by
Keywords:  integrate integral added 

comment:5 Changed 3 years ago by
I am the original poster in ask.sagemath.org for this issue. Hope I am not out of line to add a test case because I hope the eventual fix will solve my original problem:
var('x t') f = function('f')(x, t) g = function('g')(x, t) integrate(g * diff(f, x), x) ... ValueError: No differentiation variable specified.
Used to work in Sage 8.8. Not working in 8.9 or 9.0.
Thanks a lot.
comment:6 followup: 7 Changed 3 years ago by
Translation of derivatives to sympy is definitely wonky:
sage: f=function('f') sage: F=f(x,y,t).diff(t) sage: F diff(f(x, y, t), t) sage: F._sympy_() Derivative(f(x, y, t), y) sage: F=f(x,x,t).diff(t) sage: F._sympy_() ValueError: Since there is more than one variable in the expression, the variable(s) of differentiation must be supplied to differentiate f(x, x, t)
There are typos in sage/interfaces/sympy.py
:
def _sympysage_derivative(self): """ EXAMPLES:: sage: from sympy import Derivative sage: f = function('f') sage: sympy_diff = Derivative(f(x)._sympy_(), x._sympy_()) sage: assert diff(f(x),x)._sympy_() == sympy_diff sage: assert diff(f(x),x) == sympy_diff._sage_() """ from sage.calculus.functional import derivative f = self.args[0]._sage_() args = [[a._sage_() for a in arg] if isinstance(arg,tuple) else arg._sage_() for arg in self.args[2:]] return derivative(f, *args)
I'm pretty sure the secondlast line should be iterating over self.args[1:]
, but changing that doesn't actually seem to fix the problem.
comment:7 Changed 3 years ago by
Replying to nbruin:
Translation of derivatives to sympy is definitely wonky:
sage: f=function('f') sage: F=f(x,y,t).diff(t) sage: F diff(f(x, y, t), t) sage: F._sympy_() Derivative(f(x, y, t), y) sage: F=f(x,x,t).diff(t) sage: F._sympy_() ValueError: Since there is more than one variable in the expression, the variable(s) of differentiation must be supplied to differentiate f(x, x, t)
The following fixes this particular problem:

src/sage/symbolic/expression_conversions.py
a b class SympyConverter(Converter): 860 860 # retrieve order 861 861 order = operator._parameter_set 862 862 # arguments 863 _args = ex. arguments()863 _args = ex.operands() 864 864 865 865 sympy_arg = [] 866 866 for i, a in enumerate(_args):
arguments
contains the operands, but in a different order – alphabetically I assume.
comment:8 Changed 3 years ago by
Description:  modified (diff) 

Summary:  Problem with integration/differentiation of symbolic functions. → sympy: Problem with integration/differentiation of symbolic functions. 
comment:9 Changed 3 years ago by
Authors:  → Markus Wageringel 

Branch:  → u/ghmwageringel/28964 
Commit:  → 0bfd840d432c82fa84d7015736d8caff8cef8832 
Keywords:  sympy added 
Status:  new → needs_review 
The patches in comment:6 and comment:7 almost fix the problem, but there were a few more issues in the conversion of derivatives between sympy and Sage. These are now fixed by this branch.
Most notably, sympy's arguments are encoded as sympy.core.containers.Tuple
which was missed in a typecheck isinstance(arg, tuple)
, and Sage's derivative does not accept tuples as arguments, so they need to be flattened.
Moreover, the conversion from Sage to sympy was very broken and has been rewritten completely, especially to account for the case of variables with multiple occurences as in comment:1.
As far as I can see, all the test cases mentioned on this ticket work correctly now.
New commits:
42d400b  28964: fix sympy conversions of derivative

0bfd840  28964: sympy: derivatives by variables with multiple occurences

comment:10 followup: 13 Changed 3 years ago by
Reviewers:  → Vincent Delecroix 

Status:  needs_review → needs_work 
Handling the Rational this way leads to problem
sage: cos(x).diff(x, 3/2) sin(x)
If this is really needed (I don't quite understand why yet), then you should check that there is no denominator.
comment:11 Changed 3 years ago by
Commit:  0bfd840d432c82fa84d7015736d8caff8cef8832 → 57a8e83d6e1f1b6a667b444817f1314effb0f95e 

Branch pushed to git repo; I updated commit sha1. New commits:
57a8e83  28964: fix conversion of integers from sympy to sage

comment:12 Changed 3 years ago by
Description:  modified (diff) 

comment:13 Changed 3 years ago by
Status:  needs_work → needs_review 

You are right. That was not a good solution. It turns out the problem is that elements of type sympy.core.numbers.Integer
(a subtype of sympy.core.numbers.Rational
) are converted to Sage's Rational
. For example, this causes problems here:
sage: import sympy sage: sympy.sympify('Derivative(f(x, t), x, x)') Derivative(f(x, t), (x, 2)) sage: _._sage_() # the 2 is converted to Rational here ...
I have just fixed this by implementing the missing conversion routine for integers. Thanks for pointing me in the right direction. I will squash the commits if you prefer.
comment:15 Changed 3 years ago by
Branch:  u/ghmwageringel/28964 → 57a8e83d6e1f1b6a667b444817f1314effb0f95e 

Resolution:  → fixed 
Status:  positive_review → closed 
Weird. Roundtripping to maxima seems to do the right thing, so that's not where the problem seems to lie:
If you make the variable combinations a little more complicated:
These are Pynac errors. So I suspect that there is a differentiation that doesn't have its variable specified, and that with only two variables around (one for the integration!) it ends up choosing that variable. So my guess is either Pynac itself or the way we interface with it.
The same problem seems to arise the other way around as well:
we can make that cause an error by fiddling with some more variables and then we see that the error traces back in something being called from
_sympysage_integral()
, so I think this error was introduced relatively recently, when sympy was added as an integrator by default.