Opened 4 years ago
Closed 3 years ago
#9240 closed defect (fixed)
applying full_simplify() to gamma functions causes an error
Reported by: | tomc | Owned by: | tomc |
---|---|---|---|
Priority: | major | Milestone: | sage-4.7.1 |
Component: | symbolics | Keywords: | gamma function, full_simplify, factorial |
Cc: | kcrisman | Merged in: | sage-4.7.1.alpha4 |
Authors: | Tom Coates, Burcin Erocal | Reviewers: | Dan Drake, Karl-Dieter Crisman, François Bissey |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | #11415 | Stopgaps: |
Description (last modified by burcin)
Applying full_simplify() to the gamma function sometimes causes an error. This example works:
sage: gamma(4/3).full_simplify() 1/3*gamma(1/3)
but this example does not:
sage: gamma(1/3).full_simplify() ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (1254, 0)) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/tomc/sage-4.4.1/<ipython console> in <module>() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/symbolic/expression.so in sage.symbolic.expression.Expression.simplify_full (sage/symbolic/expression.cpp:21549)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/symbolic/expression.so in sage.symbolic.expression.Expression.simplify_factorial (sage/symbolic/expression.cpp:22240)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/structure/parent.so in sage.structure.parent.Parent.__call__ (sage/structure/parent.c:6332)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/structure/coerce_maps.so in sage.structure.coerce_maps.NamedConvertMap._call_ (sage/structure/coerce_maps.c:4053)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/interfaces/maxima.pyc in _symbolic_(self, R) 1810 sqrt(2) 1811 """ -> 1812 return R(self._sage_()) 1813 1814 def __complex__(self): /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/interfaces/maxima.pyc in _sage_(self) 1791 import sage.calculus.calculus as calculus 1792 return calculus.symbolic_expression_from_maxima_string(self.name(), -> 1793 maxima=self.parent()) 1794 1795 def _symbolic_(self, R): /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/calculus/calculus.pyc in symbolic_expression_from_maxima_string(x, equals_sub, maxima) 1524 # evaluation of maxima code are assumed pre-simplified 1525 is_simplified = True -> 1526 return symbolic_expression_from_string(s, syms, accept_sequence=True) 1527 except SyntaxError: 1528 raise TypeError, "unable to make sense of Maxima expression '%s' in Sage"%s /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/calculus/calculus.pyc in symbolic_expression_from_string(s, syms, accept_sequence) 1692 global _augmented_syms 1693 _augmented_syms = syms -> 1694 return parse_func(s) 1695 finally: 1696 _augmented_syms = {} /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.parse_sequence (sage/misc/parser.c:3855)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.parse_sequence (sage/misc/parser.c:3747)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.p_sequence (sage/misc/parser.c:4376)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.p_tuple (sage/misc/parser.c:5032)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.p_eqn (sage/misc/parser.c:5145)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.p_expr (sage/misc/parser.c:5465)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.p_term (sage/misc/parser.c:5690)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.p_factor (sage/misc/parser.c:6053)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/misc/parser.so in sage.misc.parser.Parser.p_power (sage/misc/parser.c:6264)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/symbolic/function.so in sage.symbolic.function.GinacFunction.__call__ (sage/symbolic/function.cpp:6321)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/symbolic/expression.so in sage.symbolic.expression.Expression.factorial (sage/symbolic/expression.cpp:20595)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/symbolic/pynac.so in sage.symbolic.pynac.py_factorial (sage/symbolic/pynac.cpp:9156)() /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/rings/arith.pyc in factorial(n, algorithm) 403 """ 404 if n < 0: --> 405 raise ValueError, "factorial -- must be nonnegative" 406 if algorithm == 'gmp': 407 return ZZ(n).factorial() ValueError: factorial -- must be nonnegative
I am running Sage 4.4.1 on Mac OS X version 10.6 (Snow Leopard), built from source. But the second example also fails on Sage 4.3.5 on 64-bit Linux, built from source. Looking at the source code suggests that the second example will fail on all platforms.
The problem occurs because full_simplify() here runs the following commands in Maxima:
(%i1) minfactorial(factcomb(makefact(gamma(1/3)))); 2 (%o1) (- -)! 3
and then the Maxima interface converts this to Sage as factorial(-2/3). This causes an error. For Sage, factorial(x) is only defined if x is a non-negative integer, whereas for Maxima factorial(x) is equivalent to gamma(1+x) and so makes sense whenever x is not in {-1, -2, -3, ...}
Apply: trac_9240_full_simplify_gamma.patch, trac_9240-factorial_evaluation.patch
Attachments (2)
Change History (17)
Changed 4 years ago by tomc
comment:1 Changed 4 years ago by ddrake
The patch applies cleanly to 4.4.4.alpha0, fixes the problem and includes some nice tests. I would give this a positive review, except that I really don't know the Pynac integration code.
Oh, and you say "For Sage, factorial(x) is only defined if x is a non-negative integer", but (with a clean 4.4.4.alpha0), it's defined the same way as Maxima:
sage: factorial(5) 120 sage: factorial(1/2) 1/2*sqrt(pi) sage: factorial(1/21) gamma(22/21)
comment:2 follow-up: ↓ 3 Changed 4 years ago by tomc
OK: I had misunderstood the docstring, which says:
sage: ? factorial String Form: factorial Namespace: Interactive File: /Users/tomc/sage-4.4.1/local/lib/python2.6/site-packages/sage/functions/other.py Definition: factorial(self, *args, coerce=True, hold=False, dont_call_method_on_arg=False) Docstring: Returns the factorial of n. INPUT: * ``n`` - an integer, or symbolic expression ...
I suppose that non-integer numerical values (rational numbers, real numbers, etc) count as symbolic expressions here, as they can be canonically coerced into the symbolic ring. But then there is definitely a bug in factorial(), because [in an unpatched version of Sage]:
sage: factorial(-1/2) ERROR: An unexpected error occurred while tokenizing input ... ValueError: factorial -- must be nonnegative
The patch fixes this.
comment:3 in reply to: ↑ 2 Changed 4 years ago by ddrake
comment:4 Changed 4 years ago by kcrisman
- Cc kcrisman added
See also this thread, where it came up again.
comment:5 Changed 3 years ago by burcin
- Milestone changed from sage-4.7 to sage-4.7.1
- Status changed from new to needs_review
I suppose this needs review.
comment:6 Changed 3 years ago by ddrake
For some reason, the patchbot isn't applying this patch, but it does apply cleanly and pass doctests in 4.7.rc1. (On 64-bit Ubuntu 10.10)
Looking at it again, I might recommend that in the TESTS, the first few examples be moved, since they already work and aren't testing to see if the problem in this ticket has been fixed. Only the full_simplify() tests are, strictly speaking, relevant to this ticket.
comment:7 Changed 3 years ago by kcrisman
- Reviewers set to Dan Drake
That's a good point, Dan. It's not bad to add them, but they should be added to the doc for gamma. A friendly reviewer patch from ddrake would probably be sufficient for this :)
comment:8 Changed 3 years ago by burcin
- Description modified (diff)
attachment:trac_9240-factorial_evaluation.patch adds further doctests and fixes for the evaluation of factorials. It should be applied after attachment:trac_9240_full_simplify_gamma.patch.
My changes depend on a small patch to pynac, where I somehow used && instead of bitwise and. This patch can be obtained from:
https://bitbucket.org/burcin/pynac-patches/src/c3c5b3b8b1eb/bitwise.patch
This ticket now depends on a new pynac release, which should be coming soon.
comment:9 Changed 3 years ago by burcin
- Dependencies set to #11415
Pynac package with the changes required by attachment:trac_9240-factorial_evaluation.patch is at #11415.
comment:10 Changed 3 years ago by kcrisman
- Reviewers changed from Dan Drake to Dan Drake, Karl-Dieter Crisman
Can you explain why the change from && to & was necessary? I don't know much CPP. Would there be a difference between bitwise 'and' and 'and' when the outputs are boolean? Or maybe they are not boolean.
Thanks!
comment:11 Changed 3 years ago by fbissey
Well I am not sure because I don't know the code well. But python_func is an unsigned and used to be a boolean according to comment in the code. What I think happens here is the code tries to match a precise type of python_func. 0 is just a c++ function but different from 0 it is a python construct and it can take different value depending on the construct. So if I am not mistaken it is bitwise because we are trying to match the construct code. This can be seen in the follwing snippet from function.cpp
function_options& function_options::eval_func(PyObject* e) { python_func |= eval_python_f; eval_f = eval_funcp(e); return *this; } function_options& function_options::evalf_func(PyObject* ef) { python_func |= evalf_python_f; evalf_f = evalf_funcp(ef); return *this; } function_options& function_options::conjugate_func(PyObject* c) { python_func |= conjugate_python_f; conjugate_f = conjugate_funcp(c); return *this; } function_options& function_options::real_part_func(PyObject* c) { python_func |= real_part_python_f; real_part_f = real_part_funcp(c); return *this; } function_options& function_options::imag_part_func(PyObject* c) { python_func |= imag_part_python_f; imag_part_f = imag_part_funcp(c); return *this; } function_options& function_options::derivative_func(PyObject* d) { python_func |= derivative_python_f; derivative_f = derivative_funcp(d); return *this; } function_options& function_options::power_func(PyObject* d) { python_func |= power_python_f; power_f = power_funcp(d); return *this; } function_options& function_options::series_func(PyObject* s) { python_func |= series_python_f; series_f = series_funcp(s); return *this; }
Notice how they match the bitwise comparison later in the patched code?
comment:12 follow-up: ↓ 13 Changed 3 years ago by kcrisman
- Reviewers changed from Dan Drake, Karl-Dieter Crisman to Dan Drake, Karl-Dieter Crisman, François Bissey
Thanks, I think that helps a little. I also found
cdef _register_function(self): # We don't need to add anything to GiNaC's function registry # However, if any custom methods were provided in the python class, # we should set the properties of the function_options object # corresponding to this function cdef GFunctionOpt opt = g_registered_functions().index(self._serial) if hasattr(self, '_eval_'): opt.eval_func(self)
which I knew about before.
I am going to have to write down exactly how all this works at Sage Days 31, because I do not want to be rediscovering this from scratch every time like I am now.
I have some more questions, presumably for Burcin. I don't think they are big deals, but I don't feel comfortable giving positive review without knowing them. Someone else who knows more might!
- why the change from the 'billions of digits' error message to the symbolic answer? This seems like a big change - someone might rely on that type of entry failing in number theory. Note that the multifactorial still has the 'billions of digits' error message, incidentally.
- what would the problem be if Ginac got symbolic answers back, if it didn't have anything for those before? (Not criticizing, just not understanding. I don't have a problem with them being numeric for ints and floats and longs.)
- Why did you remove opt.set_python_func() ? I assume this has something to do with fbissey's comment.
- Does return None just mean that Ginac will not try to evaluate things like factorial(sqrt(2)) internally?
Status:
- Positive review on Tom's patch, from Dan Drake.
- The log gamma stuff is fine.
- Apparently Francois is happy with the && to & switch. This is beyond me, though I don't see any problems with it.
- The actual changes to and new factorial and gamma functions are fine.
- Need answer to questions, or someone else to review those pieces in lieu of that.
- Finally, the big question - WHY this change? I can't find a single doctest that tells me what broke with Tom's patch but without Burcin's patch. I feel there must be some very subtle Maxima output that could have come out incorrect, but I cannot find it. All these doctests should have worked before (or were cdef functions so they couldn't be doctested).
comment:13 in reply to: ↑ 12 ; follow-up: ↓ 14 Changed 3 years ago by burcin
Replying to kcrisman:
Thanks, I think that helps a little. I also found
cdef _register_function(self): # We don't need to add anything to GiNaC's function registry # However, if any custom methods were provided in the python class, # we should set the properties of the function_options object # corresponding to this function cdef GFunctionOpt opt = g_registered_functions().index(self._serial) if hasattr(self, '_eval_'): opt.eval_func(self)which I knew about before.
Francois is right. The python_func variable is a bitset, indexed by the values here:
https://bitbucket.org/burcin/pynac/src/687b580c8c7c/ginac/function.h#cl-240
If the corresponding bit is on, then we call a python function.
When I first implemented this, defining custom evaluation etc. methods in Python was an all or nothing matter. Then I realized that overriding some of these methods and using the others defined in C++ made sense. I switched to the bitset at that point. As the change with & and && shows, that switch wasn't very successful.
I am going to have to write down exactly how all this works at Sage Days 31, because I do not want to be rediscovering this from scratch every time like I am now.
Maybe we can add some documentation to the reference manual about the general design of symbolics and in particular how the functions work.
I have some more questions, presumably for Burcin. I don't think they are big deals, but I don't feel comfortable giving positive review without knowing them. Someone else who knows more might!
- why the change from the 'billions of digits' error message to the symbolic answer? This seems like a big change - someone might rely on that type of entry failing in number theory. Note that the multifactorial still has the 'billions of digits' error message, incidentally.
The number theory people should use the functions from sage.rings.arith. In general, it's a bad idea to use the symbolics functions in library code unless you know what you're doing.
Suppose you have the expression factorial(big_number+1)/factorial(big_number). This would simplify to big_number. Telling people that they can't possibly work with numbers that big defeats the purpose of symbolic computation.
- what would the problem be if Ginac got symbolic answers back, if it didn't have anything for those before? (Not criticizing, just not understanding. I don't have a problem with them being numeric for ints and floats and longs.)
We get problems like #9913. I was being cautious.
- Why did you remove opt.set_python_func() ? I assume this has something to do with fbissey's comment.
- Does return None just mean that Ginac will not try to evaluate things like factorial(sqrt(2)) internally?
Yes. It's quite likely that this is not documented anywhere. :)
Status:
- Positive review on Tom's patch, from Dan Drake.
- The log gamma stuff is fine.
- Apparently Francois is happy with the && to & switch. This is beyond me, though I don't see any problems with it.
- The actual changes to and new factorial and gamma functions are fine.
- Need answer to questions, or someone else to review those pieces in lieu of that.
- Finally, the big question - WHY this change? I can't find a single doctest that tells me what broke with Tom's patch but without Burcin's patch. I feel there must be some very subtle Maxima output that could have come out incorrect, but I cannot find it. All these doctests should have worked before (or were cdef functions so they couldn't be doctested).
Tom's patch was great, it fixed the problem at hand. But when I first looked at it, I thought it needed more tests. Sitting down to write the tests, I found all kinds of problems, like the bitwise & issue and the big factorials raising an error. So I fixed those as well. Perhaps it was a mistake to tag these changes on this ticket, but here we are now.
comment:14 in reply to: ↑ 13 Changed 3 years ago by kcrisman
- Status changed from needs_review to positive_review
Francois is right. The python_func variable is a bitset, indexed by the values here:
https://bitbucket.org/burcin/pynac/src/687b580c8c7c/ginac/function.h#cl-240
If the corresponding bit is on, then we call a python function.
Okay, I finally understand what is going on here. I couldn't implement it myself, but it's just a really space-saving way to keep track of booleans like this. So it's just the way we tell Ginac that a custom _eval_ method or whatever has been defined. Good.
I am going to have to write down exactly how all this works at Sage Days 31, because I do not want to be rediscovering this from scratch every time like I am now.
Maybe we can add some documentation to the reference manual about the general design of symbolics and in particular how the functions work.
Absolutely - witness for instance #11143 where a new-ish developer has been stymied by this, though hopefully I was able to explain at least some of it to him.
The number theory people should use the functions from sage.rings.arith. In general, it's a bad idea to use the symbolics functions in library code unless you know what you're doing.
Yes, I agree, but sometimes the order of imports makes it hard to know which one is top-level.
Suppose you have the expression factorial(big_number+1)/factorial(big_number). This would simplify to big_number. Telling people that they can't possibly work with numbers that big defeats the purpose of symbolic computation.
Good point!
Yes. It's quite likely that this is not documented anywhere. :)
A good project as well.
- Finally, the big question - WHY this change? I can't find a single doctest that tells me what broke with Tom's patch but
So you are saying this was just an overhaul, but there is no specific error we know of that the rest fixed, it was just needed.
Good enough for me! Positive review. Long doctests finished passing late last night :)
comment:15 Changed 3 years ago by jdemeyer
- Merged in set to sage-4.7.1.alpha4
- Resolution set to fixed
- Status changed from positive_review to closed
based on Sage 4.4.1