#25501 closed defect (fixed)
Few failing internet doctests in mma_free_integrator
Reported by:  slabbe  Owned by:  

Priority:  major  Milestone:  sage8.6 
Component:  doctest coverage  Keywords:  thursdaysbdx 
Cc:  slelievre  Merged in:  
Authors:  Sébastien Labbé, Amaury Pouly  Reviewers:  Frédéric Chapoton 
Report Upstream:  N/A  Work issues:  
Branch:  f083393 (Commits, GitHub, GitLab)  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
As reported on sagerelease 8.3.beta3,
sage tp optional=sage,internet logfile=logs/25501.log src/sage/symbolic/integration/integral.py src/sage/symbolic/integration/external.py
gives
********************************************************************** File "src/sage/symbolic/integration/external.py", line 65, in sage.symbolic.integration.external.mma_free_integrator Failed example: mma_free_integrator(sin(x), x) # optional  internet Exception raised: Traceback (most recent call last): File "/home/slabbe/GitBox/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 572, in _run self.compile_and_execute(example, compiler, test.globs) File "/home/slabbe/GitBox/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 982, in compile_and_execute exec(compiled, globs) File "<doctest sage.symbolic.integration.external.mma_free_integrator[1]>", line 1, in <module> mma_free_integrator(sin(x), x) # optional  internet File "/home/slabbe/GitBox/sage/local/lib/python2.7/sitepackages/sage/symbolic/integration/external.py", line 97, in mma_free_integrator page = page[page.index('"inputForm"'):page.index('"outputForm"')] ValueError: substring not found ********************************************************************** ...  sage t src/sage/symbolic/integration/external.py # 3 doctests failed sage t src/sage/symbolic/integration/integral.py # 1 doctest failed 
It can be reproduced with:
sage: from sage.symbolic.integration.external import mma_free_integrator sage: mma_free_integrator(sin(x), x) ... ValueError: substring not found
Attachments (2)
Change History (33)
comment:1 Changed 3 years ago by
 Description modified (diff)
comment:2 Changed 3 years ago by
 Description modified (diff)
 Summary changed from Few internet doctests failed in 4 files to Few failing internet doctests in mma_free_integrator
comment:3 Changed 3 years ago by
 Description modified (diff)
comment:4 Changed 3 years ago by
It seems that Ma#######ca is now hiding carefully its results, so we can no longer parse them..
comment:5 Changed 3 years ago by
From the starting point provided in the ticket description:
sage: from sage.symbolic.integration.external import mma_free_integrator sage: mma_free_integrator(sin(x), x)  Traceback (most recent call last) <ipythoninput29d83ac0f1d23> in <module>() > 1 mma_free_integrator(sin(x), x) .../local/lib/python2.7/sitepackages/sage/symbolic/integration/external.pyc in mma_free_integrator(expression, v, a, b) 95 params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'}) 96 page = urlopen("http://integrals.wolfram.com/home.jsp", params).read() > 97 page = page[page.index('"inputForm"'):page.index('"outputForm"')] 98 page = re.sub(r"\s", "", page) 99 mexpr = re.match(r".*Integrate.*==</em><br/>(.*)</p>", page).groups()[0] ValueError: substring not found
one can check what this function is doing:
sage: mma_free_integrator??
and follow its content step by step to trace the error:
sage: expression = sin(x) sage: v = x sage: import re sage: from six.moves.urllib.request import urlopen sage: from six.moves.urllib.parse import urlencode sage: vars = [str(x) for x in expression.variables()] sage: any(len(x)>1 for x in vars) False sage: x = SR.var('x') sage: repr(v) != 'x' False sage: params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'}) sage: params 'expr=Sin%5Bx%5D&random=false' sage: page = urlopen("http://integrals.wolfram.com/home.jsp", params).read()
Since the next line is the one that gives the error:
sage: page = page[page.index('"inputForm"'):page.index('"outputForm"')] Traceback (most recent call last) ... ValueError: substring not found
we can explore page
in search of a possible reason; the last few lines
indicate that the integral calculator has moved to WolframAlpha.
sage: print(page) ... <! we inline this because of handlebars > <script type="text/javascript"> if (getURLParam("redirected") == "true") { $('#inputwrapper').before( '<div id="redirectBanner"><div id="bannerContent"><div id="bannerText">The integral calculator has moved to WolframAlpha. ' + '</div><a id="redirectLink" href="http://integrals.wolfram.com/home.jsp">Visit the old one »</a></div></div>' ); } $('form#inputform').submit(function() { var submittedData = $(this).find('input#calcinput').val(); if (!submittedData) { $(this).find('input#calcinput').val('integrate x sin(x^2)'); } }); </script> </body> </html>
Now if we use WolframAlpha to collect the page describing the integral:
sage: page = urlopen("https://www.wolframalpha.com/input/?i=integrate+x+sin(x%5E2)").read() sage: print(page)
we see that the display of the solution happens in JavaScript?.
I see no way to parse the html page for the computed integral.
comment:6 Changed 3 years ago by
Yes, the web site tries very hard to hide the result, so that no machine can extract it.
comment:7 Changed 3 years ago by
The API access for WolframAlpha requires signing in with a Wolfram ID.
It can be used for free, limited to 2000 queries per month.
comment:8 Changed 2 years ago by
 Branch set to u/slabbe/25501
 Commit set to d0c6a1a64784a0b2e58c83308fea7c45b2337892
 Status changed from new to needs_review
New commits:
0035b14  #26361: package glucose SAT solver

1159e03  #26361 : glucose depends on zlib

38d09eb  #26361 : fix glucose interface + doctests

878769d  #26361 : add a warning about the license of glucosesyrup

19ac7ec  Merge branch 'u/tmonteil/glucose_sat_solver' into 8.5.beta0

521f595  26361 adding doctests, adding glucose as an option to `SAT`

41fa426  26361: adding optional tags

bec05bb  Merge branch 'u/slabbe/glucose_sat_solver' into 8.5.beta4

d0c6a1a  25501: fixing mathematica_free optional doctests

comment:9 Changed 2 years ago by
 Commit changed from d0c6a1a64784a0b2e58c83308fea7c45b2337892 to 58e2af486095ff6f6b24e67b1c20aca548c7d4e6
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
58e2af4  25501: fixing mathematica_free optional doctests

comment:10 Changed 2 years ago by
Oups, my first push was containing commits from another ticket.
I remade the branch on top of the most recent beta.
Sorry for the noise.
comment:11 Changed 2 years ago by
Changed 2 years ago by
comment:12 Changed 2 years ago by
I was able to make the Wolfram API work without login. I basically traced what the Wolfram page does (it's full of javascript) using Firefox dev tools, and produced the smallest Python program I could to reproduce it. At the moment it only prints the result, which is bigish JSON with several results. Example use:
./wolfram.py "integrate x sin(x^2)"
Some random notes: it seems cookies are important, the "Referer" header was important, there is a proxy_code which is important but it's not clear if it can be reused. Obviously Wolfram may change this interface at any time. I have not checked if the params of the requests are the same as the official API.
EDIT: I uploaded a second program that uses the synchronous interface and drops some of the (seemingly) useless parameters of the requests. Some parameters are documented in the Full Reference of the Wolfram API.
Changed 2 years ago by
comment:13 Changed 2 years ago by
 Branch changed from u/slabbe/25501 to u/slabbe/25501fix
 Commit changed from 58e2af486095ff6f6b24e67b1c20aca548c7d4e6 to 95172443d6bf0cf6c15cf57e9a54cab0ba11ad5c
I just updated the branch with a fix based on the suggestion made by ghpamaury.
I stil need to add some doctests...
New commits:
9517244  25501: fixing the mma integrator

comment:14 Changed 2 years ago by
 Commit changed from 95172443d6bf0cf6c15cf57e9a54cab0ba11ad5c to 60bcbe7a23b5283643d4fe33d074d432a760bf9c
Branch pushed to git repo; I updated commit sha1. New commits:
60bcbe7  25501: small fix

comment:15 Changed 2 years ago by
 Keywords thursdaysbdx added
 Status changed from needs_review to needs_work
comment:16 Changed 2 years ago by
 Commit changed from 60bcbe7a23b5283643d4fe33d074d432a760bf9c to e5cb2728e42841fd76306813dbba81bef41573c9
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
e5cb272  25501: fixing the mma integrator

comment:18 Changed 2 years ago by
adding Amaury as author since the solution using cookies and json format comes from him.
comment:19 Changed 2 years ago by
 Cc slelievre added
comment:20 Changed 2 years ago by
 Status changed from needs_review to needs_work
unicode is not available in python3
comment:21 Changed 2 years ago by
 Branch changed from u/slabbe/25501fix to u/slabbe/25501
 Commit changed from e5cb2728e42841fd76306813dbba81bef41573c9 to 157850caba088143863750d603d613b0f5766878
 Status changed from needs_work to needs_review
comment:22 Changed 2 years ago by
 Commit changed from 157850caba088143863750d603d613b0f5766878 to 76ad1cb684c219ce283cbbebe7281655fd18bff7
comment:23 Changed 2 years ago by
After fixing Python 3, I needed to adapt the doctests for Python 2 again...
Rebased on 8.5.rc0.
Needs review.
comment:24 Changed 2 years ago by
 Reviewers set to Frédéric Chapoton
 Status changed from needs_review to positive_review
oh, well. Let it be..
comment:25 Changed 2 years ago by
There is a internetusing doctest thats not marked as such:
********************************************************************** File "src/sage/symbolic/integration/external.py", line 240, in sage.symbolic.integration.external.parse_moutput_from_json Failed example: page_data = request_wolfram_alpha('Integrate(Sin[z], y)') Exception raised: Traceback (most recent call last): File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 671, in _run self.compile_and_execute(example, compiler, test.globs) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 1086, in compile_and_execute exec(compiled, globs) File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[2]>", line 1, in <module> page_data = request_wolfram_alpha('Integrate(Sin[z], y)') File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/symbolic/integration/external.py", line 157, in request_wolfram_alpha resp = opener.open(req) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 429, in open response = self._open(req, data) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 447, in _open '_open', req) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1228, in http_open return self.do_open(httplib.HTTPConnection, req) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1198, in do_open raise URLError(err) URLError: <urlopen error [Errno 110] Connection timed out> ********************************************************************** File "src/sage/symbolic/integration/external.py", line 241, in sage.symbolic.integration.external.parse_moutput_from_json Failed example: parse_moutput_from_json(page_data) Expected: Traceback (most recent call last): ... ValueError: asking wolframalpha.com was not successful Got: <BLANKLINE> Traceback (most recent call last): File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 671, in _run self.compile_and_execute(example, compiler, test.globs) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 1086, in compile_and_execute exec(compiled, globs) File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[3]>", line 1, in <module> parse_moutput_from_json(page_data) NameError: name 'page_data' is not defined
comment:26 Changed 2 years ago by
 Status changed from positive_review to needs_work
comment:27 Changed 2 years ago by
 Commit changed from 76ad1cb684c219ce283cbbebe7281655fd18bff7 to f083393998f39847e4cf93a65ee72036c74506f6
Branch pushed to git repo; I updated commit sha1. New commits:
f083393  25501: adding missing optional tags

comment:28 Changed 2 years ago by
 Status changed from needs_work to needs_review
comment:30 Changed 2 years ago by
 Branch changed from u/slabbe/25501 to f083393998f39847e4cf93a65ee72036c74506f6
 Resolution set to fixed
 Status changed from positive_review to closed
comment:31 Changed 2 years ago by
 Commit f083393998f39847e4cf93a65ee72036c74506f6 deleted
 Milestone changed from sage8.3 to sage8.6
I created independent tickets #25534 and #25535 to deal with the independent issues. Only the problem based on the
mma_free_integrator
is kept in this ticket.