#25501 closed defect (fixed)
Few failing internet doctests in mma_free_integrator
Reported by: | slabbe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-8.6 |
Component: | doctest coverage | Keywords: | thursdaysbdx |
Cc: | slelievre | Merged in: | |
Authors: | Sébastien Labbé, Amaury Pouly | Reviewers: | Frédéric Chapoton |
Report Upstream: | N/A | Work issues: | |
Branch: | f083393 (Commits, GitHub, GitLab) | Commit: | |
Dependencies: | Stopgaps: |
Description (last modified by )
As reported on sage-release 8.3.beta3,
sage -tp --optional=sage,internet --logfile=logs/25501.log src/sage/symbolic/integration/integral.py src/sage/symbolic/integration/external.py
gives
********************************************************************** File "src/sage/symbolic/integration/external.py", line 65, in sage.symbolic.integration.external.mma_free_integrator Failed example: mma_free_integrator(sin(x), x) # optional - internet Exception raised: Traceback (most recent call last): File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 572, in _run self.compile_and_execute(example, compiler, test.globs) File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 982, in compile_and_execute exec(compiled, globs) File "<doctest sage.symbolic.integration.external.mma_free_integrator[1]>", line 1, in <module> mma_free_integrator(sin(x), x) # optional - internet File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/symbolic/integration/external.py", line 97, in mma_free_integrator page = page[page.index('"inputForm"'):page.index('"outputForm"')] ValueError: substring not found ********************************************************************** ... ---------------------------------------------------------------------- sage -t src/sage/symbolic/integration/external.py # 3 doctests failed sage -t src/sage/symbolic/integration/integral.py # 1 doctest failed ----------------------------------------------------------------------
It can be reproduced with:
sage: from sage.symbolic.integration.external import mma_free_integrator sage: mma_free_integrator(sin(x), x) ... ValueError: substring not found
Attachments (2)
Change History (33)
comment:1 Changed 4 years ago by
- Description modified (diff)
comment:2 Changed 4 years ago by
- Description modified (diff)
- Summary changed from Few internet doctests failed in 4 files to Few failing internet doctests in mma_free_integrator
comment:3 Changed 4 years ago by
- Description modified (diff)
comment:4 Changed 4 years ago by
It seems that Ma#######ca is now hiding carefully its results, so we can no longer parse them..
comment:5 Changed 4 years ago by
From the starting point provided in the ticket description:
sage: from sage.symbolic.integration.external import mma_free_integrator sage: mma_free_integrator(sin(x), x) --------------------------------------------------------------------------- Traceback (most recent call last) <ipython-input-2-9d83ac0f1d23> in <module>() ----> 1 mma_free_integrator(sin(x), x) .../local/lib/python2.7/site-packages/sage/symbolic/integration/external.pyc in mma_free_integrator(expression, v, a, b) 95 params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'}) 96 page = urlopen("http://integrals.wolfram.com/home.jsp", params).read() ---> 97 page = page[page.index('"inputForm"'):page.index('"outputForm"')] 98 page = re.sub(r"\s", "", page) 99 mexpr = re.match(r".*Integrate.*==</em><br/>(.*)</p>", page).groups()[0] ValueError: substring not found
one can check what this function is doing:
sage: mma_free_integrator??
and follow its content step by step to trace the error:
sage: expression = sin(x) sage: v = x sage: import re sage: from six.moves.urllib.request import urlopen sage: from six.moves.urllib.parse import urlencode sage: vars = [str(x) for x in expression.variables()] sage: any(len(x)>1 for x in vars) False sage: x = SR.var('x') sage: repr(v) != 'x' False sage: params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'}) sage: params 'expr=Sin%5Bx%5D&random=false' sage: page = urlopen("http://integrals.wolfram.com/home.jsp", params).read()
Since the next line is the one that gives the error:
sage: page = page[page.index('"inputForm"'):page.index('"outputForm"')] Traceback (most recent call last) ... ValueError: substring not found
we can explore page
in search of a possible reason; the last few lines
indicate that the integral calculator has moved to Wolfram|Alpha.
sage: print(page) ... <!-- we inline this because of handlebars --> <script type="text/javascript"> if (getURLParam("redirected") == "true") { $('#input-wrapper').before( '<div id="redirectBanner"><div id="bannerContent"><div id="bannerText">The integral calculator has moved to Wolfram|Alpha. ' + '</div><a id="redirectLink" href="http://integrals.wolfram.com/home.jsp">Visit the old one »</a></div></div>' ); } $('form#input-form').submit(function() { var submittedData = $(this).find('input#calc-input').val(); if (!submittedData) { $(this).find('input#calc-input').val('integrate x sin(x^2)'); } }); </script> </body> </html>
Now if we use Wolfram|Alpha to collect the page describing the integral:
sage: page = urlopen("https://www.wolframalpha.com/input/?i=integrate+x+sin(x%5E2)").read() sage: print(page)
we see that the display of the solution happens in JavaScript?.
I see no way to parse the html page for the computed integral.
comment:6 Changed 4 years ago by
Yes, the web site tries very hard to hide the result, so that no machine can extract it.
comment:7 Changed 4 years ago by
The API access for Wolfram|Alpha requires signing in with a Wolfram ID.
It can be used for free, limited to 2000 queries per month.
comment:8 Changed 3 years ago by
- Branch set to u/slabbe/25501
- Commit set to d0c6a1a64784a0b2e58c83308fea7c45b2337892
- Status changed from new to needs_review
New commits:
0035b14 | #26361: package glucose SAT solver
|
1159e03 | #26361 : glucose depends on zlib
|
38d09eb | #26361 : fix glucose interface + doctests
|
878769d | #26361 : add a warning about the license of glucose-syrup
|
19ac7ec | Merge branch 'u/tmonteil/glucose_sat_solver' into 8.5.beta0
|
521f595 | 26361 adding doctests, adding glucose as an option to `SAT`
|
41fa426 | 26361: adding optional tags
|
bec05bb | Merge branch 'u/slabbe/glucose_sat_solver' into 8.5.beta4
|
d0c6a1a | 25501: fixing mathematica_free optional doctests
|
comment:9 Changed 3 years ago by
- Commit changed from d0c6a1a64784a0b2e58c83308fea7c45b2337892 to 58e2af486095ff6f6b24e67b1c20aca548c7d4e6
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
58e2af4 | 25501: fixing mathematica_free optional doctests
|
comment:10 Changed 3 years ago by
Oups, my first push was containing commits from another ticket.
I remade the branch on top of the most recent beta.
Sorry for the noise.
comment:11 Changed 3 years ago by
Changed 3 years ago by
comment:12 Changed 3 years ago by
I was able to make the Wolfram API work without login. I basically traced what the Wolfram page does (it's full of javascript) using Firefox dev tools, and produced the smallest Python program I could to reproduce it. At the moment it only prints the result, which is big-ish JSON with several results. Example use:
./wolfram.py "integrate x sin(x^2)"
Some random notes: it seems cookies are important, the "Referer" header was important, there is a proxy_code which is important but it's not clear if it can be reused. Obviously Wolfram may change this interface at any time. I have not checked if the params of the requests are the same as the official API.
EDIT: I uploaded a second program that uses the synchronous interface and drops some of the (seemingly) useless parameters of the requests. Some parameters are documented in the Full Reference of the Wolfram API.
Changed 3 years ago by
comment:13 Changed 3 years ago by
- Branch changed from u/slabbe/25501 to u/slabbe/25501-fix
- Commit changed from 58e2af486095ff6f6b24e67b1c20aca548c7d4e6 to 95172443d6bf0cf6c15cf57e9a54cab0ba11ad5c
I just updated the branch with a fix based on the suggestion made by gh-pamaury.
I stil need to add some doctests...
New commits:
9517244 | 25501: fixing the mma integrator
|
comment:14 Changed 3 years ago by
- Commit changed from 95172443d6bf0cf6c15cf57e9a54cab0ba11ad5c to 60bcbe7a23b5283643d4fe33d074d432a760bf9c
Branch pushed to git repo; I updated commit sha1. New commits:
60bcbe7 | 25501: small fix
|
comment:15 Changed 3 years ago by
- Keywords thursdaysbdx added
- Status changed from needs_review to needs_work
comment:16 Changed 3 years ago by
- Commit changed from 60bcbe7a23b5283643d4fe33d074d432a760bf9c to e5cb2728e42841fd76306813dbba81bef41573c9
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
e5cb272 | 25501: fixing the mma integrator
|
comment:18 Changed 3 years ago by
adding Amaury as author since the solution using cookies and json format comes from him.
comment:19 Changed 3 years ago by
- Cc slelievre added
comment:20 Changed 3 years ago by
- Status changed from needs_review to needs_work
unicode is not available in python3
comment:21 Changed 3 years ago by
- Branch changed from u/slabbe/25501-fix to u/slabbe/25501
- Commit changed from e5cb2728e42841fd76306813dbba81bef41573c9 to 157850caba088143863750d603d613b0f5766878
- Status changed from needs_work to needs_review
comment:22 Changed 3 years ago by
- Commit changed from 157850caba088143863750d603d613b0f5766878 to 76ad1cb684c219ce283cbbebe7281655fd18bff7
comment:23 Changed 3 years ago by
After fixing Python 3, I needed to adapt the doctests for Python 2 again...
Rebased on 8.5.rc0.
Needs review.
comment:24 Changed 3 years ago by
- Reviewers set to Frédéric Chapoton
- Status changed from needs_review to positive_review
oh, well. Let it be..
comment:25 Changed 3 years ago by
There is a internet-using doctest thats not marked as such:
********************************************************************** File "src/sage/symbolic/integration/external.py", line 240, in sage.symbolic.integration.external.parse_moutput_from_json Failed example: page_data = request_wolfram_alpha('Integrate(Sin[z], y)') Exception raised: Traceback (most recent call last): File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run self.compile_and_execute(example, compiler, test.globs) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute exec(compiled, globs) File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[2]>", line 1, in <module> page_data = request_wolfram_alpha('Integrate(Sin[z], y)') File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/symbolic/integration/external.py", line 157, in request_wolfram_alpha resp = opener.open(req) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 429, in open response = self._open(req, data) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 447, in _open '_open', req) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1228, in http_open return self.do_open(httplib.HTTPConnection, req) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1198, in do_open raise URLError(err) URLError: <urlopen error [Errno 110] Connection timed out> ********************************************************************** File "src/sage/symbolic/integration/external.py", line 241, in sage.symbolic.integration.external.parse_moutput_from_json Failed example: parse_moutput_from_json(page_data) Expected: Traceback (most recent call last): ... ValueError: asking wolframalpha.com was not successful Got: <BLANKLINE> Traceback (most recent call last): File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run self.compile_and_execute(example, compiler, test.globs) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute exec(compiled, globs) File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[3]>", line 1, in <module> parse_moutput_from_json(page_data) NameError: name 'page_data' is not defined
comment:26 Changed 3 years ago by
- Status changed from positive_review to needs_work
comment:27 Changed 3 years ago by
- Commit changed from 76ad1cb684c219ce283cbbebe7281655fd18bff7 to f083393998f39847e4cf93a65ee72036c74506f6
Branch pushed to git repo; I updated commit sha1. New commits:
f083393 | 25501: adding missing optional tags
|
comment:28 Changed 3 years ago by
- Status changed from needs_work to needs_review
comment:30 Changed 3 years ago by
- Branch changed from u/slabbe/25501 to f083393998f39847e4cf93a65ee72036c74506f6
- Resolution set to fixed
- Status changed from positive_review to closed
comment:31 Changed 3 years ago by
- Commit f083393998f39847e4cf93a65ee72036c74506f6 deleted
- Milestone changed from sage-8.3 to sage-8.6
I created independent tickets #25534 and #25535 to deal with the independent issues. Only the problem based on the
mma_free_integrator
is kept in this ticket.