Few failing internet doctests in mma_free_integrator

[slabbe]

As reported on ​sage-release 8.3.beta3,

sage -tp --optional=sage,internet --logfile=logs/25501.log src/sage/symbolic/integration/integral.py src/sage/symbolic/integration/external.py

gives

**********************************************************************
File "src/sage/symbolic/integration/external.py", line 65, in sage.symbolic.integration.external.mma_free_integrator
Failed example:
    mma_free_integrator(sin(x), x) # optional - internet
Exception raised:
    Traceback (most recent call last):
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 572, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 982, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.mma_free_integrator[1]>", line 1, in <module>
        mma_free_integrator(sin(x), x) # optional - internet
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/symbolic/integration/external.py", line 97, in mma_free_integrator
        page = page[page.index('"inputForm"'):page.index('"outputForm"')]
    ValueError: substring not found
**********************************************************************

...

----------------------------------------------------------------------
sage -t src/sage/symbolic/integration/external.py  # 3 doctests failed
sage -t src/sage/symbolic/integration/integral.py  # 1 doctest failed
----------------------------------------------------------------------

It can be reproduced with:

sage: from sage.symbolic.integration.external import mma_free_integrator
sage: mma_free_integrator(sin(x), x)

...
ValueError: substring not found

[slabbe]

I created independent tickets #25534 and #25535 to deal with the independent issues. Only the problem based on the mma_free_integrator is kept in this ticket.

[chapoton]

It seems that Ma#######ca is now hiding carefully its results, so we can no longer parse them..

[slelievre]

From the starting point provided in the ticket description:

sage: from sage.symbolic.integration.external import mma_free_integrator
sage: mma_free_integrator(sin(x), x)
---------------------------------------------------------------------------
Traceback (most recent call last)
<ipython-input-2-9d83ac0f1d23> in <module>()
----> 1 mma_free_integrator(sin(x), x)

.../local/lib/python2.7/site-packages/sage/symbolic/integration/external.pyc in mma_free_integrator(expression, v, a, b)
     95     params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'})
     96     page = urlopen("http://integrals.wolfram.com/home.jsp", params).read()
---> 97     page = page[page.index('"inputForm"'):page.index('"outputForm"')]
     98     page = re.sub(r"\s", "", page)
     99     mexpr = re.match(r".*Integrate.*==</em><br/>(.*)</p>", page).groups()[0]

ValueError: substring not found

one can check what this function is doing:

sage: mma_free_integrator??

and follow its content step by step to trace the error:

sage: expression = sin(x)
sage: v = x
sage: import re
sage: from six.moves.urllib.request import urlopen
sage: from six.moves.urllib.parse import urlencode
sage: vars = [str(x) for x in expression.variables()]
sage: any(len(x)>1 for x in vars)
False
sage: x = SR.var('x')
sage: repr(v) != 'x'
False
sage: params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'})
sage: params
'expr=Sin%5Bx%5D&random=false'
sage: page = urlopen("http://integrals.wolfram.com/home.jsp", params).read()

Since the next line is the one that gives the error:

sage: page = page[page.index('"inputForm"'):page.index('"outputForm"')]
Traceback (most recent call last)
...
ValueError: substring not found

we can explore page in search of a possible reason; the last few lines indicate that the integral calculator has moved to Wolfram|Alpha.

sage: print(page)
...
    <!-- we inline this because of handlebars -->
    <script type="text/javascript">
        if (getURLParam("redirected") == "true") {
          $('#input-wrapper').before(
              '<div id="redirectBanner"><div id="bannerContent"><div id="bannerText">The integral calculator has moved to Wolfram|Alpha. &nbsp;'
              + '</div><a id="redirectLink" href="http://integrals.wolfram.com/home.jsp">Visit the old one &raquo;</a></div></div>'
          );
        }

        $('form#input-form').submit(function() {
        var submittedData = $(this).find('input#calc-input').val();
        if (!submittedData) {
          $(this).find('input#calc-input').val('integrate x sin(x^2)');
        }
      });

    </script>
  </body>
</html>

Now if we use Wolfram|Alpha to collect the page describing the integral:

sage: page = urlopen("https://www.wolframalpha.com/input/?i=integrate+x+sin(x%5E2)").read()
sage: print(page)

we see that the display of the solution happens in JavaScript?.

I see no way to parse the html page for the computed integral.

[chapoton]

Yes, the web site tries very hard to hide the result, so that no machine can extract it.

[slelievre]

The API access for Wolfram|Alpha requires signing in with a Wolfram ID.

It can be used for free, limited to 2000 queries per month.

• ​Wolfram Alpha API

[slabbe]

New commits:

​0035b14	#26361: package glucose SAT solver
​1159e03	#26361 : glucose depends on zlib
​38d09eb	#26361 : fix glucose interface + doctests
​878769d	#26361 : add a warning about the license of glucose-syrup
​19ac7ec	Merge branch 'u/tmonteil/glucose_sat_solver' into 8.5.beta0
​521f595	26361 adding doctests, adding glucose as an option to `SAT`
​41fa426	26361: adding optional tags
​bec05bb	Merge branch 'u/slabbe/glucose_sat_solver' into 8.5.beta4
​d0c6a1a	25501: fixing mathematica_free optional doctests

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​58e2af4

25501: fixing mathematica_free optional doctests

[slabbe]

Oups, my first push was containing commits from another ticket.

I remade the branch on top of the most recent beta.

Sorry for the noise.

[gh-pamaury]

I was able to make the Wolfram API work without login. I basically traced what the Wolfram page does (it's full of javascript) using Firefox dev tools, and produced the smallest Python program I could to reproduce it. At the moment it only prints the result, which is big-ish JSON with several results. Example use:

./wolfram.py "integrate x sin(x^2)"

Some random notes: it seems cookies are important, the "Referer" header was important, there is a proxy_code which is important but it's not clear if it can be reused. Obviously Wolfram may change this interface at any time. I have not checked if the params of the requests are the same as the official API.

EDIT: I uploaded a second program that uses the synchronous interface and drops some of the (seemingly) useless parameters of the requests. Some parameters are documented in the Full Reference of the Wolfram API.

[slabbe]

I just updated the branch with a fix based on the suggestion made by gh-pamaury.

I stil need to add some doctests...

---

New commits:

​9517244

25501: fixing the mma integrator

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​60bcbe7

25501: small fix

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​e5cb272

25501: fixing the mma integrator

[slabbe]

Needs Review !!!

[slabbe]

adding Amaury as author since the solution using cookies and json format comes from him.

[chapoton]

unicode is not available in python3

[slabbe]

fixed Python 3 issues, rebased on 8.5.beta6 New commits:

​6b02bc2	25501: fixing the mma integrator
​157850c	25501: Python 3 care

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​11791c5	25501: fixing the mma integrator
​2eabdcb	25501: Python 3 care
​76ad1cb	25501: fix back for Python 2

[slabbe]

After fixing Python 3, I needed to adapt the doctests for Python 2 again...

Rebased on 8.5.rc0.

Needs review.

[chapoton]

oh, well. Let it be..

[vbraun]

There is a internet-using doctest thats not marked as such:

**********************************************************************
File "src/sage/symbolic/integration/external.py", line 240, in sage.symbolic.integration.external.parse_moutput_from_json
Failed example:
    page_data = request_wolfram_alpha('Integrate(Sin[z], y)')
Exception raised:
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[2]>", line 1, in <module>
        page_data = request_wolfram_alpha('Integrate(Sin[z], y)')
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/symbolic/integration/external.py", line 157, in request_wolfram_alpha
        resp = opener.open(req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 429, in open
        response = self._open(req, data)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 447, in _open
        '_open', req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 407, in _call_chain
        result = func(*args)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1228, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1198, in do_open
        raise URLError(err)
    URLError: <urlopen error [Errno 110] Connection timed out>
**********************************************************************
File "src/sage/symbolic/integration/external.py", line 241, in sage.symbolic.integration.external.parse_moutput_from_json
Failed example:
    parse_moutput_from_json(page_data)
Expected:
    Traceback (most recent call last):
    ...
    ValueError: asking wolframalpha.com was not successful
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[3]>", line 1, in <module>
        parse_moutput_from_json(page_data)
    NameError: name 'page_data' is not defined

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f083393

25501: adding missing optional tags

[chapoton]

ok