Opened 19 months ago

Closed 12 months ago

Last modified 12 months ago

#25501 closed defect (fixed)

Few failing internet doctests in mma_free_integrator

Reported by: slabbe Owned by:
Priority: major Milestone: sage-8.6
Component: doctest coverage Keywords: thursdaysbdx
Cc: slelievre Merged in:
Authors: Sébastien Labbé, Amaury Pouly Reviewers: Frédéric Chapoton
Report Upstream: N/A Work issues:
Branch: f083393 (Commits) Commit:
Dependencies: Stopgaps:

Description (last modified by slabbe)

As reported on sage-release 8.3.beta3,

sage -tp --optional=sage,internet --logfile=logs/25501.log src/sage/symbolic/integration/integral.py src/sage/symbolic/integration/external.py

gives

**********************************************************************
File "src/sage/symbolic/integration/external.py", line 65, in sage.symbolic.integration.external.mma_free_integrator
Failed example:
    mma_free_integrator(sin(x), x) # optional - internet
Exception raised:
    Traceback (most recent call last):
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 572, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 982, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.mma_free_integrator[1]>", line 1, in <module>
        mma_free_integrator(sin(x), x) # optional - internet
      File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/symbolic/integration/external.py", line 97, in mma_free_integrator
        page = page[page.index('"inputForm"'):page.index('"outputForm"')]
    ValueError: substring not found
**********************************************************************

...

----------------------------------------------------------------------
sage -t src/sage/symbolic/integration/external.py  # 3 doctests failed
sage -t src/sage/symbolic/integration/integral.py  # 1 doctest failed
----------------------------------------------------------------------

It can be reproduced with:

sage: from sage.symbolic.integration.external import mma_free_integrator
sage: mma_free_integrator(sin(x), x)

...
ValueError: substring not found

Attachments (2)

wolfram.py (1.9 KB) - added by gh-pamaury 13 months ago.
wolfram2.py (3.5 KB) - added by gh-pamaury 13 months ago.

Download all attachments as: .zip

Change History (33)

comment:1 Changed 19 months ago by slabbe

  • Description modified (diff)

comment:2 Changed 19 months ago by slabbe

  • Description modified (diff)
  • Summary changed from Few internet doctests failed in 4 files to Few failing internet doctests in mma_free_integrator

I created independent tickets #25534 and #25535 to deal with the independent issues. Only the problem based on the mma_free_integrator is kept in this ticket.

comment:3 Changed 19 months ago by slabbe

  • Description modified (diff)

comment:4 Changed 17 months ago by chapoton

It seems that Ma#######ca is now hiding carefully its results, so we can no longer parse them..

comment:5 Changed 14 months ago by slelievre

From the starting point provided in the ticket description:

sage: from sage.symbolic.integration.external import mma_free_integrator
sage: mma_free_integrator(sin(x), x)
---------------------------------------------------------------------------
Traceback (most recent call last)
<ipython-input-2-9d83ac0f1d23> in <module>()
----> 1 mma_free_integrator(sin(x), x)

.../local/lib/python2.7/site-packages/sage/symbolic/integration/external.pyc in mma_free_integrator(expression, v, a, b)
     95     params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'})
     96     page = urlopen("http://integrals.wolfram.com/home.jsp", params).read()
---> 97     page = page[page.index('"inputForm"'):page.index('"outputForm"')]
     98     page = re.sub(r"\s", "", page)
     99     mexpr = re.match(r".*Integrate.*==</em><br/>(.*)</p>", page).groups()[0]

ValueError: substring not found

one can check what this function is doing:

sage: mma_free_integrator??

and follow its content step by step to trace the error:

sage: expression = sin(x)
sage: v = x
sage: import re
sage: from six.moves.urllib.request import urlopen
sage: from six.moves.urllib.parse import urlencode
sage: vars = [str(x) for x in expression.variables()]
sage: any(len(x)>1 for x in vars)
False
sage: x = SR.var('x')
sage: repr(v) != 'x'
False
sage: params = urlencode({'expr': expression._mathematica_init_(), 'random': 'false'})
sage: params
'expr=Sin%5Bx%5D&random=false'
sage: page = urlopen("http://integrals.wolfram.com/home.jsp", params).read()

Since the next line is the one that gives the error:

sage: page = page[page.index('"inputForm"'):page.index('"outputForm"')]
Traceback (most recent call last)
...
ValueError: substring not found

we can explore page in search of a possible reason; the last few lines indicate that the integral calculator has moved to Wolfram|Alpha.

sage: print(page)
...
    <!-- we inline this because of handlebars -->
    <script type="text/javascript">
        if (getURLParam("redirected") == "true") {
          $('#input-wrapper').before(
              '<div id="redirectBanner"><div id="bannerContent"><div id="bannerText">The integral calculator has moved to Wolfram|Alpha. &nbsp;'
              + '</div><a id="redirectLink" href="http://integrals.wolfram.com/home.jsp">Visit the old one &raquo;</a></div></div>'
          );
        }

        $('form#input-form').submit(function() {
        var submittedData = $(this).find('input#calc-input').val();
        if (!submittedData) {
          $(this).find('input#calc-input').val('integrate x sin(x^2)');
        }
      });

    </script>
  </body>
</html>

Now if we use Wolfram|Alpha to collect the page describing the integral:

sage: page = urlopen("https://www.wolframalpha.com/input/?i=integrate+x+sin(x%5E2)").read()
sage: print(page)

we see that the display of the solution happens in JavaScript?.

I see no way to parse the html page for the computed integral.

comment:6 Changed 14 months ago by chapoton

Yes, the web site tries very hard to hide the result, so that no machine can extract it.

comment:7 Changed 14 months ago by slelievre

The API access for Wolfram|Alpha requires signing in with a Wolfram ID.

It can be used for free, limited to 2000 queries per month.

comment:8 Changed 13 months ago by slabbe

  • Branch set to u/slabbe/25501
  • Commit set to d0c6a1a64784a0b2e58c83308fea7c45b2337892
  • Status changed from new to needs_review

New commits:

0035b14#26361: package glucose SAT solver
1159e03#26361 : glucose depends on zlib
38d09eb#26361 : fix glucose interface + doctests
878769d#26361 : add a warning about the license of glucose-syrup
19ac7ecMerge branch 'u/tmonteil/glucose_sat_solver' into 8.5.beta0
521f59526361 adding doctests, adding glucose as an option to `SAT`
41fa42626361: adding optional tags
bec05bbMerge branch 'u/slabbe/glucose_sat_solver' into 8.5.beta4
d0c6a1a25501: fixing mathematica_free optional doctests

comment:9 Changed 13 months ago by git

  • Commit changed from d0c6a1a64784a0b2e58c83308fea7c45b2337892 to 58e2af486095ff6f6b24e67b1c20aca548c7d4e6

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

58e2af425501: fixing mathematica_free optional doctests

comment:10 Changed 13 months ago by slabbe

Oups, my first push was containing commits from another ticket.

I remade the branch on top of the most recent beta.

Sorry for the noise.

comment:11 Changed 13 months ago by slabbe

  • Authors set to Sébastien Labbé

Changed 13 months ago by gh-pamaury

comment:12 Changed 13 months ago by gh-pamaury

I was able to make the Wolfram API work without login. I basically traced what the Wolfram page does (it's full of javascript) using Firefox dev tools, and produced the smallest Python program I could to reproduce it. At the moment it only prints the result, which is big-ish JSON with several results. Example use:

./wolfram.py "integrate x sin(x^2)"

Some random notes: it seems cookies are important, the "Referer" header was important, there is a proxy_code which is important but it's not clear if it can be reused. Obviously Wolfram may change this interface at any time. I have not checked if the params of the requests are the same as the official API.

EDIT: I uploaded a second program that uses the synchronous interface and drops some of the (seemingly) useless parameters of the requests. Some parameters are documented in the Full Reference of the Wolfram API.

Last edited 13 months ago by gh-pamaury (previous) (diff)

Changed 13 months ago by gh-pamaury

comment:13 Changed 12 months ago by slabbe

  • Branch changed from u/slabbe/25501 to u/slabbe/25501-fix
  • Commit changed from 58e2af486095ff6f6b24e67b1c20aca548c7d4e6 to 95172443d6bf0cf6c15cf57e9a54cab0ba11ad5c

I just updated the branch with a fix based on the suggestion made by gh-pamaury.

I stil need to add some doctests...


New commits:

951724425501: fixing the mma integrator

comment:14 Changed 12 months ago by git

  • Commit changed from 95172443d6bf0cf6c15cf57e9a54cab0ba11ad5c to 60bcbe7a23b5283643d4fe33d074d432a760bf9c

Branch pushed to git repo; I updated commit sha1. New commits:

60bcbe725501: small fix

comment:15 Changed 12 months ago by slabbe

  • Keywords thursdaysbdx added
  • Status changed from needs_review to needs_work

comment:16 Changed 12 months ago by git

  • Commit changed from 60bcbe7a23b5283643d4fe33d074d432a760bf9c to e5cb2728e42841fd76306813dbba81bef41573c9

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e5cb27225501: fixing the mma integrator

comment:17 Changed 12 months ago by slabbe

  • Status changed from needs_work to needs_review

Needs Review !!!

comment:18 Changed 12 months ago by slabbe

  • Authors changed from Sébastien Labbé to Sébastien Labbé, Amaury Pouly

adding Amaury as author since the solution using cookies and json format comes from him.

comment:19 Changed 12 months ago by slabbe

  • Cc slelievre added

comment:20 Changed 12 months ago by chapoton

  • Status changed from needs_review to needs_work

unicode is not available in python3

comment:21 Changed 12 months ago by slabbe

  • Branch changed from u/slabbe/25501-fix to u/slabbe/25501
  • Commit changed from e5cb2728e42841fd76306813dbba81bef41573c9 to 157850caba088143863750d603d613b0f5766878
  • Status changed from needs_work to needs_review

fixed Python 3 issues, rebased on 8.5.beta6 New commits:

6b02bc225501: fixing the mma integrator
157850c25501: Python 3 care
Last edited 12 months ago by slabbe (previous) (diff)

comment:22 Changed 12 months ago by git

  • Commit changed from 157850caba088143863750d603d613b0f5766878 to 76ad1cb684c219ce283cbbebe7281655fd18bff7

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

11791c525501: fixing the mma integrator
2eabdcb25501: Python 3 care
76ad1cb25501: fix back for Python 2

comment:23 Changed 12 months ago by slabbe

After fixing Python 3, I needed to adapt the doctests for Python 2 again...

Rebased on 8.5.rc0.

Needs review.

comment:24 Changed 12 months ago by chapoton

  • Reviewers set to Frédéric Chapoton
  • Status changed from needs_review to positive_review

oh, well. Let it be..

comment:25 Changed 12 months ago by vbraun

There is a internet-using doctest thats not marked as such:

**********************************************************************
File "src/sage/symbolic/integration/external.py", line 240, in sage.symbolic.integration.external.parse_moutput_from_json
Failed example:
    page_data = request_wolfram_alpha('Integrate(Sin[z], y)')
Exception raised:
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[2]>", line 1, in <module>
        page_data = request_wolfram_alpha('Integrate(Sin[z], y)')
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/symbolic/integration/external.py", line 157, in request_wolfram_alpha
        resp = opener.open(req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 429, in open
        response = self._open(req, data)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 447, in _open
        '_open', req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 407, in _call_chain
        result = func(*args)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1228, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/urllib2.py", line 1198, in do_open
        raise URLError(err)
    URLError: <urlopen error [Errno 110] Connection timed out>
**********************************************************************
File "src/sage/symbolic/integration/external.py", line 241, in sage.symbolic.integration.external.parse_moutput_from_json
Failed example:
    parse_moutput_from_json(page_data)
Expected:
    Traceback (most recent call last):
    ...
    ValueError: asking wolframalpha.com was not successful
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.symbolic.integration.external.parse_moutput_from_json[3]>", line 1, in <module>
        parse_moutput_from_json(page_data)
    NameError: name 'page_data' is not defined

comment:26 Changed 12 months ago by vbraun

  • Status changed from positive_review to needs_work

comment:27 Changed 12 months ago by git

  • Commit changed from 76ad1cb684c219ce283cbbebe7281655fd18bff7 to f083393998f39847e4cf93a65ee72036c74506f6

Branch pushed to git repo; I updated commit sha1. New commits:

f08339325501: adding missing optional tags

comment:28 Changed 12 months ago by slabbe

  • Status changed from needs_work to needs_review

comment:29 Changed 12 months ago by chapoton

  • Status changed from needs_review to positive_review

ok

comment:30 Changed 12 months ago by vbraun

  • Branch changed from u/slabbe/25501 to f083393998f39847e4cf93a65ee72036c74506f6
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:31 Changed 12 months ago by slabbe

  • Commit f083393998f39847e4cf93a65ee72036c74506f6 deleted
  • Milestone changed from sage-8.3 to sage-8.6
Note: See TracTickets for help on using tickets.