has_internet is broken

[jhpalmieri]

The function has_internet in doctest/external.py doesn't work, at least on OS X, because it runs these commands:

sage: from six.moves import urllib
sage: urllib.request.urlopen("http://www.sagemath.org",timeout=1)
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-6-4b7ceedc32f5> in <module>()
----> 1 urllib.request.urlopen("http://www.sagemath.org",timeout=Integer(1))

  [snip]

HTTPError: HTTP Error 403: Forbidden

[jdemeyer]

I don't understand why, because the webpage that it's trying to access does exist.

[nbruin]

It seems that the default useragent supplied by urllib isn't liked:

sage: from six.moves import urllib
sage: url="http://www.sagemath.org"
sage: request = urllib.request.Request(url)
sage: request.add_header('User-Agent',"my browser 1.0")
sage: page = urllib.request.urlopen(request)
<no problem>

If I don't specify a user agent I get a 403 as well. The error response indicates cloudflare doesn't like the "browser signature". Any User-agent starting with Python-urllib is apparently not liked.

[nbruin]

This should do the trick.

---

New commits:

​21594be

has_internet: specify user agent to avoid CloudFlare blocking

[nbruin]

Oh, side effect (but this was already the case): do we run this as part of normal doctesting/installing etc.? The logs of the server should give a very easy list of IP addresses where this command has most likely run, by searching for the user agent (previously Python-urllib, now sage-doctest). That's just a consequence of accessing the internet in a function.

[tmonteil]

Replying to nbruin:

Oh, side effect (but this was already the case): do we run this as part of normal doctesting/installing etc.? The logs of the server should give a very easy list of IP addresses where this command has most likely run, by searching for the user agent (previously Python-urllib, now sage-doctest). That's just a consequence of accessing the internet in a function.

Indeed, you should add the # optional -- internet in the doctest of that function, # random is not sufficient.

[nbruin]

someone who cares about this ticket should do what is necessary.

[tmonteil]

Replying to nbruin:

someone who cares about this ticket should do what is necessary.

Done.

[tmonteil]

The behaviour is now:

• with sage -t src/sage/doctest/external.py, has_internet is not tested.

• with sage -t --optional=sage,internet src/sage/doctest/external.py, has_internet is tested and the result is checked to be True.

(the previous behaviour was that everytime the doctest was run, there was a connection to sagemath via cloudflare, but no error could be reported, so it was doubly wrong).

[jhpalmieri]

This works for me.