Opened 2 years ago

Closed 2 years ago

#25222 closed defect (fixed)

has_internet is broken

Reported by: jhpalmieri Owned by:
Priority: major Milestone: sage-8.3
Component: doctest framework Keywords:
Cc: Merged in:
Authors: Nils Bruin, Thierry Monteil Reviewers: Thierry Monteil, John Palmieri
Report Upstream: N/A Work issues:
Branch: c68e168 (Commits) Commit: c68e1686e25812f586c2055d1adb35c19965385a
Dependencies: Stopgaps:

Description (last modified by nbruin)

The function has_internet in doctest/external.py doesn't work, at least on OS X, because it runs these commands:

sage: from six.moves import urllib
sage: urllib.request.urlopen("http://www.sagemath.org",timeout=1)
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-6-4b7ceedc32f5> in <module>()
----> 1 urllib.request.urlopen("http://www.sagemath.org",timeout=Integer(1))

  [snip]

HTTPError: HTTP Error 403: Forbidden

Change History (12)

comment:1 Changed 2 years ago by jdemeyer

I don't understand why, because the webpage that it's trying to access does exist.

comment:2 Changed 2 years ago by nbruin

It seems that the default useragent supplied by urllib isn't liked:

sage: from six.moves import urllib
sage: url="http://www.sagemath.org"
sage: request = urllib.request.Request(url)
sage: request.add_header('User-Agent',"my browser 1.0")
sage: page = urllib.request.urlopen(request)
<no problem>

If I don't specify a user agent I get a 403 as well. The error response indicates cloudflare doesn't like the "browser signature". Any User-agent starting with Python-urllib is apparently not liked.

Last edited 2 years ago by nbruin (previous) (diff)

comment:3 Changed 2 years ago by nbruin

  • Branch set to u/nbruin/has_internet_is_broken

comment:4 Changed 2 years ago by nbruin

  • Authors set to Nils Bruin
  • Commit set to 21594be2fb782bec9f4fc9e78bf0fd9292112980
  • Description modified (diff)
  • Status changed from new to needs_review

This should do the trick.


New commits:

21594behas_internet: specify user agent to avoid CloudFlare blocking

comment:5 follow-up: Changed 2 years ago by nbruin

Oh, side effect (but this was already the case): do we run this as part of normal doctesting/installing etc.? The logs of the server should give a very easy list of IP addresses where this command has most likely run, by searching for the user agent (previously Python-urllib, now sage-doctest). That's just a consequence of accessing the internet in a function.

comment:6 in reply to: ↑ 5 Changed 2 years ago by tmonteil

  • Reviewers set to Thierry Monteil
  • Status changed from needs_review to needs_work

Replying to nbruin:

Oh, side effect (but this was already the case): do we run this as part of normal doctesting/installing etc.? The logs of the server should give a very easy list of IP addresses where this command has most likely run, by searching for the user agent (previously Python-urllib, now sage-doctest). That's just a consequence of accessing the internet in a function.

Indeed, you should add the # optional -- internet in the doctest of that function, # random is not sufficient.

comment:7 follow-up: Changed 2 years ago by nbruin

someone who cares about this ticket should do what is necessary.

comment:8 Changed 2 years ago by tmonteil

  • Branch changed from u/nbruin/has_internet_is_broken to u/tmonteil/has_internet_is_broken

comment:9 in reply to: ↑ 7 Changed 2 years ago by tmonteil

  • Commit changed from 21594be2fb782bec9f4fc9e78bf0fd9292112980 to c68e1686e25812f586c2055d1adb35c19965385a
  • Status changed from needs_work to needs_review

Replying to nbruin:

someone who cares about this ticket should do what is necessary.

Done.

comment:10 Changed 2 years ago by tmonteil

The behaviour is now:

  • with sage -t src/sage/doctest/external.py, has_internet is not tested.
  • with sage -t --optional=sage,internet src/sage/doctest/external.py, has_internet is tested and the result is checked to be True.

(the previous behaviour was that everytime the doctest was run, there was a connection to sagemath via cloudflare, but no error could be reported, so it was doubly wrong).

comment:11 Changed 2 years ago by jhpalmieri

  • Authors changed from Nils Bruin to Nils Bruin, Thierry Monteil
  • Reviewers changed from Thierry Monteil to Thierry Monteil, John Palmieri
  • Status changed from needs_review to positive_review

This works for me.

comment:12 Changed 2 years ago by vbraun

  • Branch changed from u/tmonteil/has_internet_is_broken to c68e1686e25812f586c2055d1adb35c19965385a
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.