Opened 4 years ago
Closed 4 years ago
#25222 closed defect (fixed)
has_internet is broken
Reported by: | jhpalmieri | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-8.3 |
Component: | doctest framework | Keywords: | |
Cc: | Merged in: | ||
Authors: | Nils Bruin, Thierry Monteil | Reviewers: | Thierry Monteil, John Palmieri |
Report Upstream: | N/A | Work issues: | |
Branch: | c68e168 (Commits, GitHub, GitLab) | Commit: | c68e1686e25812f586c2055d1adb35c19965385a |
Dependencies: | Stopgaps: |
Description (last modified by )
The function has_internet
in doctest/external.py
doesn't work, at least on OS X, because it runs these commands:
sage: from six.moves import urllib sage: urllib.request.urlopen("http://www.sagemath.org",timeout=1) --------------------------------------------------------------------------- HTTPError Traceback (most recent call last) <ipython-input-6-4b7ceedc32f5> in <module>() ----> 1 urllib.request.urlopen("http://www.sagemath.org",timeout=Integer(1)) [snip] HTTPError: HTTP Error 403: Forbidden
Change History (12)
comment:1 Changed 4 years ago by
comment:2 Changed 4 years ago by
It seems that the default useragent supplied by urllib isn't liked:
sage: from six.moves import urllib sage: url="http://www.sagemath.org" sage: request = urllib.request.Request(url) sage: request.add_header('User-Agent',"my browser 1.0") sage: page = urllib.request.urlopen(request) <no problem>
If I don't specify a user agent I get a 403 as well. The error response indicates cloudflare doesn't like the "browser signature". Any User-agent starting with Python-urllib
is apparently not liked.
comment:3 Changed 4 years ago by
- Branch set to u/nbruin/has_internet_is_broken
comment:4 Changed 4 years ago by
- Commit set to 21594be2fb782bec9f4fc9e78bf0fd9292112980
- Description modified (diff)
- Status changed from new to needs_review
This should do the trick.
New commits:
21594be | has_internet: specify user agent to avoid CloudFlare blocking
|
comment:5 follow-up: ↓ 6 Changed 4 years ago by
Oh, side effect (but this was already the case): do we run this as part of normal doctesting/installing etc.? The logs of the server should give a very easy list of IP addresses where this command has most likely run, by searching for the user agent (previously Python-urllib
, now sage-doctest
). That's just a consequence of accessing the internet in a function.
comment:6 in reply to: ↑ 5 Changed 4 years ago by
- Reviewers set to Thierry Monteil
- Status changed from needs_review to needs_work
Replying to nbruin:
Oh, side effect (but this was already the case): do we run this as part of normal doctesting/installing etc.? The logs of the server should give a very easy list of IP addresses where this command has most likely run, by searching for the user agent (previously
Python-urllib
, nowsage-doctest
). That's just a consequence of accessing the internet in a function.
Indeed, you should add the # optional -- internet
in the doctest of that function, # random
is not sufficient.
comment:7 follow-up: ↓ 9 Changed 4 years ago by
someone who cares about this ticket should do what is necessary.
comment:8 Changed 4 years ago by
- Branch changed from u/nbruin/has_internet_is_broken to u/tmonteil/has_internet_is_broken
comment:9 in reply to: ↑ 7 Changed 4 years ago by
- Commit changed from 21594be2fb782bec9f4fc9e78bf0fd9292112980 to c68e1686e25812f586c2055d1adb35c19965385a
- Status changed from needs_work to needs_review
comment:10 Changed 4 years ago by
The behaviour is now:
- with
sage -t src/sage/doctest/external.py
,has_internet
is not tested.
- with
sage -t --optional=sage,internet src/sage/doctest/external.py
,has_internet
is tested and the result is checked to beTrue
.
(the previous behaviour was that everytime the doctest was run, there was a connection to sagemath via cloudflare, but no error could be reported, so it was doubly wrong).
comment:11 Changed 4 years ago by
- Reviewers changed from Thierry Monteil to Thierry Monteil, John Palmieri
- Status changed from needs_review to positive_review
This works for me.
comment:12 Changed 4 years ago by
- Branch changed from u/tmonteil/has_internet_is_broken to c68e1686e25812f586c2055d1adb35c19965385a
- Resolution set to fixed
- Status changed from positive_review to closed
I don't understand why, because the webpage that it's trying to access does exist.