Encoding fixes to the doctest module

[jdemeyer]

New commits:

​3eb2db6	Miscellaneous fixes to tests in the doctest module.
​51e20d9	A few other miscellaneous fixes to the doctests tests
​878a62b	This exception reads 'utf-8' on Python 3 and 'utf8' on Python 2
​1ba0e67	Fix various minor encoding issues
​dcff893	Correct some encoding issues on Python 2.

[jdemeyer]

About

+            if not isinstance(got, six.text_type):
+                # On Python 3 got should already be unicode text, but on Python
+                # 2 it is not.  For comparison's sake we want the unicode text
+                # decoded from UTF-8. If there was some error such that the
+                # output is so malformed that it does not even decode from
+                # UTF-8 at all there will be an error in the test framework
+                # here. But this shouldn't happen at all, so we want it to be
+                # understood as an error in the test framework, and no some
+                # subtle error in the code under test.
+                got = got.decode(locale.getpreferredencoding())

In the comment you are talking about UTF-8 but you actually decode using the locale. Wouldn't UTF-8 make the most sense here?

Also a typo: "no some" -> "not some"

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​12ff3be	Fix various minor encoding issues
​34876ab	Correct some encoding issues on Python 2.

[jdemeyer]

I fixed your typo and reorganized the branch but there still remains the issue of locale vs utf-8.

[embray]

Replying to jdemeyer:

About

+            if not isinstance(got, six.text_type):
+                # On Python 3 got should already be unicode text, but on Python
+                # 2 it is not.  For comparison's sake we want the unicode text
+                # decoded from UTF-8. If there was some error such that the
+                # output is so malformed that it does not even decode from
+                # UTF-8 at all there will be an error in the test framework
+                # here. But this shouldn't happen at all, so we want it to be
+                # understood as an error in the test framework, and no some
+                # subtle error in the code under test.
+                got = got.decode(locale.getpreferredencoding())

In the comment you are talking about UTF-8 but you actually decode using the locale. Wouldn't UTF-8 make the most sense here?

I think the use of "UTF-8" in the comment is an artifact--I'm using it as a placeholder for "whatever the correct encoding is". But actually I need to investigate that a bit more carefully. I think what this really ought to be is whatever encoding Python is using for sys.stdout but it may be more complicated than that.

[embray]

It seems some of the patchbots in fact, however they happen to be configured, are returning 'ascii' for locale.getpreferredencoding(). I'll just make it UTF-8 for now and we can revisit locale-specific encodings if it comes up.

[embray]

I just based this off your branch and incorporated the change to UTF-8. I think this should be good enough for now; it's not worth overthinking until there's a concrete reason to. Tests look good with this change.

Setting "positive review" assuming you agree.

---

New commits:

​bf6cf97

Correct some encoding issues on Python 2.

[jdemeyer]

Replying to embray:

Setting "positive review" assuming you agree.

That was my only complaint, so yes I agree.

[vbraun]

Reviewer name...