Python 3.7+: setlocale: LC_ALL: cannot change locale (C.UTF-8) from build/bin/sage-spkg and in doctests; disable use of system Python 3.6

[dimpase]

In #29033 in build/bin/sage-spkg LC_ALL was changed to C.UTF-8 However, not all systems have it.

See also:

• #22659

Follow-up:

• #30008 Fix sage-system-python
• #30586 macOS: Doctest failures in some locales
• #30576 Python 3.6: Fix locale/encoding issues, then re-enable Python 3.6

[dimpase]

an easy way out is just to check whether the locale change worked, and if not, use C locale, not C.UTF-8. Perhaps print a warning.

[mkoeppe]

Replying to dimpase:

an easy way out is just to check whether the locale change worked, and if not, use C locale, not C.UTF-8.

+1

[dimpase]

New commits:

​37e042c

only use locale C.UTF-8 if available, else C

[gh-kliem]

Either this ticket is a duplicate of #30008 or it should make #30008 obsolete.

[gh-kliem]

I started test runs:

​https://github.com/kliem/sage/pull/20/checks

[dimpase]

this ticket was a result of a bug report on Arch, not centos. Hopefully it works for #30008 too.

[gh-kliem]

Tests did not complete, because the 9.2.beta3 tests fail everywhere.

​https://github.com/sagemath/sage/actions/runs/157607524

Is this a github issue or have we broken sage?

 Step 13/18 : RUN ./bootstrap
 ---> Running in 89427ef5c1c4
rm -rf config configure build/make/Makefile-auto.in
rm -f src/doc/en/installation/*.txt
rm -rf src/doc/en/reference/spkg/*.rst
rm -f src/doc/en/reference/repl/*.txt
src/doc/bootstrap:48: installing src/doc/en/installation/arch.txt and src/doc/en/installation/arch-optional.txt
src/doc/bootstrap:48: installing src/doc/en/installation/debian.txt and src/doc/en/installation/debian-optional.txt
src/doc/bootstrap:48: installing src/doc/en/installation/fedora.txt and src/doc/en/installation/fedora-optional.txt
src/doc/bootstrap:48: installing src/doc/en/installation/cygwin.txt and src/doc/en/installation/cygwin-optional.txt
src/doc/bootstrap:48: installing src/doc/en/installation/homebrew.txt and src/doc/en/installation/homebrew-optional.txt
src/doc/bootstrap:55: installing src/doc/en/reference/spkg/*.rst
src/doc/bootstrap:83: installing src/doc/en/reference/repl/options.txt
src/doc/bootstrap: line 84: src/doc/en/reference/repl/options.txt: No such file or directory
The command '/bin/sh -c ./bootstrap' returned a non-zero code: 1

[gh-kliem]

I just found #30064. Edit: I was cc on that, but I didn't realize how serious this is.

Ok. I'll run a new test then.

[gh-kliem]

This breaks building sphinx on windows.

​https://github.com/kliem/sage/runs/838933940

Same error as #30008.

As far as I understand the problem is that we need some sort of UTF to make the sphinx build work. It appears that on cygwin the default is better than C and C.UTF-8 does not work. So maybe C is not the best alternative for C.UTF-8.

Btw, strangely centos 7 appears to work with the current beta. I don't know what happened. (And I don't know yet, if this behavior is stable).

[gh-kliem]

And it breaks centos 8.

[embray]

Replying to gh-kliem:

This breaks building sphinx on windows.

​https://github.com/kliem/sage/runs/838933940

Same error as #30008.

As far as I understand the problem is that we need some sort of UTF to make the sphinx build work. It appears that on cygwin the default is better than C and C.UTF-8 does not work. So maybe C is not the best alternative for C.UTF-8.

I'm not sure what you mean here. C.UTF-8 is supported on Cygwin and is in fact the default locale in absence of any other settings: ​https://www.cygwin.com/cygwin-ug-net/setup-locale.html

The default locale in the absence of the aforementioned locale environment variables is "C.UTF-8".

[dimpase]

the error UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 45: ordinal not in range(128):

2020-07-05T16:29:58.1676169Z [sphinx-3.0.4.p0] installing. Log file: /cygdrive/d/a/sage/sage/logs/pkgs/sphinx-3.0.4.p0.log
2020-07-05T16:30:01.7883565Z   [sphinx-3.0.4.p0] error installing, exit status 1. End of log file:
2020-07-05T16:30:01.8141086Z   [sphinx-3.0.4.p0]   Found local metadata for sphinx-3.0.4.p0
2020-07-05T16:30:01.8155620Z   [sphinx-3.0.4.p0]   Attempting to download package Sphinx-3.0.4.tar.gz from mirrors
2020-07-05T16:30:01.8169727Z   [sphinx-3.0.4.p0]   http://mirrors.mit.edu/sage/spkg/upstream/sphinx/Sphinx-3.0.4.tar.gz
2020-07-05T16:30:01.8174798Z   [sphinx-3.0.4.p0]   [......................................................................]
2020-07-05T16:30:01.8191111Z   [sphinx-3.0.4.p0]   sphinx-3.0.4.p0
2020-07-05T16:30:01.8193080Z   [sphinx-3.0.4.p0]   ====================================================
2020-07-05T16:30:01.8197343Z   [sphinx-3.0.4.p0]   Setting up build directory for sphinx-3.0.4.p0
2020-07-05T16:30:01.8217424Z   [sphinx-3.0.4.p0]   Traceback (most recent call last):
2020-07-05T16:30:01.8221790Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/sage-uncompress-spkg", line 23, in <module>
2020-07-05T16:30:01.8222267Z   [sphinx-3.0.4.p0]       run()
2020-07-05T16:30:01.8222576Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/../sage_bootstrap/uncompress/cmdline.py", line 72, in run
2020-07-05T16:30:01.8222857Z   [sphinx-3.0.4.p0]       unpack_archive(archive, dirname)
2020-07-05T16:30:01.8223251Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/../sage_bootstrap/uncompress/action.py", line 68, in unpack_archive
2020-07-05T16:30:01.8223583Z   [sphinx-3.0.4.p0]       archive.extractall(members=archive.names)
2020-07-05T16:30:01.8223861Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/../sage_bootstrap/uncompress/tar_file.py", line 96, in extractall
2020-07-05T16:30:01.8224117Z   [sphinx-3.0.4.p0]       **kwargs)
2020-07-05T16:30:01.8224323Z   [sphinx-3.0.4.p0]     File "/usr/lib/python3.6/tarfile.py", line 2010, in extractall
2020-07-05T16:30:01.8224793Z   [sphinx-3.0.4.p0]       numeric_owner=numeric_owner)
2020-07-05T16:30:01.8225151Z   [sphinx-3.0.4.p0]     File "/usr/lib/python3.6/tarfile.py", line 2052, in extract
2020-07-05T16:30:01.8225442Z   [sphinx-3.0.4.p0]       numeric_owner=numeric_owner)
2020-07-05T16:30:01.8225898Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/../sage_bootstrap/uncompress/tar_file.py", line 122, in _extract_member
2020-07-05T16:30:01.8226166Z   [sphinx-3.0.4.p0]       **kwargs)
2020-07-05T16:30:01.8226601Z   [sphinx-3.0.4.p0]     File "/usr/lib/python3.6/tarfile.py", line 2122, in _extract_member
2020-07-05T16:30:01.8226883Z   [sphinx-3.0.4.p0]       self.makefile(tarinfo, targetpath)
2020-07-05T16:30:01.8227488Z   [sphinx-3.0.4.p0]     File "/usr/lib/python3.6/tarfile.py", line 2163, in makefile
2020-07-05T16:30:01.8227940Z   [sphinx-3.0.4.p0]       with bltn_open(targetpath, "wb") as target:
2020-07-05T16:30:01.8228249Z   [sphinx-3.0.4.p0]   UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 45: ordinal not in range(128)
2020-07-05T16:30:01.8228542Z   [sphinx-3.0.4.p0]   ************************************************************************
2020-07-05T16:30:01.8228988Z   [sphinx-3.0.4.p0]   Error: failed to extract /cygdrive/d/a/sage/sage/upstream/Sphinx-3.0.4.tar.gz
2020-07-05T16:30:01.8229264Z   [sphinx-3.0.4.p0]   ************************************************************************
2020-07-05T16:30:01.8229537Z   [sphinx-3.0.4.p0] Full log file: /cygdrive/d/a/sage/sage/logs/pkgs/sphinx-3.0.4.p0.log

[dimpase]

could it be that locale on Cygwin is not installed by default?

However, ​https://www.cygwin.com/cygwin-ug-net/setup-locale.html says:

Note
For a list of locales supported by your Windows machine, use the new locale -a command, which is part of the Cygwin package. For a description see locale(1)

[gh-kliem]

Replying to gh-kliem:

I started test runs:

​https://github.com/kliem/sage/pull/20/checks

I just started rerunning those tests on top of the current beta. Maybe that stuff just goes away by itself.

[gh-kliem]

Still causes this error.

[arojas]

If the centos issue is caused by the sphinx upgrade (according to #30008), why is it blocking this? This is meant to fix another (very annoying) issue on Arch.

[gh-kliem]

It appears that #30008 fixed itself. However, this here broke the cygwin sphinx build, last I checked.

It have no clue what is going on, but with this ticket we go from passing to failing.

[mkoeppe]

It seems a default setting on Cygwin is LANG=en_US.UTF-8. Perhaps we can try to only set LC_ALL if LANG is not already set or something like this.

[mkoeppe]

Also it should be investigated whether it was really necessary to add this line in #29033 to achieve Python 3.6 support. In particular note that sage-uncompress-spkg uses sage-system-python (which can even be python2) -- which really has nothing to do with Python 3.6 support (which is about PYTHON_FOR_VENV).

[mjo]

What problems arise if we drop the locale mangling entirely? Trac #15791 doesn't mention a problem.

[gh-mwageringel]

I am not sure if this is related, but while compiling Cypari on macOS, every file gives a warning of this type:

     Colperl: warning: Setting locale failed.
    perl: warning: Please check that your locale settings:
        LC_ALL = "C.UTF-8",
        LC_TERMINAL = "iTerm2",
        LANG = "de_DE.UTF-8"
        are supported and installed on your system.
    perl: warning: Falling back to a fallback locale ("de_DE.UTF-8").

The fallback that is used seems to work for me, though.

[mjo]

I just did a fresh build with LC_ALL=C and see no outstanding problems (python-3.7.8).

Maybe we should just revert that line? Why rock the boat?

Erik is the only other person who might know why it was added.

[mkoeppe]

This will need testing on centos-8 with Python 3.6

[embray]

Replying to mjo:

I just did a fresh build with LC_ALL=C and see no outstanding problems (python-3.7.8).

Maybe we should just revert that line? Why rock the boat?

No. This was added for reasons. Specifically to ensure compatibility between how Python 3.6 and Python 3.7 set the default encoding. Without this, there were bugs on Python 3.6 with Python not using a unicode character encoding by default. See ​https://www.python.org/dev/peps/pep-0538/

The simplest way to deal with this problem for currently released versions of CPython is to explicitly set a more sensible locale when launching the application. For example:

LC_CTYPE=C.UTF-8 python3 ...

The C.UTF-8 locale is a full locale definition that uses UTF-8 for the LC_CTYPE category, and the same settings as the C locale for all other categories (including LC_COLLATE). It is offered by a number of Linux distributions (including Debian, Ubuntu, Fedora, Alpine and Android) as an alternative to the ASCII-based C locale. Some other platforms (such as HP-UX) offer an equivalent locale definition under the name C.utf8.

Mac OS X and other *BSD systems have taken a different approach: instead of offering a C.UTF-8 locale, they offer a partial UTF-8 locale that only defines the LC_CTYPE category. On such systems, the preferred environmental locale adjustment is to set LC_CTYPE=UTF-8 rather than to set LC_ALL or LANG.

Perhaps this should also try the LC_CTYPE=UTF-8 mentioned here. Otherwise Dima's approach makes sense, though it can't guarantee that everything will just work on those systems. I can't recall exactly what broke without it but I do recall there was something. With Python 3.7 (the default when not using the system Python) this shouldn't be a problem since Python will basically force a UTF-8 locale for itself.

[dimpase]

Replying to embray:

Perhaps this should also try the LC_CTYPE=UTF-8 mentioned here. Otherwise Dima's approach makes sense, though it can't guarantee that everything will just work on those systems. I can't recall exactly what broke without it but I do recall there was something. With Python 3.7 (the default when not using the system Python) this shouldn't be a problem since Python will basically force a UTF-8 locale for itself.

How about only doing this export LC_...= on Python3.x with x<7 ? This should in particular make Arch people (apparently Arch has no C.UTF-8 or a similar locale, everything UTF-8 there is language-specific) happy, as their Python is new enough.

[mjo]

Ok, thanks for the information. I think the major take-away from PEP538 is,

With this change, any *nix platform that does not offer at least one of the C.UTF-8, C.utf8 or UTF-8 locales as part of its standard configuration would only be considered a fully supported platform for CPython 3.7+ deployments when a suitable locale other than the default C locale is configured explicitly (e.g. en_AU.UTF-8, zh_CN.gb18030).

I'm pretty sure we have files that actually need the UTF-8 encoding by now, so that rules out the possibility of "doing nothing" (leaving LC_ALL=C or LC_CTYPE=C) on python-3.6. And if we want to make python-3.6 work the way that python-3.7 does, then we're in the same situation as upstream is with respect to C.UTF-8: we have to consider python-3.6 with no C.UTF-8 (or equivalent) unsupported.

So I see two real options left:

1. Try to set the locale to C.UTF-8 or C.utf8 or UTF-8 when python-3.6 is being used, and declare the system unsupported if we can't. If python-3.7+ is being used, we can set the locale to C, and it will coerce the locale to something utf8ish on its own. In either case, a lack of utf8 locale would be unsupported.
2. Don't set a locale at all, and rely on the distribution/user to set a utf8 locale by default. This would result in some confusing grep/sort behavior (they're locale-dependent), but maybe we could be extra careful in our SPKGs to work around that, so that e.g. en_US.UTF-8 would work too.

Long-term, as one of the largest python projects in existence, I think we probably have to suck it up and go with (1), even though it pains me to require a locale that glibc doesn't even ship and isn't POSIX. Whatever decisions python makes, we're stuck with.

[mjo]

Replying to dimpase:

How about only doing this export LC_...= on Python3.x with x<7 ? This should in particular make Arch people (apparently Arch has no C.UTF-8 or a similar locale, everything UTF-8 there is language-specific) happy, as their Python is new enough.

I think a combination of this and the current branch is the best we can do. On python-3.6, we should try to set LC_ALL to C.UTF-8, C.utf8, or UTF-8. If we can't, then we should leave it alone and pray that the user's locale is compatible with all of our SPKGs. That situation would be unsupported by sage.

On python-3.7+, we can set LC_ALL=C, and python itself will try to pick an appropriate UTF-8 version of the locale. What does python on arch do in this situation? It's possible that python itself will fail to find a suitable UTF-8 locale, but there's not a lot we can do if upstream python insists on a nonstandard locale. Arch will just have to reconsider their decision unless they want to be not-fully supported by upstream python-3.7+.

[embray]

Personally I don't care what glibc or POSIX say on this. I think 1) is a fine option.

[mjo]

Apparently we carry the C.UTF-8 patch in Gentoo for systemd, who definitely don't care about portability:

​https://github.com/systemd/systemd/pull/10742

[mkoeppe]

Is anyone working on this?

[vbraun]

Sage-the-python-library can't realistically support anything that CPython does not; Its 2020, who in their right mind doesn't support utf-8? Better diagnostics for non-compliant systems would be great but imho not a blocker.

[mjo]

Replying to vbraun:

Sage-the-python-library can't realistically support anything that CPython does not; Its 2020, who in their right mind doesn't support utf-8? Better diagnostics for non-compliant systems would be great but imho not a blocker.

These systems do support UTF-8, but not the (as of yet) non-standard C.UTF-8 locale.

The current branch has the right idea, but since python-3.6 and python-3.7 act differently, it can be made a bit more precise. With python-3.7+, we can set LC_ALL=C and let python do the guessing. (Maybe it doesn't succeed, but officially Not Our Problem at that point.) With python-3.6, we can check for the C.UTF-8 locale and set it when found, with a fallback to LC_ALL=C. The current branch does this unconditionally but, it should only do it for python-3.6 and we should check the other equivalent names C.utf8 and UTF-8 too.

I think it's worthwhile to not output a million scary error messages in the sage-9.2 release on these systems that have done nothing wrong. At the very least, we owe it to the Arch maintainers who do a lot for sage and would have to field the resulting bug reports (or patch this themselves). I'm sure there are BSDs where this is problematic too.

[dimpase]

Arch locales maintainers just need to get C.UTF-8 locale, they are being silly (their argument - "it's an evil coming from Debian" - and I'm told they don't reopen the corresponding issue, as it's "decided". Meanwhile everybody else has C.UTF-8 locale, it's just them who don't)

[mkoeppe]

It's a blocker because it is a regression regarding platform support.

Can we please get a fix done?

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​e5f6663

only use locale C.UTF-8 if available, else C

[dimpase]

rebased over the latest beta

[slabbe]

Using a freshly installed Ubuntu 18.04 (bionic) and with some french settings set somewhere so that $ git pull returns Déjà à jour, a french equivalent for Already up to date, running make on 9.2.beta12 yields the [dochtml] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2661: ordinal not in range(128) error, see ​this post, right from the start of the build of the documentation.

With beta12 + current branch, I still get the same error after make doc-clean && make.

[slabbe]

That being said, I now see where it comes from:

sage: with open('src/doc/en/reference/references/index.rst', 'r') as f: s = f.read()                              
sage: s[2600:2700]                                                                                   
' characteristic,* The Open Book Series, vol. 2, no. 1, pp. 37–53, Jan. 2019.\n\n.. [ABZ2007] \\R. Aharo'
sage: s[2661:2700]                                                                                   
'–53, Jan. 2019.\n\n.. [ABZ2007] \\R. Aharo'

The character – has many occurrences in that file and is possibly not the only occurrence of a nonascii character (for example names of authors...). So, I am not sure replacing them by -- is the correct fix. And I don't see how this is related at all with the C.UTF-8 configuration.

[dimpase]

So your machine has no C.UTF-8 locale installed, right?

[slabbe]

How can I figure this out? If it can help, I have this:

$ locale
LANG=fr_CA.UTF-8
LANGUAGE=fr_CA:fr_FR:en_GB:en
LC_CTYPE="fr_CA.UTF-8"
LC_NUMERIC=fr_FR.UTF-8
LC_TIME=fr_FR.UTF-8
LC_COLLATE="fr_CA.UTF-8"
LC_MONETARY=fr_FR.UTF-8
LC_MESSAGES="fr_CA.UTF-8"
LC_PAPER=fr_FR.UTF-8
LC_NAME=fr_FR.UTF-8
LC_ADDRESS=fr_FR.UTF-8
LC_TELEPHONE=fr_FR.UTF-8
LC_MEASUREMENT=fr_FR.UTF-8
LC_IDENTIFICATION=fr_FR.UTF-8
LC_ALL=
slabbe@miami ~ $ man locale
slabbe@miami ~ $ locale -a
C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IL
en_IL.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
fr_BE.utf8
fr_CA.utf8
fr_CH.utf8
french
fr_FR
fr_FR.iso88591
fr_FR.utf8
fr_LU.utf8
POSIX

[dimpase]

in

+if test x`locale -a | grep C\.UTF-8` != x; then
+ export LC_ALL=C.UTF-8;
+else
+ export LC_ALL=C;
+fi

bit of this branch, could you change both LC_ALL to LC_CTYPE and try if it helps?

[slabbe]

Replying to dimpase:

bit of this branch, could you change both LC_ALL to LC_CTYPE and try if it helps?

Same error after make doc-clean and make.

[slabbe]

To me, it seems like an error of the following kind. That is we are opening the src/doc/en/reference/references/index.rst file as a bytes type, and at some place (where?), we decode the bytes to ascii and then we get UnicodeDecodeError because it is not ascii at all. Here is a way to reproduce the same error message:

sage: with open('src/doc/en/reference/references/index.rst', 'rb') as f: b = f.read()                
sage: b.decode('ascii')                                                                              
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-13-498050d5a3fb> in <module>
----> 1 b.decode('ascii')

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2661: ordinal not in range(128)
sage: s = b.decode('utf-8') 

I am adding Frédéric in cc since he fixed a lot of those UnicodeDecodeError in recent times with the passage to Python 3.

Also, why this works in Python3.8 and not in Python3.6 ?

[dimpase]

perhaps Python 3.6 needs an extra punch in form of

src/sage/docs/conf.py

@@ -36 +36 @@
 # Set locale to prevent having commas in decimal numbers
 # in tachyon input (see https://trac.sagemath.org/ticket/28971)
 import locale
+locale.setlocale(LC_ALL, '')
 locale.setlocale(locale.LC_NUMERIC, 'C')
 def sphinx_plot(graphics, **kwds):
     import matplotlib.image as mpimg

could you try this, without what's suggested in comment:51 ?

There is a comment

According to POSIX, a program which has not called setlocale(LC_ALL, '') runs using the 
portable 'C' locale. Calling setlocale(LC_ALL, '') lets it use the default locale as defined by the LANG variable. Since we do not want to interfere with the current locale setting we thus emulate the behavior in the way described above.

in ​https://docs.python.org/3.6/library/locale.html which makes me think it might be what's needed.

[mjo]

The name C.UTF-8 isn't sufficient; you have to check for the others too. On Gentoo:

$ locale -a
C
C.utf8
POSIX
en_US
en_US.iso88591
en_US.utf8

[slabbe]

Replying to dimpase:

could you try this, without what's suggested in comment:51 ?

I get the same error after make doc-clean && make. In fact adding a line print('AAAAAAAAAAAAA') a this place does not print any AAAAAAAAA... on my screen, so I wonder if that code is really executed before the error occurs.

[dimpase]

AAAAA, indeed (headbang on kbd) here is the fix, I think

src/doc/en/reference/conf_sub.py

@@ -20 +20 @@
 ref_out = os.path.join(SAGE_DOC, 'html', 'en', 'reference')
 
 # We use the main document's title, if we can find it.
-rst_file = open('index.rst', 'r')
+rst_file = open('index.rst', 'r', encoding='utf-8')
 rst_lines = rst_file.read().splitlines()
 rst_file.close()
 

indeed, that's one place in our homebaked docbuilder that opens UTF-8-encoded files without specifying them as UTF, and Python 3.6 does not like it.

[dimpase]

there are also many more missing encoding='utf-8' in open() on source files in the docbuilder.

and perhaps on other text files too. all these lead to these UnicodeDecodeError: 'ascii' codec can't decode byte...

Should we just abandon the idea to support random system Python 3.6, they are just broken in this way...

[dimpase]

Worse, there are files with lots of utf-8 stuff, sitting in r"""...""" multiline comments, e.g. src/sage/algebras/lie_algebras/lie_algebra_element.pyx, and they cause problems too.

[dimpase]

this is with Ubuntu 18.04 native Python 3.6:

[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.complete_primary_decomposition: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.genus: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.primary_decomposition_complete: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)
[dochtml] [polynomia] /home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py:docstring of sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal.degree_of_semi_regularity:8: WARNING: Inline interpreted text or phrase reference start-string without end-string.
[dochtml] [polynomia] /home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py:docstring of sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal.degree_of_semi_regularity:9: WARNING: Block quote ends without a blank line; unexpected unindent.
[dochtml] [polynomia] /home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py:docstring of sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal.degree_of_semi_regularity:23: WARNING: Inline interpreted text or phrase reference start-string without end-string.
[dochtml] [polynomia] /home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py:docstring of sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal.degree_of_semi_regularity:24: WARNING: Block quote ends without a blank line; unexpected unindent.
[dochtml] [polynomia] /home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py:docstring of sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.associated_primes:4: WARNING: Inline interpreted text or phrase reference start-string without end-string.
[dochtml] [polynomia] /home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py:docstring of sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.associated_primes:7: WARNING: Block quote ends without a blank line; unexpected unindent.
[dochtml] [polynomia] /home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py:docstring of sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.primary_decomposition:4: WARNING: Inline interpreted text or phrase reference start-string without end-string.
[dochtml] [polynomia] /home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py:docstring of sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.primary_decomposition:6: WARNING: Block quote ends without a blank line; unexpected unindent.
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial.hamming_weight: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial.inverse_series_trunc: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial.is_one: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial.is_term: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial.is_zero: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial.list: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial.number_of_terms: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial.truncate: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial_generic_dense.is_term: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial_generic_dense.list: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial_generic_dense.truncate: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.generic_power_trunc: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] WARNING: error while formatting arguments for sage.rings.polynomial.polynomial_element.Polynomial._mul_trunc_: 'ascii' codec can't decode byte 0xc3 in position 8166: ordinal not in range(128)
[dochtml] [polynomia] The inventory files are in local/share/doc/sage/inventory/en/reference/polynomial_rings.
[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[dochtml]     "__main__", mod_spec)
[dochtml]   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
[dochtml]     exec(code, run_globals)
[dochtml]   File "/home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 1730, in main
[dochtml]     builder()
[dochtml]   File "/home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 344, in _wrapper
[dochtml]     getattr(get_builder(document), 'inventory')(*args, **kwds)
[dochtml]   File "/home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 570, in _wrapper
[dochtml]     self._build_everything_except_bibliography(lang, format, *args, **kwds)
[dochtml]   File "/home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 556, in _build_everything_except_bibliography
[dochtml]     build_many(build_ref_doc, non_references)
[dochtml]   File "/home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 296, in build_many
[dochtml]     _build_many(target, args, processes=NUM_THREADS)
[dochtml]   File "/home/dima/sage/dev/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/utils.py", line 291, in build_many
[dochtml]     raise worker_exc.original_exception
[dochtml] OSError: WARNING: error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.complete_primary_decomposition: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)

if anyone wants to sort all this out, be my guest.

[slabbe]

please push your branch if you started to do some changes using utf-8 encodings. I will take a look tomorrow europe time.

[slabbe]

Because build_many catches the error, it is difficult to debug. The following change allows to see the real error:

src/sage_setup/docbuild/__init__.py

@@ -292 +292 @@
     Thin wrapper around `sage_setup.docbuild.utils.build_many` which uses the
     docbuild settings ``NUM_THREADS`` and ``ABORT_ON_ERROR``.
     """
+    for arg in args:
+        target(arg)
     try:
         _build_many(target, args, processes=NUM_THREADS)
     except BaseException as exc:

which is:

[dochtml]   File "/home/slabbe/GitBox/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 296, in build_many
[dochtml]     target(arg)
[dochtml]   File "/home/slabbe/GitBox/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 79, in build_ref_doc
[dochtml]     getattr(ReferenceSubBuilder(doc, lang), format)(*args, **kwds)
[dochtml]   File "/home/slabbe/GitBox/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 763, in _wrapper
[dochtml]     for module_name in self.get_all_included_modules():
[dochtml]   File "/home/slabbe/GitBox/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 927, in get_all_included_modules
[dochtml]     for module in self.get_modules(filename):
[dochtml]   File "/home/slabbe/GitBox/sage/local/lib/python3.6/site-packages/sage_setup/docbuild/__init__.py", line 1009, in get_modules
[dochtml]     lines = f.readlines()
[dochtml]   File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
[dochtml]     return codecs.ascii_decode(input, self.errors)[0]
[dochtml] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2661: ordinal not in range(128)

But indeed, I see that adding a encoding='utf-8' at this place is not sufficient.

[dimpase]

It is not sifficient, and I get the codec errors on a file where I cleaned all non-ascii away (sic!).

I am giving up on this. I think the code is too new for Python 3.6, and I don't think we should invest time in a version that will be EOL by the end of 2021.

People who only have Python 3.6 can build Python.

[dimpase]

We already have some code in Sage that was taken from Python 3.7, in src/sage/misc/sageinspect.py, which is used in docbuilding, perhaps it is too new, e.g.

def formatannotation(annotation, base_module=None):
    """
    This is taken from Python 3.7's inspect.py; the only change is to
    add documentation.
...

code from sageinspect.py is used in "formatting arguments", the error shown in comment:60.

[slabbe]

I think I will surrender too. I posted the changes that needs to be done on the branch ​u/slabbe/30053 to get Python3.6 to work. As you, I now get the error

[dochtml] OSError: WARNING: error while formatting arguments for 
sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.complete_primary_decomposition:
'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)

[dimpase]

this ticket helps if one is on python 3.x for x>6. As to 3.6, well, it's apparently hard, as Sage code and perhaps sphinx are too new for it.

[mjo]

On python >= 3.7, LC_ALL=C is sufficient, because python itself will attempt to find a UTF-8 version of that locale.

(It may be only a semantic difference, but then the non-existence of C.UTF-8 on some systems becomes Not Our Problem.)

[dimpase]

this patch sets LC_ALL to a meaningful value. Do you want it always to be C ?

[mjo]

Replying to dimpase:

this patch sets LC_ALL to a meaningful value. Do you want it always to be C ?

Yes, if we're going to throw python-3.6 under the bus in this release.

LC_ALL=C is the standard choice, and python >= 3.7 already treat LC_ALL=C magically. That's what "PEP 538 -- Coercing the legacy C locale to a UTF-8 based locale" is about.

[mkoeppe]

Let's refocus this ticket (blocker for Sage 9.2) on solving the issues for Python 3.7+

To me Python 3.6 support is a "wishlist" item, but not a blocker. I have created a separate ticket for it: #30576

[dimpase]

what are the 3.7+ problems left here? I presume all of them have plain C locale, so it all should work, no?

[mjo]

Replying to dimpase:

what are the 3.7+ problems left here? I presume all of them have plain C locale, so it all should work, no?

None, but AFAIK there were none before we started messing with the locale, either.

If no one is going to fix the python-3.6 problems, we should just drop it from the list of supported system pythons and revert this to LC_ALL=C. That's simpler and leaves us with one fewer locale problem to debug in the future.

[mkoeppe]

I wouldn't close the door on python 3.6 yet, so let's get this patch in please.

[mjo]

Replying to mkoeppe:

I wouldn't close the door on python 3.6 yet, so let's get this patch in please.

It still isn't sufficient for python-3.6, even if someone does fix the other problems. The name C.UTF-8 isn't standard because... it's not standardized yet. On Gentoo, it's C.utf8, and elsewhere apparently (per the PEP) it's called simply UTF-8. All of them should be checked/tried, and we should really only be doing it if python-3.6 is in use. That way, when we drop python-3.6, we're left with just LC_ALL=C again.

Also keep in mind that if no one actually fixes the other problems with python-3.6, the door is closed regardless.

[mkoeppe]

Replying to mjo:

Replying to mkoeppe:

I wouldn't close the door on python 3.6 yet, so let's get this patch in please.

It still isn't sufficient for python-3.6

Yes, I know, that's why I opened the separate ticket #30576 for that.

[dimpase]

there is no harm in adding this, anyway, thus, over to bots.

[dimpase]

I've also removed #30551 as a dependency - after all whether Python 3.6 is (somewhat) supported (IMHO docbuilding is the main problem there) can be sorted out later.

[mjo]

Replying to mkoeppe:

Yes, I know, that's why I opened the separate ticket #30576 for that.

The stuff on ticket #30576 isn't what I was referring to...

Replying to dimpase:

there is no harm in adding this, anyway, thus, over to bots.

The harm is that now we have additional locale problems to debug for a change that doesn't solve anything. With python-3.7, there are now three different configurations:

• The C.UTF-8 locale exists and is used.
• No utf8 locale exists, so C is used.
• A utf8 locale is available, but your grep doesn't find it, so we wind up with C on a machine that could have a utf8 locale.

Why, when LC_ALL=C unconditionally works fine and had been in there for ages?

If someone wants to fix python-3.6, we should add these hacks only for python-3.6, and we should try the alternate names C.utf8 and UTF-8, too.

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​ff6f4a6

assume Python 3.7 or better

[dimpase]

OK, so how about this?

[mjo]

I think that's the best solution, possibly combined with dropping 3.6 from the list of system pythons that we accept. Portability is nice and all, but we're wasting a lot of time trying to support an evolutionary dead-end with py3.6.

[mkoeppe]

Fine with me - then please also adjust python3's spkg-configure...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​64c3a8a

only test python3, to be 3.7 or 3.8

[dimpase]

Replying to mkoeppe:

Fine with me - then please also adjust python3's spkg-configure...

fixed, as discussed - only test python3, to be either 3.7 or 3.8.

Please review

[mkoeppe]

I have made an adjustment. I don't like mixing the change discussed in #30546 into this ticket

---

New commits:

​be47518

build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'

[mkoeppe]

Rationale: The changed search behavior would lead to confusing reports on sage-release.