Opened 2 years ago

Closed 2 years ago

#30576 closed defect (fixed)

Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6

Reported by: Matthias Köppe Owned by:
Priority: critical Milestone: sage-9.2
Component: build Keywords:
Cc: Antonio Rojas, Dima Pasechnik, Samuel Lelièvre, Michael Orlitzky, François Bissey, Erik Bray, Frédéric Chapoton Merged in:
Authors: Matthias Koeppe Reviewers: Sébastien Labbé
Report Upstream: N/A Work issues:
Branch: 945c8c5 (Commits, GitHub, GitLab) Commit: 945c8c58b23cf7b8e5c15cb69c0ac4dbcd184cb5
Dependencies: #30008 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

Follow-up from #15791, #29033, #30053, the latter of which disabled use of system Python 3.6.

With Python 3.6, there is a failure building the documentation on ubuntu-bionic-standard (using /usr/bin/python3.6, https://github.com/mkoeppe/sage/runs/1106251169):

  [dochtml]   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2661: ordinal not in range(128)
  [dochtml] Full log file: logs/dochtml.log
Makefile:1876: recipe for target 'doc-html' failed

This is coming from #30002 "force C locale in docbuild and remove some obsolete stuff", which added this:

build/bin/sage-site:    export LANG=C # to ensure it is possible to scrape out non-EN locale warnings

In the present ticket, we rework the fix in #30002 so that it works with Python 3.6 too (in which setting the "C" locale sabotages UTF-8 operation.)

Supporting Python 3.6 allows Sage to use the system Python on some older Linux distributions that are still in widespread use in scientific computing, including centos-8 and fedora-{26,27,28} (although Python 3.7.x packages are also available for these). See #29033 for more details.

For the unrelated failures in the Sphinx package installation that #29033 tried to work around, see #30008. (#30008 is a dependency of this ticket merely for the convenience of testing.)


References:

Attachments (1)

error.log (22.1 KB) - added by Sébastien Labbé 2 years ago.

Download all attachments as: .zip

Change History (26)

comment:1 Changed 2 years ago by Sébastien Labbé

Branch: public/30576
Commit: e09d2561e5d5436717927a1e71e0551657c5164c

Posting my branch containing few changes that need to be done in order to build doc with ubuntu 18.04 bionic and python 3.6.

The changes are not sufficient. Other similar issues arise when sphinx is running...


New commits:

e09d256setting few utf-8 encodings
Last edited 2 years ago by Sébastien Labbé (previous) (diff)

comment:2 Changed 2 years ago by Sébastien Labbé

Two comments here:

  • fixing the issues in the posted branch took a while because there are many try - except clauses in the doc-building which eat up all of the useful tracebacks.
  • I think the remaining issues are just WARNINGS that breaks the docbuild because we decided at some point in the past to consider WARNINGS as errors (which is fine). I base this observation on some hacky modifications I did in the src/sage_setup/docbuild/sphinxbuild.py. To confirm this, do you know if there is a way to build sage documentation by considering warnings just as warnings?

comment:3 Changed 2 years ago by Dima Pasechnik

Warnings one keeps getting are errors in processing function arguments, i.e. in introspecting Python code. Ignoring them would give you a broken doc.

error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.complete_primary_decomposition: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)

I think the problems come from a new(ish) sphinx which is run as a backend for our custom docbuilder, and anyway the move from Python 3.6 to 3.7 was somewhat big as far as utf-8 support is concerned.

I really think that fully supporting 3.6 is hard, and we have better things to do. Python has solved this problem by moving to 3.7, let's not reinvent the wheel here.

comment:4 Changed 2 years ago by Matthias Köppe

Description: modified (diff)
Summary: Python 3.6: Fix locale/encoding issuesPython 3.6: Fix locale/encoding issues, then re-enable Python 3.6
Work issues: rebase on #30053, revert commit be47518

comment:5 Changed 2 years ago by git

Commit: e09d2561e5d5436717927a1e71e0551657c5164c9943f256d7a34c3eba9d12cde4ddeaf4260f398f

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e5f6663only use locale C.UTF-8 if available, else C
ff6f4a6assume Python 3.7 or better
64c3a8aonly test python3, to be 3.7 or 3.8
be47518build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'
af0a956Merge branch 't/30053/build/careful_with_C_UTF8' into t/30576/public/30576
8adc360Revert "build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'"
161765bRevert "only test python3, to be 3.7 or 3.8"
c75a21bsetting few utf-8 encodings
9943f25Revert "assume Python 3.7 or better"

comment:6 Changed 2 years ago by Matthias Köppe

Dependencies: #30053
Description: modified (diff)
Work issues: rebase on #30053, revert commit be47518

comment:7 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:8 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:9 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:10 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:11 in reply to:  3 ; Changed 2 years ago by Sébastien Labbé

Replying to dimpase:

Warnings one keeps getting are errors in processing function arguments, i.e. in introspecting Python code. Ignoring them would give you a broken doc.

error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.complete_primary_decomposition: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)

There are 2 things I don't understand here:

  • 4774 is an integer which is larger then the length of the string of the doc of complete_primary_decomposition method
sage: len(sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular
....: _repr.complete_primary_decomposition.__doc__)                             
3284
  • when I look at the html documentation of that method (after building the doc while ignoring the warnings with some hacky changes in sphinxbuild.py), I see no problem

comment:12 in reply to:  11 Changed 2 years ago by Sébastien Labbé

  • when I look at the html documentation of that method (after building the doc while ignoring the warnings with some hacky changes in sphinxbuild.py), I see no problem

With the current branch with no hacky changes, I can build the documentation with the flag --keep-going:

MAKE='make -j8' make build
MAKE='make -j8' ./sage --docbuild --no-pdf-links --keep-going all html

I am now trying to see what is wrong with the output doc...

Changed 2 years ago by Sébastien Labbé

Attachment: error.log added

comment:13 Changed 2 years ago by Sébastien Labbé

the command

MAKE='make -j8' build/bin/sage-logger -p './sage --docbuild --no-pdf-links --keep-going all html ' logs/dochtml-keepgoing.log
cd logs
grep error dochtml-keepgoing.log > error.log

gives relatively few errors (see 112 lines error.log attached). Very often it is the same error, for example, 60 of them are related to crystals code either can't decode byte 0xe2 in position 6257 or can't decode byte 0xc3 in position 3272 or can't decode byte 0xcc in position 7825.

I still do no know to which doc string these positions are refering to.

comment:14 Changed 2 years ago by Samuel Lelièvre

Hint:

$ python3 -q
>>> b'\xe2'.decode('latin1')
'â'
>>> b'\xc3'.decode('latin1')
'Ã'
>>> b'\xcc'.decode('latin1')
'Ì'
>>>

comment:15 Changed 2 years ago by Sébastien Labbé

But these symbols do not occur in the crystals code:

$ cd SAGE_ROOT
$ git grep à *.py *pyx
$ git grep â *.py *pyx
src/sage/combinat/ncsf_qsym/tutorial.py:MacMahon, Knuth, Kreweras, Glânffrwd Thomas, Stanley. In 1984, Gessel
$ git grep Ì *.py *pyx

comment:16 Changed 2 years ago by Matthias Köppe

Dependencies: #30053
Description: modified (diff)
Summary: Python 3.6: Fix locale/encoding issues, then re-enable Python 3.6Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6

comment:17 Changed 2 years ago by Matthias Köppe

Priority: majorcritical

comment:18 Changed 2 years ago by Matthias Köppe

Dependencies: #30008
Description: modified (diff)

comment:19 Changed 2 years ago by git

Commit: 9943f256d7a34c3eba9d12cde4ddeaf4260f398f945c8c58b23cf7b8e5c15cb69c0ac4dbcd184cb5

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

49aa061Revert "build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'"
a8b77cdRevert "only test python3, to be 3.7 or 3.8"
9403629build/bin/sage-system-python: Work around LC_ALL=C
ff0dbc6build/sage_bootstrap/uncompress/tar_file.py: Fix encoding to utf-8
f05a110Merge branch 't/30008/after__30053__sphinx_3_1_2_does_not_build_on_ubuntu__trusty_xenial_bionic___debian_jessie__centos_7__again_' into t/30576/public/30576
945c8c5build/bin/sage-site (--docbuild): Set an English locale suuch as C.utf-8, but not the C locale, to ensure correct operation on Python 3.6

comment:20 Changed 2 years ago by Matthias Köppe

Authors: Matthias Koeppe
Status: newneeds_review

comment:21 Changed 2 years ago by Matthias Köppe

Reviewers: https://github.com/mkoeppe/sage/actions/runs/267855732, ...

comment:22 Changed 2 years ago by Sébastien Labbé

On Ubuntu 18.04, after uninstalling python3.8 so that python3.6 gets picked up by ./configure:

## -------------------------------------------------------- ##                        
## Checking whether SageMath should install SPKG python3... ##                        ## -------------------------------------------------------- ##                        
configure:30688: checking whether any of sqlite libpng bzip2 xz libffi is installed as
 or will be installed as SPKG                                                         
configure:30697: result: no                                                           
configure:30707: checking for python3 >= 3.6, < 3.9 with modules sqlite3, ctypes, math
, hashlib, crypt, readline, socket, zlib, distutils.core                              
configure:30713: result:                                                              
configure:30728: checking ... whether /usr/bin/python3.6 is good                                                                      
configure:30921: result: yes                                                          
configure:30923: checking for python3 >= 3.6, < 3.9 with modules sqlite3, ctypes, math
, hashlib, crypt, readline, socket, zlib, distutils.core                              
configure:30982: result: /usr/bin/python3.6                                           
configure:30997: will use system package and not install SPKG python3    

I confirm that make doc now works ok. From me, it is a positive review.

I let someone change the status to positive review if no problem occurs on the github actions side.

comment:23 Changed 2 years ago by Sébastien Labbé

Reviewers: https://github.com/mkoeppe/sage/actions/runs/267855732, ...Sébastien Labbé, https://github.com/mkoeppe/sage/actions/runs/267855732, ...

comment:24 Changed 2 years ago by Matthias Köppe

Reviewers: Sébastien Labbé, https://github.com/mkoeppe/sage/actions/runs/267855732, ...Sébastien Labbé
Status: needs_reviewpositive_review

Thanks for testing!

Also ubuntu-bionic-standard on GH Action is fixed (https://github.com/mkoeppe/sage/runs/1152418788). The failures seen in https://github.com/mkoeppe/sage/actions/runs/267855732 (for example for ubuntu-bionic-minimal) come from an unrelated ticket that I merged in the same branch.

comment:25 Changed 2 years ago by Volker Braun

Branch: public/30576945c8c58b23cf7b8e5c15cb69c0ac4dbcd184cb5
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.