Opened 5 years ago

Closed 5 years ago

#23822 closed defect (fixed)

Python3 build error: unicode problem in pip_installed_packages()

Reported by: mderickx Owned by:
Priority: critical Milestone: sage-8.1
Component: python3 Keywords:
Cc: Merged in:
Authors: Maarten Derickx Reviewers: Frédéric Chapoton
Report Upstream: N/A Work issues:
Branch: f3e5282 (Commits, GitHub, GitLab) Commit: f3e5282c30fe45aa34b611f052341fb546e8e43b
Dependencies: Stopgaps:

Status badges


The python 3 build was broken by #23615 because of incorrect unicode usage causing the building of sage failing in the following way

chapoton@icj-laptop:~/sage3$ ./sage -br
cd . && export                                    \
    SAGE_ROOT=/doesnotexist                               \
    SAGE_SRC=/doesnotexist                                \
    SAGE_SRC_ROOT=/doesnotexist                           \
    SAGE_DOC_SRC=/doesnotexist                            \
    SAGE_BUILD_DIR=/doesnotexist                          \
    SAGE_PKGS=/home/chapoton/sage3/build/pkgs                \
    SAGE_CYTHONIZED=/home/chapoton/sage3/src/build/cythonized      \
&& sage-python23 -u --no-user-cfg build install
Traceback (most recent call last):
  File "", line 69, in <module>
    from module_list import ext_modules, library_order, aliases
  File "/home/chapoton/sage3/src/", line 166, in <module>
    from sage_setup.optional_extension import OptionalExtension
  File "/home/chapoton/sage3/src/sage_setup/", line 24, in <module>
    all_packages = list_packages(local=True)
  File "/home/chapoton/sage3/src/sage/misc/", line 226, in list_packages
    installed = installed_packages(exclude_pip)
  File "/home/chapoton/sage3/src/sage/misc/", line 286, in installed_packages
  File "/home/chapoton/sage3/src/sage/misc/", line 148, in pip_installed_packages
    return {package['name'].lower():package['version'] for package in json.loads(stdout)}
  File "/home/chapoton/sage3/local/lib/python3.6/json/", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/chapoton/sage3/local/lib/python3.6/json/", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/chapoton/sage3/local/lib/python3.6/json/", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Error building the Sage library
Makefile:34 : la recette pour la cible « sage » a échouée
make: *** [sage] Erreur 1

Change History (8)

comment:1 Changed 5 years ago by mderickx

Branch: u/mderickx/23822
Commit: f3e5282c30fe45aa34b611f052341fb546e8e43b
Status: newneeds_review

New commits:

f3e5282Trac #23822: fix python3 build error: unicode problem in pip_installed_packages()

comment:2 Changed 5 years ago by mderickx

Authors: Maarten Derickx

comment:3 Changed 5 years ago by chapoton

Reviewers: Frédéric Chapoton
Status: needs_reviewpositive_review

ok, let it be

comment:4 Changed 5 years ago by jhpalmieri

Do you need the .decode, or can you just use

stdout = proc.communicate()[0]


comment:5 Changed 5 years ago by mderickx

In python 2 it is not needed but good praxis. In python 3 it is needed since stdout is a bytestring before decoding and the json library only takes unicode strings and not bytestrings as input.

comment:6 Changed 5 years ago by jhpalmieri

When I was looking at #23876, I came across this bug and didn't use .decode, and it seemed to work with Python 3. That is, before making any changes, Sage crashed on startup, complaining about this problem, but when I switched to stdout = proc.communicate()[0], it didn't complain about that. (It crashed elsewhere, as described at #23876.) So I am not convinced that it is needed with Python 3.

Okay, I checked the Python documentation. I agree that stdout should be a binary stream, but according to the json documentation, json.loads can take something of type bytes as input.

Edit: this behavior of json is new as of Python 3.6.

Last edited 5 years ago by jhpalmieri (previous) (diff)

comment:7 Changed 5 years ago by mderickx

I could have sworn that I only added the decode() because it was failing otherwise. But sage was already on 3.6 when I wrote this. I dont think either solution is clearly better then the other. But the one with a decode already has positive review so I propose to leave it as is.

comment:8 Changed 5 years ago by vbraun

Branch: u/mderickx/23822f3e5282c30fe45aa34b611f052341fb546e8e43b
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.