optional package notedown fail to install

[vdelecroix]

the machine is running archlinux

Found local metadata for notedown-1.5.1
Using cached file /opt/sage/upstream/notedown-1.5.1.tar.gz
notedown-1.5.1
====================================================
Setting up build directory for notedown-1.5.1
Finished extraction
No patch files found in ../patches
****************************************************
Host system:
Linux mangouste 4.14.15-1-ARCH #1 SMP PREEMPT Tue Jan 23 21:49:25 UTC 2018 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --enable-libmpx --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --enable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp
Thread model: posix
gcc version 7.3.0 (GCC) 
****************************************************
Installing notedown-1.5.1
Traceback (most recent call last):
  File "setup.py", line 10, in <module>
    readme = pandoc.communicate()[0].decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3031: ordinal not in range(128)
Error: could not determine package name
********************************************************************************
Error installing notedown-1.5.1
********************************************************************************

real	0m0.896s
user	0m0.437s
sys	0m0.103s
************************************************************************
Error installing package notedown-1.5.1
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the log file
  /opt/sage/logs/pkgs/notedown-1.5.1.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/opt/sage/local/var/tmp/sage/build/notedown-1.5.1 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
  (cd '/opt/sage/local/var/tmp/sage/build/notedown-1.5.1' && '/opt/sage/sage' --sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************

Upstream report at ​https://github.com/aaren/notedown/issues/76

[fbissey]

There was a similar issue with fpylll unicode in the readme was causing a fit. This is an issue that needs fixing upstream by removing the offending stuff from the readme file.

[vdelecroix]

The README.md is perfectly ASCII. Apparently, pandoc is producing those weird characters

>>> pandoc = subprocess.Popen(['pandoc', 'README.md', '--to', 'rst'],
... stdout=subprocess.PIPE)
>>> readme = pandoc.communicate()[0]
>>> readme.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3325: ordinal not in range(128)
>>> readme[3320:3330]
'ython\xe2\x80\x99s '

[vdelecroix]

New commits:

​e60f898

24901: patch for avoiding pandoc in setup.py

[vklein]

Replying to vdelecroix:

The README.md is perfectly ASCII. Apparently, pandoc is producing those weird characters

>>> pandoc = subprocess.Popen(['pandoc', 'README.md', '--to', 'rst'],
... stdout=subprocess.PIPE)
>>> readme = pandoc.communicate()[0]
>>> readme.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3325: ordinal not in range(128)
>>> readme[3320:3330]
'ython\xe2\x80\x99s '

For the record i am not sure if this example alone show a bug in pandoc as pandoc seems to work in utf-8 by default (​https://pandoc.org/MANUAL.html#character-encoding).
Pandoc don't seem to claim to keep the input encoding. In your exemple doing readme.decode('utf-8') instead of readme.decode() works fine.

On ubuntu 16.4 with pandoc 1.16.0.2 sage -i notedown works fine (sage version 8.2.beta7)

[vdelecroix]

Replying to vklein:

Replying to vdelecroix:

The README.md is perfectly ASCII. Apparently, pandoc is producing those weird characters

>>> pandoc = subprocess.Popen(['pandoc', 'README.md', '--to', 'rst'],
... stdout=subprocess.PIPE)
>>> readme = pandoc.communicate()[0]
>>> readme.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3325: ordinal not in range(128)
>>> readme[3320:3330]
'ython\xe2\x80\x99s '

For the record i am not sure if this example alone show a bug in pandoc as pandoc seems to work in utf-8 by default (​https://pandoc.org/MANUAL.html#character-encoding).

Pandoc don't seem to claim to keep the input encoding. In your exemple doing readme.decode('utf-8') instead of readme.decode() works fine.

On ubuntu 16.4 with pandoc 1.16.0.2 sage -i notedown works fine (sage version 8.2.beta7)

I never claimed that it was a bug in pandoc. The commands are extracted from the setup.py and produce an error on my computer. As specified in ​the documentation the behavior of decode depends on some default encoding that might differ between platforms.

Replacing readme.decode() with readme.decode('utf-8') in setup.py solves the problem mentioned in the description. However, I am not sure that utf-8 characters are encouraged in the long_description argument of the setup (there is no way to specify encoding there).

[rws]

This branch installs notedown fine here, both without and with system pandoc 1.19.2.1. LGTM.