ubuntu-bionic: python 3.8.5 readline module does not build after installing ncurses, readline spkgs

[guenterrote]

running on Ubuntu 18.04. I get the same result on the "master" and the current "develop" branch

config-log and build-log is attached.

The bug seems to have something to do with these error messages:

*** WARNING: renaming "readline" since importing it failed: /home/rote/Dokumente/Software/sage/local/lib/libreadline.so.8: undefined symbol: UP


[python3-3.8.5] Following modules built successfully but were removed because they could not be imported:
[python3-3.8.5] readline   

This error can be reproduced using:

tox -e docker-ubuntu-bionic-standard -- ncurses readline python3

[guenterrote]

python build logfile

[guenterrote]

config.log

[mkoeppe]

From python3...log:

checking for getc_unlocked() and friends... yes
checking how to link readline libs... -lreadline -ltinfo
...

but when python3 actually links readline, it is doing something else:

building 'readline' extension
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -O2 -g -march=native -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include/internal -I./Include -I/home/rote/Dokumente/Software/sage/local/include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src/Include -I/home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src -c /home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src/Modules/readline.c -o build/temp.linux-x86_64-3.8/home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src/Modules/readline.o
gcc -pthread -shared -Wl,-rpath-link,/home/rote/Dokumente/Software/sage/local/lib -L/home/rote/Dokumente/Software/sage/local/lib -Wl,-rpath,/home/rote/Dokumente/Software/sage/local/lib -L. -Wl,-rpath-link,/home/rote/Dokumente/Software/sage/local/lib -L/home/rote/Dokumente/Software/sage/local/lib -Wl,-rpath,/home/rote/Dokumente/Software/sage/local/lib -L. -Wl,-rpath-link,/home/rote/Dokumente/Software/sage/local/lib -L/home/rote/Dokumente/Software/sage/local/lib -Wl,-rpath,/home/rote/Dokumente/Software/sage/local/lib build/temp.linux-x86_64-3.8/home/rote/Dokumente/Software/sage/local/var/tmp/sage/build/python3-3.8.5/src/Modules/readline.o -L/usr/lib/termcap -L. -L/home/rote/Dokumente/Software/sage/local/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -lreadline -lncursesw -o build/lib.linux-x86_64-3.8/readline.cpython-38-x86_64-linux-gnu.so

You could check if uninstalling libncursesw5-dev solves this problem

[mkoeppe]

​https://github.com/python/cpython/blob/master/configure.ac#L4887 checks it in the order "" tinfo ncursesw ncurses curses termcap

​https://github.com/python/cpython/blob/master/setup.py#L975 detect_readline_curses redoes the checks in a different way

[mkoeppe]

Günter, on a typical linux machine, one would actually expect that ncurses and readline can be used and do not have to be built from scratch. So I would recommend to do make ncurses-clean readline-clean and then re-run configure. This should be more robust than the current configuration

[mkoeppe]

root@4d2478da7517:/sage# ldd local/lib/libreadline.so.8
	linux-vdso.so.1 (0x00007ffc7a3c8000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fee30793000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fee30dd5000)
root@4d2478da7517:/sage# ldd /usr/lib/x86_64-linux-gnu/libreadline.so 
	linux-vdso.so.1 (0x00007ffe23ded000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f100ed8d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f100e99c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f100f200000)

The issue likely arises because find_library_file inspects the system library but then the Sage-built library is used for linking ​https://github.com/python/cpython/blob/master/setup.py#L977

I think this is a limitation of distutils - it does not know about LIBRARY_PATH

[mkoeppe]

I think this would need a similar fix as it needed in the packaging of python3 in homebrew...

[guenterrote]

Replying to mkoeppe:

Günter, on a typical linux machine, one would actually expect that ncurses and readline can be used and do not have to be built from scratch. So I would recommend to do make ncurses-clean readline-clean and then re-run configure. This should be more robust than the current configuration

I tried this; the python installation seemed to go through fine, but eventually the build failed.

cd ../.. && sage-logger -p './sage --docbuild --no-pdf-links all html ' logs/dochtml.log
[dochtml] Traceback (most recent call last):
...
[dochtml]   File "sage/matrix/matrix_mpolynomial_dense.pyx", line 1, in init sage.matrix.matrix_mpolynomial_dense (build/cythonized/sage/matrix/matrix_mpolynomial_dense.cpp:8672)
[dochtml] ImportError: libreadline.so.8: cannot open shared object file: No such file or directory

Trying to start sage anyway displays a nice banner but crashes with the same error

---------------------------------------------------------------------------
ImportErrorPython 3.8.5: /home/rote/Dokumente/Software/sage/local/bin/python3
                                                   Thu Jan 21 07:42:55 2021
A problem occurred executing Python code.  Here is the sequence of function
...
...
~/Dokumente/Software/sage/local/lib/python3.8/site-packages/sage/matrix/matrix_mpolynomial_dense.pyx in init sage.matrix.matrix_mpolynomial_dense (build/cythonized/sage/matrix/matrix_mpolynomial_dense.cpp:8672)()
----> 1 """
      2 Dense matrices over multivariate polynomials over fields
...
ImportError: libreadline.so.8: cannot open shared object file: No such file or directory

[mkoeppe]

Replying to guenterrote:

Replying to mkoeppe:

Günter, on a typical linux machine, one would actually expect that ncurses and readline can be used and do not have to be built from scratch. So I would recommend to do make ncurses-clean readline-clean and then re-run configure. This should be more robust than the current configuration

I tried this; the python installation seemed to go through fine, but eventually the build failed.

cd ../.. && sage-logger -p './sage --docbuild --no-pdf-links all html ' logs/dochtml.log
[dochtml] Traceback (most recent call last):
...
[dochtml]   File "sage/matrix/matrix_mpolynomial_dense.pyx", line 1, in init sage.matrix.matrix_mpolynomial_dense (build/cythonized/sage/matrix/matrix_mpolynomial_dense.cpp:8672)
[dochtml] ImportError: libreadline.so.8: cannot open shared object file: No such file or directory

Trying to start sage anyway displays a nice banner but crashes with the same error

---------------------------------------------------------------------------
ImportErrorPython 3.8.5: /home/rote/Dokumente/Software/sage/local/bin/python3
                                                   Thu Jan 21 07:42:55 2021
A problem occurred executing Python code.  Here is the sequence of function
...
...
~/Dokumente/Software/sage/local/lib/python3.8/site-packages/sage/matrix/matrix_mpolynomial_dense.pyx in init sage.matrix.matrix_mpolynomial_dense (build/cythonized/sage/matrix/matrix_mpolynomial_dense.cpp:8672)()
----> 1 """
      2 Dense matrices over multivariate polynomials over fields
...
ImportError: libreadline.so.8: cannot open shared object file: No such file or directory

This looks like make singular-clean sagelib-clean build could fix it

[guenterrote]

Replying to mkoeppe:

This looks like make singular-clean sagelib-clean build could fix it

This worked! Thanks!

[mkoeppe]

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

Probably because of our ncurses/readline update in #32137, tox -e docker-ubuntu-bionic-standard -- ncurses readline python3 now works.

[dimpase]

ok