Opened 22 months ago

Last modified 3 months ago

#29012 new defect

python3 spkg: Work around errors building/importing the _crypt module

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.5
Component: packages: standard Keywords:
Cc: isuruf, dimpase, jdemeyer Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description

Building the spkg python3 3.7.3.p1 fails on Debian with libc 2.28-10 when using the c-compiler package from conda-forge, as reported in #28745.

The _crypt extension module of Python is deprecated in 3.8 and will be removed in 3.10 according to https://www.python.org/dev/peps/pep-0594/

This ticket proposes to remove the module from our build of Python or to ignore errors found during the build.

Change History (19)

comment:1 Changed 22 months ago by mkoeppe

More information on the failure: Python3's configure script correctly determines that crypt_r is available if linking -lcrypt. But this information is not used when building the extension. Instead it links to crypt only if find_library_file can find libcrypt - which fails. This flawed logic is still present in CPython master (3.9.0 alpha 2) at https://github.com/python/cpython/blob/master/setup.py (https://github.com/python/cpython/blob/7d6378051feeadf45b4ce45b4b406b65df255648/setup.py#L1035)

  def detect_crypt(self):
        # crypt module.
       [...]

        if self.compiler.find_library_file(self.lib_dirs, 'crypt'):
            libs = ['crypt']
        else:
            libs = []

        self.add(Extension('_crypt', ['_cryptmodule.c'],
                               libraries=libs))
}}

comment:2 Changed 22 months ago by mkoeppe

comment:3 Changed 22 months ago by dimpase

isn't it easy to patch cpython to avoid said find_library_file ? E.g., make sure pyconfig.h.in gets a variable which is #defined if r_crypt is available from libcrypt, and call sysconfig.get_config_var() in setup.py. I did something like this in (rejected, for unrelated reasons) https://github.com/python/cpython/pull/12825

we can do a cpython PR for this issue...

comment:4 Changed 22 months ago by mkoeppe

My guess is that this patch will be rejected because upstream is already phasing out crypt (and also nis, which is hitting a similar problem).

comment:5 follow-up: Changed 22 months ago by mkoeppe

Update: It seems that installing the Debian package dpkg-dev (a dependency of build-essential), which provides dpkg-architecture, makes Python recognize the multiarch structure on Debian, and the problem seems to go away.

comment:7 in reply to: ↑ 5 Changed 22 months ago by mkoeppe

Replying to mkoeppe:

... and the problem seems to go away.

Actually, no.

comment:8 follow-ups: Changed 22 months ago by embray

What is the actual error from Python when building the crypt module? I've never had a problem with it before.

comment:9 in reply to: ↑ 8 Changed 22 months ago by mkoeppe

Replying to embray:

What is the actual error from Python when building the crypt module? I've never had a problem with it before.

#28745 has the Dockerfile that can be used to reproduce it, based on continuumio/miniconda3:latest (Debian)

The errors after adding dpkg-dev look like this:

[python3-3.7.3.p1] LD_LIBRARY_PATH=/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src CC='/opt/conda/envs/sage/bin/x86_64-conda_cos6-linux-gnu-cc -pthread' LDSHARED='/opt/conda/envs/sage/bin/x86_64-
conda_cos6-linux-gnu-cc -pthread -shared -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-
rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib -L. -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,re
lro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib  ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'
    _TCLTK_INCLUDES='' _TCLTK_LIBS=''       ./python -E ./setup.py  build
[python3-3.7.3.p1] running build
[python3-3.7.3.p1] running build_ext
[python3-3.7.3.p1] INFO: Can't locate Tcl/Tk libs and/or headers
[python3-3.7.3.p1] building '_struct' extension
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules
[python3-3.7.3.p1] /opt/conda/envs/sage/bin/x86_64-conda_cos6-linux-gnu-cc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -march=nocona -mtune=haswell -ftr
ee-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/conda/envs/sage/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-i
nitializers -Werror=implicit-function-declaration -I./Include -I/sage/local/include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Incl
ude -I/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src -c /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c -o build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3
-3.7.3.p1/src/Modules/_struct.o
[python3-3.7.3.p1] In file included from /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/features.h:356:0,
[python3-3.7.3.p1]                  from /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/limits.h:27,
[python3-3.7.3.p1]                  from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/limits.h:194,
[python3-3.7.3.p1]                  from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/syslimits.h:7,
[python3-3.7.3.p1]                  from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/limits.h:34,
[python3-3.7.3.p1]                  from ./Include/Python.h:11,
[python3-3.7.3.p1]                  from /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c:8:
[python3-3.7.3.p1] /usr/include/x86_64-linux-gnu/sys/cdefs.h:482:49: error: missing binary operator before token "("
[python3-3.7.3.p1]  #if __GNUC_PREREQ (4,8) || __glibc_clang_prereq (3,5)
[python3-3.7.3.p1]                                                  ^
[python3-3.7.3.p1] In file included from ./Include/Python.h:25:0,
[python3-3.7.3.p1]                  from /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c:8:
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:49:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'typedef'
[python3-3.7.3.p1]  typedef struct _IO_FILE FILE;
[python3-3.7.3.p1]  ^~~~~~~
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:50:1: error: unknown type name '__END_NAMESPACE_STD'
[python3-3.7.3.p1]  __END_NAMESPACE_STD
[python3-3.7.3.p1]  ^~~~~~~~~~~~~~~~~~~
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h: In function '__USING_NAMESPACE_STD':
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:65:25: error: storage class specified for parameter '__FILE'

followed by another 366MB of error messages

comment:10 Changed 22 months ago by embray

Yikes! I take it this is clang, not gcc?

comment:11 Changed 22 months ago by isuruf

This is mixing the sysroot in conda's gcc /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/ and the system headers in /usr/include/x86_64-linux-gnu/ which won't work.

comment:12 in reply to: ↑ 8 Changed 21 months ago by mkoeppe

Replying to embray:

What is the actual error from Python when building the crypt module? I've never had a problem with it before.

See https://github.com/mkoeppe/sage/runs/412042067 for the logs

comment:13 Changed 21 months ago by mkoeppe

isuruf's comment on #29053: Looks like python's setup.py doesn't find the conda compiler's sysroot. ​https://github.com/python/cpython/blob/v3.7.3/setup.py#L867

comment:15 Changed 20 months ago by mkoeppe

I don't think this patch can help -- we are not attempting a cross compile.

comment:16 Changed 18 months ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

Moving tickets to milestone sage-9.2 based on a review of last modification date, branch status, and severity.

comment:17 Changed 12 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:18 Changed 7 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

comment:19 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5
Note: See TracTickets for help on using tickets.