Check for more strictly required extension modules in Python build: zlib sqlite3

[embray]

When building Python 2/3 the Python build system attempts to build several extension modules that are part of the stdlib, but does not fail the build if building some of those modules fails, as many of them are optional.

There are some such modules that are mandatory for Sage, so we check after the Python build that they can be imported, and force the build to fail if not:

echo "Testing importing of various modules..."
import_errors=false
test_modules="ctypes math hashlib crypt readline socket"
if [ "$UNAME" = "Darwin" ]; then
    test_modules="$test_modules _scproxy"
fi

for module in $test_modules; do
    if $PYTHON -c "import $module"; then
        echo "$module module imported OK"
    else
        echo >&2 "$module module failed to import"
        import_errors=true
    fi
done

if $import_errors; then
    echo >&2 "Error: One or more modules failed to import."
    exit 1
fi

This came up recently in light of #26899, where the zlib module was silently failing to build on some macOS versions. Hence, we should have also added zlib to this list.

It turns out also that elliptic_curves has a build-time dependency on the sqlite3 module being built properly, so we should add it as well.

Also ssl should be added, but we reserve this for a follow-up ticket that addresses the whole openssl mess on macOS.

[embray]

Correction: It's elliptic_curves that has the sqlite3 dependency.

[embray]

As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).

[embray]

I think we should also add ssl to this list, unless a flag is passed to Sage's configure explicitly requesting an SSL-less build.

[mkoeppe]

See also #29002, #27824

[mkoeppe]

Replying to embray:

I think we should also add ssl to this list, unless a flag is passed to Sage's configure explicitly requesting an SSL-less build.

That's a good idea, I think, because it avoids nasty surprises later when people try to use pip.

[mkoeppe]

Actually, is crypt used anywhere in sage? I'm having trouble building it in #29012.

[mkoeppe]

New commits:

​da5e0fc

build/pkgs/python3/spkg-build: Check that Python build has modules zlib sqlite3 ssl

[dimpase]

lgtm

I expect some noise about ssl, but well...

[mkoeppe]

Thank you!

[vbraun]

Testing importing of various modules...
ctypes module imported OK
math module imported OK
hashlib module imported OK
crypt module imported OK
readline module imported OK
socket module imported OK
zlib module imported OK
sqlite3 module imported OK
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/buildbot/slave/sage_git/build/local/var/tmp/sage/build/python3-3.7.3.p1/src/Lib/ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
ssl module failed to import
Error: One or more modules failed to import.

[dimpase]

hmm, isn't this buildbot broken?

[jhpalmieri]

I get this same error on OS X.

[dimpase]

Does moving these tests to spkg-install from spkg-build fixes this?

[jhpalmieri]

No. From earlier in the log file:

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

It works if I first install openssl via homebrew.

[dimpase]

that is, is actually works as it should, i.e. you cannot build a broken (ssh-less) Python any more?

[jhpalmieri]

If you want to force that change, then at the very least, you have to change the documentation from saying that openssl is recommended to saying that it is required. You will also be adding another obstacle to installing Sage on OS X.

[dimpase]

Replying to jhpalmieri:

If you want to force that change, then at the very least, you have to change the documentation from saying that openssl is recommended to saying that it is required. You will also be adding another obstacle to installing Sage on OS X.

Sage without a functioning openssl (and thus not fully functioning pip, probably broken jupyter too) is pretty much broken, that is, it's probably more about saving potential users time.

I ventured into this mess on MacOS last year ​https://bugs.python.org/issue36344 and the "logic" of what's going on with openssl support made me question my own sanity :-)

But perhaps Python 3.8 is an improvement.

IMHO we'd ease the suffering of everyone wanting to build/run Sage on MacOS by telling them to use Conda or Homebrew.

[mkoeppe]

For this ticket, I'll change the error for a missing openssl module into a warning.

[jhpalmieri]

Replying to dimpase:

Replying to jhpalmieri:

If you want to force that change, then at the very least, you have to change the documentation from saying that openssl is recommended to saying that it is required. You will also be adding another obstacle to installing Sage on OS X.

Sage without a functioning openssl (and thus not fully functioning pip, probably broken jupyter too) is pretty much broken, that is, it's probably more about saving potential users time.

On the same machine, vanilla Sage 9.1.beta1 runs the Jupyter notebook just fine, despite Python's failure to build the ssl module. pip is indeed broken, but it apparently hasn't bothered me before, so I am at least one user who can survive without a functioning ssl Python module.

[mkoeppe]

dup: #22776

[mkoeppe]

New commits:

​dade952	build/pkgs/python3/spkg-build: Check that Python build has modules zlib sqlite3 ssl
​8a294fb	Remove ssl from the list of required modules

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c3542d5

Merge tag '9.1.beta4' into t/27705/da5e0fcee826ad1c667c52341220c90306f0e222

[mkoeppe]

Needs review

[dimpase]

ok

[mkoeppe]

Follow-up for ssl: #29291