Meta-ticket: SSL issues

[Samuel Lelièvre]

Building Sage with its own Python 3 can easily produce a Python 3 missing its _ssl module.

Python's _ssl module is only built if OpenSSL (with headers) is available, either via the system or via the openssl spkg.

The result is reduced functionality:

• it hinders installing pip packages
• it hinders working with Jupyter

See also:

• many reports on Ask Sage and sage-support
• ​sage-release: cannot start Jupyter due to missing _ssl module

Tickets:

• #30557: Add spkg-configure.m4 for openssl
• #31094 openssl: Make build more robust by clearing some environment variables
• #30674: Patch tornado so it can be imported in Python without an ssl module
• #29555: Upgrade: OpenSSL 3.0, make it a standard package
• #29291: Check for more strictly required extension module "ssl" in Python build
• #24107: Inclusion of OpenSSL, stage 1
• #32311: Upgrade: OpenSSL 3.0.0.beta2
• #23893: make openssl a dependency for pip packages
• #22620: Document that SSL is optional
• #17690: A SSL section in the "install from source code" documentation
• #30950: sage-download-file: Proper initialization of SSL certificates
• #32374 Remove pip package pyopenssl

References:

• ​https://www.python.org/dev/peps/pep-0644/ - Require OpenSSL 1.1 or newer

[Samuel Lelièvre]

Many builds of the python3 spkg without its _ssl module likely result from an oversight.

Could there be a configure flag to explicitly require building python3 without its _ssl module?

Without that flag, consider openssl a dependency of python3, and let the build fail if it's not there.

[Samuel Lelièvre]

Even when configuring with --enable-openssl, I think openssl may be built after python3.

So currently one really has to separate into two steps:

$ make openssl
$ make

or later have to repair with

$ sage -i openssl
$ sage -f python3

[Matthias Köppe]

Replying to slelievre:

Even when configuring with --enable-openssl, I think openssl may be built after python3.

Yes, this is a problem. We do not have a mechanism to order "optional dependencies".

[Matthias Köppe]

See #21700 - Packages with "optional" dependencies (a package manager's suggested/recommended packages)

[Matthias Köppe]

We could add openssl to TOOLCHAIN_DEPS as a workaround. Then it would be built at the very beginning.

[Matthias Köppe]

Replying to mkoeppe:

We could add openssl to TOOLCHAIN_DEPS as a workaround. Then it would be built at the very beginning.

This is now #30560

[Matthias Köppe]

If we are building python3 and openssl is not available as a system package and not enabled, then we should disable the packages that depend on openssl -- Jupyter notebook (and dependencies) and all source=pip packages. The latter are already optional packages by definition. But we would need give the Jupyter notebook packages a new package type: optional-enabled-by-default (#30383).

[Matthias Köppe]

Replying to slelievre:

Could there be a configure flag to explicitly require building python3 without its _ssl module?

This would effectively make openssl a standard package. I don't think we should do that because of license reasons. Warnings about the disabled features are better.

[Markus Wageringel]

I would like to mention that the SSL requirement for Jupyter seems to be a recent change – presumably the upgrade of tornado or other Jupyter packages in 9.2.beta9. This was not clear to me and part of why I was confused on the mailing list. I was actually using the Python 3 SPKG without SSL all along and until now I have not experienced any problem with this. In particular, this is not related to the Python 3.8 upgrade as I had assumed at first.

[Matthias Köppe]

This might be worth investigating more.

[Markus Wageringel]

Replying to mkoeppe:

This might be worth investigating more.

As suspected, this is a consequence of the Jupyter upgrades in #26919, merged in 9.2.beta9. The dependency on ssl in tornado/httpserver.py was added in ​this commit.

[Matthias Köppe]

Replying to gh-mwageringel:

Replying to mkoeppe:

This might be worth investigating more.

As suspected, this is a consequence of the Jupyter upgrades in #26919, merged in 9.2.beta9. The dependency on ssl in tornado/httpserver.py was added in ​this commit.

Looks like this would be easy to patch out.

[Matthias Köppe]

This is now #30674

[Matthias Köppe]

Building a Python without ssl module is happening on ubuntu-trusty (​https://github.com/sagemath/sage/runs/1553902103) in 9.3.beta4

Our configure finds system openssl; system python3 is too old, so python 3.8 is built from source; but apparently it fails to build the ssl module. (This should be investigated!)

The symptom is that pytest (a pip package that is a check-only dependency of some standard package) fails to install:

[pytest] installing. Log file: /sage/logs/pkgs/pytest.log
  [pytest] error installing, exit status 1. End of log file:
  [pytest]   WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
  [pytest]   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pytest/

It is likely that this problem has actually been around for a while and was only masked by the faulty SAGE_CHECK logic fixed in #31020.

[Matthias Köppe]

Same in ubuntu-xenial-standard.

Also in various -minimal builds, of course, where the SSL headers are not available. For example ubuntu-hirsute-minimal (​https://github.com/sagemath/sage/runs/1553902375)

[Matthias Köppe]

#31062 "tox / GH Actions: Disable testsuites of packages depending on pip packages (pytest, ...) if there is no ssl" proposes a workaround

[Matthias Köppe]

Moving it to 9.4; we seem to have an OK solution for now