Opened 2 years ago

Last modified 2 months ago

#30556 new defect

Meta-ticket: SSL issues

Reported by: Samuel Lelièvre Owned by:
Priority: major Milestone: sage-9.8
Component: packages: standard Keywords: openssl, python
Cc: Markus Wageringel, Matthias Köppe, Samuel Lelièvre, William Stein, Dima Pasechnik Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #30557, #30560, #30383 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

Building Sage with its own Python 3 can easily produce a Python 3 missing its _ssl module.

Python's _ssl module is only built if OpenSSL (with headers) is available, either via the system or via the openssl spkg.

The result is reduced functionality:

  • it hinders installing pip packages
  • it hinders working with Jupyter

See also:

Tickets:

  • #30557: Add spkg-configure.m4 for openssl
  • #31094 openssl: Make build more robust by clearing some environment variables
  • #30674: Patch tornado so it can be imported in Python without an ssl module
  • #29555: Upgrade: OpenSSL 3.0, make it a standard package
  • #29291: Check for more strictly required extension module "ssl" in Python build
  • #24107: Inclusion of OpenSSL, stage 1
  • #32311: Upgrade: OpenSSL 3.0.0.beta2
  • #23893: make openssl a dependency for pip packages
  • #22620: Document that SSL is optional
  • #17690: A SSL section in the "install from source code" documentation
  • #30950: sage-download-file: Proper initialization of SSL certificates
  • #32374 Remove pip package pyopenssl

References:

Change History (30)

comment:1 Changed 2 years ago by Matthias Köppe

Dependencies: #30557

comment:2 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:3 Changed 2 years ago by Samuel Lelièvre

Description: modified (diff)
Summary: Build Python's ssl moduleBuild Python's ssl module or protest loudly

Many builds of the python3 spkg without its _ssl module likely result from an oversight.

Could there be a configure flag to explicitly require building python3 without its _ssl module?

Without that flag, consider openssl a dependency of python3, and let the build fail if it's not there.

comment:4 Changed 2 years ago by Samuel Lelièvre

Description: modified (diff)

Even when configuring with --enable-openssl, I think openssl may be built after python3.

So one really has to separate into two steps:

$ make openssl
$ make

or later have to repair with

$ sage -i openssl
$ sage -f python3
Version 0, edited 2 years ago by Samuel Lelièvre (next)

comment:5 in reply to:  4 Changed 2 years ago by Matthias Köppe

Replying to slelievre:

Even when configuring with --enable-openssl, I think openssl may be built after python3.

Yes, this is a problem. We do not have a mechanism to order "optional dependencies".

comment:6 Changed 2 years ago by Matthias Köppe

See #21700 - Packages with "optional" dependencies (a package manager's suggested/recommended packages)

comment:7 Changed 2 years ago by Matthias Köppe

We could add openssl to TOOLCHAIN_DEPS as a workaround. Then it would be built at the very beginning.

comment:8 in reply to:  7 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

We could add openssl to TOOLCHAIN_DEPS as a workaround. Then it would be built at the very beginning.

This is now #30560

comment:9 Changed 2 years ago by Matthias Köppe

Dependencies: #30557#30557, #30560

comment:10 Changed 2 years ago by Matthias Köppe

Dependencies: #30557, #30560#30557, #30560, #30383

If we are building python3 and openssl is not available as a system package and not enabled, then we should disable the packages that depend on openssl -- Jupyter notebook (and dependencies) and all source=pip packages. The latter are already optional packages by definition. But we would need give the Jupyter notebook packages a new package type: optional-enabled-by-default (#30383).

comment:11 in reply to:  3 Changed 2 years ago by Matthias Köppe

Replying to slelievre:

Could there be a configure flag to explicitly require building python3 without its _ssl module?

This would effectively make openssl a standard package. I don't think we should do that because of license reasons. Warnings about the disabled features are better.

comment:12 Changed 2 years ago by Markus Wageringel

I would like to mention that the SSL requirement for Jupyter seems to be a recent change – presumably the upgrade of tornado or other Jupyter packages in 9.2.beta9. This was not clear to me and part of why I was confused on the mailing list. I was actually using the Python 3 SPKG without SSL all along and until now I have not experienced any problem with this. In particular, this is not related to the Python 3.8 upgrade as I had assumed at first.

comment:13 Changed 2 years ago by Matthias Köppe

This might be worth investigating more.

comment:14 in reply to:  13 ; Changed 2 years ago by Markus Wageringel

Replying to mkoeppe:

This might be worth investigating more.

As suspected, this is a consequence of the Jupyter upgrades in #26919, merged in 9.2.beta9. The dependency on ssl in tornado/httpserver.py was added in this commit.

comment:15 in reply to:  14 Changed 2 years ago by Matthias Köppe

Replying to gh-mwageringel:

Replying to mkoeppe:

This might be worth investigating more.

As suspected, this is a consequence of the Jupyter upgrades in #26919, merged in 9.2.beta9. The dependency on ssl in tornado/httpserver.py was added in this commit.

Looks like this would be easy to patch out.

comment:16 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

This is now #30674

comment:17 Changed 2 years ago by Matthias Köppe

Milestone: sage-9.2sage-9.3

comment:18 Changed 2 years ago by Matthias Köppe

Building a Python without ssl module is happening on ubuntu-trusty (https://github.com/sagemath/sage/runs/1553902103) in 9.3.beta4

Our configure finds system openssl; system python3 is too old, so python 3.8 is built from source; but apparently it fails to build the ssl module. (This should be investigated!)

The symptom is that pytest (a pip package that is a check-only dependency of some standard package) fails to install:

[pytest] installing. Log file: /sage/logs/pkgs/pytest.log
  [pytest] error installing, exit status 1. End of log file:
  [pytest]   WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
  [pytest]   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pytest/

It is likely that this problem has actually been around for a while and was only masked by the faulty SAGE_CHECK logic fixed in #31020.

comment:19 Changed 2 years ago by Matthias Köppe

Same in ubuntu-xenial-standard.

Also in various -minimal builds, of course, where the SSL headers are not available. For example ubuntu-hirsute-minimal (https://github.com/sagemath/sage/runs/1553902375)

comment:20 Changed 2 years ago by Matthias Köppe

#31062 "tox / GH Actions: Disable testsuites of packages depending on pip packages (pytest, ...) if there is no ssl" proposes a workaround

comment:21 Changed 2 years ago by Matthias Köppe

Cc: William Stein Dima Pasechnik added
Description: modified (diff)
Summary: Build Python's ssl module or protest loudlyMeta-ticket: SSL issues

comment:22 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:23 Changed 22 months ago by Matthias Köppe

Description: modified (diff)

comment:24 Changed 20 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4
Priority: criticalmajor

Moving it to 9.4; we seem to have an OK solution for now

comment:25 Changed 17 months ago by Matthias Köppe

Milestone: sage-9.4sage-9.5

comment:26 Changed 16 months ago by Matthias Köppe

Description: modified (diff)

comment:27 Changed 16 months ago by Matthias Köppe

Description: modified (diff)

comment:28 Changed 11 months ago by Matthias Köppe

Milestone: sage-9.5sage-9.6

comment:29 Changed 7 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:30 Changed 2 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8
Note: See TracTickets for help on using tickets.