Opened 15 months ago

Last modified 4 months ago

#30556 new defect

Meta-ticket: SSL issues

Reported by: slelievre Owned by:
Priority: major Milestone: sage-9.5
Component: packages: standard Keywords: openssl, python
Cc: gh-mwageringel, mkoeppe, slelievre, was, dimpase Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #30557, #30560, #30383 Stopgaps:

Status badges

Description (last modified by mkoeppe)

Building Sage with its own Python 3 can easily produce a Python 3 missing its _ssl module.

Python's _ssl module is only built if OpenSSL (with headers) is available, either via the system or via the openssl spkg.

The result is reduced functionality:

  • it hinders installing pip packages
  • it hinders working with Jupyter

See also:

Tickets:

  • #30557: Add spkg-configure.m4 for openssl
  • #31094 openssl: Make build more robust by clearing some environment variables
  • #30674: Patch tornado so it can be imported in Python without an ssl module
  • #29555: Upgrade: OpenSSL 3.0, make it a standard package
  • #29291: Check for more strictly required extension module "ssl" in Python build
  • #24107: Inclusion of OpenSSL, stage 1
  • #32311: Upgrade: OpenSSL 3.0.0.beta2
  • #23893: make openssl a dependency for pip packages
  • #22620: Document that SSL is optional
  • #17690: A SSL section in the "install from source code" documentation
  • #30950: sage-download-file: Proper initialization of SSL certificates
  • #32374 Remove pip package pyopenssl

References:

Change History (27)

comment:1 Changed 15 months ago by mkoeppe

  • Dependencies set to #30557

comment:2 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:3 follow-up: Changed 15 months ago by slelievre

  • Description modified (diff)
  • Summary changed from Build Python's ssl module to Build Python's ssl module or protest loudly

Many builds of the python3 spkg without its _ssl module likely result from an oversight.

Could there be a configure flag to explicitly require building python3 without its _ssl module?

Without that flag, consider openssl a dependency of python3, and let the build fail if it's not there.

comment:4 follow-up: Changed 15 months ago by slelievre

  • Description modified (diff)

Even when configuring with --enable-openssl, I think openssl may be built after python3.

So one really has to separate into two steps:

$ make openssl
$ make

or later have to repair with

$ sage -i openssl
$ sage -f python3
Version 0, edited 15 months ago by slelievre (next)

comment:5 in reply to: ↑ 4 Changed 15 months ago by mkoeppe

Replying to slelievre:

Even when configuring with --enable-openssl, I think openssl may be built after python3.

Yes, this is a problem. We do not have a mechanism to order "optional dependencies".

comment:6 Changed 15 months ago by mkoeppe

See #21700 - Packages with "optional" dependencies (a package manager's suggested/recommended packages)

comment:7 follow-up: Changed 15 months ago by mkoeppe

We could add openssl to TOOLCHAIN_DEPS as a workaround. Then it would be built at the very beginning.

comment:8 in reply to: ↑ 7 Changed 15 months ago by mkoeppe

Replying to mkoeppe:

We could add openssl to TOOLCHAIN_DEPS as a workaround. Then it would be built at the very beginning.

This is now #30560

comment:9 Changed 15 months ago by mkoeppe

  • Dependencies changed from #30557 to #30557, #30560

comment:10 Changed 15 months ago by mkoeppe

  • Dependencies changed from #30557, #30560 to #30557, #30560, #30383

If we are building python3 and openssl is not available as a system package and not enabled, then we should disable the packages that depend on openssl -- Jupyter notebook (and dependencies) and all source=pip packages. The latter are already optional packages by definition. But we would need give the Jupyter notebook packages a new package type: optional-enabled-by-default (#30383).

comment:11 in reply to: ↑ 3 Changed 15 months ago by mkoeppe

Replying to slelievre:

Could there be a configure flag to explicitly require building python3 without its _ssl module?

This would effectively make openssl a standard package. I don't think we should do that because of license reasons. Warnings about the disabled features are better.

comment:12 Changed 15 months ago by gh-mwageringel

I would like to mention that the SSL requirement for Jupyter seems to be a recent change – presumably the upgrade of tornado or other Jupyter packages in 9.2.beta9. This was not clear to me and part of why I was confused on the mailing list. I was actually using the Python 3 SPKG without SSL all along and until now I have not experienced any problem with this. In particular, this is not related to the Python 3.8 upgrade as I had assumed at first.

comment:13 follow-up: Changed 15 months ago by mkoeppe

This might be worth investigating more.

comment:14 in reply to: ↑ 13 ; follow-up: Changed 15 months ago by gh-mwageringel

Replying to mkoeppe:

This might be worth investigating more.

As suspected, this is a consequence of the Jupyter upgrades in #26919, merged in 9.2.beta9. The dependency on ssl in tornado/httpserver.py was added in this commit.

comment:15 in reply to: ↑ 14 Changed 14 months ago by mkoeppe

Replying to gh-mwageringel:

Replying to mkoeppe:

This might be worth investigating more.

As suspected, this is a consequence of the Jupyter upgrades in #26919, merged in 9.2.beta9. The dependency on ssl in tornado/httpserver.py was added in this commit.

Looks like this would be easy to patch out.

comment:16 Changed 14 months ago by mkoeppe

  • Description modified (diff)

This is now #30674

comment:17 Changed 14 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:18 Changed 12 months ago by mkoeppe

Building a Python without ssl module is happening on ubuntu-trusty (https://github.com/sagemath/sage/runs/1553902103) in 9.3.beta4

Our configure finds system openssl; system python3 is too old, so python 3.8 is built from source; but apparently it fails to build the ssl module. (This should be investigated!)

The symptom is that pytest (a pip package that is a check-only dependency of some standard package) fails to install:

[pytest] installing. Log file: /sage/logs/pkgs/pytest.log
  [pytest] error installing, exit status 1. End of log file:
  [pytest]   WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
  [pytest]   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pytest/

It is likely that this problem has actually been around for a while and was only masked by the faulty SAGE_CHECK logic fixed in #31020.

comment:19 Changed 12 months ago by mkoeppe

Same in ubuntu-xenial-standard.

Also in various -minimal builds, of course, where the SSL headers are not available. For example ubuntu-hirsute-minimal (https://github.com/sagemath/sage/runs/1553902375)

comment:20 Changed 12 months ago by mkoeppe

#31062 "tox / GH Actions: Disable testsuites of packages depending on pip packages (pytest, ...) if there is no ssl" proposes a workaround

comment:21 Changed 12 months ago by mkoeppe

  • Cc was dimpase added
  • Description modified (diff)
  • Summary changed from Build Python's ssl module or protest loudly to Meta-ticket: SSL issues

comment:22 Changed 12 months ago by mkoeppe

  • Description modified (diff)

comment:23 Changed 10 months ago by mkoeppe

  • Description modified (diff)

comment:24 Changed 8 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4
  • Priority changed from critical to major

Moving it to 9.4; we seem to have an OK solution for now

comment:25 Changed 5 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:26 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:27 Changed 4 months ago by mkoeppe

  • Description modified (diff)
Note: See TracTickets for help on using tickets.