Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#29418 closed enhancement (fixed)

sage-download-file: Fix certificate problems with https downloads from upstream_url when sage-system-python is XCode's python3 on macOS

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.1
Component: build Keywords:
Cc: vbraun, dimpase, fbissey, jhpalmieri, vdelecroix, chapoton, gh-kliem Merged in:
Authors: Matthias Koeppe Reviewers: Jonathan Kliem
Report Upstream: N/A Work issues:
Branch: 90ea00b (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

#26351 added an optional upstream_url field to build/pkgs/*/checksums.ini. It streamlines the procedure for testing upgrade tickets: Developers or automatic testing facilities can pass an extra flag -o to sage-spkg to allow downloading from upstream rather than from Sage mirrors (where the updated ticket will be made available later only).

Many upstream package URLs use the https protocol - in contrast to the http protocol used when downloading from the Sage mirrors. The downloading is done via build/bin/sage-download-file, which uses the urllib module. It supports the https protocol.

However, SSL certificate problems are common on test systems. For example, if one uses XCode's python3 as the system python, then urllib does not automatically uses the standard system certificates. (This is apparently a known issue -- which is considered "wontfix" by Apple as reported here: https://github.com/HandBrake/HandBrake/issues/2216#issuecomment-527114519)

We add an option --no-check-certificate to sage-download-file, disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification).

Developers can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when installing packages (either by make or by using sage -i).

We note that even with SSL certificates disabled, there is still cryptographic protection because of the checksums recorded in checksums.ini.


Other possible workarounds considered:

  • switching from using urllib directly to the requests library
  • passing cafile=..., capath=... to urllib, perhaps coming from an environment variable
  • using python instead of python3 on macOS as system python

As of #29090 (sage-system-python fixup) prefers /usr/bin/python3 over /usr/bin/python, leading to:

  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

(see https://github.com/mkoeppe/sage/runs/538432620)

Change History (38)

comment:1 Changed 3 years ago by mkoeppe

Description: modified (diff)

comment:4 Changed 3 years ago by mkoeppe

Description: modified (diff)
Summary: sage-download-file: Add option --no-check-certificatesage-download-file: Fix certificate problems with https downloads on macOS python3

comment:5 Changed 3 years ago by mkoeppe

Branch: u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3

comment:6 Changed 3 years ago by dimpase

Commit: bd24641560065f4ea4cd78139c11b324b98303b6

that's why I suggested not to bother with MacOS/Xcode's python(3). Can't one get Python3 from python.org in MacOS by running a script?


New commits:

bd24641build/bin/sage-download-file: Add option --no-check-certificate

comment:7 Changed 3 years ago by mkoeppe

Authors: Matthias Koeppe
Branch: u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3
Cc: dimpase fbissey jhpalmieri added
Commit: bd24641560065f4ea4cd78139c11b324b98303b6
Description: modified (diff)
Status: newneeds_review

comment:8 in reply to:  6 ; Changed 3 years ago by mkoeppe

Replying to dimpase:

that's why I suggested not to bother with MacOS/Xcode's python(3). Can't one get Python3 from python.org in MacOS by running a script?

Please... the idea is to make Sage installation easier, not harder.

comment:9 Changed 3 years ago by mkoeppe

One often runs into these certificate errors also with random Linux images when I run them on Docker. So this workaround is valuable for the automatic testing in any case.

comment:10 Changed 3 years ago by mkoeppe

Branch: u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3

comment:11 Changed 3 years ago by mkoeppe

Commit: e143939a2d434211a7607c6f9830496bce34dacd

Testing this at https://github.com/mkoeppe/sage/actions/runs/65346154


New commits:

bd24641build/bin/sage-download-file: Add option --no-check-certificate
e143939build/bin/sage-spkg: Append to SAGE_DOWNLOAD_FILE_OPTIONS instead of overwriting it

comment:12 in reply to:  8 ; Changed 3 years ago by dimpase

Replying to mkoeppe:

Replying to dimpase:

that's why I suggested not to bother with MacOS/Xcode's python(3). Can't one get Python3 from python.org in MacOS by running a script?

Please... the idea is to make Sage installation easier, not harder.

well, it seems that using system's python would make a slighly cryptographically broken Sagemath, one way or another.

comment:13 in reply to:  12 Changed 3 years ago by mkoeppe

Replying to dimpase:

well, it seems that using system's python would make a slighly cryptographically broken Sagemath, one way or another.

That's not a regression from the current status on macOS, see #29291 for example

comment:14 Changed 3 years ago by mkoeppe

And, of course, the present ticket is about sage-system-python, which is only used for bootstrapping and build. Unrelated to the use of a system python3 for venv (#27824), which has a completely separate test for what it accepts.

comment:15 Changed 3 years ago by dimpase

oy gevalt, pythons everywhere!

comment:16 Changed 3 years ago by mkoeppe

Summary: sage-download-file: Fix certificate problems with https downloads on macOS python3sage-download-file: Fix certificate problems with https downloads when sage-system-python is XCode's python3 on macOS

comment:17 in reply to:  15 Changed 3 years ago by mkoeppe

Replying to dimpase:

oy gevalt, pythons everywhere!

This will be addressed in #45678 "switch Sage from Python to Julia"

comment:18 Changed 3 years ago by git

Commit: e143939a2d434211a7607c6f9830496bce34dacd90ea00b82c4d1ade6ba5c9bbecbc6388dbedefde

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

cf27321build/bin/sage-download-file: Add option --no-check-certificate
90ea00bbuild/bin/sage-spkg: Append to SAGE_DOWNLOAD_FILE_OPTIONS instead of overwriting it

comment:19 Changed 3 years ago by mkoeppe

Rebased, needs review

comment:20 Changed 3 years ago by mkoeppe

Cc: vdelecroix added

comment:21 Changed 3 years ago by mkoeppe

Cc: chapoton added

comment:22 Changed 3 years ago by mkoeppe

Cc: gh-kliem added

comment:23 Changed 3 years ago by jhpalmieri

Okay, I'm really confused, or my Sage installation is broken, or both. If I use ./sage -i beautifulsoup4 with a build using the system Python on OS X, it fails, but not because of the issues here. Instead, I see this:

make[2]: *** No rule to make target `/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/SYSTEM/sage-9.1.beta9/local/var/lib/sage/installed/beautifulsoup4-none', needed by `all-sage'.  Stop.

If instead I run make beautifulsoup4, it succeeds. When I ran ./sage -i tornado it didn't seem to even try to install it, but ./sage -f tornado worked.

So: why does ./sage -i ... not work? And what is the point of this ticket: what problem does it solve?

comment:24 Changed 3 years ago by mkoeppe

sage -i was changed in #29113 -- unrelated to the present ticket

comment:25 Changed 3 years ago by mkoeppe

Could you try if the problem goes away if you run bootstrap?

comment:26 Changed 3 years ago by mkoeppe

Description: modified (diff)

comment:27 Changed 3 years ago by jhpalmieri

If I run bootstrap, then running ./sage -i benzene acts like ./sage -i tornado: it doesn't install. There is a message

running ./configure '--enable-beautifulsoup4' '--enable-biopython'  '--enable-benzene'

and then if I search my terminal window for benzene, the only other hit is

benzene-20130630:                            does not support check for system package; optional, use "./configure --enable-benzene" to install

I don't think I tested #29113 well enough when I was reviewing it.

comment:28 Changed 3 years ago by jhpalmieri

This fixes it for me:

  • src/bin/sage

    diff --git a/src/bin/sage b/src/bin/sage
    index 10acddcd96..6ccff3789c 100755
    a b if [ "$1" = '-i' ]; then 
    404404        #   'CC=gcc -Wall' '--enable-e_antic'
    405405        CONFIG_CMD="./configure $(./config.status --config) $ENABLE_ARGS"
    406406        echo >&2 "running $CONFIG_CMD"
    407         bash -c "$CONFIG_CMD" && $MAKE all-build
     407        bash -c "$CONFIG_CMD" && $MAKE $PACKAGES
    408408    else
    409409        echo "New packages may have been installed."
    410410        echo "Re-running configure and make in case any dependent packages need updating."

(or maybe it could be $MAKE all-build $PACKAGES?)

By the way, I still don't understand the goal of this ticket.

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:29 Changed 3 years ago by mkoeppe

Let's please take the issues with sage -i to the new ticket #29481. Thanks for catching this.

comment:30 Changed 3 years ago by mkoeppe

Description: modified (diff)
Summary: sage-download-file: Fix certificate problems with https downloads when sage-system-python is XCode's python3 on macOSsage-download-file: Fix certificate problems with https downloads from upstream_url when sage-system-python is XCode's python3 on macOS

comment:31 in reply to:  28 ; Changed 3 years ago by mkoeppe

Replying to jhpalmieri:

By the way, I still don't understand the goal of this ticket.

Reworked the ticket description, please take a look

comment:32 in reply to:  31 Changed 3 years ago by jhpalmieri

Replying to mkoeppe:

Replying to jhpalmieri:

By the way, I still don't understand the goal of this ticket.

Reworked the ticket description, please take a look

Sorry, I wasn't clear enough. What system configuration and commands lead to the problem in the description? I can't reproduce the problem, so I can't tell if the solution here works.

comment:33 Changed 3 years ago by gh-kliem

From what I understand this is a problem that only occurs when one has python3 from Xcode. This doesn't ship certify and never will (at least Apple doesn't have any intentions). So it is really difficult to establish an ssl connection with that.

Do you have successfully tested this ticket?

Do I understand correctly that the problem occurring could be theoretically fixed by manually downloading the correct packages into the upstream folder? (Yes this is not a good approach for testing environments.)

Also this flag would never be needed for a normal user as long as the sage mirrors don't use ssl?

comment:34 in reply to:  33 Changed 3 years ago by mkoeppe

Replying to gh-kliem:

From what I understand this is a problem that only occurs when one has python3 from Xcode. This doesn't ship certify and never will (at least Apple doesn't have any intentions). So it is really difficult to establish an ssl connection with that.

That's a correct description for macOS. But the problem also appears on Linux distributions if one does not install ca-certificates, or those are outdated.

Do you have successfully tested this ticket?

Yes, I have been using this as part of #29417 since end of March.

Do I understand correctly that the problem occurring could be theoretically fixed by manually downloading the correct packages into the upstream folder? (Yes this is not a good approach for testing environments.)

That's correct for the macOS tests using tox local. For the Linux tests with tox docker note that /upstream is in .gitignore (therefore in .dockerignore) and is therefore not provided to the container.

Also this flag would never be needed for a normal user as long as the sage mirrors don't use ssl?

That's right, normal users would not use the upstream_url at all; it is a feature for developers only.

comment:35 Changed 3 years ago by gh-kliem

Reviewers: Jonathan Kliem
Status: needs_reviewpositive_review

LGTM.

Btw, I have been using this ticket to test my tickets, e.g. here: https://github.com/kliem/sage-test-27122/actions/runs/72102779

comment:36 Changed 3 years ago by mkoeppe

Thank you!

comment:37 Changed 3 years ago by vbraun

Branch: u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python390ea00b82c4d1ade6ba5c9bbecbc6388dbedefde
Resolution: fixed
Status: positive_reviewclosed

comment:38 Changed 2 years ago by mkoeppe

Commit: 90ea00b82c4d1ade6ba5c9bbecbc6388dbedefde

Follow up: #30950

Note: See TracTickets for help on using tickets.