Opened 2 years ago

Last modified 2 months ago

#30950 needs_review defect

sage-download-file: Proper initialization of SSL certificates

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.8
Component: build Keywords:
Cc: John Palmieri, Dima Pasechnik, Salvatore Stella, Thierry Monteil Merged in:
Authors: Thierry Monteil Reviewers:
Report Upstream: N/A Work issues:
Branch: u/tmonteil/sage_download_file__proper_initialization_of_ssl_certificates (Commits, GitHub, GitLab) Commit: f0ded5546d77b51a0ea1da252cd2444bef7c1207
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

(from #29418)

See ticket description of #29418 and the change made in #32527.

When done, we can switch sage-bootstrap-python back to preferring python3 on macOS.

Change History (20)

comment:1 Changed 2 years ago by John Palmieri

Is this necessary? I'm not sure how to test, but here's what I did: with #29719 (new flint), I used

$ ./configure --with-python=/usr/bin/python3 --enable-download-from-upstream-url --with-system-flint=no
$ make flint

flint was downloaded properly. Since that worked, I tried a pip package:

$ ./sage -i biopython

That also worked. What should I be testing?

Edit: hold on, trying again after editing build/bin/sage-system-python.

Last edited 2 years ago by John Palmieri (previous) (diff)

comment:2 Changed 2 years ago by Matthias Köppe

This is affecting upstream URLs that are https. You can test by replacing upstream/mirror_list by something that always fails to force download from the upstream URL.

comment:3 Changed 2 years ago by John Palmieri

I put a print statement in sage-system-python to say what Python it's using (2nd line below). I removed the attrs tarball and modified mirror_list to only include https sites. I see this at the top of the attrs log file:

[attrs-19.3.0] Found local metadata for attrs-19.3.0
[attrs-19.3.0] PYTHON=/usr/bin/python3
[attrs-19.3.0] Attempting to download package attrs-19.3.0.tar.gz from mirrors
[attrs-19.3.0] https://mirror.csclub.uwaterloo.ca/sage/spkg/upstream/attrs/attrs-19.3.0.tar.gz
[attrs-19.3.0] [......................................................................]
[attrs-19.3.0] attrs-19.3.0

comment:4 Changed 2 years ago by John Palmieri

I then modified sage_bootstrap/download/mirror_list.py:

  • build/sage_bootstrap/download/mirror_list.py

    diff --git a/build/sage_bootstrap/download/mirror_list.py b/build/sage_bootstrap/download/mirror_list.py
    index 12868bf608..3a4b9d22fc 100644
    a b class MirrorList(object): 
    200200            pass
    201201        for mirror in self.mirrors:
    202202            yield mirror
    203         # If all else fails: Try the packages we host ourselves
    204         yield 'http://sagepad.org/'
    205203
    206204    @property
    207205    def fastest(self):

and changed mirror_list to include just a nonsense site. Then

Found local metadata for attrs-19.3.0
PYTHON=/usr/bin/python3
Attempting to download package attrs-19.3.0.tar.gz from mirrors
http://uw.edu/spkg/upstream/attrs/attrs-19.3.0.tar.gz
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno socket error] [Errno socket error] [Errno 404] Not Found: '//www.washington.edu/spkg/upstream/attrs/attrs-19.3.0.tar.gz'
Attempting to download from https://pypi.io/packages/source/a/attrs/attrs-19.3.0.tar.gz
[......................................................................]
attrs-19.3.0
====================================================

comment:5 Changed 2 years ago by Matthias Köppe

This looks promising... I'm surprised that this seems to be working now. I don't think we changed anything. You don't have SAGE_DOWNLOAD_FILE_OPTIONS set in your global environment by any chance?

comment:6 Changed 2 years ago by John Palmieri

I may be too optimistic, and it may depend, for example, on the version of OS X. Does the version of Python 3 depend on the version of Xcode, or the OS version, or something else? Anyway, this testing was done on Big Sur, although the same version of Python (3.8.2) is present on 10.15.7. The previous problems were reported with Python 3.7, I think.

I don't set any environment variables with names starting "SAGE".

comment:7 Changed 2 years ago by Dima Pasechnik

what is the Python3 involved here?

comment:8 Changed 2 years ago by Dima Pasechnik

You can have Python3 installed in Homebrew, Python3 installed from python.org, and system-provided Python3.

comment:9 Changed 2 years ago by John Palmieri

In my testing, it's the system-provided Python 3, /usr/bin/python3.

comment:10 Changed 2 years ago by Matthias Köppe

Cc: Salvatore Stella added

comment:11 Changed 20 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

comment:12 Changed 16 months ago by Matthias Köppe

Milestone: sage-9.4sage-9.5

comment:13 Changed 14 months ago by Matthias Köppe

Cc: Thierry Monteil added

comment:14 Changed 14 months ago by Matthias Köppe

Description: modified (diff)

comment:15 Changed 14 months ago by Thierry Monteil

Branch: u/tmonteil/sage_download_file__proper_initialization_of_ssl_certificates

comment:16 Changed 14 months ago by Thierry Monteil

Authors: Thierry Monteil
Commit: f0ded5546d77b51a0ea1da252cd2444bef7c1207
Status: newneeds_review

Could you please tell if the proposed branch fixes your issue on MacOS ?


New commits:

f0ded55#30950 : add a default SSL context for sage_bootstrap/download

comment:17 Changed 14 months ago by Matthias Köppe

Hm... I am having difficulty reproducing the original problem on my current system... this might confirm comment:1, comment:6

comment:18 Changed 11 months ago by Matthias Köppe

Milestone: sage-9.5sage-9.6

Stalled in needs_review or needs_info; likely won't make it into Sage 9.5.

comment:19 Changed 8 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:20 Changed 2 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8
Note: See TracTickets for help on using tickets.