Opened 5 years ago

Closed 5 years ago

#18397 closed enhancement (fixed)

upgrade Python to 2.7.9

Reported by: dimpase Owned by:
Priority: major Milestone: sage-6.7
Component: packages: standard Keywords:
Cc: Merged in:
Authors: Dima Pasechnik Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: b5c3c24 (Commits) Commit: b5c3c247c1f934d7f9008b792596e9573e74b091
Dependencies: Stopgaps:

Description (last modified by dimpase)

2.7.9 is out for a while already, and it fixes a number of bugs and backports stuff from 3.*.

Python 2.7.9 NEWS

The upstream tarball is here: https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz put it into SAGEROOT/upstream and rename to python-2.7.9.tar.gz

Change History (36)

comment:1 follow-up: Changed 5 years ago by leif

There's 2.7.10rc0 already...

comment:2 Changed 5 years ago by leif

  • Description modified (diff)

comment:3 Changed 5 years ago by dimpase

We must now remove our patch for Include/node.h, as it has been applied upstream. The rest of our patches apply (some with fizz).

comment:4 in reply to: ↑ 1 Changed 5 years ago by dimpase

Replying to leif:

There's 2.7.10rc0 already...

2.7.10.rc1 is on its way, cf https://hg.python.org/cpython/rev/80ccce248ba2

comment:5 follow-up: Changed 5 years ago by leif

Yep. Then wait for 2.7.10?

comment:6 in reply to: ↑ 5 Changed 5 years ago by dimpase

Replying to leif:

Yep. Then wait for 2.7.10?

well, after removing the node.h patch, the whole thing builds and make ptest passes on Linux. So it does not look hard.

comment:7 Changed 5 years ago by leif

Linux that is... ;-)

With the SSL changes, $SOMEONE should check the notebook stuff... (I don't use it at all.)

comment:8 Changed 5 years ago by dimpase

  • Branch set to u/dimpase/18397
  • Commit set to e302b1332b67660f2c51c11369983eeeb5e9af8b

New commits:

e302b13a 1st shot at upgrading to 2.7.9

comment:9 Changed 5 years ago by dimpase

  • Description modified (diff)

comment:10 Changed 5 years ago by fbissey

For the record, sage-on-gentoo has been on python 2.7.9 from at least the 27th of December 2014. I haven't heard a report of negative impact on anything. And I don't remember having to do anything special. I have a vague memory of a schedule for the release of 2.7.10 in June but I cannot find it again.

comment:11 follow-up: Changed 5 years ago by vbraun

ready for review?

comment:12 in reply to: ↑ 11 Changed 5 years ago by dimpase

Replying to vbraun:

ready for review?

I have to look at the remaining patches, if all are still needed, and rebase ones that apply with fizz. (Is the latter necessary?)

comment:13 Changed 5 years ago by leif

Fuzz isn't nice. One should look at the files after patching, then just rebasing the patches in case the context changes are really harmless.

comment:14 Changed 5 years ago by leif

If that's too tedious, I think we could simply wait for 2.7.10, in order to not have to do it twice, who knows.

comment:15 Changed 5 years ago by git

  • Commit changed from e302b1332b67660f2c51c11369983eeeb5e9af8b to f87043cf6a893b305e5208a89986047ad95015cd

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

e57fde2Merge remote-tracking branch 'trac/u/isuruf/cmake_optional_package' into develop
f725deeupdate cmake to 3.2.2
c48705cAdd lidia spkg
a034626Allow latte_int package to find the lidia package if it is installed
7f7fbc4Merge remote-tracking branch 'trac/u/mkoeppe/make_lidia_an_experimental_package' into develop
bb352e9Merge branch 'develop' of git://trac.sagemath.org/sage into develop
bc053d8Revert "update cmake to 3.2.2"
45297d5Revert "Merge remote-tracking branch 'trac/u/isuruf/cmake_optional_package' into develop"
68dd025Merge branch 'u/dimpase/18397' of git://trac.sagemath.org/sage into p279
f87043cfixed patches offset and fuzz;

comment:17 Changed 5 years ago by leif

Yeees...

Reset hard and push with force again?

comment:18 Changed 5 years ago by git

  • Commit changed from f87043cf6a893b305e5208a89986047ad95015cd to b5c3c247c1f934d7f9008b792596e9573e74b091

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b5c3c24fixed patches offset and fuzz;

comment:19 Changed 5 years ago by dimpase

  • Status changed from new to needs_review

OK, clean patch now.

comment:20 Changed 5 years ago by dimpase

On OSX I cannot install pyopenssl (it does work on Linux though), it ends with some weird error:

   sage --pip install pyopenssl
...
    building '_Cryptography_cffi_2a871178xb3816a41' extension
    gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/src/sage/sage/local/include/python2.7 -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_2a871178xb3816a41.c -o build/temp.macosx-10.9-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_2a871178xb3816a41.o
    In file included from /System/Library/Frameworks/Security.framework/Headers/SecDigestTransform.h:28:0,
                     from src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_2a871178xb3816a41.c:236:
    /System/Library/Frameworks/Security.framework/Headers/SecTransform.h:579:15: error: expected identifier or '(' before '^' token
     typedef void (^SecMessageBlock)(CFTypeRef message, CFErrorRef error, 
                   ^
    /System/Library/Frameworks/Security.framework/Headers/SecTransform.h:612:8: error: unknown type name 'SecMessageBlock'
            SecMessageBlock deliveryBlock) 
            ^
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
    Command "/usr/local/src/sage/sage/local/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/v8/mh06bfhd4j18mgftw8818gxw0000gp/T/pip-build-sVMlYI/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/v8/mh06bfhd4j18mgftw8818gxw0000gp/T/pip-iZFMxa-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/v8/mh06bfhd4j18mgftw8818gxw0000gp/T/pip-build-sVMlYI/cryptography

perhaps with 2.7.9 pyopenssl is obsolete? e.g. google finds: https://github.com/basho/riak-python-client/pull/397

Avoid dependency on pyOpenSSL when using Python 2.7.9

Many enhancements of the Python 3 ssl module have been backported to Python 2.7.9.
In particular, the class SSLContext() is available and could be used instead of relying 
on the pyOpenSSL module. 

comment:21 Changed 5 years ago by leif

Objective C again, isn't it?

comment:22 follow-up: Changed 5 years ago by vbraun

Yup, just another bug in Apple's toolchain headers...

comment:23 in reply to: ↑ 22 Changed 5 years ago by dimpase

Replying to vbraun:

Yup, just another bug in Apple's toolchain headers...

actually it is known to pyopenssl people : https://github.com/pyca/pyopenssl/issues/204

comment:24 follow-up: Changed 5 years ago by leif

But it's probably caused by misconfiguration through Python / setuptools / pip / distutils...

Does the log show anything unusual?

Does for example our old PyOpenSSL build with the same Python?

comment:25 in reply to: ↑ 24 ; follow-up: Changed 5 years ago by dimpase

Replying to leif:

But it's probably caused by misconfiguration through Python / setuptools / pip / distutils...

Does the log show anything unusual?

log shows a bunch of modules that were not needed before.

Does for example our old PyOpenSSL build with the same Python?

Old one? What one? well, I don't even know if pyopenssl builds on OSX with Python 2.7.8 in Sage.

PyOpenSSL people tell me to file a bug here: https://github.com/pyca/cryptography/issues (i.e. for https://cryptography.io) I'll see if it works if I try to install it separately.

comment:26 follow-up: Changed 5 years ago by dimpase

Opened https://github.com/pyca/cryptography/issues/1924

On the other hand, this is only needed for notebook(secure=True) which is probably not an issue on OSX, as people don't run servers on OSX nowadays.

Porting sagenb to python's SSL does not look easy, as it means porting Twisted (which is still not supporting Python 3, and relies on pyopenssl).

comment:27 in reply to: ↑ 26 Changed 5 years ago by dimpase

Replying to dimpase:

Opened https://github.com/pyca/cryptography/issues/1924

already confirmed that we are out of luck here.

Last edited 5 years ago by dimpase (previous) (diff)

comment:28 in reply to: ↑ 25 ; follow-up: Changed 5 years ago by leif

Replying to dimpase:

Does for example our old PyOpenSSL build with the same Python?

Old one? What one?

The "deleted" one, still on the mirrors. (Also with sage -i ... rather than pip.)

well, I don't even know if pyopenssl builds on OSX with Python 2.7.8 in Sage.

Well, that was my question. If it didn't build before, then it's no regression.

I was also wondering whether anybody needs/uses PyOpenSSL/a (secure) notebook server on MacOS X at all. (I recall long time ago somebody was trying to run a server under MacOS X, but gave up IIRC.)

comment:29 Changed 5 years ago by dimpase

indeed, pyopenssl not building on OSX is not a regression - it's the same with Python 2.7.8.

Any other packages that might use ssl? Otherwise it seems good to go.

comment:30 in reply to: ↑ 28 ; follow-up: Changed 5 years ago by dimpase

Replying to leif:

Replying to dimpase:

Does for example our old PyOpenSSL build with the same Python?

Old one? What one?

The "deleted" one, still on the mirrors. (Also with sage -i ... rather than pip.)

well, I don't even know if pyopenssl builds on OSX with Python 2.7.8 in Sage.

Well, that was my question. If it didn't build before, then it's no regression.

I was also wondering whether anybody needs/uses PyOpenSSL/a (secure) notebook server on MacOS X at all. (I recall long time ago somebody was trying to run a server under MacOS X, but gave up IIRC.)

probably https://groups.google.com/d/msg/sage-support/ERrL6r-u6Hc/RdlocmPuGTAJ ?

It looks like it never worked on recent OSX, as notebook.setup() calls certtools, which on OSX is a completely different beast from Linux's one.

While I am at it, https://github.com/pyca/cryptography/issues/1924#issuecomment-101266143 tells of a workaround on how to install the thing on OSX - and indeed it works, thus I was able to install pyopenssl on OSX. But the above notebook.setup() problem is still there, for sure, i.e. notebook(secure=True) still does not run. This is for another ticket.

comment:31 in reply to: ↑ 30 Changed 5 years ago by leif

Replying to dimpase:

Replying to leif:

(I recall long time ago somebody was trying to run a server under MacOS X, but gave up IIRC.)

probably https://groups.google.com/d/msg/sage-support/ERrL6r-u6Hc/RdlocmPuGTAJ ?

No, much older.

It looks like it never worked on recent OSX, as notebook.setup() calls certtools, which on OSX is a completely different beast from Linux's one.

While I am at it, https://github.com/pyca/cryptography/issues/1924#issuecomment-101266143 tells of a workaround on how to install the thing on OSX - and indeed it works, thus I was able to install pyopenssl on OSX.

Fine. So you found a solution to something we didn't want to fix... ;-)

(Ok, we don't fix it here. And after all, it's rather up to Apple.)

But the above notebook.setup() problem is still there, for sure, i.e. notebook(secure=True) still does not run. This is for another ticket.

Well, that's an "upstream" sagenb problem. Probably open an issue there.

comment:32 Changed 5 years ago by kcrisman

See https://github.com/sagemath/sagenb/issues/341 which Dima opened for that.

comment:33 Changed 5 years ago by dimpase

Any issues still to be resolved here?

comment:34 Changed 5 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

please fill in your name

comment:35 Changed 5 years ago by dimpase

  • Authors set to Dima Pasechnik

comment:36 Changed 5 years ago by vbraun

  • Branch changed from u/dimpase/18397 to b5c3c247c1f934d7f9008b792596e9573e74b091
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.