upgrade Python to 2.7.9

[Dima Pasechnik]

2.7.9 is out for a while already, and it fixes a number of bugs and backports stuff from 3.*.

​Python 2.7.9 NEWS

The upstream tarball is here: ​https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz put it into SAGEROOT/upstream and rename to python-2.7.9.tar.gz

[Leif Leonhardy]

There's 2.7.10rc0 already...

[Dima Pasechnik]

We must now remove our patch for Include/node.h, as it has been applied upstream. The rest of our patches apply (some with fizz).

[Dima Pasechnik]

Replying to leif:

There's 2.7.10rc0 already...

2.7.10.rc1 is on its way, cf ​https://hg.python.org/cpython/rev/80ccce248ba2

[Leif Leonhardy]

Yep. Then wait for 2.7.10?

[Dima Pasechnik]

Replying to leif:

Yep. Then wait for 2.7.10?

well, after removing the node.h patch, the whole thing builds and make ptest passes on Linux. So it does not look hard.

[Leif Leonhardy]

Linux that is... ;-)

With the SSL changes, $SOMEONE should check the notebook stuff... (I don't use it at all.)

[Dima Pasechnik]

New commits:

​e302b13

a 1st shot at upgrading to 2.7.9

[François Bissey]

For the record, sage-on-gentoo has been on python 2.7.9 from at least the 27th of December 2014. I haven't heard a report of negative impact on anything. And I don't remember having to do anything special. I have a vague memory of a schedule for the release of 2.7.10 in June but I cannot find it again.

[Volker Braun]

ready for review?

[Dima Pasechnik]

Replying to vbraun:

ready for review?

I have to look at the remaining patches, if all are still needed, and rebase ones that apply with fizz. (Is the latter necessary?)

[Leif Leonhardy]

Fuzz isn't nice. One should look at the files after patching, then just rebasing the patches in case the context changes are really harmless.

[Leif Leonhardy]

If that's too tedious, I think we could simply wait for 2.7.10, in order to not have to do it twice, who knows.

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​e57fde2	Merge remote-tracking branch 'trac/u/isuruf/cmake_optional_package' into develop
​f725dee	update cmake to 3.2.2
​c48705c	Add lidia spkg
​a034626	Allow latte_int package to find the lidia package if it is installed
​7f7fbc4	Merge remote-tracking branch 'trac/u/mkoeppe/make_lidia_an_experimental_package' into develop
​bb352e9	Merge branch 'develop' of git://trac.sagemath.org/sage into develop
​bc053d8	Revert "update cmake to 3.2.2"
​45297d5	Revert "Merge remote-tracking branch 'trac/u/isuruf/cmake_optional_package' into develop"
​68dd025	Merge branch 'u/dimpase/18397' of git://trac.sagemath.org/sage into p279
​f87043c	fixed patches offset and fuzz;

[Dima Pasechnik]

oops, I only meant to add ​http://git.sagemath.org/sage.git/commit/?id=f87043cf6a893b305e5208a89986047ad95015cd

So I only need ​http://git.sagemath.org/sage.git/commit/?h=f87043cf6a893b305e5208a89986047ad95015cd&id=e302b1332b67660f2c51c11369983eeeb5e9af8b and the latter.

[Leif Leonhardy]

Yeees...

Reset hard and push with force again?

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​b5c3c24

fixed patches offset and fuzz;

[Dima Pasechnik]

OK, clean patch now.

[Dima Pasechnik]

On OSX I cannot install pyopenssl (it does work on Linux though), it ends with some weird error:

   sage --pip install pyopenssl
...
    building '_Cryptography_cffi_2a871178xb3816a41' extension
    gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/src/sage/sage/local/include/python2.7 -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_2a871178xb3816a41.c -o build/temp.macosx-10.9-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_2a871178xb3816a41.o
    In file included from /System/Library/Frameworks/Security.framework/Headers/SecDigestTransform.h:28:0,
                     from src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_2a871178xb3816a41.c:236:
    /System/Library/Frameworks/Security.framework/Headers/SecTransform.h:579:15: error: expected identifier or '(' before '^' token
     typedef void (^SecMessageBlock)(CFTypeRef message, CFErrorRef error, 
                   ^
    /System/Library/Frameworks/Security.framework/Headers/SecTransform.h:612:8: error: unknown type name 'SecMessageBlock'
            SecMessageBlock deliveryBlock) 
            ^
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
    Command "/usr/local/src/sage/sage/local/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/v8/mh06bfhd4j18mgftw8818gxw0000gp/T/pip-build-sVMlYI/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/v8/mh06bfhd4j18mgftw8818gxw0000gp/T/pip-iZFMxa-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/v8/mh06bfhd4j18mgftw8818gxw0000gp/T/pip-build-sVMlYI/cryptography

perhaps with 2.7.9 pyopenssl is obsolete? e.g. google finds: ​https://github.com/basho/riak-python-client/pull/397

Avoid dependency on pyOpenSSL when using Python 2.7.9

Many enhancements of the Python 3 ssl module have been backported to Python 2.7.9.
In particular, the class SSLContext() is available and could be used instead of relying 
on the pyOpenSSL module. 

[Leif Leonhardy]

Objective C again, isn't it?

[Volker Braun]

Yup, just another bug in Apple's toolchain headers...

[Dima Pasechnik]

Replying to vbraun:

Yup, just another bug in Apple's toolchain headers...

actually it is known to pyopenssl people : ​https://github.com/pyca/pyopenssl/issues/204

[Leif Leonhardy]

But it's probably caused by misconfiguration through Python / setuptools / pip / distutils...

Does the log show anything unusual?

Does for example our old PyOpenSSL build with the same Python?

[Dima Pasechnik]

Replying to leif:

But it's probably caused by misconfiguration through Python / setuptools / pip / distutils...

Does the log show anything unusual?

log shows a bunch of modules that were not needed before.

Does for example our old PyOpenSSL build with the same Python?

Old one? What one? well, I don't even know if pyopenssl builds on OSX with Python 2.7.8 in Sage.

PyOpenSSL people tell me to file a bug here: ​https://github.com/pyca/cryptography/issues (i.e. for ​https://cryptography.io) I'll see if it works if I try to install it separately.

[Dima Pasechnik]

Opened ​https://github.com/pyca/cryptography/issues/1924

On the other hand, this is only needed for notebook(secure=True) which is probably not an issue on OSX, as people don't run servers on OSX nowadays.

Porting sagenb to python's SSL does not look easy, as it means porting Twisted (which is still not supporting Python 3, and relies on pyopenssl).

[Dima Pasechnik]

Replying to dimpase:

Opened ​https://github.com/pyca/cryptography/issues/1924

already ​confirmed that we are out of luck here.

[Leif Leonhardy]

Replying to dimpase:

Does for example our old PyOpenSSL build with the same Python?

Old one? What one?

The "deleted" one, still on the mirrors. (Also with sage -i ... rather than pip.)

well, I don't even know if pyopenssl builds on OSX with Python 2.7.8 in Sage.

Well, that was my question. If it didn't build before, then it's no regression.

I was also wondering whether anybody needs/uses PyOpenSSL/a (secure) notebook server on MacOS X at all. (I recall long time ago somebody was trying to run a server under MacOS X, but gave up IIRC.)

[Dima Pasechnik]

indeed, pyopenssl not building on OSX is not a regression - it's the same with Python 2.7.8.

Any other packages that might use ssl? Otherwise it seems good to go.

[Dima Pasechnik]

Replying to leif:

Replying to dimpase:

Does for example our old PyOpenSSL build with the same Python?

Old one? What one?

The "deleted" one, still on the mirrors. (Also with sage -i ... rather than pip.)

well, I don't even know if pyopenssl builds on OSX with Python 2.7.8 in Sage.

Well, that was my question. If it didn't build before, then it's no regression.

I was also wondering whether anybody needs/uses PyOpenSSL/a (secure) notebook server on MacOS X at all. (I recall long time ago somebody was trying to run a server under MacOS X, but gave up IIRC.)

probably ​https://groups.google.com/d/msg/sage-support/ERrL6r-u6Hc/RdlocmPuGTAJ ?

It looks like it never worked on recent OSX, as notebook.setup() calls certtools, which on OSX is a completely different beast from Linux's one.

While I am at it, ​https://github.com/pyca/cryptography/issues/1924#issuecomment-101266143 tells of a workaround on how to install the thing on OSX - and indeed it works, thus I was able to install pyopenssl on OSX. But the above notebook.setup() problem is still there, for sure, i.e. notebook(secure=True) still does not run. This is for another ticket.

[Leif Leonhardy]

Replying to dimpase:

Replying to leif:

(I recall long time ago somebody was trying to run a server under MacOS X, but gave up IIRC.)

probably ​https://groups.google.com/d/msg/sage-support/ERrL6r-u6Hc/RdlocmPuGTAJ ?

No, much older.

It looks like it never worked on recent OSX, as notebook.setup() calls certtools, which on OSX is a completely different beast from Linux's one.

While I am at it, ​https://github.com/pyca/cryptography/issues/1924#issuecomment-101266143 tells of a workaround on how to install the thing on OSX - and indeed it works, thus I was able to install pyopenssl on OSX.

Fine. So you found a solution to something we didn't want to fix... ;-)

(Ok, we don't fix it here. And after all, it's rather up to Apple.)

But the above notebook.setup() problem is still there, for sure, i.e. notebook(secure=True) still does not run. This is for another ticket.

Well, that's an "upstream" sagenb problem. Probably open an issue there.

[Karl-Dieter Crisman]

See ​https://github.com/sagemath/sagenb/issues/341 which Dima opened for that.

[Dima Pasechnik]

Any issues still to be resolved here?

[Volker Braun]

please fill in your name