Python3 is broken on OS X

[jhpalmieri]

On OS X, Python3 sometimes fails to build correctly. To fix this:

• make xz a dependency: not necessary for Python 3 to build, but this way it builds the _lzma module.

• use clang to compile Python 3, as we do for Python 2. Without this, the _scproxy module fails to build.

• patch Include/pyport.h as we do for Python 2 -- see #22147, and also ​https://trac.macports.org/ticket/44288 and ​​http://bugs.python.org/issue10910.

• do not create a symlink SAGE_LOCAL/lib/python -- see #22764. If this symlink is in place and if Python 2 is built before Python 3, then Python 3 picks up that link in its load path, and so it tries to read a lot of Python 2 packages, which is bad.

• see also #22781 which tracks a few other issues related to Python 3, OS X, and Sage packages.

[jhpalmieri]

Here is a start, but it doesn't fix the problem with Python 2 breaking Python 3, since I don't know what's going on. I bumped the version number to force a rebuild, since anyone using OS X is likely to currently have a broken Python 3 build.

---

New commits:

​c050438

trac 22756: fix python3 build on OS X:

[jhpalmieri]

By the way, some of the changes were just to decrease differences between the spkg-install scripts for python2 and python3. There is two parts in the python2 script which is not in python3:

# Use EXTRA_CFLAGS for user-defined CFLAGS since Python puts its own
# default flags like -O3 after CFLAGS but before EXTRA_CFLAGS.
# We also disable warnings about unused variables/functions which are
# common in Cython-generated code.
export EXTRA_CFLAGS="`testcflags.sh -Wno-unused` $CFLAGS"
unset CFLAGS

and the lines surrounding

    cp -rp "${xcode}"/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-migrator/sdk/MacOSX.sdk/usr/include/openssl "$CUR"/include

This second one is broken even for python2, so no need to do it for python3. I don't know about the first chunk.

[fbissey]

The error with python3 seems to be that the first python to build gets the link python to python$version. So the problem is that python2.7/encodings/__init__.py is not compatible with python3. Not sure what's happening to python2 when python3 is build first. It is a bit strange that it uses the link rather than the fully versioned directory. And obviously it doesn't happen on linux, or does it?

[jhpalmieri]

You're right. If I build python3 first, then its sys.path does not contain local/lib/python, but if I build it after python2, then the python3 sys.path does contain this. Why? And why isn't this a problem on linux?

[jhpalmieri]

Do we need the symlink (in local/lib) python2.7 -> python, or can we get by with two directories python2.7 and python3.5, no symlinks? It seems odd that we have to create it manually. If Python really needed it, it would create it on its own, wouldn't it?

Well, I'll try building without and see what happens.

[jhpalmieri]

Removing the symlink works for me. See #22764. I've made that a dependency.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e55ac1b

trac 22756: fix merge error with #22764.

[fbissey]

I am OK with this but I definitely want to wait for feedback on #22764.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7699f3f

trac 22756: check name of _scproxy .so file correctly for Python 3.

[jhpalmieri]

With the branch here, I am having problems building matplotlib and fpylll. For fpylll, the log file consists entirely of

Traceback (most recent call last):
  File "setup.py", line 131, in <module>
    long_description=open('README.rst').read(),
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2249: ordinal not in range(128)
Error: could not determine package name

The matplotlib log starts with

NOTE: Set SAGE_MATPLOTLIB_GUI to anything but 'no' to try to build the Matplotlib GUI.
Not building any matplotlib graphical backends.
Traceback (most recent call last):
  File "make-setup-config.py", line 34, in <module>
    config.write(configfile)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/configparser.py", line 916, in write
    self._sections[section].items(), d)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/configparser.py", line 920, in _write_section
    fp.write("[{}]\n".format(section_name))
TypeError: a bytes-like object is required, not 'str'
Warning: This package has a badly-behaved setup.py which outputs
more than the package name for 'setup.py --name'; using the last
line as the package name: matplotlib
Installing package matplotlib using pip
Waiting for shared lock to run pip install --ignore-installed --verbose --no-deps --no-index --isolated . ... ok
Ignoring indexes: https://pypi.python.org/simple
Processing /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/var/tmp/sage/build/matplotlib-1.5.1.p0/src
  Running setup.py (path:/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-9u0y0hgx-build/setup.py) egg_info for package from file:///Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/var/tmp/sage/build/matplotlib-1.5.1.p0/src
    Running command python setup.py egg_info

and then proceeds to try to build. Then later it has this:

    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib_backends__macosx_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/site-packages/numpy/core/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I/usr/X11/include -I/opt/X11/include -I. -Iextern/agg24-svn/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/include/python3.5m -c src/_macosx.m -o build/temp.macosx-10.9-x86_64-3.5/src/_macosx.o
    gcc: error: src/_macosx.m: Objective-C compiler not installed on this system
    error: command 'gcc' failed with exit status 1

I don't know if these are tied to the branch here or not, since I haven't been able to build python3 without the changes here.

[fbissey]

Hum... for matplotlib, that particular file is an objective-C module. We would have to build gcc with support for it or use native clang. fpylll should build with python3, although I am still at 3.4 in gentoo but I believe upstream tests 3.5. It feels like there is something missing in the python install regarding unicode support. Would have to check python's log.

[jhpalmieri]

I should have said: the problems with matplotlib and fpylll were when building Sage with SAGE_PYTHON3=yes. I think that may have been clear, but just in case.

[fbissey]

Was clear to me. My comments still stand.

[jhpalmieri]

Building matplotlib with clang works, by the way. It still has the TypeError at the beginning, which does seem like some kind of unicode thing.

[jhpalmieri]

Python 3 log attached. I noticed at the beginning:

configure: WARNING: unrecognized options: --enable-unicode, --disable-toolbox-glue

We should probably remove those options, either here or on another ticket.

[fbissey]

May as well clean up here. I'll look at the log, I cannot find what is causing trouble with fpylll - unless it is in the weird link to the sagecell server.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dcc3fb4

trac 22756: remove two invalid configure options for Python 3

[fbissey]

fpylll is a similar problem to ​https://github.com/theskumar/python-dotenv/issues/45. It could be linked to locale settings if I believe the last comment. According to file, the file is ASCII, but python thinks it has utf8 in it.

---

New commits:

​dcc3fb4

trac 22756: remove two invalid configure options for Python 3

---

New commits:

​dcc3fb4

trac 22756: remove two invalid configure options for Python 3

[jhpalmieri]

If I do ./sage --sh and then run python3, I don't get any errors if I execute the command open('README.rst').read() on the README.rst file from fpylll.

Edit: also no errors if I execute open('README.rst', encoding='ascii').read(), open('README.rst', encoding='utf8').read(), open('README.rst', encoding='latin1').read()

[jhpalmieri]

Oh, wait: the encoding for the file README.rst from fpyll version 0.2.3dev is utf8, and that's the one that is causing problems. So it's a bug in fpyll. I think it's been fixed upstream: the version in 0.2.4dev is ascii.

[jhpalmieri]

With the change

build/pkgs/matplotlib/make-setup-config.py

@@ -30 +30 @@
 for backend in ('gtk', 'gtkagg', 'tkagg', 'wxagg', 'macosx', 'windowing'):
     config.set('gui_support', backend,  graphical_backend)
 
-with open('src/setup.cfg', 'wb') as configfile:
+with open('src/setup.cfg', 'w') as configfile:
     config.write(configfile)

I don't get the error at the start of the matplotlib log. I don't know what the 'b' is doing there in the first place, since src/setup.cfg looks like an ascii file.

[jhpalmieri]

See #22772 for the matplotlib fix (eliminating the 'b').

[jhpalmieri]

By the way, the proposed change at #22772 means that we don't need to use clang to compile. Without this change, a configuration file is not written properly, and I think that led to the compilation problem. If you've tried building using python 3 on linux, I wonder if the matplotlib log shows a similar error there.

[fbissey]

I was already thinking about fpylll 0.2.4, for curiosity I should check if I have logs for 0.2.3 exhibiting any weird behavior. Unltimately I believe our continued use of gcc on OS X is a band aid, but that's another story.

[jhpalmieri]

Next I'm having trouble building pynac. Log file attached.

[fbissey]

The mountain of messages about assembly is coming from using gcc instead of clang. May disappear by upgrading gcc (I don't know for sure).

I have seen that before

In file included from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/basic_ios.h:37:0,
                 from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/ios:44,
                 from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/ostream:38,
                 from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/iostream:39,
                 from class_info.h:31,
                 from registrar.h:30,
                 from basic.h:40,
                 from numeric.h:49,
                 from inifcns.h:26,
                 from function_info.cpp:8:
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:247:53: error: macro "toupper" passed 2 arguments, but takes just 1
       toupper(char_type *__lo, const char_type* __hi) const
                                                     ^
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:276:53: error: macro "tolower" passed 2 arguments, but takes just 1
       tolower(char_type* __lo, const char_type* __hi) const
                                                     ^
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:819:53: error: macro "toupper" passed 2 arguments, but takes just 1
       toupper(char_type *__lo, const char_type* __hi) const
                                                     ^
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:852:53: error: macro "tolower" passed 2 arguments, but takes just 1
       tolower(char_type* __lo, const char_type* __hi) const
                                                     ^

May take some time to track it back.

/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/python3.5m/pyport.h:704:29: error: 'btowc' is not a type
 #define tolower(c) towlower(btowc(c))
                             ^

yes pyport.h, I fixed that in 2.7, and the fix was added in 2.7.13 upstream. May be we need to update to python-3.5.3. see #22147.

[jhpalmieri]

I see. As far as I can tell, your fix is not in Python 3.5. I'll try the patch and see what happens.

[jhpalmieri]

That patch fixes it. I'll add it to this branch.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​678fb20

trac 22756: port a fix for pyports.h from

[jhpalmieri]

With these changes (and the changes listed at #22781), I can build almost all of Sage on OS X with SAGE_PYTHON3=yes. It breaks on the Sage library in an expected place (the use of cmp in sage/rings/real_lazy.pyx -- see #22257).

Edit: oh, except for SageNB, the build of which is disabled for Python 3 in #22787.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6762fe4

trac 22756: rebase better on top of 22764

[fbissey]

I think we may be in a state where we will be able to remove all the OS X work around of setting CC=clang by sage-8.0 and just default to build everything with clang. So in that regards we may want to start mopping up. Thoughts?

[jhpalmieri]

Replying to fbissey:

I think we may be in a state where we will be able to remove all the OS X work around of setting CC=clang by sage-8.0 and just default to build everything with clang. So in that regards we may want to start mopping up. Thoughts?

I would prefer to put the mopping up on another ticket, once #12426 is ready to go, and merge this one soon (if possible), so that Python 3 can be built on OS X. Many of the changes here have to happen whether we default to clang or not.

[jhpalmieri]

Also, I am happy to review or work on any mopping up tickets.

[jhpalmieri]

ping?

[fbissey]

Sure, I think it can go in that state. Then I can some more stuff to both python2 and 3 later. Hope I can get around to it.

[jhpalmieri]

Thank you!

Regarding clang and Python on OS X: I installed gcc 5.4.0 on a machine the other day, and then with #12426, IS_CLANG was set to 'no'. So if we remove parts of Python's spkg-install telling it to build with clang on OS X, Python might break. So although it would be nice to mop up, we have to be careful. (I'll try to test whether Python breaks if I use my own gcc to build it instead of Sage's gcc, which we know doesn't work, or clang, which does.)

[fbissey]

Replying to jhpalmieri:

Thank you!

Regarding clang and Python on OS X: I installed gcc 5.4.0 on a machine the other day, and then with #12426, IS_CLANG was set to 'no'. So if we remove parts of Python's spkg-install telling it to build with clang on OS X, Python might break. So although it would be nice to mop up, we have to be careful. (I'll try to test whether Python breaks if I use my own gcc to build it instead of Sage's gcc, which we know doesn't work, or clang, which does.)

Interesting experience. #12426 can still be tweaked, but yes we may want to prevent users to shoot themselves too much in the foot. python is somewhat ok because it is C and not C++ you can compile it with clang and mostly use it with gcc. Anything that would require clang++ would be in a load of troubles.

While I don't want to impose things too strongly, I am very much of a mind to put my foot down on the compiler to be used on OS X.

[vbraun]

Reviewer name...

[fbissey]

Sorry, I really should know better.

[jhpalmieri]

Replying to fbissey:

Interesting experience. #12426 can still be tweaked, but yes we may want to prevent users to shoot themselves too much in the foot. python is somewhat ok because it is C and not C++ you can compile it with clang and mostly use it with gcc. Anything that would require clang++ would be in a load of troubles.

While I don't want to impose things too strongly, I am very much of a mind to put my foot down on the compiler to be used on OS X.

Right now we build python2, python3 (as of this ticket), curl, psutil, and R using clang rather than gcc, because of troubles with gcc. I'm going to disable the clang build for each of these to see what happens with my own gcc-5.4.0. I'll follow up with the results on #12426.

[jhpalmieri]

By the way, thank you for your help with this ticket. Many parts of the branch came from your suggestions.

[vbraun]

Author name...

[jhpalmieri]

Replying to fbissey:

Sorry, I really should know better.

Me too.