Opened 3 years ago

Closed 3 years ago

#22756 closed defect (fixed)

Python3 is broken on OS X

Reported by: jhpalmieri Owned by:
Priority: major Milestone: sage-8.0
Component: python3 Keywords:
Cc: Merged in:
Authors: John Palmieri Reviewers: François Bissey
Report Upstream: N/A Work issues:
Branch: 6762fe4 (Commits) Commit: 6762fe4bced21a095fc20942e12500a5654b4e34
Dependencies: #22764 Stopgaps:

Description (last modified by jhpalmieri)

On OS X, Python3 sometimes fails to build correctly. To fix this:

  • make xz a dependency: not necessary for Python 3 to build, but this way it builds the _lzma module.
  • use clang to compile Python 3, as we do for Python 2. Without this, the _scproxy module fails to build.
  • do not create a symlink SAGE_LOCAL/lib/python -- see #22764. If this symlink is in place and if Python 2 is built before Python 3, then Python 3 picks up that link in its load path, and so it tries to read a lot of Python 2 packages, which is bad.
  • see also #22781 which tracks a few other issues related to Python 3, OS X, and Sage packages.

Attachments (2)

python3-3.5.1.p0.log (1.3 MB) - added by jhpalmieri 3 years ago.
pynac-0.7.5.log (1.1 MB) - added by jhpalmieri 3 years ago.

Change History (54)

comment:1 Changed 3 years ago by jhpalmieri

  • Description modified (diff)

comment:2 Changed 3 years ago by jhpalmieri

  • Branch set to u/jhpalmieri/python3_osx

comment:3 Changed 3 years ago by jhpalmieri

  • Commit set to c050438fa802d56738cc7084254579c5610b7358

Here is a start, but it doesn't fix the problem with Python 2 breaking Python 3, since I don't know what's going on. I bumped the version number to force a rebuild, since anyone using OS X is likely to currently have a broken Python 3 build.


New commits:

c050438trac 22756: fix python3 build on OS X:

comment:4 Changed 3 years ago by jhpalmieri

By the way, some of the changes were just to decrease differences between the spkg-install scripts for python2 and python3. There is two parts in the python2 script which is not in python3:

# Use EXTRA_CFLAGS for user-defined CFLAGS since Python puts its own
# default flags like -O3 after CFLAGS but before EXTRA_CFLAGS.
# We also disable warnings about unused variables/functions which are
# common in Cython-generated code.
export EXTRA_CFLAGS="`testcflags.sh -Wno-unused` $CFLAGS"
unset CFLAGS

and the lines surrounding

    cp -rp "${xcode}"/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-migrator/sdk/MacOSX.sdk/usr/include/openssl "$CUR"/include

This second one is broken even for python2, so no need to do it for python3. I don't know about the first chunk.

comment:5 Changed 3 years ago by fbissey

The error with python3 seems to be that the first python to build gets the link python to python$version. So the problem is that python2.7/encodings/__init__.py is not compatible with python3. Not sure what's happening to python2 when python3 is build first. It is a bit strange that it uses the link rather than the fully versioned directory. And obviously it doesn't happen on linux, or does it?

comment:6 Changed 3 years ago by jhpalmieri

You're right. If I build python3 first, then its sys.path does not contain local/lib/python, but if I build it after python2, then the python3 sys.path does contain this. Why? And why isn't this a problem on linux?

comment:7 Changed 3 years ago by jhpalmieri

Do we need the symlink (in local/lib) python2.7 -> python, or can we get by with two directories python2.7 and python3.5, no symlinks? It seems odd that we have to create it manually. If Python really needed it, it would create it on its own, wouldn't it?

Well, I'll try building without and see what happens.

comment:8 Changed 3 years ago by jhpalmieri

  • Description modified (diff)

comment:9 Changed 3 years ago by jhpalmieri

  • Description modified (diff)

comment:10 Changed 3 years ago by jhpalmieri

  • Dependencies set to #22764
  • Description modified (diff)
  • Status changed from new to needs_review

Removing the symlink works for me. See #22764. I've made that a dependency.

comment:11 Changed 3 years ago by git

  • Commit changed from c050438fa802d56738cc7084254579c5610b7358 to e55ac1b02d1c6ca1724fc15d254cf546b956e76b

Branch pushed to git repo; I updated commit sha1. New commits:

e55ac1btrac 22756: fix merge error with #22764.

comment:12 Changed 3 years ago by fbissey

I am OK with this but I definitely want to wait for feedback on #22764.

comment:13 Changed 3 years ago by git

  • Commit changed from e55ac1b02d1c6ca1724fc15d254cf546b956e76b to 7699f3f2b6ac938b12c42644cbc2047a369a2743

Branch pushed to git repo; I updated commit sha1. New commits:

7699f3ftrac 22756: check name of _scproxy .so file correctly for Python 3.

comment:14 Changed 3 years ago by jhpalmieri

With the branch here, I am having problems building matplotlib and fpylll. For fpylll, the log file consists entirely of

Traceback (most recent call last):
  File "setup.py", line 131, in <module>
    long_description=open('README.rst').read(),
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2249: ordinal not in range(128)
Error: could not determine package name

The matplotlib log starts with

NOTE: Set SAGE_MATPLOTLIB_GUI to anything but 'no' to try to build the Matplotlib GUI.
Not building any matplotlib graphical backends.
Traceback (most recent call last):
  File "make-setup-config.py", line 34, in <module>
    config.write(configfile)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/configparser.py", line 916, in write
    self._sections[section].items(), d)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/configparser.py", line 920, in _write_section
    fp.write("[{}]\n".format(section_name))
TypeError: a bytes-like object is required, not 'str'
Warning: This package has a badly-behaved setup.py which outputs
more than the package name for 'setup.py --name'; using the last
line as the package name: matplotlib
Installing package matplotlib using pip
Waiting for shared lock to run pip install --ignore-installed --verbose --no-deps --no-index --isolated . ... ok
Ignoring indexes: https://pypi.python.org/simple
Processing /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/var/tmp/sage/build/matplotlib-1.5.1.p0/src
  Running setup.py (path:/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-9u0y0hgx-build/setup.py) egg_info for package from file:///Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/var/tmp/sage/build/matplotlib-1.5.1.p0/src
    Running command python setup.py egg_info

and then proceeds to try to build. Then later it has this:

    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib_backends__macosx_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/site-packages/numpy/core/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I/usr/X11/include -I/opt/X11/include -I. -Iextern/agg24-svn/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/include/python3.5m -c src/_macosx.m -o build/temp.macosx-10.9-x86_64-3.5/src/_macosx.o
    gcc: error: src/_macosx.m: Objective-C compiler not installed on this system
    error: command 'gcc' failed with exit status 1

I don't know if these are tied to the branch here or not, since I haven't been able to build python3 without the changes here.

comment:15 Changed 3 years ago by fbissey

Hum... for matplotlib, that particular file is an objective-C module. We would have to build gcc with support for it or use native clang. fpylll should build with python3, although I am still at 3.4 in gentoo but I believe upstream tests 3.5. It feels like there is something missing in the python install regarding unicode support. Would have to check python's log.

comment:16 Changed 3 years ago by jhpalmieri

I should have said: the problems with matplotlib and fpylll were when building Sage with SAGE_PYTHON3=yes. I think that may have been clear, but just in case.

comment:17 Changed 3 years ago by fbissey

Was clear to me. My comments still stand.

comment:18 Changed 3 years ago by jhpalmieri

Building matplotlib with clang works, by the way. It still has the TypeError at the beginning, which does seem like some kind of unicode thing.

Changed 3 years ago by jhpalmieri

comment:19 Changed 3 years ago by jhpalmieri

Python 3 log attached. I noticed at the beginning:

configure: WARNING: unrecognized options: --enable-unicode, --disable-toolbox-glue

We should probably remove those options, either here or on another ticket.

comment:20 Changed 3 years ago by fbissey

May as well clean up here. I'll look at the log, I cannot find what is causing trouble with fpylll - unless it is in the weird link to the sagecell server.

comment:21 Changed 3 years ago by git

  • Commit changed from 7699f3f2b6ac938b12c42644cbc2047a369a2743 to dcc3fb4ec40f97748d525d2e1917c747151339f8

Branch pushed to git repo; I updated commit sha1. New commits:

dcc3fb4trac 22756: remove two invalid configure options for Python 3

comment:22 Changed 3 years ago by fbissey

fpylll is a similar problem to https://github.com/theskumar/python-dotenv/issues/45. It could be linked to locale settings if I believe the last comment. According to file, the file is ASCII, but python thinks it has utf8 in it.


New commits:

dcc3fb4trac 22756: remove two invalid configure options for Python 3

New commits:

dcc3fb4trac 22756: remove two invalid configure options for Python 3

comment:23 Changed 3 years ago by jhpalmieri

If I do ./sage --sh and then run python3, I don't get any errors if I execute the command open('README.rst').read() on the README.rst file from fpylll.

Edit: also no errors if I execute open('README.rst', encoding='ascii').read(), open('README.rst', encoding='utf8').read(), open('README.rst', encoding='latin1').read()

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:24 Changed 3 years ago by jhpalmieri

Oh, wait: the encoding for the file README.rst from fpyll version 0.2.3dev is utf8, and that's the one that is causing problems. So it's a bug in fpyll. I think it's been fixed upstream: the version in 0.2.4dev is ascii.

Version 0, edited 3 years ago by jhpalmieri (next)

comment:25 Changed 3 years ago by jhpalmieri

With the change

  • build/pkgs/matplotlib/make-setup-config.py

    diff --git a/build/pkgs/matplotlib/make-setup-config.py b/build/pkgs/matplotlib/make-setup-config.py
    index ad5a0c19ab..9a86586f9b 100644
    a b config.add_section('gui_support') 
    3030for backend in ('gtk', 'gtkagg', 'tkagg', 'wxagg', 'macosx', 'windowing'):
    3131    config.set('gui_support', backend,  graphical_backend)
    3232
    33 with open('src/setup.cfg', 'wb') as configfile:
     33with open('src/setup.cfg', 'w') as configfile:
    3434    config.write(configfile)

I don't get the error at the start of the matplotlib log. I don't know what the 'b' is doing there in the first place, since src/setup.cfg looks like an ascii file.

comment:26 Changed 3 years ago by jhpalmieri

See #22772 for the matplotlib fix (eliminating the 'b').

comment:27 Changed 3 years ago by jhpalmieri

By the way, the proposed change at #22772 means that we don't need to use clang to compile. Without this change, a configuration file is not written properly, and I think that led to the compilation problem. If you've tried building using python 3 on linux, I wonder if the matplotlib log shows a similar error there.

comment:28 Changed 3 years ago by fbissey

I was already thinking about fpylll 0.2.4, for curiosity I should check if I have logs for 0.2.3 exhibiting any weird behavior. Unltimately I believe our continued use of gcc on OS X is a band aid, but that's another story.

comment:29 Changed 3 years ago by jhpalmieri

Next I'm having trouble building pynac. Log file attached.

Changed 3 years ago by jhpalmieri

comment:30 Changed 3 years ago by fbissey

The mountain of messages about assembly is coming from using gcc instead of clang. May disappear by upgrading gcc (I don't know for sure).

I have seen that before

In file included from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/basic_ios.h:37:0,
                 from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/ios:44,
                 from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/ostream:38,
                 from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/iostream:39,
                 from class_info.h:31,
                 from registrar.h:30,
                 from basic.h:40,
                 from numeric.h:49,
                 from inifcns.h:26,
                 from function_info.cpp:8:
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:247:53: error: macro "toupper" passed 2 arguments, but takes just 1
       toupper(char_type *__lo, const char_type* __hi) const
                                                     ^
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:276:53: error: macro "tolower" passed 2 arguments, but takes just 1
       tolower(char_type* __lo, const char_type* __hi) const
                                                     ^
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:819:53: error: macro "toupper" passed 2 arguments, but takes just 1
       toupper(char_type *__lo, const char_type* __hi) const
                                                     ^
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:852:53: error: macro "tolower" passed 2 arguments, but takes just 1
       tolower(char_type* __lo, const char_type* __hi) const
                                                     ^

May take some time to track it back.

/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/python3.5m/pyport.h:704:29: error: 'btowc' is not a type
 #define tolower(c) towlower(btowc(c))
                             ^

yes pyport.h, I fixed that in 2.7, and the fix was added in 2.7.13 upstream. May be we need to update to python-3.5.3. see #22147.

comment:31 Changed 3 years ago by jhpalmieri

I see. As far as I can tell, your fix is not in Python 3.5. I'll try the patch and see what happens.

comment:32 Changed 3 years ago by jhpalmieri

That patch fixes it. I'll add it to this branch.

comment:33 Changed 3 years ago by git

  • Commit changed from dcc3fb4ec40f97748d525d2e1917c747151339f8 to 678fb206f5fb7731651c80263c61ac4872915396

Branch pushed to git repo; I updated commit sha1. New commits:

678fb20trac 22756: port a fix for pyports.h from

comment:34 Changed 3 years ago by jhpalmieri

  • Description modified (diff)

comment:35 Changed 3 years ago by jhpalmieri

  • Description modified (diff)

With these changes (and the changes listed at #22781), I can build almost all of Sage on OS X with SAGE_PYTHON3=yes. It breaks on the Sage library in an expected place (the use of cmp in sage/rings/real_lazy.pyx -- see #22257).

Edit: oh, except for SageNB, the build of which is disabled for Python 3 in #22787.

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:36 Changed 3 years ago by git

  • Commit changed from 678fb206f5fb7731651c80263c61ac4872915396 to 6762fe4bced21a095fc20942e12500a5654b4e34

Branch pushed to git repo; I updated commit sha1. New commits:

6762fe4trac 22756: rebase better on top of 22764

comment:37 follow-up: Changed 3 years ago by fbissey

I think we may be in a state where we will be able to remove all the OS X work around of setting CC=clang by sage-8.0 and just default to build everything with clang. So in that regards we may want to start mopping up. Thoughts?

comment:38 in reply to: ↑ 37 Changed 3 years ago by jhpalmieri

Replying to fbissey:

I think we may be in a state where we will be able to remove all the OS X work around of setting CC=clang by sage-8.0 and just default to build everything with clang. So in that regards we may want to start mopping up. Thoughts?

I would prefer to put the mopping up on another ticket, once #12426 is ready to go, and merge this one soon (if possible), so that Python 3 can be built on OS X. Many of the changes here have to happen whether we default to clang or not.

comment:39 Changed 3 years ago by jhpalmieri

Also, I am happy to review or work on any mopping up tickets.

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:40 Changed 3 years ago by chapoton

  • Component changed from packages: standard to python3

comment:41 Changed 3 years ago by jhpalmieri

ping?

comment:42 Changed 3 years ago by fbissey

  • Status changed from needs_review to positive_review

Sure, I think it can go in that state. Then I can some more stuff to both python2 and 3 later. Hope I can get around to it.

comment:43 follow-up: Changed 3 years ago by jhpalmieri

Thank you!

Regarding clang and Python on OS X: I installed gcc 5.4.0 on a machine the other day, and then with #12426, IS_CLANG was set to 'no'. So if we remove parts of Python's spkg-install telling it to build with clang on OS X, Python might break. So although it would be nice to mop up, we have to be careful. (I'll try to test whether Python breaks if I use my own gcc to build it instead of Sage's gcc, which we know doesn't work, or clang, which does.)

comment:44 in reply to: ↑ 43 ; follow-up: Changed 3 years ago by fbissey

Replying to jhpalmieri:

Thank you!

Regarding clang and Python on OS X: I installed gcc 5.4.0 on a machine the other day, and then with #12426, IS_CLANG was set to 'no'. So if we remove parts of Python's spkg-install telling it to build with clang on OS X, Python might break. So although it would be nice to mop up, we have to be careful. (I'll try to test whether Python breaks if I use my own gcc to build it instead of Sage's gcc, which we know doesn't work, or clang, which does.)

Interesting experience. #12426 can still be tweaked, but yes we may want to prevent users to shoot themselves too much in the foot. python is somewhat ok because it is C and not C++ you can compile it with clang and mostly use it with gcc. Anything that would require clang++ would be in a load of troubles.

While I don't want to impose things too strongly, I am very much of a mind to put my foot down on the compiler to be used on OS X.

comment:45 Changed 3 years ago by vbraun

  • Status changed from positive_review to needs_work

Reviewer name...

comment:46 follow-up: Changed 3 years ago by fbissey

  • Reviewers set to François Bissey
  • Status changed from needs_work to positive_review

Sorry, I really should know better.

comment:47 in reply to: ↑ 44 Changed 3 years ago by jhpalmieri

Replying to fbissey:

Interesting experience. #12426 can still be tweaked, but yes we may want to prevent users to shoot themselves too much in the foot. python is somewhat ok because it is C and not C++ you can compile it with clang and mostly use it with gcc. Anything that would require clang++ would be in a load of troubles.

While I don't want to impose things too strongly, I am very much of a mind to put my foot down on the compiler to be used on OS X.

Right now we build python2, python3 (as of this ticket), curl, psutil, and R using clang rather than gcc, because of troubles with gcc. I'm going to disable the clang build for each of these to see what happens with my own gcc-5.4.0. I'll follow up with the results on #12426.

comment:48 Changed 3 years ago by jhpalmieri

By the way, thank you for your help with this ticket. Many parts of the branch came from your suggestions.

comment:49 Changed 3 years ago by vbraun

  • Status changed from positive_review to needs_work

Author name...

comment:50 in reply to: ↑ 46 Changed 3 years ago by jhpalmieri

Replying to fbissey:

Sorry, I really should know better.

Me too.

comment:51 Changed 3 years ago by jhpalmieri

  • Authors set to John Palmieri
  • Status changed from needs_work to positive_review

comment:52 Changed 3 years ago by vbraun

  • Branch changed from u/jhpalmieri/python3_osx to 6762fe4bced21a095fc20942e12500a5654b4e34
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.