Opened 5 years ago
Closed 5 years ago
#22756 closed defect (fixed)
Python3 is broken on OS X
Reported by: | jhpalmieri | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-8.0 |
Component: | python3 | Keywords: | |
Cc: | Merged in: | ||
Authors: | John Palmieri | Reviewers: | François Bissey |
Report Upstream: | N/A | Work issues: | |
Branch: | 6762fe4 (Commits, GitHub, GitLab) | Commit: | 6762fe4bced21a095fc20942e12500a5654b4e34 |
Dependencies: | #22764 | Stopgaps: |
Description (last modified by )
On OS X, Python3 sometimes fails to build correctly. To fix this:
- make
xz
a dependency: not necessary for Python 3 to build, but this way it builds the_lzma
module.
- use
clang
to compile Python 3, as we do for Python 2. Without this, the_scproxy
module fails to build.
- patch
Include/pyport.h
as we do for Python 2 -- see #22147, and also https://trac.macports.org/ticket/44288 and http://bugs.python.org/issue10910.
- do not create a symlink
SAGE_LOCAL/lib/python
-- see #22764. If this symlink is in place and if Python 2 is built before Python 3, then Python 3 picks up that link in its load path, and so it tries to read a lot of Python 2 packages, which is bad.
- see also #22781 which tracks a few other issues related to Python 3, OS X, and Sage packages.
Attachments (2)
Change History (54)
comment:1 Changed 5 years ago by
- Description modified (diff)
comment:2 Changed 5 years ago by
- Branch set to u/jhpalmieri/python3_osx
comment:3 Changed 5 years ago by
- Commit set to c050438fa802d56738cc7084254579c5610b7358
comment:4 Changed 5 years ago by
By the way, some of the changes were just to decrease differences between the spkg-install
scripts for python2
and python3
. There is two parts in the python2
script which is not in python3
:
# Use EXTRA_CFLAGS for user-defined CFLAGS since Python puts its own # default flags like -O3 after CFLAGS but before EXTRA_CFLAGS. # We also disable warnings about unused variables/functions which are # common in Cython-generated code. export EXTRA_CFLAGS="`testcflags.sh -Wno-unused` $CFLAGS" unset CFLAGS
and the lines surrounding
cp -rp "${xcode}"/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-migrator/sdk/MacOSX.sdk/usr/include/openssl "$CUR"/include
This second one is broken even for python2
, so no need to do it for python3
. I don't know about the first chunk.
comment:5 Changed 5 years ago by
The error with python3
seems to be that the first python to build gets the link python to python$version. So the problem is that python2.7/encodings/__init__.py
is not compatible with python3. Not sure what's happening to python2 when python3 is build first. It is a bit strange that it uses the link rather than the fully versioned directory. And obviously it doesn't happen on linux, or does it?
comment:6 Changed 5 years ago by
You're right. If I build python3
first, then its sys.path
does not contain local/lib/python
, but if I build it after python2
, then the python3
sys.path
does contain this. Why? And why isn't this a problem on linux?
comment:7 Changed 5 years ago by
Do we need the symlink (in local/lib
) python2.7 -> python
, or can we get by with two directories python2.7
and python3.5
, no symlinks? It seems odd that we have to create it manually. If Python really needed it, it would create it on its own, wouldn't it?
Well, I'll try building without and see what happens.
comment:8 Changed 5 years ago by
- Description modified (diff)
comment:9 Changed 5 years ago by
- Description modified (diff)
comment:10 Changed 5 years ago by
- Dependencies set to #22764
- Description modified (diff)
- Status changed from new to needs_review
Removing the symlink works for me. See #22764. I've made that a dependency.
comment:11 Changed 5 years ago by
- Commit changed from c050438fa802d56738cc7084254579c5610b7358 to e55ac1b02d1c6ca1724fc15d254cf546b956e76b
Branch pushed to git repo; I updated commit sha1. New commits:
e55ac1b | trac 22756: fix merge error with #22764.
|
comment:12 Changed 5 years ago by
I am OK with this but I definitely want to wait for feedback on #22764.
comment:13 Changed 5 years ago by
- Commit changed from e55ac1b02d1c6ca1724fc15d254cf546b956e76b to 7699f3f2b6ac938b12c42644cbc2047a369a2743
Branch pushed to git repo; I updated commit sha1. New commits:
7699f3f | trac 22756: check name of _scproxy .so file correctly for Python 3.
|
comment:14 Changed 5 years ago by
With the branch here, I am having problems building matplotlib
and fpylll
. For fpylll
, the log file consists entirely of
Traceback (most recent call last): File "setup.py", line 131, in <module> long_description=open('README.rst').read(), File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2249: ordinal not in range(128) Error: could not determine package name
The matplotlib
log starts with
NOTE: Set SAGE_MATPLOTLIB_GUI to anything but 'no' to try to build the Matplotlib GUI. Not building any matplotlib graphical backends. Traceback (most recent call last): File "make-setup-config.py", line 34, in <module> config.write(configfile) File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/configparser.py", line 916, in write self._sections[section].items(), d) File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/configparser.py", line 920, in _write_section fp.write("[{}]\n".format(section_name)) TypeError: a bytes-like object is required, not 'str' Warning: This package has a badly-behaved setup.py which outputs more than the package name for 'setup.py --name'; using the last line as the package name: matplotlib Installing package matplotlib using pip Waiting for shared lock to run pip install --ignore-installed --verbose --no-deps --no-index --isolated . ... ok Ignoring indexes: https://pypi.python.org/simple Processing /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/var/tmp/sage/build/matplotlib-1.5.1.p0/src Running setup.py (path:/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-9u0y0hgx-build/setup.py) egg_info for package from file:///Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/var/tmp/sage/build/matplotlib-1.5.1.p0/src Running command python setup.py egg_info
and then proceeds to try to build. Then later it has this:
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib_backends__macosx_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/site-packages/numpy/core/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/lib/python3.5/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I/usr/X11/include -I/opt/X11/include -I. -Iextern/agg24-svn/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta0/local/include/python3.5m -c src/_macosx.m -o build/temp.macosx-10.9-x86_64-3.5/src/_macosx.o gcc: error: src/_macosx.m: Objective-C compiler not installed on this system error: command 'gcc' failed with exit status 1
I don't know if these are tied to the branch here or not, since I haven't been able to build python3 without the changes here.
comment:15 Changed 5 years ago by
Hum... for matplotlib, that particular file is an objective-C module. We would have to build gcc with support for it or use native clang. fpylll should build with python3, although I am still at 3.4 in gentoo but I believe upstream tests 3.5. It feels like there is something missing in the python install regarding unicode support. Would have to check python's log.
comment:16 Changed 5 years ago by
I should have said: the problems with matplotlib
and fpylll
were when building Sage with SAGE_PYTHON3=yes
. I think that may have been clear, but just in case.
comment:17 Changed 5 years ago by
Was clear to me. My comments still stand.
comment:18 Changed 5 years ago by
Building matplotlib
with clang
works, by the way. It still has the TypeError
at the beginning, which does seem like some kind of unicode thing.
Changed 5 years ago by
comment:19 Changed 5 years ago by
Python 3 log attached. I noticed at the beginning:
configure: WARNING: unrecognized options: --enable-unicode, --disable-toolbox-glue
We should probably remove those options, either here or on another ticket.
comment:20 Changed 5 years ago by
May as well clean up here. I'll look at the log, I cannot find what is causing trouble with fpylll - unless it is in the weird link to the sagecell server.
comment:21 Changed 5 years ago by
- Commit changed from 7699f3f2b6ac938b12c42644cbc2047a369a2743 to dcc3fb4ec40f97748d525d2e1917c747151339f8
Branch pushed to git repo; I updated commit sha1. New commits:
dcc3fb4 | trac 22756: remove two invalid configure options for Python 3
|
comment:22 Changed 5 years ago by
fpylll
is a similar problem to https://github.com/theskumar/python-dotenv/issues/45. It could be linked to locale settings if I believe the last comment.
According to file, the file is ASCII, but python thinks it has utf8 in it.
New commits:
dcc3fb4 | trac 22756: remove two invalid configure options for Python 3
|
New commits:
dcc3fb4 | trac 22756: remove two invalid configure options for Python 3
|
comment:23 Changed 5 years ago by
If I do ./sage --sh
and then run python3
, I don't get any errors if I execute the command open('README.rst').read()
on the README.rst
file from fpylll
.
Edit: also no errors if I execute open('README.rst', encoding='ascii').read()
, open('README.rst', encoding='utf8').read()
, open('README.rst', encoding='latin1').read()
comment:24 Changed 5 years ago by
Oh, wait: the encoding for the file README.rst
from fpyll version 0.2.3dev is utf8, and that's the one that is causing problems. So it's a bug in fpyll. I think it's been fixed upstream: the version in 0.2.4dev is ascii.
comment:25 Changed 5 years ago by
With the change
-
build/pkgs/matplotlib/make-setup-config.py
diff --git a/build/pkgs/matplotlib/make-setup-config.py b/build/pkgs/matplotlib/make-setup-config.py index ad5a0c19ab..9a86586f9b 100644
a b config.add_section('gui_support') 30 30 for backend in ('gtk', 'gtkagg', 'tkagg', 'wxagg', 'macosx', 'windowing'): 31 31 config.set('gui_support', backend, graphical_backend) 32 32 33 with open('src/setup.cfg', 'w b') as configfile:33 with open('src/setup.cfg', 'w') as configfile: 34 34 config.write(configfile)
I don't get the error at the start of the matplotlib
log. I don't know what the 'b' is doing there in the first place, since src/setup.cfg
looks like an ascii file.
comment:26 Changed 5 years ago by
See #22772 for the matplotlib
fix (eliminating the 'b').
comment:27 Changed 5 years ago by
By the way, the proposed change at #22772 means that we don't need to use clang
to compile. Without this change, a configuration file is not written properly, and I think that led to the compilation problem. If you've tried building using python 3 on linux, I wonder if the matplotlib log shows a similar error there.
comment:28 Changed 5 years ago by
I was already thinking about fpylll 0.2.4, for curiosity I should check if I have logs for 0.2.3 exhibiting any weird behavior. Unltimately I believe our continued use of gcc on OS X is a band aid, but that's another story.
comment:29 Changed 5 years ago by
Next I'm having trouble building pynac
. Log file attached.
Changed 5 years ago by
comment:30 Changed 5 years ago by
The mountain of messages about assembly is coming from using gcc instead of clang. May disappear by upgrading gcc (I don't know for sure).
I have seen that before
In file included from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/basic_ios.h:37:0, from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/ios:44, from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/ostream:38, from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/iostream:39, from class_info.h:31, from registrar.h:30, from basic.h:40, from numeric.h:49, from inifcns.h:26, from function_info.cpp:8: /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:247:53: error: macro "toupper" passed 2 arguments, but takes just 1 toupper(char_type *__lo, const char_type* __hi) const ^ /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:276:53: error: macro "tolower" passed 2 arguments, but takes just 1 tolower(char_type* __lo, const char_type* __hi) const ^ /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:819:53: error: macro "toupper" passed 2 arguments, but takes just 1 toupper(char_type *__lo, const char_type* __hi) const ^ /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/c++/5.4.0/bits/locale_facets.h:852:53: error: macro "tolower" passed 2 arguments, but takes just 1 tolower(char_type* __lo, const char_type* __hi) const ^
May take some time to track it back.
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta1/local/include/python3.5m/pyport.h:704:29: error: 'btowc' is not a type #define tolower(c) towlower(btowc(c)) ^
yes pyport.h, I fixed that in 2.7, and the fix was added in 2.7.13 upstream. May be we need to update to python-3.5.3. see #22147.
comment:31 Changed 5 years ago by
I see. As far as I can tell, your fix is not in Python 3.5. I'll try the patch and see what happens.
comment:32 Changed 5 years ago by
That patch fixes it. I'll add it to this branch.
comment:33 Changed 5 years ago by
- Commit changed from dcc3fb4ec40f97748d525d2e1917c747151339f8 to 678fb206f5fb7731651c80263c61ac4872915396
Branch pushed to git repo; I updated commit sha1. New commits:
678fb20 | trac 22756: port a fix for pyports.h from
|
comment:34 Changed 5 years ago by
- Description modified (diff)
comment:35 Changed 5 years ago by
- Description modified (diff)
With these changes (and the changes listed at #22781), I can build almost all of Sage on OS X with SAGE_PYTHON3=yes
. It breaks on the Sage library in an expected place (the use of cmp
in sage/rings/real_lazy.pyx
-- see #22257).
Edit: oh, except for SageNB, the build of which is disabled for Python 3 in #22787.
comment:36 Changed 5 years ago by
- Commit changed from 678fb206f5fb7731651c80263c61ac4872915396 to 6762fe4bced21a095fc20942e12500a5654b4e34
Branch pushed to git repo; I updated commit sha1. New commits:
6762fe4 | trac 22756: rebase better on top of 22764
|
comment:37 follow-up: ↓ 38 Changed 5 years ago by
I think we may be in a state where we will be able to remove all the OS X work around of setting CC=clang
by sage-8.0 and just default to build everything with clang
. So in that regards we may want to start mopping up. Thoughts?
comment:38 in reply to: ↑ 37 Changed 5 years ago by
Replying to fbissey:
I think we may be in a state where we will be able to remove all the OS X work around of setting
CC=clang
by sage-8.0 and just default to build everything withclang
. So in that regards we may want to start mopping up. Thoughts?
I would prefer to put the mopping up on another ticket, once #12426 is ready to go, and merge this one soon (if possible), so that Python 3 can be built on OS X. Many of the changes here have to happen whether we default to clang
or not.
comment:39 Changed 5 years ago by
Also, I am happy to review or work on any mopping up tickets.
comment:40 Changed 5 years ago by
- Component changed from packages: standard to python3
comment:41 Changed 5 years ago by
ping?
comment:42 Changed 5 years ago by
- Status changed from needs_review to positive_review
Sure, I think it can go in that state. Then I can some more stuff to both python2 and 3 later. Hope I can get around to it.
comment:43 follow-up: ↓ 44 Changed 5 years ago by
Thank you!
Regarding clang and Python on OS X: I installed gcc 5.4.0 on a machine the other day, and then with #12426, IS_CLANG
was set to 'no'. So if we remove parts of Python's spkg-install telling it to build with clang on OS X, Python might break. So although it would be nice to mop up, we have to be careful. (I'll try to test whether Python breaks if I use my own gcc
to build it instead of Sage's gcc, which we know doesn't work, or clang, which does.)
comment:44 in reply to: ↑ 43 ; follow-up: ↓ 47 Changed 5 years ago by
Replying to jhpalmieri:
Thank you!
Regarding clang and Python on OS X: I installed gcc 5.4.0 on a machine the other day, and then with #12426,
IS_CLANG
was set to 'no'. So if we remove parts of Python's spkg-install telling it to build with clang on OS X, Python might break. So although it would be nice to mop up, we have to be careful. (I'll try to test whether Python breaks if I use my owngcc
to build it instead of Sage's gcc, which we know doesn't work, or clang, which does.)
Interesting experience. #12426 can still be tweaked, but yes we may want to prevent users to shoot themselves too much in the foot. python
is somewhat ok because it is C and not C++ you can compile it with clang and mostly use it with gcc. Anything that would require clang++ would be in a load of troubles.
While I don't want to impose things too strongly, I am very much of a mind to put my foot down on the compiler to be used on OS X.
comment:45 Changed 5 years ago by
- Status changed from positive_review to needs_work
Reviewer name...
comment:46 follow-up: ↓ 50 Changed 5 years ago by
- Reviewers set to François Bissey
- Status changed from needs_work to positive_review
Sorry, I really should know better.
comment:47 in reply to: ↑ 44 Changed 5 years ago by
Replying to fbissey:
Interesting experience. #12426 can still be tweaked, but yes we may want to prevent users to shoot themselves too much in the foot.
python
is somewhat ok because it is C and not C++ you can compile it with clang and mostly use it with gcc. Anything that would require clang++ would be in a load of troubles.While I don't want to impose things too strongly, I am very much of a mind to put my foot down on the compiler to be used on OS X.
Right now we build python2
, python3
(as of this ticket), curl
, psutil
, and R
using clang rather than gcc, because of troubles with gcc. I'm going to disable the clang build for each of these to see what happens with my own gcc-5.4.0. I'll follow up with the results on #12426.
comment:48 Changed 5 years ago by
By the way, thank you for your help with this ticket. Many parts of the branch came from your suggestions.
comment:50 in reply to: ↑ 46 Changed 5 years ago by
comment:51 Changed 5 years ago by
- Status changed from needs_work to positive_review
comment:52 Changed 5 years ago by
- Branch changed from u/jhpalmieri/python3_osx to 6762fe4bced21a095fc20942e12500a5654b4e34
- Resolution set to fixed
- Status changed from positive_review to closed
Here is a start, but it doesn't fix the problem with Python 2 breaking Python 3, since I don't know what's going on. I bumped the version number to force a rebuild, since anyone using OS X is likely to currently have a broken Python 3 build.
New commits:
trac 22756: fix python3 build on OS X: