Opened 2 years ago

Closed 2 years ago

#31227 closed enhancement (fixed)

Accept /usr/bin/python3 from XCode 12.3 on macOS 10.15 (Catalina) and 11 (Big Sur)

Reported by: Matthias Köppe Owned by:
Priority: blocker Milestone: sage-9.3
Component: build: configure Keywords:
Cc: John Palmieri, Zachary L Scherr, Dima Pasechnik, gh-kliem, Volker Braun Merged in:
Authors: Matthias Koeppe Reviewers: Jonathan Kliem
Report Upstream: N/A Work issues:
Branch: fc8b676 (Commits, GitHub, GitLab) Commit: fc8b67644bada9ea52cbb7c91f53f763ca84f4bb
Dependencies: #31132, #30725 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

(from #31132)

#27122 determines whether CC accepts -march=native and then adds it globally to CFLAGS in build/bin/sage-build-env.

#30725 fixes this for macOS, where some packages are explicitly built with clang.

However, there is another mechanism how clang can come in: Python extension modules are compiled with the compiler listed in sysconfig. On macOS with homebrew, using /usr/bin/python3, this could again be clang, which does not accept -march=native in combination with -arch arm64 (which is provided even on x86_64 if ARCHFLAGS is unset).

In this ticket, we fix this by checking whether the compiler used by distutils accepts -march=native (or more generally, the flags determined earlier by configure). If it does not, then we disable the use of these flags.

In order to test this with /usr/bin/python3 from XCode 12.3 (on macOS 10.15 (Catalina)), we fix another issue: This python3 (3.8.2) is not able to build extension modules because it emits -arch arm64 -arch x86_64 unless ARCHFLAGS is set explicitly

$ /usr/bin/python3 -m sysconfig |grep '\bCFLAGS ='
	CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64"

$ ARCHFLAGS="" /usr/bin/python3 -m sysconfig |grep '\bCFLAGS ='
	CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers  -arch x86_64 "

In this ticket, if the distutils test fails, we try if setting ARCHFLAGS="" fixes it. Also, after accepting a system python and ARCHFLAGS so far has not been set, we test whether the system python wants to do the multi-arch build.

In either of these two cases, we store a configuration variable that causes sage-env to set ARCHFLAGS as well.

This is somewhat complicated logic - we are tiptoeing around previous breakage that had been caused by setting ARCHFLAGS unconditionally (#29408).

Attachments (8)

pillow-8.0.1.log (29.1 KB) - added by John Palmieri 2 years ago.
cython-0.29.21.log (44.0 KB) - added by John Palmieri 2 years ago.
gmpy2-2.1.0b5.log (9.6 KB) - added by John Palmieri 2 years ago.
config.log (305.2 KB) - added by John Palmieri 2 years ago.
config.2.log (303.8 KB) - added by John Palmieri 2 years ago.
config.log for failed Big Sur build
config.3.log (305.1 KB) - added by John Palmieri 2 years ago.
config.log for failed Big Sur build
cysignals-1.10.2.log (20.1 KB) - added by John Palmieri 2 years ago.
scipy-1.5.4.log (497.2 KB) - added by John Palmieri 2 years ago.

Download all attachments as: .zip

Change History (91)

comment:1 Changed 2 years ago by Matthias Köppe

Dependencies: #31132

comment:2 Changed 2 years ago by Matthias Köppe

Branch: u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_

comment:3 Changed 2 years ago by Matthias Köppe

Commit: ce3481061b6308e730004ed461f8a86341249db1
Dependencies: #31132#31132, #31228

New commits:

2c60025m4/sage_check_python_for_venv.m4: Warn if python3 is misconfigured like it is currently in homebrew
ee679bdbuild/pkgs/python3/spkg-configure.m4: Do not check sysconfig CPPFLAGS so that /usr/bin/python3 is accepted on macOS; reject broken homebrew python3 unless requested explicitly with configgure --with-python=...
29aee87build/pkgs/python3: Update to 3.9.0rc2
e61d464build/pkgs/python3: Update to 3.9.0
f8bb56dbuild/pkgs/python3: Update to 3.9.1
0e3513fbuild/pkgs/pip: Update to 20.3.3
41d7e7aMerge branch 'u/jhpalmieri/new_wheel' of git://trac.sagemath.org/sage into t/30589/upgrade-python-3.9
76d5fd1Merge branch 't/30589/upgrade-python-3.9' into t/31132/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_build_when_using_homebrew_s_python3
ce34810m4/sage_check_python_for_venv.m4: Use empty ARCHFLAGS while testing distutils

comment:4 Changed 2 years ago by git

Commit: ce3481061b6308e730004ed461f8a86341249db167d66b74baa03829defe3f0721e79d8603ce34d8

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

1b16a8cbuild/pkgs/pari/spkg-install.in: Do not recompute MACOSX_VERSION
c6e30eatox.ini: Add configuration factors gcc_10, gcc_11
9b0eb15build/pkgs/{cmake,curl,psutil,python3,r}: If setting CC=clang, also set CFLAGS
05985d8Merge branch 't/30725/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_' into t/31228/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built
02c2684R is already compiled without -march=native
24b4fccset EXTRACFLAGS instead of CFLAGS
44be967make code a bit more stable against future changes
f56e65efix CFLAGS for python3/spkg-build.in by running testcflags.sh after determination of the compiler
09b03d2Merge branch 'u/gh-kliem/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_' of git://trac.sagemath.org/sage into t/31228/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built
67d66b7Merge branch 't/31228/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built' into t/31227/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_

comment:5 Changed 2 years ago by Matthias Köppe

Cc: gh-kliem added

comment:6 Changed 2 years ago by Matthias Köppe

Dependencies: #31132, #31228#31132, #30725, #31228

comment:7 Changed 2 years ago by git

Commit: 67d66b74baa03829defe3f0721e79d8603ce34d83325a4e4cef11e7b2f9946fd0db143f1c0157af3

Branch pushed to git repo; I updated commit sha1. New commits:

3325a4ebuild/pkgs/python3/spkg-configure.m4: Actually test with CFLAGS=$CFLAGS_MARCH

comment:8 Changed 2 years ago by Matthias Köppe

Authors: Matthias Koeppe
Description: modified (diff)
Status: newneeds_review

This is the branch that I had. I won't have time to work on it over the next week.

comment:9 Changed 2 years ago by gh-kliem

It was proposed to mark #31228 as invalid and do it all here. In this case (and it is in fact already done here).

In this case, this ticket here is really ready for review (and the depency of #31228 could be removed).

comment:10 Changed 2 years ago by gh-kliem

Reviewers: https://github.com/kliem/sage/pull/36/checks, Jonathan Kliem

I'm happy with this ticket, but I will have to wait and see if things work out in practice.

comment:11 Changed 2 years ago by Matthias Köppe

Priority: majorblocker

comment:12 Changed 2 years ago by Matthias Köppe

Dependencies: #31132, #30725, #31228#31132, #30725
Description: modified (diff)

comment:13 Changed 2 years ago by John Palmieri

With this branch, ./configure --with-python=/usr/bin/python3 fails for me on OS X Catalina with Xcode 12.3. See attached config.log.

comment:14 Changed 2 years ago by Matthias Köppe

Did you run bootstrap?

comment:15 in reply to:  14 Changed 2 years ago by John Palmieri

Replying to mkoeppe:

Did you run bootstrap?

No, sorry for the false report. Having run ./bootstrap, and with local/bin/python3 a symlink pointing to /Applications/Xcode.app/Contents/Developer/usr/bin/python3, the build has now failed twice: Pillow and cython have both failed to build.

Changed 2 years ago by John Palmieri

Attachment: pillow-8.0.1.log added

Changed 2 years ago by John Palmieri

Attachment: cython-0.29.21.log added

comment:16 Changed 2 years ago by John Palmieri

Oh, and gmpy2, also.

Changed 2 years ago by John Palmieri

Attachment: gmpy2-2.1.0b5.log added

comment:17 Changed 2 years ago by Matthias Köppe

OK, looks like we have to set ARCHFLAGS="" in more places. But this needs to be done very carefully - see #29408

comment:18 Changed 2 years ago by John Palmieri

I ran make -k to get a fuller report, and it ended with:

* package:         gmpy2-2.1.0b5
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/gmpy2-2.1.0b5.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/gmpy2-2.1.0b5

* package:         cython-0.29.21
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/cython-0.29.21.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/cython-0.29.21

* package:         psutil-5.2.0.p2
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/psutil-5.2.0.p2.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/psutil-5.2.0.p2

* package:         pillow-8.0.1
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/pillow-8.0.1.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/pillow-8.0.1

* package:         kiwisolver-1.0.1
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/kiwisolver-1.0.1.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/kiwisolver-1.0.1

* package:         pynac-0.7.26.sage-2020-04-03.p0
  last build time: Jan 26 15:53
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/pynac-0.7.26.sage-2020-04-03.p0.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/pynac-0.7.26.sage-2020-04-03.p0

* package:         cffi-1.14.4
  last build time: Jan 26 15:53
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/cffi-1.14.4.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/cffi-1.14.4

All show the same error, "clang: error: the clang compiler does not support '-march=native'".

comment:19 in reply to:  17 Changed 2 years ago by John Palmieri

Replying to mkoeppe:

OK, looks like we have to set ARCHFLAGS="" in more places. But this needs to be done very carefully - see #29408

How can we detect whether we're using a Python which requires this? And where is the right place to do it? build/bin/sage-build-env-config.in? There are lots of affected packages, so doing it globally would be easier. (pyzmq, cysignals, numpy, and others also seem to fail with the same error, if I recklessly add export ARCHFLAGS="" to the install scripts for the other packages which were failing.)

comment:20 in reply to:  18 Changed 2 years ago by Matthias Köppe

Replying to jhpalmieri:

All show the same error, "clang: error: the clang compiler does not support '-march=native'".

Oh, this is a different error. Let me try to fix this

comment:21 Changed 2 years ago by Matthias Köppe

Could you post config.log from this run?

Changed 2 years ago by John Palmieri

Attachment: config.log added

comment:22 in reply to:  21 Changed 2 years ago by John Palmieri

Replying to mkoeppe:

Could you post config.log from this run?

Yes. I replaced the old config.log, since that was a result of my error, not a problem with this ticket.

comment:23 Changed 2 years ago by Matthias Köppe

Thanks. The crazy thing is that passing -march=native only seems to result in the error clang: error: the clang compiler does not support '-march=native' if also -arch arm64 is passed. The current test in python3/spkg-configure.m4 does not catch this.

comment:24 Changed 2 years ago by gh-kliem

What I find a bit confusing is the following:

We do not test, whether setting ARCHFLAGS="" was necesarry in order to build this test extension during configure

Why do we expect that in real life situations (when actually compiling python extensions for sage) ARCHFLAGS="" is not a requirement anymore?

comment:25 Changed 2 years ago by John Palmieri

For what it's worth, setting ARCHFLAGS="" in sage-build-env-config.in let me build Sage. Doctesting didn't go that well: various failures because of "command 'gcc' failed with exit status 1" and "#error Unsupported architecture" and "#error architecture not supported".

comment:26 Changed 2 years ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:27 in reply to:  25 Changed 2 years ago by Matthias Köppe

Replying to jhpalmieri:

For what it's worth, setting ARCHFLAGS="" in sage-build-env-config.in let me build Sage. Doctesting didn't go that well: various failures because of "command 'gcc' failed with exit status 1" and "#error Unsupported architecture" and "#error architecture not supported".

If we set ARCHFLAGS, we will have to do this in sage-env so that it also affects runtime use of the compiler (some of which is tested in doctests) and when users install Python packages with sage -pip.

comment:28 Changed 2 years ago by git

Commit: 3325a4e4cef11e7b2f9946fd0db143f1c0157af3be3196dbfd015b9fbdbf7a06b9c953b687db3726

Branch pushed to git repo; I updated commit sha1. New commits:

be3196dbuild/pkgs/python3/spkg-configure.m4, src/bin/sage-env[-config.in]: Prepare conditional setting of ARCHFLAGS

comment:29 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:30 Changed 2 years ago by git

Commit: be3196dbfd015b9fbdbf7a06b9c953b687db37264c1596ccbf57b7d63f56a189681852e82fe5bac9

Branch pushed to git repo; I updated commit sha1. New commits:

4c1596cbuild/pkgs/python3/spkg-configure.m4, m4/sage_check_python_for_venv.m4: Refactor through new macro SAGE_PYTHON_CHECK_DISTUTILS

comment:31 Changed 2 years ago by git

Commit: 4c1596ccbf57b7d63f56a189681852e82fe5bac9dc5e22511f783391282eca6a986971e6fe6f20b3

Branch pushed to git repo; I updated commit sha1. New commits:

dc5e225build/pkgs/python3/spkg-configure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS

comment:32 Changed 2 years ago by Matthias Köppe

Status: needs_workneeds_review

Here's a new version that may be worth testing

comment:33 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:34 Changed 2 years ago by Matthias Köppe

Builds successfully for me with tox -e local-homebrew-macos-standard-python3_xcode

comment:35 Changed 2 years ago by git

Commit: dc5e22511f783391282eca6a986971e6fe6f20b3df17ad4e916d4db69a49dbe7c8082014abbcb539

Branch pushed to git repo; I updated commit sha1. New commits:

df17ad4tox.ini: Add configuration factors for specific homebrew python3.x

comment:36 Changed 2 years ago by gh-kliem

What is the purpose of this line?

+    AC_SUBST([ARCHFLAGS], [unset])

comment:37 Changed 2 years ago by gh-kliem

I think the second reason is wrong here. Isn't this the case when building a C extension fails?

dnl distutils test
AC_DEFUN([SAGE_PYTHON_CHECK_DISTUTILS], [
    m4_pushdef([PYTHON_EXE], [$1])
    m4_pushdef([COMMANDS_IF_DISTUTILS_GOOD], [$2])
    m4_pushdef([COMMANDS_IF_DISTUTILS_NOT_GOOD], [$3])
    SAGE_PYTHON_DISTUTILS_C_CONFTEST
    dnl (echo "***ENV***:"; env; echo "***SYSCONFIG***"; conftest_venv/bin/python3 -m sysconfig) >& AS_MESSAGE_LOG_FD
    echo PYTHON_EXE conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD
    AS_IF([PYTHON_EXE conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD 2>&1 ], [
        SAGE_PYTHON_DISTUTILS_CXX_CONFTEST
        echo PYTHON_EXE conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD 2>&1
        AS_IF([PYTHON_EXE conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD 2>&1 ], [
            COMMANDS_IF_DISTUTILS_GOOD], [
            reason="distutils cannot build a C++ 11 extension"
            COMMANDS_IF_DISTUTILS_NOT_GOOD
        ])
    ], [
       reason="distutils cannot build a C++ 11 extension"
       COMMANDS_IF_DISTUTILS_NOT_GOOD
    ])
    m4_popdef([PYTHON_EXE])
    m4_popdef([COMMANDS_IF_DISTUTILS_GOOD])
    m4_popdef([COMMANDS_IF_DISTUTILS_NOT_GOOD])
])

comment:38 Changed 2 years ago by Matthias Köppe

Also tox -e local-homebrew-usrlocal-minimal-python3.8 still works.

comment:39 Changed 2 years ago by git

Commit: df17ad4e916d4db69a49dbe7c8082014abbcb539914ea58a359df5478484da7fa3293007fad2b2df

Branch pushed to git repo; I updated commit sha1. New commits:

914ea58build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST

comment:40 Changed 2 years ago by git

Commit: 914ea58a359df5478484da7fa3293007fad2b2df986739b940952c7b10dfe0bd2b17dce2407f1ee0

Branch pushed to git repo; I updated commit sha1. New commits:

986739bm4/sage_check_python_for_venv.m4: Fix reason

comment:41 Changed 2 years ago by Matthias Köppe

Thanks for spotting these mistakes!

comment:42 Changed 2 years ago by gh-kliem

Otherwise, this looks very plausible.

I updated https://github.com/kliem/sage/pull/36/ (not the last two commits, but they are cosmetic).

comment:43 Changed 2 years ago by John Palmieri

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before:

sage -t --long --random-seed=0 src/sage_setup/find.py
**********************************************************************
File "src/sage_setup/find.py", line 278, in sage_setup.find.installed_files_by_module
Failed example:
    f1
Expected:
    'sage/structure/__init__.py'
Got:
    '/Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/structure/__init__.cpython-38.pyc'
**********************************************************************
File "src/sage_setup/find.py", line 280, in sage_setup.find.installed_files_by_module
Failed example:
    f2
Expected:
    'sage/structure/....pyc'
Got:
    'sage/structure/__init__.py'

and

sage -t --long --random-seed=0 src/sage_setup/clean.py
**********************************************************************
File "src/sage_setup/clean.py", line 95, in sage_setup.clean._find_stale_files
Failed example:
    for f in stale_iter:
        if f.endswith(skip_extensions): continue
        if '/ext_data/' in f: continue
        print('Found stale file: ' + f)
Expected nothing
Got:
    Found stale file: /Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/all_cmdline.cpython-38.pyc
    Found stale file: /Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/version.cpython-38.pyc

  [ and many more lines of such messages ]

Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

comment:44 Changed 2 years ago by John Palmieri

This does not work for me with OS X Big Sur, though: many packages (cython, pynac, gmpy2, psutil, pillow, kiwisolver, cffi) fail to build with the error "clang: error: the clang compiler does not support '-march=native'".

Hold on, I have to run ./bootstrap and try again.

Confirmed, these packages all fail after

make distclean && ./bootstrap && ./configure --with-python=/usr/bin/python3 && make -k
Last edited 2 years ago by John Palmieri (previous) (diff)

comment:45 in reply to:  43 ; Changed 2 years ago by Matthias Köppe

Replying to jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

Last edited 2 years ago by Matthias Köppe (previous) (diff)

comment:46 Changed 2 years ago by Matthias Köppe

config.log from the big sur system please

comment:47 in reply to:  45 ; Changed 2 years ago by John Palmieri

Replying to mkoeppe:

Replying to jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

No, not normally and not after running ./sage --sh.

Changed 2 years ago by John Palmieri

Attachment: config.2.log added

config.log for failed Big Sur build

comment:48 in reply to:  47 ; Changed 2 years ago by John Palmieri

Replying to jhpalmieri:

Replying to mkoeppe:

Replying to jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

No, not normally and not after running ./sage --sh.

However:

% /usr/bin/python3   
Python 3.8.2 (default, Dec 21 2020, 15:06:04) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.pycache_prefix
'/Users/jpalmier/Library/Caches/com.apple.python'

This is not how /usr/local/bin/python3 behaves: sys.pycache_prefix is None there. The Python documentation suggests that the default setting should be None since I am not passing a -X argument nor am I setting PYTHONPYCACHEPREFIX.

comment:49 Changed 2 years ago by Matthias Köppe

hm... config.2.log does not look like it's coming from this branch...

comment:50 in reply to:  48 Changed 2 years ago by Matthias Köppe

Replying to jhpalmieri:

% /usr/bin/python3   
Python 3.8.2 (default, Dec 21 2020, 15:06:04) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.pycache_prefix
'/Users/jpalmier/Library/Caches/com.apple.python'

Same here. Looks like Apple has preconfigured their build to do this. I'll update the description of #31314.

comment:51 in reply to:  49 Changed 2 years ago by John Palmieri

Replying to mkoeppe:

hm... config.2.log does not look like it's coming from this branch...

Sorry, you're right. Typo on my part: I checked out #31127 instead of #31227. Trying again now.

comment:52 Changed 2 years ago by John Palmieri

Now with Big Sur, cysignals and scipy fail. For scipy, the error looks like this:

    /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/lib/python3.8/site-packages/numpy/distutils/system_info.py:838: UserWarning: Specified path /usr/local/include/python3.8 is invalid.

Somehow it's picking up /usr/local. I'm not sure what's going on with cysignals. Logs attached.

Changed 2 years ago by John Palmieri

Attachment: config.3.log added

config.log for failed Big Sur build

Changed 2 years ago by John Palmieri

Attachment: cysignals-1.10.2.log added

Changed 2 years ago by John Palmieri

Attachment: scipy-1.5.4.log added

comment:53 Changed 2 years ago by Matthias Köppe

The config.log looks good.

comment:54 in reply to:  52 Changed 2 years ago by Matthias Köppe

Replying to jhpalmieri:

Now with Big Sur, cysignals and scipy fail. For scipy, the error looks like this:

    /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/lib/python3.8/site-packages/numpy/distutils/system_info.py:838: UserWarning: Specified path /usr/local/include/python3.8 is invalid.

I think the actual error is this one:

  ar: adding 50 object files to build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a
  warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive library: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a will be fat and ar(1) will not be able to operate on it
  ar: adding 50 object files to build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a
  ar: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a is a fat file (use libtool(1) or lipo(1) and ar(1) on it)
  ar: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a: Inappropriate file type or format

comment:56 Changed 2 years ago by Matthias Köppe

Is MACOSX_DEPLOYMENT_TARGET set to something on your big sur system?

comment:57 in reply to:  56 Changed 2 years ago by John Palmieri

Replying to mkoeppe:

Is MACOSX_DEPLOYMENT_TARGET set to something on your big sur system?

No, but who knows what Xcode and/or homebrew are doing these days.

comment:58 Changed 2 years ago by Matthias Köppe

from cysignals.log:

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -O2 -g -Wp,-U_FORTIFY_SOURCE -DCYTHON_CLINE_IN_TRACEBACK=0 -U_FORTIFY_SOURCE -Isrc/cysignals -Isrc -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c build/src/cysignals/signals.c -o build/temp.macosx-10.14.6-x86_64-3.8/build/src/cysignals/signals.o -pthread
  In file included from build/src/cysignals/signals.c:1570:
  In file included from build/src/cysignals/implementation.c:58:
  In file included from /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/include/pari/pari.h:49:
  ../src/kernel/none/divll_pre.h:30:10: error: invalid output constraint '=a' in asm
    return divll(~0UL, n);
           ^

This is trying to build for -arch arm64 -arch x86_64 and is tripping over the installed pari header ... which is arch-specific!

comment:59 Changed 2 years ago by John Palmieri

Before I source .homebrew-build-env, I only set these environment variables:

BIBINPUTS=/Users/palmieri/iCloud/Documents/tex/bibtex:.:
EDITOR=emacs
LESS=-R
LSCOLORS=Gxfxcxdxbxegedabagacad
MAKE='make -j6'
MAKEFLAGS=-j6
PAGER=less
PATH=/Users/palmieri/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands

(If I run export at the start of my .zshrc file and also comment out the line that runs .homebrew-build-env, and then run export again with an actual running terminal, these are the differences.) Nothing strange there, nothing that I think should cause any problems. This is on the Big Sur system, but the Catalina system has a very similar configuration.

comment:60 Changed 2 years ago by Matthias Köppe

It seems that we need to set ARCHFLAGS="" in more situations. The default set by /usr/bin/python3, trying to build for both architectures, is unrealistic to support -- this would require major changes to how we install libraries.

comment:61 Changed 2 years ago by git

Commit: 986739b940952c7b10dfe0bd2b17dce2407f1ee00b3e70d458ef88d610c6883ddedaf19565f7de3a

Branch pushed to git repo; I updated commit sha1. New commits:

0b3e70dbuild/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it

comment:62 Changed 2 years ago by Matthias Köppe

Here's a new version to try on big sur

comment:63 Changed 2 years ago by git

Commit: 0b3e70d458ef88d610c6883ddedaf19565f7de3a372adcb4b2f3de7a230b14f5c275299d7d8c50fc

Branch pushed to git repo; I updated commit sha1. New commits:

372adcb.github/workflows/tox.yml: Update xcode versions

comment:64 Changed 2 years ago by Matthias Köppe

Started tests at https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

comment:65 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:66 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:67 Changed 2 years ago by Matthias Köppe

Summary: Accept /usr/bin/python3 from XCode 12.3 on macOS 10.15 (Catalina)Accept /usr/bin/python3 from XCode 12.3 on macOS 10.15 (Catalina) and 11 (Big Sur)

comment:68 Changed 2 years ago by git

Commit: 372adcb4b2f3de7a230b14f5c275299d7d8c50fcfc8b67644bada9ea52cbb7c91f53f763ca84f4bb

Branch pushed to git repo; I updated commit sha1. New commits:

fc8b676SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting

comment:69 Changed 2 years ago by Zachary L Scherr

For the record, I was able to successfully build sage and documentation with this ticket on Big Sur using

--with-python=/usr/bin/python3

comment:70 Changed 2 years ago by Matthias Köppe

Thanks for testing!

comment:71 Changed 2 years ago by John Palmieri

Works for me with the system Python 3 on both Big Sur and Catalina.

comment:72 Changed 2 years ago by Matthias Köppe

Thanks for testing!

Let's deal with issue #31314 (messages from find_stale_files) in a follow up.

Ready for review?

comment:73 in reply to:  64 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

Started tests at https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

Some results have been coming in:

  • local-homebrew-macos-10.15-xcode-11.7-{minimal,standard}: clean
  • local-homebrew-macos-11.0-xcode-11.7-{minimal,standard}: Homebrew complains that xcode is too old

Still waiting for the tests with other xcode versions (12.2, 12.3, 12.4). No failures so far.

Note all runs are on x86_64 - we do not have test infrastructor for macos-arm64 yet.

Last edited 2 years ago by Matthias Köppe (previous) (diff)

comment:74 Changed 2 years ago by gh-kliem

Reviewers: https://github.com/kliem/sage/pull/36/checks, Jonathan KliemJonathan Kliem

I'm happy with this ticket, but I would like another opinion on this.

comment:75 Changed 2 years ago by gh-kliem

Ok, I give it another 24 hours. If nobody speaks up, I'll give a positive review (after taking a final look of course).

comment:76 Changed 2 years ago by Zachary L Scherr

I don't feel like I have enough experience to review this, but it certainly built on Big Sur and Catalina. I'm going to upgrade Big Sur to 11.2 later and see if this builds on 11.2 as well.

comment:77 in reply to:  64 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

Started tests at https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

Some more results:

comment:78 Changed 2 years ago by Matthias Köppe

Let's take care of the scipy build failures in #31326.

I think this is sufficient improvement for one ticket.

comment:79 Changed 2 years ago by Matthias Köppe

Cc: Volker Braun added

comment:80 Changed 2 years ago by Zachary L Scherr

Seems to work well on Big Sur 11.2 with standard homebrew packages. When running make ptest I get the annoying warnings:

ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (10.15) than being linked (10.14.6)

which is probably because the MACOSX_DEPLOYMENT_TARGET for /usr/bin/python3 is set to 10.14.6 which doesn't match what Homebrew sets. It's easy enough to eliminate the warnings by setting MACOSX_DEPLOYMENT_TARGET=11.0 before doing all the testing.

Otherwise, everything seems to build/test correctly.

comment:81 Changed 2 years ago by gh-kliem

Status: needs_reviewpositive_review

Ok. Thanks for fixing all of this. I think the ticket is a good improvement as it is.

comment:82 Changed 2 years ago by Matthias Köppe

Thanks!

comment:83 Changed 2 years ago by Volker Braun

Branch: u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_fc8b67644bada9ea52cbb7c91f53f763ca84f4bb
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.