Accept /usr/bin/python3 from XCode 12.3 on macOS 10.15 (Catalina) and 11 (Big Sur)

[mkoeppe]

(from #31132)

#27122 determines whether CC accepts -march=native and then adds it globally to CFLAGS in build/bin/sage-build-env.

#30725 fixes this for macOS, where some packages are explicitly built with clang.

However, there is another mechanism how clang can come in: Python extension modules are compiled with the compiler listed in sysconfig. On macOS with homebrew, using /usr/bin/python3, this could again be clang, which does not accept -march=native in combination with -arch arm64 (which is provided even on x86_64 if ARCHFLAGS is unset).

In this ticket, we fix this by checking whether the compiler used by distutils accepts -march=native (or more generally, the flags determined earlier by configure). If it does not, then we disable the use of these flags.

In order to test this with /usr/bin/python3 from XCode 12.3 (on macOS 10.15 (Catalina)), we fix another issue: This python3 (3.8.2) is not able to build extension modules because it emits -arch arm64 -arch x86_64 unless ARCHFLAGS is set explicitly

$ /usr/bin/python3 -m sysconfig |grep '\bCFLAGS ='
	CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64"

$ ARCHFLAGS="" /usr/bin/python3 -m sysconfig |grep '\bCFLAGS ='
	CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers  -arch x86_64 "

In this ticket, if the distutils test fails, we try if setting ARCHFLAGS="" fixes it. Also, after accepting a system python and ARCHFLAGS so far has not been set, we test whether the system python wants to do the multi-arch build.

In either of these two cases, we store a configuration variable that causes sage-env to set ARCHFLAGS as well.

This is somewhat complicated logic - we are tiptoeing around previous breakage that had been caused by setting ARCHFLAGS unconditionally (#29408).

[mkoeppe]

New commits:

​2c60025	m4/sage_check_python_for_venv.m4: Warn if python3 is misconfigured like it is currently in homebrew
​ee679bd	build/pkgs/python3/spkg-configure.m4: Do not check sysconfig CPPFLAGS so that /usr/bin/python3 is accepted on macOS; reject broken homebrew python3 unless requested explicitly with configgure --with-python=...
​29aee87	build/pkgs/python3: Update to 3.9.0rc2
​e61d464	build/pkgs/python3: Update to 3.9.0
​f8bb56d	build/pkgs/python3: Update to 3.9.1
​0e3513f	build/pkgs/pip: Update to 20.3.3
​41d7e7a	Merge branch 'u/jhpalmieri/new_wheel' of git://trac.sagemath.org/sage into t/30589/upgrade-python-3.9
​76d5fd1	Merge branch 't/30589/upgrade-python-3.9' into t/31132/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_build_when_using_homebrew_s_python3
​ce34810	m4/sage_check_python_for_venv.m4: Use empty ARCHFLAGS while testing distutils

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​1b16a8c	build/pkgs/pari/spkg-install.in: Do not recompute MACOSX_VERSION
​c6e30ea	tox.ini: Add configuration factors gcc_10, gcc_11
​9b0eb15	build/pkgs/{cmake,curl,psutil,python3,r}: If setting CC=clang, also set CFLAGS
​05985d8	Merge branch 't/30725/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_' into t/31228/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built
​02c2684	R is already compiled without -march=native
​24b4fcc	set EXTRACFLAGS instead of CFLAGS
​44be967	make code a bit more stable against future changes
​f56e65e	fix CFLAGS for python3/spkg-build.in by running testcflags.sh after determination of the compiler
​09b03d2	Merge branch 'u/gh-kliem/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_' of git://trac.sagemath.org/sage into t/31228/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built
​67d66b7	Merge branch 't/31228/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built' into t/31227/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3325a4e

build/pkgs/python3/spkg-configure.m4: Actually test with CFLAGS=$CFLAGS_MARCH

[mkoeppe]

This is the branch that I had. I won't have time to work on it over the next week.

[gh-kliem]

It was proposed to mark #31228 as invalid and do it all here. In this case (and it is in fact already done here).

In this case, this ticket here is really ready for review (and the depency of #31228 could be removed).

[gh-kliem]

I'm happy with this ticket, but I will have to wait and see if things work out in practice.

[jhpalmieri]

With this branch, ./configure --with-python=/usr/bin/python3 fails for me on OS X Catalina with Xcode 12.3. See attached config.log.

[mkoeppe]

Did you run bootstrap?

[jhpalmieri]

Replying to mkoeppe:

Did you run bootstrap?

No, sorry for the false report. Having run ./bootstrap, and with local/bin/python3 a symlink pointing to /Applications/Xcode.app/Contents/Developer/usr/bin/python3, the build has now failed twice: Pillow and cython have both failed to build.

[jhpalmieri]

Oh, and gmpy2, also.

[mkoeppe]

OK, looks like we have to set ARCHFLAGS="" in more places. But this needs to be done very carefully - see #29408

[jhpalmieri]

I ran make -k to get a fuller report, and it ended with:

* package:         gmpy2-2.1.0b5
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/gmpy2-2.1.0b5.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/gmpy2-2.1.0b5

* package:         cython-0.29.21
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/cython-0.29.21.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/cython-0.29.21

* package:         psutil-5.2.0.p2
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/psutil-5.2.0.p2.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/psutil-5.2.0.p2

* package:         pillow-8.0.1
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/pillow-8.0.1.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/pillow-8.0.1

* package:         kiwisolver-1.0.1
  last build time: Jan 26 15:52
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/kiwisolver-1.0.1.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/kiwisolver-1.0.1

* package:         pynac-0.7.26.sage-2020-04-03.p0
  last build time: Jan 26 15:53
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/pynac-0.7.26.sage-2020-04-03.p0.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/pynac-0.7.26.sage-2020-04-03.p0

* package:         cffi-1.14.4
  last build time: Jan 26 15:53
  log file:        /Users/jpalmier/Desktop/Sage/git/sage/logs/pkgs/cffi-1.14.4.log
  build directory: /Users/jpalmier/Desktop/Sage/git/sage/local/var/tmp/sage/build/cffi-1.14.4

All show the same error, "clang: error: the clang compiler does not support '-march=native'".

[jhpalmieri]

Replying to mkoeppe:

OK, looks like we have to set ARCHFLAGS="" in more places. But this needs to be done very carefully - see #29408

How can we detect whether we're using a Python which requires this? And where is the right place to do it? build/bin/sage-build-env-config.in? There are lots of affected packages, so doing it globally would be easier. (pyzmq, cysignals, numpy, and others also seem to fail with the same error, if I recklessly add export ARCHFLAGS="" to the install scripts for the other packages which were failing.)

[mkoeppe]

Replying to jhpalmieri:

All show the same error, "clang: error: the clang compiler does not support '-march=native'".

Oh, this is a different error. Let me try to fix this

[mkoeppe]

Could you post config.log from this run?

[jhpalmieri]

Replying to mkoeppe:

Could you post config.log from this run?

Yes. I replaced the old config.log, since that was a result of my error, not a problem with this ticket.

[mkoeppe]

Thanks. The crazy thing is that passing -march=native only seems to result in the error clang: error: the clang compiler does not support '-march=native' if also -arch arm64 is passed. The current test in python3/spkg-configure.m4 does not catch this.

[gh-kliem]

What I find a bit confusing is the following:

We do not test, whether setting ARCHFLAGS="" was necesarry in order to build this test extension during configure

Why do we expect that in real life situations (when actually compiling python extensions for sage) ARCHFLAGS="" is not a requirement anymore?

[jhpalmieri]

For what it's worth, setting ARCHFLAGS="" in sage-build-env-config.in let me build Sage. Doctesting didn't go that well: various failures because of "command 'gcc' failed with exit status 1" and "#error Unsupported architecture" and "#error architecture not supported".

[mkoeppe]

Replying to jhpalmieri:

For what it's worth, setting ARCHFLAGS="" in sage-build-env-config.in let me build Sage. Doctesting didn't go that well: various failures because of "command 'gcc' failed with exit status 1" and "#error Unsupported architecture" and "#error architecture not supported".

If we set ARCHFLAGS, we will have to do this in sage-env so that it also affects runtime use of the compiler (some of which is tested in doctests) and when users install Python packages with sage -pip.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​be3196d

build/pkgs/python3/spkg-configure.m4, src/bin/sage-env[-config.in]: Prepare conditional setting of ARCHFLAGS

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4c1596c

build/pkgs/python3/spkg-configure.m4, m4/sage_check_python_for_venv.m4: Refactor through new macro SAGE_PYTHON_CHECK_DISTUTILS

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dc5e225

build/pkgs/python3/spkg-configure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS

[mkoeppe]

Here's a new version that may be worth testing

[mkoeppe]

Builds successfully for me with tox -e local-homebrew-macos-standard-python3_xcode

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​df17ad4

tox.ini: Add configuration factors for specific homebrew python3.x

[gh-kliem]

What is the purpose of this line?

+    AC_SUBST([ARCHFLAGS], [unset])

[gh-kliem]

I think the second reason is wrong here. Isn't this the case when building a C extension fails?

dnl distutils test
AC_DEFUN([SAGE_PYTHON_CHECK_DISTUTILS], [
    m4_pushdef([PYTHON_EXE], [$1])
    m4_pushdef([COMMANDS_IF_DISTUTILS_GOOD], [$2])
    m4_pushdef([COMMANDS_IF_DISTUTILS_NOT_GOOD], [$3])
    SAGE_PYTHON_DISTUTILS_C_CONFTEST
    dnl (echo "***ENV***:"; env; echo "***SYSCONFIG***"; conftest_venv/bin/python3 -m sysconfig) >& AS_MESSAGE_LOG_FD
    echo PYTHON_EXE conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD
    AS_IF([PYTHON_EXE conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD 2>&1 ], [
        SAGE_PYTHON_DISTUTILS_CXX_CONFTEST
        echo PYTHON_EXE conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD 2>&1
        AS_IF([PYTHON_EXE conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD 2>&1 ], [
            COMMANDS_IF_DISTUTILS_GOOD], [
            reason="distutils cannot build a C++ 11 extension"
            COMMANDS_IF_DISTUTILS_NOT_GOOD
        ])
    ], [
       reason="distutils cannot build a C++ 11 extension"
       COMMANDS_IF_DISTUTILS_NOT_GOOD
    ])
    m4_popdef([PYTHON_EXE])
    m4_popdef([COMMANDS_IF_DISTUTILS_GOOD])
    m4_popdef([COMMANDS_IF_DISTUTILS_NOT_GOOD])
])

[mkoeppe]

Also tox -e local-homebrew-usrlocal-minimal-python3.8 still works.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​914ea58

build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​986739b

m4/sage_check_python_for_venv.m4: Fix reason

[mkoeppe]

Thanks for spotting these mistakes!

[gh-kliem]

Otherwise, this looks very plausible.

I updated ​https://github.com/kliem/sage/pull/36/ (not the last two commits, but they are cosmetic).

[jhpalmieri]

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before:

sage -t --long --random-seed=0 src/sage_setup/find.py
**********************************************************************
File "src/sage_setup/find.py", line 278, in sage_setup.find.installed_files_by_module
Failed example:
    f1
Expected:
    'sage/structure/__init__.py'
Got:
    '/Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/structure/__init__.cpython-38.pyc'
**********************************************************************
File "src/sage_setup/find.py", line 280, in sage_setup.find.installed_files_by_module
Failed example:
    f2
Expected:
    'sage/structure/....pyc'
Got:
    'sage/structure/__init__.py'

and

sage -t --long --random-seed=0 src/sage_setup/clean.py
**********************************************************************
File "src/sage_setup/clean.py", line 95, in sage_setup.clean._find_stale_files
Failed example:
    for f in stale_iter:
        if f.endswith(skip_extensions): continue
        if '/ext_data/' in f: continue
        print('Found stale file: ' + f)
Expected nothing
Got:
    Found stale file: /Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/all_cmdline.cpython-38.pyc
    Found stale file: /Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/version.cpython-38.pyc

  [ and many more lines of such messages ]

Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

[jhpalmieri]

This does not work for me with OS X Big Sur, though: many packages (cython, pynac, gmpy2, psutil, pillow, kiwisolver, cffi) fail to build with the error "clang: error: the clang compiler does not support '-march=native'".

Hold on, I have to run ./bootstrap and try again.

Confirmed, these packages all fail after

make distclean && ./bootstrap && ./configure --with-python=/usr/bin/python3 && make -k

[mkoeppe]

Replying to jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? ​https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

[mkoeppe]

config.log from the big sur system please

[jhpalmieri]

Replying to mkoeppe:

Replying to jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? ​https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

No, not normally and not after running ./sage --sh.

[jhpalmieri]

config.log for failed Big Sur build

[jhpalmieri]

Replying to jhpalmieri:

Replying to mkoeppe:

Replying to jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? ​https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

No, not normally and not after running ./sage --sh.

However:

% /usr/bin/python3   
Python 3.8.2 (default, Dec 21 2020, 15:06:04) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.pycache_prefix
'/Users/jpalmier/Library/Caches/com.apple.python'

This is not how /usr/local/bin/python3 behaves: sys.pycache_prefix is None there. The Python documentation suggests that the default setting should be None since I am not passing a -X argument nor am I setting PYTHONPYCACHEPREFIX.

[mkoeppe]

hm... config.2.log does not look like it's coming from this branch...

[mkoeppe]

Replying to jhpalmieri:

% /usr/bin/python3   
Python 3.8.2 (default, Dec 21 2020, 15:06:04) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.pycache_prefix
'/Users/jpalmier/Library/Caches/com.apple.python'

Same here. Looks like Apple has preconfigured their build to do this. I'll update the description of #31314.

[jhpalmieri]

Replying to mkoeppe:

hm... config.2.log does not look like it's coming from this branch...

Sorry, you're right. Typo on my part: I checked out #31127 instead of #31227. Trying again now.

[jhpalmieri]

Now with Big Sur, cysignals and scipy fail. For scipy, the error looks like this:

    /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/lib/python3.8/site-packages/numpy/distutils/system_info.py:838: UserWarning: Specified path /usr/local/include/python3.8 is invalid.

Somehow it's picking up /usr/local. I'm not sure what's going on with cysignals. Logs attached.

[jhpalmieri]

config.log for failed Big Sur build

[mkoeppe]

The config.log looks good.

[mkoeppe]

Replying to jhpalmieri:

Now with Big Sur, cysignals and scipy fail. For scipy, the error looks like this:

    /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/lib/python3.8/site-packages/numpy/distutils/system_info.py:838: UserWarning: Specified path /usr/local/include/python3.8 is invalid.

I think the actual error is this one:

  ar: adding 50 object files to build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a
  warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive library: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a will be fat and ar(1) will not be able to operate on it
  ar: adding 50 object files to build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a
  ar: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a is a fat file (use libtool(1) or lipo(1) and ar(1) on it)
  ar: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a: Inappropriate file type or format

[mkoeppe]

Similar: ​https://github.com/scipy/scipy/issues/10902

[mkoeppe]

Is MACOSX_DEPLOYMENT_TARGET set to something on your big sur system?

[jhpalmieri]

Replying to mkoeppe:

Is MACOSX_DEPLOYMENT_TARGET set to something on your big sur system?

No, but who knows what Xcode and/or homebrew are doing these days.

[mkoeppe]

from cysignals.log:

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -O2 -g -Wp,-U_FORTIFY_SOURCE -DCYTHON_CLINE_IN_TRACEBACK=0 -U_FORTIFY_SOURCE -Isrc/cysignals -Isrc -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c build/src/cysignals/signals.c -o build/temp.macosx-10.14.6-x86_64-3.8/build/src/cysignals/signals.o -pthread
  In file included from build/src/cysignals/signals.c:1570:
  In file included from build/src/cysignals/implementation.c:58:
  In file included from /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/include/pari/pari.h:49:
  ../src/kernel/none/divll_pre.h:30:10: error: invalid output constraint '=a' in asm
    return divll(~0UL, n);
           ^

This is trying to build for -arch arm64 -arch x86_64 and is tripping over the installed pari header ... which is arch-specific!

[jhpalmieri]

Before I source .homebrew-build-env, I only set these environment variables:

BIBINPUTS=/Users/palmieri/iCloud/Documents/tex/bibtex:.:
EDITOR=emacs
LESS=-R
LSCOLORS=Gxfxcxdxbxegedabagacad
MAKE='make -j6'
MAKEFLAGS=-j6
PAGER=less
PATH=/Users/palmieri/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands

(If I run export at the start of my .zshrc file and also comment out the line that runs .homebrew-build-env, and then run export again with an actual running terminal, these are the differences.) Nothing strange there, nothing that I think should cause any problems. This is on the Big Sur system, but the Catalina system has a very similar configuration.

[mkoeppe]

It seems that we need to set ARCHFLAGS="" in more situations. The default set by /usr/bin/python3, trying to build for both architectures, is unrealistic to support -- this would require major changes to how we install libraries.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0b3e70d

build/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it

[mkoeppe]

Here's a new version to try on big sur

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​372adcb

.github/workflows/tox.yml: Update xcode versions

[mkoeppe]

Started tests at ​https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fc8b676

SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting

[gh-zlscherr]

For the record, I was able to successfully build sage and documentation with this ticket on Big Sur using

--with-python=/usr/bin/python3

[mkoeppe]

Thanks for testing!

[jhpalmieri]

Works for me with the system Python 3 on both Big Sur and Catalina.

[mkoeppe]

Thanks for testing!

Let's deal with issue #31314 (messages from find_stale_files) in a follow up.

Ready for review?

[mkoeppe]

Replying to mkoeppe:

Started tests at ​https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

Some results have been coming in:

• local-homebrew-macos-10.15-xcode-11.7-{minimal,standard}: clean
• local-homebrew-macos-11.0-xcode-11.7-{minimal,standard}: Homebrew complains that xcode is too old

Still waiting for the tests with other xcode versions (12.2, 12.3, 12.4). No failures so far.

Note all runs are on x86_64 - we do not have test infrastructor for macos-arm64 yet.

[gh-kliem]

I'm happy with this ticket, but I would like another opinion on this.

[gh-kliem]

Ok, I give it another 24 hours. If nobody speaks up, I'll give a positive review (after taking a final look of course).

[gh-zlscherr]

I don't feel like I have enough experience to review this, but it certainly built on Big Sur and Catalina. I'm going to upgrade Big Sur to 11.2 later and see if this builds on 11.2 as well.

[mkoeppe]

Replying to mkoeppe:

Started tests at ​https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

Some more results:

• local-homebrew-macos-10.15-xcode-{12.2,12.3,12.4}-{minimal,standard}: clean except for doctest errors from ld: warning (#31204)
• local-homebrew-macos-11.0-xcode-{12.2,12.3}-minimal: scipy build fails (​https://github.com/mkoeppe/sage/runs/1804239019?check_suite_focus=true)
• local-homebrew-macos-11.0-xcode-12.3-standard: clean except for doctest errors from ld: warning (#31204)

[mkoeppe]

Let's take care of the scipy build failures in #31326.

I think this is sufficient improvement for one ticket.

[gh-zlscherr]

Seems to work well on Big Sur 11.2 with standard homebrew packages. When running make ptest I get the annoying warnings:

ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (10.15) than being linked (10.14.6)

which is probably because the MACOSX_DEPLOYMENT_TARGET for /usr/bin/python3 is set to 10.14.6 which doesn't match what Homebrew sets. It's easy enough to eliminate the warnings by setting MACOSX_DEPLOYMENT_TARGET=11.0 before doing all the testing.

Otherwise, everything seems to build/test correctly.

[gh-kliem]

Ok. Thanks for fixing all of this. I think the ticket is a good improvement as it is.

[mkoeppe]

Thanks!