Opened 2 years ago

Closed 2 years ago

#30589 closed enhancement (fixed)

Upgrade Python to 3.9.1, pip to 20.3.3

Reported by: Matthias Köppe Owned by:
Priority: critical Milestone: sage-9.3
Component: packages: standard Keywords:
Cc: Samuel Lelièvre, John Palmieri, Frédéric Chapoton, Dima Pasechnik, Volker Braun, François Bissey Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri, Samuel Lelièvre
Report Upstream: N/A Work issues:
Branch: 41d7e7a (Commits, GitHub, GitLab) Commit: 41d7e7a74441e5f9deb3e7a93adfccf486dde417
Dependencies: #31050 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

This contains the upgrade to Python 3.9.1

To test, use

./configure --without-system-python3 --enable-download-from-upstream-url

Change History (45)

comment:1 Changed 2 years ago by Matthias Köppe

Dependencies: #30184

comment:2 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:3 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:4 Changed 2 years ago by Matthias Köppe

Branch: u/mkoeppe/upgrade-python-3.9

comment:5 Changed 2 years ago by git

Commit: e188c5ecca99f7443cdca2d9441aabf200ab88aa

Branch pushed to git repo; I updated commit sha1. New commits:

e188c5esrc/sage/cpython/debugimpl.c: Disable _type_debug for Python 3.9

comment:6 Changed 2 years ago by git

Commit: e188c5ecca99f7443cdca2d9441aabf200ab88aa5e89ab2f5596076ec659b43612b1a84beea03b81

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

d3dd329Merge branch 't/30595/remove_deprecated_sage_libs_ppl' into t/30184/support_python_3_9
68ef4c5src/sage/cpython/debugimpl.c: Disable _type_debug for Python 3.9
f3cb571src/sage/all.py: Filter out Python 3.9 deprecation warnings from ast
e16ded2src/sage/symbolic/ring.pyx (isidentifier): Remove py2-only code, use of deprecated parser module
27a57bcsrc/sage/misc/sageinspect.py: Adjust doctest for sage_getsourcelines for Python 3.9 - decorators are now part of the definition
39416fbsrc/sage/misc/sageinspect.py (SageArgSpecVisitor): Relax doctest pattern for python 3.9
5548c0dsrc/sage/misc/latex.py (has_latex_attr): Relax doctest pattern for python 3.9
4a9362esrc/sage/combinat/root_system/pieri_factors.py: Use sorted in doctest for python 3.9
49eccd9src/sage/geometry/polyhedron/palp_database.py: Relax doctest pattern for python 3.9
5e89ab2Merge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

comment:7 Changed 2 years ago by git

Commit: 5e89ab2f5596076ec659b43612b1a84beea03b81316ae6d4881978bfcf7ed16c8f2dc2a568e930d8

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

b21ed97build/pkgs/python3/spkg-configure.m4: Ensure that PYTHON_FOR_VENV is a full path
49aa061Revert "build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'"
a8b77cdRevert "only test python3, to be 3.7 or 3.8"
9403629build/bin/sage-system-python: Work around LC_ALL=C
ff0dbc6build/sage_bootstrap/uncompress/tar_file.py: Fix encoding to utf-8
f05a110Merge branch 't/30008/after__30053__sphinx_3_1_2_does_not_build_on_ubuntu__trusty_xenial_bionic___debian_jessie__centos_7__again_' into t/30576/public/30576
945c8c5build/bin/sage-site (--docbuild): Set an English locale suuch as C.utf-8, but not the C locale, to ensure correct operation on Python 3.6
8906897Merge branch 't/30576/public/30576' into t/30546/build/py3x_conf
60137bcMerge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
316ae6dMerge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

comment:8 Changed 2 years ago by git

Commit: 316ae6d4881978bfcf7ed16c8f2dc2a568e930d85cf1bafb7692df61b243eec6fb58a09e8c2738ca

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support
9ee2110build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
d7aac84src/doc/en/developer/packaging.rst: Update sdh_... documentation
9b7c7a0build/bin/sage-pip-{install,uninstall}: Fix typo in comment
4135e8bbuild/bin/sage-pip-install: Remove an outdated comment
f2e7075Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels
a2dbb8dMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30546/build/py3x_conf
8c756edMerge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
5cf1bafMerge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

comment:9 Changed 2 years ago by git

Commit: 5cf1bafb7692df61b243eec6fb58a09e8c2738cab3c1dc1ee94b0b0cd3dd0b9ebd04ff188d26ac98

Branch pushed to git repo; I updated commit sha1. New commits:

f19d9a4allow absolute names as arguments to --with-python
4f7d588Merge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
b3c1dc1Merge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

comment:10 Changed 2 years ago by git

Commit: b3c1dc1ee94b0b0cd3dd0b9ebd04ff188d26ac986d3a0fa8249fb72dbaac0539787e870582b9cb42

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

255d22bbuild/pkgs/python3: Update to 3.9.0rc2
6d3a0fabuild/pkgs/python3: Update to 3.9.0

comment:11 Changed 2 years ago by Matthias Köppe

Dependencies: #30184
Description: modified (diff)
Summary: Test ticket: Python 3.9.0rc2Test ticket: Python 3.9.0

comment:12 Changed 2 years ago by Matthias Köppe

Reviewers: https://github.com/mkoeppe/sage/actions/runs/290402434

comment:13 Changed 2 years ago by Matthias Köppe

3.9.1 is out https://www.python.org/downloads/release/python-391/ Release Date: Dec. 7, 2020

"3.9.1 is the first version of Python to support macOS 11 Big Sur. With Xcode 11 and later it is now possible to build “Universal 2” binaries which work on Apple Silicon. We are providing such an installer as the macos11.0 variant. This installer can be deployed back to older versions, tested down to OS X 10.9. As we are waiting for an updated version of pip, please consider the macos11.0 installer experimental."

comment:14 Changed 2 years ago by Matthias Köppe

Not sure which updated version of pip they are referring to. 20.3 (2020-11-30) lists "Add support for MacOS Big Sur compatibility tags. (#9138)" https://pip.pypa.io/en/stable/news/#id15

No new news items regarding Big Sur in 20.3.3 (2020-12-15)

comment:15 Changed 2 years ago by git

Commit: 6d3a0fa8249fb72dbaac0539787e870582b9cb42f8bb56d16600c8cfe1bab1d127f3931ca280dff0

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

29aee87build/pkgs/python3: Update to 3.9.0rc2
e61d464build/pkgs/python3: Update to 3.9.0
f8bb56dbuild/pkgs/python3: Update to 3.9.1

comment:16 Changed 2 years ago by Matthias Köppe

Description: modified (diff)
Milestone: sage-pendingsage-9.3
Reviewers: https://github.com/mkoeppe/sage/actions/runs/290402434https://github.com/mkoeppe/sage/actions/runs/424627107
Summary: Test ticket: Python 3.9.0Upgrade Python to 3.9.1

comment:17 Changed 2 years ago by John Palmieri

If this works (and I'll test it out soon), it should be a blocker, since I can't build Sage's Python 3.8 on Big Sur.

comment:18 Changed 2 years ago by John Palmieri

This builds for me on Big Sur. With the old version of wheel, I (not surprisingly) hit the problem fixed by #31050. With an updated wheel, gmpy2 still fails to build, with this error:

Successfully built gmpy2
Removed build tracker: '/private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-dw7p5f40'
Using pip 20.2.3 from /Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because site-packages writeable
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-ephem-wheel-cache-nd6d05pj
Created temporary directory: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4
Initialized build tracking at /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4
Created build tracker: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4
Entered build tracker: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4
Created temporary directory: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-install-82a1edim
ERROR: gmpy2-2.1.0b5-cp39-cp39-macosx_10_9_x86_64.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 173, in resolve
    requirement_set.add_requirement(req)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/req/req_set.py", line 104, in add_requirement
    raise InstallationError(
pip._internal.exceptions.InstallationError: gmpy2-2.1.0b5-cp39-cp39-macosx_10_9_x86_64.whl is not a supported wheel on this platform.
Removed build tracker: '/private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4'
Error: installing with pip3 failed

Looks like a problem with MACOSX_DEPLOYMENT_TARGET.

comment:19 Changed 2 years ago by Matthias Köppe

We should probably also update to latest pip

comment:20 Changed 2 years ago by Matthias Köppe

Dependencies: #31050

comment:21 Changed 2 years ago by git

Commit: f8bb56d16600c8cfe1bab1d127f3931ca280dff041d7e7a74441e5f9deb3e7a93adfccf486dde417

Branch pushed to git repo; I updated commit sha1. New commits:

0e3513fbuild/pkgs/pip: Update to 20.3.3
6dc8e7ctrac 31050: upgrade wheel to 0.36.2
41d7e7aMerge branch 'u/jhpalmieri/new_wheel' of git://trac.sagemath.org/sage into t/30589/upgrade-python-3.9

comment:22 Changed 2 years ago by Matthias Köppe

Authors: Matthias Koeppe
Summary: Upgrade Python to 3.9.1Upgrade Python to 3.9.1, pip to 20.3.3

comment:23 Changed 2 years ago by John Palmieri

Great, with the new pip and new wheel, everything works until I hit scipy. The error is not one I remember seeing before:

  /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib build/temp.macosx-10.9-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.12_1/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib -Lbuild/temp.macosx-10.9-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-10.9-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so
  ld: library not found for -lSystem
  collect2: error: ld returned 1 exit status
  error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib build/temp.macosx-10.9-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.12_1/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib -Lbuild/temp.macosx-10.9-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-10.9-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

I'm guessing that this should be dealt with separately.

I would like to give this a positive review, but since I can't build Sage, I also can't build the documentation or run doctests. So for now I will have to leave the rest of the review to others.

comment:24 Changed 2 years ago by Matthias Köppe

Reviewers: https://github.com/mkoeppe/sage/actions/runs/424627107https://github.com/mkoeppe/sage/actions/runs/426748847

comment:25 Changed 2 years ago by Matthias Köppe

We can also try to throw in an upgrade of setuptools...

comment:26 Changed 2 years ago by Matthias Köppe

Cc: Frédéric Chapoton Dima Pasechnik added
Status: newneeds_review

comment:27 in reply to:  17 Changed 2 years ago by Matthias Köppe

Replying to jhpalmieri:

If this works (and I'll test it out soon), it should be a blocker, since I can't build Sage's Python 3.8 on Big Sur.

But doesn't the python3 from XCode work?

comment:28 Changed 2 years ago by John Palmieri

Right now homebrew's python3 is shadowing Xcode's, so I don't know. I would rather not uninstall homebrew's since it is used by other homebrew packages that I want to keep.

I have bigger issues right now: I can make almost no progress building Sage with 9.3.beta4. For example, with #31050 merged with beta4, gmpy2 fails to build:

  Compiling with an SDK that doesn't seem to exist: /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk
  Please check your Xcode installation
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -O2 -g -I./src -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/include -I/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/gmpy2.c -o build/temp.macosx-11-x86_64-3.9/src/gmpy2.o -DSHARED=1
  clang: warning: no such sysroot directory: '/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk' [-Wmissing-sysroot]
  In file included from src/gmpy2.c:445:
  /usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
  #include <stdio.h>

Is my command-line tools installation broken? It is certainly true that the named sysroot directory does not exist, but instead it's .../MacOSX11.1.sdk. Where is that path stored? I've tried uninstalling and reinstalling the command-line tools, uninstalling and reinstalling homebrew and its packages, but no luck.

In contrast, with just #31050 (which was based on beta3), I see the following in the gmpy2 log file:

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I./src -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/include -I/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/gmpy2.c -o build/temp.macosx-11-x86_64-3.9/src/gmpy2.o -DSHARED=1

(same wrong path given to -I) but gmpy2 builds successfully.

Separately, on a machine running Catalina, I'm having problems with cypari...

comment:29 Changed 2 years ago by Dima Pasechnik

In a (working) Homebrew installation on Catalina, I don't have these /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk paths used by the tools - ie. gcc/clang is installed in InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin and not in /Library/....

This can be seen in config.log

comment:30 Changed 2 years ago by John Palmieri

config.log is a good place to look. I see that gfortran -v prints that it is configured using --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk. I reinstalled homebrew's gcc but it didn't help. Any ideas how this is path determined? What does gfortran -v say for you?

(For gcc/clang, I see InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin.)

comment:31 Changed 2 years ago by John Palmieri

Well, this is probably not the right thing to do in the long term, but I created a symlink connecting MacOSX11.1.sdk to MacOSX11.0.sdk. The build still fails on scipy with

  /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so
  gfortran: warning: couldn't understand version 11
  ld: library not found for -lgcc_s.10.4
  collect2: error: ld returned 1 exit status
  error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

but the other problems seem to be solved.

comment:32 in reply to:  31 Changed 2 years ago by Dima Pasechnik

Replying to jhpalmieri:

Well, this is probably not the right thing to do in the long term, but I created a symlink connecting MacOSX11.1.sdk to MacOSX11.0.sdk. The build still fails on scipy with

  /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so
  gfortran: warning: couldn't understand version 11
  ld: library not found for -lgcc_s.10.4
  collect2: error: ld returned 1 exit status
  error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

but the other problems seem to be solved.

scipy probably needs an update, and/or a hack: https://github.com/scipy/scipy/issues/13102#issuecomment-744473479

comment:33 Changed 2 years ago by John Palmieri

I'm not clever enough to figure out how to hack scipy so it builds. If I issue the command export SYSTEM_VERSION_COMPAT=1 globally before building Sage, then cython doesn't build. If I just issue it in the scipy spkg-install script, then scipy doesn't build with the same error as in comment:31.

Edit: the error with cython after executing export SYSTEM_VERSION_COMPAT=1:

ERROR: Cython-0.29.21-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform.

This is with this ticket and an updated setuptools and setuptools_scm.

Last edited 2 years ago by John Palmieri (previous) (diff)

comment:34 Changed 2 years ago by John Palmieri

I guess we should move my derailment of the discussion to #30651.

comment:35 Changed 2 years ago by Matthias Köppe

Priority: majorcritical

Does someone have time to review this ticket? The tests have already run successfully.

Note that an update to latest 3.8.x is not a good alternative:

https://www.python.org/downloads/release/python-387/ Python 3.8.7 - Release Date: Dec. 21, 2020 macOS 11 Big Sur not fully supported

Python 3.8.7 is not yet fully supported on macOS 11 Big Sur. It will install on macOS 11 Big Sur and will run on Apple Silicon Macs using Rosetta 2 translation. However, a few features do not work correctly, most noticeably those involving searching for system libraries (vs user libraries) such as ctypes.util.find_library() and in Distutils. This limitation affects both Apple Silicon and Intel processors. We are looking into improving the situation for Python 3.8.8.

comment:36 Changed 2 years ago by Matthias Köppe

Cc: Volker Braun added

comment:37 Changed 2 years ago by Matthias Köppe

Cc: François Bissey added

comment:38 Changed 2 years ago by Samuel Lelièvre

On macOS 10.14.6 with lots of Homebrew packages, successfully ran

$ git trac checkout 30589
$ source .homebrew-build-env
$ ./bootstrap -q
$ ./configure --without-system-python3 --enable-download-from-upstream-url -q
$ make -s V=0

resulting in a build of Sage 9.3.beta4 based on Python 3.9.1.

But something that worked for another Sage install currently at 9.3.beta5:

$ V=0 sage -i pynormaliz

failed here while attempting to build e-antic.

comment:39 Changed 2 years ago by John Palmieri

On OS X Big Sur + various homebrew packages, if I use

% ./configure --with-system-python3=no --enable-download-from-upstream-url 

then Sage builds (if I also incorporate various other changes from #30651), but I get doctest failures of the form

File "src/sage/misc/sagedoc.py", line 646, in sage.misc.sagedoc.format
Failed example:
    cython('\n'.join(cython_code))
Expected nothing
Got:
    ld: warning: dylib (/usr/local/lib/libmpfr.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmpxx.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgsl.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libntl.dylib) was built for newer macOS version (11.0) than being linked (10.9)

We could (a) not worry about these for now, (b) filter out these warnings, (c) tinker with MACOSX_DEPLOYMENT_TARGET, or (d) something else. I don't have an opinion about which way to go.

Last edited 2 years ago by John Palmieri (previous) (diff)

comment:40 Changed 2 years ago by Matthias Köppe

These linker warnings are annoying and I think we should filter them out for the doctests. I think it should be a separate ticket. I think I have seen them on other platforms too.

comment:41 Changed 2 years ago by Matthias Köppe

This is now #31204

comment:42 in reply to:  38 Changed 2 years ago by Matthias Köppe

Replying to slelievre:

But something that worked for another Sage install currently at 9.3.beta5:

$ V=0 sage -i pynormaliz

failed here while attempting to build e-antic.

Is this on homebrew too?

comment:43 Changed 2 years ago by Matthias Köppe

In any case, let's worry about optional packages like e-antic later.

Let's get this in please.

comment:44 Changed 2 years ago by John Palmieri

Reviewers: https://github.com/mkoeppe/sage/actions/runs/426748847John Palmieri, Samuel Lelièvre
Status: needs_reviewpositive_review

Sure, let's proceed.

comment:45 Changed 2 years ago by Volker Braun

Branch: u/mkoeppe/upgrade-python-3.941d7e7a74441e5f9deb3e7a93adfccf486dde417
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.