Opened 8 months ago

Closed 4 months ago

#30589 closed enhancement (fixed)

Upgrade Python to 3.9.1, pip to 20.3.3

Reported by: mkoeppe Owned by:
Priority: critical Milestone: sage-9.3
Component: packages: standard Keywords:
Cc: slelievre, jhpalmieri, chapoton, dimpase, vbraun, fbissey Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri, Samuel Lelièvre
Report Upstream: N/A Work issues:
Branch: 41d7e7a (Commits, GitHub, GitLab) Commit: 41d7e7a74441e5f9deb3e7a93adfccf486dde417
Dependencies: #31050 Stopgaps:

Status badges

Description (last modified by mkoeppe)

This contains the upgrade to Python 3.9.1

To test, use

./configure --without-system-python3 --enable-download-from-upstream-url

Change History (45)

comment:1 Changed 8 months ago by mkoeppe

  • Dependencies set to #30184

comment:2 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:3 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:4 Changed 8 months ago by mkoeppe

  • Branch set to u/mkoeppe/upgrade-python-3.9

comment:5 Changed 8 months ago by git

  • Commit set to e188c5ecca99f7443cdca2d9441aabf200ab88aa

Branch pushed to git repo; I updated commit sha1. New commits:

e188c5esrc/sage/cpython/debugimpl.c: Disable _type_debug for Python 3.9

comment:6 Changed 8 months ago by git

  • Commit changed from e188c5ecca99f7443cdca2d9441aabf200ab88aa to 5e89ab2f5596076ec659b43612b1a84beea03b81

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

d3dd329Merge branch 't/30595/remove_deprecated_sage_libs_ppl' into t/30184/support_python_3_9
68ef4c5src/sage/cpython/debugimpl.c: Disable _type_debug for Python 3.9
f3cb571src/sage/all.py: Filter out Python 3.9 deprecation warnings from ast
e16ded2src/sage/symbolic/ring.pyx (isidentifier): Remove py2-only code, use of deprecated parser module
27a57bcsrc/sage/misc/sageinspect.py: Adjust doctest for sage_getsourcelines for Python 3.9 - decorators are now part of the definition
39416fbsrc/sage/misc/sageinspect.py (SageArgSpecVisitor): Relax doctest pattern for python 3.9
5548c0dsrc/sage/misc/latex.py (has_latex_attr): Relax doctest pattern for python 3.9
4a9362esrc/sage/combinat/root_system/pieri_factors.py: Use sorted in doctest for python 3.9
49eccd9src/sage/geometry/polyhedron/palp_database.py: Relax doctest pattern for python 3.9
5e89ab2Merge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

comment:7 Changed 8 months ago by git

  • Commit changed from 5e89ab2f5596076ec659b43612b1a84beea03b81 to 316ae6d4881978bfcf7ed16c8f2dc2a568e930d8

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

b21ed97build/pkgs/python3/spkg-configure.m4: Ensure that PYTHON_FOR_VENV is a full path
49aa061Revert "build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'"
a8b77cdRevert "only test python3, to be 3.7 or 3.8"
9403629build/bin/sage-system-python: Work around LC_ALL=C
ff0dbc6build/sage_bootstrap/uncompress/tar_file.py: Fix encoding to utf-8
f05a110Merge branch 't/30008/after__30053__sphinx_3_1_2_does_not_build_on_ubuntu__trusty_xenial_bionic___debian_jessie__centos_7__again_' into t/30576/public/30576
945c8c5build/bin/sage-site (--docbuild): Set an English locale suuch as C.utf-8, but not the C locale, to ensure correct operation on Python 3.6
8906897Merge branch 't/30576/public/30576' into t/30546/build/py3x_conf
60137bcMerge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
316ae6dMerge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

comment:8 Changed 8 months ago by git

  • Commit changed from 316ae6d4881978bfcf7ed16c8f2dc2a568e930d8 to 5cf1bafb7692df61b243eec6fb58a09e8c2738ca

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support
9ee2110build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
d7aac84src/doc/en/developer/packaging.rst: Update sdh_... documentation
9b7c7a0build/bin/sage-pip-{install,uninstall}: Fix typo in comment
4135e8bbuild/bin/sage-pip-install: Remove an outdated comment
f2e7075Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels
a2dbb8dMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30546/build/py3x_conf
8c756edMerge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
5cf1bafMerge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

comment:9 Changed 8 months ago by git

  • Commit changed from 5cf1bafb7692df61b243eec6fb58a09e8c2738ca to b3c1dc1ee94b0b0cd3dd0b9ebd04ff188d26ac98

Branch pushed to git repo; I updated commit sha1. New commits:

f19d9a4allow absolute names as arguments to --with-python
4f7d588Merge branch 't/30546/build/py3x_conf' into t/30184/support_python_3_9
b3c1dc1Merge branch 't/30184/support_python_3_9' into t/30589/upgrade-python-3.9

comment:10 Changed 7 months ago by git

  • Commit changed from b3c1dc1ee94b0b0cd3dd0b9ebd04ff188d26ac98 to 6d3a0fa8249fb72dbaac0539787e870582b9cb42

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

255d22bbuild/pkgs/python3: Update to 3.9.0rc2
6d3a0fabuild/pkgs/python3: Update to 3.9.0

comment:11 Changed 7 months ago by mkoeppe

  • Dependencies #30184 deleted
  • Description modified (diff)
  • Summary changed from Test ticket: Python 3.9.0rc2 to Test ticket: Python 3.9.0

comment:12 Changed 7 months ago by mkoeppe

  • Reviewers set to https://github.com/mkoeppe/sage/actions/runs/290402434

comment:13 Changed 5 months ago by mkoeppe

3.9.1 is out https://www.python.org/downloads/release/python-391/ Release Date: Dec. 7, 2020

"3.9.1 is the first version of Python to support macOS 11 Big Sur. With Xcode 11 and later it is now possible to build “Universal 2” binaries which work on Apple Silicon. We are providing such an installer as the macos11.0 variant. This installer can be deployed back to older versions, tested down to OS X 10.9. As we are waiting for an updated version of pip, please consider the macos11.0 installer experimental."

comment:14 Changed 5 months ago by mkoeppe

Not sure which updated version of pip they are referring to. 20.3 (2020-11-30) lists "Add support for MacOS Big Sur compatibility tags. (#9138)" https://pip.pypa.io/en/stable/news/#id15

No new news items regarding Big Sur in 20.3.3 (2020-12-15)

comment:15 Changed 5 months ago by git

  • Commit changed from 6d3a0fa8249fb72dbaac0539787e870582b9cb42 to f8bb56d16600c8cfe1bab1d127f3931ca280dff0

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

29aee87build/pkgs/python3: Update to 3.9.0rc2
e61d464build/pkgs/python3: Update to 3.9.0
f8bb56dbuild/pkgs/python3: Update to 3.9.1

comment:16 Changed 5 months ago by mkoeppe

  • Description modified (diff)
  • Milestone changed from sage-pending to sage-9.3
  • Reviewers changed from https://github.com/mkoeppe/sage/actions/runs/290402434 to https://github.com/mkoeppe/sage/actions/runs/424627107
  • Summary changed from Test ticket: Python 3.9.0 to Upgrade Python to 3.9.1

comment:17 follow-up: Changed 5 months ago by jhpalmieri

If this works (and I'll test it out soon), it should be a blocker, since I can't build Sage's Python 3.8 on Big Sur.

comment:18 Changed 5 months ago by jhpalmieri

This builds for me on Big Sur. With the old version of wheel, I (not surprisingly) hit the problem fixed by #31050. With an updated wheel, gmpy2 still fails to build, with this error:

Successfully built gmpy2
Removed build tracker: '/private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-dw7p5f40'
Using pip 20.2.3 from /Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because site-packages writeable
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-ephem-wheel-cache-nd6d05pj
Created temporary directory: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4
Initialized build tracking at /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4
Created build tracker: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4
Entered build tracker: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4
Created temporary directory: /private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-install-82a1edim
ERROR: gmpy2-2.1.0b5-cp39-cp39-macosx_10_9_x86_64.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 173, in resolve
    requirement_set.add_requirement(req)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib/python3.9/site-packages/pip/_internal/req/req_set.py", line 104, in add_requirement
    raise InstallationError(
pip._internal.exceptions.InstallationError: gmpy2-2.1.0b5-cp39-cp39-macosx_10_9_x86_64.whl is not a supported wheel on this platform.
Removed build tracker: '/private/var/folders/z6/yjw_7s357yx3_mhh81__lplc0000gn/T/pip-req-tracker-d9f2exe4'
Error: installing with pip3 failed

Looks like a problem with MACOSX_DEPLOYMENT_TARGET.

comment:19 Changed 5 months ago by mkoeppe

We should probably also update to latest pip

comment:20 Changed 5 months ago by mkoeppe

  • Dependencies set to #31050

comment:21 Changed 5 months ago by git

  • Commit changed from f8bb56d16600c8cfe1bab1d127f3931ca280dff0 to 41d7e7a74441e5f9deb3e7a93adfccf486dde417

Branch pushed to git repo; I updated commit sha1. New commits:

0e3513fbuild/pkgs/pip: Update to 20.3.3
6dc8e7ctrac 31050: upgrade wheel to 0.36.2
41d7e7aMerge branch 'u/jhpalmieri/new_wheel' of git://trac.sagemath.org/sage into t/30589/upgrade-python-3.9

comment:22 Changed 5 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Summary changed from Upgrade Python to 3.9.1 to Upgrade Python to 3.9.1, pip to 20.3.3

comment:23 Changed 5 months ago by jhpalmieri

Great, with the new pip and new wheel, everything works until I hit scipy. The error is not one I remember seeing before:

  /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib build/temp.macosx-10.9-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.12_1/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib -Lbuild/temp.macosx-10.9-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-10.9-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so
  ld: library not found for -lSystem
  collect2: error: ld returned 1 exit status
  error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib build/temp.macosx-10.9-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.12_1/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta3/local/lib -Lbuild/temp.macosx-10.9-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-10.9-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

I'm guessing that this should be dealt with separately.

I would like to give this a positive review, but since I can't build Sage, I also can't build the documentation or run doctests. So for now I will have to leave the rest of the review to others.

comment:24 Changed 5 months ago by mkoeppe

  • Reviewers changed from https://github.com/mkoeppe/sage/actions/runs/424627107 to https://github.com/mkoeppe/sage/actions/runs/426748847

comment:25 Changed 5 months ago by mkoeppe

We can also try to throw in an upgrade of setuptools...

comment:26 Changed 5 months ago by mkoeppe

  • Cc chapoton dimpase added
  • Status changed from new to needs_review

comment:27 in reply to: ↑ 17 Changed 5 months ago by mkoeppe

Replying to jhpalmieri:

If this works (and I'll test it out soon), it should be a blocker, since I can't build Sage's Python 3.8 on Big Sur.

But doesn't the python3 from XCode work?

comment:28 Changed 5 months ago by jhpalmieri

Right now homebrew's python3 is shadowing Xcode's, so I don't know. I would rather not uninstall homebrew's since it is used by other homebrew packages that I want to keep.

I have bigger issues right now: I can make almost no progress building Sage with 9.3.beta4. For example, with #31050 merged with beta4, gmpy2 fails to build:

  Compiling with an SDK that doesn't seem to exist: /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk
  Please check your Xcode installation
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -O2 -g -I./src -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/include -I/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/gmpy2.c -o build/temp.macosx-11-x86_64-3.9/src/gmpy2.o -DSHARED=1
  clang: warning: no such sysroot directory: '/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk' [-Wmissing-sysroot]
  In file included from src/gmpy2.c:445:
  /usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
  #include <stdio.h>

Is my command-line tools installation broken? It is certainly true that the named sysroot directory does not exist, but instead it's .../MacOSX11.1.sdk. Where is that path stored? I've tried uninstalling and reinstalling the command-line tools, uninstalling and reinstalling homebrew and its packages, but no luck.

In contrast, with just #31050 (which was based on beta3), I see the following in the gmpy2 log file:

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I./src -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/include -I/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/gmpy2.c -o build/temp.macosx-11-x86_64-3.9/src/gmpy2.o -DSHARED=1

(same wrong path given to -I) but gmpy2 builds successfully.

Separately, on a machine running Catalina, I'm having problems with cypari...

comment:29 Changed 5 months ago by dimpase

In a (working) Homebrew installation on Catalina, I don't have these /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk paths used by the tools - ie. gcc/clang is installed in InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin and not in /Library/....

This can be seen in config.log

comment:30 Changed 5 months ago by jhpalmieri

config.log is a good place to look. I see that gfortran -v prints that it is configured using --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk. I reinstalled homebrew's gcc but it didn't help. Any ideas how this is path determined? What does gfortran -v say for you?

(For gcc/clang, I see InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin.)

comment:31 follow-up: Changed 5 months ago by jhpalmieri

Well, this is probably not the right thing to do in the long term, but I created a symlink connecting MacOSX11.1.sdk to MacOSX11.0.sdk. The build still fails on scipy with

  /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so
  gfortran: warning: couldn't understand version 11
  ld: library not found for -lgcc_s.10.4
  collect2: error: ld returned 1 exit status
  error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

but the other problems seem to be solved.

comment:32 in reply to: ↑ 31 Changed 5 months ago by dimpase

Replying to jhpalmieri:

Well, this is probably not the right thing to do in the long term, but I created a symlink connecting MacOSX11.1.sdk to MacOSX11.0.sdk. The build still fails on scipy with

  /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so
  gfortran: warning: couldn't understand version 11
  ld: library not found for -lgcc_s.10.4
  collect2: error: ld returned 1 exit status
  error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle -bundle -undefined dynamic_lookup -L/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage/sage_builds/TESTING/TEMP/sage-9.3.beta4/local/lib build/temp.macosx-11-x86_64-3.9/scipy/integrate/_quadpackmodule.o -L/usr/local/Cellar/openblas/0.3.13/lib -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -Lbuild/temp.macosx-11-x86_64-3.9 -lquadpack -lmach -lopenblas -lopenblas -lgfortran -o build/lib.macosx-11-x86_64-3.9/scipy/integrate/_quadpack.cpython-39-darwin.so" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

but the other problems seem to be solved.

scipy probably needs an update, and/or a hack: https://github.com/scipy/scipy/issues/13102#issuecomment-744473479

comment:33 Changed 5 months ago by jhpalmieri

I'm not clever enough to figure out how to hack scipy so it builds. If I issue the command export SYSTEM_VERSION_COMPAT=1 globally before building Sage, then cython doesn't build. If I just issue it in the scipy spkg-install script, then scipy doesn't build with the same error as in comment:31.

Edit: the error with cython after executing export SYSTEM_VERSION_COMPAT=1:

ERROR: Cython-0.29.21-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform.

This is with this ticket and an updated setuptools and setuptools_scm.

Last edited 5 months ago by jhpalmieri (previous) (diff)

comment:34 Changed 5 months ago by jhpalmieri

I guess we should move my derailment of the discussion to #30651.

comment:35 Changed 5 months ago by mkoeppe

  • Priority changed from major to critical

Does someone have time to review this ticket? The tests have already run successfully.

Note that an update to latest 3.8.x is not a good alternative:

https://www.python.org/downloads/release/python-387/ Python 3.8.7 - Release Date: Dec. 21, 2020 macOS 11 Big Sur not fully supported

Python 3.8.7 is not yet fully supported on macOS 11 Big Sur. It will install on macOS 11 Big Sur and will run on Apple Silicon Macs using Rosetta 2 translation. However, a few features do not work correctly, most noticeably those involving searching for system libraries (vs user libraries) such as ctypes.util.find_library() and in Distutils. This limitation affects both Apple Silicon and Intel processors. We are looking into improving the situation for Python 3.8.8.

comment:36 Changed 5 months ago by mkoeppe

  • Cc vbraun added

comment:37 Changed 5 months ago by mkoeppe

  • Cc fbissey added

comment:38 follow-up: Changed 4 months ago by slelievre

On macOS 10.14.6 with lots of Homebrew packages, successfully ran

$ git trac checkout 30589
$ source .homebrew-build-env
$ ./bootstrap -q
$ ./configure --without-system-python3 --enable-download-from-upstream-url -q
$ make -s V=0

resulting in a build of Sage 9.3.beta4 based on Python 3.9.1.

But something that worked for another Sage install currently at 9.3.beta5:

$ V=0 sage -i pynormaliz

failed here while attempting to build e-antic.

comment:39 Changed 4 months ago by jhpalmieri

On OS X Big Sur + various homebrew packages, if I use

% ./configure --with-system-python3=no --enable-download-from-upstream-url 

then Sage builds (if I also incorporate various other changes from #30651), but I get doctest failures of the form

File "src/sage/misc/sagedoc.py", line 646, in sage.misc.sagedoc.format
Failed example:
    cython('\n'.join(cython_code))
Expected nothing
Got:
    ld: warning: dylib (/usr/local/lib/libmpfr.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmpxx.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgsl.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libntl.dylib) was built for newer macOS version (11.0) than being linked (10.9)

We could (a) not worry about these for now, (b) filter out these warnings, (c) tinker with MACOSX_DEPLOYMENT_TARGET, or (d) something else. I don't have an opinion about which way to go.

Last edited 4 months ago by jhpalmieri (previous) (diff)

comment:40 Changed 4 months ago by mkoeppe

These linker warnings are annoying and I think we should filter them out for the doctests. I think it should be a separate ticket. I think I have seen them on other platforms too.

comment:41 Changed 4 months ago by mkoeppe

This is now #31204

comment:42 in reply to: ↑ 38 Changed 4 months ago by mkoeppe

Replying to slelievre:

But something that worked for another Sage install currently at 9.3.beta5:

$ V=0 sage -i pynormaliz

failed here while attempting to build e-antic.

Is this on homebrew too?

comment:43 Changed 4 months ago by mkoeppe

In any case, let's worry about optional packages like e-antic later.

Let's get this in please.

comment:44 Changed 4 months ago by jhpalmieri

  • Reviewers changed from https://github.com/mkoeppe/sage/actions/runs/426748847 to John Palmieri, Samuel Lelièvre
  • Status changed from needs_review to positive_review

Sure, let's proceed.

comment:45 Changed 4 months ago by vbraun

  • Branch changed from u/mkoeppe/upgrade-python-3.9 to 41d7e7a74441e5f9deb3e7a93adfccf486dde417
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.