Opened 23 months ago
Closed 21 months ago
#31326 closed defect (fixed)
macos11.0: scipy build fails
Reported by:  Matthias Köppe  Owned by:  

Priority:  blocker  Milestone:  sage9.3 
Component:  porting  Keywords:  
Cc:  John Palmieri, Zachary L Scherr, Dima Pasechnik, ghkliem, Isuru Fernando  Merged in:  
Authors:  Zachary Scherr  Reviewers:  Matthias Koeppe 
Report Upstream:  N/A  Work issues:  
Branch:  4cc8c4d (Commits, GitHub, GitLab)  Commit:  4cc8c4da3ab35999e0746c498105658a24a224c0 
Dependencies:  Stopgaps: 
Description (last modified by )
(from #31227)
Affected configurations (From https://groups.google.com/g/sagerelease/c/KdSKg6RdZok/m/GXfB1rW1AgAJ)
 localmacos (homebrewmacos, minimal, default, macos11.0) https://github.com/mkoeppe/sage/runs/2103782309  Sage installs python3 from spkg
 localmacos (condaforgemacos, standard, default, macos11.0) https://github.com/mkoeppe/sage/runs/2103782333  uses python3 from conda
 localmacos (homebrewmacospython3_pythonorg, standard, default, macos11.0) https://github.com/mkoeppe/sage/runs/2103782375  uses somewhat dated python3 from python.org  https://www.python.org/ftp/python/3.7.7/python3.7.7macosx10.9.pkg
 another failure report, for a configuration possibly similar to localmacos (homebrewmacos, minimal, default, macos11.0) , in #30651:
File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.9/sitepackages/numpy/distutils/ccompiler.py", line 90, in <lambda> m = lambda self, *args, **kw: func(self, *args, **kw) File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.9/sitepackages/numpy/distutils/ccompiler.py", line 657, in CCompiler_get_version version = matcher(output) File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.9/sitepackages/numpy/distutils/fcompiler/gnu.py", line 278, in version_match v = self.gnu_version_match(version_string) File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.9/sitepackages/numpy/distutils/fcompiler/gnu.py", line 80, in gnu_version_match raise ValueError(err + version_string) ValueError: A valid Fortran version was not found in this string:
Change History (37)
comment:1 Changed 21 months ago by
Description:  modified (diff) 

comment:2 Changed 21 months ago by
Description:  modified (diff) 

Priority:  critical → blocker 
comment:3 Changed 21 months ago by
comment:4 Changed 21 months ago by
just want to point out that this problem exists on bash as well. For anyone wanting to test you can, after configuring, do:
make toolchain make gfortran FC="local/bin/gfortran" make numpy
to see the error.
comment:5 followup: 9 Changed 21 months ago by
I haven't tested the full homebrewmacos11.0xcodeminimal build, but I'm guessing it's related to MACOSX_DEPLOYMENT_TARGET.
As I noted above, I can recreate the error with numpy by using sage's gfortran 9.2.0 on my machine with homebrew. As per usual I think this is because python was built with MACOSX_DEPLOYMENT_TARGET 11.0 and gfortran 9.2.0 can't recognize this value.
Now I know that at some point scipy's spkginstall.in file was changed to contain the lines:
if [ $MACOSX_VERSION ge 20 ]; then export MACOSX_DEPLOYMENT_TARGET=11.0 fi
I'm not exactly sure what is in homebrewmacos11.0xcodeminimal, but if sage is building its own gfortran and its own python using a different MACOSX_DEPLOYMENT_TARGET then I would guess that this line might be causing the problems.
It's been a while since I've thought about this problem, but it's possible that the above export might not be needed anymore since homebrew patched GCC.
comment:6 Changed 21 months ago by
Actually it probably shouldn't matter whether or not sage is building its own python. gfortran 9.2.0 just can't recognize MACOSX_DEPLOYMENT_TARGET=11.0.
comment:7 Changed 21 months ago by
Sorry for the spam, but the ticket that introduced that if statement is #31183
comment:9 Changed 21 months ago by
Replying to ghzlscherr:
I'm not exactly sure what is in homebrewmacos11.0xcodeminimal, but if sage is building its own gfortran
Yes, that's right. This configuration uses a minimal installation of homebrew, plus the packages listed in build/pkgs/_bootstrap/distros/homebrew.txt
, namely gettext autoconf automake libtool pkgconfig
comment:10 Changed 21 months ago by
I also just saw that the error reported on top is in numpy and not in scipy. I'll take a look at #29827 when I get a chance, but I know that homebrew had to specifically patch GCC to avoid this sort of problem. A viable solution might be to track the legacy option to see if sage is building its own Fortran, and if so we set MACOSX_DEPLOYMENT_TARGET to 10.something
comment:11 Changed 21 months ago by
No, it's an error building scipy  which uses numpy.distutils, which you see in the lines of the log
comment:12 Changed 21 months ago by
Ah okay. Also, do you know if the homebrew minimal test uses system for python? That might be responsible for why the deployment Target is set to 11, whereas when sage builds it's own python the deployment Target is set lower.
comment:13 followup: 25 Changed 21 months ago by
From https://groups.google.com/g/sagerelease/c/KdSKg6RdZok/m/GXfB1rW1AgAJ: Runs in which scipy fails to build:
 localmacos (homebrewmacos, minimal, default, macos11.0) https://github.com/mkoeppe/sage/runs/2103782309  Sage installs python3 from spkg
 localmacos (condaforgemacos, standard, default, macos11.0) https://github.com/mkoeppe/sage/runs/2103782333  uses python3 from conda
 localmacos (homebrewmacospython3_pythonorg, standard, default, macos11.0) https://github.com/mkoeppe/sage/runs/2103782375  uses somewhat dated python3 from python.org  https://www.python.org/ftp/python/3.7.7/python3.7.7macosx10.9.pkg
comment:14 Changed 21 months ago by
For what it's worth, if I do
% make distclean && ./configure withsystemgfortran=no && make
then the build fails on numpy
, not scipy
, but with the same sort of error:
ValueError: A valid Fortran version was not found in this string: 9.2.0
comment:15 Changed 21 months ago by
I think there are extra newlines in this version string, which confuses the numpy.distutils code
comment:16 Changed 21 months ago by
It's actually worse than just extra newlines. I tried manually patching to strip all whitespace from both ends, but that didn't solve the problem. I dumped the contents of the version_string which is causing numpy to fail into a text file and it turned out to be
"gfortran: warning: couldn't understand version 11
9.3.0"
(I tested with gcc 9.3, but the same happens with 9.2)
I can get numpy and scipy to build by manually forcing MACOSX_DEPLOYMENT_TARGET to be 10.16 but it seems that 11 and 11.0 both cause problems.
Here is a proposal for how to deal with this:
 Remove the if statements from scipy's spkginstall to set MACOSX_DEPLOYMENT_TARGET to 11.0. I don't think this is needed anymore since Homebrew patched GCC 10 but I can do some testing on that.
 At some point in the build process we take a look at MACOSX_DEPLOYMENT_TARGET, which can be easily done with something like
python3 c "import sysconfig; print(sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET'))"
if we get back a value of 11 and yet the version of gfortran we are using is less than 10.2.0 then we change MACOSX_DEPLOYMENT_TARGET to 10.16 or something like that.
comment:17 Changed 21 months ago by
But probably the second step is not necessary, it would only be needed if the user has python3 from Homebrew and yet gfortran from sage. I think just fixing up scipy's spkginstall should fix everything.
comment:18 Changed 21 months ago by
Branch:  → u/ghzlscherr/revert_scipy_MACOSX_DEPLOYMENT_TARGET 

comment:19 Changed 21 months ago by
Commit:  → 4cc8c4da3ab35999e0746c498105658a24a224c0 

Branch pushed to git repo; I updated commit sha1. New commits:
4cc8c4d  Reverted changes in scipy's spkginstall to unset MACOSX_DEPLOYMENT_TARGET to 11 on Big Sur

comment:20 Changed 21 months ago by
Status:  new → needs_review 

comment:21 Changed 21 months ago by
Thanks for investigating!
To test this in various configurations, it would be enough to merge the branch of #31492 and then push a tag to your github fork of sage.
comment:22 Changed 21 months ago by
I think I did that correctly. I'll report on what happens in the CI on my repo.
comment:23 Changed 21 months ago by
I'm not sure if there is something I need to do to get the CI working, but some of the homebrew's are already failing because of an error like:
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to homebrew.bintray.com:443
Is there something I need to do to avoid this?
comment:24 Changed 21 months ago by
There are various ways how these CI runs can intermittently fail. It's nothing that we can fix on our side as far as I can tell.
By the way, if you want, you can remove the unnecessary runs (for linux, cygwin) by editing files in .github/workflows/
. Also maxparallel
can be edited so that more macOS jobs are run in parallel.
comment:25 Changed 21 months ago by
Cc:  Isuru Fernando added 

Replying to mkoeppe:
 localmacos (condaforgemacos, standard, default, macos11.0) https://github.com/mkoeppe/sage/runs/2103782333  uses python3 from conda
@isuruf: Also this build using condaforge is affected
comment:26 Changed 21 months ago by
Description:  modified (diff) 

Summary:  homebrewmacos11.0xcodeminimal: scipy build fails → macos11.0: scipy build fails 
comment:27 Changed 21 months ago by
yes, my student also reports getting
[scipy1.5.4] clang: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=11.0'
comment:28 Changed 21 months ago by
I tried running on github CI, but it quit after a few days and only got through 10.15. I just took out 10.15 and am trying again...
comment:29 Changed 21 months ago by
I tried to run the CI tests on 11.0 but I got the following error message:
"No runner matching the specified labels was found: macos11.0"
Is there a different way I should get the tests to run on Big Sur?
comment:30 Changed 21 months ago by
https://github.com/actions/virtualenvironments/issues/2486 "macOS 11.0 pools will be transited to private preview"
comment:31 Changed 21 months ago by
gotcha. I've tried some local testing and the reversion seems to be fine, but I'll leave it to others to do more extensive testing.
comment:32 Changed 21 months ago by
Some of my recent runs in https://github.com/mkoeppe/sage/actions/workflows/tox.yml have used branches that merged this ticket and may contain some useful results for 11.0
comment:33 Changed 21 months ago by
Authors:  → Zachary Scherr 

Reviewers:  → Matthias Koeppe 
Status:  needs_review → positive_review 
 localmacos (homebrewmacos, minimal, default, macos11.0)  https://github.com/mkoeppe/sage/runs/2156548349  OK
 localmacos (homebrewmacos, standard, default, macos11.0)  https://github.com/mkoeppe/sage/runs/2156548388  OK
comment:34 Changed 21 months ago by
Also
 localmacos (condaforgemacos, standard, default, macos11.0)  https://github.com/mkoeppe/sage/runs/2135024636  OK (except for an unrelated error in docbuild)
comment:35 followup: 36 Changed 21 months ago by
I agree it works with various Pythons on OS X 11. comment:14 has not been addressed, but I guess that's not for this ticket.
comment:36 Changed 21 months ago by
Replying to jhpalmieri:
comment:14 has not been addressed, but I guess that's not for this ticket.
Yes, to fix that we need to upgrade our gcc/gfortran  that's #29703.
comment:37 Changed 21 months ago by
Branch:  u/ghzlscherr/revert_scipy_MACOSX_DEPLOYMENT_TARGET → 4cc8c4da3ab35999e0746c498105658a24a224c0 

Resolution:  → fixed 
Status:  positive_review → closed 
It seems like
numpy.distutils
gets confused by extra whitespace in the result ofgfortran dumpversion
. This might be related to the use ofzsh
as the shell.This will need investigation by someone who has already made the switch to Big Sur...