#31326 closed defect (fixed)

macos-11.0: scipy build fails

Reported by: Matthias Köppe Owned by:
Priority: blocker Milestone: sage-9.3
Component: porting Keywords:
Cc: John Palmieri, Zachary L Scherr, Dima Pasechnik, gh-kliem, Isuru Fernando Merged in:
Authors: Zachary Scherr Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: 4cc8c4d (Commits, GitHub, GitLab) Commit: 4cc8c4da3ab35999e0746c498105658a24a224c0
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

(from #31227)

Affected configurations (From ​https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/GXfB1rW1AgAJ)

  • local-macos (homebrew-macos, minimal, default, macos-11.0) ​https://github.com/mkoeppe/sage/runs/2103782309 -- Sage installs python3 from spkg
  • local-macos (conda-forge-macos, standard, default, macos-11.0) ​https://github.com/mkoeppe/sage/runs/2103782333 - uses python3 from conda
  • local-macos (homebrew-macos-python3_pythonorg, standard, default, macos-11.0) ​https://github.com/mkoeppe/sage/runs/2103782375 - uses somewhat dated python3 from python.org - ​https://www.python.org/ftp/python/3.7.7/python-3.7.7-macosx10.9.pkg
  • another failure report, for a configuration possibly similar to local-macos (homebrew-macos, minimal, default, macos-11.0) , in #30651:
          File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.9/site-packages/numpy/distutils/ccompiler.py", line 90, in <lambda>
            m = lambda self, *args, **kw: func(self, *args, **kw)
          File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.9/site-packages/numpy/distutils/ccompiler.py", line 657, in CCompiler_get_version
            version = matcher(output)
          File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.9/site-packages/numpy/distutils/fcompiler/gnu.py", line 278, in version_match
            v = self.gnu_version_match(version_string)
          File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.9/site-packages/numpy/distutils/fcompiler/gnu.py", line 80, in gnu_version_match
            raise ValueError(err + version_string)
        ValueError: A valid Fortran version was not found in this string:
    

see also https://github.com/mkoeppe/sage/runs/2103782381

Change History (37)

comment:1 Changed 21 months ago by Matthias Köppe

Description: modified (diff)

comment:2 Changed 21 months ago by Matthias Köppe

Description: modified (diff)
Priority: criticalblocker

comment:3 Changed 21 months ago by Matthias Köppe

It seems like numpy.distutils gets confused by extra whitespace in the result of gfortran -dumpversion. This might be related to the use of zsh as the shell.

This will need investigation by someone who has already made the switch to Big Sur...

comment:4 Changed 21 months ago by Zachary L Scherr

just want to point out that this problem exists on bash as well. For anyone wanting to test you can, after configuring, do:

make toolchain
make gfortran
FC="local/bin/gfortran" make numpy

to see the error.

comment:5 Changed 21 months ago by Zachary L Scherr

I haven't tested the full homebrew-macos-11.0-xcode-minimal build, but I'm guessing it's related to MACOSX_DEPLOYMENT_TARGET.

As I noted above, I can recreate the error with numpy by using sage's gfortran 9.2.0 on my machine with homebrew. As per usual I think this is because python was built with MACOSX_DEPLOYMENT_TARGET 11.0 and gfortran 9.2.0 can't recognize this value.

Now I know that at some point scipy's spkg-install.in file was changed to contain the lines:

if [ $MACOSX_VERSION -ge 20 ]; then
     export MACOSX_DEPLOYMENT_TARGET=11.0
fi

I'm not exactly sure what is in homebrew-macos-11.0-xcode-minimal, but if sage is building its own gfortran and its own python using a different MACOSX_DEPLOYMENT_TARGET then I would guess that this line might be causing the problems.

It's been a while since I've thought about this problem, but it's possible that the above export might not be needed anymore since homebrew patched GCC.

comment:6 Changed 21 months ago by Zachary L Scherr

Actually it probably shouldn't matter whether or not sage is building its own python. gfortran 9.2.0 just can't recognize MACOSX_DEPLOYMENT_TARGET=11.0.

comment:7 Changed 21 months ago by Zachary L Scherr

Sorry for the spam, but the ticket that introduced that if statement is #31183

comment:8 Changed 21 months ago by Matthias Köppe

Could you check if the gcc upgrade in #29827 fixes this issue?

comment:9 in reply to:  5 Changed 21 months ago by Matthias Köppe

Replying to gh-zlscherr:

I'm not exactly sure what is in homebrew-macos-11.0-xcode-minimal, but if sage is building its own gfortran

Yes, that's right. This configuration uses a minimal installation of homebrew, plus the packages listed in build/pkgs/_bootstrap/distros/homebrew.txt, namely gettext autoconf automake libtool pkg-config

comment:10 Changed 21 months ago by Zachary L Scherr

I also just saw that the error reported on top is in numpy and not in scipy. I'll take a look at #29827 when I get a chance, but I know that homebrew had to specifically patch GCC to avoid this sort of problem. A viable solution might be to track the legacy option to see if sage is building its own Fortran, and if so we set MACOSX_DEPLOYMENT_TARGET to 10.something

comment:11 Changed 21 months ago by Matthias Köppe

No, it's an error building scipy -- which uses numpy.distutils, which you see in the lines of the log

comment:12 Changed 21 months ago by Zachary L Scherr

Ah okay. Also, do you know if the homebrew minimal test uses system for python? That might be responsible for why the deployment Target is set to 11, whereas when sage builds it's own python the deployment Target is set lower.

comment:13 Changed 21 months ago by Matthias Köppe

From https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/GXfB1rW1AgAJ: Runs in which scipy fails to build:

comment:14 Changed 21 months ago by John Palmieri

For what it's worth, if I do

% make distclean && ./configure --with-system-gfortran=no && make

then the build fails on numpy, not scipy, but with the same sort of error:

ValueError: A valid Fortran version was not found in this string:

9.2.0

comment:15 Changed 21 months ago by Matthias Köppe

I think there are extra newlines in this version string, which confuses the numpy.distutils code

comment:16 Changed 21 months ago by Zachary L Scherr

It's actually worse than just extra newlines. I tried manually patching to strip all whitespace from both ends, but that didn't solve the problem. I dumped the contents of the version_string which is causing numpy to fail into a text file and it turned out to be

"gfortran: warning: couldn't understand version 11

9.3.0"

(I tested with gcc 9.3, but the same happens with 9.2)

I can get numpy and scipy to build by manually forcing MACOSX_DEPLOYMENT_TARGET to be 10.16 but it seems that 11 and 11.0 both cause problems.

Here is a proposal for how to deal with this:

  1. Remove the if statements from scipy's spkg-install to set MACOSX_DEPLOYMENT_TARGET to 11.0. I don't think this is needed anymore since Homebrew patched GCC 10 but I can do some testing on that.
  1. At some point in the build process we take a look at MACOSX_DEPLOYMENT_TARGET, which can be easily done with something like
python3 -c "import sysconfig; print(sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET'))"

if we get back a value of 11 and yet the version of gfortran we are using is less than 10.2.0 then we change MACOSX_DEPLOYMENT_TARGET to 10.16 or something like that.

comment:17 Changed 21 months ago by Zachary L Scherr

But probably the second step is not necessary, it would only be needed if the user has python3 from Homebrew and yet gfortran from sage. I think just fixing up scipy's spkg-install should fix everything.

comment:18 Changed 21 months ago by Zachary L Scherr

Branch: u/gh-zlscherr/revert_scipy_MACOSX_DEPLOYMENT_TARGET

comment:19 Changed 21 months ago by git

Commit: 4cc8c4da3ab35999e0746c498105658a24a224c0

Branch pushed to git repo; I updated commit sha1. New commits:

4cc8c4dReverted changes in scipy's spkg-install to unset MACOSX_DEPLOYMENT_TARGET to 11 on Big Sur

comment:20 Changed 21 months ago by Zachary L Scherr

Status: newneeds_review

comment:21 Changed 21 months ago by Matthias Köppe

Thanks for investigating!

To test this in various configurations, it would be enough to merge the branch of #31492 and then push a tag to your github fork of sage.

comment:22 Changed 21 months ago by Zachary L Scherr

I think I did that correctly. I'll report on what happens in the CI on my repo.

comment:23 Changed 21 months ago by Zachary L Scherr

I'm not sure if there is something I need to do to get the CI working, but some of the homebrew's are already failing because of an error like:

curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to homebrew.bintray.com:443

Is there something I need to do to avoid this?

comment:24 Changed 21 months ago by Matthias Köppe

There are various ways how these CI runs can intermittently fail. It's nothing that we can fix on our side as far as I can tell.

By the way, if you want, you can remove the unnecessary runs (for linux, cygwin) by editing files in .github/workflows/. Also max-parallel can be edited so that more macOS jobs are run in parallel.

comment:25 in reply to:  13 Changed 21 months ago by Matthias Köppe

Cc: Isuru Fernando added

Replying to mkoeppe:

@isuruf: Also this build using conda-forge is affected

comment:26 Changed 21 months ago by Matthias Köppe

Description: modified (diff)
Summary: homebrew-macos-11.0-xcode-minimal: scipy build failsmacos-11.0: scipy build fails

comment:27 Changed 21 months ago by Dima Pasechnik

yes, my student also reports getting

[scipy-1.5.4]     clang: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=11.0'

comment:28 Changed 21 months ago by Zachary L Scherr

I tried running on github CI, but it quit after a few days and only got through 10.15. I just took out 10.15 and am trying again...

comment:29 Changed 21 months ago by Zachary L Scherr

I tried to run the CI tests on 11.0 but I got the following error message:

"No runner matching the specified labels was found: macos-11.0"

Is there a different way I should get the tests to run on Big Sur?

comment:30 Changed 21 months ago by Matthias Köppe

https://github.com/actions/virtual-environments/issues/2486 "macOS 11.0 pools will be transited to private preview"

comment:31 Changed 21 months ago by Zachary L Scherr

gotcha. I've tried some local testing and the reversion seems to be fine, but I'll leave it to others to do more extensive testing.

comment:32 Changed 21 months ago by Matthias Köppe

Some of my recent runs in https://github.com/mkoeppe/sage/actions/workflows/tox.yml have used branches that merged this ticket and may contain some useful results for 11.0

comment:33 Changed 21 months ago by Matthias Köppe

Authors: Zachary Scherr
Reviewers: Matthias Koeppe
Status: needs_reviewpositive_review

comment:34 Changed 21 months ago by Matthias Köppe

Also

comment:35 Changed 21 months ago by John Palmieri

I agree it works with various Pythons on OS X 11. comment:14 has not been addressed, but I guess that's not for this ticket.

Last edited 21 months ago by John Palmieri (previous) (diff)

comment:36 in reply to:  35 Changed 21 months ago by Matthias Köppe

Replying to jhpalmieri:

comment:14 has not been addressed, but I guess that's not for this ticket.

Yes, to fix that we need to upgrade our gcc/gfortran - that's #29703.

comment:37 Changed 21 months ago by Volker Braun

Branch: u/gh-zlscherr/revert_scipy_MACOSX_DEPLOYMENT_TARGET4cc8c4da3ab35999e0746c498105658a24a224c0
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.