Opened 13 months ago

Closed 12 months ago

Last modified 4 months ago

#31132 closed defect (fixed)

homebrew: Unused packages (singular, pari, ...) in /usr/local leak into build when using homebrew's python3

Reported by: mkoeppe Owned by:
Priority: critical Milestone: sage-9.3
Component: build Keywords:
Cc: mjo, jhpalmieri, gh-zlscherr, fbissey, dunfield, gh-kliem Merged in:
Authors: Matthias Koeppe Reviewers: Zachary Scherr
Report Upstream: Reported upstream. No feedback yet. Work issues:
Branch: 76d5fd1 (Commits, GitHub, GitLab) Commit:
Dependencies: #30589 Stopgaps:

Status badges

Description (last modified by mkoeppe)

As previously discussed in #30745, /usr/local is leaking into our build, through wrong orders of include and/or library directives.

In #30745, this problem was not fixed but rather some more packages were found to be usable on homebrew, so distro information was added.

The problem has been reported again with pari (building cypari2) in #31029/#30589.

This is caused by a misconfigured python3. In this ticket, we add code to detect this problem when looking for the system python at configure time.

Checking whether SageMath should install SPKG python3...
checking whether any of libpng bzip2 xz libffi is installed as or will be installed as SPKG... no
checking for python3 >= 3.6.0, < 3.10.0 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core... 
checking ... whether /usr/local/bin/python3 is good... no, this is a misconfigured Python whose sysconfig compiler/linker flags contain -I or -L options, which may cause wrong versions of libraries to leak into the build of Python packages - see https://trac.sagemath.org/ticket/31132; to use it anyway, use ./configure --with-python=/usr/local/bin/python3
checking ... whether /usr/bin/python3 is good... yes
checking for python3 >= 3.6.0, < 3.10.0 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core... /usr/bin/python3
configure: will use system package and not install SPKG python3

When --with-python=... is used, only a warning is issued.

See also:

  • #30770 cython_aliases: Use ecl-config to determine compiler/linker flags for ecl
  • #25993 Upgrade Singular

Related homebrew issues and pull requests:

Attachments (1)

config.log (180.3 KB) - added by jhpalmieri 12 months ago.

Download all attachments as: .zip

Change History (79)

comment:1 Changed 13 months ago by gh-zlscherr

I discovered another annoying leak while testing. I have octave installed via homebrew and one of its dependencies is qhull. With qhull installed via homebrew if I build sage and run

./sage -tp 8 --long --warn-long 31.1 --random-seed=0 src/sage/plot/plot3d/list_plot3d.py

then it crashes with

sage: m = matrix(RDF, 6, [sin(i^2 + j^2) for i in [0,pi/5,..,pi] for j in [0,pi/5,..,pi]]) ## line 379 ##
sage: list_plot3d(m, color='yellow', interpolation_type='linear', num_points=5) # indirect doctest ## line 380 ##
QH6249 qh_lib_check: Incorrect qhull library called.  Size of qhT for caller is 2896, but for library is 2792.
QH6256 qh_lib_check: Cannot continue.  Library 'qhull 7.2.0 (2015.2 2016/01/18)' uses a static qhT (e.g., libqhull.so)

I was able to track the problem down to matplotlib. If I do:

make matplotlib-clean
brew unlink qhull
make matplotlib
brew link qhull

then the tests pass without any issues.

comment:2 Changed 13 months ago by gh-zlscherr

I think I'll open an issue with matplotlib since it's not really a sage problem.

comment:3 Changed 13 months ago by gh-zlscherr

I created https://trac.sagemath.org/ticket/31148 to use system qhull when building matplotlib if it is present. This should fix these issues.

comment:4 Changed 13 months ago by mkoeppe

  • Cc dunfield added

In #31180, @dunfield pointed out that if we use python3 from homebrew, /usr/local/include ends up on the front of the include search path via sysconfig CFLAGS.

>>> import sysconfig
>>> sysconfig.get_config_var('CFLAGS')
'-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers'

comment:5 Changed 13 months ago by mkoeppe

Happening in setuptools._distutils.sysconfig.customize_compiler (https://github.com/pypa/setuptools/blob/main/setuptools/_distutils/sysconfig.py#L185) via __init_posix

comment:6 Changed 13 months ago by mkoeppe

By setting environment variables, one cannot override these flags, only append.

comment:7 Changed 13 months ago by mkoeppe

I think this is really a packaging bug in homebrew's python3.

comment:8 Changed 13 months ago by dunfield

I agree that this is a packaging bug in homebrew. I checked four other Python 3 .* packages (Ubuntu 20.04, CentOS 7, Python.org's macOS installer, Conda) and none of them have -Iincludes in sysconfig.get_config_var('CFLAGS').

comment:10 Changed 13 months ago by mkoeppe

  • Report Upstream changed from N/A to Reported upstream. No feedback yet.

comment:11 Changed 13 months ago by mkoeppe

  • Summary changed from homebrew: Unused packages (singular, pari, ...) in /usr/local leak into build to homebrew: Unused packages (singular, pari, ...) in /usr/local leak into build when using homebrew's python3

comment:12 Changed 13 months ago by mkoeppe

  • Branch set to u/mkoeppe/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_build_when_using_homebrew_s_python3

comment:13 Changed 13 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Commit set to 2c6002555d21e242ab5d1e25e4d1f3851e246532

Let's see what the homebrew maintainers do with this.

For now the best thing we can do is detect the issue and print a warning (or we could reject the homebrew python with this issue entirely.)


New commits:

2c60025m4/sage_check_python_for_venv.m4: Warn if python3 is misconfigured like it is currently in homebrew

comment:14 Changed 13 months ago by mkoeppe

  • Status changed from new to needs_review

comment:15 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 13 months ago by jhpalmieri

This warning is buried in config.log. Would it be better to print it at the end, along with the instructions for what additional packages should be installed, or is that too hard to implement.

comment:17 Changed 13 months ago by git

  • Commit changed from 2c6002555d21e242ab5d1e25e4d1f3851e246532 to ee679bd57302d8e422e76b422d73f83ab87a4293

Branch pushed to git repo; I updated commit sha1. New commits:

ee679bdbuild/pkgs/python3/spkg-configure.m4: Do not check sysconfig CPPFLAGS so that /usr/bin/python3 is accepted on macOS; reject broken homebrew python3 unless requested explicitly with configgure --with-python=...

comment:18 Changed 13 months ago by mkoeppe

Here's a different approach. Now actually rejecting the broken homebrew python3.

For any package, to see the reasons why a system package is rejected, one needs to scroll up in the configure output (or look into config.log)

comment:19 Changed 13 months ago by mkoeppe

  • Description modified (diff)

comment:20 Changed 13 months ago by jhpalmieri

Looks okay to me. It took one machine about 30 seconds longer to build Sage when it had to build its own Python (just one iteration, so not very scientific), so avoiding homebrew's Python for now is not placing a huge burden on developers.

comment:21 follow-up: Changed 12 months ago by mkoeppe

On my machine (I'm still running Catalina), it accepts /usr/bin/python3.

comment:22 Changed 12 months ago by mkoeppe

Let's get this in please

comment:23 Changed 12 months ago by gh-zlscherr

looks good to me on Mac. Big Sur also has /usr/bin/python3 which gets picked up and accepted.

comment:24 Changed 12 months ago by gh-zlscherr

  • Reviewers set to Zachary Scherr
  • Status changed from needs_review to positive_review

comment:25 Changed 12 months ago by gh-zlscherr

  • Status changed from positive_review to needs_work

sorry, might be premature in a positive review. I'll take a second look later on.

comment:26 in reply to: ↑ 21 Changed 12 months ago by jhpalmieri

Replying to mkoeppe:

On my machine (I'm still running Catalina), it accepts /usr/bin/python3.

Not for me:

% ./configure --with-python=/usr/bin/python3
...
checking for /usr/bin/python3... /usr/bin/python3
checking ... whether /usr/bin/python3 is good... no, the version is in the supported range, and the modules can be imported, but distutils cannot build a C extension
configure: error: the python3 selected using --with-python=/usr/bin/python3 is not suitable

(This is on a machine running Catalina.)

Last edited 12 months ago by jhpalmieri (previous) (diff)

comment:27 follow-up: Changed 12 months ago by jhpalmieri

I think that building using the included version of Python 3 on OS X is not required for this — Sage can just build its own Python — but in case anyone wants to help troubleshoot, here is an excerpt from config.log:

## -------------------------------------------------------- ##
## Checking whether SageMath should install SPKG python3... ##
## -------------------------------------------------------- ##
configure:31741: checking whether any of libpng bzip2 xz libffi is installed as or will be installed as SPKG
configure:31750: result: no
configure:31756: checking for python3 >= 3.6.0, < 3.10.0 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core
configure:31763: result: 
configure:31767: checking for /usr/bin/python3
configure:31797: result: /usr/bin/python3
configure:31812: checking ... whether /usr/bin/python3 is good
CC=gcc CXX=g++ -std=gnu++11 conftest_venv/bin/python3 conftest.py --verbose build --build-base=conftest.dir
running build
running build_ext
building 'config_check_distutils' extension
creating conftest.dir
creating conftest.dir/temp.macosx-10.14.6-x86_64-3.8
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -I/Users/jpalmier/Desktop/Sage/git/sage/conftest_venv/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c conftest.c -o conftest.dir/temp.macosx-10.14.6-x86_64-3.8/conftest.o
In file included from conftest.c:66:
In file included from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:11:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include/limits.h:21:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/limits.h:63:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
#error Unsupported architecture
 ^
In file included from conftest.c:66:
In file included from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:11:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include/limits.h:21:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/limits.h:64:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/machine/limits.h:8:2: error: architecture not supported
#error architecture not supported
 ^
In file included from conftest.c:66:
In file included from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_types.h:27:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:33:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/machine/_types.h:34:2: error: architecture not supported
#error architecture not supported
 ^
In file included from conftest.c:66:
In file included from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_types.h:27:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:55:9: error: unknown type name '__int64_t'
typedef __int64_t       __darwin_blkcnt_t;      /* total blocks */
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:56:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t       __darwin_blksize_t;     /* preferred block size */
        ^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:57:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t       __darwin_dev_t;         /* dev_t */
        ^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:60:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t      __darwin_gid_t;         /* [???] process and group IDs */
        ^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:61:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t      __darwin_id_t;          /* [XSI] pid_t, uid_t, or gid_t*/
        ^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:62:9: error: unknown type name '__uint64_t'
typedef __uint64_t      __darwin_ino64_t;       /* [???] Used for 64 bit inodes */
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:68:9: error: unknown type name '__darwin_natural_t'
typedef __darwin_natural_t __darwin_mach_port_name_t; /* Used by mach */
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:70:9: error: unknown type name '__uint16_t'; did you mean '__uint128_t'?
typedef __uint16_t      __darwin_mode_t;        /* [???] Some file attributes */
        ^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:71:9: error: unknown type name '__int64_t'
typedef __int64_t       __darwin_off_t;         /* [???] Used for file sizes */
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:72:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t       __darwin_pid_t;         /* [???] process and group IDs */
        ^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:73:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t      __darwin_sigset_t;      /* [???] signal set */
        ^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:74:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t       __darwin_suseconds_t;   /* [???] microseconds */
        ^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:75:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t      __darwin_uid_t;         /* [???] user IDs */
        ^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:76:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t      __darwin_useconds_t;    /* [???] microseconds */
        ^
note: '__uint128_t' declared here
In file included from conftest.c:66:
In file included from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_types.h:43:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t      __darwin_wctype_t;
        ^
note: '__uint128_t' declared here
In file included from conftest.c:66:
In file included from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:75:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types/_va_list.h:31:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/machine/types.h:37:2: error: architecture not supported
#error architecture not supported
 ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: command 'gcc' failed with exit status 1
configure:32019: result: no, the version is in the supported range, and the modules can be imported, but distutils cannot build a C extension
configure:32062: error: the python3 selected using --with-python=/usr/bin/python3 is not suitable

comment:28 follow-up: Changed 12 months ago by gh-zlscherr

on Big Sur, configure allowed me to use /usr/bin/python3 but then Cython failed to build.

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -O2 -g -march=native -I/Users/zscherr/sage/develop/local/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c /private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-p2i1cj72/Cython/Plex/Scanners.c -o build/temp.macosx-10.14.6-x86_64-3.8/private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-p2i1cj72/Cython/Plex/Scanners.o
  clang: error: the clang compiler does not support '-march=native'
  error: command 'gcc' failed with exit status 1

comment:29 follow-ups: Changed 12 months ago by gh-zlscherr

Also on Big Sur I am not able to build python3 through sage (with or without this ticket). Doing

source .homebrew-build-env
./configure --with-system-python3=no
make python3

ends with an error

[python3-3.8.5] ld: warning: dylib (/usr/local/lib/libintl.dylib) was built for newer macOS version (10.15) than being linked (10.9)
[python3-3.8.5] sed -e "s,@EXENAME@,/Users/zscherr/sage/develop/local/bin/python3.8," < ./Misc/python-config.in >python-config.py
[python3-3.8.5] LC_ALL=C sed -e 's,\$(\([A-Za-z0-9_]*\)),\$\{\1\},g' < Misc/python-config.sh >python-config
[python3-3.8.5] Testing importing of various modules...
[python3-3.8.5] ctypes module imported OK
[python3-3.8.5] math module imported OK
[python3-3.8.5] hashlib module imported OK
[python3-3.8.5] crypt module imported OK
[python3-3.8.5] Traceback (most recent call last):
[python3-3.8.5]   File "<string>", line 1, in <module>
[python3-3.8.5] ModuleNotFoundError: No module named 'readline'
[python3-3.8.5] readline module failed to import
[python3-3.8.5] socket module imported OK
[python3-3.8.5] Traceback (most recent call last):
[python3-3.8.5]   File "<string>", line 1, in <module>
[python3-3.8.5] ModuleNotFoundError: No module named 'zlib'
[python3-3.8.5] zlib module failed to import
[python3-3.8.5] sqlite3 module imported OK
[python3-3.8.5] _scproxy module imported OK
[python3-3.8.5] Error: One or more modules failed to import.

I don't think I've ever tried building python3 from scratch since I always have relied on homebrew's version, so I'm not sure where this is coming from.

comment:30 in reply to: ↑ 28 ; follow-up: Changed 12 months ago by mkoeppe

  • Cc gh-kliem added

Replying to gh-zlscherr:

on Big Sur, configure allowed me to use /usr/bin/python3 but then Cython failed to build.

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -O2 -g -march=native -I/Users/zscherr/sage/develop/local/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c /private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-p2i1cj72/Cython/Plex/Scanners.c -o build/temp.macosx-10.14.6-x86_64-3.8/private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-p2i1cj72/Cython/Plex/Scanners.o
  clang: error: the clang compiler does not support '-march=native'
  error: command 'gcc' failed with exit status 1

This looks like a complication that may have come in through #27122

comment:31 in reply to: ↑ 29 Changed 12 months ago by mkoeppe

Replying to gh-zlscherr:

Also on Big Sur I am not able to build python3 through sage (with or without this ticket).

I think we need the upgrade to python 3.9 for this in #30589

comment:32 in reply to: ↑ 29 ; follow-up: Changed 12 months ago by dunfield

Replying to gh-zlscherr:

I don't think I've ever tried building python3 from scratch since I always have relied on homebrew's version, so I'm not sure where this is coming from.

Looks like there are two missing libraries, readline and zlib. For brew, readline is "keg only", and I get the following message when I install it:

readline is keg-only, which means it was not symlinked into /usr/local,
because macOS provides BSD libedit.

For compilers to find readline you may need to set:
  export LDFLAGS="-L/usr/local/opt/readline/lib"
  export CPPFLAGS="-I/usr/local/opt/readline/include"

For pkg-config to find readline you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig"

I didn't check, but likely zlib is similar. The fix should be to tell configure to look in /usr/local/opt/readline in addition to /usr/local.

comment:33 in reply to: ↑ 27 ; follow-up: Changed 12 months ago by mkoeppe

Replying to jhpalmieri:

I think that building using the included version of Python 3 on OS X is not required for this — Sage can just build its own Python — but in case anyone wants to help troubleshoot, here is an excerpt from config.log:

## -------------------------------------------------------- ##
## Checking whether SageMath should install SPKG python3... ##
## -------------------------------------------------------- ##
configure:31741: checking whether any of libpng bzip2 xz libffi is installed as or will be installed as SPKG
configure:31750: result: no
configure:31756: checking for python3 >= 3.6.0, < 3.10.0 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core
configure:31763: result: 
configure:31767: checking for /usr/bin/python3
configure:31797: result: /usr/bin/python3
configure:31812: checking ... whether /usr/bin/python3 is good
CC=gcc CXX=g++ -std=gnu++11 conftest_venv/bin/python3 conftest.py --verbose build --build-base=conftest.dir
running build
running build_ext
building 'config_check_distutils' extension
creating conftest.dir
creating conftest.dir/temp.macosx-10.14.6-x86_64-3.8
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -I/Users/jpalmier/Desktop/Sage/git/sage/conftest_venv/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c conftest.c -o conftest.dir/temp.macosx-10.14.6-x86_64-3.8/conftest.o

The -arch arm64 -arch x86_64 there looks scary.

Could you post the whole config.log please

comment:34 in reply to: ↑ 32 Changed 12 months ago by mkoeppe

Replying to dunfield:

Looks like there are two missing libraries, readline and zlib. For brew, readline is "keg only", and I get the following message when I install it:

readline is keg-only, which means it was not symlinked into /usr/local,
because macOS provides BSD libedit.

For compilers to find readline you may need to set:
  export LDFLAGS="-L/usr/local/opt/readline/lib"
  export CPPFLAGS="-I/usr/local/opt/readline/include"

For pkg-config to find readline you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig"

I didn't check, but likely zlib is similar. The fix should be to tell configure to look in /usr/local/opt/readline in addition to /usr/local.

We already do that in .homebrew-build-env

comment:35 Changed 12 months ago by gh-zlscherr

I just wanted to confirm that on Big Sur I have no trouble building sage's python and Cython using ticket #30589, provided that I run configure with --with-system-python3=no.

Last edited 12 months ago by gh-zlscherr (previous) (diff)

Changed 12 months ago by jhpalmieri

comment:36 in reply to: ↑ 33 Changed 12 months ago by jhpalmieri

Replying to mkoeppe:

The -arch arm64 -arch x86_64 there looks scary.

Could you post the whole config.log please

Here it is.

comment:37 follow-up: Changed 12 months ago by gh-zlscherr

I don't know anything about this, but my guess is that -arch arm64 -arch x86_64 is because Mac is pushing to build universal binaries, so it may have been introduced in the latest Xcode/CLT on Big Sur.

comment:38 in reply to: ↑ 37 Changed 12 months ago by mkoeppe

Replying to gh-zlscherr:

I don't know anything about this, but my guess is that -arch arm64 -arch x86_64 is because Mac is pushing to build universal binaries, so it may have been introduced in the latest Xcode/CLT on Big Sur.

Yes, I have just updated my CLT to 12.3 and now also see this on my machine. (Still running on Catalina.)

comment:39 Changed 12 months ago by mkoeppe

A workaround from https://github.com/tensorflow/io/pull/1168/files is to pass ARCHFLAGS="-arch x86_64"

comment:40 Changed 12 months ago by mkoeppe

Also ARCHFLAGS="" works for me.

comment:41 Changed 12 months ago by mkoeppe

Last time ARCHFLAGS was touched - #29408

comment:42 Changed 12 months ago by mkoeppe

I have opened #31227 for trying to make python3 from CLT 12.3 usable.

comment:43 Changed 12 months ago by mkoeppe

  • Dependencies set to #30589

comment:44 Changed 12 months ago by git

  • Commit changed from ee679bd57302d8e422e76b422d73f83ab87a4293 to 76d5fd15f9bcc50a478471182f70ba8d46bf4319

Branch pushed to git repo; I updated commit sha1. New commits:

29aee87build/pkgs/python3: Update to 3.9.0rc2
e61d464build/pkgs/python3: Update to 3.9.0
f8bb56dbuild/pkgs/python3: Update to 3.9.1
0e3513fbuild/pkgs/pip: Update to 20.3.3
41d7e7aMerge branch 'u/jhpalmieri/new_wheel' of git://trac.sagemath.org/sage into t/30589/upgrade-python-3.9
76d5fd1Merge branch 't/30589/upgrade-python-3.9' into t/31132/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_build_when_using_homebrew_s_python3

comment:45 in reply to: ↑ 30 ; follow-up: Changed 12 months ago by mkoeppe

Replying to mkoeppe:

Replying to gh-zlscherr:

on Big Sur, configure allowed me to use /usr/bin/python3 but then Cython failed to build.

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -O2 -g -march=native -I/Users/zscherr/sage/develop/local/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c /private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-p2i1cj72/Cython/Plex/Scanners.c -o build/temp.macosx-10.14.6-x86_64-3.8/private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-p2i1cj72/Cython/Plex/Scanners.o
  clang: error: the clang compiler does not support '-march=native'
  error: command 'gcc' failed with exit status 1

This looks like a complication that may have come in through #27122

I have opened #31228 for this.

comment:46 Changed 12 months ago by mkoeppe

  • Status changed from needs_work to needs_review

I have merged the Python 3.9 upgrade ticket (#30589). Does anything else need changing on this ticket?

comment:47 Changed 12 months ago by gh-zlscherr

I tested this by installing as many homebrew packages as I can, and then building with --with-system-python3=no and everything builds correctly at least on Big Sur. Of course --with-system-python3=no is currently necessary since otherwise configure will use /usr/bin/python3 which needs #31228 to work.

On another note, when I test I see lots of warnings of the form:

ld: warning: dylib (/usr/local/lib/libmpfr.dylib) was built for newer macOS version (10.15) than being linked (10.9)
ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (10.15) than being linked (10.9)
ld: warning: dylib (/usr/local/lib/libgmpxx.dylib) was built for newer macOS version (10.15) than being linked (10.9)
ld: warning: dylib (/usr/local/lib/libgsl.dylib) was built for newer macOS version (10.15) than being linked (10.9)
ld: warning: dylib (/usr/local/lib/libntl.dylib) was built for newer macOS version (10.15) than being linked (10.9)

I did a little bit of poking, and it appears this might be caused by src/bin/sage-env setting MACOSX_DEPLOYMENT_TARGET to 10.9. Maybe we can open a ticket for this? The source code for sage-env mentions that this was done in response to #7095, which is a ticket from 11 years ago.

comment:48 Changed 12 months ago by jhpalmieri

#31204 silences some of those "ld: warning: ..." messages. I agree that we should also explore the proper setting (or unsetting) of MACOSX_DEPLOYMENT_TARGET.

comment:49 follow-up: Changed 12 months ago by gh-zlscherr

I don't know anything about m4 files (something to learn I guess!) but I think this is caused by build/pkgs/python3/spkg-configure.m4. It appears to set

SAGE_MACOSX_DEPLOYMENT_TARGET='legacy'

which is then picked up by sage-env.

comment:50 Changed 12 months ago by mkoeppe

We have ticket #30711 (src/bin/sage-env: Update MACOSX_DEPLOYMENT_TARGET in builds with python3 from SPKG) for this task.

comment:51 Changed 12 months ago by gh-zlscherr

Thanks. I'm happy to give this ticket a positive review since it seems to fix all issues of homebrew leaking into the builds. It would be good to sort out using /usr/bin/python3 on mac, but as far as I can tell this ticket fixes the leaking.

comment:52 Changed 12 months ago by gh-zlscherr

  • Status changed from needs_review to positive_review

comment:53 in reply to: ↑ 49 Changed 12 months ago by mkoeppe

Replying to gh-zlscherr:

I think this is caused by build/pkgs/python3/spkg-configure.m4. It appears to set

SAGE_MACOSX_DEPLOYMENT_TARGET='legacy'

which is then picked up by sage-env.

That's right, this mechanism deactivates the old code in sage-env when we use system python3 instead of building our own python.

comment:54 Changed 12 months ago by mkoeppe

Thanks!

comment:55 Changed 12 months ago by mkoeppe

  • Description modified (diff)

The Homebrew python maintainers are still struggling to fix up their python builds. I have added some links to the ticket description to track these efforts.

comment:56 in reply to: ↑ 45 Changed 12 months ago by mkoeppe

Replying to mkoeppe:

Replying to mkoeppe:

Replying to gh-zlscherr:

on Big Sur, configure allowed me to use /usr/bin/python3 but then Cython failed to build.

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -O2 -g -march=native -I/Users/zscherr/sage/develop/local/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c /private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-p2i1cj72/Cython/Plex/Scanners.c -o build/temp.macosx-10.14.6-x86_64-3.8/private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-p2i1cj72/Cython/Plex/Scanners.o
  clang: error: the clang compiler does not support '-march=native'
  error: command 'gcc' failed with exit status 1

This looks like a complication that may have come in through #27122

I have opened #31228 for this.

See also #30725.

comment:57 Changed 12 months ago by vbraun

  • Branch changed from u/mkoeppe/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_build_when_using_homebrew_s_python3 to 76d5fd15f9bcc50a478471182f70ba8d46bf4319
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:58 Changed 12 months ago by mkoeppe

  • Commit 76d5fd15f9bcc50a478471182f70ba8d46bf4319 deleted

comment:59 Changed 12 months ago by mkoeppe

And indeed the upgrade of python3 to 3.9.1_7 has repaired it.

comment:60 Changed 12 months ago by jhpalmieri

brew info python@3.9 tells me that I have 3.9.1_7 installed, but I still see this in config.log:

	CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include"
	LDFLAGS = "-L/usr/local/lib -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk"
configure:32994: result: no, this is a misconfigured Python whose sysconfig compiler/linker flags contain -I or -L options, which may cause wrong versions of libraries to leak into the build of Python packages - see https://trac.sagemath.org/ticket/31132; to use it anyway, use ./configure --with-python=/usr/local/bin/python3

comment:61 Changed 12 months ago by gh-zlscherr

very bizarre. I had to do:

brew reinstall python@3.9 --build-from-source

and that fixed it for me. Maybe the bottle hasn't been updated yet?

comment:62 Changed 12 months ago by gh-zlscherr

Although I'm not sure if I needed the --build-from-source. Maybe just try brew reinstall first?

comment:63 follow-up: Changed 12 months ago by jhpalmieri

I tried brew reinstall python@3.9 and that seems to have worked.

comment:64 Changed 12 months ago by gh-zlscherr

I mentioned this on the thread and it’s because they forgot to bump the version number I guess from 7 to 8. The next time it upgrades everything should be good without needing to reinstall.

comment:65 Changed 12 months ago by gh-zlscherr

Has anyone else tried building sage with the newest homebrew python? Configure now accepts it, but I still have issues with /usr/local leaking into the builds. Not sure why that happens now that python3-config no longer remembers /usr/local.

Last edited 12 months ago by gh-zlscherr (previous) (diff)

comment:66 Changed 12 months ago by gh-zlscherr

Although I should add that cypari build fine even with homebrew's pari was installed. The problem was that singular and pari were leaking into building sagelib, and then ecl leaked into building the manifolds documentation.

comment:67 Changed 12 months ago by mkoeppe

Follow up in #31335.

comment:68 in reply to: ↑ 63 Changed 12 months ago by jhpalmieri

Replying to jhpalmieri:

I tried brew reinstall python@3.9 and that seems to have worked.

This worked on one machine (Big Sur) but not another (Catalina). I deleted the cached download and it still didn't work, and neither did brew reinstall python@3.9 -s. Oh well, maybe 3.9.1_8 will work on this machine.

comment:69 Changed 12 months ago by mkoeppe

I think I had to do brew update first, which displayed some instructions regarding unshallowing, after which I ran brew update again.

comment:70 Changed 12 months ago by jhpalmieri

Doing brew upgrade and then forcing the upgrade to Python worked, although maybe that's because it's now version 3.9.1_8. (Pro-tip: if you've forced Sage to build with homebrew's Python, don't upgrade that Python in the middle of doctesting, at least if you want doctesting to go well.)

comment:71 follow-up: Changed 11 months ago by mjo

This is now rejecting the system python on Gentoo, I guess because of the -L in LDFLAGS:

$ python3 -m sysconfig
...
CONFIGURE_LDFLAGS = "-Wl,-O1 -Wl,--as-needed -L."
LDFLAGS = "-Wl,-O1 -Wl,--as-needed -L."

That is added to our python build for obsolete reasons (there is no Gentoo/FreeBSD anymore):

# Set LDFLAGS so we link modules with -lpython3.2 correctly.                                                                                                             
# Needed on FreeBSD unless Python 3.2 is already installed.                                                                                                              
# Please query BSD team before removing this!                                                                                                                            
append-ldflags "-L."

Can someone briefly summarize the problem that this -L. causes, so that I can get it removed?

comment:72 Changed 11 months ago by mjo

  • Cc mjo added

comment:73 in reply to: ↑ 71 Changed 11 months ago by dunfield

Replying to mjo:

Can someone briefly summarize the problem that this -L. causes, so that I can get it removed?

When building a C/C++ extension module for Python in the standard way (distutils/setuptools), compiler and linker flags are pulled from sysconfig. While you can specify additional flags for such a module, the ones in sysconfig come first in the compile/link commands, meaning that they take precedence. In the present case, that means if there is a copy of a library in . there is no way to instead use a version located elsewhere, e.g. in /usr/lib.

comment:74 Changed 11 months ago by mjo

Great, thanks. Fix is in progress.

comment:75 Changed 11 months ago by mkoeppe

Exception for -L. was done in #31358

comment:76 Changed 4 months ago by isuruf

Can't this be fixed by stripping CFLAGS and LDSHARED of -I and -L flags and then appending them to CFLAGS and LDFLAGS env variables?

Last edited 4 months ago by isuruf (previous) (diff)

comment:77 Changed 4 months ago by isuruf

-I can't be stripped, but -L can be stripped by setting LDSHARED env variable. This would fix the conda-forge issue as conda-forge only use -L and doesn't use -I.

comment:78 Changed 4 months ago by mkoeppe

Let's discuss this in #31539

Note: See TracTickets for help on using tickets.