Opened 17 months ago

Closed 16 months ago

Last modified 5 months ago

#31335 closed defect (fixed)

homebrew: Unused packages (singular, pari, ...) in /usr/local leak into sagelib/cysignals builds via distutils.cfg

Reported by: mkoeppe Owned by:
Priority: critical Milestone: sage-9.3
Component: build Keywords:
Cc: jhpalmieri, gh-zlscherr, fbissey, gh-kliem, slelievre Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: e03c5ac (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

Follow-up from #31132:

It has been reported that it's still leaking with the upgraded python3 from homebrew that includes https://github.com/Homebrew/homebrew-core/commit/784d29205ee5bd27fed226a5f96837428e171825

The sagelib build collects many -I, -L directives through pkgconfig. Such directives on the command line take precedence over the search paths set by sage-env in $CPATH, $LIBRARY_PATH. Now if -I/usr/local/ or -L/usr/local (instead of more specific paths to /usr/local/Cellar/.../...) appear, then unwanted packages that happen to be installed can leak in.

However, here the problem is caused by homebrew installing a distutils.cfg that injects /usr/local/include into our include search path. We fix the leak by switching to a modern configuration of setuptools.

This also affects cysignals, as reported in https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/91Ibwn8RAgAJ

Relevant tickets: #13348, #14709, #29562 (+), #29607 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~); possible follow-up: #31338

Change History (70)

comment:1 Changed 17 months ago by mkoeppe

  • Dependencies set to #30589, #31132, #31227

comment:2 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:3 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:4 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:5 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:6 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:7 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:8 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:9 Changed 17 months ago by mkoeppe

To check whether this comes in through pkg-config, I used:

for a in $(pkg-config --list-all | sed 's/ .*//;'); do printf "$a: "; pkg-config --cflags $a; printf "$a: "; pkg-config --libs $a; done

However, on my system, the command line in the ticket description does not find any relevant packages that directly use /usr/local/{include,lib} (only lua and fuse do)

Last edited 17 months ago by mkoeppe (previous) (diff)

comment:10 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 17 months ago by mkoeppe

  • Dependencies changed from #30589, #31132, #31227 to #30589, #31132, #31227, #30770

comment:12 Changed 17 months ago by mkoeppe

  • Branch set to u/mkoeppe/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

comment:13 Changed 17 months ago by mkoeppe

  • Branch u/mkoeppe/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build deleted
  • Description modified (diff)

With the tickets listed in Dependencies merged (that's the branch on this ticket) and after brew install singular pari ecl, I get leakage from singular:

[sagelib-9.3.beta6] [ 1/13] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -O2 -g -march=native -I./sage/cpython -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/numpy/core/include -Ibuild/cythonized -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/tcl-tk/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp -o build/temp.macosx-10.15-x86_64-3.9/build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -DSING_NDEBUG -DOM_NDEBUG -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -std=c++11
[sagelib-9.3.beta6] In file included from build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp:672:
[sagelib-9.3.beta6] In file included from /usr/local/include/singular/Singular/libsingular.h:7:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/kernel/structs.h:23:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/kernel/polys.h:15:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/polys/monomials/ring.h:13:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/coeffs/coeffs.h:19:
[sagelib-9.3.beta6] In file included from /usr/local/include/factory/factory.h:28:
[sagelib-9.3.beta6] /usr/local/include/factory/si_log2.h:6:19: error: redefinition of 'SI_LOG2'
[sagelib-9.3.beta6] static inline int SI_LOG2(int v)

comment:14 Changed 17 months ago by mkoeppe

  • Branch set to u/mkoeppe/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

comment:15 Changed 17 months ago by mkoeppe

  • Commit set to 6d3409c8cd0c9de8680025e0dbb8ddadc4c3d4bf

I think this is a specific issue with how we include singular headers.


Last 10 new commits:

32dfaddMerge branch 't/31227/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build
3049e53Use ecl-config to determine compiler/linker flags for ecl
1041128Rename ecl config variable
fd05a97Add fallback
2ac4cc8Merge branch 'develop' of git://github.com/sagemath/sage into public/build/ecl-config
6370c34src/sage/env.py: Do not use capture_output, which requires python 3.7
31f1bbbAlso use universal_newlines instead of text, for python 3.6 compatibility
11ad40aMerge branch 'public/build/ecl-config' of git://trac.sagemath.org/sage into public/build/ecl-config
046f72fMerge branch 't/31227/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' into t/30770/public/build/ecl-config
6d3409cMerge branch 't/30770/public/build/ecl-config' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

comment:16 follow-up: Changed 17 months ago by mkoeppe

-I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular is (correctly) near the beginning of the command line, but the problem is that -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include only comes at the end -- after -I/usr/local/include (which probably comes in from CPATH set in .homebrew-build-env)

comment:17 in reply to: ↑ 16 Changed 17 months ago by mkoeppe

Replying to mkoeppe:

after -I/usr/local/include (which probably comes in from CPATH set in .homebrew-build-env)

no, it's not coming from CPATH

comment:18 Changed 17 months ago by mkoeppe

it's also not coming in from sage.env.cython_aliases or sage.env.sage_include_directories

comment:19 Changed 17 months ago by mkoeppe

cat  /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/distutils.cfg 
[install]
prefix=/usr/local
[build_ext]
include_dirs=/usr/local/include:/usr/local/opt/openssl@1.1/include:/usr/local/opt/sqlite/include:/usr/local/opt/tcl-tk/include
library_dirs=/usr/local/lib:/usr/local/opt/openssl@1.1/lib:/usr/local/opt/sqlite/lib:/usr/local/opt/tcl-tk/lib

comment:20 Changed 17 months ago by mkoeppe

There we go, it's another distribution bug in homebrew. It installs a distutils.cfg that injects the paths into our build.

homebrew has been struggling with this since at least 2014 - https://github.com/Homebrew/legacy-homebrew/issues/26272

comment:21 Changed 17 months ago by git

  • Commit changed from 6d3409c8cd0c9de8680025e0dbb8ddadc4c3d4bf to 017c35126d38d01a42d0a51e7c6bfa20f23c7eac

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

6432727Merge tag '9.3.beta2' into t/30912/sagelib__update_metadata_for_pypi_deployment
4a693f2Move build/pkgs/sagelib/src/setup.cfg to SAGE_ROOT/src, replace by symlink
ea182d7Copy changes from build/pkgs/sagelib/src to src
a1a10b9src/VERSION.txt: New
5697335src/setup.cfg: Add license_file=LICENSE.txt
deb9eb3Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment
7ad4c0eMerge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment
62e1945Merge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build
5c86b59trac 31134: update setuptools, setuptools_scm
017c351Merge commit '5c86b590e64d4c18d392a14abc2129205862d12e' of git://trac.sagemath.org/sage into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

comment:22 Changed 17 months ago by mkoeppe

  • Dependencies changed from #30589, #31132, #31227, #30770 to #30589, #31132, #31227, #30770, #31134

comment:23 Changed 17 months ago by mkoeppe

  • Dependencies changed from #30589, #31132, #31227, #30770, #31134 to #30589, #31132, #31227, #30770, #30912, #31134

comment:24 Changed 17 months ago by mkoeppe

We can work around the existence of this distutils.cfg by using setuptools (#30912) - more specifically versions of setuptools that bring and use their own copy of distutils.

setuptools 49.6.0 (in Sage 9.3.beta6) does not; >=50.0.0 does; >=50.1.0 only does if SETUPTOOLS_USE_DISTUTILS=local is set. (https://setuptools.readthedocs.io/en/latest/history.html#v50-1-0)

#31134 updates setuptools to 51.1.1.

Last edited 17 months ago by mkoeppe (previous) (diff)

comment:25 Changed 17 months ago by git

  • Commit changed from 017c35126d38d01a42d0a51e7c6bfa20f23c7eac to 00b3109cd440cb56212b0a520f7e0e889356cc3c

Branch pushed to git repo; I updated commit sha1. New commits:

00b3109build/pkgs/sagelib/spkg-install: Set SETUPTOOLS_USE_DISTUTILS=local

comment:26 Changed 17 months ago by mkoeppe

This changes behavior:

[sagelib-9.3.beta6] [ 1/13] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -O2 -g -march=native -I./sage/cpython -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/numpy/core/include -Ibuild/cythonized -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp -o build/temp.macosx-10.15-x86_64-3.9/build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -DSING_NDEBUG -DOM_NDEBUG -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -std=c++11
[sagelib-9.3.beta6] build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp:671:10: fatal error: 'gmp.h' file not found
[sagelib-9.3.beta6] #include "gmp.h"
[sagelib-9.3.beta6]          ^~~~~~~
[sagelib-9.3.beta6] 1 error generated.

comment:27 Changed 17 months ago by mkoeppe

OK, I had removed /usr/local/include from CPATH in my shell.

When I restore it (using .homebrew-build-env), it builds OK!

comment:28 Changed 17 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Description modified (diff)
  • Status changed from new to needs_review
  • Summary changed from homebrew: Unused packages (singular, pari, ...) in /usr/local leak into sagelib build to homebrew: Unused packages (singular, pari, ...) in /usr/local leak into sagelib build via distutils.cfg

comment:29 Changed 17 months ago by mkoeppe

  • Description modified (diff)

Ready for review. Depends on #30912, which also needs review.

comment:30 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:31 Changed 17 months ago by gh-zlscherr

Just tried building and got the following error in sagelib:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/zscherr/sage/develop/build/pkgs/sagelib/src/sage_setup/run_parallel.py", line 50, in apply_func_progress
    return p[0](p[1])
  File "/Users/zscherr/sage/develop/build/pkgs/sagelib/src/sage_setup/command/sage_build_ext.py", line 163, in build_extension
    objects = self.compiler.compile(sources,
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/setuptools/_distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run, **kwargs)
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/setuptools/_distutils/spawn.py", line 61, in spawn
    _cfg_target_split = [int(x) for x in _cfg_target.split('.')]
AttributeError: 'int' object has no attribute 'split'
"""

comment:32 Changed 17 months ago by gh-zlscherr

I just looked at the source code for spawn.py and it seems that _cfg_target is set to

sysconfig.get_config_var("MACOSX_DEPLOYMENT_TARGET")

which is 11 and is an int type on Big Sur.

comment:33 Changed 17 months ago by gh-zlscherr

Maybe setuptools needs a version bump?

comment:35 Changed 17 months ago by mkoeppe

Unfortunately this fix that homebrew uses (https://github.com/fxcoudert/cpython/commit/6511bf56.patch) has not made its way into upstream distutils (https://github.com/pypa/distutils/blob/main/distutils/spawn.py) and thus also not into the vendored distutils in setuptools (https://github.com/pypa/setuptools/blob/main/setuptools/_distutils/spawn.py).

comment:36 Changed 17 months ago by mkoeppe

Upstream cpython issue: https://bugs.python.org/issue42504

comment:37 Changed 17 months ago by mkoeppe

As a temporary measure, let's put the patch into our setuptools.

comment:38 Changed 17 months ago by git

  • Commit changed from 00b3109cd440cb56212b0a520f7e0e889356cc3c to 3601c510bbca6155f8e99eebf55b29e8cfe51527

Branch pushed to git repo; I updated commit sha1. New commits:

fd23c0dbuild/pkgs/setuptools/patches: Add distutils patch from https://github.com/fxcoudert/cpython/commit/6511bf56.patch
3601c51Adapt the patch to apply to the vendored distutils

comment:39 Changed 17 months ago by mkoeppe

Please test on big sur

comment:40 follow-up: Changed 17 months ago by gh-zlscherr

I was able to build successfully on Big Sur with the following packages installed via homebrew:

flint arb pari singular ecl maxima fplll

There's still the issue that documentation is not building using homebrew's python 3.9, https://groups.google.com/g/sage-devel/c/9EMs9h2i_H4, but I have no idea what's going on there.

comment:41 in reply to: ↑ 40 ; follow-up: Changed 17 months ago by mkoeppe

Replying to gh-zlscherr:

There's still the issue that documentation is not building using homebrew's python 3.9, https://groups.google.com/g/sage-devel/c/9EMs9h2i_H4, but I have no idea what's going on there.

I guess it would be worth checking for libtcl with otool -L $SAGE_LOCAL/lib/* /usr/local/lib/*

Last edited 17 months ago by mkoeppe (previous) (diff)

comment:42 Changed 17 months ago by mkoeppe

I have opened #31344 for the docbuild crash - I also see it on Catalina with homebrew's python3.9

comment:43 in reply to: ↑ 41 Changed 17 months ago by mkoeppe

Replying to mkoeppe:

I guess it would be worth checking for libtcl with otool -L $SAGE_LOCAL/lib/* /usr/local/lib/*

I didn't see anything suspicious here - so it's not just a library mismatch

comment:44 Changed 17 months ago by mkoeppe

(continuing on #31344 regarding docbuild)

Last edited 17 months ago by mkoeppe (previous) (diff)

comment:45 Changed 17 months ago by mkoeppe

  • Dependencies changed from #30589, #31132, #31227, #30770, #30912, #31134 to #30589, #31132, #31227, #30770, #30912, #31134, #31344

comment:46 Changed 17 months ago by git

  • Commit changed from 3601c510bbca6155f8e99eebf55b29e8cfe51527 to 9ad84cc0105665ac9c1988a3e8fa0914fd1eac73

Branch pushed to git repo; I updated commit sha1. New commits:

80720d7src/sage/misc/cython.py: Do not run pkgconfig at import time
9ad84ccMerge branch 't/31344/homebrew__docbuild_crashes__libtcl_atforkprepare' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

comment:47 Changed 17 months ago by gh-zlscherr

I see some doctests failing on this branch that may be related to the new distutils. In particular,

./sage -t src/sage/tests/cmdline.py

fails. One of the failing tests creates a temporary spyx file, and when it tries to run it with sage, it gives the following error

Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 87, in spawn
    raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/gcc' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/misc/cython.py", line 386, in cython
    dist.run_command("build")
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
    objects = self.compiler.compile(sources,
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zscherr/sage/develop/src/bin/sage-run-cython", line 8, in <module>
    s = load_cython(sys.argv.pop(1))
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/repl/load.py", line 68, in load_cython
    mod, dir = cython(name, compile_message=True, use_cache=True)
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/misc/cython.py", line 392, in cython
    raise RuntimeError(msg.strip())
RuntimeError: command '/usr/bin/gcc' failed with exit code 1
/Users/zscherr/.sage/temp/Zacharys-MacBook-Pro.local/69918/spyx/_Users_zscherr__sage_temp_Zacharys_MacBook_Pro_local_69734_dir_y1gtp384_sage_test_file_spyx/_Users_zscherr__sage_temp_Zacharys_MacBook_Pro_local_69734_dir_y1gtp384_sage_test_file_spyx_0.c:653:10: fatal error: 'gmp.h' file not found
#include "gmp.h"

I'm wondering if the issue is somehow related to distutils not remembering where certain libraries like gmp are located.

comment:48 Changed 17 months ago by mkoeppe

gmp used to be found simply because /usr/local/ is listed in homebrew's distutils.cfg. Now we need to update sage.misc.cython._standard_libs_libdirs so that gmp library dir information is used, for example from pkgconfig.

comment:49 Changed 17 months ago by mkoeppe

Let's address this in #31348.

comment:50 follow-up: Changed 17 months ago by mkoeppe

When you ran this test, were the environment variables set by .homebrew-build-env set?

comment:51 in reply to: ↑ 50 ; follow-up: Changed 17 months ago by gh-zlscherr

Replying to mkoeppe:

When you ran this test, were the environment variables set by .homebrew-build-env set?

at testing time they were not set.

comment:52 Changed 17 months ago by gh-zlscherr

I managed to get through all of ptest, and there are a lot of issues with not being able to find -lmpfr as well. I imagine it's the same type of issue.

comment:53 Changed 17 months ago by mkoeppe

Yes, the situation for mpfr is exactly the same. We can probably take care of it in #31348 as well.

comment:54 Changed 17 months ago by git

  • Commit changed from 9ad84cc0105665ac9c1988a3e8fa0914fd1eac73 to 9f38a3bdffd2af59193c3b974f6813e798c9bc33

Branch pushed to git repo; I updated commit sha1. New commits:

f7d698dsage_setup.docbuild.AllBuilder: stop the non-reference manual docs from being built in parallel
9f38a3bsage_setup.dpcbuild.AllBuilder: Restrict workaround to macOS

comment:55 Changed 17 months ago by git

  • Commit changed from 9f38a3bdffd2af59193c3b974f6813e798c9bc33 to a1b3abcca0aa9ab1a0cf8db5aae6c2a1ca0f7a68

Branch pushed to git repo; I updated commit sha1. New commits:

a1b3abcsage_setup.docbuild: In the workaround, do not go through build_many to build serially

comment:56 Changed 17 months ago by git

  • Commit changed from a1b3abcca0aa9ab1a0cf8db5aae6c2a1ca0f7a68 to ae4ea5530cab4742272b7bb5460cacd210b63544

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

515f899sage_setup.docbuild.AllBuilder: stop the non-reference manual docs from being built in parallel
804ebd7sage_setup.dpcbuild.AllBuilder: Restrict workaround to macOS
b4ceee5sage_setup.docbuild: In the workaround, do not go through build_many to build serially
ae4ea55Merge branch 't/31344/homebrew__docbuild_crashes__libtcl_atforkprepare' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

comment:57 Changed 17 months ago by git

  • Commit changed from ae4ea5530cab4742272b7bb5460cacd210b63544 to add7db48b2138f80fda1e8f5d4e50dbff1bc2e4c

Branch pushed to git repo; I updated commit sha1. New commits:

add7db4Merge tag '9.3.beta7' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

comment:58 in reply to: ↑ 51 Changed 17 months ago by mkoeppe

Replying to gh-zlscherr:

Replying to mkoeppe:

When you ran this test, were the environment variables set by .homebrew-build-env set?

at testing time they were not set.

Shall we consider this issue -- .homebrew-build-env necessary to be set during testing -- one that can be addressed in follow-up tickets? (#31348, #31365)

comment:59 Changed 17 months ago by gh-zlscherr

That sounds fair. With .homebrew-build-env sourced I didn't have any out of the ordinary issues with make ptest, so it seems that things are working well modulo the runtime stuff.

comment:60 Changed 16 months ago by git

  • Commit changed from add7db48b2138f80fda1e8f5d4e50dbff1bc2e4c to 4e777e7c9ebe47d1b9f72d625deee6c01f77c14a

Branch pushed to git repo; I updated commit sha1. New commits:

4e777e7Merge tag '9.3.beta8' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

comment:61 Changed 16 months ago by mkoeppe

  • Dependencies #30589, #31132, #31227, #30770, #30912, #31134, #31344 deleted

All dependencies have been merged. Let's get this in?

comment:62 Changed 16 months ago by mkoeppe

  • Cc slelievre added
  • Description modified (diff)
  • Summary changed from homebrew: Unused packages (singular, pari, ...) in /usr/local leak into sagelib build via distutils.cfg to homebrew: Unused packages (singular, pari, ...) in /usr/local leak into sagelib/cysignals builds via distutils.cfg

This is arguably another homebrew distribution bug.

Last edited 16 months ago by mkoeppe (previous) (diff)

comment:63 Changed 16 months ago by git

  • Commit changed from 4e777e7c9ebe47d1b9f72d625deee6c01f77c14a to bf433e8265a0f2d6ad36491f26908e8f1be48446

Branch pushed to git repo; I updated commit sha1. New commits:

bf433e8build/bin/sage-build-env-config.in: Set SETUPTOOLS_USE_DISTUTILS here, not in build/pkgs/sagelib/spkg-install

comment:64 Changed 16 months ago by git

  • Commit changed from bf433e8265a0f2d6ad36491f26908e8f1be48446 to e03c5ac0a9d2d82dd864d6d6d9560eddabbcbba5

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e03c5acbuild/bin/sage-build-env: Set SETUPTOOLS_USE_DISTUTILS here, not in build/pkgs/sagelib/spkg-install

comment:65 Changed 16 months ago by mkoeppe

Prompted by slelievre's report on sage-devel, applying this fix to all packages by putting the setting in build/bin/sage-build-env. Needs review

comment:66 Changed 16 months ago by mkoeppe

Confirmed by @slelievre to fix the cysignals build in https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/O1VHLlUKAwAJ

comment:67 Changed 16 months ago by jhpalmieri

  • Reviewers set to John Palmieri
  • Status changed from needs_review to positive_review

I can confirm, too: if I install pari and singular via homebrew, then cysignals doesn't build without this fix.

comment:68 Changed 16 months ago by mkoeppe

Thanks!

comment:69 Changed 16 months ago by vbraun

  • Branch changed from u/mkoeppe/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build to e03c5ac0a9d2d82dd864d6d6d9560eddabbcbba5
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:70 Changed 5 months ago by mkoeppe

  • Commit e03c5ac0a9d2d82dd864d6d6d9560eddabbcbba5 deleted

The problem with distutils.cfg has now been addressed by homebrew for their Python 3.10 package, see https://trac.sagemath.org/ticket/31348#comment:38

Note: See TracTickets for help on using tickets.