homebrew: Unused packages (singular, pari, ...) in /usr/local leak into sagelib/cysignals builds via distutils.cfg

[mkoeppe]

Follow-up from #31132:

It has been reported that it's still leaking with the upgraded python3 from homebrew that includes ​https://github.com/Homebrew/homebrew-core/commit/784d29205ee5bd27fed226a5f96837428e171825

The sagelib build collects many -I, -L directives through pkgconfig. Such directives on the command line take precedence over the search paths set by sage-env in $CPATH, $LIBRARY_PATH. Now if -I/usr/local/ or -L/usr/local (instead of more specific paths to /usr/local/Cellar/.../...) appear, then unwanted packages that happen to be installed can leak in.

However, here the problem is caused by homebrew installing a distutils.cfg that injects /usr/local/include into our include search path. We fix the leak by switching to a modern configuration of setuptools.

This also affects cysignals, as reported in ​https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/91Ibwn8RAgAJ

Relevant tickets: #13348, #14709, #29562 (+), #29607 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~); possible follow-up: #31338

[mkoeppe]

To check whether this comes in through pkg-config, I used:

for a in $(pkg-config --list-all | sed 's/ .*//;'); do printf "$a: "; pkg-config --cflags $a; printf "$a: "; pkg-config --libs $a; done

However, on my system, the command line in the ticket description does not find any relevant packages that directly use /usr/local/{include,lib} (only lua and fuse do)

[mkoeppe]

With the tickets listed in Dependencies merged (that's the branch on this ticket) and after brew install singular pari ecl, I get leakage from singular:

[sagelib-9.3.beta6] [ 1/13] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -O2 -g -march=native -I./sage/cpython -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/numpy/core/include -Ibuild/cythonized -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/tcl-tk/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp -o build/temp.macosx-10.15-x86_64-3.9/build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -DSING_NDEBUG -DOM_NDEBUG -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -std=c++11
[sagelib-9.3.beta6] In file included from build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp:672:
[sagelib-9.3.beta6] In file included from /usr/local/include/singular/Singular/libsingular.h:7:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/kernel/structs.h:23:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/kernel/polys.h:15:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/polys/monomials/ring.h:13:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/coeffs/coeffs.h:19:
[sagelib-9.3.beta6] In file included from /usr/local/include/factory/factory.h:28:
[sagelib-9.3.beta6] /usr/local/include/factory/si_log2.h:6:19: error: redefinition of 'SI_LOG2'
[sagelib-9.3.beta6] static inline int SI_LOG2(int v)

[mkoeppe]

I think this is a specific issue with how we include singular headers.

---

Last 10 new commits:

​32dfadd	Merge branch 't/31227/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build
​3049e53	Use ecl-config to determine compiler/linker flags for ecl
​1041128	Rename ecl config variable
​fd05a97	Add fallback
​2ac4cc8	Merge branch 'develop' of git://github.com/sagemath/sage into public/build/ecl-config
​6370c34	src/sage/env.py: Do not use capture_output, which requires python 3.7
​31f1bbb	Also use universal_newlines instead of text, for python 3.6 compatibility
​11ad40a	Merge branch 'public/build/ecl-config' of git://trac.sagemath.org/sage into public/build/ecl-config
​046f72f	Merge branch 't/31227/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' into t/30770/public/build/ecl-config
​6d3409c	Merge branch 't/30770/public/build/ecl-config' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

[mkoeppe]

-I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular is (correctly) near the beginning of the command line, but the problem is that -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include only comes at the end -- after -I/usr/local/include (which probably comes in from CPATH set in .homebrew-build-env)

[mkoeppe]

Replying to mkoeppe:

after -I/usr/local/include (which probably comes in from CPATH set in .homebrew-build-env)

no, it's not coming from CPATH

[mkoeppe]

it's also not coming in from sage.env.cython_aliases or sage.env.sage_include_directories

[mkoeppe]

cat  /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/distutils.cfg 
[install]
prefix=/usr/local
[build_ext]
include_dirs=/usr/local/include:/usr/local/opt/openssl@1.1/include:/usr/local/opt/sqlite/include:/usr/local/opt/tcl-tk/include
library_dirs=/usr/local/lib:/usr/local/opt/openssl@1.1/lib:/usr/local/opt/sqlite/lib:/usr/local/opt/tcl-tk/lib

[mkoeppe]

There we go, it's another distribution bug in homebrew. It installs a distutils.cfg that injects the paths into our build.

homebrew has been struggling with this since at least 2014 - ​https://github.com/Homebrew/legacy-homebrew/issues/26272

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​6432727	Merge tag '9.3.beta2' into t/30912/sagelib__update_metadata_for_pypi_deployment
​4a693f2	Move build/pkgs/sagelib/src/setup.cfg to SAGE_ROOT/src, replace by symlink
​ea182d7	Copy changes from build/pkgs/sagelib/src to src
​a1a10b9	src/VERSION.txt: New
​5697335	src/setup.cfg: Add license_file=LICENSE.txt
​deb9eb3	Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment
​7ad4c0e	Merge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment
​62e1945	Merge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build
​5c86b59	trac 31134: update setuptools, setuptools_scm
​017c351	Merge commit '5c86b590e64d4c18d392a14abc2129205862d12e' of git://trac.sagemath.org/sage into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

[mkoeppe]

We can work around the existence of this distutils.cfg by using setuptools (#30912) - more specifically versions of setuptools that bring and use their own copy of distutils.

setuptools 49.6.0 (in Sage 9.3.beta6) does not; >=50.0.0 does; >=50.1.0 only does if SETUPTOOLS_USE_DISTUTILS=local is set. (​https://setuptools.readthedocs.io/en/latest/history.html#v50-1-0)

#31134 updates setuptools to 51.1.1.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​00b3109

build/pkgs/sagelib/spkg-install: Set SETUPTOOLS_USE_DISTUTILS=local

[mkoeppe]

This changes behavior:

[sagelib-9.3.beta6] [ 1/13] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -O2 -g -march=native -I./sage/cpython -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/numpy/core/include -Ibuild/cythonized -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp -o build/temp.macosx-10.15-x86_64-3.9/build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -DSING_NDEBUG -DOM_NDEBUG -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -std=c++11
[sagelib-9.3.beta6] build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp:671:10: fatal error: 'gmp.h' file not found
[sagelib-9.3.beta6] #include "gmp.h"
[sagelib-9.3.beta6]          ^~~~~~~
[sagelib-9.3.beta6] 1 error generated.

[mkoeppe]

OK, I had removed /usr/local/include from CPATH in my shell.

When I restore it (using .homebrew-build-env), it builds OK!

[mkoeppe]

Ready for review. Depends on #30912, which also needs review.

[gh-zlscherr]

Just tried building and got the following error in sagelib:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/zscherr/sage/develop/build/pkgs/sagelib/src/sage_setup/run_parallel.py", line 50, in apply_func_progress
    return p[0](p[1])
  File "/Users/zscherr/sage/develop/build/pkgs/sagelib/src/sage_setup/command/sage_build_ext.py", line 163, in build_extension
    objects = self.compiler.compile(sources,
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/setuptools/_distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run, **kwargs)
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/setuptools/_distutils/spawn.py", line 61, in spawn
    _cfg_target_split = [int(x) for x in _cfg_target.split('.')]
AttributeError: 'int' object has no attribute 'split'
"""

[gh-zlscherr]

I just looked at the source code for spawn.py and it seems that _cfg_target is set to

sysconfig.get_config_var("MACOSX_DEPLOYMENT_TARGET")

which is 11 and is an int type on Big Sur.

[gh-zlscherr]

Maybe setuptools needs a version bump?

[mkoeppe]

This is parallel to ​https://github.com/Homebrew/homebrew-core/issues/66096

[mkoeppe]

Unfortunately this fix that homebrew uses (​https://github.com/fxcoudert/cpython/commit/6511bf56.patch) has not made its way into upstream distutils (​https://github.com/pypa/distutils/blob/main/distutils/spawn.py) and thus also not into the vendored distutils in setuptools (​https://github.com/pypa/setuptools/blob/main/setuptools/_distutils/spawn.py).

[mkoeppe]

Upstream cpython issue: ​https://bugs.python.org/issue42504

[mkoeppe]

As a temporary measure, let's put the patch into our setuptools.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fd23c0d	build/pkgs/setuptools/patches: Add distutils patch from https://github.com/fxcoudert/cpython/commit/6511bf56.patch
​3601c51	Adapt the patch to apply to the vendored distutils

[mkoeppe]

Please test on big sur

[gh-zlscherr]

I was able to build successfully on Big Sur with the following packages installed via homebrew:

flint arb pari singular ecl maxima fplll

There's still the issue that documentation is not building using homebrew's python 3.9, ​https://groups.google.com/g/sage-devel/c/9EMs9h2i_H4, but I have no idea what's going on there.

[mkoeppe]

Replying to gh-zlscherr:

There's still the issue that documentation is not building using homebrew's python 3.9, ​https://groups.google.com/g/sage-devel/c/9EMs9h2i_H4, but I have no idea what's going on there.

I guess it would be worth checking for libtcl with otool -L $SAGE_LOCAL/lib/* /usr/local/lib/*

[mkoeppe]

I have opened #31344 for the docbuild crash - I also see it on Catalina with homebrew's python3.9

[mkoeppe]

Replying to mkoeppe:

I guess it would be worth checking for libtcl with otool -L $SAGE_LOCAL/lib/* /usr/local/lib/*

I didn't see anything suspicious here - so it's not just a library mismatch

[mkoeppe]

(continuing on #31344 regarding docbuild)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​80720d7	src/sage/misc/cython.py: Do not run pkgconfig at import time
​9ad84cc	Merge branch 't/31344/homebrew__docbuild_crashes__libtcl_atforkprepare' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

[gh-zlscherr]

I see some doctests failing on this branch that may be related to the new distutils. In particular,

./sage -t src/sage/tests/cmdline.py

fails. One of the failing tests creates a temporary spyx file, and when it tries to run it with sage, it gives the following error

Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 87, in spawn
    raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/gcc' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/misc/cython.py", line 386, in cython
    dist.run_command("build")
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
    objects = self.compiler.compile(sources,
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zscherr/sage/develop/src/bin/sage-run-cython", line 8, in <module>
    s = load_cython(sys.argv.pop(1))
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/repl/load.py", line 68, in load_cython
    mod, dir = cython(name, compile_message=True, use_cache=True)
  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/misc/cython.py", line 392, in cython
    raise RuntimeError(msg.strip())
RuntimeError: command '/usr/bin/gcc' failed with exit code 1
/Users/zscherr/.sage/temp/Zacharys-MacBook-Pro.local/69918/spyx/_Users_zscherr__sage_temp_Zacharys_MacBook_Pro_local_69734_dir_y1gtp384_sage_test_file_spyx/_Users_zscherr__sage_temp_Zacharys_MacBook_Pro_local_69734_dir_y1gtp384_sage_test_file_spyx_0.c:653:10: fatal error: 'gmp.h' file not found
#include "gmp.h"

I'm wondering if the issue is somehow related to distutils not remembering where certain libraries like gmp are located.

[mkoeppe]

gmp used to be found simply because /usr/local/ is listed in homebrew's distutils.cfg. Now we need to update sage.misc.cython._standard_libs_libdirs so that gmp library dir information is used, for example from pkgconfig.

[mkoeppe]

Let's address this in #31348.

[mkoeppe]

When you ran this test, were the environment variables set by .homebrew-build-env set?

[gh-zlscherr]

Replying to mkoeppe:

When you ran this test, were the environment variables set by .homebrew-build-env set?

at testing time they were not set.

[gh-zlscherr]

I managed to get through all of ptest, and there are a lot of issues with not being able to find -lmpfr as well. I imagine it's the same type of issue.

[mkoeppe]

Yes, the situation for mpfr is exactly the same. We can probably take care of it in #31348 as well.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f7d698d	sage_setup.docbuild.AllBuilder: stop the non-reference manual docs from being built in parallel
​9f38a3b	sage_setup.dpcbuild.AllBuilder: Restrict workaround to macOS

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a1b3abc

sage_setup.docbuild: In the workaround, do not go through build_many to build serially

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​515f899	sage_setup.docbuild.AllBuilder: stop the non-reference manual docs from being built in parallel
​804ebd7	sage_setup.dpcbuild.AllBuilder: Restrict workaround to macOS
​b4ceee5	sage_setup.docbuild: In the workaround, do not go through build_many to build serially
​ae4ea55	Merge branch 't/31344/homebrew__docbuild_crashes__libtcl_atforkprepare' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​add7db4

Merge tag '9.3.beta7' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

[mkoeppe]

Replying to gh-zlscherr:

Replying to mkoeppe:

When you ran this test, were the environment variables set by .homebrew-build-env set?

at testing time they were not set.

Shall we consider this issue -- .homebrew-build-env necessary to be set during testing -- one that can be addressed in follow-up tickets? (#31348, #31365)

[gh-zlscherr]

That sounds fair. With .homebrew-build-env sourced I didn't have any out of the ordinary issues with make ptest, so it seems that things are working well modulo the runtime stuff.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4e777e7

Merge tag '9.3.beta8' into t/31335/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

[mkoeppe]

All dependencies have been merged. Let's get this in?

[mkoeppe]

This is arguably another homebrew distribution bug.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bf433e8

build/bin/sage-build-env-config.in: Set SETUPTOOLS_USE_DISTUTILS here, not in build/pkgs/sagelib/spkg-install

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​e03c5ac

build/bin/sage-build-env: Set SETUPTOOLS_USE_DISTUTILS here, not in build/pkgs/sagelib/spkg-install

[mkoeppe]

Prompted by slelievre's report on sage-devel, applying this fix to all packages by putting the setting in build/bin/sage-build-env. Needs review

[mkoeppe]

Confirmed by @slelievre to fix the cysignals build in ​https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/O1VHLlUKAwAJ

[jhpalmieri]

I can confirm, too: if I install pari and singular via homebrew, then cysignals doesn't build without this fix.

[mkoeppe]

Thanks!

[mkoeppe]

The problem with distutils.cfg has now been addressed by homebrew for their Python 3.10 package, see ​https://trac.sagemath.org/ticket/31348#comment:38