Opened 8 months ago

Closed 6 months ago

#31567 closed enhancement (fixed)

tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1

Reported by: mkoeppe Owned by:
Priority: critical Milestone: sage-9.4
Component: porting Keywords:
Cc: jhpalmieri, gh-zlscherr Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: ed1a089 (Commits, GitHub, GitLab) Commit: ed1a089963e3215ddfe1216ad01237396364519b
Dependencies: #31552, #31562, #31409, #31584 Stopgaps:

Status badges

Description (last modified by mkoeppe)

This will do a "best effort" isolation to avoid using a homebrew installation in /usr/local for anything except bootstrapping and unpacking xz tarballs + a little trick for liblzma.h, which is missing on macOS.

To test:

$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode

It can then be checked that no libraries from /usr/local leak in, using

find prefix/lib -name "*.so" -o -name "*.dylib*" -exec otool -L {} \;

(Building R is disabled because the R package build still pokes around in /usr/local and finds libintl and liblzma.)

To set specific deployment targets, use:

$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.14-nohomebrew-python3_xcode -- config.status
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.15-nohomebrew-python3_xcode -- config.status
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status

... and with python3.7 from XCode:

$ tox -e local-macos-10.14-nohomebrew-python3_xcode-python3.7 -- config.status

(Remove config.status from the end of these lines to have the actual build done.)

These environments allow building Sage for older macOS versions, for example for building a 10.14 distribution on a 10.15 machine. The wheels are then correctly tagged, for example pplpy-0.8.6-cp38-cp38-macosx_10_14_x86_64.whl.

Cannot go in the opposite direction (building for 11 on a 10.15 machine) -- because pip will refuse to install a previously built wheel that is tagged for 11.)

Change History (38)

comment:1 Changed 8 months ago by mkoeppe

  • Branch set to u/mkoeppe/tox_ini__add_local_macos_nohomebrew_environments

comment:2 Changed 8 months ago by git

  • Commit set to 87e2dbaea7bf8a59d1ecb082871219e530218366

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

87e2dbatox.ini: Add local-macos-nohomebrew and configuration facors macos-{10.14,10.15,11.1}

comment:3 Changed 8 months ago by mkoeppe

  • Summary changed from tox.ini: Add local-macos-nohomebrew environments to tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1

comment:4 Changed 8 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Description modified (diff)

comment:5 Changed 8 months ago by mkoeppe

  • Cc jhpalmieri gh-zlscherr added
  • Description modified (diff)

comment:6 Changed 8 months ago by git

  • Commit changed from 87e2dbaea7bf8a59d1ecb082871219e530218366 to 1ed930292280bd3baa7cb3b72492c966fa204030

Branch pushed to git repo; I updated commit sha1. New commits:

1ed9302tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8}

comment:7 Changed 8 months ago by git

  • Commit changed from 1ed930292280bd3baa7cb3b72492c966fa204030 to 0458f2309b3f30a9445dd0196a130f4d41dafe44

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

0458f23tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8}

comment:8 Changed 8 months ago by mkoeppe

  • Description modified (diff)

This is best tested together with #31552, #31562, which do some of the necessary isolation work.

More isolation work is needed at least for pillow and freetype - as observed in https://trac.sagemath.org/ticket/31396#comment:32

comment:9 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:10 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 8 months ago by mkoeppe

  • Dependencies set to #31552, #31562

comment:12 Changed 8 months ago by git

  • Commit changed from 0458f2309b3f30a9445dd0196a130f4d41dafe44 to 9d9c562060e86156064f3647e7b223054218c93d

Branch pushed to git repo; I updated commit sha1. New commits:

8b5b273build/pkgs/singular/checksums.ini: Use 4.2.0p1+2021-03-24+sage-2
39629ddMerge #31552
1ef46d0build/pkgs/giac: Add another patch in patches/autotools
89f61d2build/pkgs/giac: Update to 1.6.0.47p3
9d9c562Merge #31562

comment:13 Changed 8 months ago by mkoeppe

  • Status changed from new to needs_review

comment:14 Changed 8 months ago by git

  • Commit changed from 9d9c562060e86156064f3647e7b223054218c93d to 236ea3c0219e5418d244c62d3289e13d14afeee1

Branch pushed to git repo; I updated commit sha1. New commits:

236ea3ctox.ini (macos-nohomebrew): Pass compiler configuration to configure using variables other than CC, CXX.

comment:15 Changed 8 months ago by git

  • Commit changed from 236ea3c0219e5418d244c62d3289e13d14afeee1 to 92db4e40bf40a73be4570e159c4e5b36c1955451

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

92db4e4tox.ini (macos-nohomebrew): Pass compiler configuration to configure using variables other than CC, CXX.

comment:16 Changed 8 months ago by mkoeppe

  • Dependencies changed from #31552, #31562 to #31552, #31562, #31409
  • Description modified (diff)

comment:17 Changed 8 months ago by git

  • Commit changed from 92db4e40bf40a73be4570e159c4e5b36c1955451 to 7966b662147c5f58807e0f9492b35c95e9322b36

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

737b21csrc/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2
4403924src/sage/interfaces/r.py: Mark all tests # optional - rpy2
405ebb9More # optional - rpy2
b7c1576Merge tag '9.3.beta8' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional
ce3b35fMore # optional - r
49c10c1Revert "build/pkgs/{r,rpy2}: Downgrade to optional"
a67300bMerge #30383
f0a5fb5configure.ac: Add option --disable-r
f04c134Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional
7966b66Merge #31409

comment:18 Changed 8 months ago by mkoeppe

  • Priority changed from major to critical

comment:19 Changed 8 months ago by git

  • Commit changed from 7966b662147c5f58807e0f9492b35c95e9322b36 to b5edde2e54cc58bd151880293e6454c3f92150fe

Branch pushed to git repo; I updated commit sha1. New commits:

b5edde2tox.ini (macos-nohomebrew): Fix typo in PILLOW_...

comment:20 Changed 8 months ago by mkoeppe

  • Status changed from needs_review to needs_work

Now pillow does not find zlib

comment:21 Changed 8 months ago by git

  • Commit changed from b5edde2e54cc58bd151880293e6454c3f92150fe to dffcdbc75e46d08bcb1fd2ac9a5677a7a7312e2f

Branch pushed to git repo; I updated commit sha1. New commits:

dffcdbctox.ini (macos-nohomebrew): Fix up pillow build by setting ZLIB_ROOT

comment:22 Changed 8 months ago by mkoeppe

pillow still finds various libraries in /usr/local because of very creative discovery code that finds everything in /usr/local because we told it that it can have /usr/local/opt/xz/include - see _add_directory (https://github.com/python-pillow/Pillow/blob/8.1.x/setup.py#L197)

comment:23 Changed 8 months ago by git

  • Commit changed from dffcdbc75e46d08bcb1fd2ac9a5677a7a7312e2f to 3779a2525d5bbf0987a1d4ad0e42b73fb5769bc0

Branch pushed to git repo; I updated commit sha1. New commits:

3779a25tox.ini (macos-nohomebrew): Disable more libraries to remove dependencies on homebrew

comment:24 Changed 8 months ago by mkoeppe

  • Status changed from needs_work to needs_review

Fixed pillow. Now I can build a wheel in #31396 that does not have any dependencies on shared libraries in /usr/local.

comment:25 follow-up: Changed 8 months ago by jhpalmieri

For what it's worth, I see a few suspicious items in the log files. (This is on a system with homebrew and with source .homebrew-build-env in my shell init file. That may be causing some of these.)

  • cvxopt includes flags -L/usr/local/lib and -I/usr/local/include when calling gcc
  • pillow says Appending path /usr/local/Cellar/xz/5.2.5/include and then the gcc flag -I/usr/local/Cellar/xz/5.2.5/include.

comment:26 Changed 8 months ago by mkoeppe

The xz include is on purpose -- see comment added in tox.ini

comment:27 in reply to: ↑ 25 Changed 8 months ago by mkoeppe

Replying to jhpalmieri:

For what it's worth, I see a few suspicious items in the log files. (This is on a system with homebrew and with source .homebrew-build-env in my shell init file. That may be causing some of these.)

Yes, this a good setting for testing this ticket. The build using tox is designed to be isolated from the effects of .homebrew-build-env.

  • cvxopt includes flags -L/usr/local/lib and -I/usr/local/include when calling gcc

This is probably https://github.com/cvxopt/cvxopt/blob/master/setup.py#L55 in combination with a bug in our build system - #31584.

comment:28 Changed 8 months ago by mkoeppe

  • Dependencies changed from #31552, #31562, #31409 to #31552, #31562, #31409, #31584

comment:29 Changed 8 months ago by git

  • Commit changed from 3779a2525d5bbf0987a1d4ad0e42b73fb5769bc0 to ed1a089963e3215ddfe1216ad01237396364519b

Branch pushed to git repo; I updated commit sha1. New commits:

7ff26dfgit grep -l SAGE_SUITESPARSE | xargs sed -i .bak s/SAGE_SUITESPARSE_LOCALINSTALL/SAGE_SUITESPARSE_PREFIX/g
52537d1build/pkgs/cvxopt/spkg-install.in: Actually use the value of SAGE_SUITESPARSE_PREFIX
ed1a089Merge #31584

comment:30 Changed 8 months ago by jhpalmieri

  • Reviewers set to John Palmieri
  • Status changed from needs_review to positive_review

Okay, this looks good to me.

comment:31 Changed 8 months ago by mkoeppe

Thanks!

comment:32 Changed 8 months ago by jhpalmieri

By the way, should commands like make distclean also clean the .tox directory?

comment:33 Changed 8 months ago by mkoeppe

I think it should not because .tox is not a "build artifact" - if this reasoning makes sense.

comment:34 follow-up: Changed 8 months ago by jhpalmieri

That makes sense. Maybe I need to learn more about tox. What command should I use if I want to restart EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode from scratch? Just delete .tox/local-macos-nohomebrew-python3?

comment:35 in reply to: ↑ 34 Changed 8 months ago by mkoeppe

Replying to jhpalmieri:

That makes sense. Maybe I need to learn more about tox. What command should I use if I want to restart EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode from scratch? Just delete .tox/local-macos-nohomebrew-python3?

That works, but you can also pass the -r option to tox, which will start from scratch.

comment:36 Changed 8 months ago by jhpalmieri

Great, thanks!

comment:37 Changed 8 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

comment:38 Changed 6 months ago by vbraun

  • Branch changed from u/mkoeppe/tox_ini__add_local_macos_nohomebrew_environments to ed1a089963e3215ddfe1216ad01237396364519b
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.