Opened 17 months ago
Closed 15 months ago
#31567 closed enhancement (fixed)
tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | critical | Milestone: | sage-9.4 |
Component: | porting | Keywords: | |
Cc: | jhpalmieri, gh-zlscherr | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | John Palmieri |
Report Upstream: | N/A | Work issues: | |
Branch: | ed1a089 (Commits, GitHub, GitLab) | Commit: | ed1a089963e3215ddfe1216ad01237396364519b |
Dependencies: | #31552, #31562, #31409, #31584 | Stopgaps: |
Description (last modified by )
This will do a "best effort" isolation to avoid using a homebrew installation in /usr/local
for anything except bootstrapping and unpacking xz tarballs + a little trick for liblzma.h, which is missing on macOS.
To test:
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode
It can then be checked that no libraries from /usr/local leak in, using
find prefix/lib -name "*.so" -o -name "*.dylib*" -exec otool -L {} \;
(Building R is disabled because the R package build still pokes around in /usr/local and finds libintl and liblzma.)
To set specific deployment targets, use:
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.14-nohomebrew-python3_xcode -- config.status $ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.15-nohomebrew-python3_xcode -- config.status $ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status
... and with python3.7 from XCode:
$ tox -e local-macos-10.14-nohomebrew-python3_xcode-python3.7 -- config.status
(Remove config.status
from the end of these lines to have the actual build done.)
These environments allow building Sage for older macOS versions, for example for building a 10.14 distribution on a 10.15 machine. The wheels are then correctly tagged, for example pplpy-0.8.6-cp38-cp38-macosx_10_14_x86_64.whl
.
Cannot go in the opposite direction (building for 11 on a 10.15 machine) -- because pip will refuse to install a previously built wheel that is tagged for 11.)
Change History (38)
comment:1 Changed 17 months ago by
- Branch set to u/mkoeppe/tox_ini__add_local_macos_nohomebrew_environments
comment:2 Changed 17 months ago by
- Commit set to 87e2dbaea7bf8a59d1ecb082871219e530218366
comment:3 Changed 17 months ago by
- Summary changed from tox.ini: Add local-macos-nohomebrew environments to tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1
comment:4 Changed 17 months ago by
- Description modified (diff)
comment:5 Changed 17 months ago by
- Cc jhpalmieri gh-zlscherr added
- Description modified (diff)
comment:6 Changed 17 months ago by
- Commit changed from 87e2dbaea7bf8a59d1ecb082871219e530218366 to 1ed930292280bd3baa7cb3b72492c966fa204030
Branch pushed to git repo; I updated commit sha1. New commits:
1ed9302 | tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8 }
|
comment:7 Changed 17 months ago by
- Commit changed from 1ed930292280bd3baa7cb3b72492c966fa204030 to 0458f2309b3f30a9445dd0196a130f4d41dafe44
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
0458f23 | tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8 }
|
comment:8 Changed 17 months ago by
- Description modified (diff)
This is best tested together with #31552, #31562, which do some of the necessary isolation work.
More isolation work is needed at least for pillow
and freetype
- as observed in https://trac.sagemath.org/ticket/31396#comment:32
comment:9 Changed 17 months ago by
- Description modified (diff)
comment:10 Changed 17 months ago by
- Description modified (diff)
comment:11 Changed 17 months ago by
- Dependencies set to #31552, #31562
comment:12 Changed 17 months ago by
- Commit changed from 0458f2309b3f30a9445dd0196a130f4d41dafe44 to 9d9c562060e86156064f3647e7b223054218c93d
comment:13 Changed 17 months ago by
- Status changed from new to needs_review
comment:14 Changed 17 months ago by
- Commit changed from 9d9c562060e86156064f3647e7b223054218c93d to 236ea3c0219e5418d244c62d3289e13d14afeee1
Branch pushed to git repo; I updated commit sha1. New commits:
236ea3c | tox.ini (macos-nohomebrew): Pass compiler configuration to configure using variables other than CC, CXX.
|
comment:15 Changed 17 months ago by
- Commit changed from 236ea3c0219e5418d244c62d3289e13d14afeee1 to 92db4e40bf40a73be4570e159c4e5b36c1955451
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
92db4e4 | tox.ini (macos-nohomebrew): Pass compiler configuration to configure using variables other than CC, CXX.
|
comment:16 Changed 17 months ago by
- Dependencies changed from #31552, #31562 to #31552, #31562, #31409
- Description modified (diff)
comment:17 Changed 17 months ago by
- Commit changed from 92db4e40bf40a73be4570e159c4e5b36c1955451 to 7966b662147c5f58807e0f9492b35c95e9322b36
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
737b21c | src/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2
|
4403924 | src/sage/interfaces/r.py: Mark all tests # optional - rpy2
|
405ebb9 | More # optional - rpy2
|
b7c1576 | Merge tag '9.3.beta8' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional
|
ce3b35f | More # optional - r
|
49c10c1 | Revert "build/pkgs/{r,rpy2}: Downgrade to optional"
|
a67300b | Merge #30383
|
f0a5fb5 | configure.ac: Add option --disable-r
|
f04c134 | Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional
|
7966b66 | Merge #31409
|
comment:18 Changed 17 months ago by
- Priority changed from major to critical
comment:19 Changed 17 months ago by
- Commit changed from 7966b662147c5f58807e0f9492b35c95e9322b36 to b5edde2e54cc58bd151880293e6454c3f92150fe
Branch pushed to git repo; I updated commit sha1. New commits:
b5edde2 | tox.ini (macos-nohomebrew): Fix typo in PILLOW_...
|
comment:20 Changed 17 months ago by
- Status changed from needs_review to needs_work
Now pillow does not find zlib
comment:21 Changed 17 months ago by
- Commit changed from b5edde2e54cc58bd151880293e6454c3f92150fe to dffcdbc75e46d08bcb1fd2ac9a5677a7a7312e2f
Branch pushed to git repo; I updated commit sha1. New commits:
dffcdbc | tox.ini (macos-nohomebrew): Fix up pillow build by setting ZLIB_ROOT
|
comment:22 Changed 17 months ago by
pillow still finds various libraries in /usr/local
because of very creative discovery code that finds everything in /usr/local
because we told it that it can have /usr/local/opt/xz/include
- see _add_directory
(https://github.com/python-pillow/Pillow/blob/8.1.x/setup.py#L197)
comment:23 Changed 17 months ago by
- Commit changed from dffcdbc75e46d08bcb1fd2ac9a5677a7a7312e2f to 3779a2525d5bbf0987a1d4ad0e42b73fb5769bc0
Branch pushed to git repo; I updated commit sha1. New commits:
3779a25 | tox.ini (macos-nohomebrew): Disable more libraries to remove dependencies on homebrew
|
comment:24 Changed 17 months ago by
- Status changed from needs_work to needs_review
Fixed pillow. Now I can build a wheel in #31396 that does not have any dependencies on shared libraries in /usr/local
.
comment:25 follow-up: ↓ 27 Changed 17 months ago by
For what it's worth, I see a few suspicious items in the log files. (This is on a system with homebrew and with source .homebrew-build-env
in my shell init file. That may be causing some of these.)
- cvxopt includes flags
-L/usr/local/lib
and-I/usr/local/include
when callinggcc
- pillow says
Appending path /usr/local/Cellar/xz/5.2.5/include
and then thegcc
flag-I/usr/local/Cellar/xz/5.2.5/include
.
comment:26 Changed 17 months ago by
The xz include is on purpose -- see comment added in tox.ini
comment:27 in reply to: ↑ 25 Changed 17 months ago by
Replying to jhpalmieri:
For what it's worth, I see a few suspicious items in the log files. (This is on a system with homebrew and with
source .homebrew-build-env
in my shell init file. That may be causing some of these.)
Yes, this a good setting for testing this ticket. The build using tox is designed to be isolated from the effects of .homebrew-build-env
.
- cvxopt includes flags
-L/usr/local/lib
and-I/usr/local/include
when callinggcc
This is probably https://github.com/cvxopt/cvxopt/blob/master/setup.py#L55 in combination with a bug in our build system - #31584.
comment:28 Changed 17 months ago by
- Dependencies changed from #31552, #31562, #31409 to #31552, #31562, #31409, #31584
comment:29 Changed 17 months ago by
- Commit changed from 3779a2525d5bbf0987a1d4ad0e42b73fb5769bc0 to ed1a089963e3215ddfe1216ad01237396364519b
comment:30 Changed 17 months ago by
- Reviewers set to John Palmieri
- Status changed from needs_review to positive_review
Okay, this looks good to me.
comment:31 Changed 17 months ago by
Thanks!
comment:32 Changed 17 months ago by
By the way, should commands like make distclean
also clean the .tox
directory?
comment:33 Changed 17 months ago by
I think it should not because .tox
is not a "build artifact" - if this reasoning makes sense.
comment:34 follow-up: ↓ 35 Changed 16 months ago by
That makes sense. Maybe I need to learn more about tox. What command should I use if I want to restart EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode
from scratch? Just delete .tox/local-macos-nohomebrew-python3
?
comment:35 in reply to: ↑ 34 Changed 16 months ago by
Replying to jhpalmieri:
That makes sense. Maybe I need to learn more about tox. What command should I use if I want to restart
EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode
from scratch? Just delete.tox/local-macos-nohomebrew-python3
?
That works, but you can also pass the -r
option to tox
, which will start from scratch.
comment:36 Changed 16 months ago by
Great, thanks!
comment:37 Changed 16 months ago by
- Milestone changed from sage-9.3 to sage-9.4
comment:38 Changed 15 months ago by
- Branch changed from u/mkoeppe/tox_ini__add_local_macos_nohomebrew_environments to ed1a089963e3215ddfe1216ad01237396364519b
- Resolution set to fixed
- Status changed from positive_review to closed
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
tox.ini: Add local-macos-nohomebrew and configuration facors macos-{10.14,10.15,11.1
}