Opened 2 years ago
Closed 2 years ago
#29104 closed enhancement (fixed)
Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.1 |
Component: | porting | Keywords: | |
Cc: | dimpase, jhpalmieri, gh-mwageringel, vbraun, tscrim | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | Dima Pasechnik, John Palmieri |
Report Upstream: | N/A | Work issues: | |
Branch: | dfce7d4 (Commits, GitHub, GitLab) | Commit: | dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc |
Dependencies: | #29335, #29361 | Stopgaps: |
Description (last modified by )
#29053 adds tox.ini
to support testing the sage distribution in isolated environments via docker.
This ticket adds toxenvs local-homebrew-macos-{minimal,standard}
for testing with (best-effort) isolated installations of homebrew on macOS (not using docker).
To test:
brew install tox tox -e local-homebrew-macos-standard
See comments in tox.ini for more examples.
An example run of that is here: https://github.com/mkoeppe/sage/actions/runs/32812271
Change History (141)
comment:1 Changed 2 years ago by
- Branch set to u/mkoeppe/tox-homebrew
comment:2 Changed 2 years ago by
- Cc dimpase jhpalmieri gh-mwageringel added
- Commit set to fca66540d85d7989d84c8551793f029533a8232d
comment:3 Changed 2 years ago by
- Description modified (diff)
- Summary changed from Add to tox.ini a homebrew toxenv (with a fresh install not in /usr/local) to Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local)
comment:4 Changed 2 years ago by
- Commit changed from fca66540d85d7989d84c8551793f029533a8232d to e0e8a4c7be40336fa140195a213f9edf3075f6ee
Branch pushed to git repo; I updated commit sha1. New commits:
e0e8a4c | tox.ini [local-homebrew]: Fixup PATH, PKG_CONFIG_PATH, set LDFLAGS, CPPFLAGS; [local-direct]: New
|
comment:5 Changed 2 years ago by
- Commit changed from e0e8a4c7be40336fa140195a213f9edf3075f6ee to 16ca88f3423ea20dcb164cd1e7b5bc0283fcec0b
Branch pushed to git repo; I updated commit sha1. New commits:
16ca88f | tox.ini: Use {envlogdir}; use ln -sf
|
comment:6 Changed 2 years ago by
- Commit changed from 16ca88f3423ea20dcb164cd1e7b5bc0283fcec0b to 6a1803dae669007dcc7005448a1e578bcc950656
Branch pushed to git repo; I updated commit sha1. New commits:
6a1803d | tox.ini [local]: First build the base-toolchain
|
comment:7 Changed 2 years ago by
- Commit changed from 6a1803dae669007dcc7005448a1e578bcc950656 to 4b72987e58ff284f57f04d4fbb5fed8f39beb36d
Branch pushed to git repo; I updated commit sha1. New commits:
4b72987 | tox.ini [local-homebrew]: Set PATH earlier and set ACLOCAL_PATH for bootstrapping
|
comment:8 Changed 2 years ago by
- Description modified (diff)
- Status changed from new to needs_review
comment:9 Changed 2 years ago by
- Cc vbraun added
comment:10 Changed 2 years ago by
Is this ticket MacOS-specific?
comment:11 Changed 2 years ago by
Hmm, I haven’t tried linuxbrew. Might be worth checking
comment:12 Changed 2 years ago by
I'll fire up my vintage OSX 10.13 macbookair...
comment:13 Changed 2 years ago by
- Commit changed from 4b72987e58ff284f57f04d4fbb5fed8f39beb36d to 2141b7536c5b99a26d16568dcc0d734518bebc58
Branch pushed to git repo; I updated commit sha1. New commits:
2141b75 | Merge tag '9.1.beta3' into t/29104/tox-homebrew
|
comment:14 Changed 2 years ago by
I tried this:
tox -p auto -e local-homebrew-macos-standard
and got
... [sagelib-9.1.beta2] user 0m10.244s [sagelib-9.1.beta2] sys 0m3.812s real 0m16.288s user 0m10.956s sys 0m4.047s Sage build/upgrade complete! ✔ OK local-homebrew-macos-standard in 1 minute, 14.362 seconds _________________________________________________________________________________________________ summary __________________________________________________________________________________________________ local-homebrew-macos-standard: commands succeeded congratulations :)
not tests were run, it seems, is this normal?
comment:15 Changed 2 years ago by
The default packages set in tox.ini
are only fflas_ffpack scipy r
, and those (and their dependencies) are checked.
I don't know why it's actually building sagelib, but I hope to fix bizarre build behavior like this in #29113.
comment:16 Changed 2 years ago by
- Dependencies #29053 deleted
comment:17 Changed 2 years ago by
- Commit changed from 2141b7536c5b99a26d16568dcc0d734518bebc58 to 4eb201b75f75fbbe526b84de9b52f2ba0ea3ef0e
Branch pushed to git repo; I updated commit sha1. New commits:
4eb201b | Merge tag '9.1.beta4' into t/29104/tox-homebrew
|
comment:18 Changed 2 years ago by
- Status changed from needs_review to needs_work
- Work issues set to rebase
comment:19 Changed 2 years ago by
- Commit changed from 4eb201b75f75fbbe526b84de9b52f2ba0ea3ef0e to 7155e7993248fcf6db057e46779ab08cd1670833
Branch pushed to git repo; I updated commit sha1. New commits:
7155e79 | Merge tag '9.1.beta7' into t/29104/tox-homebrew
|
comment:20 Changed 2 years ago by
- Status changed from needs_work to needs_review
comment:21 Changed 2 years ago by
- Commit changed from 7155e7993248fcf6db057e46779ab08cd1670833 to d7a2e726c10e58fac83b7c419973e53f92e9444f
Branch pushed to git repo; I updated commit sha1. New commits:
d7a2e72 | Fixup merge
|
comment:22 Changed 2 years ago by
- Work issues rebase deleted
comment:23 Changed 2 years ago by
- Commit changed from d7a2e726c10e58fac83b7c419973e53f92e9444f to 9e9f4c7f852c7b6f18d44e950a84317bd1de9c6b
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
3103055 | m4ri must use png on gentoo
|
1b48aa1 | Merge tag '9.1.beta7' into t/29273/packages/r_packages_in_distros
|
0fc301c | Add build/pkgs/{libgd|m4ri|ninja_build|perl_term_readline_gnu|yasm}/distros/*.txt - from repology
|
f7785ce | build/pkgs/fedora.txt: Add diffutils
|
6d2bbd8 | build/pkgs/arch.txt: Add 'which'
|
0aa90c1 | build/pkgs/debian.txt: Add ca-certificates
|
c49acdc | Merge tag '9.1.beta7' into t/29129/add_more_system_packages_to_build_pkgs_spkg_distros_
|
0f704d0 | Merge branch 't/29129/add_more_system_packages_to_build_pkgs_spkg_distros_' into t/29273/packages/r_packages_in_distros
|
f8097c4 | Merge branch 'u/mkoeppe/packages/r_packages_in_distros' of git://trac.sagemath.org/sage into t/29104/tox-homebrew
|
9e9f4c7 | Add distros/homebrew.txt for many packages, and some more distros/*.txt
|
comment:24 Changed 2 years ago by
- Dependencies set to #29273
comment:25 Changed 2 years ago by
- Summary changed from Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local) to Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages
comment:26 Changed 2 years ago by
- mpc should be
libmpc
tachyon
does not seem to be the package we usepari
cannot currently be used by Sage because the pari packages are not in Homebrewglpk
4.65 should not be used by Sage as long as it is not patched by Homebrew, see #24824
I also have these Homebrew packages installed:
cmake freetype gcc git libpng pcre pkg-config openssl xz zeromq
Some of these might already be installed automatically as dependencies of others.
Additionally there are:
sagemath/science/flint sagemath/science/arb
Though, I am not sure about how flint
in homebrew-core is different from this one.
comment:27 Changed 2 years ago by
- Commit changed from 9e9f4c7f852c7b6f18d44e950a84317bd1de9c6b to e8d1fef3818fd1d78f83df31ae237061ea089f92
Branch pushed to git repo; I updated commit sha1. New commits:
e8d1fef | Add build/pkgs/homebrew*.txt
|
comment:28 Changed 2 years ago by
- Commit changed from e8d1fef3818fd1d78f83df31ae237061ea089f92 to 242f684eac3c9a71e9562c56554a0b2cdf9e297e
Branch pushed to git repo; I updated commit sha1. New commits:
242f684 | Add/fix build/pkgs/*/homebrew.txt - from gh-mwageringel
|
comment:29 follow-up: ↓ 36 Changed 2 years ago by
https://github.com/sagemath/homebrew-science/commits/master
adds NTL support for flint, and also ensbles TLS.
we can cook up more of these formulae, theain obstacle for me there was that they ate in Ruby.
New commits:
242f684 | Add/fix build/pkgs/*/homebrew.txt - from gh-mwageringel
|
comment:30 follow-up: ↓ 32 Changed 2 years ago by
==> Installing sagemath/science/arb ==> Downloading https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz ##O=# # curl: (22) The requested URL returned error: 404 Not Found Error: Failed to download resource "arb" Download failed: https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz Warning: Bottle installation failed: building from source. ==> Downloading https://github.com/fredrik-johansson/arb/archive/2.16.0.tar.gz ==> Downloading from https://codeload.github.com/fredrik-johansson/arb/tar.gz/2.16.0 ######################################################################## 100.0%
comment:31 Changed 2 years ago by
- Commit changed from 242f684eac3c9a71e9562c56554a0b2cdf9e297e to 2449c395aad510ffc430416a06f2182b6dc0ba7c
Branch pushed to git repo; I updated commit sha1. New commits:
2449c39 | tox.ini: Get homebrew packages from *homebrew*.txt
|
comment:32 in reply to: ↑ 30 Changed 2 years ago by
Replying to mkoeppe:
==> Installing sagemath/science/arb ==> Downloading https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz ##O=# # curl: (22) The requested URL returned error: 404 Not Found Error: Failed to download resource "arb" Download failed: https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz Warning: Bottle installation failed: building from source. ==> Downloading https://github.com/fredrik-johansson/arb/archive/2.16.0.tar.gz ==> Downloading from https://codeload.github.com/fredrik-johansson/arb/tar.gz/2.16.0 ######################################################################## 100.0%
yes, this is expected. No binaries.
comment:33 Changed 2 years ago by
- Commit changed from 2449c395aad510ffc430416a06f2182b6dc0ba7c to 188994aa353dd4533f701aa986a24c1ac2cad860
Branch pushed to git repo; I updated commit sha1. New commits:
188994a | tox.ini: Use brew upgrade after brew install
|
comment:34 Changed 2 years ago by
- Commit changed from 188994aa353dd4533f701aa986a24c1ac2cad860 to 6df5104b1ed9ad4e41857cfd522ab31170b95268
Branch pushed to git repo; I updated commit sha1. New commits:
6df5104 | tox.ini [local-homebrew]: Add to PKG_CONFIG_PATH
|
comment:35 Changed 2 years ago by
Checking whether SageMath should install SPKG openblas... checking BLAS library... openblas checking whether any of gfortran is installed or will be installed as SPKG... no checking for openblas >= 0.2.20... yes checking for library containing cblas_dgemm... no checking for cblas... no checking for library containing ... no checking for lapack... no configure: no suitable system package found for SPKG openblas
The tests for cblas_dgemm
etc. do not seem to use OPENBLAS_LIBS
at all!
comment:36 in reply to: ↑ 29 ; follow-up: ↓ 76 Changed 2 years ago by
Replying to dimpase:
https://github.com/sagemath/homebrew-science/commits/master
adds NTL support for flint, and also ensbles TLS.
we can cook up more of these formulae, theain obstacle for me there was that they ate in Ruby.
There seems to be a conflict between flint
and sagemath/science/flint
, as sagemath/science/arb
installs flint
from homebrew-core as a dependency (which has been added to Homebrew very recently). I am using
brew install sagemath/science/flint brew install --ignore-dependencies sagemath/science/arb
as a workaround now, but the arb formula probably needs to be updated.
comment:37 Changed 2 years ago by
- Commit changed from 6df5104b1ed9ad4e41857cfd522ab31170b95268 to c9d5564c2b4cc4b68748c370211956984d2c986e
Branch pushed to git repo; I updated commit sha1. New commits:
c9d5564 | build/pkgs/openblas/spkg-configure.m4: Use OPENBLAS_{LIBS,CFLAGS} while checking for cblas/lapack functions
|
comment:38 Changed 2 years ago by
This line (introduced in #29071):
AC_FC_FREEFORM([AC_FC_FUNC([dgeqrf])])
is not executed at all because AC_FC_FREEFORM
is AC_DEFUN_ONCE
and is called already in gfortran/spkg-configure.m4
As a result:
Checking whether SageMath should install SPKG openblas... checking BLAS library... openblas checking whether any of gfortran is installed or will be installed as SPKG... no checking for openblas >= 0.2.20... yes checking for cblas_dgemm... yes checking for ... no checking for lapack... no configure: no suitable system package found for SPKG openblas
comment:39 follow-up: ↓ 42 Changed 2 years ago by
- Commit changed from c9d5564c2b4cc4b68748c370211956984d2c986e to 52ef59ed13d878e1a0de0bec999a9d23a442ca6f
Branch pushed to git repo; I updated commit sha1. New commits:
52ef59e | build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC.
|
comment:40 Changed 2 years ago by
Not accepted by configure yet: glpk
, r
comment:42 in reply to: ↑ 39 Changed 2 years ago by
Replying to git:
Branch pushed to git repo; I updated commit sha1. New commits:
52ef59e build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC.
why? If AC_FC_FREEFORM
can only be called once, we can move
the call to AC_FC_FUNC
to the very place it's called.
How about
-
build/pkgs/gfortran/spkg-configure.m4
a b SAGE_SPKG_CONFIGURE([gfortran], [ 6 6 # This helps verify the compiler works too, so if some idiot sets FC to 7 7 # /usr/bin/ls, we will at least know it's not a working Fortran 8 8 # compiler. 9 AC_FC_FREEFORM([SAGE_HAVE_FC_FREEFORM=yes], [ 9 AC_FC_FREEFORM([ 10 SAGE_HAVE_FC_FREEFORM=yes 11 AC_FC_FUNC([dgeqrf]) 12 ], [ 10 13 AC_MSG_NOTICE([Your Fortran compiler does not accept free-format source code]) 11 14 AC_MSG_NOTICE([which means the compiler is either seriously broken, or]) 12 15 AC_MSG_NOTICE([is too old to build Sage.]) -
build/pkgs/openblas/spkg-configure.m4
diff --git a/build/pkgs/openblas/spkg-configure.m4 b/build/pkgs/openblas/spkg-configure.m4 index c37db649be..ecc7540ee9 100644
a b SAGE_SPKG_CONFIGURE([openblas], [ 9 9 dnl openblas does not work as cblas; try to use system's cblas as is 10 10 PKG_CHECK_MODULES([CBLAS], [cblas], [], [sage_spkg_install_openblas=yes]) 11 11 ]) 12 AC_FC_FREEFORM([AC_FC_FUNC([dgeqrf])])13 12 AC_SEARCH_LIBS([$dgeqrf], [openblas], [dnl openblas works as lapack 14 13 sage_install_lapack_pc=yes 15 14 ], [
comment:43 follow-up: ↓ 44 Changed 2 years ago by
OK... let me try
comment:44 in reply to: ↑ 43 Changed 2 years ago by
Replying to mkoeppe:
OK... let me try
it appears to work with gfortan etc installed:
----------------------------------------------------------------------------- Checking whether SageMath should install SPKG openblas... checking BLAS library... openblas checking whether any of gfortran is installed or will be installed as SPKG... no checking for OPENBLAS... yes checking for library containing cblas_dgemm... -lopenblas checking for library containing dgeqrf_... none required configure: will use system package and not install SPKG openblas
but perhaps it would be throwing up a surprise if there wasn't any.
comment:45 Changed 2 years ago by
- Commit changed from 52ef59ed13d878e1a0de0bec999a9d23a442ca6f to 9dd37760062337ca4e97263c713d8a46880644b8
comment:46 Changed 2 years ago by
- Commit changed from 9dd37760062337ca4e97263c713d8a46880644b8 to 886ca2f0392764055804e44597dc38125bd93e9c
Branch pushed to git repo; I updated commit sha1. New commits:
886ca2f | tox.ini [homebrew]: make build
|
comment:47 Changed 2 years ago by
matplotlib install fails:
REQUIRED DEPENDENCIES AND EXTENSIONS numpy: yes [not found. pip may install it below.] install_requires: yes [handled by setuptools] libagg: yes [pkg-config information for 'libagg' could not be found. Using local copy.] freetype: yes [version 2.10.1] png: no [The C/C++ header for libpng (png.h) could not be found. You may need to install the development package.] qhull: yes [pkg-config information for 'libqhull' could not be found. Using local copy.] ... * The following required packages can not be built: * png * Try installing png with `brew install libpng` and * pkg-config with `brew install pkg-config` Cleaning up...
comment:48 Changed 2 years ago by
cysignals testsuite fails:
[cysignals-1.10.2] ulimit 2>/dev/null -s 1024; sage-python23 -B rundoctests.py src/cysignals/*.pyx [cysignals-1.10.2] Doctesting 5 files. [cysignals-1.10.2] Traceback (most recent call last): [cysignals-1.10.2] File "rundoctests.py", line 74, in <module> [cysignals-1.10.2] resource.setrlimit(resource.RLIMIT_STACK, (stacksize, stacksize)) [cysignals-1.10.2] ValueError: current limit exceeds maximum limit [cysignals-1.10.2] make[4]: *** [check-doctest] Error 1 [cysignals-1.10.2] cd example && sage-python23 setup.py clean build
comment:49 Changed 2 years ago by
- Commit changed from 886ca2f0392764055804e44597dc38125bd93e9c to 185dbbb346e7bea59d6108e2ca8e3b33cb734d03
comment:50 Changed 2 years ago by
- Commit changed from 185dbbb346e7bea59d6108e2ca8e3b33cb734d03 to 6750bf1d0bbe58b4ec5526af2fb50361e489a42e
Branch pushed to git repo; I updated commit sha1. New commits:
6750bf1 | tox.ini [local]: Disable linbox testsuite
|
comment:51 Changed 2 years ago by
Ready for review.
comment:52 Changed 2 years ago by
- Status changed from needs_review to needs_work
comment:53 Changed 2 years ago by
The AC_FC_FUNC business broke all -minimal
targets
(https://github.com/mkoeppe/sage/actions/runs/54084841)
comment:54 Changed 2 years ago by
- Commit changed from 6750bf1d0bbe58b4ec5526af2fb50361e489a42e to 071e0b33470bc6fc13296052a831d84d85d4a530
comment:55 Changed 2 years ago by
comment:56 Changed 2 years ago by
- Status changed from needs_work to needs_review
comment:57 Changed 2 years ago by
Needs review
comment:58 Changed 2 years ago by
Both homebrew-macos-minimal
(logs: https://github.com/mkoeppe/sage/suites/515691370/artifacts/2830410) and homebrew-macos-standard
(logs: https://github.com/mkoeppe/sage/suites/515691370/artifacts/2830411) now build correctly all the way to sagelib
.
comment:59 follow-up: ↓ 61 Changed 2 years ago by
Some doctest failures similar to this:
File "src/sage/structure/element.pyx", line 3945, in sage.structure.element.EuclideanDomainElement._mod_ Failed example: cython(''' from sage.structure.element cimport EuclideanDomainElement cdef class MyElt(EuclideanDomainElement): def quo_rem(self, other): return self._parent.var('quo,rem') ''') Expected nothing Got: ld: warning: directory not found for option '-L/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-minimal/homebrew/opt/readline/lib'
comment:60 Changed 2 years ago by
- Commit changed from 071e0b33470bc6fc13296052a831d84d85d4a530 to d1eb116499937578c84f3b27097bf4d3f4194b2f
Branch pushed to git repo; I updated commit sha1. New commits:
d1eb116 | tox.ini: Move homebrew environment settings to script .homebrew-build-env
|
comment:61 in reply to: ↑ 59 Changed 2 years ago by
Replying to mkoeppe:
Some doctest failures similar to this: [...readline...]
These are now fixed. Very few remaining doctest failures and timeouts for homebrew-macos-minimal
(https://github.com/mkoeppe/sage/runs/508217498):
sage -t src/sage/interfaces/gap.py # 1 doctest failed sage -t src/sage/manifolds/differentiable/degenerate_submanifold.py # Timed out sage -t src/sage/manifolds/differentiable/tensorfield.py # Timed out sage -t src/sage/plot/plot3d/base.pyx # Timed out sage -t src/sage/plot/plot3d/implicit_plot3d.py # Timed out sage -t src/sage/plot/plot3d/parametric_plot3d.py # Timed out sage -t src/sage/plot/plot3d/plot3d.py # Timed out sage -t src/sage/plot/plot3d/shapes.pyx # Timed out sage -t src/sage/rings/padics/padic_lattice_element.py # 3 doctests failed sage -t src/sage/plot/plot3d/shapes2.py # Timed out sage -t src/sage/rings/function_field/function_field.py # Timed out sage -t src/sage/schemes/elliptic_curves/ell_number_field.py # Timed out
homebrew-macos-standard
(https://github.com/mkoeppe/sage/runs/508217500) has many more doctest failures.
Looks like there is some problem with singular.
Some messages from singular: univariate factorization depends on NTL(missing)
In that build, Sage is using the homebrew NTL:
Checking whether SageMath should install SPKG ntl... checking installing gmp/mpir? ... no checking NTL/ZZ.h usability... yes checking NTL/ZZ.h presence... yes checking for NTL/ZZ.h... yes checking whether we can link a program using NTL... yes checking NTL version >= 10.3... 11.4.3 configure: will use system package and not install SPKG ntl using ntl library from the system
... but Singular does not find it ....
checking for NTL >= 5.0... not found configure: WARNING: Unable to find NTL (which is strongly recommended) on your machine: please use --with-ntl=PATH_TO_DIR_CONTAINING_LIB_AND_INCLUDE (see also ./configure --help if you do not understand what we are talking about)
comment:62 Changed 2 years ago by
- Dependencies changed from #29273 to #29273, #29335
comment:63 Changed 2 years ago by
- Commit changed from d1eb116499937578c84f3b27097bf4d3f4194b2f to 6410c1e02cf69213efe2e888bd63c1d1856cc135
comment:64 Changed 2 years ago by
comment:65 Changed 2 years ago by
With #29335 merged, Singular finds homebrew's NTL but then I get the following linker error (https://github.com/mkoeppe/sage/runs/508693083):
[singular-4.1.1p2.p0] ld: illegal thread local variable reference to regular symbol __ZN3NTL13ErrorCallbackE for architecture x86_64 [singular-4.1.1p2.p0] clang: error: linker command failed with exit code 1 (use -v to see invocation) [singular-4.1.1p2.p0] make[8]: *** [libSingular.la] Error 1
comment:66 follow-up: ↓ 67 Changed 2 years ago by
this is about absence of coherence on TLS between Singular and Flint.
comment:67 in reply to: ↑ 66 Changed 2 years ago by
Replying to dimpase:
this is about absence of coherence on TLS between Singular and Flint.
OK, is there a way to enable use of TLS with Singular?
comment:68 Changed 2 years ago by
- Commit changed from 6410c1e02cf69213efe2e888bd63c1d1856cc135 to ec5cd78603542b57f58f4ae0d43ba41151a55ab6
Branch pushed to git repo; I updated commit sha1. New commits:
96553db | build/pkgs/ntl/spkg-configure.m4: Check for NTL configuration
|
ec5cd78 | Merge branch 't/29339/fix_ntl_spkg_configure_m4_so_it_rejects_ntls_built_with_ntl_threads__without_ntl_gmp_lip__without_ntl_gf2x_lib' into t/29104/tox-homebrew
|
comment:69 Changed 2 years ago by
- Dependencies changed from #29273, #29335 to #29273, #29335, #29339
comment:70 follow-up: ↓ 72 Changed 2 years ago by
Build logs for homebrew-macos-standard
at https://github.com/mkoeppe/sage/runs/511533264
comment:71 Changed 2 years ago by
- Commit changed from ec5cd78603542b57f58f4ae0d43ba41151a55ab6 to 6d2f6e1fc67097613f08f326a43a6749073e7bd0
comment:72 in reply to: ↑ 70 Changed 2 years ago by
Replying to mkoeppe:
Build logs for
homebrew-macos-standard
at https://github.com/mkoeppe/sage/runs/511533264
[sagelib-9.1.beta7] ld: illegal thread local variable reference to regular symbol __ZN3NTL20ZZXFac_InitNumPrimesE for architecture x86_64 [sagelib-9.1.beta7] clang: error: linker command failed with exit code 1 (use -v to see invocation)
comment:73 Changed 2 years ago by
I guess this means that another library is picking up the system installation of NTL...
comment:74 Changed 2 years ago by
- Commit changed from 6d2f6e1fc67097613f08f326a43a6749073e7bd0 to 14e9f1270e7706e9d771ff33efe259e135fa9ec4
Branch pushed to git repo; I updated commit sha1. New commits:
14e9f12 | build/pkgs/ntl/distros/homebrew.txt: Remove ntl
|
comment:75 follow-up: ↓ 77 Changed 2 years ago by
comment:76 in reply to: ↑ 36 Changed 2 years ago by
Replying to gh-mwageringel:
There seems to be a conflict between
flint
andsagemath/science/flint
, assagemath/science/arb
installsflint
from homebrew-core as a dependency (which has been added to Homebrew very recently). I am usingbrew install sagemath/science/flint brew install --ignore-dependencies sagemath/science/arbas a workaround now, but the arb formula probably needs to be updated.
As can be seen in the build logs above, sagemath/science/arb
installs flint
from homebrew-core as a dependency. Later, sagemath/science/flint
is installed only because of brew upgrade
. This only seems to work because of the coincidence that sagemath/science/flint
has a higher version number than flint
, currently. However, sagemath/science/arb
is only compiled with flint
(without ntl support) – it is not recompiled after installing sagemath/science/flint
.
I have opened a pull request to resolve this.
comment:77 in reply to: ↑ 75 Changed 2 years ago by
Replying to mkoeppe:
If you remove ntl, you probably also need to remove flint and arb, as they depend on it.
comment:78 Changed 2 years ago by
- Commit changed from 14e9f1270e7706e9d771ff33efe259e135fa9ec4 to 70e161bdd3f29ae1b25dffab08829a20e2c53e5c
Branch pushed to git repo; I updated commit sha1. New commits:
70e161b | Remove use of homebrew packages flint, arb
|
comment:79 Changed 2 years ago by
Thank you!
comment:80 Changed 2 years ago by
New tests at https://github.com/mkoeppe/sage/runs/512243289
comment:81 Changed 2 years ago by
I had no idea that a ticket about homebrew would be touching fedora-related stuff, and in this I essentially created a duplicate of that work on #29352
It's much better to create separate tickets for unrelated problems - not the least as it gives them a chance of being discovered.
comment:82 Changed 2 years ago by
OK. What can I do to facilitate review?
comment:83 Changed 2 years ago by
I cannot review MacOS-related things - I don't have access to the needed OS. But I would be happy to look at the other things that got silently bundled here.
comment:84 follow-up: ↓ 85 Changed 2 years ago by
OK, then let me move the fedora.txt files to #29352 and the other linux .txt files to a new ticket.
comment:85 in reply to: ↑ 84 Changed 2 years ago by
comment:86 follow-up: ↓ 88 Changed 2 years ago by
@gh-mwageringel Would you be able to review this ticket?
comment:87 Changed 2 years ago by
- Cc tscrim added
- Description modified (diff)
comment:88 in reply to: ↑ 86 ; follow-up: ↓ 90 Changed 2 years ago by
Replying to mkoeppe:
@gh-mwageringel Would you be able to review this ticket?
Yes, I will look into that. Though, I only have access to macOS 10.13.6.
comment:89 Changed 2 years ago by
Tests ran at https://github.com/mkoeppe/sage/actions/runs/57924835
local-homebrew-macos-minimal
(https://github.com/mkoeppe/sage/runs/515508463) and local-homebrew-macos-standard
(https://github.com/mkoeppe/sage/runs/515508474):
All packages (except for experimental package polymake
) build, doctests:
---------------------------------------------------------------------- sage -t src/sage/game_theory/normal_form_game.py # 5 doctests failed sage -t src/sage/dynamics/arithmetic_dynamics/projective_ds.py # Timed out sage -t src/sage/interfaces/gap.py # 1 doctest failed sage -t src/sage/knots/link.py # 1 doctest failed sage -t src/sage/manifolds/differentiable/degenerate_submanifold.py # Timed out sage -t src/sage/manifolds/differentiable/tensorfield.py # Timed out sage -t src/sage/plot/plot3d/base.pyx # Timed out sage -t src/sage/plot/plot3d/implicit_plot3d.py # Timed out sage -t src/sage/plot/plot3d/parametric_plot3d.py # Timed out sage -t src/sage/plot/plot3d/plot3d.py # Timed out sage -t src/sage/plot/plot3d/shapes.pyx # Timed out sage -t src/sage/rings/padics/padic_lattice_element.py # 3 doctests failed sage -t src/sage/plot/plot3d/shapes2.py # Timed out sage -t src/sage/rings/function_field/function_field.py # Timed out sage -t src/sage/schemes/elliptic_curves/ell_number_field.py # Timed out ----------------------------------------------------------------------
comment:90 in reply to: ↑ 88 Changed 2 years ago by
Replying to gh-mwageringel:
Replying to mkoeppe:
@gh-mwageringel Would you be able to review this ticket?
Yes, I will look into that.
Thanks!
Though, I only have access to macOS 10.13.6.
That's actually great, for some broader coverage. I'm testing on 10.15.3, and also the GitHub runners are on Catalina.
comment:91 Changed 2 years ago by
lots of timeouts in tests, probably an overloaded host, or too aggressive parallelisation.
comment:92 Changed 2 years ago by
Yes, this is using 12 threads
comment:93 follow-up: ↓ 102 Changed 2 years ago by
I'm in the middle of trying this. A few comments:
- patch: this should use homebrew's patch
- pcre: same
- boost: is there a way to convince Sage to use homebrew's boost as a replacement for boost-cropped? (See #29100?)
- ntl: being discussed on another ticket, but my Sage builds on OS X regularly use homebrew's ntl without problems.
comment:94 follow-up: ↓ 98 Changed 2 years ago by
Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?
comment:95 follow-up: ↓ 99 Changed 2 years ago by
I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.
comment:96 follow-up: ↓ 97 Changed 2 years ago by
ntl failed to build. This is on OS X 10.15.3.
make setup3 g++ -std=gnu++11 -I../include -I. -O2 -g -fno-common -march=native -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/homebrew/opt/readline/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/homebrew/lib -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/include -o gen_gmp_aux gen_gmp_aux.cpp -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib -lgmp -lm ./gen_gmp_aux > ../include/NTL/gmp_aux.h NTL_GMP_LIP flag set GMP version check (6.1.2/6.2.0) *** version number mismatch: inconsistency between gmp.h and libgmp /bin/sh: line 1: 18441 Abort trap: 6 ./gen_gmp_aux > ../include/NTL/gmp_aux.h make[5]: *** [setup3] Error 134 make[4]: *** [setup-phase] Error 2 make[4]: Target `all' not remade because of errors.
comment:97 in reply to: ↑ 96 Changed 2 years ago by
Replying to jhpalmieri:
ntl failed to build. This is on OS X 10.15.3.
This happens when you have a different (older) version of gmp in your /usr/local homebrew. The isolation is unfortunately not perfect. Badly written configure scripts of packages pick up stuff from /usr/local
comment:98 in reply to: ↑ 94 ; follow-up: ↓ 118 Changed 2 years ago by
Replying to jhpalmieri:
Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?
Good idea. WIll do
comment:99 in reply to: ↑ 95 ; follow-up: ↓ 121 Changed 2 years ago by
Replying to dimpase:
I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.
Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.
comment:100 Changed 2 years ago by
- Dependencies changed from #29273, #29335, #29339 to #29273, #29335, #29339, #29361
comment:101 Changed 2 years ago by
- Commit changed from 70e161bdd3f29ae1b25dffab08829a20e2c53e5c to 11f77293e372d29e01b6525e1fb1948c4d855f04
Branch pushed to git repo; I updated commit sha1. New commits:
11f7729 | build/pkgs/patch/distros/homebrew.txt: New
|
comment:102 in reply to: ↑ 93 ; follow-up: ↓ 104 Changed 2 years ago by
Replying to jhpalmieri:
I'm in the middle of trying this. A few comments:
- pcre: same
This one actually works for me (both locally and on the GitHub runner). Please post the logs if it doesn't work for you
comment:103 Changed 2 years ago by
- Commit changed from 11f77293e372d29e01b6525e1fb1948c4d855f04 to 93a0421c7789a1ac278386f9643d87b9768a5e69
Branch pushed to git repo; I updated commit sha1. New commits:
93a0421 | build/pkgs/boost*/distros/homebrew.txt: New
|
comment:104 in reply to: ↑ 102 Changed 2 years ago by
Replying to mkoeppe:
Replying to jhpalmieri:
I'm in the middle of trying this. A few comments:
- pcre: same
This one actually works for me (both locally and on the GitHub runner). Please post the logs if it doesn't work for you
Sorry, it looks okay to me. I must have misread config.log.
comment:105 Changed 2 years ago by
Flint fails to build for me. I ran brew upgrade
, cleaned everything out, and tried again, and it still fails.
g++ -std=gnu++11 -fPIC -fno-common -pedantic -Wall -O2 -funroll-loops -g -mpopcnt -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/var/tmp/sage/build/flint-2.5.2.p5/src -I/usr/local/include -I/usr/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/include -c interfaces/NTL-interface.cpp -o build/interfaces/NTL-interface.lo ld: illegal thread local variable reference to regular symbol __ZN3NTL8ZZ_pInfoE for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make[6]: *** [libflint-13.5.2.dylib] Error 1 make[6]: Target `shared' not remade because of errors. make[5]: *** [library] Error 2 make[5]: Target `all' not remade because of errors. make[4]: *** [verbose] Error 2
comment:106 follow-up: ↓ 108 Changed 2 years ago by
Is homebrew ntl installed on this system?
comment:107 Changed 2 years ago by
(in /usr/local)
comment:108 in reply to: ↑ 106 ; follow-up: ↓ 109 Changed 2 years ago by
Replying to mkoeppe:
Is homebrew ntl installed on this system?
Yes. Do I need to delete it?
By the way, should prefix
be added to .gitignore
?
comment:109 in reply to: ↑ 108 ; follow-up: ↓ 113 Changed 2 years ago by
Replying to jhpalmieri:
Replying to mkoeppe:
Is homebrew ntl installed on this system?
Yes. Do I need to delete it?
Please try if this helps. If it does, we should fix something in the FLINT build scripts
By the way, should
prefix
be added to.gitignore
?
Yes, will do.
comment:110 Changed 2 years ago by
- Commit changed from 93a0421c7789a1ac278386f9643d87b9768a5e69 to 4c653c2759eb404f9a51baf52c260468276ea720
Branch pushed to git repo; I updated commit sha1. New commits:
4c653c2 | .gitignore: Add prefix
|
comment:111 Changed 2 years ago by
Is there a command like make distclean
that will get rid of the stuff that Sage has built but keep the homebrew stuff? I don't really want to wait for gcc to build again. (make distclean
doesn't touch prefix
.)
comment:112 Changed 2 years ago by
At the moment, nothing better than rm -rf prefix/*
comment:113 in reply to: ↑ 109 Changed 2 years ago by
Replying to mkoeppe:
Replying to jhpalmieri:
Replying to mkoeppe:
Is homebrew ntl installed on this system?
Yes. Do I need to delete it?
Please try if this helps. If it does, we should fix something in the FLINT build scripts
After doing brew uninstall ntl
, flint now builds.
comment:114 Changed 2 years ago by
- Commit changed from 4c653c2759eb404f9a51baf52c260468276ea720 to 467160cdc8402124b21c5dee60386dee03f18b67
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
2625a64 | tox.ini [homebrew]: make build
|
cf83af6 | Add zlib
|
14a3dab | tox.ini [local]: Disable testsuites for some packages
|
2553b5a | tox.ini [local]: Disable linbox testsuite
|
2f79c76 | tox.ini: Move homebrew environment settings to script .homebrew-build-env
|
77d5c11 | build/pkgs/ntl/distros/homebrew.txt: Remove ntl
|
4ec80e2 | Remove use of homebrew packages flint, arb
|
54c803b | build/pkgs/patch/distros/homebrew.txt: New
|
2ce6e6c | build/pkgs/boost*/distros/homebrew.txt: New
|
467160c | .gitignore: Add prefix
|
comment:115 Changed 2 years ago by
- Dependencies changed from #29273, #29335, #29339, #29361 to #29335, #29361
Rebased - no longer on top of #29339.
comment:116 Changed 2 years ago by
Except for the issue with FLINT not building with a pre-existing NTL, it works for me.
comment:117 follow-up: ↓ 119 Changed 2 years ago by
Let's fix the FLINT issue on a follow-up ticket.
comment:118 in reply to: ↑ 98 Changed 2 years ago by
Replying to mkoeppe:
Replying to jhpalmieri:
Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?
Good idea. WIll do
I'll do this in #29146
comment:119 in reply to: ↑ 117 Changed 2 years ago by
comment:120 Changed 2 years ago by
Ready for review
comment:121 in reply to: ↑ 99 Changed 2 years ago by
Replying to mkoeppe:
Replying to dimpase:
I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.
Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.
One can split building into make built
and make ptest
, with the latter, much more memory hungry, run with lower value of -j
.
comment:122 Changed 2 years ago by
- Commit changed from 467160cdc8402124b21c5dee60386dee03f18b67 to dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc
Branch pushed to git repo; I updated commit sha1. New commits:
dfce7d4 | tox.ini [local]: Pass MAKE environment variable, do not default to 'make -j12', change default target to 'build', add doc
|
comment:123 Changed 2 years ago by
OK I've made it more configurable already on this ticket. And I have added documentation
comment:124 Changed 2 years ago by
- Description modified (diff)
comment:125 follow-up: ↓ 135 Changed 2 years ago by
@gh-mwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.
comment:126 Changed 2 years ago by
I'm 8 timezones away from my MacOS machine.
I'd like more details/pointers on how the Github Actions testing is done - for this looks to me like "all sufficiently advanced technology is indistinguishable from magic" kind of thing :-)
comment:127 Changed 2 years ago by
comment:128 Changed 2 years ago by
how does one get the data I linked in comment:126 then?
comment:129 follow-up: ↓ 131 Changed 2 years ago by
comment:130 Changed 2 years ago by
If the GitHub Actions interface is overwhelming, just download the "Artifacts". They are tar.gz files of build logs.
comment:131 in reply to: ↑ 129 Changed 2 years ago by
comment:132 Changed 2 years ago by
I don't push logs to GitHub.
#29087 adds .github/workflows/tox.yml
(https://git.sagemath.org/sage.git/diff/.github/workflows/tox.yml?id=576a18834a16ff05cd6894172a55e86bcc95d407&id2=be1e22c441d9332fd0bf4a3066e1964deddaf8c7) which gets run on every commit pushed to GitHub and on every pull request.
The job tox-local-homebrew-macos
defined in that file provisions a macOS host, checks out the branch and runs tox -e local-homebrew-macos-standard
there. Then it packages up logs and makes them available as an artifact.
comment:133 Changed 2 years ago by
and where is that MacOS host getting provisioned?
comment:134 Changed 2 years ago by
GitHub takes care of that, I think it's using Azure.
comment:135 in reply to: ↑ 125 Changed 2 years ago by
Replying to jhpalmieri:
@gh-mwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.
I have not tested this locally, but as you have done that, I am happy with this too.
One last question though: As far as I can see, readline is not installed into /usr/local. Why do you not run into conflicts with macOS's system "readline"/libedit, as in #29000?
comment:136 Changed 2 years ago by
These tests use their own homebrew installation in a prefix .tox/local-homebrew-macos-standard/homebrew/
, and sets up environment variables using the new script .homebrew-build-env
. This sets LDFLAGS and CPPFLAGS for the readline installed into opt
under the homebrew prefix.
comment:137 Changed 2 years ago by
The problem in #29000 occured despite passing LDFLAGS and CPPFLAGS to configure
, so I guess setting these also during make
would avoid that problem, as a workaround.
comment:138 Changed 2 years ago by
I think the problem description of #29000 needs some more precision.
comment:139 Changed 2 years ago by
- Reviewers set to Dima Pasechnik, John Palmieri
- Status changed from needs_review to positive_review
OK, looks good to me.
comment:140 Changed 2 years ago by
Thanks everyone!
comment:141 Changed 2 years ago by
- Branch changed from u/mkoeppe/tox-homebrew to dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc
- Resolution set to fixed
- Status changed from positive_review to closed
Last 10 new commits:
build/bin/write-dockerfile.sh: Add commands to activate conda env
Merge tag '9.1.beta2' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time
add doc
add centos-{7,8
}on centos, IGNORE_MISSING_SYSTEM_PACKAGES=yes
Add build/pkgs/gfortran/distros/fedora.txt
build/pkgs/fedora.txt: Add python3
tox.ini: Add debian stretch, put sid last
build/pkgs/openblas/distros/arch.txt: Add lapack, cblas
tox.ini: Add toxenvs local-homebrew-macos-minimal, local-homebrew-macos-standard