Opened 5 months ago
Closed 4 months ago
#29104 closed enhancement (fixed)
Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage9.1 
Component:  porting  Keywords:  
Cc:  dimpase, jhpalmieri, ghmwageringel, vbraun, tscrim  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  Dima Pasechnik, John Palmieri 
Report Upstream:  N/A  Work issues:  
Branch:  dfce7d4 (Commits)  Commit:  dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc 
Dependencies:  #29335, #29361  Stopgaps: 
Description (last modified by )
#29053 adds tox.ini
to support testing the sage distribution in isolated environments via docker.
This ticket adds toxenvs localhomebrewmacos{minimal,standard}
for testing with (besteffort) isolated installations of homebrew on macOS (not using docker).
To test:
brew install tox tox e localhomebrewmacosstandard
See comments in tox.ini for more examples.
An example run of that is here: https://github.com/mkoeppe/sage/actions/runs/32812271
Change History (141)
comment:1 Changed 5 months ago by
 Branch set to u/mkoeppe/toxhomebrew
comment:2 Changed 5 months ago by
 Cc dimpase jhpalmieri ghmwageringel added
 Commit set to fca66540d85d7989d84c8551793f029533a8232d
comment:3 Changed 5 months ago by
 Description modified (diff)
 Summary changed from Add to tox.ini a homebrew toxenv (with a fresh install not in /usr/local) to Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local)
comment:4 Changed 5 months ago by
 Commit changed from fca66540d85d7989d84c8551793f029533a8232d to e0e8a4c7be40336fa140195a213f9edf3075f6ee
Branch pushed to git repo; I updated commit sha1. New commits:
e0e8a4c  tox.ini [localhomebrew]: Fixup PATH, PKG_CONFIG_PATH, set LDFLAGS, CPPFLAGS; [localdirect]: New

comment:5 Changed 5 months ago by
 Commit changed from e0e8a4c7be40336fa140195a213f9edf3075f6ee to 16ca88f3423ea20dcb164cd1e7b5bc0283fcec0b
Branch pushed to git repo; I updated commit sha1. New commits:
16ca88f  tox.ini: Use {envlogdir}; use ln sf

comment:6 Changed 5 months ago by
 Commit changed from 16ca88f3423ea20dcb164cd1e7b5bc0283fcec0b to 6a1803dae669007dcc7005448a1e578bcc950656
Branch pushed to git repo; I updated commit sha1. New commits:
6a1803d  tox.ini [local]: First build the basetoolchain

comment:7 Changed 5 months ago by
 Commit changed from 6a1803dae669007dcc7005448a1e578bcc950656 to 4b72987e58ff284f57f04d4fbb5fed8f39beb36d
Branch pushed to git repo; I updated commit sha1. New commits:
4b72987  tox.ini [localhomebrew]: Set PATH earlier and set ACLOCAL_PATH for bootstrapping

comment:8 Changed 5 months ago by
 Description modified (diff)
 Status changed from new to needs_review
comment:9 Changed 5 months ago by
 Cc vbraun added
comment:10 Changed 5 months ago by
Is this ticket MacOSspecific?
comment:11 Changed 5 months ago by
Hmm, I haven’t tried linuxbrew. Might be worth checking
comment:12 Changed 5 months ago by
I'll fire up my vintage OSX 10.13 macbookair...
comment:13 Changed 5 months ago by
 Commit changed from 4b72987e58ff284f57f04d4fbb5fed8f39beb36d to 2141b7536c5b99a26d16568dcc0d734518bebc58
Branch pushed to git repo; I updated commit sha1. New commits:
2141b75  Merge tag '9.1.beta3' into t/29104/toxhomebrew

comment:14 Changed 5 months ago by
I tried this:
tox p auto e localhomebrewmacosstandard
and got
... [sagelib9.1.beta2] user 0m10.244s [sagelib9.1.beta2] sys 0m3.812s real 0m16.288s user 0m10.956s sys 0m4.047s Sage build/upgrade complete! ✔ OK localhomebrewmacosstandard in 1 minute, 14.362 seconds _________________________________________________________________________________________________ summary __________________________________________________________________________________________________ localhomebrewmacosstandard: commands succeeded congratulations :)
not tests were run, it seems, is this normal?
comment:15 Changed 5 months ago by
The default packages set in tox.ini
are only fflas_ffpack scipy r
, and those (and their dependencies) are checked.
I don't know why it's actually building sagelib, but I hope to fix bizarre build behavior like this in #29113.
comment:16 Changed 5 months ago by
 Dependencies #29053 deleted
comment:17 Changed 5 months ago by
 Commit changed from 2141b7536c5b99a26d16568dcc0d734518bebc58 to 4eb201b75f75fbbe526b84de9b52f2ba0ea3ef0e
Branch pushed to git repo; I updated commit sha1. New commits:
4eb201b  Merge tag '9.1.beta4' into t/29104/toxhomebrew

comment:18 Changed 4 months ago by
 Status changed from needs_review to needs_work
 Work issues set to rebase
comment:19 Changed 4 months ago by
 Commit changed from 4eb201b75f75fbbe526b84de9b52f2ba0ea3ef0e to 7155e7993248fcf6db057e46779ab08cd1670833
Branch pushed to git repo; I updated commit sha1. New commits:
7155e79  Merge tag '9.1.beta7' into t/29104/toxhomebrew

comment:20 Changed 4 months ago by
 Status changed from needs_work to needs_review
comment:21 Changed 4 months ago by
 Commit changed from 7155e7993248fcf6db057e46779ab08cd1670833 to d7a2e726c10e58fac83b7c419973e53f92e9444f
Branch pushed to git repo; I updated commit sha1. New commits:
d7a2e72  Fixup merge

comment:22 Changed 4 months ago by
 Work issues rebase deleted
comment:23 Changed 4 months ago by
 Commit changed from d7a2e726c10e58fac83b7c419973e53f92e9444f to 9e9f4c7f852c7b6f18d44e950a84317bd1de9c6b
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
3103055  m4ri must use png on gentoo

1b48aa1  Merge tag '9.1.beta7' into t/29273/packages/r_packages_in_distros

0fc301c  Add build/pkgs/{libgdm4rininja_buildperl_term_readline_gnuyasm}/distros/*.txt  from repology

f7785ce  build/pkgs/fedora.txt: Add diffutils

6d2bbd8  build/pkgs/arch.txt: Add 'which'

0aa90c1  build/pkgs/debian.txt: Add cacertificates

c49acdc  Merge tag '9.1.beta7' into t/29129/add_more_system_packages_to_build_pkgs_spkg_distros_

0f704d0  Merge branch 't/29129/add_more_system_packages_to_build_pkgs_spkg_distros_' into t/29273/packages/r_packages_in_distros

f8097c4  Merge branch 'u/mkoeppe/packages/r_packages_in_distros' of git://trac.sagemath.org/sage into t/29104/toxhomebrew

9e9f4c7  Add distros/homebrew.txt for many packages, and some more distros/*.txt

comment:24 Changed 4 months ago by
 Dependencies set to #29273
comment:25 Changed 4 months ago by
 Summary changed from Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local) to Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages
comment:26 Changed 4 months ago by
 mpc should be
libmpc
tachyon
does not seem to be the package we usepari
cannot currently be used by Sage because the pari packages are not in Homebrewglpk
4.65 should not be used by Sage as long as it is not patched by Homebrew, see #24824
I also have these Homebrew packages installed:
cmake freetype gcc git libpng pcre pkgconfig openssl xz zeromq
Some of these might already be installed automatically as dependencies of others.
Additionally there are:
sagemath/science/flint sagemath/science/arb
Though, I am not sure about how flint
in homebrewcore is different from this one.
comment:27 Changed 4 months ago by
 Commit changed from 9e9f4c7f852c7b6f18d44e950a84317bd1de9c6b to e8d1fef3818fd1d78f83df31ae237061ea089f92
Branch pushed to git repo; I updated commit sha1. New commits:
e8d1fef  Add build/pkgs/homebrew*.txt

comment:28 Changed 4 months ago by
 Commit changed from e8d1fef3818fd1d78f83df31ae237061ea089f92 to 242f684eac3c9a71e9562c56554a0b2cdf9e297e
Branch pushed to git repo; I updated commit sha1. New commits:
242f684  Add/fix build/pkgs/*/homebrew.txt  from ghmwageringel

comment:29 followup: ↓ 36 Changed 4 months ago by
https://github.com/sagemath/homebrewscience/commits/master
adds NTL support for flint, and also ensbles TLS.
we can cook up more of these formulae, theain obstacle for me there was that they ate in Ruby.
New commits:
242f684  Add/fix build/pkgs/*/homebrew.txt  from ghmwageringel

comment:30 followup: ↓ 32 Changed 4 months ago by
==> Installing sagemath/science/arb ==> Downloading https://homebrew.bintray.com/bottlesscience/arb2.16.0.sierra.bottle.tar.gz ##O=# # curl: (22) The requested URL returned error: 404 Not Found Error: Failed to download resource "arb" Download failed: https://homebrew.bintray.com/bottlesscience/arb2.16.0.sierra.bottle.tar.gz Warning: Bottle installation failed: building from source. ==> Downloading https://github.com/fredrikjohansson/arb/archive/2.16.0.tar.gz ==> Downloading from https://codeload.github.com/fredrikjohansson/arb/tar.gz/2.16.0 ######################################################################## 100.0%
comment:31 Changed 4 months ago by
 Commit changed from 242f684eac3c9a71e9562c56554a0b2cdf9e297e to 2449c395aad510ffc430416a06f2182b6dc0ba7c
Branch pushed to git repo; I updated commit sha1. New commits:
2449c39  tox.ini: Get homebrew packages from *homebrew*.txt

comment:32 in reply to: ↑ 30 Changed 4 months ago by
Replying to mkoeppe:
==> Installing sagemath/science/arb ==> Downloading https://homebrew.bintray.com/bottlesscience/arb2.16.0.sierra.bottle.tar.gz ##O=# # curl: (22) The requested URL returned error: 404 Not Found Error: Failed to download resource "arb" Download failed: https://homebrew.bintray.com/bottlesscience/arb2.16.0.sierra.bottle.tar.gz Warning: Bottle installation failed: building from source. ==> Downloading https://github.com/fredrikjohansson/arb/archive/2.16.0.tar.gz ==> Downloading from https://codeload.github.com/fredrikjohansson/arb/tar.gz/2.16.0 ######################################################################## 100.0%
yes, this is expected. No binaries.
comment:33 Changed 4 months ago by
 Commit changed from 2449c395aad510ffc430416a06f2182b6dc0ba7c to 188994aa353dd4533f701aa986a24c1ac2cad860
Branch pushed to git repo; I updated commit sha1. New commits:
188994a  tox.ini: Use brew upgrade after brew install

comment:34 Changed 4 months ago by
 Commit changed from 188994aa353dd4533f701aa986a24c1ac2cad860 to 6df5104b1ed9ad4e41857cfd522ab31170b95268
Branch pushed to git repo; I updated commit sha1. New commits:
6df5104  tox.ini [localhomebrew]: Add to PKG_CONFIG_PATH

comment:35 Changed 4 months ago by
Checking whether SageMath should install SPKG openblas... checking BLAS library... openblas checking whether any of gfortran is installed or will be installed as SPKG... no checking for openblas >= 0.2.20... yes checking for library containing cblas_dgemm... no checking for cblas... no checking for library containing ... no checking for lapack... no configure: no suitable system package found for SPKG openblas
The tests for cblas_dgemm
etc. do not seem to use OPENBLAS_LIBS
at all!
comment:36 in reply to: ↑ 29 ; followup: ↓ 76 Changed 4 months ago by
Replying to dimpase:
https://github.com/sagemath/homebrewscience/commits/master
adds NTL support for flint, and also ensbles TLS.
we can cook up more of these formulae, theain obstacle for me there was that they ate in Ruby.
There seems to be a conflict between flint
and sagemath/science/flint
, as sagemath/science/arb
installs flint
from homebrewcore as a dependency (which has been added to Homebrew very recently). I am using
brew install sagemath/science/flint brew install ignoredependencies sagemath/science/arb
as a workaround now, but the arb formula probably needs to be updated.
comment:37 Changed 4 months ago by
 Commit changed from 6df5104b1ed9ad4e41857cfd522ab31170b95268 to c9d5564c2b4cc4b68748c370211956984d2c986e
Branch pushed to git repo; I updated commit sha1. New commits:
c9d5564  build/pkgs/openblas/spkgconfigure.m4: Use OPENBLAS_{LIBS,CFLAGS} while checking for cblas/lapack functions

comment:38 Changed 4 months ago by
This line (introduced in #29071):
AC_FC_FREEFORM([AC_FC_FUNC([dgeqrf])])
is not executed at all because AC_FC_FREEFORM
is AC_DEFUN_ONCE
and is called already in gfortran/spkgconfigure.m4
As a result:
Checking whether SageMath should install SPKG openblas... checking BLAS library... openblas checking whether any of gfortran is installed or will be installed as SPKG... no checking for openblas >= 0.2.20... yes checking for cblas_dgemm... yes checking for ... no checking for lapack... no configure: no suitable system package found for SPKG openblas
comment:39 followup: ↓ 42 Changed 4 months ago by
 Commit changed from c9d5564c2b4cc4b68748c370211956984d2c986e to 52ef59ed13d878e1a0de0bec999a9d23a442ca6f
Branch pushed to git repo; I updated commit sha1. New commits:
52ef59e  build/pkgs/openblas/spkgconfigure.m4: Do not use AC_FC_FUNC.

comment:40 Changed 4 months ago by
Not accepted by configure yet: glpk
, r
comment:42 in reply to: ↑ 39 Changed 4 months ago by
Replying to git:
Branch pushed to git repo; I updated commit sha1. New commits:
52ef59e build/pkgs/openblas/spkgconfigure.m4: Do not use AC_FC_FUNC.
why? If AC_FC_FREEFORM
can only be called once, we can move
the call to AC_FC_FUNC
to the very place it's called.
How about

build/pkgs/gfortran/spkgconfigure.m4
a b SAGE_SPKG_CONFIGURE([gfortran], [ 6 6 # This helps verify the compiler works too, so if some idiot sets FC to 7 7 # /usr/bin/ls, we will at least know it's not a working Fortran 8 8 # compiler. 9 AC_FC_FREEFORM([SAGE_HAVE_FC_FREEFORM=yes], [ 9 AC_FC_FREEFORM([ 10 SAGE_HAVE_FC_FREEFORM=yes 11 AC_FC_FUNC([dgeqrf]) 12 ], [ 10 13 AC_MSG_NOTICE([Your Fortran compiler does not accept freeformat source code]) 11 14 AC_MSG_NOTICE([which means the compiler is either seriously broken, or]) 12 15 AC_MSG_NOTICE([is too old to build Sage.]) 
build/pkgs/openblas/spkgconfigure.m4
diff git a/build/pkgs/openblas/spkgconfigure.m4 b/build/pkgs/openblas/spkgconfigure.m4 index c37db649be..ecc7540ee9 100644
a b SAGE_SPKG_CONFIGURE([openblas], [ 9 9 dnl openblas does not work as cblas; try to use system's cblas as is 10 10 PKG_CHECK_MODULES([CBLAS], [cblas], [], [sage_spkg_install_openblas=yes]) 11 11 ]) 12 AC_FC_FREEFORM([AC_FC_FUNC([dgeqrf])])13 12 AC_SEARCH_LIBS([$dgeqrf], [openblas], [dnl openblas works as lapack 14 13 sage_install_lapack_pc=yes 15 14 ], [
comment:43 followup: ↓ 44 Changed 4 months ago by
OK... let me try
comment:44 in reply to: ↑ 43 Changed 4 months ago by
Replying to mkoeppe:
OK... let me try
it appears to work with gfortan etc installed:
 Checking whether SageMath should install SPKG openblas... checking BLAS library... openblas checking whether any of gfortran is installed or will be installed as SPKG... no checking for OPENBLAS... yes checking for library containing cblas_dgemm... lopenblas checking for library containing dgeqrf_... none required configure: will use system package and not install SPKG openblas
but perhaps it would be throwing up a surprise if there wasn't any.
comment:45 Changed 4 months ago by
 Commit changed from 52ef59ed13d878e1a0de0bec999a9d23a442ca6f to 9dd37760062337ca4e97263c713d8a46880644b8
comment:46 Changed 4 months ago by
 Commit changed from 9dd37760062337ca4e97263c713d8a46880644b8 to 886ca2f0392764055804e44597dc38125bd93e9c
Branch pushed to git repo; I updated commit sha1. New commits:
886ca2f  tox.ini [homebrew]: make build

comment:47 Changed 4 months ago by
matplotlib install fails:
REQUIRED DEPENDENCIES AND EXTENSIONS numpy: yes [not found. pip may install it below.] install_requires: yes [handled by setuptools] libagg: yes [pkgconfig information for 'libagg' could not be found. Using local copy.] freetype: yes [version 2.10.1] png: no [The C/C++ header for libpng (png.h) could not be found. You may need to install the development package.] qhull: yes [pkgconfig information for 'libqhull' could not be found. Using local copy.] ... * The following required packages can not be built: * png * Try installing png with `brew install libpng` and * pkgconfig with `brew install pkgconfig` Cleaning up...
comment:48 Changed 4 months ago by
cysignals testsuite fails:
[cysignals1.10.2] ulimit 2>/dev/null s 1024; sagepython23 B rundoctests.py src/cysignals/*.pyx [cysignals1.10.2] Doctesting 5 files. [cysignals1.10.2] Traceback (most recent call last): [cysignals1.10.2] File "rundoctests.py", line 74, in <module> [cysignals1.10.2] resource.setrlimit(resource.RLIMIT_STACK, (stacksize, stacksize)) [cysignals1.10.2] ValueError: current limit exceeds maximum limit [cysignals1.10.2] make[4]: *** [checkdoctest] Error 1 [cysignals1.10.2] cd example && sagepython23 setup.py clean build
comment:49 Changed 4 months ago by
 Commit changed from 886ca2f0392764055804e44597dc38125bd93e9c to 185dbbb346e7bea59d6108e2ca8e3b33cb734d03
comment:50 Changed 4 months ago by
 Commit changed from 185dbbb346e7bea59d6108e2ca8e3b33cb734d03 to 6750bf1d0bbe58b4ec5526af2fb50361e489a42e
Branch pushed to git repo; I updated commit sha1. New commits:
6750bf1  tox.ini [local]: Disable linbox testsuite

comment:51 Changed 4 months ago by
Ready for review.
comment:52 Changed 4 months ago by
 Status changed from needs_review to needs_work
comment:53 Changed 4 months ago by
The AC_FC_FUNC business broke all minimal
targets
(https://github.com/mkoeppe/sage/actions/runs/54084841)
comment:54 Changed 4 months ago by
 Commit changed from 6750bf1d0bbe58b4ec5526af2fb50361e489a42e to 071e0b33470bc6fc13296052a831d84d85d4a530
comment:55 Changed 4 months ago by
comment:56 Changed 4 months ago by
 Status changed from needs_work to needs_review
comment:57 Changed 4 months ago by
Needs review
comment:58 Changed 4 months ago by
Both homebrewmacosminimal
(logs: https://github.com/mkoeppe/sage/suites/515691370/artifacts/2830410) and homebrewmacosstandard
(logs: https://github.com/mkoeppe/sage/suites/515691370/artifacts/2830411) now build correctly all the way to sagelib
.
comment:59 followup: ↓ 61 Changed 4 months ago by
Some doctest failures similar to this:
File "src/sage/structure/element.pyx", line 3945, in sage.structure.element.EuclideanDomainElement._mod_ Failed example: cython(''' from sage.structure.element cimport EuclideanDomainElement cdef class MyElt(EuclideanDomainElement): def quo_rem(self, other): return self._parent.var('quo,rem') ''') Expected nothing Got: ld: warning: directory not found for option 'L/Users/runner/runners/2.165.2/work/sage/sage/.tox/localhomebrewmacosminimal/homebrew/opt/readline/lib'
comment:60 Changed 4 months ago by
 Commit changed from 071e0b33470bc6fc13296052a831d84d85d4a530 to d1eb116499937578c84f3b27097bf4d3f4194b2f
Branch pushed to git repo; I updated commit sha1. New commits:
d1eb116  tox.ini: Move homebrew environment settings to script .homebrewbuildenv

comment:61 in reply to: ↑ 59 Changed 4 months ago by
Replying to mkoeppe:
Some doctest failures similar to this: [...readline...]
These are now fixed. Very few remaining doctest failures and timeouts for homebrewmacosminimal
(https://github.com/mkoeppe/sage/runs/508217498):
sage t src/sage/interfaces/gap.py # 1 doctest failed sage t src/sage/manifolds/differentiable/degenerate_submanifold.py # Timed out sage t src/sage/manifolds/differentiable/tensorfield.py # Timed out sage t src/sage/plot/plot3d/base.pyx # Timed out sage t src/sage/plot/plot3d/implicit_plot3d.py # Timed out sage t src/sage/plot/plot3d/parametric_plot3d.py # Timed out sage t src/sage/plot/plot3d/plot3d.py # Timed out sage t src/sage/plot/plot3d/shapes.pyx # Timed out sage t src/sage/rings/padics/padic_lattice_element.py # 3 doctests failed sage t src/sage/plot/plot3d/shapes2.py # Timed out sage t src/sage/rings/function_field/function_field.py # Timed out sage t src/sage/schemes/elliptic_curves/ell_number_field.py # Timed out
homebrewmacosstandard
(https://github.com/mkoeppe/sage/runs/508217500) has many more doctest failures.
Looks like there is some problem with singular.
Some messages from singular: univariate factorization depends on NTL(missing)
In that build, Sage is using the homebrew NTL:
Checking whether SageMath should install SPKG ntl... checking installing gmp/mpir? ... no checking NTL/ZZ.h usability... yes checking NTL/ZZ.h presence... yes checking for NTL/ZZ.h... yes checking whether we can link a program using NTL... yes checking NTL version >= 10.3... 11.4.3 configure: will use system package and not install SPKG ntl using ntl library from the system
... but Singular does not find it ....
checking for NTL >= 5.0... not found configure: WARNING: Unable to find NTL (which is strongly recommended) on your machine: please use withntl=PATH_TO_DIR_CONTAINING_LIB_AND_INCLUDE (see also ./configure help if you do not understand what we are talking about)
comment:62 Changed 4 months ago by
 Dependencies changed from #29273 to #29273, #29335
comment:63 Changed 4 months ago by
 Commit changed from d1eb116499937578c84f3b27097bf4d3f4194b2f to 6410c1e02cf69213efe2e888bd63c1d1856cc135
comment:64 Changed 4 months ago by
comment:65 Changed 4 months ago by
With #29335 merged, Singular finds homebrew's NTL but then I get the following linker error (https://github.com/mkoeppe/sage/runs/508693083):
[singular4.1.1p2.p0] ld: illegal thread local variable reference to regular symbol __ZN3NTL13ErrorCallbackE for architecture x86_64 [singular4.1.1p2.p0] clang: error: linker command failed with exit code 1 (use v to see invocation) [singular4.1.1p2.p0] make[8]: *** [libSingular.la] Error 1
comment:66 followup: ↓ 67 Changed 4 months ago by
this is about absence of coherence on TLS between Singular and Flint.
comment:67 in reply to: ↑ 66 Changed 4 months ago by
Replying to dimpase:
this is about absence of coherence on TLS between Singular and Flint.
OK, is there a way to enable use of TLS with Singular?
comment:68 Changed 4 months ago by
 Commit changed from 6410c1e02cf69213efe2e888bd63c1d1856cc135 to ec5cd78603542b57f58f4ae0d43ba41151a55ab6
Branch pushed to git repo; I updated commit sha1. New commits:
96553db  build/pkgs/ntl/spkgconfigure.m4: Check for NTL configuration

ec5cd78  Merge branch 't/29339/fix_ntl_spkg_configure_m4_so_it_rejects_ntls_built_with_ntl_threads__without_ntl_gmp_lip__without_ntl_gf2x_lib' into t/29104/toxhomebrew

comment:69 Changed 4 months ago by
 Dependencies changed from #29273, #29335 to #29273, #29335, #29339
comment:70 followup: ↓ 72 Changed 4 months ago by
Build logs for homebrewmacosstandard
at https://github.com/mkoeppe/sage/runs/511533264
comment:71 Changed 4 months ago by
 Commit changed from ec5cd78603542b57f58f4ae0d43ba41151a55ab6 to 6d2f6e1fc67097613f08f326a43a6749073e7bd0
comment:72 in reply to: ↑ 70 Changed 4 months ago by
Replying to mkoeppe:
Build logs for
homebrewmacosstandard
at https://github.com/mkoeppe/sage/runs/511533264
[sagelib9.1.beta7] ld: illegal thread local variable reference to regular symbol __ZN3NTL20ZZXFac_InitNumPrimesE for architecture x86_64 [sagelib9.1.beta7] clang: error: linker command failed with exit code 1 (use v to see invocation)
comment:73 Changed 4 months ago by
I guess this means that another library is picking up the system installation of NTL...
comment:74 Changed 4 months ago by
 Commit changed from 6d2f6e1fc67097613f08f326a43a6749073e7bd0 to 14e9f1270e7706e9d771ff33efe259e135fa9ec4
Branch pushed to git repo; I updated commit sha1. New commits:
14e9f12  build/pkgs/ntl/distros/homebrew.txt: Remove ntl

comment:75 followup: ↓ 77 Changed 4 months ago by
comment:76 in reply to: ↑ 36 Changed 4 months ago by
Replying to ghmwageringel:
There seems to be a conflict between
flint
andsagemath/science/flint
, assagemath/science/arb
installsflint
from homebrewcore as a dependency (which has been added to Homebrew very recently). I am usingbrew install sagemath/science/flint brew install ignoredependencies sagemath/science/arbas a workaround now, but the arb formula probably needs to be updated.
As can be seen in the build logs above, sagemath/science/arb
installs flint
from homebrewcore as a dependency. Later, sagemath/science/flint
is installed only because of brew upgrade
. This only seems to work because of the coincidence that sagemath/science/flint
has a higher version number than flint
, currently. However, sagemath/science/arb
is only compiled with flint
(without ntl support) – it is not recompiled after installing sagemath/science/flint
.
I have opened a pull request to resolve this.
comment:77 in reply to: ↑ 75 Changed 4 months ago by
Replying to mkoeppe:
If you remove ntl, you probably also need to remove flint and arb, as they depend on it.
comment:78 Changed 4 months ago by
 Commit changed from 14e9f1270e7706e9d771ff33efe259e135fa9ec4 to 70e161bdd3f29ae1b25dffab08829a20e2c53e5c
Branch pushed to git repo; I updated commit sha1. New commits:
70e161b  Remove use of homebrew packages flint, arb

comment:79 Changed 4 months ago by
Thank you!
comment:80 Changed 4 months ago by
New tests at https://github.com/mkoeppe/sage/runs/512243289
comment:81 Changed 4 months ago by
I had no idea that a ticket about homebrew would be touching fedorarelated stuff, and in this I essentially created a duplicate of that work on #29352
It's much better to create separate tickets for unrelated problems  not the least as it gives them a chance of being discovered.
comment:82 Changed 4 months ago by
OK. What can I do to facilitate review?
comment:83 Changed 4 months ago by
I cannot review MacOSrelated things  I don't have access to the needed OS. But I would be happy to look at the other things that got silently bundled here.
comment:84 followup: ↓ 85 Changed 4 months ago by
OK, then let me move the fedora.txt files to #29352 and the other linux .txt files to a new ticket.
comment:85 in reply to: ↑ 84 Changed 4 months ago by
comment:86 followup: ↓ 88 Changed 4 months ago by
@ghmwageringel Would you be able to review this ticket?
comment:87 Changed 4 months ago by
 Cc tscrim added
 Description modified (diff)
comment:88 in reply to: ↑ 86 ; followup: ↓ 90 Changed 4 months ago by
Replying to mkoeppe:
@ghmwageringel Would you be able to review this ticket?
Yes, I will look into that. Though, I only have access to macOS 10.13.6.
comment:89 Changed 4 months ago by
Tests ran at https://github.com/mkoeppe/sage/actions/runs/57924835
localhomebrewmacosminimal
(https://github.com/mkoeppe/sage/runs/515508463) and localhomebrewmacosstandard
(https://github.com/mkoeppe/sage/runs/515508474):
All packages (except for experimental package polymake
) build, doctests:
 sage t src/sage/game_theory/normal_form_game.py # 5 doctests failed sage t src/sage/dynamics/arithmetic_dynamics/projective_ds.py # Timed out sage t src/sage/interfaces/gap.py # 1 doctest failed sage t src/sage/knots/link.py # 1 doctest failed sage t src/sage/manifolds/differentiable/degenerate_submanifold.py # Timed out sage t src/sage/manifolds/differentiable/tensorfield.py # Timed out sage t src/sage/plot/plot3d/base.pyx # Timed out sage t src/sage/plot/plot3d/implicit_plot3d.py # Timed out sage t src/sage/plot/plot3d/parametric_plot3d.py # Timed out sage t src/sage/plot/plot3d/plot3d.py # Timed out sage t src/sage/plot/plot3d/shapes.pyx # Timed out sage t src/sage/rings/padics/padic_lattice_element.py # 3 doctests failed sage t src/sage/plot/plot3d/shapes2.py # Timed out sage t src/sage/rings/function_field/function_field.py # Timed out sage t src/sage/schemes/elliptic_curves/ell_number_field.py # Timed out 
comment:90 in reply to: ↑ 88 Changed 4 months ago by
Replying to ghmwageringel:
Replying to mkoeppe:
@ghmwageringel Would you be able to review this ticket?
Yes, I will look into that.
Thanks!
Though, I only have access to macOS 10.13.6.
That's actually great, for some broader coverage. I'm testing on 10.15.3, and also the GitHub runners are on Catalina.
comment:91 Changed 4 months ago by
lots of timeouts in tests, probably an overloaded host, or too aggressive parallelisation.
comment:92 Changed 4 months ago by
Yes, this is using 12 threads
comment:93 followup: ↓ 102 Changed 4 months ago by
I'm in the middle of trying this. A few comments:
 patch: this should use homebrew's patch
 pcre: same
 boost: is there a way to convince Sage to use homebrew's boost as a replacement for boostcropped? (See #29100?)
 ntl: being discussed on another ticket, but my Sage builds on OS X regularly use homebrew's ntl without problems.
comment:94 followup: ↓ 98 Changed 4 months ago by
Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?
comment:95 followup: ↓ 99 Changed 4 months ago by
I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.
comment:96 followup: ↓ 97 Changed 4 months ago by
ntl failed to build. This is on OS X 10.15.3.
make setup3 g++ std=gnu++11 I../include I. O2 g fnocommon march=native L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage9.1.beta7/.tox/localhomebrewmacosstandard/local/lib Wl,rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage9.1.beta7/.tox/localhomebrewmacosstandard/local/lib L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage9.1.beta7/.tox/localhomebrewmacosstandard/homebrew/opt/readline/lib L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage9.1.beta7/.tox/localhomebrewmacosstandard/homebrew/lib I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage9.1.beta7/.tox/localhomebrewmacosstandard/local/include o gen_gmp_aux gen_gmp_aux.cpp L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage9.1.beta7/.tox/localhomebrewmacosstandard/local/lib lgmp lm ./gen_gmp_aux > ../include/NTL/gmp_aux.h NTL_GMP_LIP flag set GMP version check (6.1.2/6.2.0) *** version number mismatch: inconsistency between gmp.h and libgmp /bin/sh: line 1: 18441 Abort trap: 6 ./gen_gmp_aux > ../include/NTL/gmp_aux.h make[5]: *** [setup3] Error 134 make[4]: *** [setupphase] Error 2 make[4]: Target `all' not remade because of errors.
comment:97 in reply to: ↑ 96 Changed 4 months ago by
Replying to jhpalmieri:
ntl failed to build. This is on OS X 10.15.3.
This happens when you have a different (older) version of gmp in your /usr/local homebrew. The isolation is unfortunately not perfect. Badly written configure scripts of packages pick up stuff from /usr/local
comment:98 in reply to: ↑ 94 ; followup: ↓ 118 Changed 4 months ago by
Replying to jhpalmieri:
Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?
Good idea. WIll do
comment:99 in reply to: ↑ 95 ; followup: ↓ 121 Changed 4 months ago by
Replying to dimpase:
I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.
Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.
comment:100 Changed 4 months ago by
 Dependencies changed from #29273, #29335, #29339 to #29273, #29335, #29339, #29361
comment:101 Changed 4 months ago by
 Commit changed from 70e161bdd3f29ae1b25dffab08829a20e2c53e5c to 11f77293e372d29e01b6525e1fb1948c4d855f04
Branch pushed to git repo; I updated commit sha1. New commits:
11f7729  build/pkgs/patch/distros/homebrew.txt: New

comment:102 in reply to: ↑ 93 ; followup: ↓ 104 Changed 4 months ago by
Replying to jhpalmieri:
I'm in the middle of trying this. A few comments:
 pcre: same
This one actually works for me (both locally and on the GitHub runner). Please post the logs if it doesn't work for you
comment:103 Changed 4 months ago by
 Commit changed from 11f77293e372d29e01b6525e1fb1948c4d855f04 to 93a0421c7789a1ac278386f9643d87b9768a5e69
Branch pushed to git repo; I updated commit sha1. New commits:
93a0421  build/pkgs/boost*/distros/homebrew.txt: New

comment:104 in reply to: ↑ 102 Changed 4 months ago by
Replying to mkoeppe:
Replying to jhpalmieri:
I'm in the middle of trying this. A few comments:
 pcre: same
This one actually works for me (both locally and on the GitHub runner). Please post the logs if it doesn't work for you
Sorry, it looks okay to me. I must have misread config.log.
comment:105 Changed 4 months ago by
Flint fails to build for me. I ran brew upgrade
, cleaned everything out, and tried again, and it still fails.
g++ std=gnu++11 fPIC fnocommon pedantic Wall O2 funrollloops g mpopcnt I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage9.1.beta7/.tox/localhomebrewmacosstandard/local/var/tmp/sage/build/flint2.5.2.p5/src I/usr/local/include I/usr/local/include I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage9.1.beta7/.tox/localhomebrewmacosstandard/local/include c interfaces/NTLinterface.cpp o build/interfaces/NTLinterface.lo ld: illegal thread local variable reference to regular symbol __ZN3NTL8ZZ_pInfoE for architecture x86_64 clang: error: linker command failed with exit code 1 (use v to see invocation) make[6]: *** [libflint13.5.2.dylib] Error 1 make[6]: Target `shared' not remade because of errors. make[5]: *** [library] Error 2 make[5]: Target `all' not remade because of errors. make[4]: *** [verbose] Error 2
comment:106 followup: ↓ 108 Changed 4 months ago by
Is homebrew ntl installed on this system?
comment:107 Changed 4 months ago by
(in /usr/local)
comment:108 in reply to: ↑ 106 ; followup: ↓ 109 Changed 4 months ago by
Replying to mkoeppe:
Is homebrew ntl installed on this system?
Yes. Do I need to delete it?
By the way, should prefix
be added to .gitignore
?
comment:109 in reply to: ↑ 108 ; followup: ↓ 113 Changed 4 months ago by
Replying to jhpalmieri:
Replying to mkoeppe:
Is homebrew ntl installed on this system?
Yes. Do I need to delete it?
Please try if this helps. If it does, we should fix something in the FLINT build scripts
By the way, should
prefix
be added to.gitignore
?
Yes, will do.
comment:110 Changed 4 months ago by
 Commit changed from 93a0421c7789a1ac278386f9643d87b9768a5e69 to 4c653c2759eb404f9a51baf52c260468276ea720
Branch pushed to git repo; I updated commit sha1. New commits:
4c653c2  .gitignore: Add prefix

comment:111 Changed 4 months ago by
Is there a command like make distclean
that will get rid of the stuff that Sage has built but keep the homebrew stuff? I don't really want to wait for gcc to build again. (make distclean
doesn't touch prefix
.)
comment:112 Changed 4 months ago by
At the moment, nothing better than rm rf prefix/*
comment:113 in reply to: ↑ 109 Changed 4 months ago by
Replying to mkoeppe:
Replying to jhpalmieri:
Replying to mkoeppe:
Is homebrew ntl installed on this system?
Yes. Do I need to delete it?
Please try if this helps. If it does, we should fix something in the FLINT build scripts
After doing brew uninstall ntl
, flint now builds.
comment:114 Changed 4 months ago by
 Commit changed from 4c653c2759eb404f9a51baf52c260468276ea720 to 467160cdc8402124b21c5dee60386dee03f18b67
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
2625a64  tox.ini [homebrew]: make build

cf83af6  Add zlib

14a3dab  tox.ini [local]: Disable testsuites for some packages

2553b5a  tox.ini [local]: Disable linbox testsuite

2f79c76  tox.ini: Move homebrew environment settings to script .homebrewbuildenv

77d5c11  build/pkgs/ntl/distros/homebrew.txt: Remove ntl

4ec80e2  Remove use of homebrew packages flint, arb

54c803b  build/pkgs/patch/distros/homebrew.txt: New

2ce6e6c  build/pkgs/boost*/distros/homebrew.txt: New

467160c  .gitignore: Add prefix

comment:115 Changed 4 months ago by
 Dependencies changed from #29273, #29335, #29339, #29361 to #29335, #29361
Rebased  no longer on top of #29339.
comment:116 Changed 4 months ago by
Except for the issue with FLINT not building with a preexisting NTL, it works for me.
comment:117 followup: ↓ 119 Changed 4 months ago by
Let's fix the FLINT issue on a followup ticket.
comment:118 in reply to: ↑ 98 Changed 4 months ago by
Replying to mkoeppe:
Replying to jhpalmieri:
Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?
Good idea. WIll do
I'll do this in #29146
comment:119 in reply to: ↑ 117 Changed 4 months ago by
comment:120 Changed 4 months ago by
Ready for review
comment:121 in reply to: ↑ 99 Changed 4 months ago by
Replying to mkoeppe:
Replying to dimpase:
I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.
Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.
One can split building into make built
and make ptest
, with the latter, much more memory hungry, run with lower value of j
.
comment:122 Changed 4 months ago by
 Commit changed from 467160cdc8402124b21c5dee60386dee03f18b67 to dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc
Branch pushed to git repo; I updated commit sha1. New commits:
dfce7d4  tox.ini [local]: Pass MAKE environment variable, do not default to 'make j12', change default target to 'build', add doc

comment:123 Changed 4 months ago by
OK I've made it more configurable already on this ticket. And I have added documentation
comment:124 Changed 4 months ago by
 Description modified (diff)
comment:125 followup: ↓ 135 Changed 4 months ago by
@ghmwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.
comment:126 Changed 4 months ago by
I'm 8 timezones away from my MacOS machine.
I'd like more details/pointers on how the Github Actions testing is done  for this looks to me like "all sufficiently advanced technology is indistinguishable from magic" kind of thing :)
comment:127 Changed 4 months ago by
comment:128 Changed 4 months ago by
how does one get the data I linked in comment:126 then?
comment:129 followup: ↓ 131 Changed 4 months ago by
comment:130 Changed 4 months ago by
If the GitHub Actions interface is overwhelming, just download the "Artifacts". They are tar.gz files of build logs.
comment:131 in reply to: ↑ 129 Changed 4 months ago by
comment:132 Changed 4 months ago by
I don't push logs to GitHub.
#29087 adds .github/workflows/tox.yml
(https://git.sagemath.org/sage.git/diff/.github/workflows/tox.yml?id=576a18834a16ff05cd6894172a55e86bcc95d407&id2=be1e22c441d9332fd0bf4a3066e1964deddaf8c7) which gets run on every commit pushed to GitHub and on every pull request.
The job toxlocalhomebrewmacos
defined in that file provisions a macOS host, checks out the branch and runs tox e localhomebrewmacosstandard
there. Then it packages up logs and makes them available as an artifact.
comment:133 Changed 4 months ago by
and where is that MacOS host getting provisioned?
comment:134 Changed 4 months ago by
GitHub takes care of that, I think it's using Azure.
comment:135 in reply to: ↑ 125 Changed 4 months ago by
Replying to jhpalmieri:
@ghmwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.
I have not tested this locally, but as you have done that, I am happy with this too.
One last question though: As far as I can see, readline is not installed into /usr/local. Why do you not run into conflicts with macOS's system "readline"/libedit, as in #29000?
comment:136 Changed 4 months ago by
These tests use their own homebrew installation in a prefix .tox/localhomebrewmacosstandard/homebrew/
, and sets up environment variables using the new script .homebrewbuildenv
. This sets LDFLAGS and CPPFLAGS for the readline installed into opt
under the homebrew prefix.
comment:137 Changed 4 months ago by
The problem in #29000 occured despite passing LDFLAGS and CPPFLAGS to configure
, so I guess setting these also during make
would avoid that problem, as a workaround.
comment:138 Changed 4 months ago by
I think the problem description of #29000 needs some more precision.
comment:139 Changed 4 months ago by
 Reviewers set to Dima Pasechnik, John Palmieri
 Status changed from needs_review to positive_review
OK, looks good to me.
comment:140 Changed 4 months ago by
Thanks everyone!
comment:141 Changed 4 months ago by
 Branch changed from u/mkoeppe/toxhomebrew to dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc
 Resolution set to fixed
 Status changed from positive_review to closed
Last 10 new commits:
build/bin/writedockerfile.sh: Add commands to activate conda env
Merge tag '9.1.beta2' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time
add doc
add centos{7,8
}on centos, IGNORE_MISSING_SYSTEM_PACKAGES=yes
Add build/pkgs/gfortran/distros/fedora.txt
build/pkgs/fedora.txt: Add python3
tox.ini: Add debian stretch, put sid last
build/pkgs/openblas/distros/arch.txt: Add lapack, cblas
tox.ini: Add toxenvs localhomebrewmacosminimal, localhomebrewmacosstandard