Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages

[mkoeppe]

#29053 adds tox.ini to support testing the sage distribution in isolated environments via docker.

This ticket adds toxenvs local-homebrew-macos-{minimal,standard} for testing with (best-effort) isolated installations of homebrew on macOS (not using docker).

To test:

    brew install tox
    tox -e local-homebrew-macos-standard

See comments in tox.ini for more examples.

An example run of that is here: ​https://github.com/mkoeppe/sage/actions/runs/32812271

[mkoeppe]

Last 10 new commits:

​7e59a6f	build/bin/write-dockerfile.sh: Add commands to activate conda env
​2ad9a97	Merge tag '9.1.beta2' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time
​9fca230	add doc
​f6cbb8a	add centos-{7,8}
​c651058	on centos, IGNORE_MISSING_SYSTEM_PACKAGES=yes
​98edff6	Add build/pkgs/gfortran/distros/fedora.txt
​843ca76	build/pkgs/fedora.txt: Add python3
​bd6a545	tox.ini: Add debian stretch, put sid last
​23c6334	build/pkgs/openblas/distros/arch.txt: Add lapack, cblas
​fca6654	tox.ini: Add toxenvs local-homebrew-macos-minimal, local-homebrew-macos-standard

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e0e8a4c

tox.ini [local-homebrew]: Fixup PATH, PKG_CONFIG_PATH, set LDFLAGS, CPPFLAGS; [local-direct]: New

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​16ca88f

tox.ini: Use {envlogdir}; use ln -sf

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6a1803d

tox.ini [local]: First build the base-toolchain

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4b72987

tox.ini [local-homebrew]: Set PATH earlier and set ACLOCAL_PATH for bootstrapping

[dimpase]

Is this ticket MacOS-specific?

[mkoeppe]

Hmm, I haven’t tried linuxbrew. Might be worth checking

[dimpase]

I'll fire up my vintage OSX 10.13 macbookair...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2141b75

Merge tag '9.1.beta3' into t/29104/tox-homebrew

[dimpase]

I tried this:

tox -p auto -e local-homebrew-macos-standard

and got

...
[sagelib-9.1.beta2] user	0m10.244s
[sagelib-9.1.beta2] sys	0m3.812s

real	0m16.288s
user	0m10.956s
sys	0m4.047s
Sage build/upgrade complete!
✔ OK local-homebrew-macos-standard in 1 minute, 14.362 seconds
_________________________________________________________________________________________________ summary __________________________________________________________________________________________________
  local-homebrew-macos-standard: commands succeeded
  congratulations :)

not tests were run, it seems, is this normal?

[mkoeppe]

The default packages set in tox.ini are only fflas_ffpack scipy r, and those (and their dependencies) are checked. I don't know why it's actually building sagelib, but I hope to fix bizarre build behavior like this in #29113.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4eb201b

Merge tag '9.1.beta4' into t/29104/tox-homebrew

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7155e79

Merge tag '9.1.beta7' into t/29104/tox-homebrew

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d7a2e72

Fixup merge

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​3103055	m4ri must use png on gentoo
​1b48aa1	Merge tag '9.1.beta7' into t/29273/packages/r_packages_in_distros
​0fc301c	Add build/pkgs/{libgd|m4ri|ninja_build|perl_term_readline_gnu|yasm}/distros/*.txt - from repology
​f7785ce	build/pkgs/fedora.txt: Add diffutils
​6d2bbd8	build/pkgs/arch.txt: Add 'which'
​0aa90c1	build/pkgs/debian.txt: Add ca-certificates
​c49acdc	Merge tag '9.1.beta7' into t/29129/add_more_system_packages_to_build_pkgs_spkg_distros_
​0f704d0	Merge branch 't/29129/add_more_system_packages_to_build_pkgs_spkg_distros_' into t/29273/packages/r_packages_in_distros
​f8097c4	Merge branch 'u/mkoeppe/packages/r_packages_in_distros' of git://trac.sagemath.org/sage into t/29104/tox-homebrew
​9e9f4c7	Add distros/homebrew.txt for many packages, and some more distros/*.txt

[gh-mwageringel]

• mpc should be libmpc
• tachyon does not seem to be the package we use
• pari cannot currently be used by Sage because the pari packages are not in Homebrew
• glpk 4.65 should not be used by Sage as long as it is not patched by Homebrew, see #24824

I also have these Homebrew packages installed:

cmake freetype gcc git libpng pcre pkg-config openssl xz zeromq

Some of these might already be installed automatically as dependencies of others.

Additionally there are:

sagemath/science/flint sagemath/science/arb

Though, I am not sure about how flint in homebrew-core is different from this one.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e8d1fef

Add build/pkgs/homebrew*.txt

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​242f684

Add/fix build/pkgs/*/homebrew.txt - from gh-mwageringel

[dimpase]

​https://github.com/sagemath/homebrew-science/commits/master

adds NTL support for flint, and also ensbles TLS.

we can cook up more of these formulae, theain obstacle for me there was that they ate in Ruby.

---

New commits:

​242f684

Add/fix build/pkgs/*/homebrew.txt - from gh-mwageringel

[mkoeppe]

==> Installing sagemath/science/arb
==> Downloading https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz
##O=#  #                                                                      
curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "arb"
Download failed: https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz
Warning: Bottle installation failed: building from source.
==> Downloading https://github.com/fredrik-johansson/arb/archive/2.16.0.tar.gz
==> Downloading from https://codeload.github.com/fredrik-johansson/arb/tar.gz/2.16.0
######################################################################## 100.0%

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2449c39

tox.ini: Get homebrew packages from *homebrew*.txt

[dimpase]

Replying to mkoeppe:

==> Installing sagemath/science/arb
==> Downloading https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz
##O=#  #                                                                      
curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "arb"
Download failed: https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz
Warning: Bottle installation failed: building from source.
==> Downloading https://github.com/fredrik-johansson/arb/archive/2.16.0.tar.gz
==> Downloading from https://codeload.github.com/fredrik-johansson/arb/tar.gz/2.16.0
######################################################################## 100.0%

yes, this is expected. No binaries.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​188994a

tox.ini: Use brew upgrade after brew install

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6df5104

tox.ini [local-homebrew]: Add to PKG_CONFIG_PATH

[mkoeppe]

Checking whether SageMath should install SPKG openblas...
checking BLAS library... openblas
checking whether any of gfortran is installed or will be installed as SPKG... no
checking for openblas >= 0.2.20... yes
checking for library containing cblas_dgemm... no
checking for cblas... no
checking for library containing ... no
checking for lapack... no
configure: no suitable system package found for SPKG openblas

The tests for cblas_dgemm etc. do not seem to use OPENBLAS_LIBS at all!

[gh-mwageringel]

Replying to dimpase:

​https://github.com/sagemath/homebrew-science/commits/master

adds NTL support for flint, and also ensbles TLS.

we can cook up more of these formulae, theain obstacle for me there was that they ate in Ruby.

There seems to be a conflict between flint and sagemath/science/flint, as sagemath/science/arb installs flint from homebrew-core as a dependency (which has been added to Homebrew very recently). I am using

brew install sagemath/science/flint
brew install --ignore-dependencies sagemath/science/arb

as a workaround now, but the arb formula probably needs to be updated.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c9d5564

build/pkgs/openblas/spkg-configure.m4: Use OPENBLAS_{LIBS,CFLAGS} while checking for cblas/lapack functions

[mkoeppe]

This line (introduced in #29071):

        AC_FC_FREEFORM([AC_FC_FUNC([dgeqrf])])

is not executed at all because AC_FC_FREEFORM is AC_DEFUN_ONCE and is called already in gfortran/spkg-configure.m4

As a result:

Checking whether SageMath should install SPKG openblas...
checking BLAS library... openblas
checking whether any of gfortran is installed or will be installed as SPKG... no
checking for openblas >= 0.2.20... yes
checking for cblas_dgemm... yes
checking for ... no
checking for lapack... no
configure: no suitable system package found for SPKG openblas

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​52ef59e

build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC.

[mkoeppe]

Not accepted by configure yet: glpk, r

[mkoeppe]

(resolved)

[dimpase]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​52ef59e

build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC.

why? If AC_FC_FREEFORM can only be called once, we can move the call to AC_FC_FUNC to the very place it's called. How about

build/pkgs/gfortran/spkg-configure.m4

@@ -6 +6 @@
     # This helps verify the compiler works too, so if some idiot sets FC to
     # /usr/bin/ls, we will at least know it's not a working Fortran
     # compiler.
-    AC_FC_FREEFORM([SAGE_HAVE_FC_FREEFORM=yes], [
+    AC_FC_FREEFORM([
+        SAGE_HAVE_FC_FREEFORM=yes
+        AC_FC_FUNC([dgeqrf])
+       ], [
        AC_MSG_NOTICE([Your Fortran compiler does not accept free-format source code])
         AC_MSG_NOTICE([which means the compiler is either seriously broken, or])
         AC_MSG_NOTICE([is too old to build Sage.])

build/pkgs/openblas/spkg-configure.m4

@@ -9 +9 @@
              dnl openblas does not work as cblas; try to use system's cblas as is
              PKG_CHECK_MODULES([CBLAS], [cblas], [], [sage_spkg_install_openblas=yes])
           ])
-       AC_FC_FREEFORM([AC_FC_FUNC([dgeqrf])])
        AC_SEARCH_LIBS([$dgeqrf], [openblas], [dnl openblas works as lapack
              sage_install_lapack_pc=yes
              ], [

[mkoeppe]

OK... let me try

[dimpase]

Replying to mkoeppe:

OK... let me try

it appears to work with gfortan etc installed:

-----------------------------------------------------------------------------
Checking whether SageMath should install SPKG openblas...
checking BLAS library... openblas
checking whether any of gfortran is installed or will be installed as SPKG... no
checking for OPENBLAS... yes
checking for library containing cblas_dgemm... -lopenblas
checking for library containing dgeqrf_... none required
configure: will use system package and not install SPKG openblas

but perhaps it would be throwing up a surprise if there wasn't any.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f049b9b	Revert "build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC."
​9dd3776	Move AC_FC_FUNC([dgeqrf]) from openblas/spkg-configure.m4 to gfortran/spkg-configure.m4

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​886ca2f

tox.ini [homebrew]: make build

[mkoeppe]

matplotlib install fails:

    REQUIRED DEPENDENCIES AND EXTENSIONS
                     numpy: yes [not found. pip may install it below.]
          install_requires: yes [handled by setuptools]
                    libagg: yes [pkg-config information for 'libagg' could not
                            be found. Using local copy.]
                  freetype: yes [version 2.10.1]
                       png: no  [The C/C++ header for libpng (png.h) could not
                            be found.  You may need to install the development
                            package.]
                     qhull: yes [pkg-config information for 'libqhull' could not
                            be found. Using local copy.]
...
                            * The following required packages can not be built:
                            * png
                            * Try installing png with `brew install libpng` and
                            * pkg-config with `brew install pkg-config`
Cleaning up...

[mkoeppe]

cysignals testsuite fails:

  [cysignals-1.10.2]   ulimit 2>/dev/null -s 1024; sage-python23 -B rundoctests.py src/cysignals/*.pyx
  [cysignals-1.10.2]   Doctesting 5 files.
  [cysignals-1.10.2]   Traceback (most recent call last):
  [cysignals-1.10.2]     File "rundoctests.py", line 74, in <module>
  [cysignals-1.10.2]       resource.setrlimit(resource.RLIMIT_STACK, (stacksize, stacksize))
  [cysignals-1.10.2]   ValueError: current limit exceeds maximum limit
  [cysignals-1.10.2]   make[4]: *** [check-doctest] Error 1
  [cysignals-1.10.2]   cd example && sage-python23 setup.py clean build

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fdeced3	Add zlib
​185dbbb	tox.ini [local]: Disable testsuites for some packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6750bf1

tox.ini [local]: Disable linbox testsuite

[mkoeppe]

Ready for review.

[mkoeppe]

The AC_FC_FUNC business broke all -minimal targets (​https://github.com/mkoeppe/sage/actions/runs/54084841)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9f486ef	Revert "Move AC_FC_FUNC([dgeqrf]) from openblas/spkg-configure.m4 to gfortran/spkg-configure.m4"
​071e0b3	Revert "Revert "build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC.""

[mkoeppe]

New tests at ​https://github.com/mkoeppe/sage/actions/runs/54093138

[mkoeppe]

Needs review

[mkoeppe]

Both homebrew-macos-minimal (logs: ​https://github.com/mkoeppe/sage/suites/515691370/artifacts/2830410) and homebrew-macos-standard (logs: ​https://github.com/mkoeppe/sage/suites/515691370/artifacts/2830411) now build correctly all the way to sagelib.

[mkoeppe]

Some doctest failures similar to this:

File "src/sage/structure/element.pyx", line 3945, in sage.structure.element.EuclideanDomainElement._mod_
Failed example:
    cython('''
    from sage.structure.element cimport EuclideanDomainElement
    cdef class MyElt(EuclideanDomainElement):
        def quo_rem(self, other):
            return self._parent.var('quo,rem')
    ''')
Expected nothing
Got:
    ld: warning: directory not found for option '-L/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-minimal/homebrew/opt/readline/lib'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d1eb116

tox.ini: Move homebrew environment settings to script .homebrew-build-env

[mkoeppe]

Replying to mkoeppe:

Some doctest failures similar to this: [...readline...]

These are now fixed. Very few remaining doctest failures and timeouts for homebrew-macos-minimal (​https://github.com/mkoeppe/sage/runs/508217498):

sage -t src/sage/interfaces/gap.py  # 1 doctest failed
sage -t src/sage/manifolds/differentiable/degenerate_submanifold.py  # Timed out
sage -t src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t src/sage/plot/plot3d/base.pyx  # Timed out
sage -t src/sage/plot/plot3d/implicit_plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/parametric_plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/shapes.pyx  # Timed out
sage -t src/sage/rings/padics/padic_lattice_element.py  # 3 doctests failed
sage -t src/sage/plot/plot3d/shapes2.py  # Timed out
sage -t src/sage/rings/function_field/function_field.py  # Timed out
sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # Timed out

homebrew-macos-standard (​https://github.com/mkoeppe/sage/runs/508217500) has many more doctest failures. Looks like there is some problem with singular. Some messages from singular: univariate factorization depends on NTL(missing)

In that build, Sage is using the homebrew NTL:

Checking whether SageMath should install SPKG ntl...
checking installing gmp/mpir? ... no
checking NTL/ZZ.h usability... yes
checking NTL/ZZ.h presence... yes
checking for NTL/ZZ.h... yes
checking whether we can link a program using NTL... yes
checking NTL version >= 10.3... 11.4.3
configure: will use system package and not install SPKG ntl
using ntl library from the system

... but Singular does not find it ....

checking for NTL >= 5.0... not found
configure: WARNING: Unable to find NTL (which is strongly recommended) on your machine: please use --with-ntl=PATH_TO_DIR_CONTAINING_LIB_AND_INCLUDE (see also ./configure --help if you do not understand what we are talking about)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​50f62c7	build/pkgs/singular: Add patches to accept system NTL in nonstandard locations
​6410c1e	Merge branch 't/29335/always_find_sage_ntl_prefix_and_call_singular_configure_with_it' into t/29104/tox-homebrew

[mkoeppe]

New tests at ​https://github.com/mkoeppe/sage/actions/runs/55944733

[mkoeppe]

With #29335 merged, Singular finds homebrew's NTL but then I get the following linker error (​https://github.com/mkoeppe/sage/runs/508693083):

  [singular-4.1.1p2.p0]   ld: illegal thread local variable reference to regular symbol __ZN3NTL13ErrorCallbackE for architecture x86_64
  [singular-4.1.1p2.p0]   clang: error: linker command failed with exit code 1 (use -v to see invocation)
  [singular-4.1.1p2.p0]   make[8]: *** [libSingular.la] Error 1

[dimpase]

this is about absence of coherence on TLS between Singular and Flint.

[mkoeppe]

Replying to dimpase:

this is about absence of coherence on TLS between Singular and Flint.

OK, is there a way to enable use of TLS with Singular?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​96553db	build/pkgs/ntl/spkg-configure.m4: Check for NTL configuration
​ec5cd78	Merge branch 't/29339/fix_ntl_spkg_configure_m4_so_it_rejects_ntls_built_with_ntl_threads__without_ntl_gmp_lip__without_ntl_gf2x_lib' into t/29104/tox-homebrew

[mkoeppe]

Build logs for homebrew-macos-standard at ​https://github.com/mkoeppe/sage/runs/511533264

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c3bc092	Remove tests for NTL_GMP_LIP, NTL_GF2X_LIB
​6d2f6e1	Merge branch 't/29339/fix_ntl_spkg_configure_m4_so_it_rejects_ntls_built_with_ntl_threads__without_ntl_gmp_lip__without_ntl_gf2x_lib' into t/29104/tox-homebrew

[mkoeppe]

Replying to mkoeppe:

Build logs for homebrew-macos-standard at ​https://github.com/mkoeppe/sage/runs/511533264

  [sagelib-9.1.beta7]   ld: illegal thread local variable reference to regular symbol __ZN3NTL20ZZXFac_InitNumPrimesE for architecture x86_64
  [sagelib-9.1.beta7]   clang: error: linker command failed with exit code 1 (use -v to see invocation)

[mkoeppe]

I guess this means that another library is picking up the system installation of NTL...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​14e9f12

build/pkgs/ntl/distros/homebrew.txt: Remove ntl

[mkoeppe]

New test: ​https://github.com/mkoeppe/sage/runs/512132585

[gh-mwageringel]

Replying to gh-mwageringel:

There seems to be a conflict between flint and sagemath/science/flint, as sagemath/science/arb installs flint from homebrew-core as a dependency (which has been added to Homebrew very recently). I am using

brew install sagemath/science/flint
brew install --ignore-dependencies sagemath/science/arb

as a workaround now, but the arb formula probably needs to be updated.

As can be seen in the build logs above, sagemath/science/arb installs flint from homebrew-core as a dependency. Later, sagemath/science/flint is installed only because of brew upgrade. This only seems to work because of the coincidence that sagemath/science/flint has a higher version number than flint, currently. However, sagemath/science/arb is only compiled with flint (without ntl support) – it is not recompiled after installing sagemath/science/flint.

I have opened a ​pull request to resolve this.

[gh-mwageringel]

Replying to mkoeppe:

New test: ​https://github.com/mkoeppe/sage/runs/512132585

If you remove ntl, you probably also need to remove flint and arb, as they depend on it.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​70e161b

Remove use of homebrew packages flint, arb

[mkoeppe]

Thank you!

[mkoeppe]

New tests at ​https://github.com/mkoeppe/sage/runs/512243289

[dimpase]

I had no idea that a ticket about homebrew would be touching fedora-related stuff, and in this I essentially created a duplicate of that work on #29352

It's much better to create separate tickets for unrelated problems - not the least as it gives them a chance of being discovered.

[mkoeppe]

OK. What can I do to facilitate review?

[dimpase]

I cannot review MacOS-related things - I don't have access to the needed OS. But I would be happy to look at the other things that got silently bundled here.

[mkoeppe]

OK, then let me move the fedora.txt files to #29352 and the other linux .txt files to a new ticket.

[mkoeppe]

Replying to mkoeppe:

OK, then let me move the fedora.txt files to #29352 and the other linux .txt files to a new ticket.

Actually, as noted on #29352, most of the files were added in #29273.

[mkoeppe]

@gh-mwageringel Would you be able to review this ticket?

[gh-mwageringel]

Replying to mkoeppe:

@gh-mwageringel Would you be able to review this ticket?

Yes, I will look into that. Though, I only have access to macOS 10.13.6.

[mkoeppe]

Tests ran at ​https://github.com/mkoeppe/sage/actions/runs/57924835

local-homebrew-macos-minimal (​https://github.com/mkoeppe/sage/runs/515508463) and local-homebrew-macos-standard (​https://github.com/mkoeppe/sage/runs/515508474):

All packages (except for experimental package polymake) build, doctests:

----------------------------------------------------------------------
sage -t src/sage/game_theory/normal_form_game.py  # 5 doctests failed
sage -t src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # Timed out
sage -t src/sage/interfaces/gap.py  # 1 doctest failed
sage -t src/sage/knots/link.py  # 1 doctest failed
sage -t src/sage/manifolds/differentiable/degenerate_submanifold.py  # Timed out
sage -t src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t src/sage/plot/plot3d/base.pyx  # Timed out
sage -t src/sage/plot/plot3d/implicit_plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/parametric_plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/shapes.pyx  # Timed out
sage -t src/sage/rings/padics/padic_lattice_element.py  # 3 doctests failed
sage -t src/sage/plot/plot3d/shapes2.py  # Timed out
sage -t src/sage/rings/function_field/function_field.py  # Timed out
sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # Timed out
----------------------------------------------------------------------

[mkoeppe]

Replying to gh-mwageringel:

Replying to mkoeppe:

@gh-mwageringel Would you be able to review this ticket?

Yes, I will look into that.

Thanks!

Though, I only have access to macOS 10.13.6.

That's actually great, for some broader coverage. I'm testing on 10.15.3, and also the GitHub runners are on Catalina.

[dimpase]

lots of timeouts in tests, probably an overloaded host, or too aggressive parallelisation.

[mkoeppe]

Yes, this is using 12 threads

[jhpalmieri]

I'm in the middle of trying this. A few comments:

• patch: this should use homebrew's patch
• pcre: same
• boost: is there a way to convince Sage to use homebrew's boost as a replacement for boost-cropped? (See #29100?)
• ntl: being discussed on another ticket, but my Sage builds on OS X regularly use homebrew's ntl without problems.

[jhpalmieri]

Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?

[dimpase]

I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.

[jhpalmieri]

ntl failed to build. This is on OS X 10.15.3.

make setup3
g++ -std=gnu++11 -I../include -I.  -O2 -g  -fno-common -march=native -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/homebrew/opt/readline/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/homebrew/lib  -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/include  -o gen_gmp_aux gen_gmp_aux.cpp -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib  -lgmp  -lm
./gen_gmp_aux > ../include/NTL/gmp_aux.h 
NTL_GMP_LIP flag set
GMP version check (6.1.2/6.2.0)
*** version number mismatch: inconsistency between gmp.h and libgmp
/bin/sh: line 1: 18441 Abort trap: 6           ./gen_gmp_aux > ../include/NTL/gmp_aux.h
make[5]: *** [setup3] Error 134
make[4]: *** [setup-phase] Error 2
make[4]: Target `all' not remade because of errors.

[mkoeppe]

Replying to jhpalmieri:

ntl failed to build. This is on OS X 10.15.3.

This happens when you have a different (older) version of gmp in your /usr/local homebrew. The isolation is unfortunately not perfect. Badly written configure scripts of packages pick up stuff from /usr/local

[mkoeppe]

Replying to jhpalmieri:

Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?

Good idea. WIll do

[mkoeppe]

Replying to dimpase:

I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.

Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​11f7729

build/pkgs/patch/distros/homebrew.txt: New

[mkoeppe]

Replying to jhpalmieri:

I'm in the middle of trying this. A few comments:

• pcre: same

This one actually works for me (both locally and on the GitHub runner). Please post the logs if it doesn't work for you

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​93a0421

build/pkgs/boost*/distros/homebrew.txt: New

[jhpalmieri]

Replying to mkoeppe:

Replying to jhpalmieri:

I'm in the middle of trying this. A few comments:

• pcre: same

This one actually works for me (both locally and on the GitHub runner). Please post the logs if it doesn't work for you

Sorry, it looks okay to me. I must have misread config.log.

[jhpalmieri]

Flint fails to build for me. I ran brew upgrade, cleaned everything out, and tried again, and it still fails.

g++ -std=gnu++11 -fPIC -fno-common -pedantic -Wall -O2 -funroll-loops -g -mpopcnt  -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/var/tmp/sage/build/flint-2.5.2.p5/src -I/usr/local/include -I/usr/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/include -c interfaces/NTL-interface.cpp -o build/interfaces/NTL-interface.lo
ld: illegal thread local variable reference to regular symbol __ZN3NTL8ZZ_pInfoE for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[6]: *** [libflint-13.5.2.dylib] Error 1
make[6]: Target `shared' not remade because of errors.
make[5]: *** [library] Error 2
make[5]: Target `all' not remade because of errors.
make[4]: *** [verbose] Error 2

[mkoeppe]

Is homebrew ntl installed on this system?

[mkoeppe]

(in /usr/local)

[jhpalmieri]

Replying to mkoeppe:

Is homebrew ntl installed on this system?

Yes. Do I need to delete it?

By the way, should prefix be added to .gitignore?

[mkoeppe]

Replying to jhpalmieri:

Replying to mkoeppe:

Is homebrew ntl installed on this system?

Yes. Do I need to delete it?

Please try if this helps. If it does, we should fix something in the FLINT build scripts

By the way, should prefix be added to .gitignore?

Yes, will do.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4c653c2

.gitignore: Add prefix

[jhpalmieri]

Is there a command like make distclean that will get rid of the stuff that Sage has built but keep the homebrew stuff? I don't really want to wait for gcc to build again. (make distclean doesn't touch prefix.)

[mkoeppe]

At the moment, nothing better than rm -rf prefix/*

[jhpalmieri]

Replying to mkoeppe:

Replying to jhpalmieri:

Replying to mkoeppe:

Is homebrew ntl installed on this system?

Yes. Do I need to delete it?

Please try if this helps. If it does, we should fix something in the FLINT build scripts

After doing brew uninstall ntl, flint now builds.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​2625a64	tox.ini [homebrew]: make build
​cf83af6	Add zlib
​14a3dab	tox.ini [local]: Disable testsuites for some packages
​2553b5a	tox.ini [local]: Disable linbox testsuite
​2f79c76	tox.ini: Move homebrew environment settings to script .homebrew-build-env
​77d5c11	build/pkgs/ntl/distros/homebrew.txt: Remove ntl
​4ec80e2	Remove use of homebrew packages flint, arb
​54c803b	build/pkgs/patch/distros/homebrew.txt: New
​2ce6e6c	build/pkgs/boost*/distros/homebrew.txt: New
​467160c	.gitignore: Add prefix

[mkoeppe]

Rebased - no longer on top of #29339.

[jhpalmieri]

Except for the issue with FLINT not building with a pre-existing NTL, it works for me.

[mkoeppe]

Let's fix the FLINT issue on a follow-up ticket.

[mkoeppe]

Replying to mkoeppe:

Replying to jhpalmieri:

Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?

Good idea. WIll do

I'll do this in #29146

[mkoeppe]

Replying to mkoeppe:

Let's fix the FLINT issue on a follow-up ticket.

This issue is probably the same as #27708/#28409.

[mkoeppe]

Ready for review

[dimpase]

Replying to mkoeppe:

Replying to dimpase:

I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.

Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.

One can split building into make built and make ptest, with the latter, much more memory hungry, run with lower value of -j.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dfce7d4

tox.ini [local]: Pass MAKE environment variable, do not default to 'make -j12', change default target to 'build', add doc

[mkoeppe]

OK I've made it more configurable already on this ticket. And I have added documentation

[jhpalmieri]

@gh-mwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.

[dimpase]

I'm 8 timezones away from my MacOS machine.

I'd like more details/pointers on how the Github Actions testing is done - for ​this looks to me like "all sufficiently advanced technology is indistinguishable from magic" kind of thing :-)

[mkoeppe]

This ticket does no GitHub Actions testing. This is all introduced in #29087.

[dimpase]

how does one get the data I linked in comment:126 then?

[mkoeppe]

By merging #29087 and pushing the branch to a GitHub repository.

[mkoeppe]

If the GitHub Actions interface is overwhelming, just download the "Artifacts". They are tar.gz files of build logs.

[dimpase]

Replying to mkoeppe:

By merging #29087 and pushing the branch to a GitHub repository.

yes,but that's how one triggers this - I'm asking rather about how stuff works, inside. Locally and then somehow pushing the logs to GitHub?

[mkoeppe]

I don't push logs to GitHub.

#29087 adds .github/workflows/tox.yml (​https://git.sagemath.org/sage.git/diff/.github/workflows/tox.yml?id=576a18834a16ff05cd6894172a55e86bcc95d407&id2=be1e22c441d9332fd0bf4a3066e1964deddaf8c7) which gets run on every commit pushed to GitHub and on every pull request.

The job tox-local-homebrew-macos defined in that file provisions a macOS host, checks out the branch and runs tox -e local-homebrew-macos-standard there. Then it packages up logs and makes them available as an artifact.

[dimpase]

and where is that MacOS host getting provisioned?

[mkoeppe]

GitHub takes care of that, I think it's using Azure.

[gh-mwageringel]

Replying to jhpalmieri:

@gh-mwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.

I have not tested this locally, but as you have done that, I am happy with this too.

One last question though: As far as I can see, readline is not installed into /usr/local. Why do you not run into conflicts with macOS's system "readline"/libedit, as in #29000?

[mkoeppe]

These tests use their own homebrew installation in a prefix .tox/local-homebrew-macos-standard/homebrew/, and sets up environment variables using the new script .homebrew-build-env. This sets LDFLAGS and CPPFLAGS for the readline installed into opt under the homebrew prefix.

[gh-mwageringel]

The problem in #29000 occured despite passing LDFLAGS and CPPFLAGS to configure, so I guess setting these also during make would avoid that problem, as a workaround.

[mkoeppe]

I think the problem description of #29000 needs some more precision.

[dimpase]

OK, looks good to me.

[mkoeppe]

Thanks everyone!