Opened 5 months ago

Closed 4 months ago

#29104 closed enhancement (fixed)

Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.1
Component: porting Keywords:
Cc: dimpase, jhpalmieri, gh-mwageringel, vbraun, tscrim Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik, John Palmieri
Report Upstream: N/A Work issues:
Branch: dfce7d4 (Commits) Commit: dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc
Dependencies: #29335, #29361 Stopgaps:

Description (last modified by mkoeppe)

#29053 adds tox.ini to support testing the sage distribution in isolated environments via docker.

This ticket adds toxenvs local-homebrew-macos-{minimal,standard} for testing with (best-effort) isolated installations of homebrew on macOS (not using docker).

To test:

    brew install tox
    tox -e local-homebrew-macos-standard

See comments in tox.ini for more examples.

An example run of that is here: https://github.com/mkoeppe/sage/actions/runs/32812271

Change History (141)

comment:1 Changed 5 months ago by mkoeppe

  • Branch set to u/mkoeppe/tox-homebrew

comment:2 Changed 5 months ago by mkoeppe

  • Cc dimpase jhpalmieri gh-mwageringel added
  • Commit set to fca66540d85d7989d84c8551793f029533a8232d

Last 10 new commits:

7e59a6fbuild/bin/write-dockerfile.sh: Add commands to activate conda env
2ad9a97Merge tag '9.1.beta2' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time
9fca230add doc
f6cbb8aadd centos-{7,8}
c651058on centos, IGNORE_MISSING_SYSTEM_PACKAGES=yes
98edff6Add build/pkgs/gfortran/distros/fedora.txt
843ca76build/pkgs/fedora.txt: Add python3
bd6a545tox.ini: Add debian stretch, put sid last
23c6334build/pkgs/openblas/distros/arch.txt: Add lapack, cblas
fca6654tox.ini: Add toxenvs local-homebrew-macos-minimal, local-homebrew-macos-standard

comment:3 Changed 5 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Add to tox.ini a homebrew toxenv (with a fresh install not in /usr/local) to Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local)

comment:4 Changed 5 months ago by git

  • Commit changed from fca66540d85d7989d84c8551793f029533a8232d to e0e8a4c7be40336fa140195a213f9edf3075f6ee

Branch pushed to git repo; I updated commit sha1. New commits:

e0e8a4ctox.ini [local-homebrew]: Fixup PATH, PKG_CONFIG_PATH, set LDFLAGS, CPPFLAGS; [local-direct]: New

comment:5 Changed 5 months ago by git

  • Commit changed from e0e8a4c7be40336fa140195a213f9edf3075f6ee to 16ca88f3423ea20dcb164cd1e7b5bc0283fcec0b

Branch pushed to git repo; I updated commit sha1. New commits:

16ca88ftox.ini: Use {envlogdir}; use ln -sf

comment:6 Changed 5 months ago by git

  • Commit changed from 16ca88f3423ea20dcb164cd1e7b5bc0283fcec0b to 6a1803dae669007dcc7005448a1e578bcc950656

Branch pushed to git repo; I updated commit sha1. New commits:

6a1803dtox.ini [local]: First build the base-toolchain

comment:7 Changed 5 months ago by git

  • Commit changed from 6a1803dae669007dcc7005448a1e578bcc950656 to 4b72987e58ff284f57f04d4fbb5fed8f39beb36d

Branch pushed to git repo; I updated commit sha1. New commits:

4b72987tox.ini [local-homebrew]: Set PATH earlier and set ACLOCAL_PATH for bootstrapping

comment:8 Changed 5 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Description modified (diff)
  • Status changed from new to needs_review

comment:9 Changed 5 months ago by mkoeppe

  • Cc vbraun added

comment:10 Changed 5 months ago by dimpase

Is this ticket MacOS-specific?

comment:11 Changed 5 months ago by mkoeppe

Hmm, I haven’t tried linuxbrew. Might be worth checking

comment:12 Changed 5 months ago by dimpase

I'll fire up my vintage OSX 10.13 macbookair...

comment:13 Changed 5 months ago by git

  • Commit changed from 4b72987e58ff284f57f04d4fbb5fed8f39beb36d to 2141b7536c5b99a26d16568dcc0d734518bebc58

Branch pushed to git repo; I updated commit sha1. New commits:

2141b75Merge tag '9.1.beta3' into t/29104/tox-homebrew

comment:14 Changed 5 months ago by dimpase

I tried this:

tox -p auto -e local-homebrew-macos-standard

and got

...
[sagelib-9.1.beta2] user	0m10.244s
[sagelib-9.1.beta2] sys	0m3.812s

real	0m16.288s
user	0m10.956s
sys	0m4.047s
Sage build/upgrade complete!
✔ OK local-homebrew-macos-standard in 1 minute, 14.362 seconds
_________________________________________________________________________________________________ summary __________________________________________________________________________________________________
  local-homebrew-macos-standard: commands succeeded
  congratulations :)

not tests were run, it seems, is this normal?

comment:15 Changed 5 months ago by mkoeppe

The default packages set in tox.ini are only fflas_ffpack scipy r, and those (and their dependencies) are checked. I don't know why it's actually building sagelib, but I hope to fix bizarre build behavior like this in #29113.

comment:16 Changed 5 months ago by mkoeppe

  • Dependencies #29053 deleted

comment:17 Changed 5 months ago by git

  • Commit changed from 2141b7536c5b99a26d16568dcc0d734518bebc58 to 4eb201b75f75fbbe526b84de9b52f2ba0ea3ef0e

Branch pushed to git repo; I updated commit sha1. New commits:

4eb201bMerge tag '9.1.beta4' into t/29104/tox-homebrew

comment:18 Changed 4 months ago by mkoeppe

  • Status changed from needs_review to needs_work
  • Work issues set to rebase

comment:19 Changed 4 months ago by git

  • Commit changed from 4eb201b75f75fbbe526b84de9b52f2ba0ea3ef0e to 7155e7993248fcf6db057e46779ab08cd1670833

Branch pushed to git repo; I updated commit sha1. New commits:

7155e79Merge tag '9.1.beta7' into t/29104/tox-homebrew

comment:20 Changed 4 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:21 Changed 4 months ago by git

  • Commit changed from 7155e7993248fcf6db057e46779ab08cd1670833 to d7a2e726c10e58fac83b7c419973e53f92e9444f

Branch pushed to git repo; I updated commit sha1. New commits:

d7a2e72Fixup merge

comment:22 Changed 4 months ago by mkoeppe

  • Work issues rebase deleted

comment:23 Changed 4 months ago by git

  • Commit changed from d7a2e726c10e58fac83b7c419973e53f92e9444f to 9e9f4c7f852c7b6f18d44e950a84317bd1de9c6b

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

3103055m4ri must use png on gentoo
1b48aa1Merge tag '9.1.beta7' into t/29273/packages/r_packages_in_distros
0fc301cAdd build/pkgs/{libgd|m4ri|ninja_build|perl_term_readline_gnu|yasm}/distros/*.txt - from repology
f7785cebuild/pkgs/fedora.txt: Add diffutils
6d2bbd8build/pkgs/arch.txt: Add 'which'
0aa90c1build/pkgs/debian.txt: Add ca-certificates
c49acdcMerge tag '9.1.beta7' into t/29129/add_more_system_packages_to_build_pkgs_spkg_distros_
0f704d0Merge branch 't/29129/add_more_system_packages_to_build_pkgs_spkg_distros_' into t/29273/packages/r_packages_in_distros
f8097c4Merge branch 'u/mkoeppe/packages/r_packages_in_distros' of git://trac.sagemath.org/sage into t/29104/tox-homebrew
9e9f4c7Add distros/homebrew.txt for many packages, and some more distros/*.txt

comment:24 Changed 4 months ago by mkoeppe

  • Dependencies set to #29273

comment:25 Changed 4 months ago by mkoeppe

  • Summary changed from Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local) to Add to tox.ini some homebrew toxenvs (with a fresh install not in /usr/local), add homebrew packages

comment:26 Changed 4 months ago by gh-mwageringel

  • mpc should be libmpc
  • tachyon does not seem to be the package we use
  • pari cannot currently be used by Sage because the pari packages are not in Homebrew
  • glpk 4.65 should not be used by Sage as long as it is not patched by Homebrew, see #24824

I also have these Homebrew packages installed:

cmake freetype gcc git libpng pcre pkg-config openssl xz zeromq

Some of these might already be installed automatically as dependencies of others.

Additionally there are:

sagemath/science/flint sagemath/science/arb

Though, I am not sure about how flint in homebrew-core is different from this one.

comment:27 Changed 4 months ago by git

  • Commit changed from 9e9f4c7f852c7b6f18d44e950a84317bd1de9c6b to e8d1fef3818fd1d78f83df31ae237061ea089f92

Branch pushed to git repo; I updated commit sha1. New commits:

e8d1fefAdd build/pkgs/homebrew*.txt

comment:28 Changed 4 months ago by git

  • Commit changed from e8d1fef3818fd1d78f83df31ae237061ea089f92 to 242f684eac3c9a71e9562c56554a0b2cdf9e297e

Branch pushed to git repo; I updated commit sha1. New commits:

242f684Add/fix build/pkgs/*/homebrew.txt - from gh-mwageringel

comment:29 follow-up: Changed 4 months ago by dimpase

https://github.com/sagemath/homebrew-science/commits/master

adds NTL support for flint, and also ensbles TLS.

we can cook up more of these formulae, theain obstacle for me there was that they ate in Ruby.


New commits:

242f684Add/fix build/pkgs/*/homebrew.txt - from gh-mwageringel

comment:30 follow-up: Changed 4 months ago by mkoeppe

==> Installing sagemath/science/arb
==> Downloading https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz
##O=#  #                                                                      
curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "arb"
Download failed: https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz
Warning: Bottle installation failed: building from source.
==> Downloading https://github.com/fredrik-johansson/arb/archive/2.16.0.tar.gz
==> Downloading from https://codeload.github.com/fredrik-johansson/arb/tar.gz/2.16.0
######################################################################## 100.0%

comment:31 Changed 4 months ago by git

  • Commit changed from 242f684eac3c9a71e9562c56554a0b2cdf9e297e to 2449c395aad510ffc430416a06f2182b6dc0ba7c

Branch pushed to git repo; I updated commit sha1. New commits:

2449c39tox.ini: Get homebrew packages from *homebrew*.txt

comment:32 in reply to: ↑ 30 Changed 4 months ago by dimpase

Replying to mkoeppe:

==> Installing sagemath/science/arb
==> Downloading https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz
##O=#  #                                                                      
curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "arb"
Download failed: https://homebrew.bintray.com/bottles-science/arb-2.16.0.sierra.bottle.tar.gz
Warning: Bottle installation failed: building from source.
==> Downloading https://github.com/fredrik-johansson/arb/archive/2.16.0.tar.gz
==> Downloading from https://codeload.github.com/fredrik-johansson/arb/tar.gz/2.16.0
######################################################################## 100.0%

yes, this is expected. No binaries.

comment:33 Changed 4 months ago by git

  • Commit changed from 2449c395aad510ffc430416a06f2182b6dc0ba7c to 188994aa353dd4533f701aa986a24c1ac2cad860

Branch pushed to git repo; I updated commit sha1. New commits:

188994atox.ini: Use brew upgrade after brew install

comment:34 Changed 4 months ago by git

  • Commit changed from 188994aa353dd4533f701aa986a24c1ac2cad860 to 6df5104b1ed9ad4e41857cfd522ab31170b95268

Branch pushed to git repo; I updated commit sha1. New commits:

6df5104tox.ini [local-homebrew]: Add to PKG_CONFIG_PATH

comment:35 Changed 4 months ago by mkoeppe

Checking whether SageMath should install SPKG openblas...
checking BLAS library... openblas
checking whether any of gfortran is installed or will be installed as SPKG... no
checking for openblas >= 0.2.20... yes
checking for library containing cblas_dgemm... no
checking for cblas... no
checking for library containing ... no
checking for lapack... no
configure: no suitable system package found for SPKG openblas

The tests for cblas_dgemm etc. do not seem to use OPENBLAS_LIBS at all!

comment:36 in reply to: ↑ 29 ; follow-up: Changed 4 months ago by gh-mwageringel

Replying to dimpase:

https://github.com/sagemath/homebrew-science/commits/master

adds NTL support for flint, and also ensbles TLS.

we can cook up more of these formulae, theain obstacle for me there was that they ate in Ruby.

There seems to be a conflict between flint and sagemath/science/flint, as sagemath/science/arb installs flint from homebrew-core as a dependency (which has been added to Homebrew very recently). I am using

brew install sagemath/science/flint
brew install --ignore-dependencies sagemath/science/arb

as a workaround now, but the arb formula probably needs to be updated.

comment:37 Changed 4 months ago by git

  • Commit changed from 6df5104b1ed9ad4e41857cfd522ab31170b95268 to c9d5564c2b4cc4b68748c370211956984d2c986e

Branch pushed to git repo; I updated commit sha1. New commits:

c9d5564build/pkgs/openblas/spkg-configure.m4: Use OPENBLAS_{LIBS,CFLAGS} while checking for cblas/lapack functions

comment:38 Changed 4 months ago by mkoeppe

This line (introduced in #29071):

        AC_FC_FREEFORM([AC_FC_FUNC([dgeqrf])])

is not executed at all because AC_FC_FREEFORM is AC_DEFUN_ONCE and is called already in gfortran/spkg-configure.m4

As a result:

Checking whether SageMath should install SPKG openblas...
checking BLAS library... openblas
checking whether any of gfortran is installed or will be installed as SPKG... no
checking for openblas >= 0.2.20... yes
checking for cblas_dgemm... yes
checking for ... no
checking for lapack... no
configure: no suitable system package found for SPKG openblas

comment:39 follow-up: Changed 4 months ago by git

  • Commit changed from c9d5564c2b4cc4b68748c370211956984d2c986e to 52ef59ed13d878e1a0de0bec999a9d23a442ca6f

Branch pushed to git repo; I updated commit sha1. New commits:

52ef59ebuild/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC.

comment:40 Changed 4 months ago by mkoeppe

Not accepted by configure yet: glpk, r

comment:41 Changed 4 months ago by mkoeppe

(resolved)

Last edited 4 months ago by mkoeppe (previous) (diff)

comment:42 in reply to: ↑ 39 Changed 4 months ago by dimpase

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

52ef59ebuild/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC.

why? If AC_FC_FREEFORM can only be called once, we can move the call to AC_FC_FUNC to the very place it's called. How about

  • build/pkgs/gfortran/spkg-configure.m4

    a b SAGE_SPKG_CONFIGURE([gfortran], [ 
    66    # This helps verify the compiler works too, so if some idiot sets FC to
    77    # /usr/bin/ls, we will at least know it's not a working Fortran
    88    # compiler.
    9     AC_FC_FREEFORM([SAGE_HAVE_FC_FREEFORM=yes], [
     9    AC_FC_FREEFORM([
     10        SAGE_HAVE_FC_FREEFORM=yes
     11        AC_FC_FUNC([dgeqrf])
     12       ], [
    1013       AC_MSG_NOTICE([Your Fortran compiler does not accept free-format source code])
    1114        AC_MSG_NOTICE([which means the compiler is either seriously broken, or])
    1215        AC_MSG_NOTICE([is too old to build Sage.])
  • build/pkgs/openblas/spkg-configure.m4

    diff --git a/build/pkgs/openblas/spkg-configure.m4 b/build/pkgs/openblas/spkg-configure.m4
    index c37db649be..ecc7540ee9 100644
    a b SAGE_SPKG_CONFIGURE([openblas], [ 
    99             dnl openblas does not work as cblas; try to use system's cblas as is
    1010             PKG_CHECK_MODULES([CBLAS], [cblas], [], [sage_spkg_install_openblas=yes])
    1111          ])
    12        AC_FC_FREEFORM([AC_FC_FUNC([dgeqrf])])
    1312       AC_SEARCH_LIBS([$dgeqrf], [openblas], [dnl openblas works as lapack
    1413             sage_install_lapack_pc=yes
    1514             ], [

comment:43 follow-up: Changed 4 months ago by mkoeppe

OK... let me try

comment:44 in reply to: ↑ 43 Changed 4 months ago by dimpase

Replying to mkoeppe:

OK... let me try

it appears to work with gfortan etc installed:

-----------------------------------------------------------------------------
Checking whether SageMath should install SPKG openblas...
checking BLAS library... openblas
checking whether any of gfortran is installed or will be installed as SPKG... no
checking for OPENBLAS... yes
checking for library containing cblas_dgemm... -lopenblas
checking for library containing dgeqrf_... none required
configure: will use system package and not install SPKG openblas

but perhaps it would be throwing up a surprise if there wasn't any.

comment:45 Changed 4 months ago by git

  • Commit changed from 52ef59ed13d878e1a0de0bec999a9d23a442ca6f to 9dd37760062337ca4e97263c713d8a46880644b8

Branch pushed to git repo; I updated commit sha1. New commits:

f049b9bRevert "build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC."
9dd3776Move AC_FC_FUNC([dgeqrf]) from openblas/spkg-configure.m4 to gfortran/spkg-configure.m4

comment:46 Changed 4 months ago by git

  • Commit changed from 9dd37760062337ca4e97263c713d8a46880644b8 to 886ca2f0392764055804e44597dc38125bd93e9c

Branch pushed to git repo; I updated commit sha1. New commits:

886ca2ftox.ini [homebrew]: make build

comment:47 Changed 4 months ago by mkoeppe

matplotlib install fails:

    REQUIRED DEPENDENCIES AND EXTENSIONS
                     numpy: yes [not found. pip may install it below.]
          install_requires: yes [handled by setuptools]
                    libagg: yes [pkg-config information for 'libagg' could not
                            be found. Using local copy.]
                  freetype: yes [version 2.10.1]
                       png: no  [The C/C++ header for libpng (png.h) could not
                            be found.  You may need to install the development
                            package.]
                     qhull: yes [pkg-config information for 'libqhull' could not
                            be found. Using local copy.]
...
                            * The following required packages can not be built:
                            * png
                            * Try installing png with `brew install libpng` and
                            * pkg-config with `brew install pkg-config`
Cleaning up...

comment:48 Changed 4 months ago by mkoeppe

cysignals testsuite fails:

  [cysignals-1.10.2]   ulimit 2>/dev/null -s 1024; sage-python23 -B rundoctests.py src/cysignals/*.pyx
  [cysignals-1.10.2]   Doctesting 5 files.
  [cysignals-1.10.2]   Traceback (most recent call last):
  [cysignals-1.10.2]     File "rundoctests.py", line 74, in <module>
  [cysignals-1.10.2]       resource.setrlimit(resource.RLIMIT_STACK, (stacksize, stacksize))
  [cysignals-1.10.2]   ValueError: current limit exceeds maximum limit
  [cysignals-1.10.2]   make[4]: *** [check-doctest] Error 1
  [cysignals-1.10.2]   cd example && sage-python23 setup.py clean build

comment:49 Changed 4 months ago by git

  • Commit changed from 886ca2f0392764055804e44597dc38125bd93e9c to 185dbbb346e7bea59d6108e2ca8e3b33cb734d03

Branch pushed to git repo; I updated commit sha1. New commits:

fdeced3Add zlib
185dbbbtox.ini [local]: Disable testsuites for some packages

comment:50 Changed 4 months ago by git

  • Commit changed from 185dbbb346e7bea59d6108e2ca8e3b33cb734d03 to 6750bf1d0bbe58b4ec5526af2fb50361e489a42e

Branch pushed to git repo; I updated commit sha1. New commits:

6750bf1tox.ini [local]: Disable linbox testsuite

comment:51 Changed 4 months ago by mkoeppe

Ready for review.

comment:52 Changed 4 months ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:53 Changed 4 months ago by mkoeppe

The AC_FC_FUNC business broke all -minimal targets (https://github.com/mkoeppe/sage/actions/runs/54084841)

Last edited 4 months ago by mkoeppe (previous) (diff)

comment:54 Changed 4 months ago by git

  • Commit changed from 6750bf1d0bbe58b4ec5526af2fb50361e489a42e to 071e0b33470bc6fc13296052a831d84d85d4a530

Branch pushed to git repo; I updated commit sha1. New commits:

9f486efRevert "Move AC_FC_FUNC([dgeqrf]) from openblas/spkg-configure.m4 to gfortran/spkg-configure.m4"
071e0b3Revert "Revert "build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC.""

comment:56 Changed 4 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:57 Changed 4 months ago by mkoeppe

Needs review

comment:58 Changed 4 months ago by mkoeppe

Both homebrew-macos-minimal (logs: https://github.com/mkoeppe/sage/suites/515691370/artifacts/2830410) and homebrew-macos-standard (logs: https://github.com/mkoeppe/sage/suites/515691370/artifacts/2830411) now build correctly all the way to sagelib.

comment:59 follow-up: Changed 4 months ago by mkoeppe

Some doctest failures similar to this:

File "src/sage/structure/element.pyx", line 3945, in sage.structure.element.EuclideanDomainElement._mod_
Failed example:
    cython('''
    from sage.structure.element cimport EuclideanDomainElement
    cdef class MyElt(EuclideanDomainElement):
        def quo_rem(self, other):
            return self._parent.var('quo,rem')
    ''')
Expected nothing
Got:
    ld: warning: directory not found for option '-L/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-minimal/homebrew/opt/readline/lib'

comment:60 Changed 4 months ago by git

  • Commit changed from 071e0b33470bc6fc13296052a831d84d85d4a530 to d1eb116499937578c84f3b27097bf4d3f4194b2f

Branch pushed to git repo; I updated commit sha1. New commits:

d1eb116tox.ini: Move homebrew environment settings to script .homebrew-build-env

comment:61 in reply to: ↑ 59 Changed 4 months ago by mkoeppe

Replying to mkoeppe:

Some doctest failures similar to this: [...readline...]

These are now fixed. Very few remaining doctest failures and timeouts for homebrew-macos-minimal (https://github.com/mkoeppe/sage/runs/508217498):

sage -t src/sage/interfaces/gap.py  # 1 doctest failed
sage -t src/sage/manifolds/differentiable/degenerate_submanifold.py  # Timed out
sage -t src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t src/sage/plot/plot3d/base.pyx  # Timed out
sage -t src/sage/plot/plot3d/implicit_plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/parametric_plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/shapes.pyx  # Timed out
sage -t src/sage/rings/padics/padic_lattice_element.py  # 3 doctests failed
sage -t src/sage/plot/plot3d/shapes2.py  # Timed out
sage -t src/sage/rings/function_field/function_field.py  # Timed out
sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # Timed out

homebrew-macos-standard (https://github.com/mkoeppe/sage/runs/508217500) has many more doctest failures. Looks like there is some problem with singular. Some messages from singular: univariate factorization depends on NTL(missing)

In that build, Sage is using the homebrew NTL:

Checking whether SageMath should install SPKG ntl...
checking installing gmp/mpir? ... no
checking NTL/ZZ.h usability... yes
checking NTL/ZZ.h presence... yes
checking for NTL/ZZ.h... yes
checking whether we can link a program using NTL... yes
checking NTL version >= 10.3... 11.4.3
configure: will use system package and not install SPKG ntl
using ntl library from the system

... but Singular does not find it ....

checking for NTL >= 5.0... not found
configure: WARNING: Unable to find NTL (which is strongly recommended) on your machine: please use --with-ntl=PATH_TO_DIR_CONTAINING_LIB_AND_INCLUDE (see also ./configure --help if you do not understand what we are talking about)
Last edited 4 months ago by mkoeppe (previous) (diff)

comment:62 Changed 4 months ago by mkoeppe

  • Dependencies changed from #29273 to #29273, #29335

comment:63 Changed 4 months ago by git

  • Commit changed from d1eb116499937578c84f3b27097bf4d3f4194b2f to 6410c1e02cf69213efe2e888bd63c1d1856cc135

Branch pushed to git repo; I updated commit sha1. New commits:

50f62c7build/pkgs/singular: Add patches to accept system NTL in nonstandard locations
6410c1eMerge branch 't/29335/always_find_sage_ntl_prefix_and_call_singular_configure_with_it' into t/29104/tox-homebrew

comment:65 Changed 4 months ago by mkoeppe

With #29335 merged, Singular finds homebrew's NTL but then I get the following linker error (https://github.com/mkoeppe/sage/runs/508693083):

  [singular-4.1.1p2.p0]   ld: illegal thread local variable reference to regular symbol __ZN3NTL13ErrorCallbackE for architecture x86_64
  [singular-4.1.1p2.p0]   clang: error: linker command failed with exit code 1 (use -v to see invocation)
  [singular-4.1.1p2.p0]   make[8]: *** [libSingular.la] Error 1

comment:66 follow-up: Changed 4 months ago by dimpase

this is about absence of coherence on TLS between Singular and Flint.

comment:67 in reply to: ↑ 66 Changed 4 months ago by mkoeppe

Replying to dimpase:

this is about absence of coherence on TLS between Singular and Flint.

OK, is there a way to enable use of TLS with Singular?

comment:68 Changed 4 months ago by git

  • Commit changed from 6410c1e02cf69213efe2e888bd63c1d1856cc135 to ec5cd78603542b57f58f4ae0d43ba41151a55ab6

Branch pushed to git repo; I updated commit sha1. New commits:

96553dbbuild/pkgs/ntl/spkg-configure.m4: Check for NTL configuration
ec5cd78Merge branch 't/29339/fix_ntl_spkg_configure_m4_so_it_rejects_ntls_built_with_ntl_threads__without_ntl_gmp_lip__without_ntl_gf2x_lib' into t/29104/tox-homebrew

comment:69 Changed 4 months ago by mkoeppe

  • Dependencies changed from #29273, #29335 to #29273, #29335, #29339

comment:70 follow-up: Changed 4 months ago by mkoeppe

Build logs for homebrew-macos-standard at https://github.com/mkoeppe/sage/runs/511533264

comment:71 Changed 4 months ago by git

  • Commit changed from ec5cd78603542b57f58f4ae0d43ba41151a55ab6 to 6d2f6e1fc67097613f08f326a43a6749073e7bd0

Branch pushed to git repo; I updated commit sha1. New commits:

c3bc092Remove tests for NTL_GMP_LIP, NTL_GF2X_LIB
6d2f6e1Merge branch 't/29339/fix_ntl_spkg_configure_m4_so_it_rejects_ntls_built_with_ntl_threads__without_ntl_gmp_lip__without_ntl_gf2x_lib' into t/29104/tox-homebrew

comment:72 in reply to: ↑ 70 Changed 4 months ago by mkoeppe

Replying to mkoeppe:

Build logs for homebrew-macos-standard at https://github.com/mkoeppe/sage/runs/511533264

  [sagelib-9.1.beta7]   ld: illegal thread local variable reference to regular symbol __ZN3NTL20ZZXFac_InitNumPrimesE for architecture x86_64
  [sagelib-9.1.beta7]   clang: error: linker command failed with exit code 1 (use -v to see invocation)

comment:73 Changed 4 months ago by mkoeppe

I guess this means that another library is picking up the system installation of NTL...

comment:74 Changed 4 months ago by git

  • Commit changed from 6d2f6e1fc67097613f08f326a43a6749073e7bd0 to 14e9f1270e7706e9d771ff33efe259e135fa9ec4

Branch pushed to git repo; I updated commit sha1. New commits:

14e9f12build/pkgs/ntl/distros/homebrew.txt: Remove ntl

comment:76 in reply to: ↑ 36 Changed 4 months ago by gh-mwageringel

Replying to gh-mwageringel:

There seems to be a conflict between flint and sagemath/science/flint, as sagemath/science/arb installs flint from homebrew-core as a dependency (which has been added to Homebrew very recently). I am using

brew install sagemath/science/flint
brew install --ignore-dependencies sagemath/science/arb

as a workaround now, but the arb formula probably needs to be updated.

As can be seen in the build logs above, sagemath/science/arb installs flint from homebrew-core as a dependency. Later, sagemath/science/flint is installed only because of brew upgrade. This only seems to work because of the coincidence that sagemath/science/flint has a higher version number than flint, currently. However, sagemath/science/arb is only compiled with flint (without ntl support) – it is not recompiled after installing sagemath/science/flint.

I have opened a pull request to resolve this.

comment:77 in reply to: ↑ 75 Changed 4 months ago by gh-mwageringel

Replying to mkoeppe:

New test: https://github.com/mkoeppe/sage/runs/512132585

If you remove ntl, you probably also need to remove flint and arb, as they depend on it.

comment:78 Changed 4 months ago by git

  • Commit changed from 14e9f1270e7706e9d771ff33efe259e135fa9ec4 to 70e161bdd3f29ae1b25dffab08829a20e2c53e5c

Branch pushed to git repo; I updated commit sha1. New commits:

70e161bRemove use of homebrew packages flint, arb

comment:79 Changed 4 months ago by mkoeppe

Thank you!

comment:81 Changed 4 months ago by dimpase

I had no idea that a ticket about homebrew would be touching fedora-related stuff, and in this I essentially created a duplicate of that work on #29352

It's much better to create separate tickets for unrelated problems - not the least as it gives them a chance of being discovered.

Last edited 4 months ago by dimpase (previous) (diff)

comment:82 Changed 4 months ago by mkoeppe

OK. What can I do to facilitate review?

comment:83 Changed 4 months ago by dimpase

I cannot review MacOS-related things - I don't have access to the needed OS. But I would be happy to look at the other things that got silently bundled here.

comment:84 follow-up: Changed 4 months ago by mkoeppe

OK, then let me move the fedora.txt files to #29352 and the other linux .txt files to a new ticket.

comment:85 in reply to: ↑ 84 Changed 4 months ago by mkoeppe

Replying to mkoeppe:

OK, then let me move the fedora.txt files to #29352 and the other linux .txt files to a new ticket.

Actually, as noted on #29352, most of the files were added in #29273.

comment:86 follow-up: Changed 4 months ago by mkoeppe

@gh-mwageringel Would you be able to review this ticket?

comment:87 Changed 4 months ago by mkoeppe

  • Cc tscrim added
  • Description modified (diff)

comment:88 in reply to: ↑ 86 ; follow-up: Changed 4 months ago by gh-mwageringel

Replying to mkoeppe:

@gh-mwageringel Would you be able to review this ticket?

Yes, I will look into that. Though, I only have access to macOS 10.13.6.

comment:89 Changed 4 months ago by mkoeppe

Tests ran at https://github.com/mkoeppe/sage/actions/runs/57924835

local-homebrew-macos-minimal (https://github.com/mkoeppe/sage/runs/515508463) and local-homebrew-macos-standard (https://github.com/mkoeppe/sage/runs/515508474):

All packages (except for experimental package polymake) build, doctests:

----------------------------------------------------------------------
sage -t src/sage/game_theory/normal_form_game.py  # 5 doctests failed
sage -t src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # Timed out
sage -t src/sage/interfaces/gap.py  # 1 doctest failed
sage -t src/sage/knots/link.py  # 1 doctest failed
sage -t src/sage/manifolds/differentiable/degenerate_submanifold.py  # Timed out
sage -t src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t src/sage/plot/plot3d/base.pyx  # Timed out
sage -t src/sage/plot/plot3d/implicit_plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/parametric_plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/plot3d.py  # Timed out
sage -t src/sage/plot/plot3d/shapes.pyx  # Timed out
sage -t src/sage/rings/padics/padic_lattice_element.py  # 3 doctests failed
sage -t src/sage/plot/plot3d/shapes2.py  # Timed out
sage -t src/sage/rings/function_field/function_field.py  # Timed out
sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # Timed out
----------------------------------------------------------------------

comment:90 in reply to: ↑ 88 Changed 4 months ago by mkoeppe

Replying to gh-mwageringel:

Replying to mkoeppe:

@gh-mwageringel Would you be able to review this ticket?

Yes, I will look into that.

Thanks!

Though, I only have access to macOS 10.13.6.

That's actually great, for some broader coverage. I'm testing on 10.15.3, and also the GitHub runners are on Catalina.

comment:91 Changed 4 months ago by dimpase

lots of timeouts in tests, probably an overloaded host, or too aggressive parallelisation.

comment:92 Changed 4 months ago by mkoeppe

Yes, this is using 12 threads

comment:93 follow-up: Changed 4 months ago by jhpalmieri

I'm in the middle of trying this. A few comments:

  • patch: this should use homebrew's patch
  • pcre: same
  • boost: is there a way to convince Sage to use homebrew's boost as a replacement for boost-cropped? (See #29100?)
  • ntl: being discussed on another ticket, but my Sage builds on OS X regularly use homebrew's ntl without problems.

comment:94 follow-up: Changed 4 months ago by jhpalmieri

Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?

comment:95 follow-up: Changed 4 months ago by dimpase

I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.

comment:96 follow-up: Changed 4 months ago by jhpalmieri

ntl failed to build. This is on OS X 10.15.3.

make setup3
g++ -std=gnu++11 -I../include -I.  -O2 -g  -fno-common -march=native -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/homebrew/opt/readline/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/homebrew/lib  -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/include  -o gen_gmp_aux gen_gmp_aux.cpp -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/lib  -lgmp  -lm
./gen_gmp_aux > ../include/NTL/gmp_aux.h 
NTL_GMP_LIP flag set
GMP version check (6.1.2/6.2.0)
*** version number mismatch: inconsistency between gmp.h and libgmp
/bin/sh: line 1: 18441 Abort trap: 6           ./gen_gmp_aux > ../include/NTL/gmp_aux.h
make[5]: *** [setup3] Error 134
make[4]: *** [setup-phase] Error 2
make[4]: Target `all' not remade because of errors.

comment:97 in reply to: ↑ 96 Changed 4 months ago by mkoeppe

Replying to jhpalmieri:

ntl failed to build. This is on OS X 10.15.3.

This happens when you have a different (older) version of gmp in your /usr/local homebrew. The isolation is unfortunately not perfect. Badly written configure scripts of packages pick up stuff from /usr/local

comment:98 in reply to: ↑ 94 ; follow-up: Changed 4 months ago by mkoeppe

Replying to jhpalmieri:

Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?

Good idea. WIll do

comment:99 in reply to: ↑ 95 ; follow-up: Changed 4 months ago by mkoeppe

Replying to dimpase:

I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.

Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.

comment:100 Changed 4 months ago by mkoeppe

  • Dependencies changed from #29273, #29335, #29339 to #29273, #29335, #29339, #29361

comment:101 Changed 4 months ago by git

  • Commit changed from 70e161bdd3f29ae1b25dffab08829a20e2c53e5c to 11f77293e372d29e01b6525e1fb1948c4d855f04

Branch pushed to git repo; I updated commit sha1. New commits:

11f7729build/pkgs/patch/distros/homebrew.txt: New

comment:102 in reply to: ↑ 93 ; follow-up: Changed 4 months ago by mkoeppe

Replying to jhpalmieri:

I'm in the middle of trying this. A few comments:

  • pcre: same

This one actually works for me (both locally and on the GitHub runner). Please post the logs if it doesn't work for you

comment:103 Changed 4 months ago by git

  • Commit changed from 11f77293e372d29e01b6525e1fb1948c4d855f04 to 93a0421c7789a1ac278386f9643d87b9768a5e69

Branch pushed to git repo; I updated commit sha1. New commits:

93a0421build/pkgs/boost*/distros/homebrew.txt: New

comment:104 in reply to: ↑ 102 Changed 4 months ago by jhpalmieri

Replying to mkoeppe:

Replying to jhpalmieri:

I'm in the middle of trying this. A few comments:

  • pcre: same

This one actually works for me (both locally and on the GitHub runner). Please post the logs if it doesn't work for you

Sorry, it looks okay to me. I must have misread config.log.

comment:105 Changed 4 months ago by jhpalmieri

Flint fails to build for me. I ran brew upgrade, cleaned everything out, and tried again, and it still fails.

g++ -std=gnu++11 -fPIC -fno-common -pedantic -Wall -O2 -funroll-loops -g -mpopcnt  -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/var/tmp/sage/build/flint-2.5.2.p5/src -I/usr/local/include -I/usr/local/include -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.beta7/.tox/local-homebrew-macos-standard/local/include -c interfaces/NTL-interface.cpp -o build/interfaces/NTL-interface.lo
ld: illegal thread local variable reference to regular symbol __ZN3NTL8ZZ_pInfoE for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[6]: *** [libflint-13.5.2.dylib] Error 1
make[6]: Target `shared' not remade because of errors.
make[5]: *** [library] Error 2
make[5]: Target `all' not remade because of errors.
make[4]: *** [verbose] Error 2

comment:106 follow-up: Changed 4 months ago by mkoeppe

Is homebrew ntl installed on this system?

comment:107 Changed 4 months ago by mkoeppe

(in /usr/local)

comment:108 in reply to: ↑ 106 ; follow-up: Changed 4 months ago by jhpalmieri

Replying to mkoeppe:

Is homebrew ntl installed on this system?

Yes. Do I need to delete it?

By the way, should prefix be added to .gitignore?

comment:109 in reply to: ↑ 108 ; follow-up: Changed 4 months ago by mkoeppe

Replying to jhpalmieri:

Replying to mkoeppe:

Is homebrew ntl installed on this system?

Yes. Do I need to delete it?

Please try if this helps. If it does, we should fix something in the FLINT build scripts

By the way, should prefix be added to .gitignore?

Yes, will do.

comment:110 Changed 4 months ago by git

  • Commit changed from 93a0421c7789a1ac278386f9643d87b9768a5e69 to 4c653c2759eb404f9a51baf52c260468276ea720

Branch pushed to git repo; I updated commit sha1. New commits:

4c653c2.gitignore: Add prefix

comment:111 Changed 4 months ago by jhpalmieri

Is there a command like make distclean that will get rid of the stuff that Sage has built but keep the homebrew stuff? I don't really want to wait for gcc to build again. (make distclean doesn't touch prefix.)

Last edited 4 months ago by jhpalmieri (previous) (diff)

comment:112 Changed 4 months ago by mkoeppe

At the moment, nothing better than rm -rf prefix/*

comment:113 in reply to: ↑ 109 Changed 4 months ago by jhpalmieri

Replying to mkoeppe:

Replying to jhpalmieri:

Replying to mkoeppe:

Is homebrew ntl installed on this system?

Yes. Do I need to delete it?

Please try if this helps. If it does, we should fix something in the FLINT build scripts

After doing brew uninstall ntl, flint now builds.

comment:114 Changed 4 months ago by git

  • Commit changed from 4c653c2759eb404f9a51baf52c260468276ea720 to 467160cdc8402124b21c5dee60386dee03f18b67

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

2625a64tox.ini [homebrew]: make build
cf83af6Add zlib
14a3dabtox.ini [local]: Disable testsuites for some packages
2553b5atox.ini [local]: Disable linbox testsuite
2f79c76tox.ini: Move homebrew environment settings to script .homebrew-build-env
77d5c11build/pkgs/ntl/distros/homebrew.txt: Remove ntl
4ec80e2Remove use of homebrew packages flint, arb
54c803bbuild/pkgs/patch/distros/homebrew.txt: New
2ce6e6cbuild/pkgs/boost*/distros/homebrew.txt: New
467160c.gitignore: Add prefix

comment:115 Changed 4 months ago by mkoeppe

  • Dependencies changed from #29273, #29335, #29339, #29361 to #29335, #29361

Rebased - no longer on top of #29339.

comment:116 Changed 4 months ago by jhpalmieri

Except for the issue with FLINT not building with a pre-existing NTL, it works for me.

comment:117 follow-up: Changed 4 months ago by mkoeppe

Let's fix the FLINT issue on a follow-up ticket.

comment:118 in reply to: ↑ 98 Changed 4 months ago by mkoeppe

Replying to mkoeppe:

Replying to jhpalmieri:

Defaulting to 12 threads seems like a bad idea. Shouldn't the user be able to configure this?

Good idea. WIll do

I'll do this in #29146

comment:119 in reply to: ↑ 117 Changed 4 months ago by mkoeppe

Replying to mkoeppe:

Let's fix the FLINT issue on a follow-up ticket.

This issue is probably the same as #27708/#28409.

comment:120 Changed 4 months ago by mkoeppe

Ready for review

comment:121 in reply to: ↑ 99 Changed 4 months ago by dimpase

Replying to mkoeppe:

Replying to dimpase:

I presume Github Actions run on a real Mac hardware, and 12 threads is probably only meaningful on a rather high end host, used exclusively.

Well, for the build I'm using high parallelization on purpose in order to catch possible parallelization bugs in the scripts.

One can split building into make built and make ptest, with the latter, much more memory hungry, run with lower value of -j.

comment:122 Changed 4 months ago by git

  • Commit changed from 467160cdc8402124b21c5dee60386dee03f18b67 to dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc

Branch pushed to git repo; I updated commit sha1. New commits:

dfce7d4tox.ini [local]: Pass MAKE environment variable, do not default to 'make -j12', change default target to 'build', add doc

comment:123 Changed 4 months ago by mkoeppe

OK I've made it more configurable already on this ticket. And I have added documentation

comment:124 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:125 follow-up: Changed 4 months ago by jhpalmieri

@gh-mwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.

comment:126 Changed 4 months ago by dimpase

I'm 8 timezones away from my MacOS machine.

I'd like more details/pointers on how the Github Actions testing is done - for this looks to me like "all sufficiently advanced technology is indistinguishable from magic" kind of thing :-)

comment:127 Changed 4 months ago by mkoeppe

This ticket does no GitHub Actions testing. This is all introduced in #29087.

comment:128 Changed 4 months ago by dimpase

how does one get the data I linked in comment:126 then?

comment:129 follow-up: Changed 4 months ago by mkoeppe

By merging #29087 and pushing the branch to a GitHub repository.

comment:130 Changed 4 months ago by mkoeppe

If the GitHub Actions interface is overwhelming, just download the "Artifacts". They are tar.gz files of build logs.

comment:131 in reply to: ↑ 129 Changed 4 months ago by dimpase

Replying to mkoeppe:

By merging #29087 and pushing the branch to a GitHub repository.

yes,but that's how one triggers this - I'm asking rather about how stuff works, inside. Locally and then somehow pushing the logs to GitHub?

comment:132 Changed 4 months ago by mkoeppe

I don't push logs to GitHub.

#29087 adds .github/workflows/tox.yml (https://git.sagemath.org/sage.git/diff/.github/workflows/tox.yml?id=576a18834a16ff05cd6894172a55e86bcc95d407&id2=be1e22c441d9332fd0bf4a3066e1964deddaf8c7) which gets run on every commit pushed to GitHub and on every pull request.

The job tox-local-homebrew-macos defined in that file provisions a macOS host, checks out the branch and runs tox -e local-homebrew-macos-standard there. Then it packages up logs and makes them available as an artifact.

Last edited 4 months ago by mkoeppe (previous) (diff)

comment:133 Changed 4 months ago by dimpase

and where is that MacOS host getting provisioned?

comment:134 Changed 4 months ago by mkoeppe

GitHub takes care of that, I think it's using Azure.

comment:135 in reply to: ↑ 125 Changed 4 months ago by gh-mwageringel

Replying to jhpalmieri:

@gh-mwageringel, @dimpase: do you think more needs to be done here, or can we proceed with a positive review? I'm happy with it.

I have not tested this locally, but as you have done that, I am happy with this too.

One last question though: As far as I can see, readline is not installed into /usr/local. Why do you not run into conflicts with macOS's system "readline"/libedit, as in #29000?

comment:136 Changed 4 months ago by mkoeppe

These tests use their own homebrew installation in a prefix .tox/local-homebrew-macos-standard/homebrew/, and sets up environment variables using the new script .homebrew-build-env. This sets LDFLAGS and CPPFLAGS for the readline installed into opt under the homebrew prefix.

comment:137 Changed 4 months ago by gh-mwageringel

The problem in #29000 occured despite passing LDFLAGS and CPPFLAGS to configure, so I guess setting these also during make would avoid that problem, as a workaround.

comment:138 Changed 4 months ago by mkoeppe

I think the problem description of #29000 needs some more precision.

comment:139 Changed 4 months ago by dimpase

  • Reviewers set to Dima Pasechnik, John Palmieri
  • Status changed from needs_review to positive_review

OK, looks good to me.

comment:140 Changed 4 months ago by mkoeppe

Thanks everyone!

comment:141 Changed 4 months ago by vbraun

  • Branch changed from u/mkoeppe/tox-homebrew to dfce7d4b3d771aaaa7b3d198ff2b1e3eeb2120fc
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.