#29102 closed defect (fixed)

fflas_ffpack installation error on new CPUs with old compilers

Reported by: mkoeppe Owned by:
Priority: critical Milestone: sage-9.1
Component: packages: standard Keywords:
Cc: dimpase, arojas, gh-timokau, fbissey, vbraun, ​tmonteil Merged in:
Authors: Matthias Koeppe Reviewers: Volker Braun, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: d2f726a (Commits, GitHub, GitLab) Commit: d2f726ab38d4a737112d49ea35a743328afb73b7
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

As observed in #29084, ​fflas_ffpack compilation fails on a processor that supports AVX512 extensions with the default compiler on Ubuntu Trusty - gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.4).

See ubuntu-trusty-minimal run at ​https://github.com/mkoeppe/sage/runs/414021553

As noted in ​https://groups.google.com/d/msg/sage-release/eMQIlhglN98/bTwDGD9HBQAJ, this problem prevents Sage from build on other major platforms: ubuntu-trusty-standard, debian-jessie-standard, centos-7-minimal/standard.

In this ticket, we check which architecture flags are supported by the compiler, and disable the unsupported ones by passing options to the configure script of fflas_ffpack. (Because of bugs in fflas_ffpack's configuration scripts, the flags for individual AVX512 variants are not handled individually.)

(Upstream indicates that they plan configuration changes in the longer term to address similar issues. Not the same upstream issue: https://github.com/linbox-team/fflas-ffpack/issues/284 (that's #29161).)

Related upstream issue: https://github.com/linbox-team/fflas-ffpack/issues/184

Change History (49)

comment:1 Changed 20 months ago by gh-mwageringel

  • Cc fbissey added

See also this thread on sage-release.

comment:2 Changed 20 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Dependencies set to #29052

comment:3 Changed 20 months ago by mkoeppe

  • Branch set to u/mkoeppe/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers

comment:4 Changed 20 months ago by mkoeppe

  • Commit set to 1cce764e78fccc709ce14d8702e9519a032e372c
  • Status changed from new to needs_review

Tests run at https://github.com/mkoeppe/sage/actions/runs/35630769


New commits:

5244060Split out build/bin/sage-build-env-config from sage-env-config
96b6c58src/bin/sage: Add -buildsh
3491a9dbuild/make/deps, build/make.Makefile.in: Source build/bin/sage-build-env-config for building sagelib and for script packages
11f6797Document 'sage -buildsh' in the output of 'sage -advanced'
ecb6a8bsrc/doc/en/reference/repl/options.rst: Document sage --buildsh
1cce764build/pkgs/fflas_ffpack: Disable arch flags that are not supported by the compiler

comment:5 Changed 20 months ago by fbissey

Are those the only options not supported by gcc-4.8.2? My first thought was testing only those flags, and not stuff like sse4.

comment:6 Changed 20 months ago by mkoeppe

I only took care of the -mavx512* flags because I knew that gcc-4.8.4 (as on Ubuntu trusty) does not have them. I threw in -mfma, -mfma4 because I was not familiar with them (though 4.8.4 supports them). All other flags are ancient and they are supported by 4.8.4.

comment:7 Changed 20 months ago by fbissey

Looks good to me then, but I cannot review the commits from the dependencies. If they are OK, I agree with the stuff that specifically is here.

comment:8 Changed 20 months ago by git

  • Commit changed from 1cce764e78fccc709ce14d8702e9519a032e372c to 82509d123567c0d653b17cbb1878585222c3b3d8

Branch pushed to git repo; I updated commit sha1. New commits:

82509d1Add AC_SUBST

comment:9 Changed 20 months ago by mkoeppe

  • Description modified (diff)

comment:10 Changed 20 months ago by git

  • Commit changed from 82509d123567c0d653b17cbb1878585222c3b3d8 to e0b620e43858a1af3eb4d67a80daeaaa7703b686

Branch pushed to git repo; I updated commit sha1. New commits:

e0b620efixup

comment:11 Changed 20 months ago by git

  • Commit changed from e0b620e43858a1af3eb4d67a80daeaaa7703b686 to 946a321a4e738e64453aec8d81d6da98e4f792d1

Branch pushed to git repo; I updated commit sha1. New commits:

946a321Fixup quoting

comment:12 Changed 20 months ago by mkoeppe

This now actually works and fixes installation on ubuntu-trusty-minimal (https://github.com/mkoeppe/sage/runs/432608426?check_suite_focus=true).

But with this fix there's an error on debian-jessie-standard https://github.com/mkoeppe/sage/runs/432608838?check_suite_focus=true Here the compiler supports only one of the avx512 flags.

comment:13 Changed 20 months ago by dimpase

I'd just drop jessie, it's too much hassle to go to for something with EOL in 5 months https://wiki.debian.org/LTS/Jessie

comment:14 Changed 20 months ago by mkoeppe

I don't think this is how we should think about it. Shouldn't we rather be increasing the minimum accepted gcc version, and install our own gcc if too old?

comment:15 Changed 20 months ago by dimpase

Well, it's possible, but newer gcc might not even get built with that old gcc.

comment:16 Changed 19 months ago by git

  • Commit changed from 946a321a4e738e64453aec8d81d6da98e4f792d1 to e17c0a5ee7d801c560657430480822fa8db47537

Branch pushed to git repo; I updated commit sha1. New commits:

a0c3c59Fix "--build-sh" -> "--buildsh" in a comment
c1f0e62Merge tag '9.1.beta4' into t/29052/create_build_bin_sage_build_env_and_call_it_from_build_make_deps__add_sage___buildsh
0dc97dbClean src/doc/en/installation/*.txt in bootstrap-clean, not misc-clean
8bd4f53Merge branch 't/29233/bootstrap____make_distclean_broken' into t/29052/create_build_bin_sage_build_env_and_call_it_from_build_make_deps__add_sage___buildsh
5d47aaeMerge branch 't/29052/create_build_bin_sage_build_env_and_call_it_from_build_make_deps__add_sage___buildsh' into t/29102/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers
e17c0a5build/pkgs/fflas_ffpack/spkg-configure.m4: Disable all avx512 variants if the compiler does not support all of them

comment:17 Changed 19 months ago by mkoeppe

Now this is more robust and also works on debian-jessie-standard.

See https://github.com/mkoeppe/sage/actions/runs/44043380

comment:18 Changed 19 months ago by mkoeppe

  • Description modified (diff)

comment:19 Changed 19 months ago by git

  • Commit changed from e17c0a5ee7d801c560657430480822fa8db47537 to 5ead72fc3369b85f03445593732c103c363f811b

Branch pushed to git repo; I updated commit sha1. New commits:

2ab9b09Move SAGE_PKG_CONFIG_PATH, SAGE_NAUTY_BINS_PREFIX back to src/bin/sage-env-config.in
5ead72fMerge branch 't/29052/create_build_bin_sage_build_env_and_call_it_from_build_make_deps__add_sage___buildsh' into t/29102/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers

comment:20 Changed 19 months ago by mkoeppe

  • Cc vbraun added

comment:21 Changed 19 months ago by git

  • Commit changed from 5ead72fc3369b85f03445593732c103c363f811b to f6870ed273ec12224f3dcb34f273ce2b2fe5097e

Branch pushed to git repo; I updated commit sha1. New commits:

f6870edMerge tag '9.1.beta6' into t/29102/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers

comment:22 Changed 19 months ago by mkoeppe

  • Dependencies #29052 deleted

comment:23 Changed 19 months ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

comment:24 Changed 19 months ago by mkoeppe

Thanks!

comment:25 Changed 19 months ago by vbraun

  • Branch changed from u/mkoeppe/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers to f6870ed273ec12224f3dcb34f273ce2b2fe5097e
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:26 Changed 19 months ago by vbraun

  • Commit f6870ed273ec12224f3dcb34f273ce2b2fe5097e deleted
  • Resolution fixed deleted
  • Status changed from closed to new

This breaks linbox on fresh builds:

-----------------------------------------------
Default path = /var/lib/buildbot/slave/sage2_git/build/local /usr /usr/local 
checking whether to compile the drivers... no
-----------------------------------------------
checking for pkg-config... /var/lib/buildbot/slave/sage2_git/build/local/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for FFLAS_FFPACK... no
configure: error: Package requirements (fflas-ffpack >= 2.4.0) were not met:
Package fflas-ffpack was not found in the pkg-config search path.
Perhaps you should add the directory containing `fflas-ffpack.pc'
to the PKG_CONFIG_PATH environment variable
Package 'fflas-ffpack', required by 'world', not found
Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.
Alternatively, you may set the environment variables FFLAS_FFPACK_CFLAGS
and FFLAS_FFPACK_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
********************************************************************************

comment:27 Changed 19 months ago by mkoeppe

On what system?

comment:28 Changed 19 months ago by mkoeppe

Full log please

comment:29 Changed 19 months ago by mkoeppe

  • Branch changed from f6870ed273ec12224f3dcb34f273ce2b2fe5097e to u/mkoeppe/f6870ed273ec12224f3dcb34f273ce2b2fe5097e

comment:30 Changed 19 months ago by mkoeppe

  • Commit set to 24192eb55748a6b70d68d13552bcdbe0f20dab28

Tests at https://github.com/mkoeppe/sage/actions/runs/51973394


New commits:

1cce764build/pkgs/fflas_ffpack: Disable arch flags that are not supported by the compiler
82509d1Add AC_SUBST
e0b620efixup
946a321Fixup quoting
5d47aaeMerge branch 't/29052/create_build_bin_sage_build_env_and_call_it_from_build_make_deps__add_sage___buildsh' into t/29102/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers
e17c0a5build/pkgs/fflas_ffpack/spkg-configure.m4: Disable all avx512 variants if the compiler does not support all of them
5ead72fMerge branch 't/29052/create_build_bin_sage_build_env_and_call_it_from_build_make_deps__add_sage___buildsh' into t/29102/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers
f6870edMerge tag '9.1.beta6' into t/29102/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers
24192ebMerge tag '9.1.beta7' into t/29102/f6870ed273ec12224f3dcb34f273ce2b2fe5097e

comment:32 Changed 19 months ago by git

  • Commit changed from 24192eb55748a6b70d68d13552bcdbe0f20dab28 to d0ac900e817536744a756dea0f08ddbdb0e38e5a

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3863984build/pkgs/fflas_ffpack/spkg-configure.m4: Set sage_spkg_install_fflas_ffpack=yes
d0ac900Merge tag '9.1.beta7' into t/29102/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers

comment:33 Changed 19 months ago by mkoeppe

  • Status changed from new to needs_review

comment:34 Changed 19 months ago by mkoeppe

@vbraun Let's get this into the next beta please

comment:35 Changed 18 months ago by git

  • Commit changed from d0ac900e817536744a756dea0f08ddbdb0e38e5a to d141b1961b46c9f0ab671e4049e199a6a0196e5f

Branch pushed to git repo; I updated commit sha1. New commits:

be3c581Configure ARB_LIBRARY through sage_conf
dcd755acosmetic change
d141b19Merge branch 't/29305/move_arb_library_back_from_sage_build_env_config_to_sage_env_config' into t/29102/fflas_ffpack_installation_error_on_new_cpus_with_old_compilers

comment:36 Changed 18 months ago by mkoeppe

  • Dependencies set to #29305

comment:37 Changed 18 months ago by mkoeppe

Merged #29305 to resolve a potential merge conflict

comment:38 Changed 18 months ago by git

  • Commit changed from d141b1961b46c9f0ab671e4049e199a6a0196e5f to d2f726ab38d4a737112d49ea35a743328afb73b7

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d2f726abuild/pkgs/fflas_ffpack: Disable arch flags that are not supported by the compiler

comment:39 Changed 18 months ago by mkoeppe

  • Dependencies #29305 deleted

Rebased/squashed on 9.1.beta8, needs review

comment:40 Changed 18 months ago by mkoeppe

  • Cc ​tmonteil added

comment:41 Changed 18 months ago by mkoeppe

  • Priority changed from major to critical

Anyone interested in completing the review of this ticket? As noted many times in sage-release, most recently in https://groups.google.com/d/msg/sage-release/eMQIlhglN98/bTwDGD9HBQAJ, this problem prevents Sage from build on major platforms: ubuntu-trusty-minimal/standard, debian-jessie-standard, centos-7-minimal/standard.

comment:42 Changed 18 months ago by mkoeppe

  • Description modified (diff)

comment:43 Changed 18 months ago by mkoeppe

The tests are done. Needs review.

comment:44 Changed 18 months ago by dimpase

All tests fail. E.g. Debian tests fail due to cannot find -larb thing - perhaps this should be rebased on top of the corresponding ticket?

Fedora builds fail due to the inability to build numpy, probably related to *blas.pc issue.

Last edited 18 months ago by dimpase (previous) (diff)

comment:45 Changed 18 months ago by mkoeppe

Yes, they fail - because of unrelated issues. What is important is that the fflas_ffpack failures are gone!

comment:46 Changed 18 months ago by mkoeppe

For example, if you look at ubuntu-trusty-minimal, which had an fflas_ffpack failure, this now builds Sage correctly (https://github.com/mkoeppe/sage/runs/524022952) and runs its test suite:

sage -t src/sage/manifolds/differentiable/degenerate_submanifold.py  # Timed out
sage -t src/sage/manifolds/differentiable/euclidean.py  # Timed out
sage -t src/sage/manifolds/differentiable/metric.py  # Timed out
sage -t src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t src/sage/plot/plot.py  # Timed out
sage -t src/sage/rings/padics/padic_lattice_element.py  # 3 doctests failed
sage -t src/sage/rings/function_field/function_field.py  # Timed out
sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # Timed out

comment:47 Changed 18 months ago by dimpase

  • Reviewers changed from Volker Braun to Volker Braun, Dima Pasechnik
  • Status changed from needs_review to positive_review

comment:48 Changed 18 months ago by mkoeppe

Thank you!

comment:49 Changed 18 months ago by vbraun

  • Branch changed from u/mkoeppe/f6870ed273ec12224f3dcb34f273ce2b2fe5097e to d2f726ab38d4a737112d49ea35a743328afb73b7
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.