#29051 closed defect (fixed)

Repair use of system BLAS broken by #29025: remove numpy site.cfg [DEFAULT/ALL] sections

Reported by: mkoeppe Owned by:
Priority: blocker Milestone: sage-9.1
Component: packages: standard Keywords:
Cc: fbissey, dimpase, gh-mwageringel, vbraun Merged in:
Authors: Dima Pasechnik Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: dba7aef (Commits, GitHub, GitLab) Commit: dba7aefc4d6678210c796aaea4edf17867be216d
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

This ticket fixes what was broken by #29025.

Previous related tickets:

  • #20157: make numpy and scipy use pkg-config to find blas/lapack
  • #22047: Scipy fails to build with ATLAS
  • #29025: numpy: site.cfg needs a [DEFAULT] section

References:

scipy/setup.py at master · scipy/scipy https://github.com/scipy/scipy/blob/master/scipy/setup.py

numpy/system_info.py at master · numpy/numpy https://github.com/numpy/numpy/blob/master/numpy/distutils/system_info.py

python - How to check BLAS/LAPACK linkage in NumPy and SciPy? - Stack Overflow https://stackoverflow.com/questions/9000164/how-to-check-blas-lapack-linkage-in-numpy-and-scipy

Attachments (2)

vdelecroix-config.log (187.3 KB) - added by vdelecroix 19 months ago.
vdelecroix-numpy-1.16.1.log (10.8 KB) - added by vdelecroix 19 months ago.

Download all attachments as: .zip

Change History (30)

comment:1 Changed 19 months ago by mkoeppe

  • Cc gh-mwageringel added

comment:2 Changed 19 months ago by dimpase

I now see this on Arch on #29071 (with openblas from the system)

comment:3 Changed 19 months ago by dimpase

  • Priority changed from major to blocker

#29025 broke buidling scipy with system's OpenBLAS. I'm now looking into how to tweak numpy/distutils/site.cfg to make it work, but it is not obvious. Probably there are more bugs there (in numpy).

comment:4 Changed 19 months ago by fbissey

Something that weirded me a bit in #29025. Why keep the [ALL] section at all? I cannot be sure that's the root cause but I would start by cleaning that.

comment:5 Changed 19 months ago by dimpase

it's a bloody mess, if you read the numpy issue linked at #29025 - some numpy config functions use [ALL] section.

Removing [DEFAULT] section allows the build with system OpenBLAS to work again, so there is clearly something amiss. I messed up while testing #29025 and so it went through.

comment:6 Changed 19 months ago by dimpase

Should we just revert #29025 ? (at the moment it's not needed anywhere expect in not yet ready tickers)

I don't see an easy fix for it, without digging into numpy mess.

comment:7 Changed 19 months ago by mkoeppe

After reverting, is [ALL] needed?

comment:8 Changed 19 months ago by dimpase

No, [ALL] with just paths in it has no effect, according to numpy people. So it should be removed.

What was the orginal need for [DEFAULT] in #29025 ?

comment:9 Changed 19 months ago by dimpase

[blas]
libraries  = openblas
[lapack]
libraries  = openblas

seems to work just as well.

comment:10 follow-up: Changed 19 months ago by fbissey

If we did the work properly with the like of CPATH and LIBRARY_PATH we should be able to drop [ALL]/[DEFAULT] completely. These section just define additional default paths for headers and libraries to be used in numpy.

comment:11 Changed 19 months ago by dimpase

  • Authors set to Dima Pasechnik
  • Branch set to public/packages/numpy/no_DEFAULT_and_ALL_numpy_site_cfg
  • Commit set to 9d1770a3ae75526fab19cff91dba6aeba1e04d97
  • Status changed from new to needs_review

New commits:

9d1770arevert #29025, remove useless [ALL] section

comment:12 in reply to: ↑ 10 Changed 19 months ago by dimpase

Replying to fbissey:

If we did the work properly with the like of CPATH and LIBRARY_PATH we should be able to drop [ALL]/[DEFAULT] completely. These section just define additional default paths for headers and libraries to be used in numpy.

IMHO all these paths are better set up per library-specific section ([blas] etc.) than via "globals" like LIBRARY_PATH.

Changed 19 months ago by vdelecroix

Changed 19 months ago by vdelecroix

comment:13 follow-up: Changed 19 months ago by vdelecroix

Thanks to Dima (and #29071 and the branch here), I got Sage compile up to numpy. It ends up with some linking trouble. See

comment:14 in reply to: ↑ 13 Changed 19 months ago by dimpase

  • Status changed from needs_review to needs_work

Replying to vdelecroix:

Thanks to Dima (and #29071 and the branch here), I got Sage compile up to numpy. It ends up with some linking trouble. See

thanks, it's the same Arch-specific issue again - more precisely, numpy does not know about cblas. Testing on Debian wasn't enough.

comment:15 Changed 19 months ago by git

  • Commit changed from 9d1770a3ae75526fab19cff91dba6aeba1e04d97 to dba7aefc4d6678210c796aaea4edf17867be216d

Branch pushed to git repo; I updated commit sha1. New commits:

dba7aefrevert from "libraries=" back to section-specific "BLAH_libs="

comment:16 Changed 19 months ago by dimpase

  • Status changed from needs_work to needs_review

the commit at comment:15 fixes the latest problem. I've also left a comment on this here: https://github.com/numpy/numpy/pull/15338#issuecomment-578516788

comment:18 Changed 19 months ago by vdelecroix

At ​dba7aef, success for me!

comment:19 follow-up: Changed 19 months ago by mkoeppe

  • Status changed from needs_review to needs_work

Errors on various platforms tested at https://github.com/mkoeppe/sage/actions/runs/31716135. Examples:

  • ubuntu-bionic-minimal

builds openblas from source, then installing R from source fails with openblas-related errors

  • fedora-32-standard

configure fails to detect openblas, error building openblas from source

comment:20 in reply to: ↑ 19 Changed 19 months ago by fbissey

Replying to mkoeppe:

Errors on various platforms tested at https://github.com/mkoeppe/sage/actions/runs/31716135. Examples:

  • ubuntu-bionic-minimal

builds openblas from source, then installing R from source fails with openblas-related errors

  • fedora-32-standard

configure fails to detect openblas, error building openblas from source

I'll be honest. None of these failures can be a result of this ticket. May be the git branch needs rebasing to include some fix for those separate issues.

comment:21 follow-up: Changed 19 months ago by mkoeppe

  • Cc vbraun added
  • Description modified (diff)
  • Reviewers set to Matthias Koeppe
  • Status changed from needs_work to positive_review
  • Summary changed from Paths configured in installed numpy site.cfg [DEFAULT/ALL] do not affect scipy to Repair use of system BLAS broken by #29025: remove numpy site.cfg [DEFAULT/ALL] sections

Yes, this fix should definitely go into the next beta.

comment:22 Changed 19 months ago by mkoeppe

Follow-up for the other build issues related to BLAS at #29088.

comment:23 in reply to: ↑ 21 Changed 19 months ago by fbissey

Replying to mkoeppe:

Yes, this fix should definitely go into the next beta.

Note that Volker silently released 9.1.beta2 yesterday. Should ping him to make a proper announcement.

comment:24 follow-up: Changed 19 months ago by jhpalmieri

I just downloaded the 9.1.beta2 tarball today, and on an Ubuntu virtual machine it fails on scipy, complaining about blas:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-7agkks4g/setup.py", line 492, in <module>
        setup_package()
      File "/tmp/pip-req-build-7agkks4g/setup.py", line 488, in setup_package
        setup(**metadata)
      File "/home/john/Sage/sage-9.1.beta2/local/lib/python3.7/site-packages/numpy/distutils/core.py", line 137, in setup
        config = configuration()
      File "/tmp/pip-req-build-7agkks4g/setup.py", line 395, in configuration
        raise NotFoundError(msg)
    numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.

The main Sage config.log file says

configure:22549: result:     openblas-0.3.6.p0 will not be installed (configure check)

Am I doing something wrong?

comment:25 in reply to: ↑ 24 ; follow-up: Changed 19 months ago by fbissey

Replying to jhpalmieri:

Am I doing something wrong?

No you are not. This ticket is supposed to deal with the stuff you just experienced. But this ticket is not included in 9.1.beta2, so it is effectively broken for external openblas.

comment:26 in reply to: ↑ 25 Changed 19 months ago by jhpalmieri

Replying to fbissey:

Replying to jhpalmieri:

Am I doing something wrong?

No you are not. This ticket is supposed to deal with the stuff you just experienced. But this ticket is not included in 9.1.beta2, so it is effectively broken for external openblas.

I just realized that, too. When I merge this, acipy builds.

comment:27 Changed 19 months ago by mkoeppe

Follow-up = #29071.

comment:28 Changed 18 months ago by vbraun

  • Branch changed from public/packages/numpy/no_DEFAULT_and_ALL_numpy_site_cfg to dba7aefc4d6678210c796aaea4edf17867be216d
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.