Opened 3 years ago
Closed 3 years ago
#29051 closed defect (fixed)
Repair use of system BLAS broken by #29025: remove numpy site.cfg [DEFAULT/ALL] sections
Reported by:  mkoeppe  Owned by:  

Priority:  blocker  Milestone:  sage9.1 
Component:  packages: standard  Keywords:  
Cc:  fbissey, dimpase, ghmwageringel, vbraun  Merged in:  
Authors:  Dima Pasechnik  Reviewers:  Matthias Koeppe 
Report Upstream:  N/A  Work issues:  
Branch:  dba7aef (Commits, GitHub, GitLab)  Commit:  dba7aefc4d6678210c796aaea4edf17867be216d 
Dependencies:  Stopgaps: 
Description (last modified by )
This ticket fixes what was broken by #29025.
Previous related tickets:
 #20157: make numpy and scipy use
pkgconfig
to findblas/lapack
 #22047: Scipy fails to build with ATLAS
 #29025: numpy:
site.cfg
needs a[DEFAULT]
section
References:
scipy/setup.py at master · scipy/scipy https://github.com/scipy/scipy/blob/master/scipy/setup.py
numpy/system_info.py at master · numpy/numpy https://github.com/numpy/numpy/blob/master/numpy/distutils/system_info.py
python  How to check BLAS/LAPACK linkage in NumPy and SciPy?  Stack Overflow https://stackoverflow.com/questions/9000164/howtocheckblaslapacklinkageinnumpyandscipy
Attachments (2)
Change History (30)
comment:1 Changed 3 years ago by
 Cc ghmwageringel added
comment:2 Changed 3 years ago by
comment:3 Changed 3 years ago by
 Priority changed from major to blocker
#29025 broke buidling scipy with system's OpenBLAS. I'm now looking into how to tweak
numpy/distutils/site.cfg
to make it work, but it is not obvious. Probably there are more bugs there (in numpy).
comment:4 Changed 3 years ago by
Something that weirded me a bit in #29025. Why keep the [ALL]
section at all? I cannot be sure that's the root cause but I would start by cleaning that.
comment:5 Changed 3 years ago by
comment:6 Changed 3 years ago by
Should we just revert #29025 ? (at the moment it's not needed anywhere expect in not yet ready tickers)
I don't see an easy fix for it, without digging into numpy mess.
comment:7 Changed 3 years ago by
After reverting, is [ALL] needed?
comment:8 Changed 3 years ago by
No, [ALL] with just paths in it has no effect, according to numpy people. So it should be removed.
What was the orginal need for [DEFAULT] in #29025 ?
comment:9 Changed 3 years ago by
[blas] libraries = openblas [lapack] libraries = openblas
seems to work just as well.
comment:10 followup: ↓ 12 Changed 3 years ago by
If we did the work properly with the like of CPATH
and LIBRARY_PATH
we should be able to drop [ALL]
/[DEFAULT]
completely. These section just define additional default paths for headers and libraries to be used in numpy.
comment:11 Changed 3 years ago by
 Branch set to public/packages/numpy/no_DEFAULT_and_ALL_numpy_site_cfg
 Commit set to 9d1770a3ae75526fab19cff91dba6aeba1e04d97
 Status changed from new to needs_review
New commits:
9d1770a  revert #29025, remove useless [ALL] section

comment:12 in reply to: ↑ 10 Changed 3 years ago by
Replying to fbissey:
If we did the work properly with the like of
CPATH
andLIBRARY_PATH
we should be able to drop[ALL]
/[DEFAULT]
completely. These section just define additional default paths for headers and libraries to be used in numpy.
IMHO all these paths are better set up per libraryspecific section ([blas]
etc.)
than via "globals" like LIBRARY_PATH
.
Changed 3 years ago by
Changed 3 years ago by
comment:13 followup: ↓ 14 Changed 3 years ago by
Thanks to Dima (and #29071 and the branch here), I got Sage compile up to numpy. It ends up with some linking trouble. See
comment:14 in reply to: ↑ 13 Changed 3 years ago by
 Status changed from needs_review to needs_work
Replying to vdelecroix:
Thanks to Dima (and #29071 and the branch here), I got Sage compile up to numpy. It ends up with some linking trouble. See
thanks, it's the same Archspecific issue again  more precisely, numpy does not know about cblas. Testing on Debian wasn't enough.
comment:15 Changed 3 years ago by
 Commit changed from 9d1770a3ae75526fab19cff91dba6aeba1e04d97 to dba7aefc4d6678210c796aaea4edf17867be216d
Branch pushed to git repo; I updated commit sha1. New commits:
dba7aef  revert from "libraries=" back to sectionspecific "BLAH_libs="

comment:16 Changed 3 years ago by
 Status changed from needs_work to needs_review
the commit at comment:15 fixes the latest problem. I've also left a comment on this here: https://github.com/numpy/numpy/pull/15338#issuecomment578516788
comment:17 Changed 3 years ago by
comment:18 Changed 3 years ago by
At dba7aef
, success for me!
comment:19 followup: ↓ 20 Changed 3 years ago by
 Status changed from needs_review to needs_work
Errors on various platforms tested at https://github.com/mkoeppe/sage/actions/runs/31716135. Examples:
 archlinuxlateststandard at https://github.com/mkoeppe/sage/runs/410329835: configure finds system openblas, then installing R from source fails with openblasrelated errors
 ubuntubionicminimal
builds openblas from source, then installing R from source fails with openblasrelated errors
 fedora32standard
configure fails to detect openblas, error building openblas from source
comment:20 in reply to: ↑ 19 Changed 3 years ago by
Replying to mkoeppe:
Errors on various platforms tested at https://github.com/mkoeppe/sage/actions/runs/31716135. Examples:
 archlinuxlateststandard at https://github.com/mkoeppe/sage/runs/410329835: configure finds system openblas, then installing R from source fails with openblasrelated errors
 ubuntubionicminimal
builds openblas from source, then installing R from source fails with openblasrelated errors
 fedora32standard
configure fails to detect openblas, error building openblas from source
I'll be honest. None of these failures can be a result of this ticket. May be the git branch needs rebasing to include some fix for those separate issues.
comment:21 followup: ↓ 23 Changed 3 years ago by
 Cc vbraun added
 Description modified (diff)
 Reviewers set to Matthias Koeppe
 Status changed from needs_work to positive_review
 Summary changed from Paths configured in installed numpy site.cfg [DEFAULT/ALL] do not affect scipy to Repair use of system BLAS broken by #29025: remove numpy site.cfg [DEFAULT/ALL] sections
Yes, this fix should definitely go into the next beta.
comment:22 Changed 3 years ago by
Followup for the other build issues related to BLAS at #29088.
comment:23 in reply to: ↑ 21 Changed 3 years ago by
Replying to mkoeppe:
Yes, this fix should definitely go into the next beta.
Note that Volker silently released 9.1.beta2 yesterday. Should ping him to make a proper announcement.
comment:24 followup: ↓ 25 Changed 3 years ago by
I just downloaded the 9.1.beta2 tarball today, and on an Ubuntu virtual machine it fails on scipy
, complaining about blas
:
Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pipreqbuild7agkks4g/setup.py", line 492, in <module> setup_package() File "/tmp/pipreqbuild7agkks4g/setup.py", line 488, in setup_package setup(**metadata) File "/home/john/Sage/sage9.1.beta2/local/lib/python3.7/sitepackages/numpy/distutils/core.py", line 137, in setup config = configuration() File "/tmp/pipreqbuild7agkks4g/setup.py", line 395, in configuration raise NotFoundError(msg) numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
The main Sage config.log
file says
configure:22549: result: openblas0.3.6.p0 will not be installed (configure check)
Am I doing something wrong?
comment:25 in reply to: ↑ 24 ; followup: ↓ 26 Changed 3 years ago by
Replying to jhpalmieri:
Am I doing something wrong?
No you are not. This ticket is supposed to deal with the stuff you just experienced. But this ticket is not included in 9.1.beta2, so it is effectively broken for external openblas.
comment:26 in reply to: ↑ 25 Changed 3 years ago by
Replying to fbissey:
Replying to jhpalmieri:
Am I doing something wrong?
No you are not. This ticket is supposed to deal with the stuff you just experienced. But this ticket is not included in 9.1.beta2, so it is effectively broken for external openblas.
I just realized that, too. When I merge this, acipy
builds.
comment:27 Changed 3 years ago by
Followup = #29071.
comment:28 Changed 3 years ago by
 Branch changed from public/packages/numpy/no_DEFAULT_and_ALL_numpy_site_cfg to dba7aefc4d6678210c796aaea4edf17867be216d
 Resolution set to fixed
 Status changed from positive_review to closed
I now see this on Arch on #29071 (with openblas from the system)