Opened 10 years ago

Closed 9 years ago

#8086 closed defect (fixed)

numpy fails to build on OpenSolaris or Solaris 10 64-bit

Reported by: drkirkby Owned by: drkirkby
Priority: major Milestone: sage-4.5.3
Component: porting: Solaris Keywords:
Cc: jsp Merged in: sage-4.5.3.alpha0
Authors: David Kirkby Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by drkirkby)

Build environment

  • Sun Ultra 27 3.33 GHz Intel W3580 Xeon. Quad core. 8 threads. 12 GB RAM
  • OpenSolaris 2009.06 snv_111b X86
  • Sage 4.3.1 (with a few packages hacked to work on 64-bit)
  • gcc 4.3.4 configured with Sun linker and GNU assembler from binutils version 2.20.
  • 64-bit build. SAGE64 was set to yes, plus various other tricks to get -m64 into packages.

The problem

numpy is failing to build. It looks like there is a mix of 32 and 64-bit objects.

numpy-1.3.0.p2/src/setup.py
numpy-1.3.0.p2/src/MANIFEST.in
numpy-1.3.0.p2/.hgtags
Finished extraction
****************************************************
Host system
uname -a:
SunOS hawk 5.11 snv_111b i86pc i386 i86pc
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
Target: i386-pc-solaris2.11
Configured with: ../gcc-4.3.4/configure --prefix=/usr/local/gcc-4.3.4-GNU-assembler-Sun-linker --with-as=/usr/local/binutils-2.20/bin/as --with-ld=/usr/ccs/bin/ld --with-gmp=/usr/local --with-mpfr=/usr/local
Thread model: posix
gcc version 4.3.4 (GCC) 
****************************************************
Running from numpy source directory.
F2PY Version 2
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /export/home/drkirkby/sage-4.3.1/local/lib
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in /export/home/drkirkby/sage-4.3.1/local/lib
  NOT AVAILABLE

atlas_blas_info:
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/export/home/drkirkby/sage-4.3.1/local/lib']
    language = c
    include_dirs = ['/export/home/drkirkby/sage-4.3.1/local/include']

/export/home/drkirkby/sage-4.3.1/spkg/build/numpy-1.3.0.p2/src/numpy/distutils/command/config.py:361: DeprecationWarning: 
+++++++++++++++++++++++++++++++++++++++++++++++++
Usage of get_output is deprecated: please do not 
use it anymore, and avoid configuration checks 
involving running executable on the target machine.
+++++++++++++++++++++++++++++++++++++++++++++++++

  DeprecationWarning)
customize Sage_FCompiler_1
customize Sage_FCompiler_1
customize Sage_FCompiler_1 using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc _configtest.o -L/export/home/drkirkby/sage-4.3.1/local/lib -lf77blas -lcblas -latlas -o _configtest
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libf77blas.so: wrong ELF class: ELFCLASS64
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libcblas.so: wrong ELF class: ELFCLASS64
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libatlas.so: wrong ELF class: ELFCLASS64
ld: fatal: file processing errors. No output written to _configtest
collect2: ld returned 1 exit status
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libf77blas.so: wrong ELF class: ELFCLASS64
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libcblas.so: wrong ELF class: ELFCLASS64
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libatlas.so: wrong ELF class: ELFCLASS64
ld: fatal: file processing errors. No output written to _configtest
collect2: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
Status: 255
Output: 
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/export/home/drkirkby/sage-4.3.1/local/lib']
    language = c
    define_macros = [('NO_ATLAS_INFO', 2)]
    include_dirs = ['/export/home/drkirkby/sage-4.3.1/local/include']

lapack_opt_info:
lapack_mkl_info:

Attachments (1)

8086.patch (2.0 KB) - added by drkirkby 9 years ago.
Mercurial patch which solves the Numpy build issues without any sort of hack. I think this would work on OS X too, but I'm not changing the OS X code now.

Download all attachments as: .zip

Change History (13)

comment:1 Changed 10 years ago by drkirkby

  • Description modified (diff)

comment:2 Changed 10 years ago by jsp

I do have a working numpy on your machine.

There is a numpy-1.3.0.p3.spkg in trac waiting to be merged.

Here is a pre version of mine: http://boxen.math.washington.edu/home/jsp/ports/numpy-1.3.0.p3.spkg

Works on Open Solaris.

Jaap

comment:3 Changed 10 years ago by jsp

  • Authors set to Jaap Spies
  • Status changed from new to needs_review

comment:4 Changed 9 years ago by drkirkby

  • Status changed from needs_review to needs_info

Hi Jaap,

I'm having a problem with this. I got a message about "could not fork". I even rebooted the machine once, thinking something was wrong, but everything else seems to work.

Is this bit of code in the patch right? It looks wrong to me:

echo "#! /usr/bin/env bash" > gcc 
echo "`which gcc` -m64 " | sed 's/$/\$@/' >> fake_gcc 

Should the first line not write to the file 'fake_gcc' rather than 'gcc'? If so, since it gets copied to $SAGE_LOCAL/bin/gcc, why not write it directly there?

i.e.

echo "#! /usr/bin/env bash" > $SAGE_LOCAL/bin/gcc
echo "`which gcc` -m64 " | sed 's/$/\$@/' >> $SAGE_LOCAL/bin/gcc 

Also, why use 'sed'? If I understand your intentions correctly,

echo "`which gcc` -m64 \$@"

would achieve the same, but without invoking 'sed'.

I would also add that 'which' is not a POSIX command, whereas 'command -v' gives you the same information, but in a standards complient way. This can be important, especially on Solaris 10, as the return code of 'which' is undefined on there.

drkirkby@hawk:~$ echo "`command -v gcc` -m64 \$@" 
/usr/local/gcc-4.4.4/bin/gcc -m64 $@

is more portable.

However, I can't seem to get it to work. I'm not sure if I understand exactly what you are aiming to do here. Why do we need a fake gcc?

Dave

comment:5 Changed 9 years ago by drkirkby

  • Authors changed from Jaap Spies to David Kirkby

Jaap,

I can't get your patch to work, so have reimplemented this. I've completely removed the file fake_gcc from Numpy, and instead create the fake file dynamically with the correct path.

I've also reported the issue to the Numpy bug database.

http://projects.scipy.org/numpy/ticket/1525

which should have been done years ago when the problem was first realised in OS X.

Here's a package waiting for review, which I believe does fix the problem.

http://boxen.math.washington.edu/home/kirkby/patches/numpy-1.3.0.p4.spkg

Dave

comment:6 Changed 9 years ago by drkirkby

  • Status changed from needs_info to needs_review

comment:7 Changed 9 years ago by drkirkby

  • Description modified (diff)
  • Summary changed from numpy fails to build on Open Solaris x64 - 32 / 64-bit mixup to numpy fails to build on OpenSolaris or Solaris 10 64-bit

comment:8 Changed 9 years ago by drkirkby

  • Status changed from needs_review to needs_work

Great news

I've found a much simpler solution to this, which totally avoids using any fake gcc files. Just compile with

CC="gcc -m64" export CC

and it works! I will attach a simpler patch in 15 minutes. I'm marking as "needs work" for now, as I need to do something else just now.

Dave

Changed 9 years ago by drkirkby

Mercurial patch which solves the Numpy build issues without any sort of hack. I think this would work on OS X too, but I'm not changing the OS X code now.

comment:9 Changed 9 years ago by drkirkby

  • Status changed from needs_work to needs_review

Here's the updated package. I believe this is a decent fix (at last).

http://boxen.math.washington.edu/home/kirkby/patches/numpy-1.3.0.p4.spkg

I've reported this upstream

http://projects.scipy.org/numpy/ticket/1525

Dave

comment:10 Changed 9 years ago by jhpalmieri

  • Reviewers set to John Palmieri
  • Status changed from needs_review to positive_review

The changes look good. The spkg builds on a number of different machines, both 32-bit and 64-bit, and doctests pass.

By the way, with the old spkg, numpy didn't build properly on 64-bit Solaris (either sparc -- t2 -- or x86 -- fulvia), so it's not exclusively an OpenSolaris issue. With the new spkg, it works on fulvia. I haven't gotten t2 (with SAGE64='yes') to build the prerequisite packages yet. I'm working on that, and if there are issues building numpy, I'll revert the positive review.

(Tested on t2 (32-bit), mark2 (32-bit), Mac OS X 10.6 (64-bit), sage.math (both with SAGE64='yes' and with SAGE64 unset), taurus (both with SAGE64='yes' and SAGE64 unset), cicero (32-bit), fulvia (both settings for SAGE64), and iras (64-bit machine? but with SAGE64 unset).)

comment:11 Changed 9 years ago by drkirkby

Note to the release manager

All changes are committed in the repository of the .spkg file. No library changes need to be made. Just drop in

http://boxen.math.washington.edu/home/kirkby/patches/numpy-1.3.0.p4.spkg

comment:12 Changed 9 years ago by mpatel

  • Merged in set to sage-4.5.3.alpha0
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.