Rewrite ATLAS spkg-install

[vbraun]

I rewrote the ATLAS spkg-install to

• allow customization of architectural defaults via the new SAGE_ATLAS_ARCH environment variable.
• still build a reasonable library if the automatic tuning process fails to find accurate timings.
• SAGE_ATLAS_LIB=path now searches in path/libatlas.so instead of path/lib/libatlas.so so it works for people with atlas in /lib64, too.
• Threading is now enabled by default. Whats the point of trying to squeeze more performance out of modern processors if you run on one core only?

By default, ATLAS will now try twice to get timings, if that fails build with SAGE_ATLAS_ARCH=fast, and if that fails with SAGE_ATLAS_ARCH=base.

From the new SPKG.txt:

The package can be configured via two environment variables:

 * SAGE_ATLAS_LIB=path
   If this environment variable is set, path/libatlas.so is used
   and ATLAS is not compiled.

 * SAGE_ATLAS_ARCH=arch[,isaext1][,isaext2]...[,isaextN]
   The given architectural default and instruction set extensions are
   used instead of the empirical tuning. Available architectures are 

     POWER3, POWER4, POWER5, PPCG4, PPCG5, P5, P5MMX, PPRO, PII, PIII,
     PM, CoreSolo, CoreDuo, Core2Solo, Core2, Corei7, P4, P4E,
     Efficeon, K7, HAMMER, AMD64K10h, UNKNOWNx86, IA64Itan, IA64Itan2,
     USI, USII, USIII, USIV, UnknownUS, MIPSR1xK, MIPSICE9

   and instruction set extensions are 
   
     AltiVec, SSE3, SSE2, SSE1, 3DNow   

   In addition, you can also set

    - SAGE_ATLAS_ARCH=fast picks defaults for a modern (2-3 year old)
      CPU of your processor line, and 

    - SAGE_ATLAS_ARCH=base picks defaults that should work for a ~10
      year old CPU.

   For example, 

     SAGE_ATLAS_ARCH=Corei7,SSE3,SSE2,SSE1

   would be appropriate for a Core i7 CPU.

I implemented the spkg-install script in python without relying on any external programs (except for make, of course). The configuration-detecting part is in the configuration.py module. For easier review I will attach the files to this ticket. The complete spkg is here:

​http://www.stp.dias.ie/~vbraun/Sage/spkg/atlas-3.8.4.spkg

[vbraun]

This should fix #10051, #9385.

Is the make_lapack_library() function in spkg-install (originally from make_correct_shared.sh) necessary on Solaris? It was just bitrot on Linux...

[vbraun]

updated spkg-install to correctly build shared libraries

[jhpalmieri]

This fails to build on t2.math.washington.edu (Solaris on sparc); I've posted the log at ​http://sage.math.washington.edu/home/palmieri/misc/atlas-3.8.3.p17.log. I get a similar failure on the skynet machines mark (Solaris on sparc) and fulvia (Solaris on x86). I am not seeing this problem on David Kirkby's machine hawk (OpenSolaris on x86).

[vbraun]

Solaris ld only understands -V and not --version`, it seems. I've updated the spkg.

[drkirkby]

IMHO, it would be good if we could remove the small amount of python that was in this package, we could start building this as soon as Fortran is built. This would allow us to start building ATLAS much earlier (before building Python).

We really should update ATLAS too.

I do wonder if such a large change is appropriate without updating ATLAS. It will make an update even more difficult.

Given ATLAS takes longer than any other package to build on many systems, getting it building early, which means avoiding Python, would seem a good idea IMHO.

Dave

[jhpalmieri]

I'm still having problems with this on t2; now I get the following error message, immediately after the description of the platform and the C compiler:

****************************************************
ld: Software Generation Utilities - Solaris Link Editors: 5.10-1.496
ld: fatal: no files on input command line
Cannot execute ld!

With the previous version, the build also failed on hawk eventually, although with a different problem. I'll try to post the log somewhere soon.

We really should update ATLAS too.

We seem to be using the most recent "stable" version, whatever stable means. (I'm using the language from the Atlas website.) It's over a year and a half old, though, with 25 unstable releases since then. Should we try 3.9.25, released in early November? Or is it too risky to try something labeled "unstable" by the developers?

[vbraun]

ATLAS 3.8.3 is the latest stable release. The next release should come soon but until its out we should stick with 3.8.3

ATLAS is also very difficult to build and it would be a nightmare to rewrite the spkg-install in POSIX shell script. This is why I rewrote the mixture of shell script, Perl, and Python in pure Python. Python is actually very well suited to detect the architectural defaults necessary for building ATLAS.

I also take it from jhpalmieri's comment that "ld -V" exits with a non-zero exit code on t2 (where I don't have an account). I'll add a workaround later...

[jhpalmieri]

(I thought that if you had a sage.math account, then you automatically had a t2.math account, same username and same password. But maybe I'm wrong about that.)

[drkirkby]

Replying to vbraun:

ATLAS 3.8.3 is the latest stable release. The next release should come soon but until its out we should stick with 3.8.3

Normally I would agree with you, but Clint, who is the ATLAS developer, has advised us not to use the latest stable release, but instead use the development release. He has also said off list that he will not be making many changes to ATLAS due to other commitments. The Sage on gentoo project is using the latest ATLAS without issue

​https://groups.google.com/group/sage-devel/browse_thread/thread/f3fa4f8737437d7f/5528028cc0c16975?lnk=gst&q=atlas&hl=en#5528028cc0c16975

ATLAS is also very difficult to build and it would be a nightmare to rewrite the spkg-install in POSIX shell script.

As it is now, I agree. But there's an awful lot of junk in there. It is a non-trivial task, but really we should update ATLAS.

This is why I rewrote the mixture of shell script, Perl, and Python in pure Python.

Yes, but it seriously increases the time to to build Sage, as numerous packages have to be build before Python is built, but if

• The small bit of Python in ATLAS is removed
• The small bit of Python in the Fortran package is removed

then it will be possible to start the build of ATLAS much earlier. Currently, builds are often building ATLAS for a long time, with nothing else going on.

I also take it from jhpalmieri's comment that "ld -V" exits with a non-zero exit code on t2 (where I don't have an account). I'll add a workaround later...

I can create you an account on t2.math.

To answer John's point, no the passwords are not shared with t2, as the Linux based NIS is not working happily with the Sun NIS. It is possible to get around, but it is a non-trivial task. William asked someone to set it up, without mentioning this to me. The guy left the machine unusable. The only way was to log in on the console. I sorted that out, but have never looked at the NIS issues.

Apparenlty the latest ATLAS builds in under an hour on t2.math, compared to the 7 hours it takes now. But I think that is only as a 64-bit application, as I don't think ATLAS has the 32-bit tuning parameters for t2.math. But I think sorting that out will be somewhat easier than with the current ATLAS, as the new ATLAS has the code to detect the T2+ processors on t2.math. Dave

[leif]

Replying to vbraun:

I implemented the spkg-install script in python without relying on any external programs (except for make, of course). The configuration-detecting part is in the configuration.py module.

I aimed at removing the Python dependency of ATLAS. ;-)

I'm not in principle against using Python for installation scripts, but unless Python gets a prerequisite (or of course is required by the package or its prerequisites anyway)...

Maybe we somehow should support SAGE_ALLOW_SYSTEM_PYTHON_FOR_BUILD.

[leif]

The way parallel makes are "disabled" is certainly wrong for a couple of reasons.

(This was in before "of course", too.)

[yledu]

Replying to vbraun:

I implemented the spkg-install script in python without relying on any external programs (except for make, of course). The configuration-detecting part is in the configuration.py module. For easier review I will attach the files to this ticket. The complete spkg is here:

​http://www.stp.dias.ie/~vbraun/Sage/spkg/atlas-3.8.3.p17.spkg

Hi,

I had problems compiling Sage-4.6 both on i7_860 and i7-740QM, whereas compilation on i7-950, Xeon-E5530, Quad-9550 ran smoothly. I use environment variables:

export MAKE='make -jN'

export SAGE64='yes'

export SAGE_PARALLEL_SPKG_BUILD='yes'

with N=number of cores (not hw threads).

Ok, so that didn't quite work out on the 860 and 740QM, I mean it took ages to compile, spending lots of time in Atlas. So I applied the p17 change, added the variables mentioned for i7, and I also as obliged to use the performance governor :

sudo cpufreq-set -r -g performance

Otherwise, even the p17 didn't detect the top frequency and only compiled with a flag indicating the lowest frequency available.

So now all my CPUs compile Sage in a reasonably coherent amount of time.

[vbraun]

export MAKE='make -jN'

Parallel make is disabled in the ATLAS spkg

export SAGE64='yes'

Not used by the ATLAS spkg

export SAGE_PARALLEL_SPKG_BUILD='yes'

Might throw off ATLAS timings if you build stuff in the background. But since ATLAS takes such a long time to build it probably does not matter.

sudo cpufreq-set -r -g performance

What governor are you using usually? I'm hoping that "ondemand" is good enough on modern CPU's since the tuning process will kick them into high gear. Any other governor besides performance and ondemand is useless unless you have an old CPU.

[vbraun]

Updated configuration.py helper file

[vbraun]

#10508: Update ATLAS to version 3.9.32

is not ready yet as it makes linbox crash. Its not yet clear whether this is the fault of atlas-3.9.32, linbox, or ntl but something is wrong. In the meantime, somebody should review the new build system as it is the way to move forward.

This ticket also fixes #9382: atlas not respecting SAGE_FAT_BINARY on i686 systems.

[ddrake]

I've looked over the code here pretty carefully and I anticipate this can get a positive review. I just need to try an actual build and see how it works.

[drkirkby]

I'm still puzzled by the logic of making changes which:

• Make it much more difficult to speed up the Sage build process by removing the small amount of Python necessary to start the ATLAS build earlier.
• Implementing changes, some of which are unnecessary if ATLAS is updated.

IMGHO, the two most logical things to do the the ATLAS package are:

1) Update the ATLAS version, which has support for many new CPUs and so allow a lot of the Sage-specfic CPU hacks to be removed. It also has the facility to use a generic set of tuning parameters. 2) Remove the few lines of Python, so ATLAS no longer depends on Python. Then the build of ATLAS can be started much earlier. This would speed up the Sage build, as ATLAS would be building for longer. ATLAS is ofgten the only item being built, so is a bottleneck.

This patch seems a backward step to me.

Dave

[vbraun]

We talked about the ATLAS install script here at bugdays, and I did raise your point that it introduces an extra python dependency. But nobody minded that much. William thinks that, if anything, more scripts should be written in python. The build scripts don't use any Sage modules, so they can easily be executed by any system-wide install of python. Eventually we'll just the system python interpreter, potentially falling back to compiling our own if no python is preinstalled. We already depend on a working gcc and perl installation, so adding python to that list isn't much of a stretch. In any case, the goal of this ticket is to first make ATLAS build reliably.

The new ATLAS version makes linbox segfault, this is why I haven't gone ahead and updated it. But again that task is a lot easier with the new spkg-install script, so I think it makes sense to push this ticket through until we figure out whats wroing with linbox/ATLAS-3.9/NTL. This will be worked on in #10508.

[ddrake]

Replying to drkirkby:

I'm still puzzled by the logic of making changes which:

• Make it much more difficult to speed up the Sage build process by removing the small amount of Python necessary to start the ATLAS build earlier.
• Implementing changes, some of which are unnecessary if ATLAS is updated.

Right now the build process is run by a Python script that calls a shell script that calls a Python script that mostly just runs shell commands and also a shell script that calls a Perl script.

Understanding what's going on is about as easy as understanding that sentence. It's crazy. The proposed solution here is two easy-to-understand Python files. If we upgrade ATLAS, it will be easy to remove the unnecessary bits. And since the files only require Python 2.3 (I think), and there's no problem demanding Python as a build dependency for Sage, so with a little extra work, we can get the ATLAS building very early in the process.

[ddrake]

With the new scripts dropped into the existing .p16 spkg, this builds ATLAS and all doctests pass on sage.math (Ubuntu 8.04 amd64). I'm going to test on t2.math and bsd.math today.

[dimpase]

Replying to ddrake:

With the new scripts dropped into the existing .p16 spkg, this builds ATLAS and all doctests pass on sage.math (Ubuntu 8.04 amd64). I'm going to test on t2.math and bsd.math today.

Volker has now updated the spkg so that it does not get confused by "ld -V" on MacOSX (this is of course very minor). It works on MacOSX 10.6 just fine.

[ddrake]

I built Sage from scratch with an atlas spkg that has these scripts. First, with SAGE_ROOT/spkg/standard/deps unchanged, and then the second time I removed the Python dependency. Both builds worked and passed doctests.

I would like to see something that shows me that SAGE_FAT_BINARY works in i686 (to make sure that we've fixed #9382) but I don't have access to any suitable machines at the moment. I'll look into this.

[ddrake]

I don't think Volker will forget about these two things, but just to make a note of it:

• the time.sleep doesn't work; time doesn't seem to be imported
• to insure that this will work with Python 2.4, you'll need to change the tuples to lists, since that version of Python strangely didn't support .index() on tuples.

[vbraun]

You are late Dan, I had the spkg already updated five minutes after you told me about the issues :-)

[ddrake]

I tested SAGE_FAT_BINARY on skynet/cicero and it works properly. So positive review here, and we can close #9382 when this is merged.

[ddrake]

#10666 is a followup ticket for documenting SAGE_ATLAS_ARCH.

[jdemeyer]

I know it's a detail, but I would prefer the ATLAS spkg-install does NOT print the message

Finished installing ATLAS.

Simply because grepping logs for '^Finished installing ' now gives

[...]
Finished installing python-2.6.4.p9.spkg
Finished installing fortran-20100629.spkg
Finished installing lapack-20071123.p2.spkg
Finished installing ATLAS.
Finished installing atlas-3.8.3.p17.spkg
Finished installing blas-20070724.spkg
[...]

[vbraun]

Ok, done. Updated spkg at the same location.

[jdemeyer]

Thanks!

[jdemeyer]

On Mac OS X 10.6, 32 bit:

atlas-3.8.3.p17
Machine:
Darwin bsd.local 10.6.0 Darwin Kernel Version 10.6.0: Wed Nov 10 18:13:17 PST 2010; root:xnu-1504.9.26~3/RELEASE_I386 i386
Deleting directories from past builds of previous/current versions of atlas-3.8.3.p17
Extracting package /Users/buildbot/build/sage/bsd-1/bsd_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg ...
-rw-r--r--  1 buildbot  staff  2819924 Mar  8 06:30 /Users/buildbot/build/sage/bsd-1/bsd_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg
Finished extraction
****************************************************
Host system
uname -a:
Darwin bsd.local 10.6.0 Darwin Kernel Version 10.6.0: Wed Nov 10 18:13:17 PST 2010; root:xnu-1504.9.26~3/RELEASE_I386 i386
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
Target: i686-apple-darwin10
Configured with: /var/tmp/gcc/gcc-5664~105/src/configure --disable-checking --enable-werror --prefix=/usr --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin10 --program-prefix=i686-apple-darwin10- --host=x86_64-apple-darwin10 --target=i686-apple-darwin10 --with-gxx-include-dir=/include/c++/4.2.1
Thread model: posix
gcc version 4.2.1 (Apple Inc. build 5664)
****************************************************
ld: unknown option: -V
Cannot execute ld!

[jdemeyer]

On OpenSolaris? 06.2009-32 (hawk):

atlas-3.8.3.p17
Machine:
SunOS hawk 5.11 snv_134 i86pc i386 i86pc
Deleting directories from past builds of previous/current versions of atlas-3.8.3.p17
Extracting package /export/home/buildbot/build/sage/hawk-1/hawk_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg ...
-rw-r--r--   1 buildbot other    2819924 Mar  8 14:30 /export/home/buildbot/build/sage/hawk-1/hawk_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg
Finished extraction
****************************************************
Host system
uname -a:
SunOS hawk 5.11 snv_134 i86pc i386 i86pc
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc-4.5.0/libexec/gcc/i386-pc-solaris2.10/4.5.0/lto-wrapper
Target: i386-pc-solaris2.10
Configured with: ../gcc-4.5.0/configure --prefix=/usr/local/gcc-4.5.0 --build=i386-pc-solaris2.10 --enable-languages=c,c++,fortran --with-gmp=/usr/local/gcc-4.5.0 --with-mpfr=/usr/local/gcc-4.5.0 --disable-nls --enable-checking=release --enable-werror=no --enable-multilib -with-system-zlib --enable-bootstrap --with-gnu-as --with-as=/usr/local/binutils-2.20/bin/as --without-gnu-ld --with-ld=/usr/ccs/bin/ld
Thread model: posix
gcc version 4.5.0 (GCC) 
****************************************************
ld: Software Generation Utilities - Solaris Link Editors: 5.11-1.1689
sh: line 1: readelf: not found
Unknown linker: 
Configuration:
    linker_GNU?: False
    CYGWIN?: False
    ld: None
    FreeBSD?: False
    32bit?: True
    linker_Solaris?: False
    bits: 32bit
    system: SunOS
    Darwin?: False
    machine: i86pc
    fortran: gfortran
    PPC?: False
    64bit?: False
    fortran_GNU?: True
    fortran_g95?: False
    Intel?: True
    Linux?: False
    SPARC?: False
    processor: i386
    Solaris?: True
Unable to find one of liblapack.a, libcblas.a, libatlas.a or libf77blas.a
in the directory /ATLAS32
Set SAGE_ATLAS_LIB to the parent directory of the directory containing
liblapack.a, libcblas.a, libatlas.a and libf77blas.a if you wish to use
existing ATLAS libraries.
So if for example those four libraries are in /usr/local/atlas/lib64
then set SAGE_ATLAS_LIB to be /usr/local/atlas/lib64
Unset SAGE_ATLAS_LIB to build ATLAS from source.
Then type make.

[jdemeyer]

On SunOS 5.10-32 (fulvia):

atlas-3.8.3.p17
Machine:
SunOS fulvia 5.10 Generic_127128-11 i86pc i386 i86pc
Deleting directories from past builds of previous/current versions of atlas-3.8.3.p17
Extracting package /home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg ...
-rw-r--r--   1 buildbot sage     2819924 Mar  8 09:30 /home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg
Finished extraction
****************************************************
Host system
uname -a:
SunOS fulvia 5.10 Generic_127128-11 i86pc i386 i86pc
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc-4.5.1/x86_64-SunOS-core2-sun-ld/libexec/gcc/i386-pc-solaris2.10/4.5.1/lto-wrapper
Target: i386-pc-solaris2.10
Configured with: /usr/local/gcc-4.5.1/src/gcc-4.5.1/configure --enable-languages=c,c++,fortran --with-gnu-as --with-as=/usr/local/binutils-2.20.1/x86_64-SunOS-core2-gcc-4.4.3/bin/as --with-ld=/usr/ccs/bin/ld --with-gmp=/usr/local/gmp-5.0.1/x86_64-SunOS-core2-gcc-4.5.0-abi32 --with-mpfr=/usr/local/mpfr-3.0.0/x86_64-SunOS-core2-gmp-5.0.1-gcc-4.5.0-abi32 --with-mpc=/usr/local/mpc-0.8.2/x86_64-SunOS-core2-mpfr-3.0.0-gmp-5.0.1-gcc-4.5.0-abi32 --prefix=/usr/local/gcc-4.5.1/x86_64-SunOS-core2-sun-ld
Thread model: posix
gcc version 4.5.1 (GCC) 
****************************************************
ld: Software Generation Utilities - Solaris Link Editors: 5.10-1.489
[...]
STAGE 4: POST-BUILD TUNING
make -f Makefile IPostTune 2>&1 | ./xatlas_tee INSTALL_LOG/POSTTUNE.LOG
   done.


STAGE 4: Threading install
make -f Makefile IBuildPtlibs 2>&1 | ./xatlas_tee INSTALL_LOG/LIBPTBUILD.LOG




ATLAS install complete.  Examine 
ATLAS/bin/<arch>/INSTALL_LOG/SUMMARY.LOG for details.
make[3]: Leaving directory `/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build'
make clean
make[3]: Entering directory `/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build'
rm -f *.o x* config?.out *core*
make[3]: Leaving directory `/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build'
make[2]: Leaving directory `/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build'
make[2]: Entering directory `/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build/lib'
make[2]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
rm -f libatlas.so liblapack.so
make libatlas.so liblapack.so libf77blas.so libcblas.so liblapack.so
make[3]: Entering directory `/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build/lib'
ld -melf_i386 -shared -soname libatlas.so -o libatlas.so \
        --whole-archive libatlas.a --no-whole-archive -lc -lpthread -lm
ld: fatal: file libatlas.so: open failed: No such file or directory
make[3]: *** [libatlas.so] Error 1
make[3]: Leaving directory `/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build/lib'
make[2]: *** [shared] Error 2
make[2]: Leaving directory `/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build/lib'
Unknown linker: 
Configuration:
    linker_GNU?: False
    CYGWIN?: False
    ld: None
    FreeBSD?: False
    32bit?: True
    linker_Solaris?: False
    bits: 32bit
    system: SunOS
    Darwin?: False
    machine: i86pc
    fortran: gfortran
    PPC?: False
    64bit?: False
    fortran_GNU?: True
    fortran_g95?: False
    Intel?: True
    Linux?: False
    SPARC?: False
    processor: i386
    Solaris?: True
Copying patches/archinfo_linux.c to src/CONFIG/src/backend/archinfo_linux.c
Copying patches/archinfo_x86.c to src/CONFIG/src/backend/archinfo_x86.c
Copying patches/probe_comp.c to src/CONFIG/src/probe_comp.c
Copying patches/Make.top to src
First automatic tuning attempt.
Running ../src/configure --prefix=/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/local --with-netlib-lapack=/home/buildbot/build/sage/fulvia-1/fulvia_full/build/sage-4.7.alpha1/local/lib/liblapack.a -Si cputhrchk 0 -Fa alg -fPIC -C if sage_fortran -b 32 -O 2
Finished configuring ATLAS.
Finished building ATLAS core.
Traceback (most recent call last):
  File "./spkg-install", line 406, in <module>
    assert rc==0, 'Building shared ATLAS library failed.'
AssertionError: Building shared ATLAS library failed.

[jdemeyer]

On SunOS 5.10-32 (t2):

atlas-3.8.3.p17
Machine:
SunOS t2 5.10 Generic_141414-02 sun4v sparc SUNW,T5240
Deleting directories from past builds of previous/current versions of atlas-3.8.3.p17
Extracting package /scratch/buildbot/sage/t2-1/t2_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg ...
-rw-r--r--   1 buildbot buildbot 2819924 Mar  8 06:30 /scratch/buildbot/sage/t2-1/t2_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg
Finished extraction
****************************************************
Host system
uname -a:
SunOS t2 5.10 Generic_141414-02 sun4v sparc SUNW,T5240
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc-4.5.1/libexec/gcc/sparc-sun-solaris2.10/4.5.1/lto-wrapper
Target: sparc-sun-solaris2.10
Configured with: ../gcc-4.5.1/configure --prefix=/usr/local/gcc-4.5.1 --with-gmp=/usr/local/gcc-4.5.1 --with-mpfr=/usr/local/gcc-4.5.1 --with-mpc=/usr/local/gcc-4.5.1 --enable-languages=c,c++,fortran --disable-nls --enable-checking=release --enable-werror=no --enable-multilib --with-system-zlib --enable-bootstrap --without-gnu-as --with-as=/usr/ccs/bin/as --without-gnu-ld --with-ld=/usr/ccs/bin/ld --with-pkgversion='Built by D. Kirkby gmp-5.0.1 mpfr-3.0.0 mpc-0.8.2'
Thread model: posix
gcc version 4.5.1 (Built by D. Kirkby gmp-5.0.1 mpfr-3.0.0 mpc-0.8.2) 
****************************************************
ld: Software Generation Utilities - Solaris Link Editors: 5.10-1.496
ld: fatal: no files on input command line
Cannot execute ld!

[jdemeyer]

Non-reproducable failure on Fedora 14-64 (taurus):

atlas-3.8.3.p17
Machine:
Linux taurus 2.6.35.10-74.fc14.x86_64 #1 SMP Thu Dec 23 16:04:50 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
Deleting directories from past builds of previous/current versions of atlas-3.8.3.p17
Extracting package /home/buildbot/build/sage/taurus-1/taurus_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg ...
-rw-r--r-- 1 buildbot sage 2819924 Mar  8 09:30 /home/buildbot/build/sage/taurus-1/taurus_full/build/sage-4.7.alpha1/spkg/standard/atlas-3.8.3.p17.spkg
Finished extraction
****************************************************
Host system
uname -a:
Linux taurus 2.6.35.10-74.fc14.x86_64 #1 SMP Thu Dec 23 16:04:50 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc-4.5.1/x86_64-Linux-nehalem-fc/libexec/gcc/x86_64-unknown-linux-gnu/4.5.1/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /usr/local/gcc-4.5.1/src/gcc-4.5.1/configure --enable-languages=c,c++,fortran --with-gnu-as --with-gnu-as=/usr/local/binutils-2.20.1/x86_64-Linux-nehalem-fc-gcc-4.4.3-rh/bin/as --with-gnu-ld --with-ld=/usr/local/binutils-2.20.1/x86_64-Linux-nehalem-fc-gcc-4.4.3-rh/bin/ld --with-gmp=/usr/local/gmp-5.0.1/x86_64-Linux-nehalem-fc-gcc-4.4.3-rh --with-mpfr=/usr/local/mpfr-3.0.0/x86_64-Linux-nehalem-fc-gmp-5.0.1-gcc-4.4.4-rh --with-mpc=/usr/local/mpc-0.8.2/x86_64-Linux-nehalem-fc-mpfr-3.0.0-gmp-5.0.1-gcc-4.4.4-rh --prefix=/usr/local/gcc-4.5.1/x86_64-Linux-nehalem-fc
Thread model: posix
gcc version 4.5.1 (GCC) 
****************************************************
[...]
   STAGE 2-2-5: GEMV TUNE
make -f Makefile INSTALL_LOG/sMVRES pre=s 2>&1 | ./xatlas_tee INSTALL_LOG/sMVTUNE.LOG
      gemvN : chose routine 3:ATL_gemvN_1x1_1a.c written by R. Clint Whaley
              Yunroll=32, Xunroll=1, using 87 percent of L1
              Performance = 4858.37 (39.48 of copy matmul, 166.04 of clock)
      gemvT : chose routine 104:ATL_gemvT_4x16_1.c written by R. Clint Whaley
              Yunroll=4, Xunroll=16, using 87 percent of L1
              Performance = 3299.85 (26.81 of copy matmul, 112.78 of clock)


   STAGE 2-2-6: GER TUNE
make -f Makefile INSTALL_LOG/sR1RES pre=s 2>&1 | ./xatlas_tee INSTALL_LOG/sR1TUNE.LOG
make[3]: *** [build] Error 255
make[3]: Leaving directory `/home/buildbot/build/sage/taurus-1/taurus_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build'
make[2]: *** [build] Error 2
make[2]: Leaving directory `/home/buildbot/build/sage/taurus-1/taurus_full/build/sage-4.7.alpha1/spkg/build/atlas-3.8.3.p17/ATLAS-build'
Configuration:
    linker_GNU?: True
    CYGWIN?: False
    ld: GNU
    FreeBSD?: False
    32bit?: False
    linker_Solaris?: False
    bits: 64bit
    system: Linux
    Darwin?: False
    machine: x86_64
    fortran: gfortran
    PPC?: False
    64bit?: True
    fortran_GNU?: True
    fortran_g95?: False
    Intel?: True
    Linux?: True
    SPARC?: False
    processor: x86_64
    Solaris?: False
Copying patches/archinfo_linux.c to src/CONFIG/src/backend/archinfo_linux.c
Copying patches/archinfo_x86.c to src/CONFIG/src/backend/archinfo_x86.c
Copying patches/probe_comp.c to src/CONFIG/src/probe_comp.c
Copying patches/Make.top to src
First automatic tuning attempt.
Running ../src/configure --prefix=/home/buildbot/build/sage/taurus-1/taurus_full/build/sage-4.7.alpha1/local --with-netlib-lapack=/home/buildbot/build/sage/taurus-1/taurus_full/build/sage-4.7.alpha1/local/lib/liblapack.a -Si cputhrchk 0 -Fa alg -fPIC -C if sage_fortran -b 64 -O 1
Finished configuring ATLAS.
Traceback (most recent call last):
  File "./spkg-install", line 402, in <module>
    assert rc==0, 'Failed to build ATLAS.'
AssertionError: Failed to build ATLAS.

[vbraun]

Sorry the atlas spkg was an older version, and I accidentally dropped a couple of compile fixes. I've updated the spkg (same place) with the most recent version.

The OSX issue has been fixed previously. It also definitely works on t2. Is fulvia a supported platform? Sage refuses to compile there unless SAGE_PORT is set.

[vbraun]

I tried to build Sage on fulvia but it fails to build mpir. How does the buildbot manage to build Sage for starters?

On hawk (where is that machine, its apparently not on skynet) it seems like SAGE_ATLAS_LIB is set to something but at least one of the libraries is not in the specified path.

[vbraun@eno ~]$ ssh hawk
ssh: Could not resolve hostname hawk: Name or service not known

[jdemeyer]

Replying to vbraun:

I tried to build Sage on fulvia but it fails to build mpir. How does the buildbot manage to build Sage for starters?

On hawk (where is that machine, its apparently not on skynet)

Mitesh Patel should know the answer to both of these questions.

[drkirkby]

'fulvia' is a Solaris 10 x86 machine on Skynet. Sage builds on that fine.

'hawk' is a Sun Ultra 27 running OpenSolaris and is my personal machine.

The buildbot regularly builds Sage on both of these machines.

Dave

[jhpalmieri]

For more on fulvia and the other skynet machines, see ​http://wiki.sagemath.org/skynet. In particular, this should tell you how to set environment variables to build Sage on fulvia.

[vbraun]

I found a couple more environment variables that might help on ​http://wiki.sagemath.org/skynet so I'll try again to build on fulvia.

Dave, can you try the atlas spkg? And if it fails again, can you tell me the directory structure of /ATLAS32? Then I'll modify the spkg-install to be not too picky. Do you have an automated script to build a suitable gcc on OpenSolaris? by any chance, then I could use my virtual machine.

For the record, the updated atlas spkg builds fine on OSX and t2.

[jhpalmieri]

Either here or on a followup ticket, the environment variables should be documented in the Sage installation guide. See ​http://www.sagemath.org/doc/installation/source.html#environment-variables, in particular. Since SAGE_ATLAS_LIB now has a different usage, this especially needs to be documented. Is this lack of backwards compatibility okay, by the way? For someone like me, if we don't realize that the variable has been changed, then the time to build Sage on t2.math.washington.edu will go from 4 hours to 15, or something like that. (This might also explain the situation with hawk: perhaps SAGE_ATLAS_LIB is set to the parent directory of .../lib.)

[jhpalmieri]

Replying to jhpalmieri:

Either here or on a followup ticket, the environment variables should be documented in the Sage installation guide.

Oh, I see that SAGE_ATLAS_ARCH has been documented already in #10666. But my comments about SAGE_ATLAS_LIB still stand.

[jdemeyer]

I have made a new version of sage-4.7.alpha1 without this ticket which should be tested today on the buildbots (and hopefully released soon). After sage-4.7.alpha1 is released, I can test the updated version of the atlas spkg which can then maybe go in a later alpha.

[vbraun]

Oh, I remember now. I changed the SAGE_ATLAS_LIB semantics such that one has to specify /path/lib instead of /path. This was changed because otherwise the spkg-install has to hardcode /lib but some OS'es use /lib32, /lib64, etc. The header must now be in SAGE_ATLAS_LIB/../include.

[vbraun]

Though thinking about it, thats also insufficient. For example, Fedora puts the ATLAS shared libraries into /usr/lib64/atlas, so the headers are not in ../include. How about we'll have a library/header search path. If it consists of a single entry, it'll try to append /lib and /include. Useful yet backward-compatible.

[drkirkby]

​http://wiki.sagemath.org/solaris has information about building on Solaris on SPARC. The main difference on x86 is that one must build binutils, to get the GNU assembler 'as', then configure gcc to use that assembler, rather than the Sun assembler. But use the Sun linker, as one does on SPARC.

Building on OpenSolaris is the same as on Solaris 10 x86 - just see what's done on skynet.

The directory structure on 'hawk' of /ATLAS32 is

/ATLAS32/lib
/ATLAS32/include

That corresponds with the current behavior in Sage, where one sets SAGE_ATLAS_LIB to be /ATLAS32.

I don't know if we actually need the include files, in which case there may be an argument for making SAGE_ATLAS_LIB point to /ATLAS32/lib or /ATLAS32/lib64 or /usr/lib64/atlas etc. I'm fairly sure one can delete the include files on a system-wide install of ATLAS and Sage will still build with just the ATLAS libraries.

[drkirkby]

Replying to vbraun:

Do you have an automated script to build a suitable gcc on OpenSolaris? by any chance, then I could use my virtual machine.

No, but I just wrote this, which should be enough information. If there's something missing, let me know. It's all from memory.

​http://wiki.sagemath.org/Solaris86

Dave

[drkirkby]

It fails on OpenSolaris in 64-bit mode. I reports not all 4 libraries are present, though I'm not convinced all 4 were needed. Anyway, that might be an installation issue at this end with the 64-bit libraries, as I've not used them much.

Here's the failure message:

****************************************************
Platform detected to be 64 bits
system_atlas.py:6: DeprecationWarning: os.popen2 is deprecated.  Use the subprocess module.
  fortran = os.popen2(os.environ['SAGE_LOCAL']+'/bin/'+'which_fortran')[1].read()

Unable to find one of liblapack.a, libcblas.a, libatlas.a or libf77blas.a
in the directory /ATLAS64/lib

As you know, I think the idea of writing this in Python is a bad one, but if its going to be done, I would have thought it better not to use depreciated code. Ideally one should use something that works in Python 2.6 and well as 3.x, but at least make sure it is not depreciated already, as that will only make it more difficult to maintain as Python changes.

Dave

[vbraun]

Do you know which libraries are necessary? I'm pretty sure we need liblapack, and at least the liblapack.so requires the other three shared libraries.

[drkirkby]

drkirkby@hawk:~$ ls /ATLAS32/lib
libatlas.a     libcblas.a     libf77blas.a   liblapack.a
libatlas.so    libcblas.so    libf77blas.so

I'm not convinced we need any shared libraries. I don't believe ATLAS builds any, and certainly Mathematica comes with libatlas.a, but no libatlas.so.

Note there is no liblapack.so, yet this builds fine.

On the 64-bit version,

drkirkby@hawk:~$ ls /ATLAS64/lib
libatlas.a     libcblas.a     libf77blas.a
libatlas.so    libcblas.so    libf77blas.so

but I must have omitted to copy liblapack.a there, which is why I had to build ATLAS from scratch, which takes about 15 minutes on this machine in 64-bit mode, but a few hours in 32-bit mode, since the ATLAS package does not have the right tuning parameters for 32-bit code.

I've since copied the 64-bit liblapack.a to my global installation directory, so now the directory has

drkirkby@hawk:~/pre-release/64/sage-4.7.alpha1/local/lib$  ls /ATLAS64/lib
libatlas.a     libcblas.a     libf77blas.a   liblapack.a
libatlas.so    libcblas.so    libf77blas.so

which is the same number and names as the 32-bit version.

Dave

[vbraun]

For Sage, or any other code that uses blas/lapack, it really does not matter if these libraries exist as static or shared form. Of course static libraries are cruft from the dark ages and shouldn't be used nowadays, but I digress. As long as either is available, the linker will do the right thing and hook up the code with blas/lapack.

With this in mind, I changed the spgk-install to just use whatever is in the SAGE_ATLAS_LIB directory, either static or shared libraries. Fedora ships dynamic atlas libraries ONLY and Sage compiles fine with SAGE_ATLAS_LIB=/usr/lib64/atlas. I also removed the part where headers from SAGE_ATLAS_LIB were symlinked to SAGE_LOCAL_LIB and apparently to no ill effect. Finally, I'm now also looking into SAGE_ATLAS_LIB/lib for libraries in order to be backward compatible.

I've updated the atlas spkg (same place), please give it a try :-)

[vbraun]

For review purposes

[drkirkby]

I'll test this as soon as I can, but for now I have other things that must take priority. It's going to take some effort to test this with the different possibilities for the environment variables.

A couple of points.

• Did you sort out the depreciated Python code?
• Is there revised documentation which explains the role of SAGE_ATLAS_LIB? (I don't see a patch attached to the ticket, and as John remarks, this is quite important).

Dave

[vbraun]

Documentation changes for SAGE_ATLAS_LIB are at #10666.

I haven't changed the os.popen2 call since a) its also in the sage library and b) its probably better to be backward than future compatible if we ever want to attempt to use the system python for executing spkg scripts. But then, I don't care too much about popen vs. subprocess and will be happy to rewrite it if you insist.

[vbraun]

Dave: I just checked and I'm actually using subprocess.Popen instead of os.popen2. The warning that you posted is from the system_atlas.py script which has never been part of the rewrite. You must be using an old version of the atlas spkg before I rewrote anything in its spkg-install.

[vbraun]

For testing purposes I built a source distribution with the updated atlas spkg and documentation fixes:

​http://www.stp.dias.ie/~vbraun/Sage/sage-4.6.2-atlas1.tar

[vbraun]

I updated the spkg to also symlink the parallel libraries libptcblas, libptf77blas from SAGE_ATLAS_LIB if they are available.

[jdemeyer]

Planning to test this on the buildbot...

[jdemeyer]

Things are looking very good on the buildbot, no problems with ATLAS!

[vbraun]

I created an updated version

​http://www.stp.dias.ie/~vbraun/Sage/spkg/atlas-3.8.3.p18.spkg

to address #11326 (make ATLAS respect CC environment variable).

[drkirkby]

I'll reiterate the point that ATLAS takes a long time to build, which could be speeded up considerably if a couple of minor changes were made to the Fortran package to remove its unnecessary reliance on Python. If we build Fortran earlier, we could start the build of ATLAS much earlier, so reduce the time to build Sage. For much of the time building Sage, ATLAS is the only package being built.

As such, I feel this change is a retrograde step.

IMHO, the best approach for ATLAS would be to

• Update it to use the pre-release/alpha version, as advised by the ATLAS developer and has been used in Gentoo.
• Check what (if any) changes need to be made to ATLAS. I think the new ATLAS will address some of the issues this update attempts to solve.
• Rewrite this package to use a POSIX shell commands
• Rewrite the Fortran package to avoid the very small amount of Python.
• Change the deps file to remove the dependency of ATLAS on Python.

That would allow ATLAS to be built before (or in parallel with Python), so speeding up the Sage build time.

I feel these changes are a retrograde step.

Dave

[vbraun]

• The development version of ATLAS currently does not work for Sage purposes, see #10508.
• Python takes essentially no time to compile vs. ATLAS, especially if ATLAS is going through its tuning process.
• If anything, more spkg-install scripts should be written in Python. Its easier to maintain and avoids may common pitfalls that can make shell scripts not POSIX-compliant. It also gives you portable functions to find out bit width and number of CPU cores, say.
• Nowadays every distribution already comes with Python, we can just use the system Python for building Sage.

[jdemeyer]

Regarding Python, I agree with Volker's opinion. I would also like to add that I don't find the Sage build time so important. And I'm saying this as the person who probably does the most compiles of a full Sage from source.

[vbraun]

New stable release ATLAS-3.8.4 is out! Spkg updated to reflect this.

Now would be a good time for somebody to review this so that it can make its way into Sage-4.7.1 ...

[fbissey]

preliminary testing on sage-on-gentoo are all good. Now to test the spkg on a couple of linux box with "vanilla" installs. I have a few other things I want to review so there may be a couple of days delay.

[drkirkby]

If the version of ATLAS is updated, many if not all of the Sage defaults for different processors can be removed. I know many of them in Sage were available in the alpha releases ages ago.

The buildbots on "hawk" and "t2.math" both use a preinstalled version of ATLAS, so testing on those machine would have to be done with unsetting SAGE_ATLAS_LIB in the buildbot's configuration.

Dave

[fbissey]

Looks like Volker took care of all the old patches already. I applied to a 4.7.1.alpha1 that I had lying around then rebuilt all ATLAS using packages I could think of: iml, linbox, cvxopt, numpy, scipy and R. I followed that by sage -ba and run the long tests. The only tests that failed were unrelated ones that failed before (german tutorial that were removed from alpha1 in the end because they were failing on this very machine).

So far it looks good to me on amd64.

[fbissey]

It would be good if we had some reports from the build boot. I hesitate to put on positive review on a package like this one which has an enormous platform dependency just based on my tests on 1 or 2 machines. It looks good, people who used the 3.8.3 version were satisfied (but unfortunately didn't always report their success story on the ticket). If we had a round of testing on the bot I would fell more comfortable pushing the positive review button.

[ddrake]

Replying to fbissey:

It would be good if we had some reports from the build boot. I hesitate to put on positive review on a package like this one which has an enormous platform dependency just based on my tests on 1 or 2 machines. It looks good, people who used the 3.8.3 version were satisfied (but unfortunately didn't always report their success story on the ticket). If we had a round of testing on the bot I would fell more comfortable pushing the positive review button.

This is a good idea. In fact, I'd like a way to take any spkg, put it the most recent alpha, and submit it to the buildbot. But for now I would be very happy to see this ATLAS upgrade get into 4.7.1.

[jdemeyer]

François, Dan: may I take your comments as meaning "if no problems arise on the buildbot, then positive_review for this ticket"?

[ddrake]

Replying to jdemeyer:

François, Dan: may I take your comments as meaning "if no problems arise on the buildbot, then positive_review for this ticket"?

Yes.

[fbissey]

Replying to ddrake:

Replying to jdemeyer:

François, Dan: may I take your comments as meaning "if no problems arise on the buildbot, then positive_review for this ticket"?

Yes.

I agree.

[jdemeyer]

This fails on the buildbot machine iras (Linux ia64): see ​http://build.sagemath.org/sage/builders/SUSE%20ES10-64%20%28iras%29/builds/129/steps/shell_1/logs/atlas

If I recall correctly, an earlier version of the Atlas spkg on this ticket did work correctly.

[vbraun]

I built Sage-4.7 with my atlas-3.8.4.spkg on iras overnight and it worked fine. Looking at the log file, it seems like atlas just failed to get timings because of system load.

I updated my atlas spkg to force the architecture to IA64Itan2/IA64Itan if the automatic tuning fails, this would give us two more shots at a successful build on Itanium and match the behaviour of the old atlas spkg (4 build attempts altogether). But when I built atlas overnight, the automatic tuning worked on the first try. I did not set SAGE_PARALLEL_SPKG_BUILD=yes, but the buildbot does. It might be a good idea to not set it on iras if the atlas timing problem surface often.

To summarize, this failure would have happened with the old atlas spkg as well. Can you try again with the updated spkg (same location)?

[jdemeyer]

Volker, I tried again with the old atlas-3.8.4 (i.e. the package which I originally said didn't work) and after 4 attempts at building Sage, it finally succeeded.

However, I think this is still a regression, because I did not get atlas build failures usually (even with SAGE_PARALLEL_SPKG_BUILD=yes).

I will need to do some more testing.

[vbraun]

I went back and re-checked the differences with the previous atlas-3.8.3 and found that I accidentally dropped some patches that deal with itanium detection. This manifests itself in the build log as Architecture configured as UNKNOWN whereas it previously was Architecture configured as IA64Itan2. I've re-added the Itanium patches and updated the atlas-3.8.4.spkg. I'm pretty sure that it'll work now.

[fbissey]

We really needed that run on the build bot otherwise we wouldn't have caught it. I had a quick look at the spkg Volker, did you update SPKG.txt with what you just did? It doesn't look like it to me. It looks like it could use some clean up so we know exactly what is currently needed versus what was needed in the past.

[vbraun]

Thanks, I updated the SPKG.txt to properly reflect which patches were removed. The new spkg is in the usual location. No changes except to the SPKG.txt log.

[fbissey]

Sorry to be a pain in the ass but it seems that this spkg-install is looking for the wrong thing under cygwin according to #11494.

[vbraun]

I've updated the atlas spkg to include the Cygwin path fix.

[fbissey]

OK so ia64 should be fixed and we have feedback from #11494 that it works on cygwin with windows XP so I think we can switch to positive review. *Pushing the button*

[rkirov]

one more positive "review" (haven't read the actual patch). Had issues with Atlas building on fresh Sage on ubuntu 11.04 x86. Downloaded Volker's new spkg and put it in spkg/standard.

First 'make' run failed again for 'probe/aux' problem. Setting SAGE_ATLAS_ARCH=fast made everything go smooth.