Update ATLAS to stable version 3.10

[Volker Braun]

The new atlas release now builds netlib lapack itself, so the lapack tarball is now included in /patches. The lapack spkg is no longer needed and will be removed.

• Updated to newest upstream source, various patches are no longer required
• SAGE_ATLAS_LIB=path now searches in path/libatlas.so instead of path/lib/libatlas.so so it works for people with atlas in /lib64, too.
• Threading is now enabled by default
• Flush before os.system (#13210)

Upstream has made some attempt at changing the layout of the shared libraries, which is now different from the static libraries. The atlas spkg contains a stub autoconf/libtools project that unpacks the static libraries and repacks them into equivalent shared libraries.

By default, ATLAS will now try twice to get timings and fail immediately if throttling is enabled. If auto-tuning fails build with SAGE_ATLAS_ARCH=fast, and if that fails with SAGE_ATLAS_ARCH=base. On x86, the fast and base targets are the new ATLAS generic targets x86SSE3 and x86SSE2/x86x87.

The current spkg version is at

​http://www.stp.dias.ie/~vbraun/Sage/spkg/atlas-3.10.0.spkg

Apply trac_10508_root_repo.patch to the SAGE_ROOT repository and 10508_doctest.patch to the Sage repository.

[Volker Braun]

Note that you cannot just use sage -f atlas-3.9.32.spkg to update atlas only. Many other spkgs use blas/lapack and must be rebuilt. The easiest way is to do a separate Sage installation...

The cvxopt spkg needs to be updated to link correctly with this atlas release, see #10509.

[Dima Pasechnik]

Does it mean that LAPACK spkg can be removed, too?

[Volker Braun]

Yes, the lapack spkg can be removed.

I'm still trying to debug some issues with linbox...

[François Bissey]

BLAS can also be removed if we go with this. f2c which was used before we got ATLAS to provide cblas by f2c-ing the BLAS package should also be removed (I think it listed in scipy's dependency only).

[Dima Pasechnik]

on 32-bit x86 Linux (Debian squeeze) I get the following, when trying to install the spkg (applied the patches to a pristine Sage 4.6.1.alpha3):

...
make -j1 libatlas.so libptf77blas.so libf77blas.so \
                libptcblas.so libcblas.so liblapack.so
make[3]: Entering directory `/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/lib'
ld -melf_i386 -shared -soname /usr/local/src/sage/sage-4.6.1.alpha3/local/lib/libatlas.so -o libatlas.so \
           -rpath-link /usr/local/src/sage/sage-4.6.1.alpha3/local/lib \
           --whole-archive libatlas.a --no-whole-archive -lc -lm
make[3]: *** No rule to make target `libptf77blas.a', needed by `libptf77blas.so'.  Stop.
make[3]: Leaving directory `/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/at
las-3.9.32/ATLAS-build/lib'
make[2]: *** [ptshared] Error 2
make[2]: Leaving directory `/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/at
las-3.9.32/ATLAS-build/lib'
Configuration:
    SAGE_LOCAL: /usr/local/src/sage/sage-4.6.1.alpha3/local
    linker_Solaris?: False
    PPC?: False
    SPKG_DIR: /usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32
    linker_GNU?: True
    ld: GNU
    system: Linux
    Darwin?: False
    machine: i686
    fortran: gfortran
    Solaris?: False
    fortran_g95?: False
    bits: 32bit
    CYGWIN?: False
    SPARC?: False
    fortran_GNU?: True
    FreeBSD?: False
    32bit?: True
    Linux?: True
    64bit?: False
    Intel?: False
    processor: 

[Dima Pasechnik]

Replying to dimpase:

on 32-bit x86 Linux (Debian squeeze) I get the following, when trying to install the spkg (applied the patches to a pristine Sage 4.6.1.alpha3):

complete install.log is here: ​http://boxen.math.washington.edu/home/dima/tmp/install-alt3.9.log.gz

[Volker Braun]

Hi Dima,

I think the problem is

make[6]: Entering directory `/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/tune/sysinfo'
gcc -c -DL2SIZE=4194304 -I/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/include -I/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/../src//include -I/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/../src//include/contrib -DAdd_ -DF77_INTEGER=int -DStringSunStyle -DATL_OS_Linux -DATL_ARCH_CoreDuo -DATL_CPUMHZ=800 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_GAS_x8632  -fomit-frame-pointer -O3 -mfpmath=387 -fPIC -m32 /usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/../src//tune/sysinfo/findNT.c
gcc -DL2SIZE=4194304 -I/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/include -I/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/../src//include -I/usr/local/src/sage/sage-4.6.1.alpha3/spkg/build/atlas-3.9.32/ATLAS-build/../src//include/contrib -DAdd_ -DF77_INTEGER=int -DStringSunStyle -DATL_OS_Linux -DATL_ARCH_CoreDuo -DATL_CPUMHZ=800 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_GAS_x8632  -fomit-frame-pointer -O3 -mfpmath=387 -fPIC -m32 -o xfindNT findNT.o ATL_walltime.o -lm
/usr/lib/crt1.o: In function `_start':
(.text+0x18): undefined reference to `main'
collect2: ld returned 1 exit status
make[6]: *** [xfindNT] Error 1

This fails because ATL_NCPU is not set. Do you have a single-core processor? I guess the threaded libraries are not built in that case.

[Dima Pasechnik]

Replying to vbraun:

This fails because ATL_NCPU is not set. Do you have a single-core processor? I guess the threaded libraries are not built in that case.

yes, it's single-core. An old Pentium M. (Atlas 3.8 spkg does not build on it, at all)

[Volker Braun]

I changed the spkg-install to make only single-threaded shared libraries if necessary. Should work now.

For simplicity, I made spkgs for cvxopts and sage_scripts. So you need to add

​http://www.stp.dias.ie/~vbraun/Sage/spkg/atlas-3.9.32.spkg ​http://www.stp.dias.ie/~vbraun/Sage/spkg/cvxopt-1.1.3.p0.spkg ​http://www.stp.dias.ie/~vbraun/Sage/spkg/sage_scripts-4.6.1.alpha3.p0.spkg

to $SAGE_ROOT/spkg/standard and then

• replace spkg/install with the attached version (Note that sage_scripts overwrites this file during installation, aargh)
• replace spkg/standard/deps with the attached version.

I'm still having doctest errors with linbox...

[Dima Pasechnik]

Replying to vbraun:

I changed the spkg-install to make only single-threaded shared libraries if necessary. Should work now.

It builds OK, but then testlong gives quite a bit of failures:

The following tests failed:
        sage -t  -long -force_lib "devel/sage/doc/en/bordeaux_2008/elliptic_curves.rst"
        sage -t  -long -force_lib "devel/sage/sage/modular/modsym/space.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/modsym/tests.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/modsym/subspace.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/modsym/ambient.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/modform/space.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/modform/element.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/modform/ambient.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/hecke/submodule.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/abvar/abvar.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/abvar/torsion_subgroup.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/abvar/cuspidal_subgroup.py"
        sage -t  -long -force_lib "devel/sage/sage/modular/abvar/finite_subgroup.py"
        sage -t  -long -force_lib "devel/sage/sage/matrix/matrix_integer_dense.pyx"
        sage -t  -long -force_lib "devel/sage/sage/matrix/matrix_integer_dense_hnf.py"
        sage -t  -long -force_lib "devel/sage/sage/rings/qqbar.py"
        sage -t  -long -force_lib "devel/sage/sage/finance/time_series.pyx"
        sage -t  -long -force_lib "devel/sage/sage/schemes/elliptic_curves/padic_lseries.py"
        sage -t  -long -force_lib "devel/sage/sage/schemes/elliptic_curves/ell_modular_symbols.py"
        sage -t  -long -force_lib "devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py"
        sage -t  -long -force_lib "devel/sage/sage/schemes/elliptic_curves/sha_tate.py"
Total time for all tests: 30045.4 seconds

I do not know how many of them are Atlas related, though. The log is here: ​http://boxen.math.washington.edu/home/dima/tmp/testlong-atl3.9.log

[François Bissey]

A lot of it is probably related. Was this a build from scratch? linbox calls seem to be particularly affected, there are a lot of failures in that path: sage.matrix.matrix_integer_dense.Matrix_integer_dense._charpoly_linbox stuff going through: sage.matrix.matrix_rational_dense.Matrix_rational_dense.right_kernel is also affected and that smells like atlas at work too.

I must say we have observed a number of failures related to ATLAS-3.9.23 in sage-on-gentoo: ​https://github.com/cschwan/sage-on-gentoo/issues#issue/3 you have failure there too but not due to iml/ATLAS as far as I can see. ​https://github.com/cschwan/sage-on-gentoo/issues#issue/6 more subtle. I see you have the failure with devel/sage/sage/finance/time_series.pyx you may want to look the comment I wrote about it in the section "known test failures for Sage on Gentoo": ​https://github.com/cschwan/sage-on-gentoo/wiki/Known-test-failures note that sage-on-gentoo can use (c)blas-reference, ATLAS or gsl-cblas or some combinations - in fact amd and intel libraries could in principle be used but no one I know has tried. ​http://www.gentoo.org/proj/en/science/blas-lapack.xml

Not sure about the SIGFPE, it could come from a result rounded to 0 or the like.

[Dima Pasechnik]

Replying to fbissey:

A lot of it is probably related. Was this a build from scratch?

yes, it's a build from scratch, on the same machine (old Pentium M) as already discussed above.

linbox calls seem to be particularly affected, there are a lot of failures in that path:

I wonder if these are Atlas bugs, or Linbox bugs...

One should try an OSX build.

[François Bissey]

why OS X in particular? I could do that but not before the 5th of January, when my university reopens. Ok I could do it before that but I'll enjoy the break a little bit more :)

[Dima Pasechnik]

Replying to fbissey:

why OS X in particular? I could do that but not before the 5th of January, when my university reopens. Ok I could do it before that but I'll enjoy the break a little bit more :)

Or does OSX remain disabled for Atlas, i.e. it's not built?

[François Bissey]

Replying to dimpase:

Replying to fbissey:

why OS X in particular? I could do that but not before the 5th of January, when my university reopens. Ok I could do it before that but I'll enjoy the break a little bit more :)

Or does OSX remain disabled for Atlas, i.e. it's not built?

I had forgotten about that. I checked the spkg and ATLAS is not built on cygwin and OS X. So it makes sense.

[Volker Braun]

I get the same doctest errors. Most of them are linbox related. The SIGFPE is from converting a NAN into a GMP integer, but I haven't gotten to the root of the NAN yet. In trying to debug this I've noticed that there are a bunch of valgrind warnings in the linbox code path we are using. I've asked some more specific questions on the linbox-use mailinglist:

​https://groups.google.com/d/topic/linbox-use/N3QNNOQuTAc/discussion

But so far no final conclusion.

[François Bissey]

Replying to dimpase:

Replying to vbraun:

This fails because ATL_NCPU is not set. Do you have a single-core processor? I guess the threaded libraries are not built in that case.

yes, it's single-core. An old Pentium M. (Atlas 3.8 spkg does not build on it, at all)

I notice you say it is a Pentium M, yet ATLAS is compiling things with the assumption that it is a coreduo: -DATL_ARCH_CoreDuo -DATL_CPUMHZ=800 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_GAS_x8632 I am guessing the speed is right but the rest is not. Was your successful build using these parameters? I had in one instance a non-working ATLAS because the cpu type was misdetected (wanted to use sse3 when it didn't have them). The build was curiously successful but the library was unusable.

[Dima Pasechnik]

Replying to fbissey:

Replying to dimpase:

Replying to vbraun:

This fails because ATL_NCPU is not set. Do you have a single-core processor? I guess the threaded libraries are not built in that case.

yes, it's single-core. An old Pentium M. (Atlas 3.8 spkg does not build on it, at all)

I notice you say it is a Pentium M, yet ATLAS is compiling things with the assumption that it is a coreduo: -DATL_ARCH_CoreDuo -DATL_CPUMHZ=800 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_GAS_x8632 I am guessing the speed is right but the rest is not. Was your successful build using these parameters? I had in one instance a non-working ATLAS because the cpu type was misdetected (wanted to use sse3 when it didn't have them). The build was curiously successful but the library was unusable.

The processor is Pentium M Banias 1.1GHz, ​http://ark.intel.com/Product.aspx?id=27600 (according to GotoBlas? installation procedure :-))

The Atlas built is not totally useless, it works for many doctests. By the way, 3.8 also thinks it's a CoreDuo?. And it works. I saw somewhere a note that it's OK.

[François Bissey]

Replying to vbraun:

I get the same doctest errors. Most of them are linbox related. The SIGFPE is from converting a NAN into a GMP integer, but I haven't gotten to the root of the NAN yet. In trying to debug this I've noticed that there are a bunch of valgrind warnings in the linbox code path we are using. I've asked some more specific questions on the linbox-use mailinglist:

​https://groups.google.com/d/topic/linbox-use/N3QNNOQuTAc/discussion

But so far no final conclusion.

Converting a NAN into a GMP integer is exactly what was happening in ​https://github.com/cschwan/sage-on-gentoo/issues/3 and it didn't happen when using gslcblas. I will do a full build of sage-on-gentoo with 3.9.40 (or 41) and see if I can see anything.

[François Bissey]

Yuck, still got problems leading to linbox, I got the ones leading to iml too. Note that using another cblas like gslcblas/openblas/reference(netlib) all these problems disappear which seem to indicate that there is something going on in ATLAS itself or that all the others gets it wrong (I realise that on a stock sage you may have trouble compiling iml with anything else than ATLAS, it requires some patching to the configure script to be able to do so).

[Tim Dumol]

It may be worth noting that new released are available for both stable and development:

​http://sourceforge.net/projects/math-atlas/files/Stable/3.8.4/|Stable ​http://sourceforge.net/projects/math-atlas/files/Developer%20%28unstable%29/3.9.41/|Unstable

[François Bissey]

I have 3.9.41 now, the only difference was it compiled faster because there was tuning for my cpu. But we could investigate 3.8.4. Thanks for pointing it is out, some of us (me at least) didn't think it would ever happen. We may have a stable ATLAS supporting newer CPUs at last, and it looks like I could test it quickly.

[François Bissey]

OK 3.8.4 doesn't suffer from any of the drawbacks that are apparent in 3.9.23+.

The big drawback is that it doesn't build lapack nicely on its own, provided we point to the sources like the latest 3.9.xx do. That means spkg-install would need a little bit more work.

Opinions?

[Volker Braun]

Can we keep this ticket focused on the development version of atlas and discuss the stable ATLAS on #10226? I updated the spkg on the latter ticket to the new stable ATLAS release.

[François Bissey]

Sure. My current opinion and that's something I am pushing to sage-on-gentoo users is to avoid ATLAS 3.9.xx for the time being. It is possible that ATLAS-3.9.xx is doing something permissible that iml and linbox are not ready to catch. My line of thought is that I remember that some algorithm in iml takes the inverse of a result returned by cblas. If the result is 0 instead of a small value we may have our NaN, more likely some result from ATLAS is a NaN.

[Leif Leonhardy]

Note that you can copy updated spkgs into spkg/standard/ and then do

env SAGE_UPGRADING=yes make build

to rebuild all dependent packages.

(We should perhaps add make targets for that to the top-level Makefile and document them in the Developer's Guide, as this is currently just a side-effect of the effort to make upgrading work.)

If you at the same time need to apply patches to the Sage library, things get a bit more complicated, as e.g. Sage switches to the main branch before reinstalling the Sage library package. One safe way is to first apply the patches, create a new sage-x.y.z-whatever spkg (with devel/sage/spkg-dist) and replace the one in spkg/standard/ with that one (or at least make sure newest_version will pick up the right one).

Note that the extension modules' dependencies in module_list.py are currently far from complete. #8664 adds some in a generic way by adding them automatically in setup.py, i.e. lets modules also depend on the headers of the libraries they use (which [only] works if the headers' mtimes get modified / updated during installation of their corresponding libraries). The dumb alternative is to run sage -ba-force after an "upgrade" process.

[Leif Leonhardy]

Ping.

[Dima Pasechnik]

Replying to leif:

Ping.

just fired up my vintage MacOSX 32-bit Powerbbok PPC... Will know more in, like, 12 hours...

[François Bissey]

Several issues:

• atlas is now at version 3.9.49 (last I checked)
• I have reports of it failling to build on some old opterons
• @dimpase: do you need atlas on OS X ppc? Don't you use vectorize like the other OS X?
• I haven't checked but I am quite sure from other reports that cblas_dgem{m,v} is still buggy (someone posted that R built against that Atlas lapack was giving them trouble and R upstream pointed the finger to Atlas).

[Dima Pasechnik]

Replying to fbissey:

Several issues:

• atlas is now at version 3.9.49 (last I checked)
• I have reports of it failling to build on some old opterons

so what? I have a 5-year old 32-bit Intel laptop on which the sage-current Atlas does not build.

• @dimpase: do you need atlas on OS X ppc? Don't you use vectorize like the other OS X?

indeed. But that's for #10509. Oops, was posting on the wrong ticket...

[Karl-Dieter Crisman]

With respect to comment:33:ticket:12011, should this be closed as a dup of #12011?

[Dima Pasechnik]

propose to close this one and refer to #12011 the the continuation of the upgrade work.

[Volker Braun]

I would say #12011 is a duplicate of my ticket but oh well ;-)

[Jeroen Demeyer]

Seems like #12011 isn't a duplicate after all...

[Jeroen Demeyer]

Obviously, the patches to spkg/install and spkg/standard/deps must be rebased.

[Jeroen Demeyer]

It makes a lot more sense to me to put the LAPACK tarball at the top level of the spkg instead of in patches/.

patches/ATLAS-lib/autom4te.cache should be removed.

[Jeroen Demeyer]

I don't like using assert for control flow, because that's not what it's meant for.

Why not replace those by (see #13210)

if rc != 0:
    print "Error: foo"
    sys.exit(rc)

[Jeroen Demeyer]

There is something wrong with the history in SPKG.txt (atlas-3.8.4 is completely missing and there is atlas-3.9.68 for #12011 which never got merged)

[Volker Braun]

I don't have any strong opinion on where to put the lapack tarball, except that our convention of only allowing a single src/ directory is shortsighted.

Note that I'm not using assertions for control flow, i.e. I'm not using

  try:
    <command>
  except AssertionError:
    <alternative>

Note that you could theoretically also catch the SystemExit exception, so sys.exit() isn't different from assert in that regard. I'm only using assertions to ensure the following contract holds: spkg-install completes successfully only if the relevant atlas configure/make completed with rc==0

I'll replace the asserts by something that returns rc as exit code, though.

[Jeroen Demeyer]

Replying to vbraun:

I'm only using assertions to ensure the following contract holds: spkg-install completes successfully only if the relevant atlas configure/make completed with rc==0

I think (IMHO but I might be wrong) that assertions should be used only in a situation where an assertion being false indicates a bug in the program. If rc != 0 in spkg-install, that is an ordinary error condition, not a bug in the spkg-install script.

[Jeroen Demeyer]

Starting from sage-5.1, I get one doctest failure:

File "/release/merger/sage-5.1-atlas/devel/sage-main/sage/rings/polynomial/polynomial_element.pyx", line 1039:
    sage: parent(poly)([ 0.0 if abs(c)<=1e-14 else c for c in poly.coeffs() ])
Expected:
    1.0
Got:
    1.02140518266e-14*x^2 + 1.0

[Dima Pasechnik]

Replying to jdemeyer:

Starting from sage-5.1, I get one doctest failure:

     sage: parent(poly)([ 0.0 if abs(c)<=1e-14 else c for c in poly.coeffs() ])

it's most probably not at Atlas problem, but rather that 1e-14 (any solid rationale behind this choice? I guess not.) needs to be adjusted.

[Benjamin Jones]

Testing in sage-5.1.rc1 on x86_64 debian Linux:

$ uname -a
Linux sage 2.6.32 #1 SMP Fri Sep 2 21:08:57 CDT 2011 x86_64 GNU/Linux

$ head -18 /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           X5690  @ 3.47GHz
stepping        : 2
cpu MHz         : 3465.790
cache size      : 12288 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat
bogomips        : 6931.58
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

BEFORE new ATLAS spkg

sage -t  "devel/sage-main/sage/modular/modsym/ambient.py"   
         [11.6 s]

sage -t  "devel/sage-main/sage/modular/hecke/ambient_module.py"
         [4.2 s]

sage -t  "devel/sage-main/sage/modular/hecke/hecke_operator.py"
         [3.0 s]

AFTER new spkg

1. untar'd fresh sage-5.1.rc1
2. replaced atlas-3.8.4.p1.spkg with atlas-3.9.85.spkg
3. make build

SPKG build log: ​http://sage.math.washington.edu/home/bjones/atlas-3.9.85.log

Build atlas-3.9.85

real    11m18.595s
user    11m15.906s
sys     2m56.099s
Successfully installed atlas-3.9.85

Tests

sage -t  "devel/sage-main/sage/modular/modsym/ambient.py"   
         [11.6 s]

sage -t  "devel/sage-main/sage/modular/hecke/ambient_module.py"
         [4.2 s]

sage -t  "devel/sage-main/sage/modular/hecke/hecke_operator.py"
         [3.1 s]

All tests

$ make ptestlong
...
All tests passed!
Total time for all tests: 1282.1 seconds

[François Bissey]

3.10.0 has just been released a few hours ago. Do we try to go for it? Like Jeroen remarked there may be tolerance issues depending on the hardware. I know I have one when using openblas instead of ATLAS.

[Volker Braun]

The new spkg will attempt to build on OSX if SAGE_ATLAS_ARCH is set. Untested, but a start.

I also am planning to allow SAGE_ATLAS_LIB=system and use ldconfig -p to get the system atlas, if available. Though perhaps we should do that in a later version. This spkg seems to fix some build issues that people reported on the sage mailinglists so it might be good to get it in sooner rather than later.

[Jeroen Demeyer]

Replying to vbraun:

so it might be good to get it in sooner rather than later.

+1, let's not worry now about changing the working of SAGE_ATLAS_LIB.