cvxopt: fix illegal BLAS call and fix Solaris build

[jhpalmieri]

On OS X Lion, if you set SAGE_CHECK=yes and build cvxopt, some tests fail and then it hangs before completing the tests. See ​http://sage.math.washington.edu/home/palmieri/misc/cvxopt-1.1.3.log for the install log.

CVXOPT devs claimed that Apple shipped a broken BLAS. But it turned out that they cannot provide valid examples showing this; they did not adhere to a de facto BLAS standard, and they agreed to fix the corresponding CVXOPT bug quickly.

This spkg also fixes an issue building with GCC on Solaris 10. There already was a fix for this in the spkg, but it was only enabled for GCC versions < 4.5 (without any mention of why). Enable this fix for all GCC versions.

---

New spkg: ​http://boxen.math.washington.edu/home/jdemeyer/spkg/cvxopt-1.1.4.p1.spkg

[jhpalmieri]

In a related matter, we should patch spkg-check to exit with a nonzero return value if tests fail. We can either exit immediately once there's a failure, or we can run all tests and then exit. I'm not sure which of these is better. In general I would suggest the latter, but since it's actually hanging on OS X Lion (after at least one failed test), I might lean toward the former...

Also, should we upgrade to version 1.1.4?

[jdemeyer]

See #12519 for a new cvxopt spkg. I fixed spkg-check such that it now actually exists when there is a failure. However, it still fails on OS X Lion.

[jhpalmieri]

Dima: any ideas about the self-test failures?

[dimpase]

Replying to jhpalmieri:

Dima: any ideas about the self-test failures?

I just asked upstream ​here. Is there a way to get access to 10.7 for me?

Dima

[dimpase]

Replying to dimpase:

Replying to jhpalmieri:

Dima: any ideas about the self-test failures?

I just asked upstream ​here. Is there a way to get access to 10.7 for me?

Dima

​here it is written that it's probably a bug in Apple's blas/lapack. A workaround is to use Atlas. I have no way to check this at the moment, but it looks very reasonable to me.

It would be great if you tested this with the newest Xcode...

[jhpalmieri]

Replying to dimpase:

​here it is written that it's probably a bug in Apple's blas/lapack. A workaround is to use Atlas. I have no way to check this at the moment, but it looks very reasonable to me.

Do you mean Apple's ATLAS? We don't build ATLAS on Mac OS X, we just use the system's libraries. What should I do to try this out? Can I just make some changes in the spkg-install script?

It would be great if you tested this with the newest Xcode...

I've tried this with various versions of Xcode, including the newly released 4.3, and I get failures in all of them.

[jhpalmieri]

By the way, it might make sense to base further discussion on the new cvxopt package at #12519.

[dimpase]

Replying to jhpalmieri:

Replying to dimpase:

​here it is written that it's probably a bug in Apple's blas/lapack. A workaround is to use Atlas. I have no way to check this at the moment, but it looks very reasonable to me.

Do you mean Apple's ATLAS? We don't build ATLAS on Mac OS X, we just use the system's libraries. What should I do to try this out? Can I just make some changes in the spkg-install script?

Well, no. We need to build ATLAS spkg (or get some other non-Apple lapack/blas somehow) and use it. Actually, if I recall right, this (native lapack/blas) is an artefact of the PPC support. We can perfectly well use Atlas on x86, right?

It would be great if you tested this with the newest Xcode...

I've tried this with various versions of Xcode, including the newly released 4.3, and I get failures in all of them.

[dimpase]

doesnt work on OSX10.6:

Calling sage-spkg on 'http://boxen.math.washington.edu/home/jdemeyer/spkg/atlas-3.8.4.p2.spkg'
atlas-3.8.4.p2
Machine:
Darwin nash 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
Deleting directories from past builds of previous/current versions of atlas-3.8.4.p2
/usr/local/src/sage/sage-4.8.alpha4/local/bin/sage-spkg: file atlas-3.8.4.p2 does not exist
Attempting to download it.
http://boxen.math.washington.edu/home/jdemeyer/spkg/atlas-3.8.4.p2.spkg --> atlas-3.8.4.p2.spkg
[..................................................]
Extracting package /usr/local/src/sage/sage-4.8.alpha4/spkg/optional/atlas-3.8.4.p2.spkg ...
-rw-r--r--  1 dima  staff  2866339 Mar  6 19:27 /usr/local/src/sage/sage-4.8.alpha4/spkg/optional/atlas-3.8.4.p2.spkg
Finished extraction
****************************************************
Host system
uname -a:
Darwin nash 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
Target: i686-apple-darwin10
Configured with: /var/tmp/gcc/gcc-5666.3~6/src/configure --disable-checking --enable-werror --prefix=/usr --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin10 --program-prefix=i686-apple-darwin10- --host=x86_64-apple-darwin10 --target=i686-apple-darwin10 --with-gxx-include-dir=/include/c++/4.2.1
Thread model: posix
gcc version 4.2.1 (Apple Inc. build 5666) (dot 3)
****************************************************
Traceback (most recent call last):
  File "./spkg-install", line 13, in <module>
    from configuration import conf, cp, try_run, edit_in_place
  File "/usr/local/src/sage/sage-4.8.alpha4/spkg/build/atlas-3.8.4.p2/configuration.py", line 108, in <module>
    conf['release'] = subprocess.check_output(['uname', '-r'])
AttributeError: 'module' object has no attribute 'check_output'

real	0m0.092s
user	0m0.038s
sys	0m0.044s
************************************************************************
Error installing package atlas-3.8.4.p2
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
  /usr/local/src/sage/sage-4.8.alpha4/spkg/logs/atlas-3.8.4.p2.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/usr/local/src/sage/sage-4.8.alpha4/spkg/build/atlas-3.8.4.p2 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
  (cd '/usr/local/src/sage/sage-4.8.alpha4/spkg/build/atlas-3.8.4.p2' && '/usr/local/src/sage/sage-4.8.alpha4/sage' -sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************
Error: Failed to install package 'atlas-3.8.4.p2'.
nash:sage-5.0.beta6 dima$ 

[jdemeyer]

You're using a too old version of Sage.

[dimpase]

Replying to jdemeyer:

You're using a too old version of Sage.

/me bangs head on the kbd...

[kcrisman]

Does this only build ATLAS in Lion, or all Mac? Seems silly to do it from scratch on others, maybe we can check Dima's program mentioned at #12519 on older versions...

[jdemeyer]

Replying to kcrisman:

Does this only build ATLAS in Lion, or all Mac?

Only on Lion.

[jdemeyer]

After more than 3 hours, it finally managed to fail to build on sqrt5:

ATLAS install complete.  Examine
ATLAS/bin/<arch>/INSTALL_LOG/SUMMARY.LOG for details.
make -j1 clean
rm -f *.o x* config?.out *core*
rm: xarchinfo_freebsd.dSYM: is a directory
rm: xarchinfo_x86.dSYM: is a directory
rm: xconfig.dSYM: is a directory
rm: xprobe_gas_x8632.dSYM: is a directory
rm: xprobe_gas_x8664.dSYM: is a directory
rm: xspew.dSYM: is a directory
make[1]: *** [clean] Error 1
make: *** [build] Error 2
ATLAS failed to build, possibly because of a loaded system.
Traceback (most recent call last):
  File "./spkg-install", line 473, in <module>
    assert rc==0, 'Failed to build ATLAS.'
AssertionError: Failed to build ATLAS.

real    202m1.081s
user    182m33.861s
sys     10m48.195s

Weird that it reported that the build succeeded, but still it failed somehow.

[jdemeyer]

Replace a "rm -f" by "rm -rf" probably will solve this problem.

This explain the dSYM directories: ​http://stackoverflow.com/questions/584825/dsym-directories-while-compiling-c-code-in-macos

[kcrisman]

Aah, you beat me by five seconds! I concur.

[jdemeyer]

I put up a new version hopefully fixing this, currently building on sqrt5...

[jdemeyer]

:-(

ATLAS install complete.  Examine
ATLAS/bin/<arch>/INSTALL_LOG/SUMMARY.LOG for details.
make -j1 clean
# Xcode (OS X) may create directories like xspew.dSYM.
# Add -r option to delete also these.
rm -rf *.o x* config?.out *core*
Finished building ATLAS core.
Finished building ATLAS core.
Running make -j1 shared cshared
rm -f libatlas.so liblapack.so
make -j1 libatlas.so liblapack.so libf77blas.so libcblas.so liblapack.so
ld -melf_x86_64 -shared -soname libatlas.so -o libatlas.so \
        --whole-archive libatlas.a --no-whole-archive -lc -lpthread -lm
ld: unknown option: -melf_x86_64
make[1]: *** [libatlas.so] Error 1
make: *** [shared] Error 2
Traceback (most recent call last):
  File "./spkg-install", line 480, in <module>
    assert rc==0, 'Building shared ATLAS library failed.'
AssertionError: Building shared ATLAS library failed.

[jdemeyer]

We need to totally rewrite the command line

ld -melf_x86_64 -shared -soname libatlas.so -o libatlas.so \
        --whole-archive libatlas.a --no-whole-archive -lc -lpthread -lm

I unfortunately don't know enough about Apple's linker to know what to do here...

[vbraun]

According to Clint Whaley, only ATLAS 3.9+ supports building a shared library on OSX:

​http://sourceforge.net/tracker/?func=detail&aid=3178751&group_id=23725&atid=379483

[vbraun]

I've made a new atlas spkg here:

​http://www.stp.dias.ie/~vbraun/Sage/spkg/atlas-3.9.68.spkg

Testing will take some time...

[dimpase]

Replying to vbraun:

I've made a new atlas spkg here:

​http://www.stp.dias.ie/~vbraun/Sage/spkg/atlas-3.9.68.spkg

Testing will take some time...

on sqrt5 with Sage 5.0.beta7:

...
    Intel?: True
    processor: i386
Installing ATLAS on OS X Lion, as the BLAS shipped with the system
is broken, see http://trac.sagemath.org/sage_trac/ticket/12011
Copying patches/Makefile to src/CONFIG/src/Makefile
Copying patches/archinfo_linux.c to src/CONFIG/src/backend/archinfo_linux.c
Copying patches/archinfo_x86.c to src/CONFIG/src/backend/archinfo_x86.c
Copying patches/probe_comp.c to src/CONFIG/src/probe_comp.c
First attempt: automatic tuning.
Running configure with arch = None and isa extensions None
Traceback (most recent call last):
  File "./spkg-install", line 408, in <module>
    rc = build()
  File "./spkg-install", line 396, in build
    rc = configure(arch, isa_ext)    
  File "./spkg-install", line 278, in configure
    cmd += ' -O'+atlas_system
TypeError: cannot concatenate 'str' and 'int' objects

[dimpase]

Replying to dimpase:

on sqrt5 with Sage 5.0.beta7:

  File "./spkg-install", line 278, in configure
    cmd += ' -O'+atlas_system

Replace this line with

cmd += ' -O '+str(atlas_system)

(note the space after O!)

The result can be obtained here: ​http://boxen.math.washington.edu/home/dima/packages/atlas-3.9.68.spkg

Then is starts building. I'll report when it's done.

[vbraun]

I've also fixed the spkg in the meantime, for the record

[dimpase]

Replying to vbraun:

I've also fixed the spkg in the meantime, for the record

sqrt5:standard dima$ mailx vbraun
Subject: let's not try to build 2 atlaces at the same time 
OK? :)
EOT

[jdemeyer]

Unrelated to this ticket, but why include LAPACK inside the ATLAS spkg?

[dimpase]

The following code (compile with Sage's gfortran, with -lblas library option) crashes not only on OSX 10.7, but also on OSX 10.6.

Should we also provide Atlas on 10.6 ?

        PROGRAM TEST
        COMPLEX*16 A(1), B(1),C
        COMPLEX*16 ZDOTC
        EXTERNAL ZDOTC
        A(1)=1;
        B(1)=2;
        C=ZDOTC( 1, A, 1, B, 1 )
        WRITE (*,*) C
        END

[dimpase]

well, no, no luck yet. atlas-3.9.68 fails on sqrt5 with plenty of ld: unknown option: -melf_x86_64 messages at the end.

...
make[2]: *** [LDTRY] Error 1
ld -melf_x86_64 -shared -soname /Users/dima/Sage/sage-5.0.beta7/local/lib/libtatlas.so -o libtatlas.so \
           -rpath-link /Users/dima/Sage/sage-5.0.beta7/local/lib \
           --whole-archive libptlapack.a libptf77blas.a libptcblas.a libatlas.a --no-whole-archive -lpthread -lm
ld: unknown option: -melf_x86_64
make[2]: *** [LDTRY] Error 1
gcc -fPIC -m64 -shared -o libtatlas.so -Wl,"rpath-link /Users/dima/Sage/sage-5.0.beta7/local/lib" \
           -Wl,--whole-archive libptlapack.a libptf77blas.a libptcblas.a libatlas.a -Wl,--no-whole-archive -lpthread -lm
ld: file not found: rpath-link /Users/dima/Sage/sage-5.0.beta7/local/lib
collect2: ld returned 1 exit status
make[2]: *** [GCCTRY] Error 1
make[1]: *** [TRYALL] Error 2
make: *** [fat_ptshared] Error 2
Traceback (most recent call last):
  File "./spkg-install", line 431, in <module>
    assert rc==0, 'Building shared ATLAS library failed.'
AssertionError: Building shared ATLAS library failed.

real    683m42.661s
user    200m29.372s
sys     36m57.615s
************************************************************************
Error installing package atlas-3.9.68
************************************************************************

[vbraun]

atlas-3.9 is designed to build lapack as part of its build process. This is why lapack is now included and can be removed as separate spkg (see discussion at #10508).

Fortunately, atlas-3.9.68 has actually make targets for OSX dylibs so fixing the remaining issue should be easy. Updated spkg at ​http://www.stp.dias.ie/~vbraun/Sage/spkg/atlas-3.9.68.spkg, though its still compiling of course...

[fbissey]

I am not thrilled by atlas-3.9.xx. Do we know if they have have finally solved some of the problems they have in dgemm? At least 3.9.68 seems to have all the symbols unlike 3.9.67 which somehow left a pile of lapack symbols behind.

[vbraun]

For the record, I've tentatively run some tests on Linux haven't found any breakage so far with atlas-3.9.68.

[fbissey]

Didn't test 3.9.68 (last one I checked was 3.9.49). The following tests were historically failing with 3.9.xx:

sage -t -force_lib "devel/sage/sage/modular/modsym/space.py"
sage -t -force_lib "devel/sage/sage/calculus/interpolators.pyx"

[jdemeyer]

Replying to vbraun:

atlas-3.9 is designed to build lapack as part of its build process. This is why lapack is now included and can be removed as separate spkg (see discussion at #10508).

Does this imply we have to build ATLAS+LAPACK on every system (we used to build LAPACK on every system)? Not saying this is a bad thing, just asking...

Putting lapack in patches is just weird, you should probably distribute it extracted (either inside src or in a lapack top-level directory in the spkg).

[jdemeyer]

If you patch spkg/install, please keep in mind #10492 (which has positive_review).

[dimpase]

Replying to jdemeyer:

Replying to vbraun:

atlas-3.9 is designed to build lapack as part of its build process. This is why lapack is now included and can be removed as separate spkg (see discussion at #10508).

Does this imply we have to build ATLAS+LAPACK on every system (we used to build LAPACK on every system)? Not saying this is a bad thing, just asking...

hmm, all? Are we dropping OSX PPC support? IMHO Atlas doesn't run on OSX PPC. Also, I don't think we build Atlas on ARM. (And Linux ARM support sort of works, it's matter of just a little bit more work)

Putting lapack in patches is just weird, you should probably distribute it extracted (either inside src or in a lapack top-level directory in the spkg).

[jhpalmieri]

The new ATLAS spkg builds for me on several OS X Lion machines. The self-tests are disabled on Darwin (I realized after installing it), so they didn't run; I'm trying again after modifying spkg-check. In an existing Sage build (one in which lapack and blas were already installed), after building ATLAS, the cvxopt spkg still fails self-tests. Do we need to modify that spkg to tell it where to find the ATLAS libraries? Or could there be a problem with the old lapack or blas?

[jhpalmieri]

As far as installing this on OS X: I think we can test the existing ATLAS and/or BLAS libraries with Python, and Python is a prerequisite for this spkg. So why not test to see if the system libraries seem to be functioning; if so, don't install this. Then we shouldn't install this on pre-Lion machines, and if Apple ever fixes their system libraries, we won't install it then either.

[vbraun]

I think we have some more work to do until we can ship the new atlas on any system, you are a bit premature in discussing how to support old systems.

The only way to be sure that you don't have statically linked to another blas is to build Sage from source with the new atlas spkg. This doesn't add much overhead over building atlas.

Also, the atlas shared libraries are now named lib{t,s}atlas.{so,dylib} for threaded/singlethreaded and unix/osx respectively. We need to change modules_list.py and sage/misc/cython.py to actually use the new libraries. How about one of you with an actual OSX machine makes these config changes.

Also, on Linux I'm getting

/home/vbraun/opt/sage-atlas/sage-5.0.beta7/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: dpotrf_

when installing scipy. I'll try to track this down.

[fbissey]

Replying to vbraun:

I think we have some more work to do until we can ship the new atlas on any system, you are a bit premature in discussing how to support old systems.

The only way to be sure that you don't have statically linked to another blas is to build Sage from source with the new atlas spkg. This doesn't add much overhead over building atlas.

Also, the atlas shared libraries are now named lib{t,s}atlas.{so,dylib} for threaded/singlethreaded and unix/osx respectively. We need to change modules_list.py and sage/misc/cython.py to actually use the new libraries. How about one of you with an actual OSX machine makes these config changes.

Also, on Linux I'm getting

/home/vbraun/opt/sage-atlas/sage-5.0.beta7/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: dpotrf_

when installing scipy. I'll try to track this down.

Which lapack version are you using in the package. We had quite a few problem of that kind in Gentoo and Sebastien Fabbro (the package maintainer of atlas) did go from lapack-3.4.0_p20120215 back to lapack-3.4.0 to solve the problem.

[vbraun]

I'm using lapack-3.4.0 in atlas-3.9.68 (did you mean lapack-3.3.0?)

[fbissey]

Replying to vbraun:

I'm using lapack-3.4.0 in atlas-3.9.68 (did you mean lapack-3.3.0?)

No I meant 3.4.0, the reports that we had were for a different symbols but there could have been several missing for all I know. dpotrf_ is definitely present in my system install of atlas-lapack which is atlas 3.9.68 + lapack-3.4.0. But it may be worth a try with 3.3.1. I am looking at the spkg (I am on irc right now).

[jhpalmieri]

Now on OS X Lion, with Sage built from scratch (using the root repo patch from this ticket), this spkg passes its self-tests, but cvxopt doesn't:

Running the test suite for cvxopt-1.1.3.p1...
Testing in /Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/spkg/build/cvxopt-1.1.3.p1/src/examples/doc/chap10
Testing l1svc.py ...
Traceback (most recent call last):
  File "l1svc.py", line 12, in <module>
    op(sum(abs(x)) + sum(u), [A*x >= 1-u, u >= 0]).solve()
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/local/lib/python2.7/site-packages/cvxopt/modeling.py", line 2575, in solve
    sol = solvers.lp(c[:], G, h, A, b, solver=solver)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/local/lib/python2.7/site-packages/cvxopt/coneprog.py", line 2794, in lp
    from cvxopt import base, blas, misc
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/local/lib/python2.7/site-packages/cvxopt/misc.py", line 20, in <module>
    from cvxopt import base, blas, lapack, cholmod, misc_solvers
ImportError: dlopen(/Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/local/lib/python2.7/site-packages/cvxopt/lapack.so, 2): Symbol not found: _ATL_cGetNB
  Referenced from: /Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/local/lib/python2.7/site-packages/cvxopt/lapack.so
  Expected in: flat namespace
 in /Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/local/lib/python2.7/site-packages/cvxopt/lapack.so
Error: test /Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/spkg/build/cvxopt-1.1.3.p1/src/examples/doc/chap10/l1svc.py failed

[fbissey]

We talked on irc with Volker it is not impossible that you didn't link against the right library because there is a slightly difference in naming scheme in 3.9.68. What does

otool -L /Users/palmieri/Desktop/Sage_stuff/sage_builds/new-atlas/sage-5.0.beta7-gcc/local/lib/python2.7/site-packages/cvxopt/lapack.so

says?

[dimpase]

In fact, there is no urgent need to build Atlas on OSX 10.7, as I just learned. it turns out to be a CVXOPT bug. Please see this ​message and this ​discussion)

[dimpase]

Replying to dimpase:

In fact, there is no urgent need to build Atlas on OSX 10.7, as I just learned. it turns out to be a CVXOPT bug. Please see this ​message and this ​discussion)

what is true is that in OSX 10.7 there is a change in behavour of Apple's BLAS on a technically illegal input. What used to work as (wrongly!) expected does not work any more.

[jdemeyer]

Any news from upstream CVXOPT?

[dimpase]

Replying to jdemeyer:

Any news from upstream CVXOPT?

They have released 1.1.5 ​fixing this issue. Any takers to update the cvxopt spkg ? I, right now, have too many other things to do.

[jhpalmieri]

Replying to dimpase:

They have released 1.1.5 ​fixing this issue. Any takers to update the cvxopt spkg ? I, right now, have too many other things to do.

I'm working on it. More details in a few minutes.

[jhpalmieri]

I'm posting a new spkg, but not a perfect one. The flaw is that it is version 1.1.4 rather than 1.1.5. I looked at 1.1.4 and 1.1.5, and (using diff -r ...) I found two changes in the code: the file src/C/dense.c was modified, and I imported that change into the 1.1.4 spkg. With this change, it builds and self-tests pass on OS X Lion, and also on sage.math and OpenSolaris. So that's the version I'm posting here.

The other change was to setup.py, and I don't understand enough of the changes in our patch to modify it. That is, setup.py in version 1.1.4 was almost identical to the version in 1.1.3, so it was easy to rebase our patch, but I couldn't figure out how to rebase to 1.1.5. C'est la vie.

[jhpalmieri]

patch for cvxopt spkg; for review only

[jdemeyer]

Unfortunately, I don't have access anymore to a OS X 10.7 machine.

[jdemeyer]

This spkg is included in ​http://boxen.math.washington.edu/home/jdemeyer/release/sage-5.0.beta12-gcc/ for testing.

[jdemeyer]

Diff for the cvxopt spkg (p0->p1). For review only

[jdemeyer]

Reviewer spkg: ​http://boxen.math.washington.edu/home/jdemeyer/spkg/cvxopt-1.1.4.p1.spkg

Changes: cvxopt-1.1.4.p1.diff​

John, if you confirm that my spkg builds with SAGE_CHECK=yes on OS X Lion and you agree with my changes, you may set this to positive review.

[jhpalmieri]

Changes look good and this passes tests on OS X Lion (and OpenSolaris, which I tried just for fun).