Opened 8 years ago

Closed 8 years ago

#11920 closed defect (fixed)

Sympow needs to disable fused-multiply-add and should create datafiles

Reported by: jdemeyer Owned by: tbd
Priority: major Milestone: sage-5.0
Component: packages: standard Keywords:
Cc: leif Merged in: sage-5.0.beta3
Authors: Jeroen Demeyer Reviewers: Leif Leonhardy, Volker Braun
Report Upstream: None of the above - read trac for reasoning. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

See #11226 and http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48823.

For sympow, we need to use the compiler option -ffp-contract=on, -mno-fused-madd or -ffloat-store (if the compiler supports it).

Sympow should also create its datafiles, see #5155.

spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/sympow-1.018.1.p11.spkg

Attachments (2)

sympow-1.018.1.p9-p10.diff (32.9 KB) - added by jdemeyer 8 years ago.
Diff for the new Sympow spkg, for review only
sympow-1.018.1.p10-p11.diff (10.9 KB) - added by jdemeyer 8 years ago.
Diff for the sympow spkg p10 -> p11, for review only

Download all attachments as: .zip

Change History (44)

comment:1 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:2 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:3 Changed 8 years ago by leif

  • Cc leif added

comment:4 Changed 8 years ago by jdemeyer

  • Milestone changed from sage-4.7.2 to sage-4.7.3
  • Summary changed from Sympow needs to specify -ffp-contract=on to Sympow needs specify -ffp-contract=on and should create datafiles

comment:5 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:6 Changed 8 years ago by jdemeyer

  • Report Upstream changed from N/A to None of the above - read trac for reasoning.

Not reported upstream because upstream project is dead (we should probably have this as an option in the "Upstream" field).

comment:7 Changed 8 years ago by jdemeyer

  • Authors set to Jeroen Demeyer
  • Description modified (diff)
  • Status changed from new to needs_review

Tested on Core2, pentium4, ia64 with fairly recent versions of gcc.

comment:8 Changed 8 years ago by Snark

sympow-1.018.1.p10.spkg compiles on my ARM box.

comment:9 follow-ups: Changed 8 years ago by leif

With GCC versions prior to 4.6, you might have to specify e.g. -fno-fused-madd instead on some platforms, so you could also check whether that's supported and use it in case it is.

[Although it's not immediately clear to me whether that's desirable at all; with (fused) multiply-accumulate, the results are more accurate than without. They'll of course frequently differ slightly from what you get with strict IEEE 754 compliance.]


spkg-install should use $MAKE instead of make.

$SAGE_LOCAL should be quoted in the generated script.

There are a few typos in the comments, e.g. "default mode of operation" and "precision" (instead of "precion").

comment:10 Changed 8 years ago by leif

P.S.:

Linux silius 2.6.32.46-0.3-ppc64 #1 SMP 2011-09-29 17:49:31 +0200 ppc64 ppc64 ppc64 GNU/Linux
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
Target: powerpc64-unknown-linux-gnu
Configured with: ../gcc-4.4.6/configure -v --prefix=/home/leif/local/silius/ --program-suffix=-4.4.6 --enable-languages=c,c++,fortran --enable-plugins --enable-lto --enable-gold --enable-version-specific-runtime-libs --with-cpu=power7 --with-tune=power7 --with-gmp=/home/leif/local/silius/ --with-mpfr=/home/leif/local/silius/ --with-mpc=/home/leif/local/silius/
Thread model: posix
gcc version 4.4.6 (GCC) 
****************************************************
patching file Configure
patching file generate.c
patching file fpu.c
The double precision of your FPU is 105 bits.
The Quad Double library used by SYMPOW assumes IEEE-754 double precision
numbers with exactly 53 bits in the mantissa (64 bits in total).

Unfortunately, we currently have no workaround for your system.
Running SYMPOW will surely fail. Please report this problem to
sage-devel (http://groups.google.com/group/sage-devel),
mentioning in particular your operating system, processor type
and compiler version (run gcc --version).

comment:11 Changed 8 years ago by leif

  • Reviewers set to Leif Leonhardy
  • Status changed from needs_review to needs_work

With -mno-fused-add, it builds. (I get "The double precision of your FPU is 53 bits.")

comment:12 in reply to: ↑ 9 Changed 8 years ago by leif

Replying to leif:

With GCC versions prior to 4.6, you might have to specify e.g. -fno-fused-madd [...]

Note that it's of course -m, not -f.

comment:13 follow-ups: Changed 8 years ago by leif

FWIW, both with the p9 (with -O3, and without -mno-fused-madd, which is AFAIK the default) and with the p10 (with -mno-fused-madd) all (long) tests in

sage/lfunctions/sympow.py

pass. So you should probably change your check to detect whether the doubles are 64-bit / have a 53-bit mantissa.

Only the optional ones fail (differently); one has to compute extra data (with PARI/GP) for these anyway. With the p10 I get:

sage -t -long -optional "devel/sage/sage/lfunctions/sympow.py"
**********************************************************************
File ".../sage/lfunctions/sympow.py", line 127:
    sage: a = sympow.L(EllipticCurve('11a'), 2, 16); a   # optional
Expected:
    '1.057599244590958E+00'
Got:
    '-4.967915042710588E+11'
**********************************************************************
File ".../sage/lfunctions/sympow.py", line 129:
    sage: RR(a)                    # optional -- requires precomputations
Expected:
    1.05759924459096
Got:
    -4.96791504271059e11
**********************************************************************
1 items had failures:
   2 of   5 in __main__.example_4
***Test Failed*** 2 failures.
For whitespace errors, see the file /home/leif/.sage//tmp/sympow_52568.py
	 [12.7 s]

Fairly loud noise... :-)

(Might be that the data in the "manually" computed file is just wrong.)

comment:14 in reply to: ↑ 13 Changed 8 years ago by leif

Replying to leif:

FWIW, both with the p9 (with -O3, and without -mno-fused-madd, which is AFAIK the default) and with the p10 (with -mno-fused-madd) all (long) tests in

sage/lfunctions/sympow.py

pass. So you should probably change your check to detect whether the doubles are 64-bit / have a 53-bit mantissa.

With the p9 (and with -mfused-add):

$ time env SAGE_TEST_ITER=20 ./sage -tp 1 -long devel/sage/sage/lfunctions/sympow.py 
Testing that Sage starts...
Yes, Sage starts.
Global iterations: 1
File iterations: 20
No long cached timings exist; will create for successful files.
Doctesting 1 file using 1 thread
sage -t -long devel/sage/sage/lfunctions/sympow.py
         [10.2 s]
 
----------------------------------------------------------------------
All tests passed!
Timings have been updated.
Total time for all tests: 232.1 seconds

real    3m54.484s
user    3m39.864s
sys     0m11.631s

With the p10:

$ time env SAGE_TEST_ITER=20 ./sage -tp 1 -long devel/sage/sage/lfunctions/sympow.py 
Testing that Sage starts...
Yes, Sage starts.
Global iterations: 1
File iterations: 20
Using long cached timings to run longest doctests first.
Doctesting 1 file using 1 thread
sage -t -long devel/sage/sage/lfunctions/sympow.py
         [11.4 s]
 
----------------------------------------------------------------------
All tests passed!
Timings have been updated.
Total time for all tests: 238.5 seconds

real    4m0.566s
user    3m44.169s
sys     0m11.777s

(Both packages compiled with -O3 -march=power7 -mtune=power7.)

comment:15 in reply to: ↑ 9 Changed 8 years ago by jdemeyer

Replying to leif:

With GCC versions prior to 4.6, you might have to specify e.g. -fno-fused-madd instead on some platforms, so you could also check whether that's supported and use it in case it is.

Good point, I will add that flag.

[Although it's not immediately clear to me whether that's desirable at all; with (fused) multiply-accumulate, the results are more accurate than without. They'll of course frequently differ slightly from what you get with strict IEEE 754 compliance.]

This is certainly desirable. SYMPOW really requires strict IEEE-754 compliance, anything more precise is bad.

comment:16 in reply to: ↑ 13 Changed 8 years ago by jdemeyer

Replying to leif:

FWIW, both with the p9 (with -O3, and without -mno-fused-madd, which is AFAIK the default) and with the p10 (with -mno-fused-madd) all (long) tests in

sage/lfunctions/sympow.py

pass.

The fact that all tests pass, does not necessarily mean that SYMPOW is operating correctly. It has been shown that SYMPOW sometimes computes incorrect results when fused-madd is used. I agree, this was with gcc 4.6.0 on ia64, not gcc 4.4.6 on powerpc, but still...

Changed 8 years ago by jdemeyer

Diff for the new Sympow spkg, for review only

comment:17 Changed 8 years ago by jdemeyer

  • Status changed from needs_work to needs_review

New spkg, builds correctly on silius, both with gcc 4.6.1 and with gcc 4.3.4. Needs review.

comment:18 Changed 8 years ago by jdemeyer

  • Description modified (diff)
  • Summary changed from Sympow needs specify -ffp-contract=on and should create datafiles to Sympow needs to disable fused-multiply-add and should create datafiles

comment:19 follow-up: Changed 8 years ago by jdemeyer

The new spkg does give harmless deprecation warnings on newer gcc versions. Should I add the flag -Wno-deprecated?

comment:20 in reply to: ↑ 19 Changed 8 years ago by leif

Replying to jdemeyer:

The new spkg does give harmless deprecation warnings on newer gcc versions. Should I add the flag -Wno-deprecated?

Why? Because upstream died and nobody is going to fix the code?

IMHO it's better to not suppress such warnings, even though they might confuse or annoy users. (It's ok to suppress some other warnings, e.g. for Cython-generated files when you're 100% sure that GCC is just not smart enough to deduce that they're non-issues. But in most cases, one should avoid the warnings by changing the code.)

comment:21 Changed 8 years ago by jdemeyer

  • Milestone sage-4.7.3 deleted

Milestone sage-4.7.3 deleted

comment:22 Changed 8 years ago by cremona

  • Milestone set to sage-5.0

I happen to be at a conference with Mark Watkins (sympow author) for the next 5 days. Is there anything useful I can ask him?

comment:23 follow-up: Changed 8 years ago by cremona

What is required of a reviewer? I have no idea what is going on with these gcc issues. But I successfully built the spkg on 4.8.alpha6 on a 64-bit ubuntu machine; and not being sure what the relevant things to test are I am testing sage/schemes/elliptic_curves since I know that some sympow is used in there.

But I may be wasting my time since this is not the kind of machine on which there were problems, and the gcc version is 4.3.3.

comment:24 in reply to: ↑ 23 ; follow-up: Changed 8 years ago by leif

Replying to cremona:

What is required of a reviewer? I have no idea what is going on with these gcc issues. But I successfully built the spkg on 4.8.alpha6 on a 64-bit ubuntu machine; and not being sure what the relevant things to test are I am testing sage/schemes/elliptic_curves since I know that some sympow is used in there.

But I may be wasting my time since this is not the kind of machine on which there were problems, and the gcc version is 4.3.3.

Well, in addition to the platforms where Sympow previously caused trouble (those with [optional] non-standard floating-point and/or MAC/FMA instructions), the new spkg should also (still) work on more common platforms, so testing it on the latter doesn't hurt either.

Perhaps you could give more details w.r.t. the processor and your Ubuntu version; GCC 4.3.3 smells like Ubuntu 9.04. Otherwise -- if your time allows -- you could run a full test (ptestlong) with a recent Sage version and the new spkg here.

According to deps, only the Sage library depends on Sympow, so you don't have to rebuild any other spkgs. (You could do a ./sage -ba-force after installing the new spkg, but this shouldn't be necessary.)


I don't think there's anything we (at least me) could or should ask Mark, but Jeroen will know better.

comment:25 Changed 8 years ago by leif

P.S.: I can perhaps (re-)review the new spkg tomorrow evening; I'm not sure whether there've been changes since I last looked at it.

comment:26 in reply to: ↑ 24 ; follow-ups: Changed 8 years ago by Snark

Replying to leif:

According to deps, only the Sage library depends on Sympow, so you don't have to rebuild any other spkgs. (You could do a ./sage -ba-force after installing the new spkg, but this shouldn't be necessary.)

According to deps, nothing depends on sympow ; see my recent mail to the sage-devel mailing-list where I'm surprised by the many roots of the dependency graph.

comment:27 in reply to: ↑ 26 Changed 8 years ago by leif

Replying to Snark:

Replying to leif:

According to deps, only the Sage library depends on Sympow, so you don't have to rebuild any other spkgs. (You could do a ./sage -ba-force after installing the new spkg, but this shouldn't be necessary.)

According to deps, nothing depends on sympow ; see my recent mail to the sage-devel mailing-list where I'm surprised by the many roots of the dependency graph.

Doesn't really surprise me... ;-)

At least target all (which is the real root of the dependency graph) depends on Sympow, and the Sage library does use the Sympow executable, so could or shouldTM depend on it (although there's no need to rebuild the library upon a change/update of Sympow, but sage -b usually doesn't do much anyway).

comment:28 in reply to: ↑ 26 Changed 8 years ago by jdemeyer

Replying to Snark:

According to deps, nothing depends on sympow ;

True, no package depends on Sympow and the Sage library doesn't depend on Sympow in the sense that building it (./sage -b) doesn't need Sympow. But of course, the Sage runtime needs Sympow.

comment:29 Changed 8 years ago by Snark

Thanks for your remarks -- I can't help but notice two things:

  1. this question is crucial when one wants to work on cross-compilation, as a build-time dep must be built for the host and a runtime for the target!
  2. this bug report might not be the most appropriate place for this ;-)

comment:30 follow-up: Changed 8 years ago by leif

Replying to leif:

P.S.: I can perhaps (re-)review the new spkg tomorrow evening; I'm not sure whether there've been changes since I last looked at it.

At first glance looks ok, modulo some cosmetic inconsistencies and the like (in spkg-install):

  • Indentation
  • "Sympow" vs. "sympow" vs. "SYMPOW"; SPKG.txt (mostly) uses the latter.
  • Not all error messages are redirected to stderr.
  • Some error messages don't start with "Error" (and don't contain the word elsewhere).
  • Superfluous if [ -d $FOO ]; then rm -rf $FOO; fi (instead of just rm -rf $FOO) etc.
  • The viral trailing semicolon in
       echo "SAGE_LOCAL undefined ... exiting";
    
  • A few typos (either "[fused] multiply-add instructions" or "multiply-and-add..." -- also without a hyphen before "instructions"; "almost surely on an x86 system", ...?)

A little more relevant:

  • To go triple safe, you could add -fno-fast-math to your FLAG list.
  • $SCRIPT (which contains $SAGE_LOCAL) should be quoted near the end of spkg-install; perhaps also $file, which in contrast just depends on Sympow['s data file names], not a user setting.
  • The generated script should perhaps test $SAGE_LOCAL to give a meaningful error message in case it is undefined.
  • if [ ! -f "$TARGET/sympow" ] ... could be if [ ! -x "$TARGET/sympow" ] ....


I cannot (re)test this much; will probably do on cleo, perhaps silius, and some "unaffected" systems; interesting would also be newer Intel CPUs with AVX, although the early Sandy Bridges (for example) do not support multiply-accumulate IIRC.

I also haven't looked at the new C code recently, assuming it hasn't changed since I last did (a few month ago).

comment:31 in reply to: ↑ 30 ; follow-up: Changed 8 years ago by leif

Replying to leif:

I cannot (re)test this much; will probably do on cleo [...]

...
Linux cleo 2.6.18-128.1.1.el5 #1 SMP Mon Jan 26 13:57:09 EST 2009 ia64 ia64 ia64 GNU/Linux
Deleting directories from past builds of previous/current versions of sympow-1.018.1.p10
Extracting package /home/leif/Sage/spkgs/sympow-1.018.1.p10.spkg ...
-rw-r--r-- 1 leif sage 2212655 Oct 31 07:15 /home/leif/Sage/spkgs/sympow-1.018.1.p10.spkg
Finished extraction
****************************************************
Host system
uname -a:
Linux cleo 2.6.18-128.1.1.el5 #1 SMP Mon Jan 26 13:57:09 EST 2009 ia64 ia64 ia64 GNU/Linux
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
Target: ia64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --host=ia64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
****************************************************
patching file Configure
patching file generate.c
patching file fpu.c
The double precision of your FPU is 105 bits.
The Quad Double library used by SYMPOW assumes IEEE-754 double precision
numbers with exactly 53 bits in the mantissa (64 bits in total).

Unfortunately, we currently have no workaround for your system.
Running SYMPOW will almost certainly fail on some inputs.
Please report this problem to
sage-devel (http://groups.google.com/group/sage-devel),
mentioning in particular your operating system, processor type
and compiler version (run gcc --version).

real	0m1.626s
user	0m0.128s
sys	0m0.072s
************************************************************************
Error installing package sympow-1.018.1.p10
************************************************************************
...
leif@cleo:~/Sage/release/build/cleo/sage-4.8.rc0-gcc-4.2.1> echo $CFLAGS 
-O3 -g -fno-strict-aliasing

I think older GCCs (>= 4.0.1 or 4.2.1) should be supported as well, especially on platforms like this. Haven't tracked down what exactly goes wrong...


Ooops, just noticed John C. infected me: Ignore the prompt / directory name -- it's of course GCC 4.1.2, not 4.2.1. Do we want to support this version (on Itanium)?

comment:32 in reply to: ↑ 31 ; follow-up: Changed 8 years ago by leif

Replying to leif:

...
************************************************************************
Error installing package sympow-1.018.1.p10
************************************************************************
...
leif@cleo:~/Sage/release/build/cleo/sage-4.8.rc0-gcc-4.2.1> echo $CFLAGS 
-O3 -g -fno-strict-aliasing

More detailed:

...
gcc -v
Using built-in specs.
Target: ia64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --host=ia64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
****************************************************
patching file Configure
patching file generate.c
patching file fpu.c

Initial CFLAGS='-O3 -g -fno-strict-aliasing'
Testing '-ffp-contract=on': ...not recognized (compilation failed)
Testing '-mno-fused-madd': ...not recognized (compilation failed)
Testing '-mfpmath=sse': ...not recognized (compilation failed)
Testing '-mpc64': ...not recognized (compilation failed)
Accumulated CFLAGS='-O3 -g -fno-strict-aliasing'

The double precision of your FPU is 105 bits.
The Quad Double library used by SYMPOW assumes IEEE-754 double precision
numbers with exactly 53 bits in the mantissa (64 bits in total).

Unfortunately, we currently have no workaround for your system.
Running SYMPOW will almost certainly fail on some inputs.
...

comment:33 in reply to: ↑ 32 Changed 8 years ago by leif

  • Status changed from needs_review to needs_work

Replying to leif:

More detailed:

...
gcc -v
Using built-in specs.
Target: ia64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --host=ia64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
****************************************************
patching file Configure
patching file generate.c
patching file fpu.c
 
Initial CFLAGS='-O3 -g -fno-strict-aliasing'
Testing '-ffp-contract=on': ...not recognized (compilation failed)
Testing '-mno-fused-madd': ...not recognized (compilation failed)
Testing '-mfpmath=sse': ...not recognized (compilation failed)
Testing '-mpc64': ...not recognized (compilation failed)
Accumulated CFLAGS='-O3 -g -fno-strict-aliasing'

The double precision of your FPU is 105 bits.
The Quad Double library used by SYMPOW assumes IEEE-754 double precision
numbers with exactly 53 bits in the mantissa (64 bits in total).

Unfortunately, we currently have no workaround for your system.
Running SYMPOW will almost certainly fail on some inputs.
...

In this case -ffloat-store helps (-fno-fast-math doesn't):

...
patching file Configure
patching file generate.c
patching file fpu.c

Initial CFLAGS='-O3 -g -fno-strict-aliasing'
Testing '-ffp-contract=on': ...not recognized (compilation failed)
Testing '-mno-fused-madd': ...not recognized (compilation failed)
Testing '-mfpmath=sse': ...not recognized (compilation failed)
Testing '-mpc64': ...not recognized (compilation failed)
Testing '-ffloat-store': ...compiles ...runs (exit code 0)
Testing '-fno-fast-math': ...compiles ...runs (exit code 0)
Accumulated CFLAGS='-O3 -g -fno-strict-aliasing -ffloat-store -fno-fast-math'

The double precision of your FPU is 53 bits.
...

comment:34 Changed 8 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to needs_review

leif: I took care of most of your comments. It now uses "-ffloat-store" if needed.

Changed 8 years ago by jdemeyer

Diff for the sympow spkg p10 -> p11, for review only

comment:35 follow-up: Changed 8 years ago by jdemeyer

leif, could you please review?

comment:36 in reply to: ↑ 35 ; follow-up: Changed 8 years ago by leif

Replying to jdemeyer:

leif, could you please review?

I'm "at it"TM (already took a brief look); will yet take some time, hopefully later today...

comment:37 in reply to: ↑ 36 Changed 8 years ago by leif

Replying to leif:

I'm "at it"TM (already took a brief look); will yet take some time, hopefully later today...

P.S.: FWIW, now of course also builds and passes tests on cleo with GCC 4.1.2.

comment:38 Changed 8 years ago by leif

I wonder if we should try -ffloat-store before "resorting" to -fno-fast-math (which, depending on the platform, disables a lot), but I don't really have an idea w.r.t. the performance impact.

comment:39 Changed 8 years ago by jdemeyer

-fno-fast-math is clearly an option we want in all cases. The opposite -ffast-math makes code not IEEE compliant.

comment:40 follow-up: Changed 8 years ago by vbraun

  • Reviewers changed from Leif Leonhardy to Leif Leonhardy, Volker Braun
  • Status changed from needs_review to positive_review

Looks good to me.

Incidentally, did you look at the sympow source? Do you have an opinion on how easy it would be to replace their troublesome quad math implementation with libquadmath?

comment:41 in reply to: ↑ 40 Changed 8 years ago by jdemeyer

Replying to vbraun:

Looks good to me.

Incidentally, did you look at the sympow source?

No.

Do you have an opinion on how easy it would be to replace their troublesome quad math implementation with libquadmath?

I think libquadmath does something different from SYMPOW's "quad double". libquadmath implements Quadruple precision numbers (113 bits) whereas SYMPOW's quad double implements something like 4*53 = 212 bits of precision.

Anyway, I don't think it's that troublesome, once you understand that it needs IEEE-compliants doubles.

Thanks for the review!

comment:42 Changed 8 years ago by jdemeyer

  • Merged in set to sage-5.0.beta3
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.