Opened 13 years ago

Closed 9 years ago

#1206 closed enhancement (duplicate)

doc testing support for numerical computations with randomish output is insufficient

Reported by: jkantor Owned by: failure
Priority: minor Milestone: sage-duplicate/invalid/wontfix
Component: doctest coverage Keywords:
Cc: Merged in:
Authors: Reviewers: Maarten Derickx
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description

The current way in which doc tests of numerical computations that produce randomish output are performed is not sufficient to actually detect regressions.

Currently if the word #random follows the test, it is run but the output is not compared, this only tests that there was no failure doing something but not that what was computed was in any way correct or what was expected.

For computations with randomish output, what should be checked is that the difference between all the floating values in what is computed and in the doc string are less than some bound 1e-8 or something, or maybe the bound should be specified, so that

#random 1e-8 would check that the the computation differs from the output in the doc string by 1e-8.

Change History (10)

comment:1 Changed 13 years ago by mabshoff

We have the "12.123.." notation to indicate that the last n, in this case 2 decimals are affected by numerical precision issues. Shouldn't that work for you, too?

Cheers,

Michael

comment:2 Changed 13 years ago by jkantor

That should be sufficient, it doesn't seem to be documented in the programming guide, though I may have just missed it.

comment:3 Changed 13 years ago by mabshoff

Ok, since was doubted that the following case could happen:

Expected:
    1.0000000000000000
Got:
    0.9999999999999999

here a real world example from 2.8.14 on Solaris:

File "complex_double.pyx", line 1496:
    sage: z^2 - z + 1
Expected:
    2.22044604925e-16 + 1.11022302463e-16*I
Got:
    2.22044604925e-16 + 2.22044604925e-16*I

Cheers,

Michael

comment:4 Changed 13 years ago by mabshoff

Actually, the case

Expected:
    1.0000000000000000
Got:
    0.9999999999999999

Did happen a couple times in the 2.8.15 release cycle. The usual approach was to change the doctest to avoid results where such small rounding issues would cause "..." the doctest to be useless.

Cheers,

Michael

comment:5 Changed 13 years ago by mabshoff

  • Milestone changed from sage-feature to sage-2.9

comment:6 Changed 13 years ago by mhansen

I believe implementing this would require a major restructuring of how doctesting is carried out.

comment:7 Changed 12 years ago by AlexGhitza

  • Type changed from defect to enhancement

comment:8 Changed 10 years ago by mderickx

  • Milestone changed from sage-4.7.2 to sage-duplicate/invalid/wontfix
  • Report Upstream set to N/A
  • Status changed from new to needs_review

There is a patch up for the same problem at #10952

comment:9 Changed 9 years ago by kini

  • Status changed from needs_review to positive_review

Setting this to positive_review so the release manager will see it.

comment:10 Changed 9 years ago by jdemeyer

  • Resolution set to duplicate
  • Reviewers set to Maarten Derickx
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.