Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#13278 closed enhancement (fixed)

Allow reporting of skipped optional tests

Reported by: roed Owned by: mvngu
Priority: major Milestone: sage-5.9
Component: doctest framework Keywords:
Cc: Merged in: sage-5.9.beta2
Authors: David Roe Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #12415, #14331 Stopgaps:

Description (last modified by jdemeyer)

From a discussion on sage-combinat-devel:

Optionally print extra information after doctesting: how many optional doctests were skipped, and how they were labeled ("skipped 5 doctests marked "optional -- bug").

See also #14276.

Apply 13278.2.patch

Apply 13278_script.patch to the script repo

Attachments (5)

13278_script.patch (1007 bytes) - added by roed 7 years ago.
13278.patch (55.1 KB) - added by roed 7 years ago.
13278_review.patch (4.7 KB) - added by roed 7 years ago.
13278_review2.patch (7.2 KB) - added by jdemeyer 7 years ago.
13278.2.patch (55.2 KB) - added by jdemeyer 7 years ago.

Download all attachments as: .zip

Change History (23)

comment:1 Changed 7 years ago by roed

  • Authors set to David Roe
  • Status changed from new to needs_review

It was necessary to do some cleanup in sage.doctest.sources.DocTestSource: now it just stores an options object rather than separate options that it cares about.

There will be some double counting if optional tests are marked with multiple tags.

For ReST files with inner sources we'll also miss the optional tests in the inner sources.

Both issues seem minor enough that they're not worth the extra code complexity.

Changed 7 years ago by roed

comment:2 Changed 7 years ago by roed

  • Description modified (diff)

comment:3 Changed 7 years ago by roed

  • Description modified (diff)

comment:4 follow-up: Changed 7 years ago by jdemeyer

Could you add a test to sage/doctest/test.py to test a complete sage -t --show-skipped run (also checking that optional tests which are run are not reported).

And I would enable --show-skipped automatically when --verbose is given.

Changed 7 years ago by roed

comment:5 in reply to: ↑ 4 Changed 7 years ago by roed

Replying to jdemeyer:

Could you add a test to sage/doctest/test.py to test a complete sage -t --show-skipped run (also checking that optional tests which are run are not reported).

And I would enable --show-skipped automatically when --verbose is given.

Done.

comment:6 Changed 7 years ago by jdemeyer

Why still use useabspath here (but remove it from the documentation)?

def __init__(self, path, options, useabspath=False)

especially since you replace

if self.useabspath: 

by

if self.options.abspath: 

comment:7 Changed 7 years ago by jdemeyer

And why isn't --long displayed in the output here:

sage: subprocess.call(["sage", "-t", "--long", "--show-skipped", "--optional=sage,gap", "show_skipped.rst"], cwd=tests_dir) 
Running doctests ... 
Doctesting 1 file. 
sage -t show_skipped.rst 

comment:8 Changed 7 years ago by jdemeyer

Line 405, you could replace

untested = optionals["not tested"] + optionals["not implemented"]

by

untested = optionals.pop("not tested") + optionals.pop("not implemented")

Then you don't need to check for "not tested" or "not implemented" further.

And I think the translation "known bug" -> "bug" should better be moved inside the parse_optional_tags() function itself.

comment:9 Changed 7 years ago by jdemeyer

  • Reviewers set to Jeroen Demeyer
  • Status changed from needs_review to needs_work

comment:10 Changed 7 years ago by roed

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Fixed.

Changed 7 years ago by roed

comment:11 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to needs_work
**********************************************************************
File "devel/sage/sage/doctest/test.py", line 240, in sage.doctest.test
Failed example:
    subprocess.call(["sage", "-t", "--long", "--show-skipped", "--optional=sage,gap", "show_skipped.rst"], cwd=tests_dir)
Expected:
    Running doctests ...
    Doctesting 1 file.
    sage -t --long show_skipped.rst
        1 unlabeled test not run
        2 tests not run due to known bugs
        1 other test skipped
        [3 tests, ... s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    ...
    0
Got:
    Running doctests with ID 2013-03-27-00-20-38-1d0ae1fc.
    Doctesting 1 file.
    sage -t --long show_skipped.rst
        1 unlabeled test not run
        2 tests not run due to known bugs
        0 not tested tests not run
        1 other test skipped
        [3 tests, 0.0 s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 0.0 seconds
        cpu time: 0.0 seconds
        cumulative wall time: 0.0 seconds
    0
**********************************************************************

I'll have a look.

comment:12 Changed 7 years ago by jdemeyer

  • Dependencies changed from #12415 to #12415, #14331

Changed 7 years ago by jdemeyer

comment:13 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to needs_review

13278_review2.patch (which replaces 13278_review.patch) needs review.

comment:14 Changed 7 years ago by roed

  • Status changed from needs_review to positive_review

Works for me.

comment:15 Changed 7 years ago by jdemeyer

This conflicts massively with #14331, rebasing... :-(

Changed 7 years ago by jdemeyer

comment:16 Changed 7 years ago by jdemeyer

  • Description modified (diff)

Rebased and squashed the two patches.

comment:17 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.9.beta2
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:18 Changed 7 years ago by roed

  • Component changed from doctest to doctest framework
Note: See TracTickets for help on using tickets.