Opened 12 years ago

Closed 12 years ago

#7390 closed enhancement (fixed)

Generate a HTML report for SageNB tests

Reported by: mpatel Owned by: boothby
Priority: minor Milestone: sage-4.3
Component: notebook Keywords:
Cc: was, timdumol Merged in:
Authors: Mitesh Patel Reviewers: Tim Dumol
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description

It would be useful to have a HTML report summarizing the results of Sage Notebook tests, including the output and traceback, if any, of each test.

We can begin with the notebook's functional test suites (cf. #7343), but we could add doctests, eventually. And with a backend server, we might select, run, and monitor tests remotely.

Attachments (5)

trac_7390-sagenb_test_report_A.patch (24.1 KB) - added by mpatel 12 years ago.
HTML test report. Part A. Apply to sagenb repo.
trac_7390-sagenb_test_report_B.patch (43.8 KB) - added by mpatel 12 years ago.
HTML test report. Part B. Apply to sagenb repo after part A.
trac_7390-sagenb_test_report_B_v2.patch (44.4 KB) - added by mpatel 12 years ago.
HTML test report. Part B Version 2. Apply to sagenb repo after part A.
trac_7390-sagenb_test_report_referee.patch (4.2 KB) - added by timdumol 12 years ago.
Removes unicode characters from the documentation. Restyles color scheme to a lighter layout.
trac_7390-sagenb_test_report_referee_v2.patch (4.9 KB) - added by mpatel 12 years ago.
Update TODO list. Version 2 of referee patch.

Download all attachments as: .zip

Change History (17)

Changed 12 years ago by mpatel

HTML test report. Part A. Apply to sagenb repo.

Changed 12 years ago by mpatel

HTML test report. Part B. Apply to sagenb repo after part A.

comment:1 Changed 12 years ago by mpatel

  • Cc was timdumol added
  • Status changed from new to needs_review

Note: The patches depend on #7343. Please apply parts A and B, in order. To test the test report generator, try

sage: from sagenb.testing.run_tests import run_and_report; run_and_report()

This should run the notebook's Selenium test suite, generate and write a self-contained report.html, and open the report in SAGE_BROWSER.

Remarks:

  • Part A adds HTMLTestRunner.
  • Part B makes significant changes to HTMLTestRunner.py and sets up the test runner for sagenb.
  • The runner captures all output, e.g., we can display information from passing tests, too.

To do, but not necessarily in this ticket:

  • Stabilize the results table's column widths, especially when toggling tracebacks.
  • Support multiple results tables.
  • Use the runner to display doctest results.
  • Use a backend server to select, run, and monitor live tests.

comment:2 Changed 12 years ago by mpatel

Mistake: run_and_report's docstring should say open_viewer defaults to True.

comment:3 Changed 12 years ago by mpatel

Doctesting may be straightforward. Given foo.py, sage-doctest preparses its triple-quoted blocks and writes them as docstrings of example_* functions in .doctest_foo.py. This file calls on itself a subclass of doctest.DocTestRunner. Since Python's doctest module can also generate unittests, we can "replace" the end of .doctest_foo.py with, e.g.,

if __name__ ==  '__main__':
    from sagenb.testing.run_tests import run_and_report
    import doctest
    run_and_report(doctest.DocTestSuite())
    sys.exit(0)

Although the test names example_* are not informative, sage -python .doctest_foo_mod.py runs the tests and makes a report!

Changed 12 years ago by mpatel

HTML test report. Part B Version 2. Apply to sagenb repo after part A.

comment:4 Changed 12 years ago by mpatel

Version 2 (of part B):

  • Solves the table width problem for Cr3, FF3.5, IE8, O10, S4 on Windows XP and Cr4, FF3.5, O10 on Linux.
  • Works around a jQuery / IE8 toggle bug.
  • Fixes run_and_report's docstring.

comment:5 Changed 12 years ago by mpatel

  • Milestone set to sage-4.3

comment:6 Changed 12 years ago by mpatel

Feel free to suggest improvements to the form and function of the report!

comment:8 in reply to: ↑ 7 Changed 12 years ago by timdumol

Replying to mpatel:

There's sample report at

If you don't mind, I can try restyling the tests. The colors are a bit jarring, in my opinion.

comment:9 Changed 12 years ago by mpatel

Feel free.

Changed 12 years ago by timdumol

Removes unicode characters from the documentation. Restyles color scheme to a lighter layout.

comment:10 Changed 12 years ago by timdumol

  • Reviewers set to Tim Dumol

This patch removes unicode characters from the documentation that would otherwise require magic comments (feel free to do so instead). The color scheme has been inverted to a lighter layout, with colors taken from the color palette on Wikipedia.

Things look great. The only thing I might want added are timings for each individual test, but that can go in another patch. Positive review. All that's needed now is for someone to review the referee patch.

Changed 12 years ago by mpatel

Update TODO list. Version 2 of referee patch.

comment:11 Changed 12 years ago by mpatel

  • Status changed from needs_review to positive_review

Looks good to me. I'll change this status to "positive review."

comment:12 Changed 12 years ago by was

  • Report Upstream set to N/A
  • Resolution set to fixed
  • Status changed from positive_review to closed

merged into sagenb for sage-4.3

Note: See TracTickets for help on using tickets.