Opened 10 years ago

Closed 8 years ago

#9225 closed enhancement (invalid)

New doctesting features

Reported by: mpatel Owned by: mvngu
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: doctest coverage Keywords:
Cc: wjp, leif, rbeezer Merged in:
Authors: Reviewers: David Roe, Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by roed)

This is an umbrella ticket for various improvements to the Sage doctesting framework. For example:

When we doctest several files in parallel or in sequence, it might be helpful to print stats relating to the test suite's progress, e.g.,

$ sage -t monoids/
sage -t  "monoids/monoid.py"
 1/10    1.9 s / 1.9 s
sage -t  "monoids/free_monoid.py"
 2/10    2.0 s / 3.9 s
[...]

What other columns would be useful? Coverage?

Please see the comments for more ideas. Suggestions are welcome!

Related: #838, #4294, #7493, #8385, #9224, #9739, #10458.

Change History (19)

comment:1 Changed 10 years ago by wjp

  • Cc wjp added

comment:2 follow-up: Changed 10 years ago by mpatel

If we widen the scope here a bit: We should use optparse, at least, to handle command-line arguments.

comment:3 Changed 10 years ago by leif

  • Cc leif added

comment:4 Changed 10 years ago by mpatel

  • Priority changed from minor to major

Other possibilities, probably mentioned before and probably for other tickets:

  • "Librarify:" Make it possible to doctest files and objects from the Sage command-line or notebook, e.g.,
    sage: doctest('file.sage', long=True, optional=['axiom', 'magma'])
    sage: doctest(os.path.join(SAGE_ROOT, 'devel', 'sage', 'sage', 'monoids'), processes=4)
    sage: stats = {}     # Collect errors, counts, timings, etc.
    sage: doctest(['foo.py', 'bar.pyx'], stats=stats)
    sage: def f():
    ....:     """
    ....:     sage: f()
    ....:     1
    ....:     """
    ....:     return 1
    ....: 
    sage: doctest(f)
    
  • Doctest the doctesting framework! I think we could include this example, at least.
  • An option to run an individual file.py's "examples" (i.e., the files example_* functions) in parallel.

comment:5 follow-up: Changed 10 years ago by mpatel

A way to upload anonymized test stats and system information to a remote server?

comment:6 follow-up: Changed 10 years ago by mpatel

Also potentially useful: doctesting the same file multiple times in parallel. I think this is just a matter of generating in sage-doctest a unique name, e.g., .doctest_foo-R3X87S.py for each run on foo.py.

comment:7 Changed 10 years ago by drkirkby

I have a suspicion that some doctests fail when the system is under heavy load. It would be good if the one-minute and perhaps 5 minute load average could be recorded with any failures, along with the dates and time as I mentioned before.

comment:8 in reply to: ↑ 5 Changed 10 years ago by drkirkby

Replying to mpatel:

A way to upload anonymized test stats and system information to a remote server?

Yes, but with the option of giving a name if one wishes to. It will make problems easier to manage if people did give a name + email, but certainly make it optional.

comment:9 in reply to: ↑ 6 Changed 10 years ago by mpatel

Replying to mpatel:

Also potentially useful: doctesting the same file multiple times in parallel. I think this is just a matter of generating in sage-doctest a unique name, e.g., .doctest_foo-R3X87S.py for each run on foo.py.

This is #9739.

comment:10 Changed 10 years ago by mpatel

  • Description modified (diff)
  • Summary changed from Indicate progress and elapsed time when running multiple doctests to New doctesting features

comment:11 in reply to: ↑ 2 ; follow-up: Changed 10 years ago by drkirkby

Replying to mpatel:

If we widen the scope here a bit: We should use optparse, at least, to handle command-line arguments.

Though if you look at that link, it says its been depreciated in favor of argparse.

Dave

comment:12 in reply to: ↑ 11 Changed 10 years ago by mpatel

Replying to drkirkby:

Replying to mpatel:

If we widen the scope here a bit: We should use optparse, at least, to handle command-line arguments.

Though if you look at that link, it says its been depreciated in favor of argparse.

That's why I inserted "at least," but I should have more specific. We'll need to upgrade to Python 2.7 to use argparse readily. argparse used to be an independent project.

comment:13 Changed 10 years ago by leif

  • Cc rbeezer added

Rob, perhaps you could state your personal wishes here, too.

(E.g. ./sage -testall ... with an option to specify the number of threads IIRC rather than always doing that serially.)

comment:14 Changed 10 years ago by leif

  • Description modified (diff)

#10458 improves cut & paste of doctest code & output from an actual Sage session, in that it allows also "....: " for line continuations. (Previously, for some reason only "..." was supported in docstrings; also the confusion with an ellipsis intended to mean arbitrary output should be reduced by this ticket.)

Another problem still has to be addressed: The original (non-preparsed) source code of continuation lines gets "lost", i.e. doesn't end up in the end-of-line comments of the generated test files (.doctest_*) as it does for ordinary lines, hence also won't be shown on doctest errors or when one doctests in verbose mode.

comment:15 Changed 9 years ago by roed

  • Description modified (diff)

comment:16 Changed 9 years ago by roed

#12415 will add many of these features.

comment:17 Changed 8 years ago by jdemeyer

  • Milestone set to sage-duplicate/invalid/wontfix
  • Reviewers set to David Roe, Jeroen Demeyer
  • Status changed from new to needs_review

As David mentioned, there is a lot of work at #12415. And apart from that, this ticket is extremely vague.

comment:18 Changed 8 years ago by jdemeyer

  • Status changed from needs_review to positive_review

comment:19 Changed 8 years ago by jdemeyer

  • Resolution set to invalid
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.