New doctesting features

[mpatel]

This is an umbrella ticket for various improvements to the Sage doctesting framework. For example:

When we doctest several files in parallel or in sequence, it might be helpful to print stats relating to the test suite's progress, e.g.,

$ sage -t monoids/
sage -t  "monoids/monoid.py"
 1/10    1.9 s / 1.9 s
sage -t  "monoids/free_monoid.py"
 2/10    2.0 s / 3.9 s
[...]

What other columns would be useful? Coverage?

Please see the comments for more ideas. Suggestions are welcome!

Related: #838, #4294, #7493, #8385, #9224, #9739, #10458.

[mpatel]

If we widen the scope here a bit: We should use ​optparse, at least, to handle command-line arguments.

[mpatel]

Other possibilities, probably mentioned before and probably for other tickets:

• "Librarify:" Make it possible to doctest files and objects from the Sage command-line or notebook, e.g.,

  sage: doctest('file.sage', long=True, optional=['axiom', 'magma'])
  sage: doctest(os.path.join(SAGE_ROOT, 'devel', 'sage', 'sage', 'monoids'), processes=4)
  sage: stats = {}     # Collect errors, counts, timings, etc.
  sage: doctest(['foo.py', 'bar.pyx'], stats=stats)
  sage: def f():
  ....:     """
  ....:     sage: f()
  ....:     1
  ....:     """
  ....:     return 1
  ....: 
  sage: doctest(f)
  

• Doctest the doctesting framework! I think we could include this example, at least.

• An option to run an individual file.py's "examples" (i.e., the files example_* functions) in parallel.

[mpatel]

A way to upload anonymized test stats and system information to a remote server?

[mpatel]

Also potentially useful: doctesting the same file multiple times in parallel. I think this is just a matter of generating in sage-doctest a unique name, e.g., .doctest_foo-R3X87S.py for each run on foo.py.

[drkirkby]

I have a suspicion that some doctests fail when the system is under heavy load. It would be good if the one-minute and perhaps 5 minute load average could be recorded with any failures, along with the dates and time as I mentioned before.

[drkirkby]

Replying to mpatel:

A way to upload anonymized test stats and system information to a remote server?

Yes, but with the option of giving a name if one wishes to. It will make problems easier to manage if people did give a name + email, but certainly make it optional.

[mpatel]

Replying to mpatel:

Also potentially useful: doctesting the same file multiple times in parallel. I think this is just a matter of generating in sage-doctest a unique name, e.g., .doctest_foo-R3X87S.py for each run on foo.py.

This is #9739.

[drkirkby]

Replying to mpatel:

If we widen the scope here a bit: We should use ​optparse, at least, to handle command-line arguments.

Though if you look at that link, it says its been depreciated in favor of argparse.

Dave

[mpatel]

Replying to drkirkby:

Replying to mpatel:

If we widen the scope here a bit: We should use ​optparse, at least, to handle command-line arguments.

Though if you look at that link, it says its been depreciated in favor of argparse.

That's why I inserted "at least," but I should have more specific. We'll need to upgrade to Python 2.7 to use ​argparse readily. ​argparse used to be an independent project.

[leif]

Rob, perhaps you could state your personal wishes here, too.

(E.g. ./sage -testall ... with an option to specify the number of threads IIRC rather than always doing that serially.)

[leif]

#10458 improves cut & paste of doctest code & output from an actual Sage session, in that it allows also "....: " for line continuations. (Previously, for some reason only "..." was supported in docstrings; also the confusion with an ellipsis intended to mean arbitrary output should be reduced by this ticket.)

Another problem still has to be addressed: The original (non-preparsed) source code of continuation lines gets "lost", i.e. doesn't end up in the end-of-line comments of the generated test files (.doctest_*) as it does for ordinary lines, hence also won't be shown on doctest errors or when one doctests in verbose mode.

[roed]

#12415 will add many of these features.

[jdemeyer]

As David mentioned, there is a lot of work at #12415. And apart from that, this ticket is extremely vague.