Opened 9 years ago
Last modified 8 years ago
#12415 closed enhancement
Update doctesting framework — at Version 88
Reported by:  robertwb  Owned by:  mvngu 

Priority:  major  Milestone:  sage5.9 
Component:  doctest framework  Keywords:  
Cc:  kini, ohanar, jhpalmieri  Merged in:  
Authors:  David Roe, Robert Bradshaw  Reviewers:  Jeroen Demeyer 
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384  Stopgaps: 
Description (last modified by )
There are several improvements that would be good to make, including but not limited to:
 Forking rather than starting up a new Sage each time.
 Customizable comparison rather than mutating doctests into new doctests (e.g. followup to #10952).
 Eliminate need to write temporary files for every doctested file.
 Constant headaches caused by the difference in doctesting library vs. nonlibrary files.
 The ability to drop into a debugger if a doctest raises an exception.
Robert's code is at
http://code.google.com/p/sagemathtimer/
and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)
Apply 12415_stderr_vs_51b5.patch, 12415_doctest_fixes.patch, 12415_framework.patch, 12415_doc.patch
Apply 12415_script.patch to the scripts repo
Apply 12415_sagenb_fixes.patch to the sagenb repo (fixes doctest errors)
Apply 12415_spkg_bin_sage.patch to the root repo
Change History (88)
comment:1 Changed 9 years ago by
 Description modified (diff)
comment:2 Changed 9 years ago by
 Description modified (diff)
comment:3 Changed 9 years ago by
 Description modified (diff)
comment:4 Changed 9 years ago by
 Description modified (diff)
comment:5 Changed 9 years ago by
 Description modified (diff)
comment:6 Changed 9 years ago by
 Dependencies set to #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384
comment:7 Changed 9 years ago by
 Dependencies changed from #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384
comment:8 Changed 9 years ago by
 Cc kini added
comment:9 Changed 9 years ago by
12415_2.patch seems to contain a lot of tempfile changes, could you split out the doctest changes into a separate patch (or are these affecting doctests?)
comment:10 Changed 9 years ago by
 Cc ohanar added
comment:11 Changed 9 years ago by
apparently even I forget my trac username is not what it should be
comment:12 Changed 9 years ago by
 Dependencies changed from #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384
comment:13 Changed 9 years ago by
 Dependencies changed from #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384
comment:14 Changed 9 years ago by
 Dependencies changed from #13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384
comment:15 Changed 9 years ago by
I've moved the patches dealing with temporary files to #13147.
comment:16 Changed 9 years ago by
 Cc jhpalmieri added
comment:17 Changed 9 years ago by
 Description modified (diff)
comment:18 Changed 9 years ago by
 Description modified (diff)
comment:19 Changed 9 years ago by
 Status changed from new to needs_review
Ready for review!
There are still a few known issues:
 valgrinding under sage t doesn't work. I haven't used valgrind much and don't know how to resolve this.
 The fix at #13147 probably needs work: I'm getting a failure in
sage.rings.polynomial.multi_polynomial_libsingular
to do with garbage collection. I'm working on it.  I'm in the process of updating the developer guide.
But I wanted to get some other people involved in reviewing.
comment:20 Changed 9 years ago by
This may be a silly question, but how do I use the new framework? Don't we need changes to the scripts repo, too? (If I just run sage t FILES
, I don't see the various messages, like Doctesting ...
, that should be printed by run_doctests
are not printed, so sage t
is not yet using the new framework.)
One of the patches also didn't apply cleanly to 5.1.beta5:
hg qimport P ~/Downloads/12415_stderr.patch adding 12415_stderr.patch to series file applying 12415_stderr.patch patching file sage/lfunctions/lcalc.py Hunk #1 FAILED at 226 1 out of 2 hunks FAILED  saving rejects to file sage/lfunctions/lcalc.py.rej patch failed, unable to continue (try v) patch failed, rejects left in working dir errors during apply, please fix and refresh 12415_stderr.patch
comment:21 Changed 9 years ago by
comment:22 Changed 9 years ago by
 Description modified (diff)
comment:23 Changed 9 years ago by
The description is updated: let me know if you have any more problems. I think you need to apply 12415_stderr_vs_51b5.patch instead of 12415_stderr.patch.
comment:24 Changed 9 years ago by
Regarding 12415_spkg_bin_sage.patch, the file in question should be part of the root repo.
comment:25 Changed 9 years ago by
One regression: with the old setup, running sage tp FILES
(with no argument for tp
) would choose the number of threads automatically. With the new setup, this command gives an error, because FILES
is not an integer. Therefore I think that, for example, make ptestlong
will fail. (The file sageruntests says that the default value for nthreads is 1, but I don't think optparse is dealing with the missing argument properly here. Also, passing the argument 0 causes doctests to never happen: maybe it's using 0 threads instead of minimum(8, cpu_count())
.
comment:26 Changed 9 years ago by
I would suggest the following two changes: for the scripts repo:

sageruntests
diff git a/sageruntests b/sageruntests
a b import optparse, os, sys 5 5 if __name__ == "__main__": 6 6 parser = optparse.OptionParser() 7 7 8 parser.add_option("p", "nthreads", type= int, default=1, metavar="N", help="tests in parallel using N threads with 0 interpreted as minimum(8, cpu_count())")8 parser.add_option("p", "nthreads", type="string", default=1, metavar="N", help="tests in parallel using N threads with 0 interpreted as minimum(8, cpu_count())") 9 9 parser.add_option("serial", action="store_true", default=False, help="run tests in a single process in series") 10 10 parser.add_option("timeout", type=int, default=1, help="timeout (in seconds) for doctesting one file") 11 11 parser.add_option("a", "all", action="store_true", default=False, help="test all files in the Sage library") … … if __name__ == "__main__": 49 49 parser.add_option("stats_path", "statspath", default=os.path.join(os.path.expanduser("~/.sage/timings2.json")), \ 50 50 help="path to a json dictionary for the latest run storing a timing for each file") 51 51 52 options, args = parser.parse_args() 53 try: 54 options.nthreads = int(options.nthreads) 55 except ValueError: 56 args.insert(0, options.nthreads.strip()) 57 options.nthreads = 0 58 59 if options.nthreads == 0: 60 try: 61 options.nthreads = int(os.environ['SAGE_NUM_THREADS_PARALLEL']) 62 except KeyError: 63 options.nthreads = 1 64 52 65 from sage.doctest.control import DocTestController 53 DC = DocTestController( *parser.parse_args())66 DC = DocTestController(options, args) 54 67 err = DC.run() 55 68 sys.exit(err)
and for the Sage library:

sage/doctest/control.py
diff git a/sage/doctest/control.py b/sage/doctest/control.py
a b class DocTestController(SageObject): 433 433 else: 434 434 nother += 1 435 435 if nfiles + ndbsources + nother: 436 self.log("Doctesting %s."%(", ".join((["%s file%s"%(nfiles, "s" if nfiles > 1 else "")] if nfiles else []) + 437 (["%s other sources"%nother] if nother else [])))) 436 self.log("Doctesting %s"%(", ".join((["%s file%s"%(nfiles, "s" if nfiles > 1 else "")] if nfiles else []) + 437 (["%s other sources"%nother] if nother else []))) 438 + " using %s threads."%self.options.nthreads if self.options.nthreads > 1 else ".") 438 439 self.reporter = DocTestReporter(self) 439 440 self.dispatcher = DocTestDispatcher(self) 440 441 try:
Or something like that. I guess you should also print the number of threads when you print "Doctesting entire Sage library.", etc.
comment:27 Changed 9 years ago by
I've updated the option parsing to allow p to take no options (though I used a callback so that the p doesn't have to be last). I changed 12415_spkg_bin_sage.patch to be an actual patch against the root repository. I also made a small change to the debug functionality (printing the doctest that failed before dropping into the debugger).
Since I'm modifying four repositories, I'll just update the patches rather than creating lots of referee patches (unless the referee objects!).
comment:28 Changed 9 years ago by
 Description modified (diff)
comment:29 Changed 9 years ago by
sage/parallel/decorate.py
: Use
Unhandled SIG...
(on systems, this would give a SIGBUS: Bus Error instead)
comment:30 Changed 9 years ago by
Please explain why you're adding nodoctest
to some of the files in doc/
?
comment:31 Changed 9 years ago by
Are all the options in sageruntests
actually implemented? (it seems some of them are meant for #12720, these should not be added here)
comment:32 Changed 9 years ago by
I guess you're no longer using the scripts sagetest
, sageptest
, sagedoctest
, sagemaketest
, perhaps sagetestnew
and sagetestimport
. Remove these.
comment:33 followup: ↓ 34 Changed 9 years ago by
Why does the doctesting framework need to be imported from sage.all
?
comment:34 in reply to: ↑ 33 Changed 9 years ago by
Replying to jdemeyer:
Why does the doctesting framework need to be imported from
sage.all
?
I don't think it needs to be, but it makes it easy to test (and debug) doctests from within Sage:
sage: run_doctests(sage.rings.integer) ...
comment:35 Changed 9 years ago by
You have two instances of
except:
in the new code.
The one for os.kill()
should just catch OSError
, for the other one replace
except KeyboardInterrupt: pass except:
by
except Exception:
comment:36 Changed 9 years ago by
Replace
import multiprocessing options.nthreads = os.getenv('SAGE_NUM_THREADS_PARALLEL',min(8, multiprocessing.cpu_count()))
by
options.nthreads = int(os.getenv('SAGE_NUM_THREADS_PARALLEL', 1))
(don't reinvent the sagenumthreads.py
wheel)
comment:37 Changed 9 years ago by
 Status changed from needs_review to needs_work
comment:38 followup: ↓ 42 Changed 9 years ago by
 Reviewers set to Jeroen Demeyer
Planning to make a reviewer patch, so hang on...
comment:39 Changed 9 years ago by
comment:40 Changed 9 years ago by
This actually fails on the very first thing I tried (after applying all patches and doing sage b
):
jdemeyer@sage:/release/sage5.1.beta5doctest$ ./sage t devel/sage/sage/rings/padics/ argv ['/release/sage5.1.beta5doctest/local/bin/sageruntests', 'devel/sage/sage/rings/padics/'] Traceback (most recent call last): File "/release/sage5.1.beta5doctest/local/bin/sageruntests", line 68, in <module> DC = DocTestController(*parser.parse_args()) File "/release/sage5.1.beta5doctest/local/lib/python2.7/sitepackages/sage/doctest/control.py", line 116, in __init__ if options.nthreads == 0: AttributeError: Values instance has no attribute 'nthreads'
Am I doing something wrong?
comment:41 Changed 9 years ago by
Building from scratch and doing make ptest
gives:
argv ['/release/merger/sage5.1.beta512415/local/bin/sageruntests', 'p', 'sagenb', 'devel/sage/doc/common', 'devel/sage/doc/de', 'devel/sage/doc/en', 'devel/sage/doc/fr', 'devel/sage/doc/ru', 'devel/sage/doc/tr', 'devel/sage/sage'] next_arg: sagenb Running doctests with ID 20120625125722d71deecd. Doctesting the Sage notebook. Doctesting 2067 files using 6 threads Usage: sage docbuild [OPTIONS] DOCUMENT (FORMAT  COMMAND) sageruntests: error: no such option: p ********************************************************************** File "programming", line 620, in programming Failed example: for i in range(5): Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 572, in _run compileflags, 1) File "<doctest programming[0]>", line 1 for i in range(5): ^ SyntaxError: unexpected EOF while parsing ********************************************************************** 1 items had failures: 1 of 113 in programming ***Test Failed*** 1 failures. ********************************************************************** File "programming", line 578, in programming Failed example: for i in range(5): Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 572, in _run compileflags, 1) File "<doctest programming[0]>", line 1 for i in range(5): ^ SyntaxError: unexpected EOF while parsing ********************************************************************** 1 items had failures: 1 of 113 in programming ***Test Failed*** 1 failures. ********************************************************************** File "sage.misc.sh", line 20, in sage.misc.sh.Sh.eval Failed example: sh.eval('''echo "Hello there"\nif [ $? eq 0 ]; then\necho "good"\nfi''') Expected: /... '' Got: Hello there good /release/merger/sage5.1.beta512415 '' ********************************************************************** File "interfaces", line 331, in interfaces Failed example: maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\ y*sin(x/2)]", "[x, 4, 4]", "[y, 4, 4]",\ '[plot_format, openmath]') # not tested Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 572, in _run compileflags, 1) File "<doctest interfaces[0]>", line 2 maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\ y*sin(x/2)]", "[x, 4, 4]", "[y, 4, 4]",\ ^ SyntaxError: unexpected character after line continuation character ********************************************************************** 1 items had failures: 1 of 57 in interfaces ***Test Failed*** 1 failures. ********************************************************************** 1 items had failures: 1 of 1 in sage.misc.sh.Sh.eval ***Test Failed*** 1 failures. ********************************************************************** File "sage.doctest.control", line 246, in sage.doctest.control.DocTestController.add_files Failed example: DC = DocTestController(DD, []) Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 573, in _run self.execute(example, compiled, test.globs) File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 835, in execute exec compiled in globs File "<doctest sage.doctest.control.DocTestController.add_files[2]>", line 1, in <module> DC = DocTestController(DD, []) File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/control.py", line 127, in __init__ self.logfile = open(options.logfile, 'a') IOError: [Errno 2] No such file or directory: '/release/merger/sage5.1.beta512415/home/.sage//tmp/test.log' ********************************************************************** File "sage.doctest.control", line 247, in sage.doctest.control.DocTestController.add_files Failed example: DC.add_files() Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 573, in _run self.execute(example, compiled, test.globs) File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 835, in execute exec compiled in globs File "<doctest sage.doctest.control.DocTestController.add_files[3]>", line 1, in <module> DC.add_files() NameError: name 'DC' is not defined ********************************************************************** File "sage.doctest.control", line 249, in sage.doctest.control.DocTestController.add_files Failed example: (os.path.join(os.environ['SAGE_ROOT'], 'devel', 'sage', 'sage'), True) in DC.files Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 573, in _run self.execute(example, compiled, test.globs) File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 835, in execute exec compiled in globs File "<doctest sage.doctest.control.DocTestController.add_files[4]>", line 1, in <module> (os.path.join(os.environ['SAGE_ROOT'], 'devel', 'sage', 'sage'), True) in DC.files NameError: name 'DC' is not defined ********************************************************************** File "sage.doctest.control", line 418, in sage.doctest.control.DocTestController.run_doctests Failed example: DC.run_doctests() Expected: Doctesting 1 file. sage t .../sage/rings/homset.py [... tests, ... s]  All tests passed!  Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Got: . sage t /release/merger/sage5.1.beta512415/devel/sage/sage/rings/homset.py [15 tests, 0.3 s]  All tests passed!  Total time for all tests: 0.8 seconds cpu time: 0.1 seconds cumulative wall time: 0.3 seconds ********************************************************************** File "interfaces", line 318, in interfaces Failed example: maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\ y*sin(x/2)]", "[x, 4, 4]", "[y, 4, 4]",\ '[plot_format, openmath]') # not tested Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 572, in _run compileflags, 1) File "<doctest interfaces[0]>", line 2 maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\ y*sin(x/2)]", "[x, 4, 4]", "[y, 4, 4]",\ ^ SyntaxError: unexpected character after line continuation character ********************************************************************** 1 items had failures: 1 of 57 in interfaces ***Test Failed*** 1 failures. ********************************************************************** File "sage.doctest.control", line 479, in sage.doctest.control.DocTestController.cleanup Failed example: DC.run() Expected: Running doctests with ID ... Doctesting 1 file. sage t .../rings/infinity.py [... tests, ... s]  All tests passed!  Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds 0 Got: Running doctests with ID 20120625125735cf5e164b. . sage t /release/merger/sage5.1.beta512415/devel/sage/sage/rings/infinity.py [220 tests, 12.8 s]  All tests passed!  Total time for all tests: 13.8 seconds cpu time: 1.0 seconds cumulative wall time: 12.8 seconds 0 ********************************************************************** File "sage.doctest.control", line 619, in sage.doctest.control.DocTestController.run Failed example: DC.run() Expected: Running doctests with ID ... Doctesting 1 file. sage t .../sage/sets/non_negative_integers.py [... tests, ... s]  All tests passed!  Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds 0 Got: Running doctests with ID 20120625125749f475b4d8. . sage t /release/merger/sage5.1.beta512415/devel/sage/sage/sets/non_negative_integers.py [41 tests, 0.5 s]  All tests passed!  Total time for all tests: 1.4 seconds cpu time: 0.1 seconds cumulative wall time: 0.5 seconds 0 ********************************************************************** File "sage.doctest.control", line 660, in sage.doctest.control.run_doctests Failed example: run_doctests(sage.rings.infinity) Expected: Doctesting .../sage/rings/infinity.py Running doctests with ID ... Doctesting 1 file. sage t .../sage/rings/infinity.py [... tests, ... s]  All tests passed!  Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Got: Doctesting /release/merger/sage5.1.beta512415/devel/sage/sage/rings/infinity.py Running doctests with ID 2012062512575038f106c7. . sage t /release/merger/sage5.1.beta512415/devel/sage/sage/rings/infinity.py [220 tests, 20.3 s]  All tests passed!  Total time for all tests: 22.3 seconds cpu time: 1.0 seconds cumulative wall time: 20.3 seconds ********************************************************************** 5 items had failures: 3 of 13 in sage.doctest.control.DocTestController.add_files 1 of 10 in sage.doctest.control.DocTestController.cleanup 1 of 6 in sage.doctest.control.DocTestController.run 1 of 7 in sage.doctest.control.DocTestController.run_doctests 1 of 1 in sage.doctest.control.run_doctests ***Test Failed*** 7 failures. ********************************************************************** File "sage.parallel.decorate", line 559, in sage.parallel.decorate.fork Failed example: g() Expected: <BLANKLINE>  Unhandled SIGSEGV: A segmentation fault occurred in Sage. This probably occurred because a *compiled* component of Sage has a bug in it and is not properly wrapped with sig_on(), sig_off(). You might want to run Sage under gdb with 'sage gdb' to debug this. Sage will now terminate.  'NO DATA' Got: /release/merger/sage5.1.beta512415/local/lib/libcsage.so(print_backtrace+0x2b)[0x2b39ebbaf66e] /release/merger/sage5.1.beta512415/local/lib/libcsage.so(sigdie+0x14)[0x2b39ebbaf69b] /release/merger/sage5.1.beta512415/local/lib/libcsage.so(sage_signal_handler+0x20b)[0x2b39ebbaf189] /lib/libpthread.so.0[0x2b39e86417d0] /lib/libc.so.6(strlen+0x30)[0x2b39e8f532f0] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyString_FromString+0x1b)[0x2b39e82e486b] /release/merger/sage5.1.beta512415/home/.sage//temp/sage.math.washington.edu/28931//spyx//_release_merger_sage_5_1_beta5_12415_home__sage_temp_sage_math_washington_edu_28931_tmp_0_spyx/_release_merger_sage_5_1_beta5_12415_home__sage_temp_sage_math_washington_edu_28931_tmp_0_spyx_0.so[0x2aaaaaab0abd] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a63)[0x2b39e833b2a3] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82c1979] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x37bf)[0x2b39e8338fff] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82b6b09] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82c952d] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82c9acb] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82f0568] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1299)[0x2b39e8336ad9] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ae4)[0x2b39e833b324] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x2b39e833d432] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x714e)[0x2b39e833c98e] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ae4)[0x2b39e833b324] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ae4)[0x2b39e833b324] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82c1979] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82a487f] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82f563a] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x37bf)[0x2b39e8338fff] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82c1979] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82a487f] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82f523c] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82f0568] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1299)[0x2b39e8336ad9] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82c1979] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82a487f] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82f523c] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0[0x2b39e82f0568] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1299)[0x2b39e8336ad9] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x2b39e833d432] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xc1)[0x2b39e83611b1] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0x1f9)[0x2b39e8361489] /release/merger/sage5.1.beta512415/local/lib/libpython2.7.so.1.0(Py_Main+0xb15)[0x2b39e8373e65] /lib/libc.so.6(__libc_start_main+0xf4)[0x2b39e8ef61f4] python[0x400679] <BLANKLINE>  Unhandled SIGSEGV: A segmentation fault occurred in Sage. This probably occurred because a *compiled* component of Sage has a bug in it and is not properly wrapped with sig_on(), sig_off(). You might want to run Sage under gdb with 'sage gdb' to debug this. Sage will now terminate.  'NO DATA' ********************************************************************** 1 items had failures: 1 of 17 in sage.parallel.decorate.fork ***Test Failed*** 1 failures. ********************************************************************** File "sage.tests.startup", line 6, in sage.tests.startup Failed example: if os.uname()[1] == 'sage.math.washington.edu': print float(os.popen("sage startuptime>/dev/null; sage startuptimegrep sage.all").readlines()[0].split()[1]) < 2.0 else: print True # nothing when not on sage.math Expected: True Got: False ********************************************************************** File "sage.rings.polynomial.multi_polynomial_libsingular", line 420, in sage.rings.polynomial.multi_polynomial_libsingular.MPolynomialRing_libsingular.__copy__ Failed example: len(ring_refcount_dict) == n Expected: True Got: False ********************************************************************** File "sage.rings.polynomial.plural", line 2738, in sage.rings.polynomial.plural.new_NRing Failed example: z*x Expected: x*z Got: Exception KeyError: (The ring pointer 0x2b3a04886e90,) in 'sage.libs.singular.ring.singular_ring_delete' ignored x*z ********************************************************************** File "sage.rings.polynomial.plural", line 2856, in sage.rings.polynomial.plural.SCA Failed example: E = SCA(QQ, ['x', 'y', 'z'], [0, 1], order = 'degrevlex') Expected nothing Got: Exception KeyError: (The ring pointer 0x2b3a048af1a0,) in 'sage.libs.singular.ring.singular_ring_delete' ignored ********************************************************************** File "sage.matrix.benchmark", line 11, in sage.matrix.benchmark Failed example: b.report([b.det_ZZ], 'Test', systems=['sage']) Expected: ====================================================================== Test ====================================================================== ... ====================================================================== Got: Linux sage.math.washington.edu 2.6.2428server #1 SMP Fri Feb 11 18:08:32 UTC 2011 x86_64 GNU/Linux ====================================================================== Test ====================================================================== <BLANKLINE> <BLANKLINE>  Dense integer determinant over ZZ. Given an n x n matrix A over ZZ with random entries between min and max, inclusive, compute det(A). <BLANKLINE> ********************************************************************** 2 items had failures: 1 of 8 in sage.rings.polynomial.plural.SCA 1 of 17 in sage.rings.polynomial.plural.new_NRing ***Test Failed*** 2 failures. INPUT: <BLANKLINE>  ``n``  matrix dimension (default: ``200``)  ``min``  minimal value for entries of matrix (default: ``1``)  ``max``  maximal value for entries of matrix (default: ``100``)  ``system``  either 'sage' or 'magma' (default: 'sage') <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.det_ZZ(200) sage: tm = b.det_ZZ(200, system='magma') # optional  magma sage 1.560 ====================================================================== ********************************************************************** 1 items had failures: 1 of 16 in sage.rings.polynomial.multi_polynomial_libsingular.MPolynomialRing_libsingular.__copy__ ***Test Failed*** 1 failures. ********************************************************************** 1 items had failures: 1 of 2 in sage.tests.startup ***Test Failed*** 1 failures. ********************************************************************** File "sage.matrix.benchmark", line 38, in sage.matrix.benchmark.report Failed example: b.report([b.det_ZZ], 'Test', systems=['sage']) Expected: ====================================================================== Test ====================================================================== ... ====================================================================== Got: Linux sage.math.washington.edu 2.6.2428server #1 SMP Fri Feb 11 18:08:32 UTC 2011 x86_64 GNU/Linux ====================================================================== Test ====================================================================== <BLANKLINE> <BLANKLINE>  Dense integer determinant over ZZ. Given an n x n matrix A over ZZ with random entries between min and max, inclusive, compute det(A). <BLANKLINE> INPUT: <BLANKLINE>  ``n``  matrix dimension (default: ``200``)  ``min``  minimal value for entries of matrix (default: ``1``)  ``max``  maximal value for entries of matrix (default: ``100``)  ``system``  either 'sage' or 'magma' (default: 'sage') <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.det_ZZ(200) sage: tm = b.det_ZZ(200, system='magma') # optional  magma sage 1.330 ====================================================================== ********************************************************************** File "sage.doctest.forker", line 925, in sage.doctest.forker.SageDocTestRunner.report_unexpected_exception Failed example: "ArithmeticError" in tb Expected: True Got: False ********************************************************************** File "sage.doctest.forker", line 1185, in sage.doctest.forker.DocTestWorker.__init__ Failed example: run_doctests(sage.rings.big_oh) # indirect doctest Expected: Doctesting .../sage/rings/big_oh.py Running doctests with ID ... Doctesting 1 file. sage t .../sage/rings/big_oh.py [... tests, ... s]  All tests passed!  Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Got: Doctesting /release/merger/sage5.1.beta512415/devel/sage/sage/rings/big_oh.py Running doctests with ID 20120625125929e5a039b5. . sage t /release/merger/sage5.1.beta512415/devel/sage/sage/rings/big_oh.py [17 tests, 1.4 s]  All tests passed!  Total time for all tests: 1.9 seconds cpu time: 0.4 seconds cumulative wall time: 1.4 seconds ********************************************************************** File "sage.doctest.forker", line 1211, in sage.doctest.forker.DocTestWorker.run Failed example: run_doctests(sage.symbolic.units) # indirect doctest Expected: Doctesting .../sage/symbolic/units.py Running doctests with ID ... Doctesting 1 file. sage t .../sage/symbolic/units.py [... tests, ... s]  All tests passed!  Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Got: Doctesting /release/merger/sage5.1.beta512415/devel/sage/sage/symbolic/units.py Running doctests with ID 20120625125931e804e534. . sage t /release/merger/sage5.1.beta512415/devel/sage/sage/symbolic/units.py [82 tests, 0.4 s]  All tests passed!  Total time for all tests: 0.7 seconds cpu time: 0.2 seconds cumulative wall time: 0.4 seconds ********************************************************************** 3 items had failures: 1 of 1 in sage.doctest.forker.DocTestWorker.__init__ 1 of 1 in sage.doctest.forker.DocTestWorker.run 1 of 18 in sage.doctest.forker.SageDocTestRunner.report_unexpected_exception ***Test Failed*** 3 failures. ********************************************************************** File "sage.tests.cmdline", line 270, in sage.tests.cmdline.test_executable Failed example: ret Expected: 0 Got: 1 ********************************************************************** File "sage.tests.cmdline", line 272, in sage.tests.cmdline.test_executable Failed example: out.find("All tests passed!") >= 0 Expected: True Got: False ********************************************************************** File "sage.tests.cmdline", line 275, in sage.tests.cmdline.test_executable Failed example: ret Expected: 0 Got: 1 ********************************************************************** File "sage.tests.cmdline", line 277, in sage.tests.cmdline.test_executable Failed example: out.find("All tests passed!") >= 0 Expected: True Got: False ********************************************************************** File "sage.tests.cmdline", line 291, in sage.tests.cmdline.test_executable Failed example: ret Expected: 128 Got: 1 ********************************************************************** File "sage.tests.cmdline", line 293, in sage.tests.cmdline.test_executable Failed example: out.find("1 items had failures:") >= 0 Expected: True Got: False ********************************************************************** 1 items had failures: 6 of 183 in sage.tests.cmdline.test_executable ***Test Failed*** 6 failures. ********************************************************************** File "sage.matrix.benchmark", line 566, in sage.matrix.benchmark.report_GF Failed example: b.report_GF(systems=['sage']) Expected: ====================================================================== Dense benchmarks over GF with prime 16411 ====================================================================== ... ====================================================================== Got: Linux sage.math.washington.edu 2.6.2428server #1 SMP Fri Feb 11 18:08:32 UTC 2011 x86_64 GNU/Linux ====================================================================== Dense benchmarks over GF with prime 16411 ====================================================================== <BLANKLINE> <BLANKLINE>  Rank over GF(p): Given a n x (n+10) matrix over GF(p) with random entries, compute the rank. <BLANKLINE> INPUT: <BLANKLINE>  ``n``  matrix dimension (default: 300)  ``p``  prime number (default: ``16411``)  ``system``  either 'magma' or 'sage' (default: 'sage') <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.rank_GF(1000) sage: tm = b.rank_GF(1000, system='magma') # optional  magma sage 0.160  Rank over GF(p): Given a (n + 10) x n matrix over GF(p) with random entries, compute the rank. <BLANKLINE> INPUT: <BLANKLINE>  ``n``  matrix dimension (default: 300)  ``p``  prime number (default: ``16411``)  ``system``  either 'magma' or 'sage' (default: 'sage') <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.rank2_GF(500) sage: tm = b.rank2_GF(500, system='magma') # optional  magma sage 0.170  Given a n+1 x n matrix over GF(p) with random entries, compute the nullspace. <BLANKLINE> INPUT: <BLANKLINE>  ``n``  matrix dimension (default: 300)  ``p``  prime number (default: ``16411``)  ``system``  either 'magma' or 'sage' (default: 'sage') <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.nullspace_GF(300) sage: tm = b.nullspace_GF(300, system='magma') # optional  magma sage 0.820  Given a n x n matrix over GF with random entries, compute the charpoly. <BLANKLINE> INPUT: <BLANKLINE>  ``n``  matrix dimension (default: 100)  ``p``  prime number (default: ``16411``)  ``system``  either 'magma' or 'sage' (default: 'sage') <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.charpoly_GF(100) sage: tm = b.charpoly_GF(100, system='magma') # optional  magma sage 0.060  Given an n x n matrix A over GF(p) with random entries, compute A * (A+1). <BLANKLINE> INPUT: <BLANKLINE>  ``n``  matrix dimension (default: 100)  ``p``  prime number (default: ``16411``)  ``system``  either 'magma' or 'sage' (default: 'sage')  ``times``  number of experiments (default: ``3``) <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.matrix_multiply_GF(100, p=19) sage: tm = b.matrix_multiply_GF(100, p=19, system='magma') # optional  magma sage 0.003  Dense determinant over GF(p). Given an n x n matrix A over GF with random entries compute det(A). <BLANKLINE> INPUT: <BLANKLINE>  ``n``  matrix dimension (default: 300)  ``p``  prime number (default: ``16411``)  ``system``  either 'magma' or 'sage' (default: 'sage') <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.det_GF(1000) sage: tm = b.det_GF(1000, system='magma') # optional  magma sage 0.090 ====================================================================== ********************************************************************** 3 items had failures: 1 of 2 in sage.matrix.benchmark 1 of 2 in sage.matrix.benchmark.report 1 of 2 in sage.matrix.benchmark.report_GF ***Test Failed*** 3 failures.  Doctests interrupted: 0/2067 files tested  Total time for all tests: 181.4 seconds cpu time: 0.0 seconds cumulative wall time: 0.0 seconds Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/bin/sageruntests", line 69, in <module> err = DC.run() File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/control.py", line 645, in run self.run_doctests() File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/control.py", line 446, in run_doctests self.dispatcher.dispatch() File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 1130, in dispatch self._parallel_dispatch() File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 1072, in _parallel_dispatch w.start() File "/release/merger/sage5.1.beta512415/local/lib/python/multiprocessing/process.py", line 130, in start self._popen = Popen(self) File "/release/merger/sage5.1.beta512415/local/lib/python/multiprocessing/forking.py", line 120, in __init__ self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory
comment:42 in reply to: ↑ 38 Changed 9 years ago by
Replying to jdemeyer:
Planning to make a reviewer patch
...after you guys fix the more serious errors.
comment:43 followup: ↓ 47 Changed 9 years ago by
Some of these are due to a bug in the change I just made to p. I'm working on fixing those.
Some look like you didn't apply the patch to the sagenb repo. Did you forget, or is something else going on?
Some are due to the fact that in your tests, os.system calls are happening before the python process gets far enough, so results that were printed to the screen in the wrong order. I'm guessing this may relate to the fact that you're running on /release, since my tests on sage.math didn't show the same behavior.
I'm in the process of fixing stuff.
comment:44 Changed 9 years ago by
There's some code in the current doctesting code to output xml results, which I'm not currently duplicating. Is this used somewhere?
comment:45 Changed 9 years ago by
Currently sage t new is broken. I'm trying to figure out why.
comment:46 followup: ↓ 78 Changed 9 years ago by
Alright, I'm going to start running tests. There are still some remaining issues, but the patches are in a state where someone else might want to look at them. :)
Remaining problems:
 #13145 needs work, generating some doctest failures related to deallocating multivariate polynomials
 hg_sage seems to be broken, causing the new option to fail
 valgrinding
sage t
doesn't work; I'm not sure whether it did before  The new code doesn't support xml output
I'll see if I find more when running sage t all
.
comment:47 in reply to: ↑ 43 Changed 9 years ago by
Replying to roed:
Some are due to the fact that in your tests, os.system calls are happening before the python process gets far enough, so results that were printed to the screen in the wrong order.
Use sys.stdout.flush()
(and/or the same for stderr
) before os.system()
to overcome this. The difference is probably that my output was sent to a pipe, as opposed to directly to the terminal. In the latter case, flushing is automatic.
True, I might not have applied all patches correctly.
comment:48 followup: ↓ 49 Changed 9 years ago by
comment:49 in reply to: ↑ 48 Changed 9 years ago by
Replying to kini:
It doesn't make sense to patch the old notebook. Please work with the flask notebook instead (i.e. make this ticket depend on #11080 and preferably #13121).
I agree that it doesn't make much sense to patch the old notebook, but if this can get a positive review quickly I'd rather not wait on #11080 and #13121. The changes to the old notebook are pretty minor.
comment:50 followups: ↓ 52 ↓ 53 Changed 9 years ago by
Design question: when running tests in parallel, failed examples are printed as they arise, meaning that examples from different files are intermingled. Is this a problem that needs to be fixed?
comment:51 Changed 9 years ago by
 Status changed from needs_work to needs_review
The only tests I have failing on sage.math are sage.tests.startup
and sage.rings.polynomial.multi_polynomial_libsingular
(due to #13145).
comment:52 in reply to: ↑ 50 Changed 9 years ago by
Replying to roed:
Design question: when running tests in parallel, failed examples are printed as they arise, meaning that examples from different files are intermingled. Is this a problem that needs to be fixed?
It might not be truly a problem, but I prefer the current model where the tests for one file are grouped.
comment:53 in reply to: ↑ 50 Changed 9 years ago by
Replying to roed:
Design question: when running tests in parallel, failed examples are printed as they arise, meaning that examples from different files are intermingled. Is this a problem that needs to be fixed?
Even better might be an option to show only the first failure in each file.
comment:54 Changed 9 years ago by
I've updated the code to group results for each file, and added an option (sage t initial
) so that only the first failure is reported.
comment:55 Changed 9 years ago by
The following is cut from a make ptestlong
run:
sage t devel/sage/sage/symbolic/expression.pyx [1960 tests, 38.2 s] sage t devel/sage/sage/pl********************************************************************** File "sage.schemes.elliptic_curves.ell_rational_field", line 1468, in sage.schemes.elliptic_curves.ell_rational_field.EllipticCurve_ration al_field.simon_two_descent Failed example: E.simon_two_descent() # long time (9s on sage.math, 2011) Expected: (5, 5, [(5 : 8 : 1), (4 : 9 : 1), (3 : 11 : 1), (1 : 20 : 1), (6 : 25 : 1)]) Got: Saturation index bound = 265 WARNING: saturation at primes p > 97 will not be done; points may be unsaturated at primes between 97 and index bound Failed to saturate MW basis at primes [ ] Saturation index bound = 265 WARNING: saturation at primes p > 199 will not be done; points may be unsaturated at primes between 199 and index bound Failed to saturate MW basis at primes [ ] (5, 5, [(5 : 8 : 1), (4 : 9 : 1), (3 : 11 : 1), (1 : 20 : 1), (6 : 25 : 1)]) ********************************************************************** 1 items had failures: 1 of 38 in sage.schemes.elliptic_curves.ell_rational_field.EllipticCurve_rational_field.simon_two_descent ***Test Failed*** 1 failures. ot/plot3d/implicit_plot3d.py [57 tests, 26.9 s] sage t devel/sage/sage/modular/modsym/ambient.py [397 tests, 26.4 s]
There are 3 issues:
 The doctest failure itself.
 The fact that
sage t foo
is written instead ofsage t long foo
.
 Output flushing: note the
sage t devel/sage/sage/pl
and then later
ot/plot3d/implicit_plot3d.py [57 tests, 26.9 s]
By all means please keep working on this, it already looks very cool!
comment:56 Changed 9 years ago by
Some more doctest failures from make ptest
:
sage t devel/sage/doc/common/__init__.py [0 tests, 0.0 s] sage t devel/sage/doc/common/build_optiUsage: sage docbuild [OPTIONS] DOCUMENT (FORMAT  COMMAND) sageruntests: error: no such option: p ons.py [0 tests, 0.0 s] sage t devel/sage/doc/common/builder.py Bad exit: 2 Tests run before process failed: sage t devel/sage/doc/common/conf.py [0 tests, 0.0 s]
sage t devel/sage/doc/de/tutorial/interactive_shell.rst Error: TAB character found at line 656 [8 tests, 0.2 s] sage t devel/sage/doc/en/tutorial/interactive_shell.rst [8 tests, 0.2 s] sage t devel/sage/doc/en/bordeaux_2008/birds_other.rst [26 tests, 5.2********************************************************************** File "interfaces", line 331, in interfaces Failed example: maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\ y*sin(x/2)]", "[x, 4, 4]", "[y, 4, 4]",\ '[plot_format, openmath]') # not tested Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 590, in _run compileflags, 1) File "<doctest interfaces[0]>", line 2 maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\ y*sin(x/2)]", "[x, 4, 4]", "[y, 4, 4]",\ ^ SyntaxError: unexpected character after line continuation character ********************************************************************** 1 items had failures: 1 of 57 in interfaces ***Test Failed*** 1 failures. ********************************************************************** File "interfaces", line 318, in interfaces Failed example: maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\ y*sin(x/2)]", "[x, 4, 4]", "[y, 4, 4]",\ '[plot_format, openmath]') # not tested Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 590, in _run compileflags, 1) File "<doctest interfaces[0]>", line 2 maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\ y*sin(x/2)]", "[x, 4, 4]", "[y, 4, 4]",\ ^ SyntaxError: unexpected character after line continuation character ********************************************************************** 1 items had failures: 1 of 57 in interfaces ***Test Failed*** 1 failures. s] sage t devel/sage/doc/fr/tutorial/interactive_shell.rst [8 tests, 0.2 s]
(why is the filename simply "interfaces")
sage t devel/sage/doc/en/reference/probability.rst [0 tests, 0.0 s] sage t de********************************************************************** File "programming", line 620, in programming Failed example: for i in range(5): Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 590, in _run compileflags, 1) File "<doctest programming[0]>", line 1 for i in range(5): ^ SyntaxError: unexpected EOF while parsing ********************************************************************** 1 items had failures: 1 of 113 in programming ***Test Failed*** 1 failures.
sage t devel/sage/doc/en/reference/rings_standard.rst [0 tests, 0.0 s] sage t devel/sage/doc/en/reference/combinat/root_syste********************************************************************** File "programming", line 578, in programming Failed example: for i in range(5): Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 590, in _run compileflags, 1) File "<doctest programming[0]>", line 1 for i in range(5): ^ SyntaxError: unexpected EOF while parsing ********************************************************************** 1 items had failures: 1 of 113 in programming ***Test Failed*** 1 failures. ms.rst [0 tests, 0.0 s]
sage t devel/sage/sage/databases/odlyzko.py [0 tests, 0.0 s] sag********************************************************************** File "sage.doctest.reporting", line 93, in sage.doctest.reporting.DocTestReporter.report Failed example: DTR.report(FDS, True, 0, None, "Output so far...") Expected: sage t .../devel/sage/sage/doctest/reporting.py Timed out! Output so far... Got: sage t /release/merger/sage5.1.beta512415/devel/sage/sage/doctest/reporting.py Timed out! Tests run before process froze: Output so far... ********************************************************************** File "sage.doctest.reporting", line 102, in sage.doctest.reporting.DocTestReporter.report Failed example: DTR.report(FDS, False, 3, None, "Output before bad exit") Expected: sage t .../devel/sage/sage/doctest/reporting.py Bad exit: 3 Output before bad exit Got: sage t /release/merger/sage5.1.beta512415/devel/sage/sage/doctest/reporting.py Bad exit: 3 Tests run before process failed: Output before bad exit ********************************************************************** File "sage.doctest.reporting", line 215, in sage.doctest.reporting.DocTestReporter.finalize Failed example: DTR.report(FDS, True, 0, None, "Output so far...") Expected: sage t .../devel/sage/sage/doctest/reporting.py Timed out! Output so far... Got: sage t /release/merger/sage5.1.beta512415/devel/sage/sage/doctest/reporting.py Timed out! Tests run before process froze: Output so far... ********************************************************************** File "sage.doctest.reporting", line 219, in sage.doctest.reporting.DocTestReporter.finalize Failed example: DTR.report(FDS, False, 3, None, "Output before bad exit") Expected: sage t .../devel/sage/sage/doctest/reporting.py Bad exit: 3 Output before bad exit Got: sage t /release/merger/sage5.1.beta512415/devel/sage/sage/doctest/reporting.py Bad exit: 3 Tests run before process failed: Output before bad exit ********************************************************************** 2 items had failures: 2 of 26 in sage.doctest.reporting.DocTestReporter.finalize 2 of 23 in sage.doctest.reporting.DocTestReporter.report ***Test Failed*** 4 failures. e t devel/sage/sage/databases/sloane.py [1 tests, 0.1 s]
sage t devel/sage/sage/ext/cdefs.pxi [0 tests, 0.0 s] sage t devel/sage/sage/docte********************************************************************** File "sage.doctest.control", line 252, in sage.doctest.control.DocTestController.add_files Failed example: DC = DocTestController(DD, []) Exception raised: Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 591, in _run self.execute(example, compiled, test.globs) File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 873, in execute exec compiled in globs File "<doctest sage.doctest.control.DocTestController.add_files[2]>", line 1, in <module> DC = DocTestController(DD, []) File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/control.py", line 132, in __init__ print "Unable to open logfile at %s\nProceeding without logging."%(self.logfile) AttributeError: 'DocTestController' object has no attribute 'logfile' **********************************************************************
sage t devel/sage/sage/misc/profiler.py [0 tests, 0.0 s] sage t devel/sage/********************************************************************** File "sage.matrix.benchmark", line 11, in sage.matrix.benchmark Failed example: print "starting"; b.report([b.det_ZZ], 'Test', systems=['sage']) Expected: starting... ====================================================================== Test ====================================================================== ... ====================================================================== Got: Linux sage.math.washington.edu 2.6.2428server #1 SMP Fri Feb 11 18:08:32 UTC 2011 x86_64 GNU/Linux starting ====================================================================== Test ====================================================================== <BLANKLINE> <BLANKLINE>  Dense integer determinant over ZZ. Given an n x n matrix A over ZZ with random entries between min and max, inclusive, compute det(A). <BLANKLINE> INPUT: <BLANKLINE>  ``n``  matrix dimension (default: ``200``)  ``min``  minimal value for entries of matrix (default: ``1``)  ``max``  maximal value for entries of matrix (default: ``100``)  ``system``  either 'sage' or 'magma' (default: 'sage') <BLANKLINE> EXAMPLES:: <BLANKLINE> sage: import sage.matrix.benchmark as b sage: ts = b.det_ZZ(200) sage: tm = b.det_ZZ(200, system='magma') # optional  magma sage 1.370 ====================================================================== **********************************************************************
comment:57 Changed 9 years ago by
Another annoyance, compare in the output:
sage t devel/sage/sage/rings/polynomial/padics/polynomial_padic_capped_relative_dense.py
sage t /release/merger/sage5.1.beta512415/devel/sagenbmain/sagenb/interfaces/expect.py
Why a relative path in the first case but a full path in the second case? Also with a
, only absolute paths are shown. I would prefer all paths to be relative to the current directory.
comment:58 Changed 9 years ago by
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage tp 1000000000 devel/sage/sage/rings/padics/ Running doctests with ID 201206261158023accd40f. Doctesting 32 files using 1000000000 threads
The previous doctester would reduce the number of threads to be at most the number of files to test.
comment:59 Changed 9 years ago by
A very useful feature from the previous doctester which is missing here is that verbose
actually shows the output as the test is running. So when it gets stuck somewhere, you actually see that it gets stuck. Currently, the whole test result is simply dumped at the end.
Also: previously, line numbers were shown with verbose
(albeit in a very bad way).
One more featurerequest: with verbose
, show the time taken for each single test, i.e. I would like
Trying (line 242): EllipticCurve('900d1').integral_points() Expecting: [(11 : 27 : 1), (4 : 34 : 1), (4 : 18 : 1), (16 : 54 : 1)] ok [0.10s] Trying (line 243): E=EllipticCurve([879984,319138704]) Expecting nothing ok [1.92s] Trying (line 255): P1=E.point((540,1188)); P2=E.point((576,1836)) Expecting nothing ok [4.59s]
It's totally okay for me to disallow verbose
in parallel tests.
comment:60 followup: ↓ 65 Changed 9 years ago by
What does the serial
option do and how is it different from p 1
?
comment:61 Changed 9 years ago by
I would expect sage t f
not to need any filename arguments (similarly to a
), just run all the failed tests.
However:
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t f Usage: sage t [options] filenames
comment:62 followup: ↓ 66 Changed 9 years ago by
Perhaps add a T
option as a synonym for timeout
?
comment:63 followups: ↓ 68 ↓ 71 Changed 9 years ago by
I'm collecting some "special cases" for doctests in http://boxen.math.washington.edu/home/jdemeyer/doctest/ (maybe these should eventually become "metadoctests").
The doctesting framework seems to hang on
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t /home/jdemeyer/doctest/sigdie.py Running doctests with ID 20120626141255fbefc7c4. Doctesting 1 file.
While the following gives an error inside the doctesting framework (this also used to fail before):
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t /home/jdemeyer/doctest/keyboardinterrupt.py Running doctests with ID 20120626141238b70e54c4. Doctesting 1 file. sage t /home/jdemeyer/doctest/keyboardinterrupt.py Traceback (most recent call last): File "/release/merger/sage5.1.beta512415/local/lib/python2.7/sitepackages/sage/doctest/reporting.py", line 146, in report ntests, runner = results TypeError: 'NoneType' object is not iterable  Doctests interrupted: 0/1 files tested  Total time for all tests: 1.0 seconds cpu time: 0.0 seconds cumulative wall time: 0.0 seconds
The following should report an error but doesn't:
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t /home/jdemeyer/doctest/sig_on.py Running doctests with ID 2012062614140806a01dbf. Doctesting 1 file. sage t /home/jdemeyer/doctest/sig_on.py [1 tests, 1.1 s]  All tests passed!  Total time for all tests: 2.1 seconds cpu time: 0.1 seconds cumulative wall time: 1.1 seconds
comment:64 Changed 9 years ago by
I'm working on fixing these. Thanks for the feedback!
comment:65 in reply to: ↑ 60 ; followup: ↓ 80 Changed 9 years ago by
Replying to jdemeyer:
What does the
serial
option do and how is it different fromp 1
?
The p 1
has one worker: there's another thread which collates and prints answers. p 1
is the default.
comment:66 in reply to: ↑ 62 Changed 9 years ago by
Replying to jdemeyer:
Perhaps add a
T
option as a synonym fortimeout
?
Done. Are there other options that should have a short form? Maybe L
for long
? I'd like to avoid too much mixing of upper and lower case....
comment:67 Changed 9 years ago by
There doesn't seem to be an objection to diverting stderr
, so I'm going to not worry about keeping changes resulting from that decision separate (I don't want to update that patch since it requires rebuilding a lot of files).
comment:68 in reply to: ↑ 63 ; followups: ↓ 69 ↓ 74 Changed 9 years ago by
Replying to jdemeyer:
The following should report an error but doesn't:
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t /home/jdemeyer/doctest/sig_on.py Running doctests with ID 2012062614140806a01dbf. Doctesting 1 file. sage t /home/jdemeyer/doctest/sig_on.py [1 tests, 1.1 s]  All tests passed!  Total time for all tests: 2.1 seconds cpu time: 0.1 seconds cumulative wall time: 1.1 seconds
If I do cython('sig_on()')
at the command line nothing happens. Is it supposed to crash Sage?
comment:69 in reply to: ↑ 68 Changed 9 years ago by
Replying to roed:
Replying to jdemeyer:
The following should report an error but doesn't:
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t /home/jdemeyer/doctest/sig_on.py Running doctests with ID 2012062614140806a01dbf. Doctesting 1 file. sage t /home/jdemeyer/doctest/sig_on.py [1 tests, 1.1 s]  All tests passed!  Total time for all tests: 2.1 seconds cpu time: 0.1 seconds cumulative wall time: 1.1 secondsIf I do
cython('sig_on()')
at the command line nothing happens. Is it supposed to crash Sage?
I can even run sage commands afterward, like factor(2^1991)
. But if I hit CtlC, I get a segfault (and it screws up my terminal).
comment:70 followup: ↓ 72 Changed 9 years ago by
New version posted. sage t long a
passes all tests except the one related to #13145.
/home/jdemeyer/doctest/keyboardinterrupt.py
passes if you remove the trailing colon. I've updated the framework to recover a bit more gracefully if a doctest raises an unexpected KeyboardInterrupt?, but it behaves a bit strangely since it's difficult to distinguish between an actual interrupt (in which case the testing should terminate) and one raised by a doctest (in which case tests of other files could continue). I think the new behavior is actually fairly reasonable: try it on your file without removing the colon.
comment:71 in reply to: ↑ 63 ; followup: ↓ 73 Changed 9 years ago by
Replying to jdemeyer:
The doctesting framework seems to hang on
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t /home/jdemeyer/doctest/sigdie.py Running doctests with ID 20120626141255fbefc7c4. Doctesting 1 file.
Is this behavior acceptable? Eventually the test will time out (you can try running sage t T 5 /home/jdemeyer/doctest/sigdie.py
) and the ending summary will be printed with a time out.
comment:72 in reply to: ↑ 70 ; followup: ↓ 75 Changed 9 years ago by
Replying to roed:
it's difficult to distinguish between an actual interrupt (in which case the testing should terminate) and one raised by a doctest (in which case tests of other files could continue).
A "real" interrupt would interrupt also the master process, a fake doctestinterrupt only the child process. That's an easy way to distinguish.
comment:73 in reply to: ↑ 71 ; followup: ↓ 76 Changed 9 years ago by
Replying to roed:
Replying to jdemeyer:
The doctesting framework seems to hang on
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t /home/jdemeyer/doctest/sigdie.py Running doctests with ID 20120626141255fbefc7c4. Doctesting 1 file.Is this behavior acceptable?
Not really. One should get a message that Sage crashed (compare with the old doctesting framework).
comment:74 in reply to: ↑ 68 ; followup: ↓ 77 Changed 9 years ago by
Replying to roed:
If I do
cython('sig_on()')
at the command line nothing happens. Is it supposed to crash Sage?
Let me elaborate: #10030 added a check to the doctesting code which checks
sage: sig_on_count() 0
after each doctest (i.e. it appends this to every doctest), to ensure there are no unbalanced sig_on()
s. Could you add this also to your doctesting framework?
comment:75 in reply to: ↑ 72 ; followup: ↓ 79 Changed 9 years ago by
Replying to jdemeyer:
Replying to roed:
it's difficult to distinguish between an actual interrupt (in which case the testing should terminate) and one raised by a doctest (in which case tests of other files could continue).
A "real" interrupt would interrupt also the master process, a fake doctestinterrupt only the child process. That's an easy way to distinguish.
Sure, and for reporting what happened to the user this is fine. But from the child process' perspective, an unexpected KeyboardInterrupt
has been raised and it should stop testing that file. The parent process can continue the doctests though....
comment:76 in reply to: ↑ 73 Changed 9 years ago by
Replying to jdemeyer:
Replying to roed:
Replying to jdemeyer:
The doctesting framework seems to hang on
jdemeyer@sage:/release/merger/sage5.1.beta512415$ ./sage t /home/jdemeyer/doctest/sigdie.py Running doctests with ID 20120626141255fbefc7c4. Doctesting 1 file.Is this behavior acceptable?
Not really. One should get a message that Sage crashed (compare with the old doctesting framework).
Okay, I will look into it. I assume that the underlying process has terminated and I should be able to get at the exit code somehow.
comment:77 in reply to: ↑ 74 Changed 9 years ago by
Replying to jdemeyer:
Replying to roed:
If I do
cython('sig_on()')
at the command line nothing happens. Is it supposed to crash Sage?Let me elaborate: #10030 added a check to the doctesting code which checks
sage: sig_on_count() 0after each doctest (i.e. it appends this to every doctest), to ensure there are no unbalanced
sig_on()
s. Could you add this also to your doctesting framework?
This shouldn't be too hard.
comment:78 in reply to: ↑ 46 Changed 9 years ago by
Replying to roed:
Alright, I'm going to start running tests. There are still some remaining issues, but the patches are in a state where someone else might want to look at them. :)
Remaining problems:
 #13145 needs work, generating some doctest failures related to deallocating multivariate polynomials
 hg_sage seems to be broken, causing the new option to fail
 valgrinding
sage t
doesn't work; I'm not sure whether it did before The new code doesn't support xml output
I'll see if I find more when running
sage t all
.
I'm going to sleep; let me know which of these need to be fixed in order for this ticket to get a positive review. I will also keep working on the tests in /home/jdemeyer/doctest/
. And of course if you find more problems or have other feature requests, keep those coming too!
comment:79 in reply to: ↑ 75 ; followup: ↓ 83 Changed 9 years ago by
Replying to roed:
Sure, and for reporting what happened to the user this is fine. But from the child process' perspective, an unexpected
KeyboardInterrupt
has been raised and it should stop testing that file.
Okay, I see what you mean. I would propose not to specialcase KeyboardInterrupt
at all in the child process. When the parent process gets a KeyboardInterrupt
, kill all child processes (without showing their output) and exit.
(note: since the old doctesting framework didn't do this either, I don't consider this a prerequisite for positive review, but a nice feature to have.)
comment:80 in reply to: ↑ 65 ; followup: ↓ 81 Changed 9 years ago by
comment:81 in reply to: ↑ 80 ; followup: ↓ 82 Changed 9 years ago by
Replying to jdemeyer:
Replying to roed:
Replying to jdemeyer:
What does the
serial
option do and how is it different fromp 1
?The
p 1
has one worker: there's another thread which collates and prints answers.p 1
is the default.I suppose you mean "process", not thread? Still, I don't see the point of
serial
.
The main point of serial
is that it allows you to debug problems in the doctesting framework without worrying about multiprocessing. :)
I'm fine making it less prominent (having it not show up in the help display for sage t
for example, and not using it for verbose
mode). But I think it should stick around since it currently exists, and it makes some kinds of bugs much easier to track down.
comment:82 in reply to: ↑ 81 Changed 9 years ago by
Replying to roed:
Replying to jdemeyer:
Replying to roed:
Replying to jdemeyer:
What does the
serial
option do and how is it different fromp 1
?The
p 1
has one worker: there's another thread which collates and prints answers.p 1
is the default.I suppose you mean "process", not thread? Still, I don't see the point of
serial
.The main point of
serial
is that it allows you to debug problems in the doctesting framework without worrying about multiprocessing. :)I'm fine making it less prominent (having it not show up in the help display for
sage t
for example, and not using it forverbose
mode). But I think it should stick around since it currently exists, and it makes some kinds of bugs much easier to track down.
I'm also using it for gdb
since it makes the tracebacks simpler, though I'm not sure that's necessary.
Do you ever use sage t
with valgrind? I'm not sure exactly what's wrong with sage t valgrind <FILENAME>
.
comment:83 in reply to: ↑ 79 Changed 9 years ago by
Replying to jdemeyer:
Replying to roed:
Sure, and for reporting what happened to the user this is fine. But from the child process' perspective, an unexpected
KeyboardInterrupt
has been raised and it should stop testing that file.Okay, I see what you mean. I would propose not to specialcase
KeyboardInterrupt
at all in the child process. When the parent process gets aKeyboardInterrupt
, kill all child processes (without showing their output) and exit.(note: since the old doctesting framework didn't do this either, I don't consider this a prerequisite for positive review, but a nice feature to have.)
I just tried changing the interrupt handling so that KeyboardInterrupt
is handled in the parent process and it doesn't work as well: there's extraneous printing happening when the doctest framework thinks it's just printing an unexpected exception, and I don't see a good way to get rid of it.
I think the current handling of KeyboardInterrupts? is fine.
comment:84 Changed 9 years ago by
I'm going to be offline until Tuesday. More feedback while I'm gone would be great. :)
comment:85 Changed 9 years ago by
As far as I understand it, this new framework should allow for the same commands (e.g., sage tp 6 long devel/sage/sage/matrix/
). In addition, though, there are new commandline options to sageruntest
, which should available through sage t ...
, right? Which of these new options should be documented in the help messages usage
and/or usage_advanced
at the top of spkg/bin/sage
? Which should be documented in the developer's guide?
comment:86 Changed 9 years ago by
I'm back, and I've been working on a patch to the developer's guide. I'll post something soon, and more comments on which options will be shown in usage.
comment:87 Changed 9 years ago by
I'm going to build and test Sage again with these patches. Stay tuned...
comment:88 Changed 9 years ago by
 Description modified (diff)
I'm working on this. Email me if you want to collaborate.