Opened 6 years ago

Closed 6 years ago

#22832 closed enhancement (fixed)

Add an option to the test runner to exit immediately upon the first failure

Reported by: Erik Bray Owned by:
Priority: major Milestone: sage-8.0
Component: doctest framework Keywords:
Cc: Merged in:
Authors: Erik Bray Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: c6a5b40 (Commits, GitHub, GitLab) Commit: c6a5b40146506795873a1cb7b64e645258251a90
Dependencies: Stopgaps:

Status badges

Description (last modified by Jeroen Demeyer)

Inspired by the --exitfirst option to py.test, this adds a --exitfirst option to Sage's test runner to exit immediately upon the first test failure.

I've wanted something like this at several points, but one reason in particular is that part of the build of the patchbot Docker image is to perform the baseline test run, so that the image itself can be reused without having to rebuild the baseline each time. Obviously, if a test fails during the baseline test run then I want the Docker image build to fail immediately, and not wait until the rest of the test suite has run.

Change History (11)

comment:1 Changed 6 years ago by Erik Bray

Authors: Erik Bray
Branch: u/embray/tests/fail-once
Commit: 2f359e8a5e5da8b8dba9475e7b168b2cf74c0e65
Status: newneeds_review

New commits:

c12a4d3Initial attempt to add a test runner option to stop a test run as soon as a single failure is encountered.
2f359e8Make --fail-once work with parallel doctests.

comment:2 in reply to:  description ; Changed 6 years ago by Jeroen Demeyer

Replying to embray:

Inspired by the --exitfirst option to py.test, this adds a --fail-once option to Sage's test runner to exit immediately upon the first test failure.

Obvious question: why not call the Sage option exitfirst? I find it a better name than fail-once.

comment:3 Changed 6 years ago by Jeroen Demeyer

Status: needs_reviewneeds_work

I'm making some changes, hang on...

comment:4 Changed 6 years ago by Jeroen Demeyer

Branch: u/embray/tests/fail-onceu/jdemeyer/tests/fail-once

comment:5 Changed 6 years ago by Jeroen Demeyer

Commit: 2f359e8a5e5da8b8dba9475e7b168b2cf74c0e65c2bff84e640507306d52b108e3ee440eb555ddba
Description: modified (diff)
Reviewers: Jeroen Demeyer
Status: needs_workneeds_review

New commits:

f26f64eInitial attempt to add a test runner option to stop a test run as soon as a single failure is encountered.
2e9a80cMake --fail-once work with parallel doctests.
c2bff84Rename --fail-once to --exitfirst; minor fixes; add test

comment:6 Changed 6 years ago by Erik Bray

Status: needs_reviewpositive_review

I'm fine with that, thanks.

comment:7 in reply to:  2 Changed 6 years ago by Erik Bray

Replying to jdemeyer:

Replying to embray:

Inspired by the --exitfirst option to py.test, this adds a --fail-once option to Sage's test runner to exit immediately upon the first test failure.

Obvious question: why not call the Sage option exitfirst? I find it a better name than fail-once.

I personally find "fail once" to be clearer than "exit first"--"Exit first what?" is what it makes me ask. That said, the documentation makes it clear enough, and I don't mind the consistency.

comment:8 Changed 6 years ago by Erik Bray

Status: positive_reviewneeds_work

Oops, one little nitpick--you forgot to remove the line fail_immediately=False after renaming it to abort_now everywhere else.

comment:9 Changed 6 years ago by git

Commit: c2bff84e640507306d52b108e3ee440eb555ddbac6a5b40146506795873a1cb7b64e645258251a90

Branch pushed to git repo; I updated commit sha1. New commits:

c6a5b40Remove forgotten line

comment:10 Changed 6 years ago by Jeroen Demeyer

Status: needs_workpositive_review

comment:11 Changed 6 years ago by Volker Braun

Branch: u/jdemeyer/tests/fail-oncec6a5b40146506795873a1cb7b64e645258251a90
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.