The Catch3 runner

Contents
How a run works
Retries, flaky tests and order dependence
History, quarantine and analytics
Importing the results of other frameworks
Choosing and ordering tests
Tags the runner reads
Time limits
Temporary files
What a failure report says
Reports and traces
Per-test coverage
Formats other tools read
Project configuration
Bazel and CTest
Exit codes
Coming from Catch2's runner

Every Catch3 test binary has the runner built in. It replaced Catch2's session, command line and reporters; the API tests are written with stays Catch2's: TEST_CASE, SECTION, the assertions, matchers, generators, benchmarks, fixtures and listeners. The options are listed in the command line reference.

$ ./tests
Catch3: 446 test(s), up to 8 worker(s), seed 2153251184

446 test(s): 423 passed, 16 failed as expected, 7 skipped
84193 assertion(s), 446 attempt(s); 1.2 s (6.0 s of tests, 5.0x in parallel)

How a run works

The runner reads the registered tests, decides which to run and in which order, and hands them to worker processes: the same binary, started again with an internal option. A free worker takes the next test, so the work spreads by itself and a long test does not hold back a shard of short ones. --jobs=N sets the number of workers; the default is one per hardware thread.

Running tests in their own processes gives the run what one process cannot:

  • A crash ends the test, not the run. A worker that dies is replaced; the test it was running is reported as crashed, with the signal or structured exception, the last assertion it started and the section it was in.
  • A hang is stopped. A test that runs past its time limit is ended together with every process it started, and reported as timed out, with the section it was in.
  • Tests that need it run alone. --isolate, or the tag [!isolated], gives a test a fresh process.

The runner, its workers and every process a test starts form one group: a job object on Windows, a process group elsewhere. Stopping a worker stops all of it.

What a test writes to std::cout, std::cerr and std::clog is captured and belongs to the test. Output of a worker outside any test, such as what a listener prints when the run starts, is passed on to the runner's standard error.

--in-process runs the tests in the runner's own process, one after another, for a debugger. --break implies it. Crashes and hangs are then not contained.

Listeners registered with CATCH_REGISTER_LISTENER run in every worker: each worker is one test run for them, from testRunStarting to testRunEnded.

Retries, flaky tests and order dependence

With --retries=auto, the default, a failed test is tried once more, alone in a fresh process. What happens then tells three kinds of failure apart:

  • It fails again: the failure is deterministic, and is reported.
  • It passes, and it had failed alone too: the test is flaky. It counts as passed, is marked flaky in every report, and its history remembers it.
  • It passes alone, but had failed after other tests in a shared process: the runner runs the same tests again in a fresh process, in the same order. If the test fails again, it depends on state another test leaves behind, and the runner bisects the tests that ran before it to find which one. The report names it: fails after 'X', which leaves state behind. Such a failure fails the run; --no-bisect turns the search off.

Crashes, timeouts, tests in quarantine and tests with a flaky history get up to two retries. --retries=N retries every failure up to N times, --retries=off never retries, and [!retries:N] sets the limit of one test. A run spends at most a tenth of its tests on retries, so a broken build does not run everything three times.

--repeat=N runs every test N times; a test whose verdicts differ is flaky.

History, quarantine and analytics

The runner remembers the results of the last 50 runs of each test: verdict, duration, flakiness, failure signature and a fingerprint of the inputs. The history of a binary lives in one JSON file in .catch3 beside the binary, or in the directory --history or the environment variable CATCH3_HISTORY names. --no-history neither reads nor writes it. Runs that finish at the same time all reach the file: it is updated under a lock, read again, and replaced in one rename.

The history drives several features:

  • Adaptive time limits. See time limits.
  • Predictive ordering. Each test gets a probability of failing: its failure rate, with recent runs weighing more, and at least 70% after a failure. Tests that are likely to fail run first, the most failures per second of running time first, so a failing run fails early. The rest run longest first, which keeps every worker busy until the end.
  • Quarantine. A test with two flaky results in its last 20 runs goes into quarantine. It still runs and is reported, but its failure does not fail the run. Ten clean passes in a row release it. [!quarantine] quarantines a test by hand; --quarantine=report reports quarantine without excusing failures, and --quarantine=off ignores it.
  • Classification. A failure is new (it passed last time), persistent (it failed last time too) or intermittent (it flakes).
  • Replay. --replay runs the last failures again, each with the seed it failed with and after the same tests, in a fresh process.
  • Incremental runs. --incremental skips tests that passed last time with the same fingerprint; they are reported as up to date.
  • Analytics. --analytics prints what the history says: flaky tests, tests in quarantine, the tests that fail most often, the slowest tests and the tests getting slower. --analytics=json prints it as JSON.

Under bazel test there is no history unless CATCH3_HISTORY names a directory that outlives the sandbox.

Importing the results of other frameworks

--import=FILE reads the report of another test framework instead of running tests, so that tests Catch3 does not run get the same history, flaky detection, quarantine and analytics. Any Catch3 test binary can do it, and the option can be repeated:

./tests --import=results/pytest.xml --import=results/vitest.ctrf.json \
        --import-name=web --junit=merged.xml
--import-format=Reads
autowhichever the contents show (the default)
catch3a --json report of Catch3, or catch3-results
junitJUnit XML, with Surefire's reruns; Gradle, pytest, Jest, GoogleTest and most others write it
xunitxUnit.net v2 XML
trxVisual Studio's TRX, with data-driven rows
tapTAP 12, 13 and 14, with subtests and YAML diagnostics
ctrfCTRF JSON, with retries

The results go into a history of their own, named by --import-name and imported by default; --analytics --import-name=NAME reads it. The console, --junit, --json, --trace, --quarantine, --history and the exit code work as they do for a run. Reports that were read count as one run.

A test is identified by its class and its name. Results with the same identifier, in one report or in several, are attempts at one test: a test that both failed and passed is flaky, passes and is remembered as flaky, which is how a report of a rerun is imported. A failure stays a failure when a later report skips the test. Reports of the same tests on different platforms are not reruns of each other: import each with an --import-name of its own, or a failure on one reads as flakiness.

TAP names tests only within one stream, so the path given to --import is the class of its tests: keep it the same from run to run for the history to follow them. A name that comes back in one stream is another test and gets a number.

--import-revision=REVISION names what the reports tested, and a verdict that changes on one revision then counts as flaky too. Without it only retries in the reports do, because a working directory can have edits or be another checkout.

Nothing is run, so an imported failure has no replay command, and only what the report says is known: it has no order, seed or sections. A TAP stream that bails out or reports fewer tests than it planned adds a failed test called the TAP stream, so that it does not read as a run that passed.

The catch3-results format

For a framework without a report of these kinds, catch3-results is the smallest thing to write: JSON Lines with one test per line, after a line that names the format.

{"format": "catch3-results", "version": 1}
{"name": "parses a date", "className": "Dates", "outcome": "passed", "durationMicroseconds": 1200}
{"name": "rounds down", "outcome": "failed", "message": "1.5 != 2", "details": "at Dates.rounds:12"}

name or id, and outcome, are required; outcome is a name from the JSON report, such as passed, failed, skipped, crashed or timed-out. The other members are id, className, file, line, tags, category, message, details, standardOutput and standardError. A line can give attempts instead, as the JSON report does.

Choosing and ordering tests

Test specs work as in Catch2: names with * as a wildcard, [tags], ~ to exclude, a comma for "or". --input-file reads exact names from a file. --select narrows the matching tests further:

--select=Runs
allevery matching test (default)
failedtests that failed, crashed, timed out or flaked last time
changedtests defined in files Git reports as changed since --changes-since (default HEAD), staged or not, and untracked files
newtests without a recorded result
smartchanged, failed and new tests, those in quarantine, and those at least 50% likely to fail

Each selected test carries the reason it was selected into the reports.

--order=priority (default) orders by the prediction above; declared, name and random are also available. Every test is seeded with the run's seed, wherever it runs, so the order does not change its random numbers.

--shard=INDEX/COUNT runs one of COUNT shards. A test goes to the shard a stable hash of its name chooses, so every machine agrees without talking to the others; tests linked by [!depends] stay in one shard. With --shard-durations=FILE, a history or a JSON report, shards are balanced by duration instead, longest first into the least loaded shard. --list honours --shard.

Tags the runner reads

Tags that start with ! are reserved, as in Catch2. The runner adds these:

TagEffect
[!serial]nothing else runs while this test runs
[!isolated]the test runs in a process of its own
[!resource:NAME]tests that share a resource never run at the same time
[!depends:TEST]runs after TEST, and only if TEST passes; otherwise it is reported as blocked. TEST is selected along with it. Cycles are an error.
[!timeout:SECONDS]the time limit of this test
[!retries:N]the retry limit of this test
[!quarantine]the test is in quarantine
TEST_CASE("migrates the schema", "[db][!resource:database]") { ... }
TEST_CASE("reads after migration", "[db][!resource:database][!depends:migrates the schema]") { ... }
TEST_CASE("renders the full catalogue", "[!timeout:120]") { ... }

Time limits

A test with history gets a limit learned from it: four times its 95th percentile or ten times its median, whichever is larger, plus two seconds, and at least five seconds. A test without history gets 300 seconds. --timeout=SECONDS sets the limit for tests without history and caps the learned ones; --timeout=0 turns limits off; --no-adaptive-timeout uses --timeout for every test. [!timeout:SECONDS] overrides all of these. Benchmarks get no learned limit.

Temporary files

Every attempt of every test gets a temporary directory of its own. While the test runs, TMPDIR, TMP and TEMP point at it, so std::filesystem::temp_directory_path(), tmpfile and most libraries use it, and tests that run at the same time never share a scratch file by accident. Progmasoft::Catch3::TestTemporaryDirectory() in <Progmasoft/Catch3/Runner.hpp> returns it.

The directory is removed when the attempt passes and kept when it fails; the report says where.

Environment variables

Workers, and the processes the runner starts for itself, inherit its environment without the variables whose names look like secrets: those that contain, in any case, TOKEN, SECRET, PASSWORD, PASSWD, CREDENTIAL, API_KEY, APIKEY, PRIVATE_KEY or ACCESS_KEY. A test that needs one gets it with --pass-env=NAME, which can be repeated, or "pass-env" in the project configuration. The runner itself starts with the whole environment, so code that runs before the first worker starts still sees them.

What a failure report says

FAIL: parses nested lists
tests/parser.cpp:120
  tests/parser.cpp:131: FAILED:
    REQUIRE( tokens.size() == 3 )
  with expansion:
    2 == 3
  in section with commas
  analysis: new failure after 14 passing run(s); it last passed at commit 4c40a97b2f1e
  analysis: fails the same way as 'parses flat lists': probably one cause
  analysis: changed since HEAD: src/lexer.cpp
  replay: ./tests "parses nested lists" --seed=2153251184

Every failure has a category: assertion, unexpected exception, missing exception, explicit failure, crash, timeout, dependency or infrastructure. It has a signature: the category, the location of the first failure and its message with numbers and addresses removed. Tests that share a signature are named together, since they probably share a cause.

The analysis adds what the runner knows: how the failure relates to the history, the commit the test last passed at, the change it follows (the test's file, or the file of the failing assertion, changed since --changes-since), where a crash or a hang happened, the test that leaves state behind, the quarantine, and where the temporary files were kept. The replay line runs the failure again as it happened.

Reports and traces

  • The console report shows progress and the failures with their analysis, then a summary. --verbose shows every test and what passing tests printed; --success also shows every assertion; --quiet shows only failures. --slowest=N lists the slowest tests.
  • --junit=FILE writes JUnit XML. Retries are <rerunFailure> elements, and a flaky test is a success with <flakyFailure> elements, which Jenkins, GitLab and most CI dashboards understand.
  • --json=FILE writes everything the runner knows: every attempt, its assertions, sections and benchmarks, the captured output, the analysis.
  • --trace=FILE writes a trace for chrome://tracing or Perfetto: one track per worker, one slice per attempt and section.

Per-test coverage

--coverage=DIR runs every test in a process of its own and keeps what each one reached. Build the tests with coverage instrumentation:

  • Clang: -fprofile-instr-generate -fcoverage-mapping, LLVM source-based coverage;
  • GCC: --coverage, gcov's format.
$ ./tests --coverage=coverage
Catch3: 120 test(s), up to 8 worker(s), seed 2153251184
note: --coverage: 120 test profile(s); what each test reached is in coverage/coverage.json, the run combined in coverage/coverage.lcov

120 test(s): 120 passed

Each attempt gets a process of its own, so the profile its process writes at exit is the coverage of that one test: DIR/profiles/t<index>-a<attempt> with .profraw for LLVM, or a .gcov directory below which gcov writes its .gcda files under their absolute paths. The runner sets LLVM_PROFILE_FILE and GCOV_PREFIX for the process; each runtime reads its own. A process that runs other tests first, to find a test that leaves state behind, writes beside the profiles, never into one. The profiles and reports of an earlier run in DIR are removed when the run starts: gcov adds to the data it finds, and a run that writes no reports leaves none to be read as its own.

After the run, the runner reads, on as many threads as the run had workers, the profile of the last attempt of every test that left one; when a retry or repeat crashed before writing its profile, the attempt before it counts:

  • DIR/coverage.json holds, for each test, its outcome, its profile and the files it reached, with the lines it ran and, per line, how many of its branches it took:
    { "version": 1, "format": "llvm", "binary": "./tests",
      "tests": [ { "name": "Adds one", "outcome": "passed",
                   "profile": "coverage/profiles/t0-a1.profraw",
                   "files": [ { "file": "/src/calc.cpp", "lines": [ 12, 13 ],
                                "branches": [ { "line": 12, "taken": 1, "total": 2 } ] } ] } ] }
    
  • DIR/coverage.lcov is the coverage of the whole run as an LCOV tracefile, which genhtml, Codecov and editors read.

LLVM profiles are read with llvm-profdata and llvm-cov, gcov data with gcov. They are looked up on PATH, through xcrun on macOS, or named by CATCH3_LLVM_PROFDATA, CATCH3_LLVM_COV and CATCH3_GCOV. Without them the profiles stay in DIR/profiles and the run says so, as it does when the tests wrote no profiles; neither fails the run. A profile the tools cannot read, or a report that cannot be written, fails a run whose tests passed, like any report the run was asked for. A test that crashes or is stopped at its time limit leaves no profile. gcov is run with --branch-probabilities, without which its JSON has no branches. The runner's own process is instrumented too and writes its profile where any instrumented program does, default.profraw in the working directory unless LLVM_PROFILE_FILE says otherwise; it holds no test.

--coverage cannot be used with --in-process or --break, which run the tests in one process. Per-test coverage is tested on Linux and macOS; Windows builds with clang-cl are next.

Formats other tools read

These formats carry a format and a version. Within a version they only gain members; a reader ignores members it does not know and refuses a newer version.

  • Discovery, --list=json: "format": "catch3-tests", version 1. One object per test with id, name, className, file, line, tags, hidden, and what the runner tags say: serial, isolated, quarantined, resources, dependsOn, timeoutSeconds and retries. CMake's catch3_discover_tests reads it.
  • Run report, --json: "format": "catch3-run", version 1.
  • Imported results, --import: "format": "catch3-results", version 1; see its format.
  • Analytics, --analytics=json: "format": "catch3-analytics", version 1.
  • History: "format": "catch3-history", version 1. It is the runner's own file; read it through --analytics.

The messages between the runner and its workers are internal and change with the binary.

Project configuration

Options a project always wants go in a catch3.json:

{
  "version": 1,
  "tests": ["~[slow]"],
  "jobs": 8,
  "timeout": 60,
  "warn": ["no-assertions"],
  "junit": "build/test-results/junit.xml"
}

The runner looks for the file in the working directory and in each directory above it, up to the root of the Git repository, and reads the nearest one. Outside a repository it looks only in the working directory. --config=FILE names a file instead, and --no-config reads none.

The keys are the long options, and their values go through the same checks as on the command line, with the key named in the message:

ValueKeys
true turns the switch onincremental, filenames-as-tags, in-process, isolate, success, no-throw, allow-no-tests, skip-benchmarks, benchmark-no-analysis
false turns it offadaptive-timeout, bisect
a stringselect, changes-since, order, quarantine, color
a whole numberjobs, repeat, fail-fast, slowest, benchmark-samples, benchmark-resamples, benchmark-warmup-time
a whole number or a stringretries ("auto", "off"), seed ("random")
a numbertimeout (seconds), benchmark-confidence-interval
a path, relative to the filejunit, json, trace, shard-durations; history, or false for none
an array of stringstests (test specs), warn, pass-env
"quiet", "normal" or "verbose"verbosity

"version": 1 is required; "$schema" may name a schema for editors. The file is strict JSON: no comments and no trailing commas. A key it does not know is an error, with a suggestion when it looks like a known one. Options that describe one run, such as --shard, --section, --list or --replay, belong on the command line only.

Options on the command line win over the file, and test specs on the command line replace the file's tests rather than adding to them. A switch the file turns on stays on; --no-config leaves the file out. --verbose names the file that was read. Workers, --help and --version read no file.

Bazel and CTest

Under bazel test the runner reads Bazel's environment: XML_OUTPUT_FILE for a JUnit report, TESTBRIDGE_TEST_ONLY for --test_filter, TEST_SHARD_INDEX, TEST_TOTAL_SHARDS and TEST_SHARD_STATUS_FILE for sharding, TEST_RANDOM_SEED for the seed, TEST_PREMATURE_EXIT_FILE for the exit guard and TEST_UNDECLARED_OUTPUTS_DIR for a JSON report.

catch3_discover_tests registers each test with CTest through the discovery format. The runner also parallelises a whole binary under one CTest test, which usually starts fewer processes.

Exit codes

CodeMeaning
0every test passed or failed as allowed by [!mayfail] or [!shouldfail]; flaky tests and failures in quarantine do not fail the run
1the command line was wrong, the tests could not be registered, or workers could not start
2no test ran (--allow-no-tests accepts this)
3a test spec matched nothing and --warn=unmatched-specs was given
4every test that ran was skipped
5a test spec is invalid
42a test failed, crashed, timed out or was blocked

Coming from Catch2's runner

The test-writing API is unchanged. The runner is new:

  • Options are Catch3's own; --help lists them. Long options take their value after = or as the next argument. -s, -b, -e, -a, -c, -p, -f, -# and -w keep their meaning, under the names in the reference.
  • Reporters and CATCH_REGISTER_REPORTER are gone. The console, JUnit and JSON reports replace the built-in reporters; for anything else, read the JSON report, or write a listener, which still runs in every worker.
  • Catch::Session remains for a main of your own: run(argc, argv) hands the command line to the runner. Session::cli() and configData() are gone with Clara. See supplying main yourself.
  • main runs in every worker too, since workers are the same binary.
  • Tests run in parallel by default. Tests that share files or other state outside the process need [!serial] or [!resource:NAME]; per-test temporary directories cover the common case.

Эта страница на GitHubЛицензии документации