Command line

Contents
Test specs
Selecting tests
Running
How tests behave
History and analysis
Output
Configuration
Information
Environment

This is the command line of a test binary, which the Catch3 runner reads. --help prints it.

Long options take their value after = or as the next argument: --jobs=4 and --jobs 4 are the same. Short options take it attached or after =: -j4, -j=4. An option the runner does not know is an error, with a suggestion when it looks like a known one:

$ ./tests --jbos=4
error: '--jbos=4' is not an option; did you mean --jobs?

-- ends the options: what follows is test specs, even when it starts with -.

Test specs

Arguments that are not options select tests by name or tag, as in Catch2:

SpecSelects
"parses lists"the test of that name
"parses*"tests whose name starts with "parses"; * may stand at either end
[parser]tests tagged [parser]
[parser][fast]tests with both tags
~[slow]every test but those tagged [slow]
[parser],[lexer]tests with either tag
"a b" [x]specs separated by spaces must all match
[@alias]the tags of a tag alias

Without a spec, every test runs except hidden ones, whose tags start with .; a spec that names a hidden test runs it. A backslash makes ,, [, ], *, ~ and " literal: "commas\, and more".

Selecting tests

Option
-c, --section=NAMErun only this section; repeat for nested sections
-p, --path-filter=c:SECTION|g:INDEXrun one path through sections (c:) and generator values (g:, an index or *); repeat for each level. See running a section.
-f, --input-file=FILErun the tests named in FILE, one exact name per line; lines starting with # are comments
--select=WHICHall (default), failed, changed, new or smart; see choosing tests
--changes-since=REVISIONthe Git revision changed and smart compare with (default HEAD)
--incrementalskip tests that passed last time and whose inputs did not change
--shard=INDEX/COUNTrun one of COUNT shards, counting from 0
--shard-durations=FILEbalance shards by the durations in a JSON report or a history
-#, --filenames-as-tagstag every test with # and the name of its file, as [#parser.tests]

Running

Option
-j, --jobs=Nworker processes; 0, the default, uses one per hardware thread
--in-processrun the tests in this process, one after another
--isolatestart a fresh process for every test
--pass-env=NAMEgive the tests the variable NAME, which they do not get because its name looks like a secret (repeatable)
--timeout=SECONDSthe limit for a test without history, and the cap of learned limits; 0 for none
--no-adaptive-timeoutuse --timeout for every test
--retries=N|auto|offauto (default) retries what may pass again; N retries every failure up to N times
--repeat=Nrun every test N times, to find flaky tests
-a, --fail-fast[=N]start no test after N failed tests (default 1)
--order=ORDERpriority (default), declared, name or random
--seed=N|randomthe random seed; a new one each run by default
--no-bisectdo not search for the test that leaves state behind

How tests behave

Option
-s, --successreport passing assertions too, and show every assertion
-b, --breakbreak into the debugger on a failure; runs in process
-e, --no-throwskip the assertions that expect exceptions
-w, --warn=WHATfail on no-assertions, unmatched-specs or infinite-generators
--allow-no-testssucceed when no test runs
--skip-benchmarksdo not run BENCHMARK blocks
--benchmark-samples=Nsamples per benchmark (default 100)
--benchmark-resamples=Nbootstrap resamples (default 100000)
--benchmark-confidence-interval=Xbetween 0 and 1 (default 0.95)
--benchmark-no-analysisreport the mean of the samples without statistics
--benchmark-warmup-time=MILLISECONDSwarm-up time per benchmark (default 100)

History and analysis

Option
--history=DIRECTORYwhere results are remembered (default .catch3 beside the binary, or CATCH3_HISTORY)
--no-historyneither read nor write the history
--quarantine=MODEauto (default), report or off
--replayrun last run's failures again, each in the order and with the seed it failed with
--analytics[=json]print what the history says about the tests, and exit

Output

Option
-v, --verboseprint every test as it ends, with what it printed
-q, --quietprint only failures and the summary
--color=WHENauto (default), always or never
--junit=FILEwrite a JUnit XML report
--json=FILEwrite a JSON report with every attempt and its analysis
--trace=FILEwrite a trace for chrome://tracing or Perfetto
--coverage=DIRrun every test in a process of its own and keep its coverage in DIR, with a map of what each test reached; see Per-test coverage
--slowest[=N]list the N slowest tests (default 10)
--premature-exit-guard-file=FILEcreate FILE, private to the user, when the run starts and remove it when the run ends normally

Configuration

A project can keep its options in a catch3.json; options on the command line win over it.

Option
--config=FILEread this catch3.json instead of searching for one
--no-configread no catch3.json

Information

Option
-l, --list[=FORMAT]list the matching tests: text (default), names or json (format)
-t, --list-tagslist the tags of the matching tests
-h, --helpprint the options
--versionprint the version of Catch3

Environment

Variable
CATCH3_HISTORYthe history directory
CATCH3_PROPERTY_DATABASEthe example database of property tests
CATCH3_WORKERset in each worker to its number
TMPDIR, TMP, TEMPset while a test runs to its own directory
Bazel's test variablessee Bazel

Эта страница на GitHubЛицензии документации