[{"data":1,"prerenderedAt":2067},["ShallowReactive",2],{"doc-progmasoft-runner":3},{"id":4,"title":5,"body":6,"description":2057,"extension":2058,"meta":2059,"navigation":2060,"path":2061,"seo":2062,"slug":2063,"source":2064,"stem":2065,"__hash__":2066},"docs\u002Fdocs\u002Fprogmasoft-runner.md","The Catch3 runner",{"type":7,"value":8,"toc":2036},"minimark",[9,13,120,137,147,151,158,161,195,198,212,222,236,239,246,270,285,291,294,313,316,400,410,413,419,440,548,577,583,590,596,603,608,614,637,704,707,729,801,804,819,840,843,850,931,991,994,1016,1019,1047,1050,1054,1093,1096,1102,1112,1118,1121,1182,1185,1191,1207,1213,1244,1247,1297,1344,1354,1357,1368,1472,1475,1478,1485,1533,1544,1547,1762,1782,1801,1804,1842,1850,1853,1938,1941,1944,2032],[10,11,5],"h1",{"id":12},"the-catch3-runner",[14,15,16,20,23,28,30,34,36,40,42,46,48,52,54,58,60,64,66,70,72,76,78,82,84,88,90,94,96,100,102,106,108,112,114,118],"p",{},[17,18,19],"strong",{},"Contents",[21,22],"br",{},[24,25,27],"a",{"href":26},"#how-a-run-works","How a run works",[21,29],{},[24,31,33],{"href":32},"#retries-flaky-tests-and-order-dependence","Retries, flaky tests and order dependence",[21,35],{},[24,37,39],{"href":38},"#history-quarantine-and-analytics","History, quarantine and analytics",[21,41],{},[24,43,45],{"href":44},"#importing-the-results-of-other-frameworks","Importing the results of other frameworks",[21,47],{},[24,49,51],{"href":50},"#choosing-and-ordering-tests","Choosing and ordering tests",[21,53],{},[24,55,57],{"href":56},"#tags-the-runner-reads","Tags the runner reads",[21,59],{},[24,61,63],{"href":62},"#time-limits","Time limits",[21,65],{},[24,67,69],{"href":68},"#temporary-files","Temporary files",[21,71],{},[24,73,75],{"href":74},"#what-a-failure-report-says","What a failure report says",[21,77],{},[24,79,81],{"href":80},"#reports-and-traces","Reports and traces",[21,83],{},[24,85,87],{"href":86},"#per-test-coverage","Per-test coverage",[21,89],{},[24,91,93],{"href":92},"#formats-other-tools-read","Formats other tools read",[21,95],{},[24,97,99],{"href":98},"#project-configuration","Project configuration",[21,101],{},[24,103,105],{"href":104},"#bazel-and-ctest","Bazel and CTest",[21,107],{},[24,109,111],{"href":110},"#exit-codes","Exit codes",[21,113],{},[24,115,117],{"href":116},"#coming-from-catch2s-runner","Coming from Catch2's runner",[21,119],{},[14,121,122,123,127,128,131,132,136],{},"Every Catch3 test binary has the runner built in. It replaced Catch2's\nsession, command line and reporters; the API tests are written with stays\nCatch2's: ",[124,125,126],"code",{},"TEST_CASE",", ",[124,129,130],{},"SECTION",", the assertions, matchers, generators,\nbenchmarks, fixtures and listeners. The options are listed in the\n",[24,133,135],{"href":134},"\u002Fdocs\u002Fcommand-line\u002F","command line reference",".",[138,139,145],"pre",{"className":140,"code":142,"language":143,"meta":144},[141],"language-text","$ .\u002Ftests\nCatch3: 446 test(s), up to 8 worker(s), seed 2153251184\n\n446 test(s): 423 passed, 16 failed as expected, 7 skipped\n84193 assertion(s), 446 attempt(s); 1.2 s (6.0 s of tests, 5.0x in parallel)\n","text","",[124,146,142],{"__ignoreMap":144},[148,149,27],"h2",{"id":150},"how-a-run-works",[14,152,153,154,157],{},"The runner reads the registered tests, decides which to run and in which\norder, and hands them to worker processes: the same binary, started again\nwith an internal option. A free worker takes the next test, so the work\nspreads by itself and a long test does not hold back a shard of short ones.\n",[124,155,156],{},"--jobs=N"," sets the number of workers; the default is one per hardware\nthread.",[14,159,160],{},"Running tests in their own processes gives the run what one process\ncannot:",[162,163,164,171,181],"ul",{},[165,166,167,170],"li",{},[17,168,169],{},"A crash ends the test, not the run."," A worker that dies is replaced;\nthe test it was running is reported as crashed, with the signal or\nstructured exception, the last assertion it started and the section it\nwas in.",[165,172,173,176,177,180],{},[17,174,175],{},"A hang is stopped."," A test that runs past its ",[24,178,179],{"href":62},"time limit","\nis ended together with every process it started, and reported as timed\nout, with the section it was in.",[165,182,183,186,187,190,191,194],{},[17,184,185],{},"Tests that need it run alone."," ",[124,188,189],{},"--isolate",", or the tag ",[124,192,193],{},"[!isolated]",",\ngives a test a fresh process.",[14,196,197],{},"The runner, its workers and every process a test starts form one group: a\njob object on Windows, a process group elsewhere. Stopping a worker stops\nall of it.",[14,199,200,201,127,204,207,208,211],{},"What a test writes to ",[124,202,203],{},"std::cout",[124,205,206],{},"std::cerr"," and ",[124,209,210],{},"std::clog"," is captured\nand belongs to the test. Output of a worker outside any test, such as what a\nlistener prints when the run starts, is passed on to the runner's standard\nerror.",[14,213,214,217,218,221],{},[124,215,216],{},"--in-process"," runs the tests in the runner's own process, one after\nanother, for a debugger. ",[124,219,220],{},"--break"," implies it. Crashes and hangs are then not\ncontained.",[14,223,224,225,228,229,232,233,136],{},"Listeners registered with ",[124,226,227],{},"CATCH_REGISTER_LISTENER"," run in every worker:\neach worker is one test run for them, from ",[124,230,231],{},"testRunStarting"," to\n",[124,234,235],{},"testRunEnded",[148,237,33],{"id":238},"retries-flaky-tests-and-order-dependence",[14,240,241,242,245],{},"With ",[124,243,244],{},"--retries=auto",", the default, a failed test is tried once more, alone\nin a fresh process. What happens then tells three kinds of failure apart:",[162,247,248,251,258],{},[165,249,250],{},"It fails again: the failure is deterministic, and is reported.",[165,252,253,254,257],{},"It passes, and it had failed alone too: the test is ",[17,255,256],{},"flaky",". It counts as\npassed, is marked flaky in every report, and its history remembers it.",[165,259,260,261,265,266,269],{},"It passes alone, but had failed after other tests in a shared process:\nthe runner runs the same tests again in a fresh process, in the same\norder. If the test fails again, it depends on state another test leaves\nbehind, and the runner bisects the tests that ran before it to find which\none. The report names it: ",[262,263,264],"em",{},"fails after 'X', which leaves state behind",".\nSuch a failure fails the run; ",[124,267,268],{},"--no-bisect"," turns the search off.",[14,271,272,273,276,277,280,281,284],{},"Crashes, timeouts, tests in quarantine and tests with a flaky history get up\nto two retries. ",[124,274,275],{},"--retries=N"," retries every failure up to N times,\n",[124,278,279],{},"--retries=off"," never retries, and ",[124,282,283],{},"[!retries:N]"," sets the limit of one\ntest. A run spends at most a tenth of its tests on retries, so a broken\nbuild does not run everything three times.",[14,286,287,290],{},[124,288,289],{},"--repeat=N"," runs every test N times; a test whose verdicts differ is flaky.",[148,292,39],{"id":293},"history-quarantine-and-analytics",[14,295,296,297,300,301,304,305,308,309,312],{},"The runner remembers the results of the last 50 runs of each test: verdict,\nduration, flakiness, failure signature and a fingerprint of the inputs. The\nhistory of a binary lives in one JSON file in ",[124,298,299],{},".catch3"," beside the binary,\nor in the directory ",[124,302,303],{},"--history"," or the environment variable ",[124,306,307],{},"CATCH3_HISTORY","\nnames. ",[124,310,311],{},"--no-history"," neither reads nor writes it. Runs that finish at the\nsame time all reach the file: it is updated under a lock, read again, and\nreplaced in one rename.",[14,314,315],{},"The history drives several features:",[162,317,318,327,333,351,369,378,387],{},[165,319,320,323,324,136],{},[17,321,322],{},"Adaptive time limits."," See ",[24,325,326],{"href":62},"time limits",[165,328,329,332],{},[17,330,331],{},"Predictive ordering."," Each test gets a probability of failing: its\nfailure rate, with recent runs weighing more, and at least 70% after a\nfailure. Tests that are likely to fail run first, the most failures per\nsecond of running time first, so a failing run fails early. The rest run\nlongest first, which keeps every worker busy until the end.",[165,334,335,338,339,342,343,346,347,350],{},[17,336,337],{},"Quarantine."," A test with two flaky results in its last 20 runs goes\ninto quarantine. It still runs and is reported, but its failure does not\nfail the run. Ten clean passes in a row release it. ",[124,340,341],{},"[!quarantine]","\nquarantines a test by hand; ",[124,344,345],{},"--quarantine=report"," reports quarantine\nwithout excusing failures, and ",[124,348,349],{},"--quarantine=off"," ignores it.",[165,352,353,356,357,360,361,364,365,368],{},[17,354,355],{},"Classification."," A failure is ",[262,358,359],{},"new"," (it passed last time),\n",[262,362,363],{},"persistent"," (it failed last time too) or ",[262,366,367],{},"intermittent"," (it flakes).",[165,370,371,186,374,377],{},[17,372,373],{},"Replay.",[124,375,376],{},"--replay"," runs the last failures again, each with the seed it\nfailed with and after the same tests, in a fresh process.",[165,379,380,186,383,386],{},[17,381,382],{},"Incremental runs.",[124,384,385],{},"--incremental"," skips tests that passed last time\nwith the same fingerprint; they are reported as up to date.",[165,388,389,186,392,395,396,399],{},[17,390,391],{},"Analytics.",[124,393,394],{},"--analytics"," prints what the history says: flaky tests,\ntests in quarantine, the tests that fail most often, the slowest tests and\nthe tests getting slower. ",[124,397,398],{},"--analytics=json"," prints it as JSON.",[14,401,402,403,406,407,409],{},"Under ",[124,404,405],{},"bazel test"," there is no history unless ",[124,408,307],{}," names a\ndirectory that outlives the sandbox.",[148,411,45],{"id":412},"importing-the-results-of-other-frameworks",[14,414,415,418],{},[124,416,417],{},"--import=FILE"," reads the report of another test framework instead of running\ntests, so that tests Catch3 does not run get the same history, flaky detection,\nquarantine and analytics. Any Catch3 test binary can do it, and the option can\nbe repeated:",[138,420,424],{"className":421,"code":422,"language":423,"meta":144,"style":144},"language-sh shiki shiki-themes github-dark",".\u002Ftests --import=results\u002Fpytest.xml --import=results\u002Fvitest.ctrf.json \\\n        --import-name=web --junit=merged.xml\n","sh",[124,425,426,434],{"__ignoreMap":144},[427,428,431],"span",{"class":429,"line":430},"line",1,[427,432,433],{},".\u002Ftests --import=results\u002Fpytest.xml --import=results\u002Fvitest.ctrf.json \\\n",[427,435,437],{"class":429,"line":436},2,[427,438,439],{},"        --import-name=web --junit=merged.xml\n",[441,442,443,458],"table",{},[444,445,446],"thead",{},[447,448,449,455],"tr",{},[450,451,452],"th",{},[124,453,454],{},"--import-format=",[450,456,457],{},"Reads",[459,460,461,472,492,502,512,522,532],"tbody",{},[447,462,463,469],{},[464,465,466],"td",{},[124,467,468],{},"auto",[464,470,471],{},"whichever the contents show (the default)",[447,473,474,479],{},[464,475,476],{},[124,477,478],{},"catch3",[464,480,481,482,485,486],{},"a ",[124,483,484],{},"--json"," report of Catch3, or ",[24,487,489],{"href":488},"#the-catch3-results-format",[124,490,491],{},"catch3-results",[447,493,494,499],{},[464,495,496],{},[124,497,498],{},"junit",[464,500,501],{},"JUnit XML, with Surefire's reruns; Gradle, pytest, Jest, GoogleTest and most others write it",[447,503,504,509],{},[464,505,506],{},[124,507,508],{},"xunit",[464,510,511],{},"xUnit.net v2 XML",[447,513,514,519],{},[464,515,516],{},[124,517,518],{},"trx",[464,520,521],{},"Visual Studio's TRX, with data-driven rows",[447,523,524,529],{},[464,525,526],{},[124,527,528],{},"tap",[464,530,531],{},"TAP 12, 13 and 14, with subtests and YAML diagnostics",[447,533,534,539],{},[464,535,536],{},[124,537,538],{},"ctrf",[464,540,541,547],{},[24,542,546],{"href":543,"rel":544},"https:\u002F\u002Fctrf.io",[545],"nofollow","CTRF"," JSON, with retries",[14,549,550,551,554,555,558,559,562,563,127,566,127,568,127,571,127,574,576],{},"The results go into a history of their own, named by ",[124,552,553],{},"--import-name"," and\n",[124,556,557],{},"imported"," by default; ",[124,560,561],{},"--analytics --import-name=NAME"," reads it. The console,\n",[124,564,565],{},"--junit",[124,567,484],{},[124,569,570],{},"--trace",[124,572,573],{},"--quarantine",[124,575,303],{}," and the exit code\nwork as they do for a run. Reports that were read count as one run.",[14,578,579,580,582],{},"A test is identified by its class and its name. Results with the same\nidentifier, in one report or in several, are attempts at one test: a test that\nboth failed and passed is flaky, passes and is remembered as flaky, which is how\na report of a rerun is imported. A failure stays a failure when a later report\nskips the test. Reports of the same tests on different platforms are not reruns\nof each other: import each with an ",[124,581,553],{}," of its own, or a failure on one\nreads as flakiness.",[14,584,585,586,589],{},"TAP names tests only within one stream, so the path given to ",[124,587,588],{},"--import"," is the\nclass of its tests: keep it the same from run to run for the history to follow\nthem. A name that comes back in one stream is another test and gets a number.",[14,591,592,595],{},[124,593,594],{},"--import-revision=REVISION"," names what the reports tested, and a verdict that\nchanges on one revision then counts as flaky too. Without it only retries in the\nreports do, because a working directory can have edits or be another checkout.",[14,597,598,599,602],{},"Nothing is run, so an imported failure has no replay command, and only what the\nreport says is known: it has no order, seed or sections. A TAP stream that bails\nout or reports fewer tests than it planned adds a failed test called\n",[124,600,601],{},"the TAP stream",", so that it does not read as a run that passed.",[604,605,607],"h3",{"id":606},"the-catch3-results-format","The catch3-results format",[14,609,610,611,613],{},"For a framework without a report of these kinds, ",[124,612,491],{}," is the\nsmallest thing to write: JSON Lines with one test per line, after a line that\nnames the format.",[138,615,619],{"className":616,"code":617,"language":618,"meta":144,"style":144},"language-json shiki shiki-themes github-dark","{\"format\": \"catch3-results\", \"version\": 1}\n{\"name\": \"parses a date\", \"className\": \"Dates\", \"outcome\": \"passed\", \"durationMicroseconds\": 1200}\n{\"name\": \"rounds down\", \"outcome\": \"failed\", \"message\": \"1.5 != 2\", \"details\": \"at Dates.rounds:12\"}\n","json",[124,620,621,626,631],{"__ignoreMap":144},[427,622,623],{"class":429,"line":430},[427,624,625],{},"{\"format\": \"catch3-results\", \"version\": 1}\n",[427,627,628],{"class":429,"line":436},[427,629,630],{},"{\"name\": \"parses a date\", \"className\": \"Dates\", \"outcome\": \"passed\", \"durationMicroseconds\": 1200}\n",[427,632,634],{"class":429,"line":633},3,[427,635,636],{},"{\"name\": \"rounds down\", \"outcome\": \"failed\", \"message\": \"1.5 != 2\", \"details\": \"at Dates.rounds:12\"}\n",[14,638,639,642,643,646,647,650,651,653,654,127,657,127,660,127,663,642,666,669,670,127,672,127,675,127,678,127,680,127,683,127,686,689,690,127,693,207,696,699,700,703],{},[124,640,641],{},"name"," or ",[124,644,645],{},"id",", and ",[124,648,649],{},"outcome",", are required; ",[124,652,649],{}," is a name from the JSON\nreport, such as ",[124,655,656],{},"passed",[124,658,659],{},"failed",[124,661,662],{},"skipped",[124,664,665],{},"crashed",[124,667,668],{},"timed-out",". The other\nmembers are ",[124,671,645],{},[124,673,674],{},"className",[124,676,677],{},"file",[124,679,429],{},[124,681,682],{},"tags",[124,684,685],{},"category",[124,687,688],{},"message",",\n",[124,691,692],{},"details",[124,694,695],{},"standardOutput",[124,697,698],{},"standardError",". A line can give ",[124,701,702],{},"attempts","\ninstead, as the JSON report does.",[148,705,51],{"id":706},"choosing-and-ordering-tests",[14,708,709,710,713,714,127,717,720,721,724,725,728],{},"Test specs work as in Catch2: names with ",[124,711,712],{},"*"," as a wildcard, ",[124,715,716],{},"[tags]",[124,718,719],{},"~","\nto exclude, a comma for \"or\". ",[124,722,723],{},"--input-file"," reads exact names from a file.\n",[124,726,727],{},"--select"," narrows the matching tests further:",[441,730,731,743],{},[444,732,733],{},[447,734,735,740],{},[450,736,737],{},[124,738,739],{},"--select=",[450,741,742],{},"Runs",[459,744,745,755,764,782,791],{},[447,746,747,752],{},[464,748,749],{},[124,750,751],{},"all",[464,753,754],{},"every matching test (default)",[447,756,757,761],{},[464,758,759],{},[124,760,659],{},[464,762,763],{},"tests that failed, crashed, timed out or flaked last time",[447,765,766,771],{},[464,767,768],{},[124,769,770],{},"changed",[464,772,773,774,777,778,781],{},"tests defined in files Git reports as changed since ",[124,775,776],{},"--changes-since"," (default ",[124,779,780],{},"HEAD","), staged or not, and untracked files",[447,783,784,788],{},[464,785,786],{},[124,787,359],{},[464,789,790],{},"tests without a recorded result",[447,792,793,798],{},[464,794,795],{},[124,796,797],{},"smart",[464,799,800],{},"changed, failed and new tests, those in quarantine, and those at least 50% likely to fail",[14,802,803],{},"Each selected test carries the reason it was selected into the reports.",[14,805,806,809,810,689,813,207,815,818],{},[124,807,808],{},"--order=priority"," (default) orders by the prediction above; ",[124,811,812],{},"declared",[124,814,641],{},[124,816,817],{},"random"," are also available. Every test is seeded with the\nrun's seed, wherever it runs, so the order does not change its random\nnumbers.",[14,820,821,824,825,828,829,832,833,836,837,136],{},[124,822,823],{},"--shard=INDEX\u002FCOUNT"," runs one of COUNT shards. A test goes to the shard a\nstable hash of its name chooses, so every machine agrees without talking to\nthe others; tests linked by ",[124,826,827],{},"[!depends]"," stay in one shard. With\n",[124,830,831],{},"--shard-durations=FILE",", a history or a JSON report, shards are balanced by\nduration instead, longest first into the least loaded shard. ",[124,834,835],{},"--list","\nhonours ",[124,838,839],{},"--shard",[148,841,57],{"id":842},"tags-the-runner-reads",[14,844,845,846,849],{},"Tags that start with ",[124,847,848],{},"!"," are reserved, as in Catch2. The runner adds these:",[441,851,852,862],{},[444,853,854],{},[447,855,856,859],{},[450,857,858],{},"Tag",[450,860,861],{},"Effect",[459,863,864,874,883,893,903,913,922],{},[447,865,866,871],{},[464,867,868],{},[124,869,870],{},"[!serial]",[464,872,873],{},"nothing else runs while this test runs",[447,875,876,880],{},[464,877,878],{},[124,879,193],{},[464,881,882],{},"the test runs in a process of its own",[447,884,885,890],{},[464,886,887],{},[124,888,889],{},"[!resource:NAME]",[464,891,892],{},"tests that share a resource never run at the same time",[447,894,895,900],{},[464,896,897],{},[124,898,899],{},"[!depends:TEST]",[464,901,902],{},"runs after TEST, and only if TEST passes; otherwise it is reported as blocked. TEST is selected along with it. Cycles are an error.",[447,904,905,910],{},[464,906,907],{},[124,908,909],{},"[!timeout:SECONDS]",[464,911,912],{},"the time limit of this test",[447,914,915,919],{},[464,916,917],{},[124,918,283],{},[464,920,921],{},"the retry limit of this test",[447,923,924,928],{},[464,925,926],{},[124,927,341],{},[464,929,930],{},"the test is in quarantine",[138,932,936],{"className":933,"code":934,"language":935,"meta":144,"style":144},"language-cpp shiki shiki-themes github-dark","TEST_CASE(\"migrates the schema\", \"[db][!resource:database]\") { ... }\nTEST_CASE(\"reads after migration\", \"[db][!resource:database][!depends:migrates the schema]\") { ... }\nTEST_CASE(\"renders the full catalogue\", \"[!timeout:120]\") { ... }\n","cpp",[124,937,938,959,975],{"__ignoreMap":144},[427,939,940,943,947,951,953,956],{"class":429,"line":430},[427,941,126],{"class":942},"svObZ",[427,944,946],{"class":945},"s95oV","(",[427,948,950],{"class":949},"sU2Wk","\"migrates the schema\"",[427,952,127],{"class":945},[427,954,955],{"class":949},"\"[db][!resource:database]\"",[427,957,958],{"class":945},") { ... }\n",[427,960,961,963,965,968,970,973],{"class":429,"line":436},[427,962,126],{"class":942},[427,964,946],{"class":945},[427,966,967],{"class":949},"\"reads after migration\"",[427,969,127],{"class":945},[427,971,972],{"class":949},"\"[db][!resource:database][!depends:migrates the schema]\"",[427,974,958],{"class":945},[427,976,977,979,981,984,986,989],{"class":429,"line":633},[427,978,126],{"class":942},[427,980,946],{"class":945},[427,982,983],{"class":949},"\"renders the full catalogue\"",[427,985,127],{"class":945},[427,987,988],{"class":949},"\"[!timeout:120]\"",[427,990,958],{"class":945},[148,992,63],{"id":993},"time-limits",[14,995,996,997,1000,1001,1004,1005,1008,1009,1012,1013,1015],{},"A test with history gets a limit learned from it: four times its 95th\npercentile or ten times its median, whichever is larger, plus two seconds,\nand at least five seconds. A test without history gets 300 seconds.\n",[124,998,999],{},"--timeout=SECONDS"," sets the limit for tests without history and caps the\nlearned ones; ",[124,1002,1003],{},"--timeout=0"," turns limits off; ",[124,1006,1007],{},"--no-adaptive-timeout"," uses\n",[124,1010,1011],{},"--timeout"," for every test. ",[124,1014,909],{}," overrides all of these.\nBenchmarks get no learned limit.",[148,1017,69],{"id":1018},"temporary-files",[14,1020,1021,1022,127,1025,207,1028,1031,1032,127,1035,1038,1039,1042,1043,1046],{},"Every attempt of every test gets a temporary directory of its own. While the\ntest runs, ",[124,1023,1024],{},"TMPDIR",[124,1026,1027],{},"TMP",[124,1029,1030],{},"TEMP"," point at it, so\n",[124,1033,1034],{},"std::filesystem::temp_directory_path()",[124,1036,1037],{},"tmpfile"," and most libraries use\nit, and tests that run at the same time never share a scratch file by\naccident. ",[124,1040,1041],{},"Progmasoft::Catch3::TestTemporaryDirectory()"," in\n",[124,1044,1045],{},"\u003CProgmasoft\u002FCatch3\u002FRunner.hpp>"," returns it.",[14,1048,1049],{},"The directory is removed when the attempt passes and kept when it fails;\nthe report says where.",[148,1051,1053],{"id":1052},"environment-variables","Environment variables",[14,1055,1056,1057,127,1060,127,1063,127,1066,127,1069,689,1072,127,1075,127,1078,642,1081,1084,1085,1088,1089,1092],{},"Workers, and the processes the runner starts for itself, inherit its\nenvironment without the variables whose names look like secrets: those that\ncontain, in any case, ",[124,1058,1059],{},"TOKEN",[124,1061,1062],{},"SECRET",[124,1064,1065],{},"PASSWORD",[124,1067,1068],{},"PASSWD",[124,1070,1071],{},"CREDENTIAL",[124,1073,1074],{},"API_KEY",[124,1076,1077],{},"APIKEY",[124,1079,1080],{},"PRIVATE_KEY",[124,1082,1083],{},"ACCESS_KEY",". A test that needs one\ngets it with ",[124,1086,1087],{},"--pass-env=NAME",", which can be repeated, or ",[124,1090,1091],{},"\"pass-env\""," in\nthe project configuration. The runner itself starts with the whole\nenvironment, so code that runs before the first worker starts still sees\nthem.",[148,1094,75],{"id":1095},"what-a-failure-report-says",[138,1097,1100],{"className":1098,"code":1099,"language":143,"meta":144},[141],"FAIL: parses nested lists\ntests\u002Fparser.cpp:120\n  tests\u002Fparser.cpp:131: FAILED:\n    REQUIRE( tokens.size() == 3 )\n  with expansion:\n    2 == 3\n  in section with commas\n  analysis: new failure after 14 passing run(s); it last passed at commit 4c40a97b2f1e\n  analysis: fails the same way as 'parses flat lists': probably one cause\n  analysis: changed since HEAD: src\u002Flexer.cpp\n  replay: .\u002Ftests \"parses nested lists\" --seed=2153251184\n",[124,1101,1099],{"__ignoreMap":144},[14,1103,1104,1105,1107,1108,1111],{},"Every failure has a ",[17,1106,685],{},": assertion, unexpected exception, missing\nexception, explicit failure, crash, timeout, dependency or infrastructure.\nIt has a ",[17,1109,1110],{},"signature",": the category, the location of the first failure and\nits message with numbers and addresses removed. Tests that share a\nsignature are named together, since they probably share a cause.",[14,1113,1114,1115,1117],{},"The analysis adds what the runner knows: how the failure relates to the\nhistory, the commit the test last passed at, the change it follows (the\ntest's file, or the file of the failing assertion, changed since\n",[124,1116,776],{},"), where a crash or a hang happened, the test that leaves\nstate behind, the quarantine, and where the temporary files were kept. The\nreplay line runs the failure again as it happened.",[148,1119,81],{"id":1120},"reports-and-traces",[162,1122,1123,1146,1160,1166],{},[165,1124,1125,1126,1129,1130,1133,1134,1137,1138,1141,1142,1145],{},"The ",[17,1127,1128],{},"console report"," shows progress and the failures with their\nanalysis, then a summary. ",[124,1131,1132],{},"--verbose"," shows every test and what passing\ntests printed; ",[124,1135,1136],{},"--success"," also shows every assertion; ",[124,1139,1140],{},"--quiet"," shows\nonly failures. ",[124,1143,1144],{},"--slowest=N"," lists the slowest tests.",[165,1147,1148,1151,1152,1155,1156,1159],{},[124,1149,1150],{},"--junit=FILE"," writes JUnit XML. Retries are ",[124,1153,1154],{},"\u003CrerunFailure>"," elements,\nand a flaky test is a success with ",[124,1157,1158],{},"\u003CflakyFailure>"," elements, which\nJenkins, GitLab and most CI dashboards understand.",[165,1161,1162,1165],{},[124,1163,1164],{},"--json=FILE"," writes everything the runner knows: every attempt, its\nassertions, sections and benchmarks, the captured output, the analysis.",[165,1167,1168,1171,1172,1175,1176,1181],{},[124,1169,1170],{},"--trace=FILE"," writes a trace for ",[124,1173,1174],{},"chrome:\u002F\u002Ftracing"," or\n",[24,1177,1180],{"href":1178,"rel":1179},"https:\u002F\u002Fui.perfetto.dev",[545],"Perfetto",": one track per worker, one slice per\nattempt and section.",[148,1183,87],{"id":1184},"per-test-coverage",[14,1186,1187,1190],{},[124,1188,1189],{},"--coverage=DIR"," runs every test in a process of its own and keeps what each\none reached. Build the tests with coverage instrumentation:",[162,1192,1193,1200],{},[165,1194,1195,1196,1199],{},"Clang: ",[124,1197,1198],{},"-fprofile-instr-generate -fcoverage-mapping",", LLVM source-based\ncoverage;",[165,1201,1202,1203,1206],{},"GCC: ",[124,1204,1205],{},"--coverage",", gcov's format.",[138,1208,1211],{"className":1209,"code":1210,"language":143,"meta":144},[141],"$ .\u002Ftests --coverage=coverage\nCatch3: 120 test(s), up to 8 worker(s), seed 2153251184\nnote: --coverage: 120 test profile(s); what each test reached is in coverage\u002Fcoverage.json, the run combined in coverage\u002Fcoverage.lcov\n\n120 test(s): 120 passed\n",[124,1212,1210],{"__ignoreMap":144},[14,1214,1215,1216,1219,1220,1223,1224,1227,1228,1231,1232,1235,1236,1239,1240,1243],{},"Each attempt gets a process of its own, so the profile its process writes at\nexit is the coverage of that one test: ",[124,1217,1218],{},"DIR\u002Fprofiles\u002Ft\u003Cindex>-a\u003Cattempt>","\nwith ",[124,1221,1222],{},".profraw"," for LLVM, or a ",[124,1225,1226],{},".gcov"," directory below which gcov writes its\n",[124,1229,1230],{},".gcda"," files under their absolute paths. The runner sets ",[124,1233,1234],{},"LLVM_PROFILE_FILE","\nand ",[124,1237,1238],{},"GCOV_PREFIX"," for the process; each runtime reads its own. A process\nthat runs other tests first, to find a test that leaves state behind, writes\nbeside the profiles, never into one. The profiles and reports of an earlier\nrun in ",[124,1241,1242],{},"DIR"," are removed when the run starts: gcov adds to the data it finds,\nand a run that writes no reports leaves none to be read as its own.",[14,1245,1246],{},"After the run, the runner reads, on as many threads as the run had workers,\nthe profile of the last attempt of every test that left one; when a retry or\nrepeat crashed before writing its profile, the attempt before it counts:",[162,1248,1249,1287],{},[165,1250,1251,1254,1255],{},[124,1252,1253],{},"DIR\u002Fcoverage.json"," holds, for each test, its outcome, its profile and the\nfiles it reached, with the lines it ran and, per line, how many of its\nbranches it took:",[138,1256,1258],{"className":616,"code":1257,"language":618,"meta":144,"style":144},"{ \"version\": 1, \"format\": \"llvm\", \"binary\": \".\u002Ftests\",\n  \"tests\": [ { \"name\": \"Adds one\", \"outcome\": \"passed\",\n               \"profile\": \"coverage\u002Fprofiles\u002Ft0-a1.profraw\",\n               \"files\": [ { \"file\": \"\u002Fsrc\u002Fcalc.cpp\", \"lines\": [ 12, 13 ],\n                            \"branches\": [ { \"line\": 12, \"taken\": 1, \"total\": 2 } ] } ] } ] }\n",[124,1259,1260,1265,1270,1275,1281],{"__ignoreMap":144},[427,1261,1262],{"class":429,"line":430},[427,1263,1264],{},"{ \"version\": 1, \"format\": \"llvm\", \"binary\": \".\u002Ftests\",\n",[427,1266,1267],{"class":429,"line":436},[427,1268,1269],{},"  \"tests\": [ { \"name\": \"Adds one\", \"outcome\": \"passed\",\n",[427,1271,1272],{"class":429,"line":633},[427,1273,1274],{},"               \"profile\": \"coverage\u002Fprofiles\u002Ft0-a1.profraw\",\n",[427,1276,1278],{"class":429,"line":1277},4,[427,1279,1280],{},"               \"files\": [ { \"file\": \"\u002Fsrc\u002Fcalc.cpp\", \"lines\": [ 12, 13 ],\n",[427,1282,1284],{"class":429,"line":1283},5,[427,1285,1286],{},"                            \"branches\": [ { \"line\": 12, \"taken\": 1, \"total\": 2 } ] } ] } ] }\n",[165,1288,1289,1292,1293,1296],{},[124,1290,1291],{},"DIR\u002Fcoverage.lcov"," is the coverage of the whole run as an LCOV tracefile,\nwhich ",[124,1294,1295],{},"genhtml",", Codecov and editors read.",[14,1298,1299,1300,207,1303,1306,1307,1310,1311,1314,1315,1318,1319,127,1322,207,1325,1328,1329,1332,1333,1336,1337,1340,1341,1343],{},"LLVM profiles are read with ",[124,1301,1302],{},"llvm-profdata",[124,1304,1305],{},"llvm-cov",", gcov data with\n",[124,1308,1309],{},"gcov",". They are looked up on ",[124,1312,1313],{},"PATH",", through ",[124,1316,1317],{},"xcrun"," on macOS, or named by\n",[124,1320,1321],{},"CATCH3_LLVM_PROFDATA",[124,1323,1324],{},"CATCH3_LLVM_COV",[124,1326,1327],{},"CATCH3_GCOV",". Without them the\nprofiles stay in ",[124,1330,1331],{},"DIR\u002Fprofiles"," and the run says so, as it does when the\ntests wrote no profiles; neither fails the run. A profile the tools cannot\nread, or a report that cannot be written, fails a run whose tests passed,\nlike any report the run was asked for. A test that crashes or is stopped at\nits time limit leaves no profile. gcov is run with ",[124,1334,1335],{},"--branch-probabilities",",\nwithout which its JSON has no branches. The runner's own process is\ninstrumented too and writes its profile where any instrumented program does,\n",[124,1338,1339],{},"default.profraw"," in the working directory unless ",[124,1342,1234],{}," says\notherwise; it holds no test.",[14,1345,1346,1348,1349,642,1351,1353],{},[124,1347,1205],{}," cannot be used with ",[124,1350,216],{},[124,1352,220],{},", which run the\ntests in one process. Per-test coverage is tested on Linux and macOS; Windows\nbuilds with clang-cl are next.",[148,1355,93],{"id":1356},"formats-other-tools-read",[14,1358,1359,1360,1363,1364,1367],{},"These formats carry a ",[124,1361,1362],{},"format"," and a ",[124,1365,1366],{},"version",". Within a version they only\ngain members; a reader ignores members it does not know and refuses a newer\nversion.",[162,1369,1370,1425,1436,1450,1461],{},[165,1371,1372,127,1375,1378,1379,1382,1383,127,1385,127,1387,127,1389,127,1391,127,1393,689,1395,1398,1399,127,1402,689,1405,127,1408,127,1411,127,1414,207,1417,1420,1421,1424],{},[17,1373,1374],{},"Discovery",[124,1376,1377],{},"--list=json",": ",[124,1380,1381],{},"\"format\": \"catch3-tests\"",", version 1. One\nobject per test with ",[124,1384,645],{},[124,1386,641],{},[124,1388,674],{},[124,1390,677],{},[124,1392,429],{},[124,1394,682],{},[124,1396,1397],{},"hidden",", and what the runner tags say: ",[124,1400,1401],{},"serial",[124,1403,1404],{},"isolated",[124,1406,1407],{},"quarantined",[124,1409,1410],{},"resources",[124,1412,1413],{},"dependsOn",[124,1415,1416],{},"timeoutSeconds",[124,1418,1419],{},"retries",".\nCMake's ",[124,1422,1423],{},"catch3_discover_tests"," reads it.",[165,1426,1427,127,1430,1378,1432,1435],{},[17,1428,1429],{},"Run report",[124,1431,484],{},[124,1433,1434],{},"\"format\": \"catch3-run\"",", version 1.",[165,1437,1438,127,1441,1378,1443,1446,1447,136],{},[17,1439,1440],{},"Imported results",[124,1442,588],{},[124,1444,1445],{},"\"format\": \"catch3-results\"",", version 1;\nsee ",[24,1448,1449],{"href":488},"its format",[165,1451,1452,127,1455,1378,1457,1460],{},[17,1453,1454],{},"Analytics",[124,1456,398],{},[124,1458,1459],{},"\"format\": \"catch3-analytics\"",",\nversion 1.",[165,1462,1463,1378,1466,1469,1470,136],{},[17,1464,1465],{},"History",[124,1467,1468],{},"\"format\": \"catch3-history\"",", version 1. It is the runner's\nown file; read it through ",[124,1471,394],{},[14,1473,1474],{},"The messages between the runner and its workers are internal and change\nwith the binary.",[148,1476,99],{"id":1477},"project-configuration",[14,1479,1480,1481,1484],{},"Options a project always wants go in a ",[124,1482,1483],{},"catch3.json",":",[138,1486,1488],{"className":616,"code":1487,"language":618,"meta":144,"style":144},"{\n  \"version\": 1,\n  \"tests\": [\"~[slow]\"],\n  \"jobs\": 8,\n  \"timeout\": 60,\n  \"warn\": [\"no-assertions\"],\n  \"junit\": \"build\u002Ftest-results\u002Fjunit.xml\"\n}\n",[124,1489,1490,1495,1500,1505,1510,1515,1521,1527],{"__ignoreMap":144},[427,1491,1492],{"class":429,"line":430},[427,1493,1494],{},"{\n",[427,1496,1497],{"class":429,"line":436},[427,1498,1499],{},"  \"version\": 1,\n",[427,1501,1502],{"class":429,"line":633},[427,1503,1504],{},"  \"tests\": [\"~[slow]\"],\n",[427,1506,1507],{"class":429,"line":1277},[427,1508,1509],{},"  \"jobs\": 8,\n",[427,1511,1512],{"class":429,"line":1283},[427,1513,1514],{},"  \"timeout\": 60,\n",[427,1516,1518],{"class":429,"line":1517},6,[427,1519,1520],{},"  \"warn\": [\"no-assertions\"],\n",[427,1522,1524],{"class":429,"line":1523},7,[427,1525,1526],{},"  \"junit\": \"build\u002Ftest-results\u002Fjunit.xml\"\n",[427,1528,1530],{"class":429,"line":1529},8,[427,1531,1532],{},"}\n",[14,1534,1535,1536,1539,1540,1543],{},"The runner looks for the file in the working directory and in each\ndirectory above it, up to the root of the Git repository, and reads the\nnearest one. Outside a repository it looks only in the working directory.\n",[124,1537,1538],{},"--config=FILE"," names a file instead, and ",[124,1541,1542],{},"--no-config"," reads none.",[14,1545,1546],{},"The keys are the long options, and their values go through the same checks\nas on the command line, with the key named in the message:",[441,1548,1549,1559],{},[444,1550,1551],{},[447,1552,1553,1556],{},[450,1554,1555],{},"Value",[450,1557,1558],{},"Keys",[459,1560,1561,1598,1614,1636,1664,1688,1702,1727,1744],{},[447,1562,1563,1569],{},[464,1564,1565,1568],{},[124,1566,1567],{},"true"," turns the switch on",[464,1570,1571,127,1574,127,1577,127,1580,127,1583,127,1586,127,1589,127,1592,127,1595],{},[124,1572,1573],{},"incremental",[124,1575,1576],{},"filenames-as-tags",[124,1578,1579],{},"in-process",[124,1581,1582],{},"isolate",[124,1584,1585],{},"success",[124,1587,1588],{},"no-throw",[124,1590,1591],{},"allow-no-tests",[124,1593,1594],{},"skip-benchmarks",[124,1596,1597],{},"benchmark-no-analysis",[447,1599,1600,1606],{},[464,1601,1602,1605],{},[124,1603,1604],{},"false"," turns it off",[464,1607,1608,127,1611],{},[124,1609,1610],{},"adaptive-timeout",[124,1612,1613],{},"bisect",[447,1615,1616,1619],{},[464,1617,1618],{},"a string",[464,1620,1621,127,1624,127,1627,127,1630,127,1633],{},[124,1622,1623],{},"select",[124,1625,1626],{},"changes-since",[124,1628,1629],{},"order",[124,1631,1632],{},"quarantine",[124,1634,1635],{},"color",[447,1637,1638,1641],{},[464,1639,1640],{},"a whole number",[464,1642,1643,127,1646,127,1649,127,1652,127,1655,127,1658,127,1661],{},[124,1644,1645],{},"jobs",[124,1647,1648],{},"repeat",[124,1650,1651],{},"fail-fast",[124,1653,1654],{},"slowest",[124,1656,1657],{},"benchmark-samples",[124,1659,1660],{},"benchmark-resamples",[124,1662,1663],{},"benchmark-warmup-time",[447,1665,1666,1669],{},[464,1667,1668],{},"a whole number or a string",[464,1670,1671,1673,1674,127,1677,1680,1681,1673,1684,1687],{},[124,1672,1419],{}," (",[124,1675,1676],{},"\"auto\"",[124,1678,1679],{},"\"off\"","), ",[124,1682,1683],{},"seed",[124,1685,1686],{},"\"random\"",")",[447,1689,1690,1693],{},[464,1691,1692],{},"a number",[464,1694,1695,1698,1699],{},[124,1696,1697],{},"timeout"," (seconds), ",[124,1700,1701],{},"benchmark-confidence-interval",[447,1703,1704,1707],{},[464,1705,1706],{},"a path, relative to the file",[464,1708,1709,127,1711,127,1713,127,1716,1719,1720,1723,1724,1726],{},[124,1710,498],{},[124,1712,618],{},[124,1714,1715],{},"trace",[124,1717,1718],{},"shard-durations","; ",[124,1721,1722],{},"history",", or ",[124,1725,1604],{}," for none",[447,1728,1729,1732],{},[464,1730,1731],{},"an array of strings",[464,1733,1734,1737,1738,127,1741],{},[124,1735,1736],{},"tests"," (test specs), ",[124,1739,1740],{},"warn",[124,1742,1743],{},"pass-env",[447,1745,1746,1757],{},[464,1747,1748,127,1751,642,1754],{},[124,1749,1750],{},"\"quiet\"",[124,1752,1753],{},"\"normal\"",[124,1755,1756],{},"\"verbose\"",[464,1758,1759],{},[124,1760,1761],{},"verbosity",[14,1763,1764,1767,1768,1771,1772,127,1774,127,1777,1175,1779,1781],{},[124,1765,1766],{},"\"version\": 1"," is required; ",[124,1769,1770],{},"\"$schema\""," may name a schema for editors.\nThe file is strict JSON: no comments and no trailing commas. A key it does\nnot know is an error, with a suggestion when it looks like a known one.\nOptions that describe one run, such as ",[124,1773,839],{},[124,1775,1776],{},"--section",[124,1778,835],{},[124,1780,376],{},", belong on the command line only.",[14,1783,1784,1785,1787,1788,1790,1791,1793,1794,554,1797,1800],{},"Options on the command line win over the file, and test specs on the\ncommand line replace the file's ",[124,1786,1736],{}," rather than adding to them. A\nswitch the file turns on stays on; ",[124,1789,1542],{}," leaves the file out.\n",[124,1792,1132],{}," names the file that was read. Workers, ",[124,1795,1796],{},"--help",[124,1798,1799],{},"--version"," read no file.",[148,1802,105],{"id":1803},"bazel-and-ctest",[14,1805,402,1806,1808,1809,1812,1813,1816,1817,689,1820,127,1823,207,1826,1829,1830,1833,1834,1837,1838,1841],{},[124,1807,405],{}," the runner reads Bazel's environment: ",[124,1810,1811],{},"XML_OUTPUT_FILE","\nfor a JUnit report, ",[124,1814,1815],{},"TESTBRIDGE_TEST_ONLY"," for ",[124,1818,1819],{},"--test_filter",[124,1821,1822],{},"TEST_SHARD_INDEX",[124,1824,1825],{},"TEST_TOTAL_SHARDS",[124,1827,1828],{},"TEST_SHARD_STATUS_FILE"," for\nsharding, ",[124,1831,1832],{},"TEST_RANDOM_SEED"," for the seed, ",[124,1835,1836],{},"TEST_PREMATURE_EXIT_FILE"," for\nthe exit guard and ",[124,1839,1840],{},"TEST_UNDECLARED_OUTPUTS_DIR"," for a JSON report.",[14,1843,1844,1849],{},[24,1845,1847],{"href":1846},"\u002Fdocs\u002Fcmake-integration\u002F#automatic-test-registration",[124,1848,1423],{},"\nregisters each test with CTest through the discovery format. The runner\nalso parallelises a whole binary under one CTest test, which usually starts\nfewer processes.",[148,1851,111],{"id":1852},"exit-codes",[441,1854,1855,1865],{},[444,1856,1857],{},[447,1858,1859,1862],{},[450,1860,1861],{},"Code",[450,1863,1864],{},"Meaning",[459,1866,1867,1882,1890,1902,1914,1922,1930],{},[447,1868,1869,1872],{},[464,1870,1871],{},"0",[464,1873,1874,1875,642,1878,1881],{},"every test passed or failed as allowed by ",[124,1876,1877],{},"[!mayfail]",[124,1879,1880],{},"[!shouldfail]","; flaky tests and failures in quarantine do not fail the run",[447,1883,1884,1887],{},[464,1885,1886],{},"1",[464,1888,1889],{},"the command line was wrong, the tests could not be registered, or workers could not start",[447,1891,1892,1895],{},[464,1893,1894],{},"2",[464,1896,1897,1898,1901],{},"no test ran (",[124,1899,1900],{},"--allow-no-tests"," accepts this)",[447,1903,1904,1907],{},[464,1905,1906],{},"3",[464,1908,1909,1910,1913],{},"a test spec matched nothing and ",[124,1911,1912],{},"--warn=unmatched-specs"," was given",[447,1915,1916,1919],{},[464,1917,1918],{},"4",[464,1920,1921],{},"every test that ran was skipped",[447,1923,1924,1927],{},[464,1925,1926],{},"5",[464,1928,1929],{},"a test spec is invalid",[447,1931,1932,1935],{},[464,1933,1934],{},"42",[464,1936,1937],{},"a test failed, crashed, timed out or was blocked",[148,1939,117],{"id":1940},"coming-from-catch2s-runner",[14,1942,1943],{},"The test-writing API is unchanged. The runner is new:",[162,1945,1946,1987,1994,2019,2024],{},[165,1947,1948,1949,1951,1952,1955,1956,127,1959,127,1962,127,1965,127,1968,689,1971,127,1974,127,1977,207,1980,1983,1984,136],{},"Options are Catch3's own; ",[124,1950,1796],{}," lists them. Long options take their\nvalue after ",[124,1953,1954],{},"="," or as the next argument. ",[124,1957,1958],{},"-s",[124,1960,1961],{},"-b",[124,1963,1964],{},"-e",[124,1966,1967],{},"-a",[124,1969,1970],{},"-c",[124,1972,1973],{},"-p",[124,1975,1976],{},"-f",[124,1978,1979],{},"-#",[124,1981,1982],{},"-w"," keep their meaning, under the names in the\n",[24,1985,1986],{"href":134},"reference",[165,1988,1989,1990,1993],{},"Reporters and ",[124,1991,1992],{},"CATCH_REGISTER_REPORTER"," are gone. The console, JUnit and\nJSON reports replace the built-in reporters; for anything else, read the\nJSON report, or write a listener, which still runs in every worker.",[165,1995,1996,1999,2000,2003,2004,2007,2008,207,2011,2014,2015,136],{},[124,1997,1998],{},"Catch::Session"," remains for a ",[124,2001,2002],{},"main"," of your own: ",[124,2005,2006],{},"run(argc, argv)"," hands\nthe command line to the runner. ",[124,2009,2010],{},"Session::cli()",[124,2012,2013],{},"configData()"," are\ngone with Clara. See ",[24,2016,2018],{"href":2017},"\u002Fdocs\u002Fown-main\u002F","supplying main yourself",[165,2020,2021,2023],{},[124,2022,2002],{}," runs in every worker too, since workers are the same binary.",[165,2025,2026,2027,642,2029,2031],{},"Tests run in parallel by default. Tests that share files or other state\noutside the process need ",[124,2028,870],{},[124,2030,889],{},"; per-test\ntemporary directories cover the common case.",[2033,2034,2035],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}",{"title":144,"searchDepth":436,"depth":436,"links":2037},[2038,2039,2040,2041,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056],{"id":150,"depth":436,"text":27},{"id":238,"depth":436,"text":33},{"id":293,"depth":436,"text":39},{"id":412,"depth":436,"text":45,"children":2042},[2043],{"id":606,"depth":633,"text":607},{"id":706,"depth":436,"text":51},{"id":842,"depth":436,"text":57},{"id":993,"depth":436,"text":63},{"id":1018,"depth":436,"text":69},{"id":1052,"depth":436,"text":1053},{"id":1095,"depth":436,"text":75},{"id":1120,"depth":436,"text":81},{"id":1184,"depth":436,"text":87},{"id":1356,"depth":436,"text":93},{"id":1477,"depth":436,"text":99},{"id":1803,"depth":436,"text":105},{"id":1852,"depth":436,"text":111},{"id":1940,"depth":436,"text":117},"ContentsHow a run worksRetries, flaky tests and order dependenceHistory, quarantine and analyticsImporting the results of other frameworksChoosing and ordering testsTags the runner readsTime limitsTemporary filesWhat a failure report saysReports and tracesPer-test coverageFormats other tools readProject configurationBazel and CTestExit codesComing from Catch2's runner","md",{},true,"\u002Fdocs\u002Fprogmasoft-runner",{"title":5,"description":2057},"progmasoft-runner","docs\u002Fprogmasoft-runner.md","docs\u002Fprogmasoft-runner","CoJsmryCTs-7zp6TsPEGx31N2FRN7-Rr72BpJx-Ophc",1794961592469]