[{"data":1,"prerenderedAt":674},["ShallowReactive",2],{"doc-benchmarks":3},{"id":4,"title":5,"body":6,"description":150,"extension":666,"meta":667,"navigation":215,"path":668,"seo":669,"slug":670,"source":671,"stem":672,"__hash__":673},"docs\u002Fdocs\u002Fbenchmarks.md","Authoring benchmarks",{"type":7,"value":8,"toc":658},"minimark",[9,50,54,65,68,71,98,103,106,127,130,134,141,144,173,179,312,315,330,333,341,346,357,391,398,407,417,426,436,456,459,472,481,485,488,491,548,562,568,585,589,596,605,608,642,645,648,654],[10,11,12],"blockquote",{},[13,14,15,19,20,24,25,28,29,32,33,36,37,36,40,43,44,49],"p",{},[16,17,18],"strong",{},"Catch3 compatibility reference:"," This page describes the test-writing API Catch3 shares with Catch2: the macros, such as ",[21,22,23],"code",{},"TEST_CASE"," and ",[21,26,27],{},"REQUIRE",", and the ",[21,30,31],{},"Catch::"," namespace keep Catch2's names. Headers and build names are Catch3's own (",[21,34,35],{},"\u003CProgmasoft\u002FCatch3\u002FTestMacros.hpp>",", ",[21,38,39],{},"find_package(Catch3)",[21,41,42],{},"Catch3::Catch3WithMain","). See ",[45,46,48],"a",{"href":47},"\u002Fdocs\u002Fprogmasoft-catch3\u002F","Catch3 additions"," for Progmasoft APIs.",[51,52,5],"h1",{"id":53},"authoring-benchmarks",[10,55,56],{},[13,57,58,64],{},[45,59,63],{"href":60,"rel":61},"https:\u002F\u002Fgithub.com\u002Fcatchorg\u002FCatch2\u002Fissues\u002F1616",[62],"nofollow","Introduced"," in Catch2 2.9.0.",[13,66,67],{},"Writing benchmarks is not easy. Catch simplifies certain aspects but you'll\nalways need to take care about various aspects. Understanding a few things about\nthe way Catch runs your code will be very helpful when writing your benchmarks.",[13,69,70],{},"First off, let's go over some terminology that will be used throughout this\nguide.",[72,73,74,82,92],"ul",{},[75,76,77,81],"li",{},[78,79,80],"em",{},"User code",": user code is the code that the user provides to be measured.",[75,83,84,87,88,91],{},[78,85,86],{},"Run",": one run is one execution of the user code. Sometimes also referred\nto as an ",[78,89,90],{},"iteration",".",[75,93,94,97],{},[78,95,96],{},"Sample",": one sample is one data point obtained by measuring the time it takes\nto perform a certain number of runs. One sample can consist of more than one\nrun if the clock available does not have enough resolution to accurately\nmeasure a single run. All samples for a given benchmark execution are obtained\nwith the same number of runs.",[99,100,102],"h2",{"id":101},"execution-procedure","Execution procedure",[13,104,105],{},"Now I can explain how a benchmark is executed in Catch. There are three main\nsteps, though the first does not need to be repeated for every benchmark.",[107,108,109,115,121],"ol",{},[75,110,111,114],{},[78,112,113],{},"Environmental probe",": before any benchmarks can be executed, the clock's\nresolution is estimated. A few other environmental artifacts are also estimated\nat this point, like the cost of calling the clock function, but they almost\nnever have any impact in the results.",[75,116,117,120],{},[78,118,119],{},"Estimation",": the user code is executed a few times to obtain an estimate of\nthe amount of runs that should be in each sample. This also has the potential\neffect of bringing relevant code and data into the caches before the actual\nmeasurement starts.",[75,122,123,126],{},[78,124,125],{},"Measurement",": all the samples are collected sequentially by performing the\nnumber of runs estimated in the previous step for each sample.",[13,128,129],{},"This already gives us one important rule for writing benchmarks for Catch: the\nbenchmarks must be repeatable. The user code will be executed several times, and\nthe number of times it will be executed during the estimation step cannot be\nknown beforehand since it depends on the time it takes to execute the code.\nUser code that cannot be executed repeatedly will lead to bogus results or\ncrashes.",[99,131,133],{"id":132},"benchmark-specification","Benchmark specification",[13,135,136,137,140],{},"Benchmarks can be specified anywhere inside a Catch test case.\nThere is a simple and a slightly more advanced version of the ",[21,138,139],{},"BENCHMARK"," macro.",[13,142,143],{},"Let's have a look how a naive Fibonacci implementation could be benchmarked:",[145,146,151],"pre",{"className":147,"code":148,"language":149,"meta":150,"style":150},"language-c++ shiki shiki-themes github-dark","std::uint64_t Fibonacci(std::uint64_t number) {\n    return number \u003C 2 ? 1 : Fibonacci(number - 1) + Fibonacci(number - 2);\n}\n","c++","",[21,152,153,161,167],{"__ignoreMap":150},[154,155,158],"span",{"class":156,"line":157},"line",1,[154,159,160],{},"std::uint64_t Fibonacci(std::uint64_t number) {\n",[154,162,164],{"class":156,"line":163},2,[154,165,166],{},"    return number \u003C 2 ? 1 : Fibonacci(number - 1) + Fibonacci(number - 2);\n",[154,168,170],{"class":156,"line":169},3,[154,171,172],{},"}\n",[13,174,175,176,178],{},"Now the most straight forward way to benchmark this function, is just adding a ",[21,177,139],{}," macro to our test case:",[145,180,182],{"className":147,"code":181,"language":149,"meta":150,"style":150},"TEST_CASE(\"Fibonacci\") {\n    CHECK(Fibonacci(0) == 1);\n    \u002F\u002F some more asserts..\n    CHECK(Fibonacci(5) == 8);\n    \u002F\u002F some more asserts..\n\n    \u002F\u002F now let's benchmark:\n    BENCHMARK(\"Fibonacci 20\") {\n        return Fibonacci(20);\n    };\n\n    BENCHMARK(\"Fibonacci 25\") {\n        return Fibonacci(25);\n    };\n\n    BENCHMARK(\"Fibonacci 30\") {\n        return Fibonacci(30);\n    };\n\n    BENCHMARK(\"Fibonacci 35\") {\n        return Fibonacci(35);\n    };\n}\n",[21,183,184,189,194,199,205,210,217,223,229,235,241,246,252,258,263,268,274,280,285,290,296,302,307],{"__ignoreMap":150},[154,185,186],{"class":156,"line":157},[154,187,188],{},"TEST_CASE(\"Fibonacci\") {\n",[154,190,191],{"class":156,"line":163},[154,192,193],{},"    CHECK(Fibonacci(0) == 1);\n",[154,195,196],{"class":156,"line":169},[154,197,198],{},"    \u002F\u002F some more asserts..\n",[154,200,202],{"class":156,"line":201},4,[154,203,204],{},"    CHECK(Fibonacci(5) == 8);\n",[154,206,208],{"class":156,"line":207},5,[154,209,198],{},[154,211,213],{"class":156,"line":212},6,[154,214,216],{"emptyLinePlaceholder":215},true,"\n",[154,218,220],{"class":156,"line":219},7,[154,221,222],{},"    \u002F\u002F now let's benchmark:\n",[154,224,226],{"class":156,"line":225},8,[154,227,228],{},"    BENCHMARK(\"Fibonacci 20\") {\n",[154,230,232],{"class":156,"line":231},9,[154,233,234],{},"        return Fibonacci(20);\n",[154,236,238],{"class":156,"line":237},10,[154,239,240],{},"    };\n",[154,242,244],{"class":156,"line":243},11,[154,245,216],{"emptyLinePlaceholder":215},[154,247,249],{"class":156,"line":248},12,[154,250,251],{},"    BENCHMARK(\"Fibonacci 25\") {\n",[154,253,255],{"class":156,"line":254},13,[154,256,257],{},"        return Fibonacci(25);\n",[154,259,261],{"class":156,"line":260},14,[154,262,240],{},[154,264,266],{"class":156,"line":265},15,[154,267,216],{"emptyLinePlaceholder":215},[154,269,271],{"class":156,"line":270},16,[154,272,273],{},"    BENCHMARK(\"Fibonacci 30\") {\n",[154,275,277],{"class":156,"line":276},17,[154,278,279],{},"        return Fibonacci(30);\n",[154,281,283],{"class":156,"line":282},18,[154,284,240],{},[154,286,288],{"class":156,"line":287},19,[154,289,216],{"emptyLinePlaceholder":215},[154,291,293],{"class":156,"line":292},20,[154,294,295],{},"    BENCHMARK(\"Fibonacci 35\") {\n",[154,297,299],{"class":156,"line":298},21,[154,300,301],{},"        return Fibonacci(35);\n",[154,303,305],{"class":156,"line":304},22,[154,306,240],{},[154,308,310],{"class":156,"line":309},23,[154,311,172],{},[13,313,314],{},"There's a few things to note:",[72,316,317,323],{},[75,318,319,320,322],{},"As ",[21,321,139],{}," expands to a lambda expression it is necessary to add a semicolon after\nthe closing brace (as opposed to the first experimental version).",[75,324,325,326,329],{},"The ",[21,327,328],{},"return"," is a handy way to avoid the compiler optimizing away the benchmark code.",[13,331,332],{},"Running this already runs the benchmarks and outputs something similar to:",[145,334,339],{"className":335,"code":337,"language":338},[336],"language-text","-------------------------------------------------------------------------------\nFibonacci\n-------------------------------------------------------------------------------\nC:\\path\\to\\Catch2\\Benchmark.tests.cpp(10)\n...............................................................................\nbenchmark name                                  samples       iterations    est run time\n                                                mean          low mean      high mean\n                                                std dev       low std dev   high std dev\n-------------------------------------------------------------------------------\nFibonacci 20                                            100       416439   83.2878 ms\n                                                       2 ns         2 ns         2 ns\n                                                       0 ns         0 ns         0 ns\n\nFibonacci 25                                            100       400776   80.1552 ms\n                                                       3 ns         3 ns         3 ns\n                                                       0 ns         0 ns         0 ns\n\nFibonacci 30                                            100       396873   79.3746 ms\n                                                      17 ns        17 ns        17 ns\n                                                       0 ns         0 ns         0 ns\n\nFibonacci 35                                            100       145169   87.1014 ms\n                                                     468 ns       464 ns       473 ns\n                                                      21 ns        15 ns        34 ns\n","text",[21,340,337],{"__ignoreMap":150},[342,343,345],"h3",{"id":344},"advanced-benchmarking","Advanced benchmarking",[13,347,348,349,352,353,356],{},"The simplest use case shown above, takes no arguments and just runs the user code that needs to be measured.\nHowever, if using the ",[21,350,351],{},"BENCHMARK_ADVANCED"," macro and adding a ",[21,354,355],{},"Catch::Benchmark::Chronometer"," argument after\nthe macro, some advanced features are available. The contents of the simple benchmarks are invoked once per run,\nwhile the blocks of the advanced benchmarks are invoked exactly twice:\nonce during the estimation phase, and another time during the execution phase.",[145,358,360],{"className":147,"code":359,"language":149,"meta":150,"style":150},"BENCHMARK(\"simple\"){ return long_computation(); };\n\nBENCHMARK_ADVANCED(\"advanced\")(Catch::Benchmark::Chronometer meter) {\n    set_up();\n    meter.measure([] { return long_computation(); });\n};\n",[21,361,362,367,371,376,381,386],{"__ignoreMap":150},[154,363,364],{"class":156,"line":157},[154,365,366],{},"BENCHMARK(\"simple\"){ return long_computation(); };\n",[154,368,369],{"class":156,"line":163},[154,370,216],{"emptyLinePlaceholder":215},[154,372,373],{"class":156,"line":169},[154,374,375],{},"BENCHMARK_ADVANCED(\"advanced\")(Catch::Benchmark::Chronometer meter) {\n",[154,377,378],{"class":156,"line":201},[154,379,380],{},"    set_up();\n",[154,382,383],{"class":156,"line":207},[154,384,385],{},"    meter.measure([] { return long_computation(); });\n",[154,387,388],{"class":156,"line":212},[154,389,390],{},"};\n",[13,392,393,394,397],{},"These advanced benchmarks no longer consist entirely of user code to be measured.\nIn these cases, the code to be measured is provided via the\n",[21,395,396],{},"Catch::Benchmark::Chronometer::measure"," member function. This allows you to set up any\nkind of state that might be required for the benchmark but is not to be included\nin the measurements, like making a vector of random integers to feed to a\nsorting algorithm.",[13,399,400,401,403,404,91],{},"A single call to ",[21,402,396],{}," performs the actual measurements\nby invoking the callable object passed in as many times as necessary. Anything\nthat needs to be done outside the measurement can be done outside the call to\n",[21,405,406],{},"measure",[13,408,409,410,412,413,416],{},"The callable object passed in to ",[21,411,406],{}," can optionally accept an ",[21,414,415],{},"int","\nparameter.",[145,418,420],{"className":147,"code":419,"language":149,"meta":150,"style":150},"meter.measure([](int i) { return long_computation(i); });\n",[21,421,422],{"__ignoreMap":150},[154,423,424],{"class":156,"line":157},[154,425,419],{},[13,427,428,429,431,432,435],{},"If it accepts an ",[21,430,415],{}," parameter, the sequence number of each run will be passed\nin, starting with 0. This is useful if you want to measure some mutating code,\nfor example. The number of runs can be known beforehand by calling\n",[21,433,434],{},"Catch::Benchmark::Chronometer::runs","; with this one can set up a different instance to be\nmutated by each run.",[145,437,439],{"className":147,"code":438,"language":149,"meta":150,"style":150},"std::vector\u003Cstd::string> v(meter.runs());\nstd::fill(v.begin(), v.end(), test_string());\nmeter.measure([&v](int i) { in_place_escape(v[i]); });\n",[21,440,441,446,451],{"__ignoreMap":150},[154,442,443],{"class":156,"line":157},[154,444,445],{},"std::vector\u003Cstd::string> v(meter.runs());\n",[154,447,448],{"class":156,"line":163},[154,449,450],{},"std::fill(v.begin(), v.end(), test_string());\n",[154,452,453],{"class":156,"line":169},[154,454,455],{},"meter.measure([&v](int i) { in_place_escape(v[i]); });\n",[13,457,458],{},"Note that it is not possible to simply use the same instance for different runs\nand resetting it between each run since that would pollute the measurements with\nthe resetting code.",[13,460,461,462,464,465,468,469,471],{},"It is also possible to just provide an argument name to the simple ",[21,463,139],{}," macro to get\nthe same semantics as providing a callable to ",[21,466,467],{},"meter.measure"," with ",[21,470,415],{}," argument:",[145,473,475],{"className":147,"code":474,"language":149,"meta":150,"style":150},"BENCHMARK(\"indexed\", i){ return long_computation(i); };\n",[21,476,477],{"__ignoreMap":150},[154,478,479],{"class":156,"line":157},[154,480,474],{},[342,482,484],{"id":483},"constructors-and-destructors","Constructors and destructors",[13,486,487],{},"All of these tools give you a lot mileage, but there are two things that still\nneed special handling: constructors and destructors. The problem is that if you\nuse automatic objects they get destroyed by the end of the scope, so you end up\nmeasuring the time for construction and destruction together. And if you use\ndynamic allocation instead, you end up including the time to allocate memory in\nthe measurements.",[13,489,490],{},"To solve this conundrum, Catch provides class templates that let you manually\nconstruct and destroy objects without dynamic allocation and in a way that lets\nyou measure construction and destruction separately.",[145,492,494],{"className":147,"code":493,"language":149,"meta":150,"style":150},"BENCHMARK_ADVANCED(\"construct\")(Catch::Benchmark::Chronometer meter) {\n    std::vector\u003CCatch::Benchmark::storage_for\u003Cstd::string>> storage(meter.runs());\n    meter.measure([&](int i) { storage[i].construct(\"thing\"); });\n};\n\nBENCHMARK_ADVANCED(\"destroy\")(Catch::Benchmark::Chronometer meter) {\n    std::vector\u003CCatch::Benchmark::destructable_object\u003Cstd::string>> storage(meter.runs());\n    for(auto&& o : storage)\n        o.construct(\"thing\");\n    meter.measure([&](int i) { storage[i].destruct(); });\n};\n",[21,495,496,501,506,511,515,519,524,529,534,539,544],{"__ignoreMap":150},[154,497,498],{"class":156,"line":157},[154,499,500],{},"BENCHMARK_ADVANCED(\"construct\")(Catch::Benchmark::Chronometer meter) {\n",[154,502,503],{"class":156,"line":163},[154,504,505],{},"    std::vector\u003CCatch::Benchmark::storage_for\u003Cstd::string>> storage(meter.runs());\n",[154,507,508],{"class":156,"line":169},[154,509,510],{},"    meter.measure([&](int i) { storage[i].construct(\"thing\"); });\n",[154,512,513],{"class":156,"line":201},[154,514,390],{},[154,516,517],{"class":156,"line":207},[154,518,216],{"emptyLinePlaceholder":215},[154,520,521],{"class":156,"line":212},[154,522,523],{},"BENCHMARK_ADVANCED(\"destroy\")(Catch::Benchmark::Chronometer meter) {\n",[154,525,526],{"class":156,"line":219},[154,527,528],{},"    std::vector\u003CCatch::Benchmark::destructable_object\u003Cstd::string>> storage(meter.runs());\n",[154,530,531],{"class":156,"line":225},[154,532,533],{},"    for(auto&& o : storage)\n",[154,535,536],{"class":156,"line":231},[154,537,538],{},"        o.construct(\"thing\");\n",[154,540,541],{"class":156,"line":237},[154,542,543],{},"    meter.measure([&](int i) { storage[i].destruct(); });\n",[154,545,546],{"class":156,"line":243},[154,547,390],{},[13,549,550,553,554,557,558,561],{},[21,551,552],{},"Catch::Benchmark::storage_for\u003CT>"," objects are just pieces of raw storage suitable for ",[21,555,556],{},"T","\nobjects. You can use the ",[21,559,560],{},"Catch::Benchmark::storage_for::construct"," member function to call a constructor and\ncreate an object in that storage. So if you want to measure the time it takes\nfor a certain constructor to run, you can just measure the time it takes to run\nthis function.",[13,563,564,565,567],{},"When the lifetime of a ",[21,566,552],{}," object ends, if an actual object was\nconstructed there it will be automatically destroyed, so nothing leaks.",[13,569,570,571,574,575,577,578,580,581,584],{},"If you want to measure a destructor, though, we need to use\n",[21,572,573],{},"Catch::Benchmark::destructable_object\u003CT>",". These objects are similar to\n",[21,576,552],{}," in that construction of the ",[21,579,556],{}," object is manual, but\nit does not destroy anything automatically. Instead, you are required to call\nthe ",[21,582,583],{},"Catch::Benchmark::destructable_object::destruct"," member function, which is what you\ncan use to measure the destruction time.",[342,586,588],{"id":587},"the-optimizer","The optimizer",[13,590,591,592,595],{},"Sometimes the optimizer will optimize away the very code that you want to\nmeasure. There are several ways to use results that will prevent the optimiser\nfrom removing them. You can use the ",[21,593,594],{},"volatile"," keyword, or you can output the\nvalue to standard output or to a file, both of which force the program to\nactually generate the value somehow.",[13,597,598,599,601,602,604],{},"Catch adds a third option. The values returned by any function provided as user\ncode are guaranteed to be evaluated and not optimised out. This means that if\nyour user code consists of computing a certain value, you don't need to bother\nwith using ",[21,600,594],{}," or forcing output. Just ",[21,603,328],{}," it from the function.\nThat helps with keeping the code in a natural fashion.",[13,606,607],{},"Here's an example:",[145,609,611],{"className":147,"code":610,"language":149,"meta":150,"style":150},"\u002F\u002F may measure nothing at all by skipping the long calculation since its\n\u002F\u002F result is not used\nBENCHMARK(\"no return\"){ long_calculation(); };\n\n\u002F\u002F the result of long_calculation() is guaranteed to be computed somehow\nBENCHMARK(\"with return\"){ return long_calculation(); };\n",[21,612,613,618,623,628,632,637],{"__ignoreMap":150},[154,614,615],{"class":156,"line":157},[154,616,617],{},"\u002F\u002F may measure nothing at all by skipping the long calculation since its\n",[154,619,620],{"class":156,"line":163},[154,621,622],{},"\u002F\u002F result is not used\n",[154,624,625],{"class":156,"line":169},[154,626,627],{},"BENCHMARK(\"no return\"){ long_calculation(); };\n",[154,629,630],{"class":156,"line":201},[154,631,216],{"emptyLinePlaceholder":215},[154,633,634],{"class":156,"line":207},[154,635,636],{},"\u002F\u002F the result of long_calculation() is guaranteed to be computed somehow\n",[154,638,639],{"class":156,"line":212},[154,640,641],{},"BENCHMARK(\"with return\"){ return long_calculation(); };\n",[13,643,644],{},"However, there's no other form of control over the optimizer whatsoever. It is\nup to you to write a benchmark that actually measures what you want and doesn't\njust measure the time to do a whole bunch of nothing.",[13,646,647],{},"To sum up, there are two simple rules: whatever you would do in handwritten code\nto control optimization still works in Catch; and Catch makes return values\nfrom user code into observable effects that can't be optimized away.",[13,649,650],{},[651,652,653],"i",{},"Adapted from nonius' documentation.",[655,656,657],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":150,"searchDepth":163,"depth":163,"links":659},[660,661],{"id":101,"depth":163,"text":102},{"id":132,"depth":163,"text":133,"children":662},[663,664,665],{"id":344,"depth":169,"text":345},{"id":483,"depth":169,"text":484},{"id":587,"depth":169,"text":588},"md",{},"\u002Fdocs\u002Fbenchmarks",{"title":5,"description":150},"benchmarks","docs\u002Fbenchmarks.md","docs\u002Fbenchmarks","NA2O4VDh1rG9S6ktxY-t9fZX2_ANmzsmwXsiWlZTVrU",1794961594895]