Progmasoft Catch3 additions

Progmasoft Catch3 keeps Catch2's API for writing tests, its macros and the Catch:: namespace, and replaces Catch2's runner with its own. The Catch3 runner runs tests in parallel worker processes and adds retries, flaky-test detection, quarantine, history, predictive ordering, test selection, sharding, time limits and failure analysis. Every header is in the Progmasoft/Catch3 include tree; the files of the engine that come from Catch2 keep its BSL-1.0 license, and Progmasoft's own API is in the Progmasoft::Catch3 namespace.

Property testing

Catch3 has two property APIs. Native property testing with ForAll and strategies is the full engine: every combination of strategies shrinks, failures replay from a token and from an example database, and it adds state machines, statistics, targeted search and differential and metamorphic oracles. New tests should use it. The value-shrinking CheckProperty API described here remains supported for existing tests. The roadmap lists what comes next.

<Progmasoft/Catch3.hpp> is the umbrella header. It brings in the engine's Progmasoft/Catch3/All.hpp as well as the Progmasoft extensions, so it is convenient but increases compile time. Include individual headers when a translation unit only needs a narrow part of the API.

#include <Progmasoft/Catch3.hpp>

TEST_CASE("integer values stay in their requested domain", "[property]") {
    CATCH3_CHECK_PROPERTY(
        Progmasoft::Catch3::Integer<int>(1, 100),
        [](int value) { return value > 0; },
        Progmasoft::Catch3::PropertyOptions{
            .Trials = 250,
            .Seed = 0xC0FFEE,
            .MaxShrinkSteps = 200});
}

The current property API is split into these headers:

  • <Progmasoft/Catch3/Random.hpp>: portable SplitMix64 sequence, unbiased bounded sampling, inclusive integer ranges, and per-trial seed derivation.
  • <Progmasoft/Catch3/Generator.hpp>: user-defined generation/shrinking pair.
  • <Progmasoft/Catch3/Generators/Integer.hpp> and Boolean.hpp: built-in scalar strategies with deterministic shrink candidates.
  • <Progmasoft/Catch3/Generators/Vector.hpp>, Tuple.hpp, Element.hpp and Text.hpp: composite strategies described under Composite generators.
  • <Progmasoft/Catch3/Property.hpp>: trial execution and structured results.
  • <Progmasoft/Catch3/Assertions.hpp>: CATCH3_CHECK_PROPERTY, which reports the failing trial, replay seed, reason, and minimized counterexample through Catch2's normal assertion/reporting path.

PropertyOptions::Seed is a stable root seed. Each trial receives a seed derived from the root and its zero-based trial index, so the reported replay seed can be used to reproduce that trial independently. The default generator is also stable across standard-library implementations; it does not rely on the implementation- defined output sequence of std::uniform_int_distribution.

Shrinking is deterministic and greedy: candidates are tested in the order provided by the generator, and the first still-failing candidate is selected. MaxShrinkSteps bounds predicate evaluations during minimization. The result is therefore a useful locally minimized counterexample, not a claim of a globally minimal value. A false predicate or an exception from the predicate fails the property. Exceptions from generation or shrinking are reported as execution errors, distinct from an ordinary failing property.

A failing report ends with the line replay: set PropertyOptions::ReplayTrialSeed to <seed>. Setting that option runs exactly one trial with the given seed, including its shrinking, and ignores Seed and Trials; the reported trial number is then 1. This reproduces a failure from a CI log without rerunning the trials before it.

Progmasoft::Catch3::PropertyOptions{ .ReplayTrialSeed = 1234567890123456789ULL }

Composite generators

A property still receives one generated value. Several arguments are passed as one tuple:

using namespace Progmasoft::Catch3;

CATCH3_CHECK_PROPERTY(
    Tuple(Vector(Integer<int>(0, 100), 0, 20), Text("abc", 1, 8)),
    [](const std::tuple<std::vector<int>, std::string> &input) {
        return Accepts(std::get<0>(input), std::get<1>(input));
    },
    PropertyOptions{ .Trials = 500 });
  • Tuple(generators...) draws its components strictly left to right from the trial's random source and shrinks one component at a time: every candidate of the first component, then every candidate of the second, and so on.
  • Vector(element, minimumSize, maximumSize) draws a length in the inclusive range and then the elements in index order. Text(alphabet, minimumLength, maximumLength) does the same for byte strings over the given alphabet; PrintableAscii() is the default alphabet. Both shrink in a fixed order: the shortest allowed prefix, the first and second half, the value with one element removed, and finally the value with one element simplified. A shrunk value is never shorter than its minimum, each step offers at most 256 candidates, and a size limit above 1,048,576 elements is rejected.
  • Element(values) picks uniformly from a fixed non-empty list. The list order is the simplicity order: a value shrinks to the values listed before it. Text treats its alphabet the same way, so put the simplest value or character first.

Counterexamples of tuples, pairs and vectors are printed element by element, for example { 7, { true, false }, "text" }. This does not depend on Catch2's optional tuple or range string makers, so the report is the same whatever configuration macros a client defines. Scalars still use Catch::StringMaker, including user specializations.

Invalid factory arguments, such as an empty alphabet or a minimum above its maximum, throw std::invalid_argument; a client built without exceptions gets the message on standard error followed by std::abort.

Mapped, filtered and dependent generators and state machines are not part of this module; the strategies of native property testing provide them. Shrinking a composite value here is greedy and local like scalar shrinking: it finds a small counterexample, not a provably smallest one.

Snapshots

<Progmasoft/Catch3/Snapshot.hpp> provides file-backed text snapshots through CompareSnapshot(name, actual, options). The directory is supplied explicitly; names are restricted to portable ASCII filename characters and cannot contain path separators. CRLF and CR are normalized to LF by default, with no trimming or other content rewriting.

Ordinary comparisons never write files. SnapshotUpdateMode::kCreateMissing creates only absent snapshots, while kAlways explicitly permits replacement. The result reports a match, create/update, mismatch with a one-based first difference location, invalid input, or I/O failure. CATCH3_CHECK_SNAPSHOT combines this result with Catch2's normal assertion reporting. Keep snapshots under a directory owned by the test suite and enable replacement only for an intentional update run.

const Progmasoft::Catch3::SnapshotOptions options{
    .Directory = "tests/snapshots",
};
CATCH3_CHECK_SNAPSHOT("rendered-report", RenderReport(), options);

Structured results and XML

<Progmasoft/Catch3/Results.hpp> defines the model the runner's reports are written from: a RunResult holds a TestResult per test, with its verdict, failure category, history classification, flakiness, quarantine, failure signature, analysis and replay command, and every Attempt with its assertions, sections, benchmarks, captured output and the tests that ran before it in the same process. Summarize counts the verdicts, and DescribeFailure writes a failed attempt as text. <Progmasoft/Catch3/XmlWriter.hpp> writes a run as deterministic JUnit XML through JUnitXmlWriter::Write or ToString, with <rerunFailure> and <flakyFailure> for retries. It escapes XML markup, replaces malformed UTF-8 and XML-forbidden control characters, and omits timestamps so identical results produce identical output.

The results, the JUnit writer, the runner, property testing and the snapshot API are compiled into the Catch3::Catch3 library. Generator templates and convenience macros remain in headers where their types or call-site expansion require it.

Build and integration

The CMake package is Catch3, with the targets Catch3::Catch3 and Catch3::Catch3WithMain. The library is C++17, and its headers work in C++17 and later. After installation, consumers can use:

find_package(Catch3 0.3 CONFIG REQUIRED)
target_link_libraries(MyTests PRIVATE Catch3::Catch3WithMain)

Without a build system, an installed Catch3 links by its name: the library is Catch3 and the one that brings main is Catch3Main, as libCatch3.a and libCatch3Main.a (Catch3.lib and Catch3Main.lib on Windows). The released archives and packages hold these release builds; a debug build made with CMake is named apart, Catch3d and Catch3Maind. Catch3Main comes first, since it uses Catch3:

c++ -std=c++17 tests.cpp -lCatch3Main -lCatch3

When Catch3 is installed outside the compiler's search paths, pkg-config gives the same flags with the include and library directories:

c++ -std=c++17 tests.cpp $(pkg-config --cflags --libs catch3-with-main)

The pkg-config modules are catch3 and catch3-with-main. CMake, Meson, Xmake and Premake build the files under the same names.

In the Bazel Central Registry, from 0.5.0 on, the module is progmasoft_catch3, for Bazel 8 and later:

bazel_dep(name = "progmasoft_catch3", version = "0.5.0")

Its targets are @progmasoft_catch3//:catch3 and @progmasoft_catch3//:catch3_main, which brings main. Code that includes the headers is C++17 or later, so a module that is not already set to it passes --cxxopt=-std=c++17 (--cxxopt=/std:c++17 with MSVC, which defaults to C++14).

In this repository the Bazel target is //:catch3; its tests are //tests/Progmasoft:property_tests, and tests/Progmasoft/BazelModule is a module that depends on Catch3 as a user's does. Bazel is the primary build path; CMake, Meson, Xmake, and Premake are also maintained as alternate ways to build the same engine and property tests. Their root files are CMakeLists.txt, meson.build, xmake.lua, and premake5.lua. Xmake and Premake materialize the default generated configuration header, Progmasoft/Catch3/UserConfig.hpp, from the checked-in template. Meson 1.1.0 or newer is required for the Clang-CL C++17 test target.

For project history, compatibility intent, license split, and current feature limits, read Why Progmasoft Catch3?.

Build quick start

Bazel is the primary build path:

bazel test //tests/Progmasoft:property_tests

The alternate project builds exercise the same property suite:

# CMake
cmake -S . -B build/cmake -DCATCH3_DEVELOPMENT_BUILD=ON -DCATCH3_BUILD_TESTING=ON
cmake --build build/cmake --target ProgmasoftSelfTest
ctest --test-dir build/cmake -R Progmasoft --output-on-failure

# Xmake
xmake build ProgmasoftPropertyTests
xmake run ProgmasoftPropertyTests

# Meson (Windows PowerShell; requires the Visual C++ SDK/CRT)
$env:CC = 'clang-cl'; $env:CXX = 'clang-cl'
meson setup build/meson --vsenv -Dtests=true -Dinstall=false
meson compile -C build/meson
meson test -C build/meson --print-errorlogs

# Premake (choose an action supported by the host)
premake5 vs2022

Premake generates projects rather than compiling directly; build the generated solution with the corresponding host toolchain. On macOS, use premake5 gmake, then build the generated makefiles with make. Meson users on macOS can select Apple Clang with CC=clang CXX=clang++ before configuration.

Compatibility and licensing boundary

Do not rename or rewrite Catch2 headers, Catch symbols, macros, runner options, or upstream coding style as part of this extension. New functionality belongs in separately licensed Progmasoft headers and uses Progmasoft's PascalCase C++ types/functions and project naming conventions. Catch2-originated source files remain under BSL-1.0; Progmasoft-authored source files are under LGPL-3.0-or-later, as their headers say. The two areas are not relicensed as a whole. Any proposed upstream PR must be prepared under Catch2's contribution rules as a separate contribution, rather than reusing a source file whose license header says otherwise.

This page on GitHubLicenses of the documentation