Learning Resources

Unit Testing Part 1

Introduction

Today, C++ developers can choose from several great Open Source unit testing frameworks such as Catch2, doctest, and Boost.Test (just to name a few). All of them are good choices with different strengths and weaknesses.

When it comes to testing CAF applications, however, we recommend using the native CAF unit testing API. Not because of particular design choices but because of its tight integration that enables fully deterministic execution of actors.

In part 1 of this guide, we will introduce the unit testing API and show how to write tests, scenarios and outlines as well as custom fixtures. In part 2, we will show how to use the predefined fixture in CAF for writing deterministic unit tests for actors.

The source code for all of the examples in this guide as well as the full setup for compiling and running the tests can be found on GitHub. We strongly recommend cloning the repository and following along with the examples.

Basic Setup and Test Binaries

Before discussing the unit testing API in detail, we first look at a minimal test that we can compile and run.

#include <caf/test/test.hpp>
#include <caf/test/caf_test_main.hpp>

TEST("a simple test") {
  check_eq(1, 1);
}

CAF_TEST_MAIN()

To compile and run the test, we will use CMake. Boilerplate code aside, the relevant part of the CMake code looks like this:

add_executable(simple-test main.test.cpp)

target_compile_features(simple-test PRIVATE cxx_std_${CXX_VERSION})

target_link_libraries(simple-test PRIVATE CAF::test)

add_test(NAME simple-test COMMAND simple-test)

This first example is located in the directory simple-test of the repository.

By convention, we use the suffix .test.cpp for all source files that belong to unit tests. Hence, the file that contains the main function for the test binary is called main.test.cpp. In this case, since it is a trivial first test, we only have this one file. The main function itself is generated by the CAF_TEST_MAIN macro. This macro works similar to the CAF_MAIN macro and allows us to pass custom type ID blocks to the macro for initializing the global meta objects table. The main function generated from this macro also takes care of parsing command line arguments and running the tests.

To tell CMake what version of C++ we want to use, our setup uses the variable CXX_VERSION. This variable is set in the main CMakeLists.txt file and defaults to 17. Hence, we will pass cxx_std_17 to the target_compile_features function by default, which enables C++17 features.

The CAF::test target is a CMake target that links against the CAF unit testing library. This target is provided by CAF and includes all necessary dependencies to write and run unit tests with CAF. Lastly, we add a test to the CMake setup with the add_test function. This will allow us to run the test with ctest.

After building the project, this test will run when we execute ctest. However, we can also run the test binary directly. Unless configured otherwise, CMake will put the binary in the build directory, e.g., build/simple-example/simple-test.

When running the test binary manually, the output will look like this:

$ simple-test
Summary:
  Time:   0.000s
  Suites: 1 / 1
  Checks: 1 / 1
  Status: passed

As we can see, the unit test framework will omit most of the output if all tests pass and we will only see a summary. If a test fails, the framework will print more detailed information about the failure. This is because output is generated lazily and only printed if the severity of the log message is included in the verbosity level of the reporter.

The test binary also supports several command line options. We can see all available options by running the test binary with the --help flag:

$ simple-test --help
global options:
  (-a|--available-suites)              : print all available suites
  (-h|-?|--help)                       : print this help text
  (-n|--no-colors)                     : disable coloring (ignored on Windows)
  (-r|--max-runtime) <uint64_t>        : set a maximum runtime in seconds
  (-s|--suites) <std::string>          : regex for selecting suites
  (-t|--tests) <std::string>           : regex for selecting tests
  (-A|--available-tests) <std::string> : print tests for a suite
  (-v|--verbosity) <std::string>       : set verbosity level of the reporter
  (-l|--log-component-filter) <list>   : set log filter

Here, we can find an option to set the verbosity level of the reporter. By using the -v flag, we can control how much output we want to see. For example, we can set the verbosity level to debug to see more log messages (CAF recognizes the verbosity levels quiet, error, warning, info, debug, and trace):

$ simple-test -v debug
Test: a simple test
  pass /Users/neverlord/github/unit-test-example/simple-example/main.test.cpp:5

Suite Summary: (default suite)
  Time:   0.000s
  Checks: 1 / 1
  Status: passed

Summary:
  Time:   0.000s
  Suites: 1 / 1
  Checks: 1 / 1
  Status: passed

This time, we see the test name, followed by the file and line number where the successful check was made. There is also a new section called Suite Summary that summarizes the results for a single suite. This is useful when we have multiple suites in a single test binary, which we will see soon.

By default, CAF will put all tests into the default suite. This implicit default suite has the name $. With the -A flag, we can list all available tests in a suite:

$ simple-test -A $
available tests in suite $:
- a simple test

CAF also implements filter options with the -s and -t flags. With these flags, we can select specific suites or tests to run. We can also combine the two flags to run a single test:

$ simple-test -s '\$' -t 'simple'
Summary:
  Time:   0.000s
  Suites: 1 / 1
  Checks: 1 / 1
  Status: passed

When using the -s option, we have to escape the $ character because it is a special character in regular expressions. Filtering suites and tests is useful to having fine-grained control over which tests to run. Either on the command line directly, or when adding tests in CMake (on a suite or test level).

The remaining options should be self-explanatory, except for the -l option. As we have mentioned earlier, the test framework is tightly integrated with CAF. The output of the tests also includes log messages from the CAF runtime. This can be quite overwhelming when running tests, especially when running tests with a high verbosity level. Hence, most log components are disabled by default via this filter. To disable the filter and see all log messages, we can use the -l option:

$ simple-test -l '[]'
Summary:
  Time:   0.000s
  Suites: 1 / 1
  Checks: 1 / 1
  Status: passed

In this first example, no log messages are generated, so the output is the same as before. Usually, setting this filter manually is not necessary except when debugging CAF itself or when needing more low-level context information about a failing test.

Checks and Requirements

The DSL for writing tests in CAF distinguishes between checks and requirements. A check is a statement that is either true or false. If the statement is false, the test fails and an error is reported. However, the test execution continues after a check failure. A requirement is a statement that must be true for the test to continue. If a requirement is false, the test fails and the execution stops immediately.

The following checks are available in CAF:

  • check(arg): Checks whether the argument is true.
  • check_eq(lhs, rhs): Checks whether the left-hand side is equal to the right-hand side.
  • check_ne(lhs, rhs): Checks whether the left-hand side is not equal to the right-hand side.
  • check_lt(lhs, rhs): Checks whether the left-hand side is less than the right-hand side.
  • check_le(lhs, rhs): Checks whether the left-hand side is less than or equal to the right-hand side.
  • check_gt(lhs, rhs): Checks whether the left-hand side is greater than the right-hand side.
  • check_ge(lhs, rhs): Checks whether the left-hand side is greater than or equal to the right-hand side.
  • check_throws(fn): Checks whether fn() throws an exception.
  • check_throws<E>(fn): Checks whether fn() throws an exception of type E.

Note that all of these are proper C++ functions, not macros. All check functions also return a boolean value, which is true if the check succeeded and false if the check failed. This allows us to use the check functions in if statements to conditionally execute code. A common pattern is to use a check function in an if statement and then place dependent checks inside the if block. For example:

if (check_eq(values.size(), 3u)) {
  check_eq(values[0], 1);
  check_eq(values[1], 2);
  check_eq(values[2], 3);
}

Here, we first check the size of some container, e.g., a std::vector, and then check the values of the elements only if the size check succeeded. We could also use a requirement instead of the check:

require_eq(values.size(), 3u);
check_eq(values[0], 1);
check_eq(values[1], 2);
check_eq(values[2], 3);

In this case, the test would abort if the size check fails and the following checks would not run. Both versions have their use cases and it depends on the context (and personal preference) which one to use.

The function names for the requiements are the same as for the checks, simply replace check with require. The requirements are also proper C++ functions.

Test Suites

There are two ways to group tests into suites in CAF. The first way is to use the explicit SUITE macro. This macro takes a suite name and a block of tests. The suite name is a string literal that can be used to filter tests on the command line.

Here is an example of how to use the SUITE macro:

SUITE("algorithms") {

TEST("std::sort sorts a sequence of elements") {
  auto xs = std::vector{3, 1, 4, 1, 5, 9, 2, 6};
  std::sort(xs.begin(), xs.end());
  check_eq(xs, std::vector{1, 1, 2, 3, 4, 5, 6, 9});
}

TEST("std::find searches for an element in a sequence") {
  auto xs = std::vector{3, 1, 4, 1, 5, 9, 2, 6};
  auto i = std::find(xs.begin(), xs.end(), 5);
  if (check_ne(i, xs.end())) {
    check_eq(*i, 5);
  }
  check_eq(std::find(xs.begin(), xs.end(), 42), xs.end());
}

} // SUITE("algorithms")

SUITE("containers") {

TEST("a default-construted vector is empty") {
  auto xs = std::vector<int>{};
  check(xs.empty());
  check_eq(xs.size(), 0u);
}

TEST("push_back increases the size of a vector") {
  auto xs = std::vector<int>{};
  xs.push_back(42);
  check_eq(xs.size(), 1u);
}

TEST("push_back adds elements to the end of a vector") {
  auto xs = std::vector<int>{};
  xs.push_back(42);
  xs.push_back(43);
  if (check_eq(xs.size(), 2u)) {
    check_eq(xs[0], 42);
    check_eq(xs[1], 43);
  }
}

} // SUITE("containers")

The upside of using the SUITE macro is that we can define multiple suites in a single file. Having the suite name explicitly in the test code also makes it easier to understand the context of the tests. The downside is that we have to maintain the suite names by hand. If the suite name is derived from the file name, it is easy to forget to update the suite name when renaming the file.

For example, in CAF, files are named after the class they contain. If we want to rename a class, we also have to rename the files. When using this explicit method, we also have to update the suite name in the test code. Since the name of the test suite by convention is derived from the file name, we can automate the generation of the suite name with CMake by providing a definition for the macro CAF_TEST_SUITE_NAME when compiling a .test.cpp file.

In the repository, we have an example of this setup in the directory multi-suite-generated. The directory contains the two files algorithms.test.cpp and containers.test.cpp that contain the tests from the previous example. We also have a trivial main.test.cpp that only includes these three lines:

#include <caf/test/caf_test_main.hpp>

CAF_TEST_MAIN()

Instead of using the SUITE macro, we set the CAF_TEST_SUITE_NAME macro in CMake as follows:

set(test_suites algorithms containers)

add_executable(multi-suites-generated main.test.cpp)

foreach(test_suite IN LISTS test_suites)
    set_property(SOURCE ${test_suite}.test.cpp PROPERTY COMPILE_DEFINITIONS
                 CAF_TEST_SUITE_NAME=${test_suite})
    target_sources(multi-suites-generated PRIVATE ${test_suite}.test.cpp)
    add_test(NAME multi-suites-generated-${test_suite}
             COMMAND multi-suites-generated -s "^${test_suite}\$")
endforeach()

target_compile_features(multi-suites-generated PRIVATE cxx_std_${CXX_VERSION})

target_link_libraries(multi-suites-generated PRIVATE CAF::test)

First, we store the suite names in a variable test_suites. Then, we iterate over the suite names and set the CAF_TEST_SUITE_NAME macro for each source file while also adding the source file to the target. We also add a test for each suite with the -s flag to filter the tests. The suite name is a regular expression that matches the suite name exactly. The ^ and $ characters are used to match the beginning and end of the string, respectively.

Which method to prefer of course depends on the project and personal preference. Having the build system generate the suite names is more robust and less error prone. However, of course it also obfuscates the suite names by basically hiding them in a CMake script.

Log Messages and Custom Types

Even when running tests without constructing an actor system, the unit test framework installs a logger that will print log messages to the console. CAF also provides a set of logging functions that can be used in tests. These functions live in the namespace caf::log::test, are simply named after the log levels and take a format string as well as corresponding arguments.

The format string uses the same syntax as std::format from C++20. CAF can be configured to use std::format if the compiler supports it. If not, CAF will fall back to a custom implementation. In any case, CAF recognizes not only the types recognized by the formatters but any type with an inspect overload.

In the following example, we define a custom type point and provide an inspect overload for it. We also give the type a proper CAF type ID so that we could use it in CAF messages if we wanted to. Just like with the regular CAF_MAIN macro, we can pass a custom type ID block to the CAF_TEST_MAIN macro to initialize the global meta objects table. After this setup, we can pass an instance of point to the caf::log::test functions:

struct point {
  int x;
  int y;
};

CAF_BEGIN_TYPE_ID_BLOCK(log_test, caf::first_custom_type_id)

  CAF_ADD_TYPE_ID(log_test, (point))

CAF_END_TYPE_ID_BLOCK(log_test)

template <class Inspector>
bool inspect(Inspector& f, point& x) {
  return f.object(x).fields(f.field("x", x.x), f.field("y", x.y));
}

TEST("a simple test") {
  auto res = 0;
  caf::log::test::debug("run the for loop in line {}", __LINE__);
  for (int i = 0; i < 3; ++i) {
    caf::log::test::debug("i = {}", i);
    res += i;
  }
  check_eq(res, 3);
  auto p = point{1, 2};
  caf::log::test::info("{}", p);
  check_eq(p.x, 1);
  check_eq(p.y, 2);
}

CAF_TEST_MAIN(caf::id_block::log_test)

Running this example will produce the following output:

Test: a simple test
  info:
    loc: /Users/neverlord/github/unit-test-example/log/main.test.cpp:31
    msg: point(1, 2)

Summary:
  Time:   0.000s
  Suites: 1 / 1
  Checks: 3 / 3
  Status: passed

To display the debug log messages as well, we need to raise the verbosity level of the reporter to debug.

Sections and Control Flow

In CAF tests, we can use sections using the SECTION macro to structure the test code. Sections form a tree-like structure and can be nested arbitrarily deep. The SECTION macro takes a string literal as an argument that will be displayed in the test output. CAF will run only one "leaf" section at a time.

Structuring tests in this way makes it easy to first initialize some state in the test and then run independent checks on that state, whereas each section receives a fresh state.

In the following example, we use the path variable to highlight the path that each test run takes through the sections:

TEST("sections showscase") {
  auto path = "test"s;
  SECTION("section 1") {
    path += " « section 1";
    SECTION("section 1.1") {
      path += ".1";
      SECTION("section 1.1.1") {
        path += ".1";
        caf::log::test::info("{}", path);
      }
      SECTION("section 1.1.2") {
        path += ".2";
        caf::log::test::info("{}", path);
      }
    }
    SECTION("section 1.2") {
      path += ".1";
      caf::log::test::info("{}", path);
    }
  }
  SECTION("section 2") {
    path += " « section 2";
    SECTION("section 2.1") {
      path += ".1";
      caf::log::test::info("{}", path);
    }
    SECTION("section 2.2") {
      path += ".2";
      caf::log::test::info("{}", path);
    }
  }
}

When running the executable, this test runs six times in total, once for each leaf section. The output will look like this:

Test: sections showcase
  Section: section 1
    Section: section 1.1
      Section: section 1.1.1
        info:
          loc: /Users/neverlord/github/unit-test-example/sections/main.test.cpp:15
          msg: test « section 1.1.1

 Test: sections showcase
  Section: section 1
    Section: section 1.1
      Section: section 1.1.2
        info:
          loc: /Users/neverlord/github/unit-test-example/sections/main.test.cpp:19
          msg: test « section 1.1.2

 Test: sections showcase
  Section: section 1
    Section: section 1.2
      info:
        loc: /Users/neverlord/github/unit-test-example/sections/main.test.cpp:24
        msg: test « section 1.2

 Test: sections showcase
  Section: section 2
    Section: section 2.1
      info:
        loc: /Users/neverlord/github/unit-test-example/sections/main.test.cpp:31
        msg: test « section 2.1

 Test: sections showcase
  Section: section 2
    Section: section 2.2
      info:
        loc: /Users/neverlord/github/unit-test-example/sections/main.test.cpp:35
        msg: test « section 2.2

This also means that a failed requirement in one section does not affect the execution of other sections.

Scenarios: Given-When-Then

Writing tests using the given-when-then pattern is a common practice and CAF supports this pattern as an alternative to using regular tests and sections.

Scenarios use the SCENARIO macro instead of the TEST macro. Inside a scenario, we can use the GIVEN, WHEN, and THEN macros to structure the test code. The GIVEN macro is used to set up the initial state of the test. Inside the GIVEN block, we can define one or more WHEN blocks that describe the actions that we want to test. Finally, the WHEN block contains a THEN block that checks the outcome of the actions.

Just like with sections, each leaf node of the scenario tree is run as a separately. To following example revisits our previous test for sorting and searching in a vector:

SCENARIO("std::sort sorts a sequence of elements") {
  GIVEN("a vector of integers") {
    auto xs = std::vector{3, 1, 4, 1, 5, 9, 2, 6};
    WHEN("sorting the vector using std::sort") {
      std::sort(xs.begin(), xs.end());
      THEN("the vector is sorted in ascending order") {
        check_eq(xs, std::vector{1, 1, 2, 3, 4, 5, 6, 9});
      }
    }
  }
}

SCENARIO("std::find searches for an element in a sequence") {
  GIVEN("a vector of integers") {
    auto xs = std::vector{3, 1, 4, 1, 5, 9, 2, 6};
    WHEN("searching for an existing element") {
      THEN("std::find returns an iterator to the element") {
        auto i = std::find(xs.begin(), xs.end(), 5);
        if (check_ne(i, xs.end())) {
          check_eq(*i, 5);
        }
      }
    }
    WHEN("searching for a non-existing element") {
      THEN("std::find returns an iterator to the end of the sequence") {
        check_eq(std::find(xs.begin(), xs.end(), 42), xs.end());
      }
    }
  }
}

Note that a scenario cannot contain sections. However, a single scenario can contain multiple GIVEN blocks that will run independently.

Outlines

Outlines in CAF are parameterized scenarios that allow us to run the same scenario with different input values. We can think of them as scenario templates that are instantiated with different arguments. The sources for the examples in this section can be found in the directory outlines.

An outline is defined with the OUTLINE macro. Inside the outline, we can use the GIVEN, WHEN, and THEN macros just like in a regular scenario. In the description of the individual blocks, we can use placeholders that will be replaced with the actual values when the outline is instantiated. Placeholds use the syntax <name>, where name is an arbitrary identifier.

The key-value pairs for the instantiation of the outline are passed to an EXAMPLES block as markdown-formatted table. The first row of the table contains the names of the placeholders and the following rows contain the values for each example. Each row in the table corresponds to a single instantiation of the outline.

To retrieve the values for the placeholders, we can use the function block_parameters, which takes the types of the placeholders as template arguments. The function will scan the description of the current block and create a tuple with the values for the placeholders in the order they appear in the block. If the block only contains a single placeholder, the function will return the value directly (not as a tuple).

In the following example, we define an outline with three examples for adding two numbers:

OUTLINE("adding two numbers") {
  GIVEN("the numbers <a> and <b>") {
    auto [a, b] = block_parameters<int, int>();
    WHEN("adding the numbers") {
      auto result = a + b;
      THEN("the result is <sum>") {
        auto sum = block_parameters<int>();
        check_eq(result, sum);
      }
    }
  }
  EXAMPLES = R"_(
    | a  | b  | sum |
    | 1  | 7  | 8   |
    | 7  | 1  | 8   |
    | -7 | 11 | 4   |
  )_";
}

When running the test, CAF will instantiate the outline three times, once for each row in the table. As we can see, when calling block_parameters with only a single template argument, the function will return the value directly. When calling the function with multiple template arguments, the function will return a tuple with the values in the order they appear in the block description.

The type conversion for the placeholders is done automatically by CAF. In this example, we convert all values to int. However, we can also use other types. CAF will try to convert the values to the requested type by using the same parser that CAF would use for parsing command line arguments.

Lists generally use the syntax [...]. However, a list of comma-separated values is interpreted as a list when CAF tries to convert the value to a container type like std::vector:

OUTLINE("std::sort sorts a sequence of elements") {
  GIVEN("a vector containing the values <input>") {
    auto input = block_parameters<std::vector<int>>();
    WHEN("sorting the vector using std::sort") {
      std::sort(input.begin(), input.end());
      THEN("the vector should contain <output>") {
        auto output = block_parameters<std::vector<int>>();
        check_eq(input, output);
      }
    }
  }
  EXAMPLES = R"_(
    | input                    | output                   |
    | 1                        | [1]                      |
    | [1, 3, 2]                | [1, 2, 3]                |
    | 3, 1, 4, 1, 5, 9, 2, 6   | 1, 1, 2, 3, 4, 5, 6, 9   |
  )_";
}

In our EXAMPLES section above, we mix explicitly defined lists using the [...] syntax and "plain" comma-separated values. Since we define the target type as vector<int>, CAF will interpret the comma-separated values as a list of integers. However, we do recommend using the [...] syntax for lists to make the intent clear and also to be consistent.

User-defined types can also be used in outlines. We simply define the values by writing a dictionary into the table that contains the values for the fields of the type. CAF will then use the inspect overload to convert the dictionary to the user-defined type.

In our final example, we define an outline for scaling points by adding a scalar to both coordinates. The type point is a user-defined type with two integer fields x and y (please see the repository for the full definition of the type):

OUTLINE("points can be scaled by adding a scalar to both coordinates") {
  GIVEN("the point <input>") {
    auto value = block_parameters<point>();
    WHEN("adding the scalar <scalar> to it") {
      auto scalar = block_parameters<int>();
      value += scalar;
      THEN("the result point is <output>") {
        auto output = block_parameters<point>();
        check_eq(value, output);
      }
    }
  }
  EXAMPLES = R"_(
    | input            | scalar | output           |
    | {"x": 7, "y": 5} |  2     | {"x": 9, "y": 7} |
    | {"x": 0, "y": 0} |  0     | {"x": 0, "y": 0} |
    | {"x": 1, "y": 2} | -1     | {"x": 0, "y": 1} |
  )_";
}

Fixtures

Like many other unit testing frameworks, CAF also supports fixtures. Fixtures allow us to set up a common state or to provide common functionality for a set of tests. A fixture is simply a class that will be injected into the test code as a base class. The test code can then access the members and functions of the fixture class.

To add a fixture to a test, we use the WITH_FIXTURE macro. This macro takes the fixture class as a template argument and a block of tests. Inside the block, we can access the members and functions of the fixture class through the this pointer.

struct my_fixture {
  int x = 0;
  int y = 0;

  void setup() {
    x = 42;
    y = 23;
  }
};

WITH_FIXTURE(my_fixture) {

TEST("tests can access fixture members") {
  check_eq(x, 0);
  check_eq(y, 0);
  setup();
  check_eq(x, 42);
  check_eq(y, 23);
}

} // WITH_FIXTURE(my_fixture)

In this example, we define a fixture class my_fixture and then inject it into the test code using the WITH_FIXTURE macro. Inside the test block, we can access all members and functions of the fixture class. The fixture class is constructed before the test block is executed and destructed after the test block has run. Hence, each run of the test block will have a fresh instance of the fixture class.

Fixtures cannot be nested, i.e., a test block can only have a single fixture. However, this is rarely a limitation in practice since we can simply define a fixture that inherits from multiple other fixtures we want to use.

Of course, the example we have shown here is a toy example. In practice, we would not use a fixture to set up a simple state like this. If we need to have some state for multiple test blocks, we would usually simply use sections. However, fixtures provide a powerful customization point for setting up a domain-specific language (DSL) to write tests. CAF also provides a predefined fixture for deterministic testing of actors, as we will see in part 2 of this guide.

Conclusion

In this part of the guide, we have introduced the CAF unit testing API. We have shown how to write checks, requirements, tests, scenarios, and outlines as well as how to use custom fixtures.

The main benefit of using the CAF unit testing API is the tight integration with the framework. For example, the logging API is directly available in the test code and automatically picks up inspect overloads for user-defined types.

In part 2, we will delve into the deterministic testing of actors with CAF.