http://anadoxin.org/blog

Type-parametrized unit testing in GTest

Mon, 12 January 2015 :: #cpp :: #testing

Google's testing framework, GTest, goes beyond writing basic unit tests. It has some more advanced functionality that allows the user to cover more code with less tests, which -- considering the fact that unit tests always make a significant share of the source code ratio in a project -- is always a good thing.

Test parametrization reduces the need to duplicate code inside tests. It's really best suited for testing interfaces and underlying implementations of these interfaces.

Consider a following example. We have a few container classes inside our application. To keep things simple let's use std containers to implement our own. So, we will build three containers:

VectorContainer, based on `std::vector`,
ListContainer, based on `std::list`,
QueueContainer, based on `std::deque`.

As for generalization, let's create a simple interface IContainer which will define all the functionality needed to form a simple container for our needs. We'll use only a few methods:

  1. Adding to the container by the add() method,
  2. Iterating through the container by the forEach() method,
  3. Reporting the size of the container by the count() method.

Here's our interface:

#pragma once
#include <functional>

typedef std::function<void (const size_t num)> IteratorFuncT;

class IContainer {
public:
    virtual void add(size_t num) = 0;
    virtual size_t count() const = 0;
    virtual void forEach(IteratorFuncT closure) const = 0;
};

The interface will allow the user of our container to add some files into it, ask about the number of elements currently stored inside, and iterate through all added elements. Pretty basic stuff.

Let's create our first implementation, based on std::vector. Please note that "pure" TDD philosophy advocates writing tests before the implementation, but since we're not producing production code here, we'll try to get away with Test Aware Development, instead of Test Driven Development ;).

// VectorContainer.h

#pragma once
#include "IContainer.h"
#include <vector>

class VectorContainer: public IContainer {
private:
    std::vector<size_t> items;

public:
    void add(size_t num) override {
        items.push_back(num);
    }

    size_t count() const override {
        return items.size();
    }

    void forEach(IteratorFuncT closure) const override {
        for(const size_t item: items) {
            closure(item);
        }
    }
};

OK, since we have our first implementation, it's possible to write some tests.

One way would be to create multiple tests tailored for VectorContainer, and later copy and paste them to another file, while performing refactoring of the VectorContainer type to ListContainer, like this:

// Test.cpp

#include <gtest/gtest.h>
#include "VectorContainer.h"

TEST(ContainerTest, ShouldBeEmptyOnStartup) {
    VectorContainer container;
    ASSERT_EQ(container.count(), 0);
}

TEST(ContainerTest, ShouldAdd1) {
    VectorContainer container;
    container.add(1);
    ASSERT_EQ(container.count(), 1);
}

But, of course, we can do better.

We can define a generic test fixture first. Then, we can bind multiple types to that fixture, so that when it will finally be instantiated by the test runner, the runner will create multiple instances, each instance for each declared type of the test fixture. Then, each test for each fixture will be invoked and executed, testing all of the types that we declared.

In other words, we can bind multiple types to a test class. Later, we can bind multiple tests to the same class. When we do both things, each test will be executed multiple times, depending on how many types we declared.

So, scratch your current Test.cpp, and let's start again. First, usual set of preprocessor includes:

#include <gtest/gtest.h>
#include "VectorContainer.h"

Then, we need to create our fixture class. This class will be the central point of our test collection.

template <typename T>
class ContainerTest: public ::testing::Test { };
TYPED_TEST_CASE_P(ContainerTest);

Our fixture class should be a subclass of ::testing::Test class, which is a part of GTest framework. For our current purposes it can be empty, that's why it has no body, not even a constructor nor destructor.

Then, the TYPED_TEST_CASE_P macro initializes the code needed to turn this fixture class into a parametrized one.

Now we can write our tests. But let's skip that for a minute and do a different thing: let's specify what types the framework should instantiate in order to test the interface. This is done by the following code:

// placeholder for test bodies

typedef ::testing::Types<VectorContainer> ContainerTypes;
INSTANTIATE_TYPED_TEST_CASE_P(ContainerTypesInstantiation, ContainerTest, ContainerTypes);

First we need to create a typedef of the ::testing::Types class. I will name this new type ContainerTypes, and the type of it will be specified as:

    ::testing::Types<VectorContainer>

So, we will specify here that we want to instantiate the VectorContainer class inside our parametrized tests, that we'll write in a minute or two.

After the typedef has been created, it needs to be bound to our fixture class by using the INSTANTIATE_TYPED_TEST_CASE_P macro:

    INSTANTIATE_TYPED_TEST_CASE_P(PlaceholderName, OurFixtureClass, OurTypedef);

PlaceholderName is a name of some kind. To be completely honest, I have no idea what is its purpose, it seems that it can be anything.

OurFixtureClass is a name of our fixture class: ContainerTest.

OurTypedef is a typedef we have just created, that points to the types we would like to be instantiated inside test cases.

Now, we can write some unit tests, but instead of specifying TEST macro, we need to use TYPED_TEST_P. Note: Remember to put those tests before the typedef we just declared, not after. See the "placeholder for test bodies" comment above -- that's the spot you need to put your tests into.

TYPED_TEST_P(ContainerTest, ShouldBeEmptyOnStartup) {
    TypeParam container;
    ASSERT_EQ(container.count(), 0);
}

TYPED_TEST_P(ContainerTest, ShouldAdd1) {
    TypeParam container;
    container.add(1);
    ASSERT_EQ(container.count(), 1);
}

REGISTER_TYPED_TEST_CASE_P(ContainerTest,
    ShouldBeEmptyOnStartup,
    ShouldAdd1
);

Inside each test I use TypeParam to specify the type of the container to be tested. In runtime, it will point to a concrete type, specified in the ::testing::Types<> typedef that was created earlier.

Also, have you seen the last 4 lines? Each test should be included inside the REGISTER_TYPED_TEST_CASE_P macro. Without it, the test runner will abort in runtime reminding you about any test cases that aren't specified here.

Let's compile whole project and see if it works:

g++ Test.cpp -o Test -lgtest -lgtest_main -std=c++11

Linking gtest along with gtest_main will allow you to skip defining the main() function. gtest_main has its own main().

$ ./Test
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from ContainerTypesInstantiation/ContainerTest/0, where TypeParam = VectorContainer
[ RUN      ] ContainerTypesInstantiation/ContainerTest/0.ShouldBeEmptyOnStartup
[       OK ] ContainerTypesInstantiation/ContainerTest/0.ShouldBeEmptyOnStartup (0 ms)
[ RUN      ] ContainerTypesInstantiation/ContainerTest/0.ShouldAdd1
[       OK ] ContainerTypesInstantiation/ContainerTest/0.ShouldAdd1 (0 ms)
[----------] 2 tests from ContainerTypesInstantiation/ContainerTest/0 (0 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (0 ms total)
[  PASSED  ] 2 tests.

Well, nothing extraordinary here, the test runner has executed two tests that we have defined. So where is the justification for all those macros that we had to use?

It will be more clear when we'll create another class implementing the IContainer interface, ListContainer. Here's the code:

// ListContainer.h
#pragma once

#include "IContainer.h"
#include <list>

class ListContainer: public IContainer {
private:
    std::list<size_t> items;

public:
    void add(size_t num) override {
        items.push_back(num);
    }

    size_t count() const override {
        return items.size();
    }

    void forEach(IteratorFuncT closure) const override {
        for(const size_t item: items) {
            closure(item);
        }
    }
};

It looks like the previous class, but this one uses std::list, so it's different.

Now what do we do to test this class? We modify our ContainerTypes typedef by adding a new type:

typedef ::testing::Types<VectorContainer, ListContainer> ContainerTypes;

And we recompile the project. After we run the test runner, it will automatically call the test cases for our new interface:

$ ./Test
Running main() from gtest_main.cc
[==========] Running 4 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from ContainerTypesInstantiation/ContainerTest/0, where TypeParam = VectorContainer
[ RUN      ] ContainerTypesInstantiation/ContainerTest/0.ShouldBeEmptyOnStartup
[       OK ] ContainerTypesInstantiation/ContainerTest/0.ShouldBeEmptyOnStartup (0 ms)
[ RUN      ] ContainerTypesInstantiation/ContainerTest/0.ShouldAdd1
[       OK ] ContainerTypesInstantiation/ContainerTest/0.ShouldAdd1 (0 ms)
[----------] 2 tests from ContainerTypesInstantiation/ContainerTest/0 (0 ms total)

[----------] 2 tests from ContainerTypesInstantiation/ContainerTest/1, where TypeParam = ListContainer
[ RUN      ] ContainerTypesInstantiation/ContainerTest/1.ShouldBeEmptyOnStartup
[       OK ] ContainerTypesInstantiation/ContainerTest/1.ShouldBeEmptyOnStartup (0 ms)
[ RUN      ] ContainerTypesInstantiation/ContainerTest/1.ShouldAdd1
[       OK ] ContainerTypesInstantiation/ContainerTest/1.ShouldAdd1 (0 ms)
[----------] 2 tests from ContainerTypesInstantiation/ContainerTest/1 (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 2 test cases ran. (0 ms total)
[  PASSED  ] 4 tests.

Much better than copy, paste and refactor!

We can create another test just to see that it's really working:

TYPED_TEST_P(ContainerTest, ShouldIterate) {
    TypeParam container;
    container.add(1);
    container.add(2);
    container.add(3);

    std::vector<size_t> verification;
    auto proc = [& verification, & container] (const size_t item) -> void {
        verification.push_back(item);
    };

    container.forEach(proc);

    ASSERT_EQ(verification.size(), 3);
    ASSERT_EQ(verification[0], 1);
    ASSERT_EQ(verification[1], 2);
    ASSERT_EQ(verification[2], 3);
}

We compile the project and run it:

$ g++ Test.cpp -o Test -lgtest_main -lgtest -std=c++11 && ./Test
Test.cpp:41: You forgot to list test ShouldIterate.
[1]    12001 abort (core dumped)  ./Test

Yes, I forgot to include ShouldIterate in REGISTER_TYPED_TEST_CASE_P...

REGISTER_TYPED_TEST_CASE_P(ContainerTest,
    ShouldBeEmptyOnStartup,
    ShouldAdd1,
    ShouldIterate
);

And the test has been automatically bound to both types.

$ g++ Test.cpp -o Test -lgtest_main -lgtest -std=c++11 && ./Test
Running main() from gtest_main.cc
[==========] Running 6 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from ContainerTypesInstantiation/ContainerTest/0, where TypeParam = VectorContainer
[ RUN      ] ContainerTypesInstantiation/ContainerTest/0.ShouldBeEmptyOnStartup
[       OK ] ContainerTypesInstantiation/ContainerTest/0.ShouldBeEmptyOnStartup (0 ms)
[ RUN      ] ContainerTypesInstantiation/ContainerTest/0.ShouldAdd1
[       OK ] ContainerTypesInstantiation/ContainerTest/0.ShouldAdd1 (0 ms)
[ RUN      ] ContainerTypesInstantiation/ContainerTest/0.ShouldIterate
[       OK ] ContainerTypesInstantiation/ContainerTest/0.ShouldIterate (0 ms)
[----------] 3 tests from ContainerTypesInstantiation/ContainerTest/0 (0 ms total)

[----------] 3 tests from ContainerTypesInstantiation/ContainerTest/1, where TypeParam = ListContainer
[ RUN      ] ContainerTypesInstantiation/ContainerTest/1.ShouldBeEmptyOnStartup
[       OK ] ContainerTypesInstantiation/ContainerTest/1.ShouldBeEmptyOnStartup (0 ms)
[ RUN      ] ContainerTypesInstantiation/ContainerTest/1.ShouldAdd1
[       OK ] ContainerTypesInstantiation/ContainerTest/1.ShouldAdd1 (0 ms)
[ RUN      ] ContainerTypesInstantiation/ContainerTest/1.ShouldIterate
[       OK ] ContainerTypesInstantiation/ContainerTest/1.ShouldIterate (0 ms)
[----------] 3 tests from ContainerTypesInstantiation/ContainerTest/1 (0 ms total)

[----------] Global test environment tear-down
[==========] 6 tests from 2 test cases ran. (0 ms total)
[  PASSED  ] 6 tests.

Now you can create the third type, QueueContainer, based on std::deque, and see how easy it is to create new parametrized tests!