Writing unit tests for glue code
Introduction
This tutorial will lay out the steps for adding unit tests to the Open IoT SDK glue code. It uses the GoogleTest test runner along with the Fake Function Framework (fff) for creating mock C functions for the tests.
fff is a C/C++ micro-framework that allows users to create fake C functions for use in unit tests. When unit testing, this allows for functions to be mocked, with different levels of manipulation to be applied to functions such as:
- setting return values,
- counting the number of calls of a function,
- examining parameters that functions are called with.
There are many more functionalities available within
fff, consult the framework documentation here for further information.
The Open IoT SDK aims to provide mocked symbols for all components it fetches. This allows for unit tests to be written for the component's glue code as well as for other components's glue code that depend on it.
MbedTLS mocks will be used as an example for which symbols to mock and how to mock them. The mocks can be found here.
The MbedTLS mocks are used to write unit test for the aws-iot-device-sdk-embedded-c component. The unit test can be found here.
Unit test scope
Within the Open IoT SDK, we unit test any glue code that is added to integrate a component, this ensures the code works as intended. The code originating from the component itself is not tested by unit test, this is mocked instead. The component code is tested with integration tests by running applications on targets.
Mocks
What is a mock?
A mock is defined by the user and is intended to simulate existing symbols from a component. This allows for the unit test to precisely control the execution path in the file under test for each test case. The mocks are defined in the components sub-directory for the component mocked. The sub-directory structure is as follows:
├── ${REAL_LIBRARY_NAME}/
│ ├── mocks/
│ │ ├── inc/
│ │ ├── src/
│ │ ├── CMakeLists.txt
│ ├── CMakeLists.txt
The mock should be made available as a CMake library target so they can be linked by unit test CMake executable targets that require the mocked symbols. The library target is named after the real component where the symbols would be with the added -mock suffix. For example, the mocks for MbedTLS are made available via the mbedtls-mock CMake target.
Adding mocked header files
Header files should be added following the sub-directory structure in the component being mocked. This is to ensure that their inclusion in the glue code does not generate compiler errors.
For example, if a header file is included as such:
The folder structure within the inc directory should include any sub-directory, where the header file is located. In this case, the structure would be as follows:
├── mbedtls/
│ ├── mocks/
│ │ ├── inc/
│ │ | ├── mbedtls/
│ │ | | ├── ssl.h
│ │ ├── src/
│ │ ├── CMakeLists.txt
│ ├── CMakeLists.txt
...
The inc sub-directory should be the directory exposed by the mocked component CMake target see Creating mocked component CMake target library.
Creating mocked component CMake target library
As the Open IoT SDK utilises CMake as its build system, all mocks are provided as CMake library targets which can the be used throughout the project.
Listing the mock subdirectory
To ensure the mocks directory is built only for unit testing, list it in CMake with a condition. The MbedTLS component CMake illustrates this:
if(BUILD_TESTING AND NOT CMAKE_CROSSCOMPILING)
add_subdirectory(mocks)
else()
<code relating to the real component>
endif()
BUILD_TESTING is set and CMAKE_CROSSCOMPILING is disabled. When the condition is not met, the component real files are built instead of the mock.
Mocked component library
The library for the mocked component should be added in the CMakeLists.txt within the mocks sub-directory. The name of the mocked component should follow the name of the CMake library where the real symbols normally are with the added -mock suffix.
Use the CMake add_library() function to create the CMake library target. Source files that implement the mocked symbols should be added to that library, as seen for the MbedTLS Mocks [here][mbedtls-mocks-cmake].
Example for adding CMake library
Firstly, add the library as shown below:
${REAL_LIBRARY_NAME}is the name of the real CMake library target being mocked.
Source files that implement the mocked library should be located wihtin a src subdirectory within the mocks subdirectory, as laid out in the tree here.
Next, list the inc subdirectory which contains all of the components mocked header files:
inc sub-directory should only include header files required by the mocks. Additional sub-directories may be added in inc if the real library header files are accessible only by also including the path to the header file. The access specifier should be set to PUBLIC to enable consumers of the mocked library to access the mocked symbols.
Finally, link any required libraries for the mocks to function. fff is a must for mocks, as this is the framework used. Any other libraries linked here must be mock libraries, add only the ones that are required. The access specifier here should be set to PRIVATE to ensure only this library has access to the libraries being linked.
Creating header files
When creating the header file, the aim is to create one which closely matches the real file. Below will lay out the process of adding items to the header files, with the aim of only adding what is strictly necessary.
Header file protection
The header file should include the same #define protection as the real file, for example, ssl.h within the MBedTLS uses the same protection as the real file, see below.
Mocking datatypes
When mocking datatypes, it is only required to mock Datatypes that are unique to a component and are defined as part of the header files. Standard C datatypes can be used as normal and do not require mocking.
When defining the mocked datatype, for enum and struct types, ensure you only mock what is required for the unit tests. Datatypes that are not globally accessible can be skipped.
Struct declarations
The Datatype mbedtls_ssl_context is defined as a struct here as part of the real header file, however within the glue code we do not access the parameters of this struct. Hence we can define it as such in the mocked ssl.h file.
If we only needed to access certain members of the struct, we would define it as
This ensures we do not have unnecessary members defined for structs.Enumeration declarations
For enumerations, the same philosophy applies here that we should only mock what is required. However, you should ensure that the values assigned to each enumerator matches the real file enumeration values for consistency. It may be required to explicitly define values if only a partial definition is being done to avoid different values. See an example below:
Above is an example of an enum that might be used within the glue code, with the values assigned to each enum increasing by one each time. However, if the glue code only usesSTATUS_OK and STATUS_TIMEOUT, it will be required to explicitly define the value assigned. See below
By doing this, it is ensured that the value is consitstent in both the mock and real file.
Mocking functions
Functions should be mocked using both a header and source file. The declaration of a function will be in the header file, with the definition residing in the source file.
To mock a function, fff's cheat sheet can be used as a template for the macro to use for a specific function. This gives all the available options for handling functions and the expected application of these with examples.
To declare a function, one of four Macros can be used:
DECLARE_FAKE_VOID_FUNC- Forvoidfunctions with a fixed number of parameters.DECLARE_FAKE_VOID_FUNC_VARARG- Forvoidfunctions with a variable number of parameters.DECLARE_FAKE_VALUE_FUNC- For value returning functions with a fixed number of paramertes.DECLARE_FAKE_VALUE_FUNC_VARARG- For value returning functions with a variable number of parameters.
An example from the MbedTLS mocks:
The following functions are defined in the ssl.h header file within MbedTLS:
void mbedtls_ssl_init(mbedtls_ssl_context *ssl);
int mbedtls_ssl_setup(mbedtls_ssl_context *ssl, const mbedtls_ssl_config *conf);
ssl.h mocked header file would be as follows:
DECLARE_FAKE_VOID_FUNC(mbedtls_ssl_init, mbedtls_ssl_context *);
DECLARE_FAKE_VALUE_FUNC(int, mbedtls_ssl_setup, mbedtls_ssl_context *, const mbedtls_ssl_config *);
DECLARE_FAKE_VALUE_FUNC requires the return type to be the first argument in the Macro, whereas DECLARE_FAKE_VOID_FUNC has the name of the function as the first argument.
MbedTLS then defines these functions within the ssl_tls.c file, where the mock has the same name in the Open IoT SDK. The definition is as follows:
DEFINE_FAKE_VOID_FUNC(mbedtls_ssl_init, mbedtls_ssl_context *);
DEFINE_FAKE_VALUE_FUNC(int, mbedtls_ssl_setup, mbedtls_ssl_context *, const mbedtls_ssl_config *);
There may be situations where functions are declared and defined in header and source files which have file names that do not match. In this case, ensure that the file names used for the mocks match the names used in the real files throughout.
Mocking macros
There may be a need to mock defines that are defined in the component for use in the glue code, for defines such as the example below, from here, can simply be copied.
For macros that provide functionality, these should be mocked as a function.
Creating unit test files
Unit test files should be created within a tests sub-directory adjacent to the file being tested, as follows:
test_${FILE_UNDER_TEST}.cpp where ${FILE_UNDER_TEST} is the name of the source file for the glue code.
For example, mbedtls_threading_cmsis_rtos.c will have the test file test_mbedtls_threading_cmsis_rtos.cpp. Every source file for the glue code should have its own test file. The source file will be referred from this point onwards as the "file under test".
Including C header files in the test file
Use the extern "C" pre-processor directive to include C header files within the test file for the source file to be visible to the toolchain.
E.g.
Creating fff test class
Add the fff global variables by adding the DEFINE_FFF_GLOBALS macro.
GoogleTest requires the use of a test suites classes, inheriting from ::testing::Test, for running the unit tests. The class should be named as follows, in PascalCase form:
So for the file mbedtls_threading_cmsis_rtos.c, the test class will be called TestMbedTlsThreadingCmsisRtos.
Child test suites can inherit from other test suites, with some setup already configured. For example, test_mbedtls_threading_cmsis_rtos.cpp has a second class, TestMbedTlsThreadingCmsisRtosInitializedMutex which has already initialized the mutex being used so this does not need to be done every time within the test itself. These test suites should inherit from existing test suites and any mocked functions used in them should also be reset.
Resetting mocked functions
All the mocked functions called during the execution of the test cases should be reset in the test suite constructor, fff provides the RESET_FAKE macro to do this.
E.g.
class TestFileUnderTest : public ::testing::Test{
public:
TestFileUnderTest()
{
RESET_FAKE(executed_mocked_function_1);
RESET_FAKE(executed_mocked_function_2);
}
}
Creating test cases
Test cases are written using GoogleTest's macro for writing tests macro.
The test name should be describing what you are testing.
For example, a test for checking if a function for initializing a mutex returns an error if a parameter (ctx) is null could be named initializing_mutex_fails_when_ctx_is_null
This tells a user exactly what is being tested and what the expected outcome is.
The test case is written in snake case with no space characters.
An example of a test, taken from AWS IoT Device SDK Embedded C Unit Tests can be seen below:
TEST_F(TestAwsNetworkManager, sending_message_fails_when_buf_is_nullptr)
{
NetworkContext_t network_context = {};
EXPECT_NE(0, AwsNetwork_send(&network_context, nullptr, 0));
}
Set the expected return values of the mocked functions before calling a function to test.
E.g
TEST_F(TestAwsNetworkManager, network_connection_fails_if_configuring_tls_contexts_fails)
{
NetworkContext_t network_context = {};
AwsNetworkContextConfig_t network_config = {};
AwsNetwork_init(&network_context, &network_config);
// Prevents successful TLS context configuration
mbedtls_ctr_drbg_seed_fake.return_val = -1;
EXPECT_NE(0, AwsNetwork_connect(&network_context, [](NetworkContext_t *context) {
(void)context;
return 0;
}));
}
You can see from the above example that a TLS configuration is set to fail before calling the function to start the network connection.
Avoiding implementation specific tests
It is the preferred to test functionality rather implementation details. If values are returned (by value or reference) by the function called in the test case, test against the returned value. This allows for the implementation of a function to be changed and the functionality tested without breaking the test case.
An example of a good test would be as follows. This will test the return value of the function using GoogleTest's EXPECT_EQ assertion macro.
EXCEPT_EQandEXCEPT_NEthat compare values so the error message in case of failure better informs about the cause of the failure.
Whereas a test as follows would not be as good as it is reliant on the implementation of the function. If the function cloud_mutex_init is changed or removed, this test is no longer applicable and requires extra maintenance.
Functions that do not have observable primary effects should be tested for their side effects to test the behavior expected. If the side effects of the functions cannot be tested there should not be test cases for the behaviour.
An example of using another function to verify the functionality would be from here.
TEST_F(TestAwsNetworkManager, network_initialization_fails_when_config_is_nullptr)
{
NetworkContext_t network_context = {};
AwsNetwork_init(&network_context, nullptr);
EXPECT_NE(0, AwsNetwork_connect(&network_context, [](NetworkContext_t *context) {
(void)context;
return 0;
}));
}
This is testing the function AwsNetwork_init, however this function does not return a value. We can however use the function AwsNetwork_connect to verify if the initialization has been successful. In this case, we expect an error value to be returned from AwsNetwork_connect as the initialization process has not been successful.
Integrating unit test file with CMake
Adding subdirectories
The unit test file is included in the test build using the CMake build system.
The tests sub-directory which contains the test file is to be added near the glue code file to be tested.
Just like the mocks sub-directory, the tests directory is listed conditionally for non cross-compiling test build as follows:
if(BUILD_TESTING AND NOT CMAKE_CROSSCOMPILING)
add_subdirectory(mocks)
add_subdirectory(tests)
else()
<code relating to the real component>
endif()
The mocks directory must be added before the tests directory. This ensures the mocked CMake library target can be referenced in the test's CMakeLists.txt to link it.
Unit test CMakeLists.txt
The snippet below shows a template for the tests sub-directory CMakeLists.txt file:
add_executable(${FILE_UNDER_TEST}-test
test_${FILE_UNDER_TEST}.cpp
../${FILE_UNDER_TEST}.c
)
target_include_directories(${FILE_UNDER_TEST}-test
PRIVATE
${PATH_TO_FILE_UNDER_TEST_HEADER_FILE_WITH_PUBLIC_API}
)
target_link_libraries(${FILE_UNDER_TEST}-test
PRIVATE
fff
project_options
${REQUIRED_MOCKED_LIBRARY}
)
iotsdk_add_test(${FILE_UNDER_TEST}-test)
add_executable: Creates the CMake target executable and allows the listing of the test file and the file under test.target_include_directories: Includes the sub-directories that contain the header file(s) that exposes the function of the file under test. See the next bullet point for header files that are part of a mocked CMake target library.target_link_libraries: Links any library required to successfully build the executable. This includesfffas well as any required mocked CMake library.project-optionsis also added to ensure the best practices are followed.iotsdk_add_test: A helper function to make the test suite discoverable by GoogleTest.
Building and running unit tests
Two commands will be required to build and run the unit tests. The first step is to configure the unit test build directory, this is done with the following CMake command.
cmake -S . -B __unit_build -GNinja -DENABLE_DEVELOPER_MODE=ON -DOPT_ENABLE_DOXYGEN=OFF -DOPT_ENABLE_SANITIZER_LEAK=OFF -DOPT_ENABLE_CLANG_TIDY=OFF -DFETCHCONTENT_QUIET=OFF
To build and run the unit tests, use the following command:
This will build the tests and then run these. You will see an output for each test, with either pass or fail, and an output at the end stating the total number of tests passed. For any tests that fail, an output will be present for debugging purposes.