Test Framework

The Zephyr Test Framework (Ztest) provides a simple testing framework intended to be used during development. It provides basic assertion macros and a generic test structure.

The framework can be used in two ways, either as a generic framework for integration testing, or for unit testing specific modules.

Quick start - Integration testing

A simple working base is located at samples/testing/integration. Just copy the files to tests/ and edit them for your needs. The test will then be automatically built and run by the sanitycheck script. If you are testing the bar component of foo, you should copy the sample folder to tests/foo/bar. It can then be tested with:

./scripts/sanitycheck -s tests/foo/bar/test-identifier

In the example above tests/foo/bar signifies the path to the test and the test-identifier references a test defined in the testcase.yaml file.

To run all tests defined in a test project, run:

./scripts/sanitycheck -T tests/foo/bar/

The sample contains the following files:

CMakeLists.txt

1
2
3
4
5
6
cmake_minimum_required(VERSION 3.13.1)
include($ENV{ZEPHYR_BASE}/cmake/app/boilerplate.cmake NO_POLICY_SCOPE)
project(integration)

FILE(GLOB app_sources src/*.c)
target_sources(app PRIVATE ${app_sources})

testcase.yaml

1
2
3
4
tests:
  section.subsection:
    build_only: true
    tags: my_tags

prj.conf

1
CONFIG_ZTEST=y

src/main.c (see best practices)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
/*
 * Copyright (c) 2016 Intel Corporation
 *
 * SPDX-License-Identifier: Apache-2.0
 */

#include <ztest.h>

/**
 * @brief Test Asserts
 *
 * This test verifies various assert macros provided by ztest.
 *
 */
static void test_assert(void)
{
	zassert_true(1, "1 was false");
	zassert_false(0, "0 was true");
	zassert_is_null(NULL, "NULL was not NULL");
	zassert_not_null("foo", "\"foo\" was NULL");
	zassert_equal(1, 1, "1 was not equal to 1");
	zassert_equal_ptr(NULL, NULL, "NULL was not equal to NULL");
}

void test_main(void)
{
	ztest_test_suite(framework_tests,
		ztest_unit_test(test_assert)
	);

	ztest_run_test_suite(framework_tests);
}

A test case project may consist of multiple sub-tests or smaller tests that either can be testing functionality or APIs. Functions implementing a test should follow the guidelines below:

  • Test cases function names should be prefix with test_
  • Test cases should be documented using doxygen
  • Test function names should be unique within the section or component being tested

An example can be seen below:

/**
 * @brief Test Asserts
 *
 * This test verifies the zassert_true macro.
 */
static void test_assert(void)
{
        zassert_true(1, "1 was false");
}

The above test is then enabled as part of the testsuite using:

ztest_unit_test(test_assert)

Listing Tests

Tests (test projects) in the Zephyr tree consist of many testcases that run as part of a project and test similar functionality, for example an API or a feature. The sanitycheck script can parse the testcases in all test projects or a subset of them, and can generate reports on a granular level, i.e. if cases have passed or failed or if they were blocked or skipped.

Sanitycheck parses the source files looking for test case names, so you can list all kernel test cases, for example, by entering:

sanitycheck --list-tests -T tests/kernel

Skipping Tests

Special- or architecture-specific tests cannot run on all platforms and architectures, however we still want to count those and report them as being skipped. Because the test inventory and the list of tests is extracted from the code, adding conditionals inside the test suite is sub-optimal. Tests that need to be skipped for a certain platform or feature need to explicitly report a skip using ztest_test_skip(). If the test runs, it needs to report either a pass or fail. For example:

#ifdef CONFIG_TEST1
void test_test1(void)
{
        zassert_true(1, "true");
}
#else
void test_test1(void)
{
        ztest_test_skip();
}
#endif


void test_main(void)
{
        ztest_test_suite(common,
                         ztest_unit_test(test_test1),
                         ztest_unit_test(test_test2)
                         );
        ztest_run_test_suite(common);
}

Quick start - Unit testing

Ztest can be used for unit testing. This means that rather than including the entire Zephyr OS for testing a single function, you can focus the testing efforts into the specific module in question. This will speed up testing since only the module will have to be compiled in, and the tested functions will be called directly.

Since you won’t be including basic kernel data structures that most code depends on, you have to provide function stubs in the test. Ztest provides some helpers for mocking functions, as demonstrated below.

In a unit test, mock objects can simulate the behavior of complex real objects and are used to decide whether a test failed or passed by verifying whether an interaction with an object occurred, and if required, to assert the order of that interaction.

Best practices for declaring the test suite

sanitycheck and other validation tools need to obtain the list of subcases that a Zephyr ztest test image will expose.

Rationale

This all is for the purpose of traceability. It’s not enough to have only a semaphore test project. We also need to show that we have testpoints for all APIs and functionality, and we trace back to documentation of the API, and functional requirements.

The idea is that test reports show results for every sub-testcase as passed, failed, blocked, or skipped. Reporting on only the high-level test project level, particularly when tests do too many things, is too vague.

Here is a generic template for a test showing the expected use of ztest_test_suite():

#include <ztest.h>

extern void test_sometest1(void);
extern void test_sometest2(void);
#ifndef CONFIG_WHATEVER              /* Conditionally skip test_sometest3 */
void test_sometest3(void)
{
     ztest_test_skip();
}
#else
extern void test_sometest3(void);
#endif
extern void test_sometest4(void);
...

void test_main(void)
{
     ztest_test_suite(common,
                         ztest_unit_test(test_sometest1),
                         ztest_unit_test(test_sometest2),
                         ztest_unit_test(test_sometest3),
                         ztest_unit_test(test_sometest4),
                );
     ztest_run_test_suite(common);
}

For sanitycheck to parse source files and create a list of subcases, the declarations of ztest_test_suite() must follow a few rules:

  • one declaration per line
  • conditional execution by using ztest_test_skip()

What to avoid:

  • packing multiple testcases in one source file

    void test_main(void)
    {
    #ifdef TEST_feature1
            ztest_test_suite(feature1,
                             ztest_unit_test(test_1a),
                             ztest_unit_test(test_1b),
                             ztest_unit_test(test_1c),
            ztest_run_test_suite(feature1);
    #endif
    
    #ifdef TEST_feature2
            ztest_test_suite(feature2,
                             ztest_unit_test(test_2a),
                             ztest_unit_test(test_2b),
            ztest_run_test_suite(feature2);
    #endif
    }
    
  • Do not use #if

            ztest_test_suite(common,
                             ztest_unit_test(test_sometest1),
                             ztest_unit_test(test_sometest2),
    #ifdef CONFIG_WHATEVER
                             ztest_unit_test(test_sometest3),
    #endif
                             ztest_unit_test(test_sometest4),
            ...
    
  • Do not add comments on lines with a call to ztest_unit_test():

    ztest_test_suite(common,
                     ztest_unit_test(test_sometest1),
                     ztest_unit_test(test_sometest2) /* will fail */,
    /* will fail! */ ztest_unit_test(test_sometest3),
                     ztest_unit_test(test_sometest4),
    ...
    
  • Do not define multiple definitions of unit / user unit test case per line

    ztest_test_suite(common,
                     ztest_unit_test(test_sometest1), ztest_unit_test(test_sometest2),
                     ztest_unit_test(test_sometest3),
                     ztest_unit_test(test_sometest4),
    ...
    

Other questions:

  • Why not pre-scan with CPP and then parse? or post scan the ELF file?

    If C pre-processing or building fails because of any issue, then we won’t be able to tell the subcases.

  • Why not declare them in the YAML testcase description?

    A separate testcase description file would be harder to maintain than just keeping the information in the test source files themselves – only one file to update when changes are made eliminates duplication.

API reference

Running tests

group ztest_test

This module eases the testing process by providing helpful macros and other testing structures.

Defines

ztest_unit_test_setup_teardown(fn, setup, teardown)

Define a test with setup and teardown functions.

This should be called as an argument to ztest_test_suite. The test will be run in the following order: setup, fn, teardown.

Parameters
  • fn: Main test function
  • setup: Setup function
  • teardown: Teardown function

ztest_user_unit_test_setup_teardown(fn, setup, teardown)

Define a user mode test with setup and teardown functions.

This should be called as an argument to ztest_test_suite. The test will be run in the following order: setup, fn, teardown. ALL test functions will be run in user mode, and only if CONFIG_USERSPACE is enabled, otherwise this is the same as ztest_unit_test_setup_teardown().

Parameters
  • fn: Main test function
  • setup: Setup function
  • teardown: Teardown function

ztest_unit_test(fn)

Define a test function.

This should be called as an argument to ztest_test_suite.

Parameters
  • fn: Test function

ztest_user_unit_test(fn)

Define a test function that should run as a user thread.

This should be called as an argument to ztest_test_suite. If CONFIG_USERSPACE is not enabled, this is functionally identical to ztest_unit_test().

Parameters
  • fn: Test function

ZTEST_DMEM

Define a test suite.

This function should be called in the following fashion:

ztest_test_suite(test_suite_name,
        ztest_unit_test(test_function),
        ztest_unit_test(test_other_function)
);

ztest_run_test_suite(test_suite_name);

Parameters
  • name: Name of the testing suite

ZTEST_BMEM
ZTEST_SECTION
ztest_test_suite(name, ...)
ztest_run_test_suite(suite)

Run the specified test suite.

Parameters
  • suite: Test suite to run.

Functions

void ztest_test_fail(void)

Fail the currently running test.

This is the function called from failed assertions and the like. You probably don’t need to call it yourself.

void ztest_test_pass(void)

Pass the currently running test.

Normally a test passes just by returning without an assertion failure. However, if the success case for your test involves a fatal fault, you can call this function from z_SysFatalErrorHandler to indicate that the test passed before aborting the thread.

void ztest_test_skip(void)

Skip the current test.

static void unit_test_noop(void)

Do nothing, successfully.

Unit test / setup function / teardown function that does nothing, successfully. Can be used as a parameter to ztest_unit_test_setup_teardown().

Variables

struct k_mem_partition ztest_mem_partition
struct k_mem_domain ztest_mem_domain

Assertions

These macros will instantly fail the test if the related assertion fails. When an assertion fails, it will print the current file, line and function, alongside a reason for the failure and an optional message. If the config option:CONFIG_ZTEST_ASSERT_VERBOSE is 0, the assertions will only print the file and line numbers, reducing the binary size of the test.

Example output for a failed macro from zassert_equal(buf->ref, 2, "Invalid refcount"):

Assertion failed at main.c:62: test_get_single_buffer: Invalid refcount (buf->ref not equal to 2)
Aborted at unit test function
group ztest_assert

This module provides assertions when using Ztest.

Defines

zassert(cond, default_msg, msg, ...)

Fail the test, if cond is false.

You probably don’t need to call this macro directly. You should instead use zassert_{condition} macros below.

Parameters
  • cond: Condition to check
  • msg: Optional, can be NULL. Message to print if cond is false.
  • default_msg: Message to print if cond is false

zassert_unreachable(msg, ...)

Assert that this function call won’t be reached.

Parameters
  • msg: Optional message to print if the assertion fails

zassert_true(cond, msg, ...)

Assert that cond is true.

Parameters
  • cond: Condition to check
  • msg: Optional message to print if the assertion fails

zassert_false(cond, msg, ...)

Assert that cond is false.

Parameters
  • cond: Condition to check
  • msg: Optional message to print if the assertion fails

zassert_is_null(ptr, msg, ...)

Assert that ptr is NULL.

Parameters
  • ptr: Pointer to compare
  • msg: Optional message to print if the assertion fails

zassert_not_null(ptr, msg, ...)

Assert that ptr is not NULL.

Parameters
  • ptr: Pointer to compare
  • msg: Optional message to print if the assertion fails

zassert_equal(a, b, msg, ...)

Assert that a equals b.

a and b won’t be converted and will be compared directly.

Parameters
  • a: Value to compare
  • b: Value to compare
  • msg: Optional message to print if the assertion fails

zassert_not_equal(a, b, msg, ...)

Assert that a does not equal b.

a and b won’t be converted and will be compared directly.

Parameters
  • a: Value to compare
  • b: Value to compare
  • msg: Optional message to print if the assertion fails

zassert_equal_ptr(a, b, msg, ...)

Assert that a equals b.

a and b will be converted to void * before comparing.

Parameters
  • a: Value to compare
  • b: Value to compare
  • msg: Optional message to print if the assertion fails

Functions

static void zassert_mem_equal(void *buf, void *exp, size_t size, const char *msg)

Assert that 2 memory buffers have the same contents.

Parameters
  • buf: Buffer to compare
  • exp: Buffer with expected contents
  • size: Size of buffers
  • msg: Optional message to print if the assertion fails

Mocking

These functions allow abstracting callbacks and related functions and controlling them from specific tests. You can enable the mocking framework by setting CONFIG_ZTEST_MOCKING to “y” in the configuration file of the test. The amount of concurrent return values and expected parameters is limited by CONFIG_ZTEST_PARAMETER_COUNT.

Here is an example for configuring the function expect_two_parameters to expect the values a=2 and b=3, and telling returns_int to return 5:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#include <ztest.h>

static void expect_two_parameters(int a, int b)
{
	ztest_check_expected_value(a);
	ztest_check_expected_value(b);
}

static void parameter_tests(void)
{
	ztest_expect_value(expect_two_parameters, a, 2);
	ztest_expect_value(expect_two_parameters, b, 3);
	expect_two_parameters(2, 3);
}

static int returns_int(void)
{
	return ztest_get_return_value();
}

static void return_value_tests(void)
{
	ztest_returns_value(returns_int, 5);
	zassert_equal(returns_int(), 5, NULL);
}

void test_main(void)
{
	ztest_test_suite(mock_framework_tests,
		ztest_unit_test(parameter_test),
		ztest_unit_test(return_value_test)
	);

	ztest_run_test_suite(mock_framework_tests);
}
group ztest_mock

This module provides simple mocking functions for unit testing. These need CONFIG_ZTEST_MOCKING=y.

Defines

ztest_expect_value(func, param, value)

Tell function func to expect the value value for param.

When using ztest_check_expected_value(), tell that the value of param should be value. The value will internally be stored as an uintptr_t.

Parameters
  • func: Function in question
  • param: Parameter for which the value should be set
  • value: Value for param

ztest_check_expected_value(param)

If param doesn’t match the value set by ztest_expect_value(), fail the test.

This will first check that does param have a value to be expected, and then checks whether the value of the parameter is equal to the expected value. If either of these checks fail, the current test will fail. This must be called from the called function.

Parameters
  • param: Parameter to check

ztest_returns_value(func, value)

Tell func that it should return value.

Parameters
  • func: Function that should return value
  • value: Value to return from func

ztest_get_return_value

Get the return value for current function.

The return value must have been set previously with ztest_returns_value(). If no return value exists, the current test will fail.

Return
The value the current function should return

ztest_get_return_value_ptr

Get the return value as a pointer for current function.

The return value must have been set previously with ztest_returns_value(). If no return value exists, the current test will fail.

Return
The value the current function should return as a void *