Twister blackbox tests

This guide aims to explain the structure of a test file so the reader will be able to understand existing files and create their own. All developers should fix any tests they break and create new ones when introducing new features, so this knowledge is important for any Twister developer.

Basics

Twister blackbox tests are written in python, using the pytest library. Read up on it here . Auxiliary test data follows whichever format it was in originally. Tests and data are wholly contained in the scripts/tests/twister_blackbox directory and prepended with test_.

Blackbox tests should not be aware of the internal twister code. Instead, they should call twister as user would and check the results.

Sample test file

#!/usr/bin/env python3
# Copyright (c) 2024 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0

import importlib
import json
import os
import sys
from unittest import mock

import pytest
from conftest import TEST_DATA, ZEPHYR_BASE, testsuite_filename_mock
from twisterlib.testplan import TestPlan


class TestDummy:
    TESTDATA_X = [
        ("smoke", 5),
        ("acceptance", 6),
    ]

    @classmethod
    def setup_class(cls):
        apath = os.path.join(ZEPHYR_BASE, "scripts", "twister")
        cls.loader = importlib.machinery.SourceFileLoader("__main__", apath)
        cls.spec = importlib.util.spec_from_loader(cls.loader.name, cls.loader)
        cls.twister_module = importlib.util.module_from_spec(cls.spec)

    @classmethod
    def teardown_class(cls):
        pass

    @pytest.mark.parametrize(
        "level, expected_tests", TESTDATA_X, ids=["smoke", "acceptance"]
    )
    @mock.patch.object(TestPlan, "TESTSUITE_FILENAME", testsuite_filename_mock)
    def test_level(self, capfd, out_path, level, expected_tests):
        # Select platforms used for the tests
        test_platforms = ["qemu_x86", "frdm_k64f"]
        # Select test root
        path = os.path.join(TEST_DATA, "tests")
        config_path = os.path.join(TEST_DATA, "test_config.yaml")

        # Set flags for our Twister command as a list of strs
        args = (
            # Flags related to the generic test setup:
            # * Control the level of detail in stdout/err
            # * Establish the output directory
            # * Select Zephyr tests to use
            # * Control whether to only build or build and run aforementioned tests
            ["-i", "--outdir", out_path, "-T", path, "-y"]
            # Flags under test
            + ["--level", level]
            # Flags required for the test
            + ["--test-config", config_path]
            # Flags related to platform selection
            + [
                val
                for pair in zip(["-p"] * len(test_platforms), test_platforms, strict=False)
                for val in pair
            ]
        )

        # First, provide the args variable as our Twister command line arguments.
        # Then, catch the exit code in the sys_exit variable.
        with mock.patch.object(sys, "argv", [sys.argv[0]] + args), pytest.raises(
            SystemExit
        ) as sys_exit:
            # Execute the Twister call itself.
            self.loader.exec_module(self.twister_module)

        # Check whether the Twister call succeeded
        assert str(sys_exit.value) == "0"

        # Access to the test file output
        with open(os.path.join(out_path, "testplan.json")) as f:
            j = json.load(f)
        filtered_j = [
            (ts["platform"], ts["name"], tc["identifier"])
            for ts in j["testsuites"]
            for tc in ts["testcases"]
            if "reason" not in tc
        ]

        # Read stdout and stderr to out and err variables respectively
        out, err = capfd.readouterr()
        # Rewrite the captured buffers to stdout and stderr so the user can still read them
        sys.stdout.write(out)
        sys.stderr.write(err)

        # Test-relevant checks
        assert expected_tests == len(filtered_j)

Comparison with CLI

Test above runs the command

twister -i --outdir $OUTDIR -T $TEST_DATA/tests -y --level $LEVEL
--test-config $TEST_DATA/test_config.yaml -p qemu_x86 -p frdm_k64f

It presumes a CLI with the zephyr-env.sh or zephyr-env.cmd already run.

Such a test provides us with all the outputs we typically expect of a Twister run thanks to importlib ‘s exec_module() [1] . We can easily set up all flags that we expect from a Twister call via args variable [2] . We can check the standard output or stderr in out and err variables.

Beside the standard outputs, we can also investigate the file outputs, normally placed in twister-out directories. Most of the time, we will use the out_path fixture in conjunction with --outdir flag (L52) to keep test-generated files in temporary directories. Typical files read in blackbox tests are testplan.json , twister.xml and twister.log .

Other functionalities

Decorators

• @pytest.mark.usefixtures('clear_log')

  • allows us to use clear_log fixture from conftest.py . The fixture is to become autouse in the future. After that, this decorator can be removed.

• @pytest.mark.parametrize('level, expected_tests', TESTDATA_X, ids=['smoke', 'acceptance'])

  • this is an example of pytest ‘s test parametrization. Read up on it here. TESTDATAs are most often declared as class fields.

• @mock.patch.object(TestPlan, 'TESTSUITE_FILENAME', testsuite_filename_mock)

  • this decorator allows us to use only tests defined in the test_data and ignore the Zephyr testcases in the tests directory. Note that all ``test_data`` tests use test_data.yaml as a filename, not testcase.yaml ! Read up on the mock library here.

Fixtures

Blackbox tests use pytest ‘s fixtures, further reading on which is available here.

If you would like to add your own fixtures, consider whether they will be used in just one test file, or in many.

• If in many, create such a fixture in the scripts/tests/twister_blackbox/conftest.py file.

  • scripts/tests/twister_blackbox/conftest.py already contains some fixtures - take a look there for an example.

• If in just one, declare it in that file.

  • Consider using class fields instead - look at TESTDATAs for an example.

How do I…

Call Twister multiple times in one test?

Sometimes we want to test something that requires prior Twister use. --test-only flag would be a typical example, as it is to be coupled with previous --build-only Twister call. How should we approach that?

If we just call the importlib ‘s exec_module two times, we will experience log duplication. twister.log will duplicate every line (triplicate if we call it three times, etc.) instead of overwriting the log or appending to the end of it.

It is caused by the use of logger module variables in the Twister files. Thus us executing the module again causes the loggers to have multiple handles.

To overcome this, between the calls you ought to use

capfd.readouterr()   # To remove output from the buffer
                     # Note that if you want output from all runs after each other,
                     # skip this line.
clear_log_in_test()  # To remove log duplication

---

Footnotes

[1]

Take note of the setup_class() class function, which allows us to run twister python file as if it were called directly (bypassing the __name__ == '__main__' check).

[2]

We advise you to keep the first section of args definition intact in almost all of your tests, as it is used for the common test setup.