Model-Based Testing for Embedded Systems

Justyna Zander, Ina Schieferdecker, Pieter J. Mosterman, Justyna Zander, Ina Schieferdecker, Pieter J. Mosterman

• Publisher: CRC Press
• Year: 2017
• ISBN:
  9781351833912
• Edition: 1
• Topic: Informatik
• Subtopic: Computertechnik

What the experts have to say about Model-Based Testing for Embedded Systems:

"This book is exactly what is needed at the exact right time in this fast-growing area. From its beginnings over 10 years ago of deriving tests from UML statecharts, model-based testing has matured into a topic with both breadth and depth. Testing embedded systems is a natural application of MBT, and this book hits the nail exactly on the head. Numerous topics are presented clearly, thoroughly, and concisely in this cutting-edge book. The authors are world-class leading experts in this area and teach us well-used and validated techniques, along with new ideas for solving hard problems.

"It is rare that a book can take recent research advances and present them in a form ready for practical use, but this book accomplishes that and more. I am anxious to recommend this in my consulting and to teach a new class to my students."

—Dr. Jeff Offutt, professor of software engineering, George Mason University, Fairfax, Virginia, USA

"This handbook is the best resource I am aware of on the automated testing of embedded systems. It is thorough, comprehensive, and authoritative. It covers all important technical and scientific aspects but also provides highly interesting insights into the state of practice of model-based testing for embedded systems."

—Dr. Lionel C. Briand, IEEE Fellow, Simula Research Laboratory, Lysaker, Norway, and professor at the University of Oslo, Norway

"As model-based testing is entering the mainstream, such a comprehensive and intelligible book is a must-read for anyone looking for more information about improved testing methods for embedded systems. Illustrated with numerous aspects of these techniques from many contributors, it gives a clear picture of what the state of the art is today."

—Dr. Bruno Legeard, CTO of Smartesting, professor of Software Engineering at the University of Franche-Comté, Besançon, France, and co-author of Practical Model-Based Testing

Part I

Introduction

1

A Taxonomy of Model-Based Testing for Embedded Systems from Multiple Industry Domains

Justyna Zander, Ina Schieferdecker and Pieter J. Mosterman

CONTENTS

1.1 Introduction

1.2 Definition of Model-Based Testing

1.2.1 Test dimensions

1.2.1.1 Test goal

1.2.1.2 Test scope

1.2.1.3 Test abstraction

1.3 Taxonomy of Model-Based Testing

1.3.1 Model

1.3.2 Test generation

1.3.2.1 Test selection criteria

1.3.2.2 Test generation technology

1.3.2.3 Result of the generation

1.3.3 Test execution

1.3.4 Test evaluation

1.3.4.1 Specification

1.3.4.2 Technology

1.4 Summary

References

1.1 Introduction

This chapter provides a taxonomy of Model-Based Testing (MBT) based on the approaches that are presented throughout this book as well as in the related literature. The techniques for testing are categorized using a number of dimensions to familiarize the reader with the terminology used throughout the chapters that follow.

In this chapter, after a brief introduction, a general definition of MBT and related work on available MBT surveys is provided. Next, the various test dimensions are presented. Subsequently, an extensive taxonomy is proposed that classifies the MBT process according to the MBT foundation (referred to as MBT basis), definition of various test generation techniques, consideration of test execution methods, and the specification of test evaluation. The taxonomy is an extension of previous work by Zander and Schieferdecker (2009) and it is based on contributions of Utting, Pretschner, and Legeard (2006). A summary concludes the chapter with the purpose of encouraging the reader to further study the contributions of the collected chapters in this book and the specific aspects of MBT that they address in detail.

1.2 Definition of Model-Based Testing

This section provides a brief survey of the selected definitions of MBT available in the literature. Next, certain aspects of MBT are highlighted in the discussion on test dimensions and their categorization is illustrated.

MBT relates to a process of test generation from models of/related to a system under test (SUT) by applying a number of sophisticated methods. The basic idea of MBT is that instead of creating test cases manually, a selected algorithm is generating them automatically from a model. MBT usually comprises the automation of black-box test design (Utting and Legeard 2006), however recently it has been used to automate white-box tests as well. Several authors such as Utting (2005) and Kamga, Hermann, and Joshi (2007) define MBT as testing in which test cases are derived in their entirety or in part from a model that describes some aspects of the SUT based on selected criteria. Utting, Pretschner, and Legeard (2006) elaborate that MBT inherits the complexity of the domain or, more specifically, of the related domain models. Dai (2006) refers to MBT as model-driven testing (MDT) because of the context of the model-driven architecture (MDA) (OMG 2003) in which MBT is proposed.

Advantages of MBT are that it allows tests to be linked directly to the SUT requirements, which renders readability, understandability, and maintainability of tests easier. It helps ensure a repeatable and scientific basis for testing. Furthermore, MBT has been shown to provide good coverage of all the behaviors of the SUT (Utting 2005) and to reduce the effort and cost for testing (Pretschner et al. 2005).

The term MBT is widely used today with subtle differences in its meaning. Surveys on different MBT approaches are provided by Broy et al. (2005), Utting, Pretschner, and Legeard (2006), and the D-Mint Project (2008), and Schieferdecker et al. (2011). In the automotive industry, MBT describes all testing activities in the context of Model-Based Design (MBD), as discussed for example, by Conrad, Fey, and Sadeghipour (2004) and Lehmann and Krämer (2008). Rau (2002), Lamberg et al. (2004), and Conrad (2004a, 2004b) define MBT as a test process that encompasses a combination of different test methods that utilize the executable model in MBD as a source of information. As a single testing technique is insufficient to achieve a desired level of test coverage, different test methods are usually combined to complement each other across all the specified test dimensions (e.g., functional and structural testing techniques are frequently applied together). If sufficient test coverage has been achieved on the model level, properly designed test cases can be reused for testing the software created based on or generated from the models within the framework of back-to-back tests as proposed by Wiesbrock, Conrad, and Fey (2002). With this practice, the functional equivalence between the specification, executable model, and code can be verified and validated (Conrad, Fey, and Sadeghipour 2004).

The most generic definition of MBT is testing in which the test specification is derived in its entirety or in part from both the system requirements and a model that describe selected functional and nonfunctional aspects of the SUT.

The test specification can take the form of a model, executable model, script, or computer program code. The resulting test specification is intended to ultimately be executed together with the SUT so as to provide the test results. The SUT again can exist in the form of a model, code, or even hardware.

For example, in Conrad (2004b) and Conrad, Fey, and Sadeghipour (2004), no additional test models are created, but the already existing functional system models are utilized for test purposes. In the test approach proposed by Zander-Nowicka (2009), the system models are exploited as well. In addition, however, a test specification model (also called test case specification, test model, or test design in the literature (Pretschner 2003b, Zander et al. 2005, and Dai 2006) is created semi-automatically. Concrete test data variants are then automatically derived from this test specification model.

The application of MBT is as proliferate as the interest in building embedded systems. For example, case studies borrowed from such widely varying domains as medicine, automotive, control engineering, telecommunication, entertainment, or aerospace can be found in this book. MBT then appears as part of specific techniques that are proposed for testing a medical device, the GSM 11.11 protocol for mobile phones, a smartphone graphical user interface (GUI), a steam boiler, smartcard, a robot-control application, a kitchen toaster, automated light control, analog- and mixed-signal electrical circuits, a feeder-box controller of a city lighting system, and other complex software systems.

1.2.1 Test dimensions

Tests can be classified depending on the characteristics of the SUT and the test system. In this book, such SUT features comprise, for example, safety-critical properties, deterministic and nondeterministic behavior, load and performance, analog characteristics, network-related, and user-friendliness qualities. Furthermore, systems that exhibit behavior of a discrete, continuous, or hybrid nature are analyzed in this book. The modeling paradigms for capturing a model of the SUT and tests combine different approaches, such as history-based, functional data flow combined with transition-based semantics. As it is next to impossible for one single classification scheme to successfully apply to such a wide range of attributes, selected dimensions have been introduced in previous work to isolate certain aspects. For example, Neukirchen (2004) aims at testing communication systems and categorizes testing in the dimensions of test goals, test scope, and test distribution. Dai (2006) replaces the test distribution by a dimension describing the different test development phases, since she is testing both local and distributed systems. Zander-Nowicka (2009) refers to test goals, test abstraction, test execution platforms, test reactiveness, and test scope in the context of embedded automotive systems.

In the following, the specifics related to test goal, test scope, and test abstraction (see Figure 1.1) are introduced to provide a basis for a common vocabulary, simplicity, and a better understanding of the concepts discussed in the rest of this book.

FIGURE 1.1
Selected test dimensions.

1.2.1.1 Test goal

During software development, systems are tested with different purposes (i.e., goals). These goals can be categorized as static testing, also called review, and dynamic testing, where the latter is based on test execution and further distinguishes between structural, functional, and nonfunctional testing. After the review phase, the test goal is usually to check the functional behavior of the system. Nonfunctional tests appear in later development stages.

• Static test: Testing is often defined as the process of finding errors, failures, and faults. Errors in a program can be revealed without execution by just examining its source code (International Software Testing Qualification Board 2006). Similarly, other development artifacts can be reviewed (e.g., requirements, models, or the test specification itself).
• Structural test: Structural tests cover the structure of the SUT during test execution (e.g., control or data flow), and so the internal structure of the system (e.g., code or model) must be known. As such, structural tests are also called white-box or glass-box tests (Myers 1979; International Software Testing Qualification Board 2006).
• Functional test: Functional testing is concerned with assessing the functional behavior of an SUT against the functional requirements. In contrast to structural tests, functional tests do not require any knowledge about system internals. They are therefore called black-box tests (Beizer 1995). A systematic, planned, executed, and documented procedure is desirable to make them successful. In this category, ...