Testing Complex and Embedded Systems

Kim H. Pries, Jon M. Quigley

• Publisher: CRC Press
• Year: 2018
• Topic: Computer Science
• eBook ISBN:
  9781351688994
• Subtopic: Computer Science General
• Index: Computer Science

Many enterprises regard system-level testing as the final piece of the development effort, rather than as a tool that should be integrated throughout the development process. As a consequence, test teams often execute critical test plans just before product launch, resulting in much of the corrective work being performed in a rush and at the last minute.

Presenting combinatorial approaches for improving test coverage, Testing Complex and Embedded Systems details techniques to help you streamline testing and identify problems before they occur—including turbocharged testing using Six Sigma and exploratory testing methods. Rather than present the continuum of testing for particular products or design attributes, the text focuses on boundary conditions. Examining systems and software testing, it explains how to use simulation and emulation to complement testing.

• Details how to manage multiple test hardware and software deliveries
• Examines the contradictory perspectives of testing—including ordered/ random, structured /unstructured, bench/field, and repeatable/non repeatable
• Covers essential planning activities prior to testing, how to scope the work, and how to reach a successful conclusion
• Explains how to determine when testing is complete

Where you find organizations that are successful at product development, you are likely to find groups that practice disciplined, strategic, and thorough testing. Tapping into the authors' decades of experience managing test groups in the automotive industry, this book provides the understanding to help ensure your organization joins the likes of these groups.

Chapter 1

Does Your Testing Look Like This?

1.1 Last-Minute Flailing

Some enterprises regard testing as the final piece of the development effort rather than as a competitive tool. As a consequence, the test team executes critical test plans as the project is closing—just before launch and well after the time when the design and development teams can perform any kind of rational corrective actions. This behavior affords little time for the test engineers to understand the product at a sufficiently detailed level to perform useful testing. Testing groups may adopt a fatalistic approach to their craft by realizing that dilatory sample delivery and schedule crashing is part of their destiny or possibly the result of truly incompetent planning. The situation is aggravated by disengagement between the development and verification groups, as well as a frequent disconnect between project management, development engineering, and test engineering. The seriousness of this situation is illustrated when you hear project managers lament the test group “blowing” the schedule when they refuse the request of testing the system before the constituent system components are even available for test.

We owe it to our customers to provide them with a high-quality, reasonably priced, on-schedule, and safe products. Test engineering is a huge driver for achieving this goal because it is through testing that we reveal the character of our product. If we are professional enough and careful enough, we can cautiously predict the general quality of the product we are about to sell. Intelligent product testing should eliminate surprises in the field.

Intelligent testing means we are not waiting until the eleventh hour to make an acquaintance with our product. The kinds of last-minute flailing we have seen reflect a kind of cavalier attitude toward the quality of the product that never ceases to arrive as a surprise. In our own positions, we have done whatever we can to implement intelligent and complete testing.

Figure 1.1 Typical testing fiascos.

1.2 Fiascos Uncovered Weeks before Launch

Critical dependencies in the product development process often climax during testing—generally at the end of the project. Embedded software often takes a long time to mature; meanwhile, the delivery date rarely changes. We have seen the “we can test late” syndrome in several industries. So, the testing is either much more abbreviated than originally planned or continues after the first products are shipped. Then, the problems are already shipping to the customer. Numerous problems (see Figure 1.1) in the early product delivered to the field can contaminate the customer’s perspective of the product indefinitely.

While an organization may think it is saving money by delivering a product on time with minimal testing, the reality is that a poor-quality product may have a long-term impact on the company’s ability to receive revenue and improve margins with the product. Additionally, it will be expensive to fix field problems that are not contained in the plant. This will cost engineering time, sales and marketing time, and production time.

1.3 Huge Warranty Problems

Even when an organization tests a product to specification (the minimalist’s approach to testing), we generally expect to see subsequent field problems. Some products present difficulties when endeavoring to quantify the field exposure. This ranges from environmental exposure through customer use impacts on the quality. Not only can it be difficult to fully quantify the demands on a product up front, but there can be variation in the product itself and there certainly is variation in how the customer will use the product. These ingredients all add up to a nasty stew that, if taken to the tested-to-specification route, could be detrimental to the future of sales. If the warranty dollars for the product are greater than planned (see Figure 1.2), the profit margins on the product are less than planned. It can be a difficult sell to improve the product margins by increasing the product cost when the product quality is below customer expectations or less than originally touted by the supplying organization.

Figure 1.2 Warranty cost and project payback.

Testing to specification, especially if each line item of the specifications is performed independent of the other requirements, can present the delusion of successful testing. Suppose an electronic component output was tested to a certain voltage requirement at an ambient temperature of 25°C. Suppose then that the operating environment for the product has a range of −40°C to +85°C. Just because the component is able to deliver the desired current/voltage at the ambient temperature of 25°C does not mean it will be able to do so at the other temperatures (the upper level often being the most risky). The elevated temperatures on electronic equipment require derating after understanding the design work. If we do not follow this level of diligence, it will be largely accidental that the product will be able to deliver the required load current or voltage without some negative consequence upon the product. If this combination is not confirmed, it will be found in the field when we have the customer experience this set of stimuli.

Figure 1.3 Customer dissatisfaction.

1.4 Customer Dissatisfaction

Customer satisfaction is generated, at least in part, by the quality of the product. There are a number of other aspects, but we are discussing testing and how that drives product quality. It is difficult to gauge the impact of customer dissatisfaction in relation to a mediocre product. If our organization is the only supplying organization for the product, then we can still make some progress with the customer relationship for some period, an interval that is diminishing as the customer feels increasingly slighted by the inferior product. If the market size is large enough for the product, eventually some other manufacturers may come to believe they can make a better product. We can then expect our product volumes to decrease as these new, more reliable entries to the market take away our customers with largely the same feature set. Figure 1.3 depicts the process of increasing customer dissatisfaction as well as the shift to another supplier.

Chapter 2

Benefits of Improved Testing

We once heard a product development neophyte say, “Testing is only checking to see if you developed things correctly.” While there is some truth to this statement, the reality is that the possibility of getting everything (every task, every interpretation, and every deliverable item that could impact quality) correct in the development of a product is unlikely at best. There are hundreds and thousands of interactions and interpretations required. With transnational enterprises, the work is divided up all over the globe, with conflicting responsibilities, perspectives, and authorities. Additionally, decisions or interpretations are made with limited time and information. It really is not possible to get all these exchanges perfect every time.

Consider testing as a control system (see Figure 2.1). The system is designed to control the results (output) based upon constraints and capabilities of the organization as well as other factors such as risk. This is no different than any other system or activity where we want to control the outcome or at least be able to predict the results.

Making decisions about the economic value of a project is difficult to measure, and adding unpredictable quality issues makes this even more difficult. Whether the calculation is internal rate of return (IRR), net present value (NPV), or return on investment (ROI), the expected economic benefit to the organization is offset by the economic loss of maintaining the product (see Figure 2.2). If this product maintenance carries the additional burden of poor quality, the expected profit may vanish. This loss can be severe if the product has severe defects that harm the customer, initiating legal actions.

Figure 2.1 Testing as a control system.

2.1 Product Problems Revealed Early

Unlike the aforementioned scenario where testing is the last activity in a long line of activities, testing should be integrated into the development effort. The reason is that we want corrective actions that are made by developers as well as feedback from the testing and verification group. This does not absolve the developer from performing an appropriate level of diligence. Figure 2.3 shows the process for the developer’s level of verification of the product.

Figure 2.2 Benefits of improved testing.

Figure 2.3 Development and verification.

Developer testing typically uses the development tools to verify. In-circuit emulators (ICEs) are used to verify the performance of the software m...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• List of Figures
• List of Tables
• Preface
• Acknowledgment
• About the Authors
• 1 Does Your Testing Look Like This?
• 2 Benefits of Improved Testing
• 3 Overview
• 4 Basic Principles
• 5 The Question
• 6 Contradictory Perspectives of Testing
• 7 The Use of Noise
• 8 How to Perform “Bad” Tests
• 9 Documenting the Testing
• 10 Test Administration
• 11 Advanced Concepts
• 12 Software Test Documentation
• 13 Configuration Management
• 14 Software Testing
• 15 Systems Testing
• 16 Simulation and Emulation
• 17 Span of Tests
• 18 Exit Criteria
• Bibliography
• Index