Software Testing

An ISTQB-BCS Certified Tester Foundation guide

Brian Hambling, Peter Morgan, Angelina Samaroo, Geoff Thompson, Peter Williams, Brian Hambling

• Publisher: BCS, The Chartered Institute for IT
• Year: 2015
• ISBN:
  9781780173016
• Edition: 3
• Topic: Informatik
• Subtopic: Qualitätssicherung & Prüfung

• Best-selling software testing title
• The only official textbook of the ISTQB-BCS Certified Tester Foundation Level
• Updated with examples and exercises reflecting current technology and applications
• ISBN of previous edition - 9781906124762
  9781906124762 9781906124762 9781906124762 9781906124762

1 THE FUNDAMENTALS OF TESTING Peter Morgan

BACKGROUND

If you were buying a new car, you would not expect to take delivery from the showroom with a scratch down the side of the vehicle. The car should have five wheels, a steering wheel, an engine and all the other essential components, and it should come with appropriate documentation, with all pre-sales checks completed and passed satisfactorily. The car you receive should be the car described in the sales literature; it should have the correct engine size, the correct colour scheme, and whatever extras you have ordered, and performance in areas such as fuel consumption and maximum speed should match published figures. In short, a level of expectation is set by brochures, by your experience of sitting in the driving seat, and probably by a test drive. If your expectations are not met you will feel justifiably aggrieved.

This kind of expectation seems not to apply to new software installations; examples of software being delivered not working as expected, or not working at all, are common. Why is this? There is no single cause that can be rectified to solve the problem, but one important contributing factor is the inadequacy of the testing to which software applications are exposed.

Software testing is neither complex nor difficult to implement, yet it is a discipline that is seldom applied with anything approaching the necessary rigour to provide confidence in delivered software. Software testing is costly in human effort or in the technology that can multiply the effect of human effort, yet is seldom implemented at a level that will provide any assurance that software will operate effectively, efficiently or even correctly.

This book explores the fundamentals of this important but neglected discipline to provide a basis on which a practical and cost-effective software testing regime can be constructed.

INTRODUCTION

In this opening chapter we have three very important objectives to achieve. First, we will introduce you to the fundamental ideas that underpin the discipline of software testing, and this will involve the use and explanation of some new terminology. Secondly, we will establish the structure that we have used throughout the book to help you to use the book as a learning and revision aid. Thirdly, we will use this chapter to point forward to the content of later chapters.

The key ideas of software testing are applicable irrespective of the software involved and any particular development methodology (waterfall, Agile etc.). Software development methodologies are discussed in detail in Chapter 2.

We begin by defining what we expect you to get from reading this chapter. The learning objectives below are based on those defined in the Software Foundation Certificate syllabus, so you need to ensure that you have achieved all of these objectives before attempting the examination.

Learning objectives

The learning objectives for this chapter are listed below. You can confirm that you have achieved these by using the self-assessment questions at the start of the chapter, the ‘Check of understanding’ boxes distributed throughout the text, and the example examination questions provided at the end of the chapter. The chapter summary will remind you of the key ideas.

The sections are allocated a K number to represent the level of understanding required for that section; where an individual topic has a lower K number than the section as a whole, this is indicated for that topic; for an explanation of the K numbers see the Introduction.

Why is testing necessary? (K2)

• Describe, with examples, the way in which a defect in software can cause harm to a person, to the environment or to a company.
• Distinguish between the root cause of a defect and its effects.
• Give reasons why testing is necessary by giving examples.
• Describe why testing is part of quality assurance and give examples of how testing contributes to higher quality.
• Recall the terms error, defect, fault, failure and the corresponding terms mistake and bug, using examples.

What is testing? (K2)

• Recall the common objectives of testing. (K1)
• Provide examples for the objectives of testing in different phases of the software life cycle.
• Differentiate testing from debugging.

General testing principles (K2)

• Explain the seven principles in testing.

Fundamental test process (K1)

• Recall the five fundamental test activities and respective tasks from planning to closure.

The psychology of testing (K2)

• Recall the psychological factors that influence the success of testing. (K1)
• Contrast the mindset of a tester and of a developer.

Self-assessment questions

The following questions have been designed to enable you to check your current level of understanding for the topics in this chapter. The answers are given at the end of the chapter.

Question SA1 (K1)

A non-functional requirement concerning performance is missing from the system documentation. How should this be described?

1. It is a mistake.
2. It is a failure.
3. It is a fault.
4. It is an error.

Question SA2 (K1)

Which of the following gives MOST independence in testing?

1. Testing performed by the person who wrote the code.
2. Testing performed by an external consultancy.
3. Testing performed by another member of the same Agile team.
4. Testing performed by a separate department.

Question SA3 (K1)

Running the same set of tests will not continue to find new defects. Which of the seven testing principles does this illustrate?

1. Defect clustering.
2. Testing shows the presence of bugs.
3. Absence of errors fallacy.
4. The pesticide paradox.

WHY SOFTWARE FAILS

Examples of software failure are depressingly common. Here are some you may recognise:

• After successful test flights and air worthiness accreditation, problems arose in the manufacture of the Airbus A380 aircraft. Assembly of the large subparts into the completed aircraft revealed enormous cabling and wiring problems. The wiring of large subparts could not be joined together. It has been estimated that the direct or indirect costs of rectification was $6.1 billion. (Note: this problem was quickly fixed and the aircraft entered into service within 18 months of the cabling difficulties being identified.)
• When the UK Government introduced online filing of tax returns, a user could sometimes see the amount that a previous user earned. This was regardless of the physical location of the two applicants.
• In November 2005, information on the UK’s top 10 wanted criminals was displayed on a website. The publication of this information was described in newspapers and on morning radio and television and, as a result, many people attempted to access the site. The performance of the website proved inadequate under this load and the website had to be taken offline. The publicity created performance peaks beyond the capacity of the website.
• A new smartphone mapping application (app) was introduced in September 2012. Among many other problems, a museum was incorrectly located in the middle of a river, and Sweden’s second city, Gothenburg, seemed to have disappeared from at least one map.
• A small, one-line, change in the billing system of an electrical provider blacked out the whole of a major US city.

What is it about these examples that make them so startling? Is it a sense that something fairly obvious was missed? Is it the feeling that, expensive and important as they were, the systems were allowed to enter service before they were ready? Do you think these systems were adequately tested? Obviously they were not, but in this book we want to explore why this was the case and why these kinds of failure continue to plague us.

To understand what is going on we need to start at the beginning, with the people who design systems. Do they make mistakes? Of course they do. People make mistakes because they are fallible, but there are also many pressures that make mistakes more likely. Pressures such as deadlines, complexity of systems and organisations, and changing technology all bear down on designers of systems and increase the likelihood of errors in specifications, in designs and in software code. These errors are where major system failures usually begin. If a document with an error in it is used to specify a component the component will be faulty and will probably exhibit incorrect behaviour. If this faulty component is built into a system the system may fail. While failure is not always guaranteed, it is likely that errors in specifications will lead to faulty components and faulty components will cause system failure.

An error (or mistake) leads to a defect, which can cause an observed failure (Figure 1.1).

Figure 1.1 Effect of an error

There are other reasons why systems fail. Environmental conditions such as the presence of radiation, magnetism, electronic fields or pollution can affect the operation of hardware and firmware and lead to system failure.

If we want to avoid failure we must either avoid errors and faults or find them and rectify them. Testing can contribute to both avoidance and rectification, as we will see when we have looked at the testing process in a little more detail. One thing is clear: if we wish to influence errors with testing we need to begin testing as soon as we begin making errors – right at the beginning of the de...