Fundamentals of Software Testing

Bernard Homès

• Publisher: Wiley-ISTE
• Year: 2013
• ISBN:
  9781118603093
• Edition: 1
• Topic: Informatique
• Subtopic: Assurance qualité et tests

The testing market is growing at a fast pace and ISTQB certifications are being increasingly requested, with more than 180, 000 persons currently certified throughout the world. The ISTQB Foundations level syllabus was updated in 2011, and this book provides detailed course study material including a glossary and sample questions to help adequately prepare for the certification exam.
The fundamental aspects of testing are approached, as is testing in the lifecycles from Waterfall to Agile and iterative lifecycles. Static testing, such as reviews and static analysis, and their benefits are examined as well as techniques such as Equivalence Partitioning, Boundary Value Analysis, Decision Table Testing, State Transitions and use cases, along with selected white box testing techniques. Test management, test progress monitoring, risk analysis and incident management are covered, as are the methods for successfully introducing tools in an organization.

Contents

1. Fundamentals of Testing.
2. Testing Throughout the Software Life Cycle.
3. Static Techniques (FL 3.0).
4. Test Design Techniques (FL 4.0).
5. Test Management (FL 5.0).
6. Tools support for Testing (FL 6.0).
7. Mock Exam.
8. Templates and Models.
9. Answers to the Questions.

Chapter 1

Fundamentals of Testing

1.1. Why is testing necessary? (FL1.1)

FLO-1.1.1. Describe, with examples, the way in which a defect in software can cause harm to a person, to the environment, or to a company (K2)

In our everyday life, we are more and more dependent on the correct execution of software, whether it is in our equipment (cell phones, engine injection, etc.), in the transactions we undertake each day (credit or debit card purchases, fund transfers, Internet usage, electronic mail, etc.), or even those that are hidden from view (back office software for transaction processing), software simplifies our daily lives. When it goes awry, the impact can be devastating.

1.1.1. Software systems context

Testing software and systems is necessary to avoid failures visible to customers and avoid bad publicity for the organizations involved. This is the case for service companies responsible for the development or testing of third-party software, because the customer might not renew the contract, or might sue for damages.

We can imagine how millions of Germans felt on January 1^st, 2010, when their credit cards failed to work properly. No early warning sign informed them, and they found themselves, the day after New Year celebrations, with an empty fridge, totally destitute, without the possibility of withdrawing cash from ATMs or purchasing anything from retail outlets. Those most pitied were probably those who took advantage of the holiday period to go abroad; they did not even have the possibility to go to their bank to withdraw cash.

On November 20^th, 2009, during its first week of commercial operation on the Paris to New York route, the autopilot function of the Airbus A380, the pride of the Air France fleet, suffered a software failure such that it was forced to return to New York. The passengers were dispatched to other flights. Such a software problem could have been a lot more serious.

Software problems can also have an impact on an individual’s rights and freedom, be it in the USA, where voting machines failed during the presidential elections, preventing a large number of votes from being included [KER 04], or in France where, during local elections in 2008, a candidate from the Green party obtained 1,765 votes from 17,656 registered voters, and the software from the Ministry of Interior allowed the person to sit in the next stage of the election as the 10% threshold was reached. However, the software did not compute three digits after the decimal point and an “unfortunate rounding error to 10% was computed while the candidate only had 9.998% of the registered voters”. The end result was that the candidate was not allowed to participate in the next stage of the election. [ELE].

Software problems are not limited to small inconveniences, such as those listed above. They can be the root cause of accidents and even fatalities. This happened with the radiotherapy system Therac-25 [LEV 93, pp. 18-41], which led to six accidental releases of massive overdoses of radiation between 1985 and 1987, leading to the death of three patients. In the case of Therac-25, the root cause of the software failures — and of the death of the patients — were determined as being:

– a lack of code reviews by independent personnel;

– software design methods that were not adapted and thus incorrectly implemented for safety critical systems;

– lack of awareness regarding system reliability for evaluation of software defects;

– unclear error messages and usability problems in the software;

– a lack of full acceptance tests for the complete system (hardware and software).

Other examples of software failures that have caused major incidents have occurred in the space industry, such as:

– the first flight of the Ariane 5 launcher, where a component that was developed and used reliably on the Ariane 4 launchers was used outside its normal operational context and led to the loss of the launcher and all the satellites it carried;

– NASA’s (National Aeronautics and Space Administration) Mars Climate Orbiter mission, where a unit conversion problem, between the units used by the European Space Agency (ESA; using metric-based units) and the units used by NASA (nautical mile) led to the loss of the spaceship and the full mission;

– NASA’s Mars Polar Lander, where a speck of dust led to an incorrect response from one of the three landing gear, and a lack of software testing led to the shutdown of the probe’s engine some 40 meters above the surface, leading to the loss of the probe and the mission.

These three examples each cost hundreds of millions of Euros and US dollars, even with the high level of quality and tests done on such systems. Every year software failures generate financial losses evaluated to be hundreds of millions of Euros. Correct testing of software is necessary to avoid frustration, lost financial expenditure, damages to property, or even death; all this due to failures in software.

1.1.2. Causes of software defects

FLO-1.1.2 Distinguish between the root cause of a defect and its effects (K2)

There is a causality link between errors and defects, and between defects and failures generated. The initial cause — the root cause — of defects is often found to be caused by the actions (or lack of action) of humans:

– misunderstanding of the specifications by functional analysts, resulting in a software design or architecture that prevents the expected goals from being reached or objectives stated by the customers;

– mistakes, such as replacing a greater than sign by a greater than or equal to sign, resulting in abnormal behavior when both variables are equal.

Some failures are not directly caused by human action, but are caused by the interactions between the test object and its environment:

– software malfunctions when electronic components overheat abnormally due to dust;

– electrical or electronic interferences produced by power cables near unshielded data cables;

– solar storms or other activities generating ionizing radiation that impacts on electronic components (this is important for satellite and airborne equipment);

– impact of magnets or electromagnetic fields on data storage devices (magnetic disks or tapes, etc.).

FLO-1.1.5 Explain and compare the terms error, defect, fault, failure, and the corresponding terms mistake and bug, using examples (K2)

Many terms describe the incorrect behavior of software: bug, error, failure, defect, fault, mistake, etc. These terms are sometimes considered as equivalent, which may generate misunderstandings. In this book, just as for the International Software Testing Qualifications Board (ISTQB), we will use the following terms and definitions:

– error: human action at the root of a defect;

– defect: result, present in the test object, of a human action (i.e. error);

– failure: result from the execution of a defect by a process (whether the process is automated or not).

These terminological differences are important and will result in different activities to limit their occurrence and/or impact.

In order to reduce human errors — and thus the number of defects introduced in the software — training activities can be implemented, or more strict processes can be set in place. Tests are executed to identify the failures, by displaying abnormal behavior. Using the information provided by testers, designers can identify and remove defects that cause incorrect behavior. The software defects can be identified by submitting the software to reviews (code or architecture reviews, etc.), or by executing the software and identifying failures that result from the presence of defects.

NOTE: Defects may be located in software, but can also be present in documents. A large number of software problems are caused by requirements or specifications that may be ambiguous, or even incoherent or incompatible. The error is thus made by those who write these requirements, the defect in the specification, and in the code, before the failure is identified during test execution.

FLO-1.1.3 Give reasons why testing is necessary by giving examples (K2)

Our software and systems become more and more complex, and we rely more and more on their faultless operation. Our cell phones and personal digital assistants (PDAs) are more powerful than the mainframes of 30 years ago, simultaneously integrating agenda, notepad, and calendar functions, plus global positioning systems (GPSs), cameras, emails, instant messaging, games, voice recorders, music and video players, etc., not forgetting the telephone functionalities of course. Vehicles are equipped with more and more electronic circuits and data processing systems (ESP (trajectory control for vehicles, anti-skid), GPS, fuel injection, airbags, course control, cruise control, etc.), our cell phones connect automatically (via Bluetooth) to our vehicles and its audio system. We only need a small software problem and our vehicle or our cell phone becomes unusable.

We also rely on other software, such as those in our credit or debit cards, where a defect can directly impact millions of users [LEM 10], such as occurred in early 2010 where German users were victims of a major failure for over a week. We have also seen exploding virtual high-speed trains [LEP 10] (without actual victims) or the availability of customer data for rail companies available on ...