Managing the Testing Process

Practical Tools and Techniques for Managing Hardware and Software Testing

Rex Black

• Publisher: Wiley
• Year: 2011
• Print ISBN: 9780470404157
• eBook ISBN: 9781118074015
• Edition: 3
• Topic: Computer Science
• Subtopic: Software Development
• Index: Computer Science

New edition of one of the most influential books on managing software and hardware testing

In this new edition of his top-selling book, Rex Black walks you through the steps necessary to manage rigorous testing programs of hardware and software. The preeminent expert in his field, Mr. Black draws upon years of experience as president of both the International and American Software Testing Qualifications boards to offer this extensive resource of all the standards, methods, and tools you'll need.

The book covers core testing concepts and thoroughly examines the best test management practices and tools of leading hardware and software vendors. Step-by-step guidelines and real-world scenarios help you follow all necessary processes and avoid mistakes.

• Producing high-quality computer hardware and software requires careful, professional testing; Managing the Testing Process, Third Edition explains how to achieve that by following a disciplined set of carefully managed and monitored practices and processes
• The book covers all standards, methods, and tools you need for projects large and small
• Presents the business case for testing products and reviews the author's latest test assessments
• Topics include agile testing methods, risk-based testing, IEEE standards, ISTQB certification, distributed and outsourced testing, and more
• Over 100 pages of new material and case studies have been added to this new edition

If you're responsible for managing testing in the real world, Managing the Testing Process, Third Edition is the valuable reference and guide you need.

Chapter 1
Defining What's on Your Plate: The Foundation of a Test Project

Testing requires a tight focus. It's easy to try to do too much. You could run an infinite number of tests against any nontrivial piece of software or hardware. Even if you try to focus on what you think might be “good enough” quality, you can find that such testing is too expensive or that you have trouble figuring out what “good enough” means for your customers and users. Before I start to develop the test system—the testware, the test environment, and the test process—and before I hire the test team, I figure out what I might test, then what I should test, and finally what I can test. Determining the answers to these questions helps me plan and focus my test efforts.

What I might test are all those untested areas that fall within the purview of my test organization. On every project in which I've been involved, some amount of the test effort fell to organizations outside my area of responsibility. Testing an area that another group already covered adds little value, wastes time and money, and can create political problems for you.

What I should test are those untested areas that directly affect the customers' and users' experience of quality. People often use buggy software and computers and remain satisfied nevertheless. Either they never encounter the bugs or the bugs don't significantly hinder their work. Our test efforts should focus on finding the critical defects that will limit people's ability to get work done with our products.

What I can test are those untested, critical areas on which my limited resources are best spent. Can I test everything I should? Not likely, given the schedule and budget I usually have available.¹ On most projects, I must make tough choices, using limited information, on a tight schedule. I also need to sell the test project to my managers to get the resources and the time I need.

What You Might Test: The Extended Test Effort

On my favorite software and system projects, testing was pervasive. By this, I mean that a lot of testing went on outside the independent test team. In addition, testing started early. This arrangement not only made sense technically, but also kept my team's workload manageable. This section uses two lenses to examine how groups outside the formal test organization contribute to testing. The first lens is the level of focus—the granularity—of a test. The second is the type of testing performed within various test phases. Perhaps other organizations within your company could be (or are) helping you test.

From Microscope to Telescope: Test Granularity

Test granularity refers to the fineness or coarseness of a test's focus. A fine-grained test case allows the tester to check low-level details, often internal to the system. A coarse-grained test case provides the tester with information about general system behavior. You can think of test granularity as running along a spectrum ranging from structural (white-box) to behavioral (black-box and live) tests, as shown in Figure 1-1.

Figure 1-1 The test granularity spectrum and owners

Structural (White-Box) Tests

Structural tests (also known as white-box tests and glass-box tests) find bugs in low-level structural elements such as lines of code, database schemas, chips, subassemblies, and interfaces. The tester bases structural tests on how a system operates. For example, a structural test might reveal that the database that stores user preferences has space to store an 80-character username, but that the field allows the user to enter only 40 characters.

Structural testing involves a detailed technical knowledge of the system. For software, testers create structural tests by looking at the code and the data structures themselves. For hardware, testers create structural tests to compare chip specifications to readings on oscilloscopes or voltage meters. Structural tests thus fit well in the development area. Testers in an independent test team—who often have little exposure to low-level details and might lack programming or engineering skills—find it difficult to perform structural testing.

Structural tests also involve knowledge of structural testing techniques. Not all programmers learn these techniques as part of their initial education and ongoing skills growth. In such cases, having a member of the test team work with the programmers as a subject-matter expert can promote good structural testing. This person can help train the programmers in the techniques needed to find bugs at a structural level.

Behavioral (Black-Box) Tests

Testers use behavioral tests (also known as black-box tests) to find bugs in high-level operations, such as major features, operational profiles, and customer scenarios. Testers can create black-box functional tests based on what a system should do. For example, if SpeedyWriter should include a feature that saves files in XML format, then you should test whether it does so. Testers can also create black-box non-functional tests based on how a system should do what it does. For example, if DataRocket can achieve an effective throughput of only 10 Mbps across two 1-gigabit Ethernet connections acting as a bridge, a black-box network-performance test can find this bug.

Behavioral testing involves a detailed understanding of the application domain, the business problem that the system solves, and the mission the system serves. When testers understand the design of the system, at least at a high level, they can augment their behavioral tests to effectively find bugs common to that type of design. For example, programs implemented in languages like C and C++ can—depending on the programmers' diligence—suffer from serious security bugs related to buffer overflows.

In addition to the application domain and some of the technological issues surrounding the system under test, behavioral testers must understand the special behavioral test techniques that are most effective at finding such bugs. While some behavioral tests look at typical user scenarios, many tests exercise extremes, interfaces, boundaries, and error conditions. Bugs thrive in such boundaries, and behavioral testing involves searching for defects, just as structural testing does. Good behavioral testers use scripts, requirements, documentation, and testing skills to guide them to these bugs. Simply playing around with the system or demonstrating that the system works under average conditions are not effective techniques for behavioral testing, although many test teams make the mistake of adopting these as the sole test techniques. Good behavioral tests, like good structural tests, are structured, methodical, and often repeatable sequences of tester-created conditions that probe suspected system weaknesses and strive to find bugs, but through the external interfaces of the system under test. Most independent test organizations perform primarily behavioral testing.

Live Tests

Live tests involve putting customers, content experts, early adopters, and other end users in front of the system. In some cases, we encourage the testers to try to break the system. Beta testing is a well-known form of bug-driven live testing. For example, if the SpeedyWriter product has certain configuration-specific bugs, live testing might be the best way to catch those bugs specific to unusual or obscure configurations. In other cases, the testers try to demonstrate conformance to requirements, as in acceptance testing, another common form of live testing.

Live tests can follow general scripts or checklists, but live tests are often ad hoc (worst case) or exploratory (best case). They don't focus on system weaknesses except for the “error guessing” that comes from experience. Live testing is a perfect fit for technical support, marketing, and sales organizations whose members don't know formal test techniques but do know the application domain and the product intimately. This understanding, along with recollections of the nasty bugs that have bitten them before, allows them to find bugs that developers and testers miss.

The Complementary and Continuous Nature of Test Granularity

The crew of a fishing boat uses a tight-mesh net to catch 18-inch salmon and a loose-mesh net to catch six-foot tuna. They might be able to catch a tuna in a salmon net or vice versa, but it would probably make them less efficient. Likewise, structural, behavioral, and live tests each are most effective at finding certain types of bugs. Many great test efforts include a mix of all three types.

While my test teams focus on behavioral testing typically, I don't feel bound to declare my test group “the black-box bunch.” I've frequently used structural test tools and cases effectively as part of my system test efforts. I've also used live production data in system testing. Both required advanced planning, but paid off handsomely in terms of efficiency (saved time and effort) and effectiveness (bugs found that we might have missed). Test granularity is a spectrum, not an either/or categorization. Mixing these elements can be useful in creating test conditions or assessing results. I also mix planned test scenarios with exploratory live testing. I use whatever works.

A Stampede or a March? Test Phases

The period of test execution activity during development or maintenance is sometimes an undifferentiated blob. Testing begins, testers run some (vaguely defined) tests and identify some bugs, and then, at some point, project management declares testing complete. As development and maintenance processes mature, however, companies tend to...

Table of contents

• Cover
• Table of Contents
• Title
• Copyright
• About the Author
• Credits
• Acknowledgments
• Introduction
• Chapter 1: Defining What's on Your Plate: The Foundation of a Test Project
• Chapter 2: Plotting and Presenting Your Course: The Test Plan
• Chapter 3: Test System Architecture, Cases, and Coverage
• Chapter 4: An Exciting Career in Entomology Awaits You: A Bug-Tracking Database
• Chapter 5: Managing Test Cases: The Test Tracking Spreadsheet
• Chapter 6: Tips and Tools for Crunch Mode: Managing the Dynamic
• Chapter 7: Stocking and Managing a Test Lab
• Chapter 8: Staffing and Managing a Test Team
• Chapter 9: The Triumph of Politics: Organizational Challenges for Test Managers
• Chapter 10: Involving Other Players: Distributed Testing, Outsourcing, and Related Topics
• Chapter 11: Economics of Testing: Fiscal Context
• Chapter 12: Testing Implications of Project and Process: Situational Context
• Appendices
• Glossary
• Index
• End User License Agreement