How to Reduce the Cost of Software Testing

Matthew Heusser, Govind Kulkarni, Matthew Heusser, Govind Kulkarni

• Publisher: Auerbach Publications
• Year: 2018
• Print ISBN: 9781439861554
• eBook ISBN: 9781351689083
• Topic: Business
• Subtopic: Project Management
• Index: Business

Plenty of software testing books tell you how to test well; this one tells you how to do it while decreasing your testing budget. A series of essays written by some of the leading minds in software testing, How to Reduce the Cost of Software Testing provides tips, tactics, and techniques to help readers accelerate the testing process, improve the performance of the test teams, and lower costs.

The distinguished team of contributors—that includes corporate test leaders, best paper authors, and keynote speakers from leading software testing conferences—supply concrete suggestions on how to find cost savings without sacrificing outcome. Detailing strategies that testers can immediately put to use to reduce costs, the book explains how to make testing nimble, how to remove bottlenecks in the testing process, and how to locate and track defects efficiently and effectively.

Written in language accessible to non-technical executives, as well as those doing the testing, the book considers the latest advances in test automation, ideology, and technology. Rather than present the perspective of one or two experts in software testing, it supplies the wide-ranging perspectives of a team of experts to help ensure your team can deliver a completed test cycle in less time, with more confidence, and reduced costs.

Part 1

What Will This Cost Us?

1

Is This the Right Question?

Matt Heusser

This book starts with the question “How do we decrease the cost of testing?” In this chapter, I’ll push back a little on that question, asking if it is, in fact, the right one—and putting it in its place. After all, one rational answer to the question on how to reduce the cost of testing is the following:

The easiest way to make testing cheap is to do a quick, slapdash job. Easier than that, don’t test at all. Your testing costs would go down, but overall costs would go up. And customers? Those would just go away.

—Anonymous

1.1 CONTROLLING COSTS: WHAT DOES IT MEAN?

Say, for a moment, that you are a test director and successfully implementing the ideas in this book. That means you can do the same work with half the test or quality assurance (QA) staff. Assuming the work coming in from development stays constant, what happens, what is the result, the outcome? That’s right. “Success” at lowering test costs might just mean having the dubious honor of laying off half your staff.

This sort of thing has happened before. Eli Goldratt saw it happen in the twentieth century in manufacturing. According to Goldratt, one early consequence of his “theory of constraints” was massive increases in throughput for American factories. This actually turned out to be a problem, because the companies now had more supply than demand, so they began shutting down factories due to overcapacity. Goldratt documents these sorts of system effects in the second edition of his book The Goal: A Process of Ongoing Improvement.

Laying off half the staff might lead to a bonus or promotion, but it doesn’t sit well with me; it doesn’t serve the community we work in. No, I want to look at something beyond pure cost. In general business we look at cost and sales (or the “top line” numbers); it may be acceptable to double our costs if sales quadruple. We have nothing like that top line in testing. There is no easy metric for it.

Instead of value, we focus on cost. And again, we know where that ends—success means the unemployment line.

I do agree that brining down testing costs in general is a thing, but is is only part of the equation. In this chapter I will lay some ground rules for evaluating cost, while also including value within the discussion. Without looking at value, we’re back to cutting testing completely.

So first, let’s look at what limiting, containing, or decreasing cost means, exactly.

1.2 LOOKING DEEPER INTO LIMITING COSTS

Before we discuss value, let’s take a moment and look at limiting costs. After all, it’s straightforward. It’s easy. It’s simple. What can be wrong with that? Consider for a moment a typical cost-cutting strategy. We could, for example, cut benefits, bonuses, and salaries for the workers. Over time, this tends to decrease morale, increase turnover, and decrease productivity. When salaries go down, the best and brightest leave for greener pastures—leaving the company with, to paraphrase Ronald Padavan, “the dumbest and the stupidest.” I hope you’ll agree that’s not a useful long-term strategy [1].

The next level of cutting costs is a tad more refined. Instead of cutting benefits or laying off the highly paid staff, we can instead try to break the work down into extremely simple, well-defined component tasks. Then we can hire people for specific, repeatable, well-described jobs [2]. For example, we can separate our front office from the back office, then hire a low-paid “call center” or “help desk” person to manage the queue of incoming work. That allows our more experienced, more well-paid technicians to stay focused on one problem at a time without interruption.

But there’s a problem. Psychologist and systems thinker John Seddon labels the problem “failure demand.” As Seddon explains it, there are two types of demand in the system: regular demand, where a customer wants something; and failure demand, where a customer comes back a second, third, fourth, and fifth time to ask again for a solution because the initial “solution” did not meet the initial demand.

If you’ve ever seen a call tracking ticket bounced around a dozen, two dozen, or three dozen times without resolving the issue, you’ve seen failure demand. And you know that failure demand can be more expensive than the original request in the first place. Seddon goes on to document that this separation of front office from back office and movement of jobs, often to low-cost-of-living, low-wage areas, creates enough failure demand to remove any cost savings. Seddon uses one phrase to describe all this. In his words, “When you try to control costs, costs go up” [3]. We need a better way.

1.3 WAYS TO LOOK AT VALUE

We want to report something to management to get that “exceeds expectations” on the annual review, and we want it to be a number, right? Value in testing defies simple numeric measurement. This is not a new problem. The great W. Edwards Deming, guru of total quality management, used a quote from Lloyd Nelson to explain this: “The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them” [4].

The classic example Deming gives is training. If you simply look at numbers on a spreadsheet, training has an immediate cost without clear value. Of course, you could make up numbers in a complex spreadsheet about turnover, or productivity, but those would just be an estimate, a guess, wishful thinking. Likewise, if you only look at a spreadsheet, it makes sense to only hire part-time employees (to save the cost of insurance and other benefits), or to cut benefits and retirement, or to fire the entire staff and rehire cheaper people. Yet we know from the previous section that those strategies fail. So we have a paradox: We have to manage to value, yet value defies numerical measurement.

Managers have been struggling with measuring value for centuries. Even measurements as deceptively simple as cold hard cash in the bank can be manipulated. One sales technique that creates sales right now, without creating real long-term value is known as “channel stuffing.” When a manufacturer takes a channel stuffing approach, they convince customers (usually a retail store) to buy many more products than they actually need for a given time period, then allow the other company to return the extra equipment by a certain day. This allows sales executives to “make” monthly or quarterly sales targets without having to do the work of, you know, actually selling more things. This also causes the manufacturer to pay out large bonuses without “real” sales attached. Gil Amelio, former CEO of Apple Computer, claims that channel stuffing was one of the practices that led to the near-destruction of Apple in the 1990s [5].

Another example currently playing out in the courts is Lehman Brothers. According to the U.S. Securities and Exchange Commission, Lehman Brothers manipulated its balance sheet by taking on loans right before its quarterly report, thus creating the appearance of more cash on hand. And then there is Enron, WorldCom, or a host of other frauds.

So much for cold hard cash as an absolute metric. But who said we need metrics, anyway? The most common line used as an argument in favor of metrics is from Tom DeMarco, in his book Controlling Software Projects [6]. The actual line from that book was not “without measurement you can not manage,” but instead “without measurement you can not control.” DeMarco he spent the rest of his career pointing out that control in software projects isn’t all that important. Twenty years after that original article, he published an article in IEEE Spectrum that formally refuted the importance of controlling software projects [7]. In his words:

My early metrics book, “Controlling Software Projects: Management, Measurement, and Estimation,” played a role in the way many budding software engineers quantified work and planned their projects. In my reflective mood, I’m wondering, was its advice correct at the time, is it still relevant, and do I still believe that metrics are a must for any software development effort? My answers are no, no, and no.

Think about it: You manage to keep your hair looking good without taking a ruler to it every day, right? This can be done by looking at it, understand what is happening, and paying attention. We call this qualitative evaluation. You may not agree with me and I can understand that. I hope you will at least suspend your disbelief long enough to finish this chapter. Let me talk about other tools for your toolbox, to see if they might be helpful. So let’s forget about this metrics madness for a bit and instead talk about what actually happens while doing testing, in order to understand. Once we understand testing a bit better, perhaps then we can generalize some rules for management.

1.4 THE REALITY OF TESTS AND TEST CASES

When I talk to managers and executives I get this vague general impression that there is some correct set of test cases. It seems to be kind of like the holy grail or the North American Big Foot, except that many leaders think they have captured it.

The thinking goes that if the team had this set of correct test cases, we could separate test design from test execution, and pay the test designer more wages as the one “ just executing” the tests. Of course that whole idea would send Seddon into convulsions; but, I am sad to say, it doesn’t even work.

First, for any given piece of software, there may be an infinite number of possible test cases. If you give the requirement to two test engineers and ask them to create test cases, you’d likely get two entirely different sets of documents. Each would address the risks that the author perceives. Likewise, test cases tend to try to predict errors. Consider the simple web application that converts degrees Fahrenheit to Celsius and back, illustrated in Figure 1.1 [8]. A classic approach to this problem would break the functionality into equivalence classes and boundaries. It might look something like this:

Celsius Value	Expected Fahrenheit Value
0	32
37	100
100	212
One	ERROR
(blank)	(blank)

We could make quite a list, and it might be a good one.

Framed this way, the tests have a flaw. They assume the software will be used by something like a computer, with a specific input and output. This frames the problem like a function in computer programming. But applications are not consumed by software, they are used by humans. And the human has many more ways to interact with the software than simplistic input–output.

Enter a number in a field below to convert it to either Fahrenheit or Celsius:

FIGURE 1.1
Matthew Heusser’s host of Martin Carlsson’s F to C convers...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Foreword
• Preface
• Acknowledgments
• Authors
• Part 1 What Will This Cost Us?
• Part 2 What Should We Do?
• Part 3 How Should We Do It?
• Afterword
• Appendix A: Immediate Strategies to Reduce Test Cost
• Appendix B: 25 Tips to Reduce Testing Cost Today
• Appendix C: Is It about Cost or Value?
• Appendix D: Cost of Starting Up a Test Team
• Index