News and Numbers

A Writer's Guide to Statistics

Victor Cohn, Lewis Cope, Deborah Cohn Runkle

• Publisher: Wiley-Blackwell
• Year: 2011
• Print ISBN: 9781405160964, 9781444361889
• eBook ISBN: 9781444344349
• Edition: 3
• Topic: Languages & Linguistics
• Subtopic: Journalism
• Index: Languages & Linguistics

Numbers and statistical claims dominate today's news. Politics, budgets, crime analysis, medical issues, and sports reporting all demand numbers. Now in its third edition, News & Numbers focuses on how to evaluate statistical claims in science, health, medicine, and politics. It does so by helping readers answer three key questions about all scientific studies, polls, and other statistical claims: "What can I believe?" "What does it mean?" and "How can I explain it to others?"

Updated throughout, this long overdue third edition brings this classic text up-to-date with the 21st century with a complete updating of examples, case studies, and stories. The text emphasises clear thinking and common sense approaches for understanding, analyzing and explaining statistics, and terms throughout the book are explained in easy-to-understand, nontechnical language. Much new material has been added to ensure the text maintains its pertinent approach to the subject, including:

• A section on computer modelling
• Additional chapters on risks and 'missing numbers'
• Updated sections on health plans and insurance, including updates on President Obama's health system overhaul & new material on health care costs and quality

Part I

Learning the Basics

A Guide to Part I of News & Numbers

In the first five chapters, we cover the basics:

1. Where We Can Do Better

Improving how stories with numbers are reported.

2. The Certainty of Uncertainty

Scientists are always changing their minds.

3. Testing the Evidence

Thinking clearly about scientific studies.

4. What Makes a Good Study?

Separating the wheat from the chaff.

5. Your Questions and Peer Review

What to ask the experts.

1

Where We Can Do Better

Almost everyone has heard that “figures don’t lie, but liars can figure.” We need statistics, but liars give them a bad name, so to be able to tell the liars from the statisticians is crucial.

Dr. Robert Hook

A team of medical researchers reports that it has developed a promising, even exciting, new treatment. Is the claim justified, or could there be some other explanation for their patients’ improvement? Are there too few patients to justify any claim?

An environmentalist says that a certain toxic waste will cause many cases of cancer. An industry spokesman denies it. What research has been done? What are the numbers? How valid are they?

We watch the numbers fly in debates ranging from pupil-testing to global warming to the cost of health insurance reforms, and from influenza threats to shocking events such as the catastrophic Deepwater Horizon oil spill in the Gulf of Mexico in April 2010.

Even when we journalists say that we are dealing in facts and ideas, much of what we report is based on numbers. Politics comes down to votes. Dollar-figures dominate business and government news – and stir hot-button issues such as sports stadium proposals. Numbers are at the heart of crime rates, nutritional advice, unemployment reports, weather forecasts, and much more.

But numbers offered by experts sometimes conflict, triggering confusion and controversies. Statistics are used or misused even by people who tell us, “I don’t believe in statistics,” then claim that all of us, or most people, or many do such and such. We should not merely repeat such numbers, but interpret them to deliver the best possible picture of reality.

And the really good news: We can do this without any heavy-lifting math. We do need to learn how the best statisticians – the best figurers – think. They can show us how to detect possible biases and bugaboos in numbers. And they can teach us how to consider alternate explanations, so that we won’t be stuck with the obvious when the obvious is wrong.

Clear thinking is more important than any figuring. There’s only one math equation in this book: 1 = 200,000. This is our light-hearted way of expressing how one person in a poll can represent the views of up to 200,000 Americans. That is, when the poll is done right. The chapter on polling tells you how to know when things go wrong.

Although News & Numbers is written primarily to aid journalists in ferreting meaning out of numbers, this book can help anyone answer three questions about all sorts of studies and statistical claims:

What can I believe? What does it mean? How can I explain it to others?

The Journalistic Challenges

The very way in which we journalists tell our readers and viewers about a medical, environmental, or other controversy can affect the outcome.

If we ignore a bad situation, the public may suffer. If we write “danger,” the public may quake. If we write “no danger,” the public may be falsely reassured.

If we paint an experimental medical treatment too brightly, the public is given false hope. If we are overly critical of some drug that lots of people take, people may avoid a treatment that could help them, maybe even save their lives.

Simply using your noggin when you view the numbers can help you travel the middle road.

And whether we journalists will it or not, we have in effect become part of the regulatory apparatus. Dr. Peter Montague at Princeton University tells us: “The environmental and toxic situation is so complex, we can’t possibly have enough officials to monitor it. Reporters help officials decide where to focus their activity.”

And when to kick some responses into high gear. During the early days after the Deepwater Horizon oil well explosion, both company (BP) and federal officials tended to soft-pedal and underestimate the extent of the problem. Aggressive reporters found experts who sharply hiked the estimates of how much oil was pouring into the Gulf. These journalists provided running tallies of the miles of shorelines where gooey globs were coming ashore, the numbers of imperiled birds and wildlife, and the number of cleanup workers feeling ill effects. Nightly news broadcasts showed graphic video of oil gushing out of the ground a mile under the Gulf. National attention was soon galvanized on the crisis; both government and industry action intensified.

Five Areas for Improvement

As we reporters seek to make page one or the six o’clock news:

1. We sometimes overstate and oversimplify. We may report, “A study showed that black is white,” when a study merely suggested there was some evidence that such might be the case. We may slight or omit the fact that a scientist calls a result “preliminary,” rather than saying that it offers strong and convincing evidence.

Dr. Thomas Vogt, at the Kaiser Permanente Center for Health Research, tells of seeing the headline “Heart Attacks from Lack of ‘C’ ” and then, two months later, “People Who Take Vitamin C Increase Their Chances of a Heart Attack.”¹ Both stories were based on limited, far-from-conclusive animal studies.

Philip Meyer, veteran reporter and author of Precision Journalism, writes, “Journalists who misinterpret statistical data usually tend to err in the direction of over-interpretation … The reason for this professional bias is self-evident; you usually can’t write a snappy lead upholding [the negative]. A story purporting to show that apple pie makes you sterile is more interesting than one that says there is no evidence that apple pie changes your life.”²

We’ve joked that there are only two types of health news stories – New Hope and No Hope. In truth, we must remember that the truth usually lies somewhere in the middle.

2. We work fast, sometimes too fast, with severe limits on the space or airtime we may fill. We find it hard to tell editors or news directors, “I haven’t had enough time. I don’t have the story yet.” Even a long-term project or special may be hurriedly done. In a newsroom, “long term” may mean a few weeks.

A major Southern newspaper had to print a front-page retraction after a series of stories alleged that people who worked at or lived near a plutonium plant suffered in excess numbers from a blood disease. “Our reporters obviously had confused statistics and scientific data,” the editor admitted. “We did not ask enough questions.”³

3. We too often omit needed cautions and perspective. We tend to rely too much on “authorities” who are either most quotable or quickly available or both. They may get carried away with their own sketchy, unconfirmed but “exciting” data – or have big axes to grind, however lofty their motives. The cautious, unbiased scientist who says, “Our results are inconclusive” or “We don’t have enough data yet to make any strong statement” or “I don’t know” tends to be omitted or buried deep down in the story.

Some scientists who overstate their results deserve part of the blame. But bad science is no excuse for bad journalism.

We may write too glowingly about some experimental drug to treat a perilous disease, without needed perspective about what hurdles lie ahead. We may over-worry our readers about preliminary evidence of a possible new carcinogen – yet not write often enough about what the U.S. surgeon general calls the “now undisputable” evidence that secondhand tobacco smoke “is a serious health hazard.”⁴

4. Seeking balance in our reporting on controversial issues, we sometimes forget to emphasize where the scientific evidence points.

Study after study has found no evidence that childhood immunizations can cause autism – yet lay promoters (and some doctors) continue to garner ink and airtime on a popular daytime TV show (see Chapter 12).

On the hot-button issue of “global warming,” we must not get carried away by occasional super-cold winters. Year-to-year temperatures vary by their very nature. Climate experts had to study decades of weather and interpret data going back thousands of years to detect the slow, yet potentially dangerous, warming of our planet. The bottom line: Most scientists now agree that global warming is real, and is linked to the burning of fossil fuels. The scientific and societal debate continues over details such as how urgent the threat might be – and precisely what to do about it. Another bottom line: Don’t write the nay-sayers off as kooks and tools of industry. Sometimes even very minority views turn out to be right (see Chapter 9).

5. We are influenced by intense competition and other pressures to tell the story first and tell it most dramatically. One reporter said, “The fact is, you are going for the strong [lead and story]. And, while not patently absurd, it may not be the lead you would go for a year later.”⁵

Or even a few hours later. Witness the competitive rush to declare election-night winners, and the mistakes that sometimes result.

We are also subject to human hope and human fear. A new “cure” comes along, and we want to believe it. A new alarm is sounded, and we too tremble – and may overstate the risk. Dr. H. Jack Geiger, a respected former science writer who became a professor of medicine, says:

I know I wrote stories in which I explained or interpreted the results wrongly. I wrote stories that didn’t have the disclaimers I should have written. I wrote stories under competitive pressure, when it became clear later that I shouldn’t have written them. I wrote stories when I hadn’t asked – because I didn’t know enough to ask – “Was your study capable of getting the answers you wanted? Could it be interpreted to say something else? Did you take into account possible confounding factors?”

How can we learn to do better? How do we separate the wheat from the chaff in all ...

Table of contents

• Cover
• Title page
• Copyright page
• A Note to Our Readers
• A Tribute to Victor Cohn, 1919–2000
• Foreword
• Acknowledgments
• Notes on Sources
• Part I: Learning the Basics
• Part II: Now Down to Specifics
• Epilogue The Making of Good Journalists
• Glossary
• Bibliography
• Index