Standard Deviations

Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics

Gary Smith

• Publisher: ABRAMS Press
• Year: 2014
• Print ISBN: 9781468311020
• eBook ISBN: 9781468310689
• Topic: Social Sciences
• Subtopic: Data Mining
• Index: Social Sciences

How statistical data is used, misused, and abused every day to fool us: "A very entertaining book about a very serious problem." —Robert J. Shiller, winner of the Nobel Prize in Economics and author of Irrational Exuberance Did you know that baseball players whose names begin with "D" are more likely to die young? That Asian Americans are most susceptible to heart attacks on the fourth day of the month? That drinking a full pot of coffee every morning adds years to your life, but one cup a day increases your pancreatic cancer risk? These "facts" have been argued with a straight face by credentialed researchers and backed up with reams of data and convincing statistics. As Nobel Prize–winning economist Ronald Coase cynically observed, "If you torture data long enough, it will confess." Lying with statistics is a time-honored con. In Standard Deviations, economics professor Gary Smith walks us through the various tricks and traps that people use to back up their own crackpot theories. Sometimes, the unscrupulous deliberately try to mislead us. Other times, the well-intentioned are blissfully unaware of the mischief they are committing. Today, data is so plentiful that researchers spend precious little time distinguishing between good, meaningful indicators and total rubbish. Not only do others use data to fool us, we fool ourselves. Drawing on breakthrough research in behavioral economics and using clear examples, Standard Deviations demystifies the science behind statistics and makes it easy to spot the fraud all around us. "An entertaining primer... packed with figures, tables, graphs and ludicrous examples from people who know better (academics, scientists) and those who don't (political candidates, advertisers)." — Kirkus Reviews (starred review)

1

PATTERNS, PATTERNS, PATTERNS

YOUTH SOCCER IS A VERY BIG DEAL WHERE I LIVE IN SOUTHERN California. It’s a fun, inexpensive sport that can be played by boys and girls of all sizes and shapes. I initially didn’t know anything about soccer. All I knew was that, every weekend, the city parks and school grounds were filled with kids in brightly colored uniforms chasing soccer balls while their parents cheered. When my son was old enough, we were in.

By the time the 2010 World Cup came around, my son was playing on one of the top soccer teams in Southern California. I was the manager and a fanatic about soccer, so naturally he and I watched every World Cup match we could. The opponents in the 2010 championship game were Netherlands and Spain, two extraordinarily talented teams from underachieving nations that often disappointed their supporters. Which country would finally win the World Cup? I loved the Dutch, who had won all six of their World Cup games, scoring twelve goals while allowing only five, and had knocked out the mighty Brazil and Uruguay. But then I heard about Paul the octopus, who had correctly predicted the winners of seven World Cup games by choosing food from plastic boxes with the nations’ flags on them. Paul the Oracle had picked Spain, and the world now seemed certain of a Spanish victory.

What the heck was going on? How could a slimy, pea-brained invertebrate know more about soccer than I did? I laughed and waited for Paul the Omniscient to get his comeuppance. Except he didn’t. The Dutch did not play with their usual creativity and flair. In a brutal, cynical match, with fourteen yellow cards—nine given to the dirty Dutchmen—Spain scored the winning goal with four minutes left in the game.

How could an octopus living in a tank have predicted any of this? Had Paul ever seen a soccer game? Did Paul even have a brain?

It turns out that octopuses are among the most intelligent invertebrates, but that isn’t saying much—sort of like being the world’s tallest midget. Still, Paul made eight World Cup predictions and got every single one right. Not only that, Paul made six predictions during the 2008 European Football Championships and got four right. Overall, that’s twelve out of fourteen correct, which in the eyes of many would be considered statistical proof of Paul’s psychic abilities. But were there really enough data?

If a fair coin is flipped fourteen times, the chances of twelve or more heads are less than one percent. In the same way, if Paul were just a hapless guesser with a 50 percent chance of making a correct prediction, the probability that he would make so many correct predictions is less than 1 percent, a probability so low that it is considered “statistically significant.” The chances of Paul being correct so many times are so small that, logically, we can rule out luck as an explanation. With his consistency, Paul had demonstrated that he was not merely a lucky guesser. He was truly Paul the Psychic Octopus!

And yet, something didn’t seem quite right. Is it really possible for an octopus to predict the future? Paul’s performance raises several issues that are endemic in statistical studies. Paul was not a psychic (surprise, surprise), but he is a warning of things to watch out for the next time you hear some fanciful claim.

CONFOUNDING EFFECTS

First, let’s look at how Paul made his predictions. At feeding time, he was shown two clear plastic boxes with the national flags of the opposing teams glued to the front of the boxes. The boxes contained identical yummy treats, such as a mussel or an oyster. Whichever box Paul opened first was the predicted winner.

Octopuses don’t know much about soccer, but they do have excellent eyesight and good memories. One time, an octopus at the New England Aquarium decided he didn’t like a volunteer and shot salt water at her whenever he saw her. She left the aquarium to go to college, but when she returned months later, the octopus remembered her and immediately drenched her with salt water again. In an experiment at a Seattle aquarium, one volunteer fed the octopuses while another wearing identical clothes irritated the octopuses with a stick. After a week of this, most of the octopuses could tell who was who. When they saw the good person, they moved closer; when they saw the bad person, they moved away (and sometimes shot water at him for good measure).

Paul the Psychic Octopus happened to be living in an aquarium in Germany and, except for the Spain-Netherlands World Cup final, Paul only predicted games involving Germany. In eleven of the thirteen games involving Germany, Paul picked Germany—and Germany won nine of these eleven games. Was Paul picking Germany because he had analyzed their opponents carefully or because he had an affinity for the German flag? Paul was almost certainly color blind, but experiments have shown that octopuses recognize brightness and are attracted to horizontal shapes. Germany’s flag has three vivid horizontal stripes, as do the flags of Serbia and Spain, the only other countries Paul selected. Indeed, the Spanish and German flags are pretty similar, which may explain why Paul picked Spain over Germany in one of the two matches they played and picked Spain over the Netherlands in the World Cup final. The only game in which Paul did not choose the German or Spanish flag was a match between Serbia and Germany.

The flag was apparently a confounding factor in that Paul wasn’t picking the best soccer team. He was choosing his favorite flag. Paul the Omniscient was just a pea-brained octopus after all.

Figure 1.1: Paul’s Favorite Flags

Germany (eleven times)

Spain (twice)

Serbia (once)

SELECTIVE REPORTING AND MISREPORTING

Another explanation for Paul’s success is that too many people with too much time on their hands try stupid pet tricks, using animals to predict sports, lottery, and stock market winners.

Some will inevitably succeed, just like among thousands of people flipping coins, some people will inevitably flip heads ten times in a row. Who do you think gets reported, the octopus who picked winners or the ostrich who didn’t?

Several years ago, a sports columnist for The Dallas Morning News had a particularly bad week picking the winners of National Football League (NFL) football games—he got one right and twelve wrong, with one tie. He wrote that, “Theoretically, a baboon at the Dallas Zoo can look at a schedule of 14 NFL games, point to one team for each game and come out with at least seven winners.” The next week, Kanda the Great, a gorilla at the Dallas Zoo, made his predictions by selecting pieces of paper from his trainer. Kanda got nine right and four wrong, better than all six Morning News sportswriters. The media descended on the story like hungry wolves, but would Kanda’s performance have been reported if he had gotten, say, six right and seven wrong?

Not to be outdone, officials at the Minnesota Zoo in Apple Valley, Minnesota, reported that a dolphin named Mindy successfully predicted the outcomes of NFL games by choosing among pieces of Plexiglas, each bearing a different team’s name. The opponents’ Plexiglas sheets were dropped into Mindy’s pool and the one she brought back to her handler was considered to be her “prediction.” The handlers reported that Mindy had gotten thirty-two of fifty-three games correct. If so, that’s 60 percent, enough to make a profit betting on football games.

How many other birds, bees, and beasts tried and failed to predict NFL games and went unreported because they failed? We don’t know, and that’s precisely the point. If hundreds of pets are forced to make pointless predictions, we will be misled by the successful ones that get reported because we don’t take into account the hundreds of unsuccessful pets that were not reported.

This doesn’t just happen in football games. A Minneapolis stock broker once boasted that he selected stocks by spreading The Wall Street Journal on the floor and buying the stock touched by the first nail on the right paw of his golden retriever. The fact that he thought this would attract investors says something about him—and perhaps his customers.

Another factor is that people seeking fifteen minutes of fame are tempted to fudge the data to attract attention. Was there an impartial observer monitoring the Minneapolis stockbroker and his dog each morning? Back when bridge was the most popular card game in America, a mathematically inclined bridge player estimated that far too many people were reporting to their local paper that they had been dealt a hand with thirteen cards of the same suit. Given the chances of being dealt such a hand, there were not nearly enough games being played to yield so many wacky hands. Tellingly, the suit reported was usually spades. People were evidently embellishing their experiences in order to get their names in the paper.

After Paul the octopus received worldwide attention, a previously obscure Singapore fortune teller reported that his assistant, Mani the parakeet, had correctly predicted all four winners of the World Cup quarterfinal matches. Mani was given worldwide publicity, and then predicted that Uruguay would beat Netherlands and that Spain would beat Germany in the semifinals, with Spain defeating Uruguay in the championship game. After Netherlands defeated Uruguay, Mani changed his finals prediction, choosing Netherlands, which turned out to be incorrect. Nonetheless, the number of customers visiting this fortune teller’s shop increased from ten a day to ten an hour—which makes you wonder whether the owner’s motives were purely sporting and whether his initial reports of Mani’s quarterfinal predictions were accurate.

Why did Paul and Mani become celebrities who were taken seriously by soccer fans celebrating and cursing their predictions? Why didn’t they stay unnoticed in the obscurity they deserved? It’s not them, it’s us.

HARDWIRED TO BE DECEIVED

More than a century ago, Sherlock Holmes pleaded to his long-suffering friend Watson, “Data! Data! Data! I can’t make bricks without clay.” Today, Holmes’s wish has been granted in spades. Powerful computers sift through data, data, and more data. The problem is not that we don’t have enough data, but that we are misled by what we have in front of us. It is not entirely our fault. You can blame it on our ancestors.

The evolution of certain traits is relatively simple. Living things with inheritable traits that help them survive and reproduce are more likely to pass these traits on to future generations than are otherwise similar beings that do not have these traits. Continued generation after generation, these valuable inherited traits become dominant.

The well-known history of the peppered moth is a simple, straightforward example. These moths are generally light-colored and spend most of their days on trees where they are camouflaged from the birds that prey on them. The first dark-colored peppered moths were reported in England in 1848, and by 1895, 98 percent of the peppered moths in Manchester were dark-colored. In the 1950s, the pendulum started swinging back. Dark-colored moths are now so rare that they may soon be extinct.

The evolutionary explanation is that the rise of dark-colored moths coincided with the pollution caused by the Industrial Revolution. The blackening of trees from soot and smog gave dark-colored moths the advantage of being better camouflaged and less likely to be noticed by predators. Because dark-colored moths were more likely to survive long enough to reproduce, they came to dominate the gene pool. England’s clean-air laws reversed the situation, as light-colored moths are camouflaged better on pollution-free trees. Their survival advantage now allows them to flourish.

Other examples of natural selection are more subtle. For example, studies have consistently found that men and women are more attracted to people with symmetrical faces and bodies. This isn’t just cultural—it is true across different societies, true of babies, and even found in other animals. In one experiment, researchers clipped the tail feathers of some male barn swallows to make them asymmetrical. Other males kept their symmetrical tail feathers. When female swallows were let loose in this mating pool, they favored the males with symmetrical feathers. This preference for symmetry is not just a superficial behavior. Symmetry evidently indicates an absence of genetic defects that might hamper a potential mate’s strength, health, and fertility. Those who prefer symmetry eventually dominate the gene pool because those who don’t are less likely to have offspring that are strong, healthy, and fertile.

Believe it or not, evolution is also the reason why many people took Paul and Mani seriously. Our ingrained preference for symmetry is an example of how recognizing patterns helped our human ancestors survive and reproduce in an unforgiving world. Dark clouds often bring rain. A sound in the brush may be a predator. Hair quality is a sign of fertility. Those distant ancestors who recognized patterns that helped them find food and water, warned them of danger, and attracted them to fertile mates passed this aptitude on to future generations. Those who were less adept at recognizing patterns that would help them survive and reproduce had less chance of passing on their genes. Through countless generations of natural selection, we have become hardwired to look for patterns and to think of explanations for the patterns we find. Storm clouds bring rain. Predators make noise. Fertile adults have nice hair.

Unfortunately, the pattern-recognition skills that were valuable for our long-ago ancestors are ill-suited for our modern lives, where the data we encounter are complex and not easily interpreted. Our inherited desire to explain what we see fuels two kinds of cognitive errors. First, we are too easily seduced by patterns and by the theories that explain them. Second, we latch onto data that support our theories and discount contradicting evidence. We believe stories simply because they are consistent with the patterns we observe and, once we have a story, we are reluctant to let it go.

When you keep rolling sevens at the craps table, you believe you are on a hot streak because you want to keep winning. When you keep throwing snake eyes, you believe you are due for a win because you want to start winning. We don’t think hard enough about the fact that dice do not remember the past and do not care about the future. They are inanimate; the only meaning they carry is what we hopeful humans ascribe to them. If the hot streak continues or the cold streak ends, we are even more convinced that our fanciful theory is correct. If it doesn’t, we invent excuses so that we can cling to our nonsensical story.

We see the same behavior when athletes wear unwashed lucky socks, when investors buy hot stocks, or when people throw good money after bad, confident that things must take a turn for the better. We yearn to make an uncertain world more certain, to gain control over things that we do not control, to predict the unpredictable. If we did well wearing these socks, then it must be that these socks help us do well. If other people made money buying this stock, then we can make money buying this stock. If we have had bad luck, our luck has to change, right? Order is more comforting than chaos.

These cognitive errors make us susceptible to all sorts of statistical deceptions. We are too quick to assume that meaningless patterns are meaningful when they are pres...

Table of contents

• Cover
• MORE ADVANCE PRAISE FOR STANDARD DEVIATIONS
• COPYRIGHT
• DEDICATION
• CONTENTS
• INTRODUCTION
• 1. Patterns, Patterns, Patterns
• 2. Garbage In, Gospel Out
• 3. Apples and Prunes
• 4. Oops!
• 5. Graphical Gaffes
• 6. Common Nonsense
• 7. Confound It!
• 8. When You’re Hot, You’re Not
• 9. Regression
• 10. Even Steven
• 11. The Texas Sharpshooter
• 12. The Ultimate Procrastination
• 13. Serious Omissions
• 14. Flimsy Theories and Rotten Data
• 15. Don’t Confuse Me With Facts
• 16. Data Without Theory
• 17. Betting the Bank
• 18. Theory Without Data
• 19. When to Be Persuaded and When to Be Skeptical
• SOURCES
• INDEX