Making Sense of Data in the Media

Andrew Bell, Todd Hartman, Aneta Piekut, Alasdair Rae, Mark Taylor

• Publisher: SAGE Publications Ltd
• Year: 2019
• ISBN:
  9781526493002
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

The amount of data produced, captured and transmitted through the media has never been greater. But for this data to be useful, it needs to be properly understood and claims made about or with data need to be properly scrutinized. Through a series of examples of statistics in the media, this book shows you how to critically assess the presentation of data in the media, to identify what is significant and to sort verifiable conclusions from misleading claims.How accurate are polls, and how should we know? How should league tables be read? Are numbers presented as 'large' really as big as they may seem at first glance?

By answering these questions and more, readers will learn a number of statistical concepts central to many undergraduate social science statistics courses. By tying them in to real life examples, the importance and relevance of these concepts comes to life. As such, this book does more than teaches techniques needed for a statistics course; it teaches you life skills that we need to use every single day.

1 Introduction

Statistics are everywhere in the news media. And yet they are widely misunderstood, poorly reported and often downright false. Not only that, but consumers of the news are not well prepared to understand the figures used by reporters and journalists. Just consider how many people openly flaunt their poor quantitative skills. Maths teachers regularly hear phrases from parents like, ‘We’re terrible at Maths, we can’t help at all.’ Why is it that English teachers rarely get told by parents, ‘We can’t read very well, so we can’t help Delila with her homework’?

Politicians in the news haven’t helped remedy this quantitative skills deficit. In the run-up to the 2016 EU Referendum in the UK, Conservative MP Michael Gove defended Brexit by arguing that ‘people in this country have had enough of experts’. In the United States, President Donald Trump routinely attacks the news media when they don’t report favourable coverage of his administration. For instance, he once tweeted that ‘any negative polls are fake news, just like the CNN, ABC, NBC polls in the election’.

Part of the problem is that people think statistics should provide an objective truth – one that can be used to definitively prove things. In fact, this is far from the case. Sometimes statistics are blatantly misused. But there are many other examples where figures can actually be interpreted in more than one way. As it turns out, statistics are sometimes as fuzzy and subjective as any other research methods.

Given this state of affairs, perhaps you can forgive Trump and Gove in the above examples. With so much bad statistics, isn’t it best to just ignore them entirely? We can’t trust the experts because of all the bad statistics, so surely we’re better off just going with our intuition? An extra bonus of this course of action is that our views are never challenged, so we can continue believing whatever warm, fuzzy thing we want to believe.

This is one option. The problem is that it leaves us open to being manipulated and used by people who have their own, often sinister, motives. Corporations, news organisations and governments have realised that there are ways to subtly change and undermine public opinion, and to encourage people to buy a certain product, believe a certain viewpoint or vote a certain way. If we leave statistics in the gutter, and just go with our gut feelings, we leave ourselves open to those feelings being manipulated.

So what’s the other option? The other option is a bit more difficult, requires a little more work and starts with reading this book. It involves learning how to understand statistics. It involves learning how to spot when statistics are done well, and when they are done poorly. It also involves reading news articles and poll results with a critical eye, and coming up with your own, sometimes subjective, fuzzy conclusions. And it involves being willing to let your preconceptions about your beliefs be challenged.

Fortunately, all this isn’t as difficult as you might think. It turns out that by learning this one neat trick, you can understand everything that is written in the news and …

OK, maybe it’s not that easy! But the ideas behind statistics don’t need to be shrouded in complex equations and impenetrable language. Statistics are really just advanced common sense. That means there aren’t any tricks that will immediately give you an answer about the world or about statistics, or about anything else. But it also means that the same abilities you have in thinking critically about other things in the world can be applied to statistics as well.

Having said that, there are patterns in the things that people often get wrong in statistics, and it is some of the more common issues that we focus on in this book. Over the course of the next 250-or-so pages, you’ll see examples of statistics being used well, as well as being used badly. You’ll see what people did right, and what they did wrong. And by doing so, you’ll see the sorts of things that will help you spot good and bad statistics in the news and media in the future.

As a result, this book will be useful for people who want to be journalists, so they know how to report data in the news in a way that is responsible and effective. But it will also be useful to anyone that reads the news, or watches adverts, or goes on the Internet. In other words, it is potentially useful to anyone. Understanding data is not a niche thing that should be left to a few nerds in universities. It is everyone’s responsibility to be able to understand the information that is thrown at us, and to critique it when it is used badly. We all need to do this to stop businesses cheating us out of money or politicians lying to us. In other words, understanding statistics helps our societies and our democracies to work properly.

That’s the other thing about this book – it contains few equations. There will be some numbers, and occasionally there will be an equals sign, but for the most part we’re going to tell you things in words. This is not because we think that equations are unnecessary. Sometimes, for some people, they make things easier to understand. But usually those things can be explained in words too, and that’s usually better for most people who shy away from mathematics and complex equations.

Throughout this book, you’ll see some boxes on a variety of different things. Some of these present examples that are relevant, but not essential, to the topic of the book. Some present interesting stories that help put some flesh on the bones of the overall points in the book. Others deal with advanced topics that many won’t need or won’t want to know, while some more advanced readers might like to learn more about them. So if you don’t understand the content of some of these boxes, you shouldn’t worry too much (but also, don’t feel afraid to give them a go and see what you can get out of them!).

The rest of this introduction will give an example of bad statistics – just really a bit of fun. It will then go over what the rest of the book looks like. We hope you enjoy it!

1.1 ‘30,000 pigs float down the Dawson’

It might surprise you that, in a book about data in the media, we would start with the Rockhampton (Australia) Morning Bulletin. However, in February 2011, the Bulletin gained a brief moment of fame as a result of an unfortunate and comedic error. At the time, Queensland was struggling with flooding affecting hundreds of thousands of people, leading to the evacuation of a number of towns and cities. However, it also affected farming and livestock with, according to the Bulletin, 30,000 pigs being washed away from one farm alone.

Hang on, though: 30,000 pigs? In a book about being sceptical about numbers, here’s a starting point. That number doesn’t sound right. Even the biggest pig farms in the world have only around 2,000 animals. 30,000 pigs would, if lined up curly tail to snout, stretch about 50 km: that is, about 500 football pitches. While the image of that many pigs floating down the river is an impressive one, it is, sadly, probably implausible.

It turns out that the mistake resulted from a miscommunication between a journalist and the farmer. While the journalist had heard ‘30,000 pigs’, the farmer had actually said ‘30 sows and pigs’ – sows, as in female pigs! The Morning Bulletin issued a correction the following day.

This mistake is perhaps not the most regularly encountered data error you will ever see. But it is a reminder that not all errors are disastrous – some are just a bit funny. This isn’t always the case, however. Sometimes getting statistics wrong in the media can have quite scary consequences. In this book we’ll see an example where people have ended up convicted of murder because of a misunderstood statistic. We’ll see that statistics can convince people to vote a certain way, meaning that misleading statistics can actively undermine democracy. And we’ll see that misinterpreted statistics can be used to stir up people’s prejudices and potentially increase hate crime against certain groups.

In other words, while errors in statistics can be funny, they can also be important. As we will see in this book, they can lead people with a disease to get the wrong treatment, or put an innocent person in prison. They can literally be a matter of life and death.

1.2 How the Book Progresses

This book is organised in chapters based around particular statistical ideas. However, within each chapter, there are actually very few statistics at all. The book focuses on examples which will illustrate statistical concepts in what we hope is a clear and easy-to-understand way. So, unlike a more standard statistics textbook, where you might start with a statistical method, in this book you will examine some examples, and in the process learn some statistical concepts perhaps without even realising it!

Chapter 2 asks perhaps the most basic question we need to answer when seeing a number in the news: is that a lot? It’s a surprisingly difficult question to answer, and numbers can be made to seem big or small simply by changing the way they are framed. So, 0.0001 might be very big indeed, and 30 billion might, in fact, be pretty small. It all depends on the context of those numbers, and what they are measuring.

While lots of statistics books consider carefully how numbers can be manipulated once we have them, Chapters 3 and 4 take a step back and focus on how data is collected. Chapter 3 helps us think about the sources of data because some data sources are more valuable, accurate and informative than others. The chapter will help you spot bad data and consider some of the ways that people collecting data can make their data better.

Chapter 4 focuses on representativeness – that is, for data about people, how can we be sure that it’s talking about the people we want it to be about? A survey of men might not tell us much about the views of women; a survey of politicians probably doesn’t tell us much about the views of people in the country more widely. The chapter will help you understand how researchers try to make surveys representative, and the problems that failing to do this properly can cause.

Chapter 5 is about graphs and shows that, even if your data is accurate, a graph can mislead the person looking at it if it’s designed in a certain way (either deliberately or accidentally). The chapter goes through some graphs that have been used in the news media, and shows how easily they can mislead people. It will also show you how you can avoid being hoodwinked by such graphs, by spotting some simple but subtle tricks that can make data show something that isn’t there.

Chapters 6 and 7 are about the geographic representation of data, or, to put it more simply, maps. The fundamental principle underlying these two chapters is that, almost without exception, everything has a location and the numbers we see in the media relate to real-world places. A good example of this is voting patterns, which tend to cluster geographically at the neighbourhood level. So, with this in mind, Chapter 6 is based on the fundamental question of where things happen and why maps matter. We also discuss what you might want to consider before making the decision to map something and whether a map is the right tool in the first place. Maps are only one way to represent data and we’re careful to note that they are neither value neutral nor inherently ‘truthful’. Yet they remain immensely powerful, so it’s important to understand them. The example of political gerrymandering is used to make this point.

In Chapter 7 we go into more detail of why geography matters and look at slightly more complex topics such as spatial aggregation and why the geographic units we use to analyse phenomena can have a profound impact on our results. The key message we want readers to take from these chapters is that mapping matters, but also that the way we make maps can influence what they appear to show.

Numbers are often reported in the media as absolute, specific and certain values, suggesting that they cannot be questioned. But uncertainty is central to statistics, and it should be central to the way that numbers are understood. Chapter 8 focuses on uncertainty: how sure should we be about numbers in the media? It will give examples of numbers that, though reported as truthful, in fact have a lot of uncertainty, because apparent patterns can occur simply by chance.

Chapter 9 takes an example to help illustrate this. League tables are often presented in the media, both in sport and in other aspects of life (schools, universities, hospitals, and so on). They are presented as if the top-ranked team/school/university is definitely the best, the next as second best, and so on. But it’s rarely clear what the assumptions underlying these league tables are, or how certain we are that the rankings are correct. Chapter 9 considers how league tables are far from objective and are often manipulated for political ends, meaning they should be interpreted cautiously.

Chapter 10 considers relationships between different things. Is a particular drug related to better health? Do parks make people happier? Does mayonnaise increase suicide? You may be sceptical about some or all of these (hopefully at least the last one!), but as it turns out, it’s very difficult to say that something causes something else. Chapter 10 gives examples of news stories that assume a relationship is causal, when, actually, it’s likely to have been a result of something else.

Chapter 11 considers some of the more surprising results that statistics can produce. It will show that things that sound impossible actually can be very possible indeed. A drug can be 99% effective, yet a positive test result is probably wrong? Sure. Employment rates can simultaneously be going up and going down? Yep. Two people in your class share a birthday? Almost certainly.

Finally, we sum up the book in chapter 12, by using an example that illustrates many of the concepts learned throughout the book and shows how important those concepts can be. We finish with a call for all readers to help in fighting against bad statistics and misleading uses of data in the media.

Throughout the chapters, you will learn a lot about scary-sounding statistical concepts. But this book will hopefully show that they aren’t that scary. In fact, pretty much all statistical concepts can be boiled down to common-sense ideas. If the idea of statistics really fills you with dread, we hope this book will help you do something impossible (or at least, very, very unlikely): you might actually enjoy statistics!

1.3 A Note to Teachers

This book can be used in a variety of ways in teaching both about data in the media and quantitative methods/statistics, more generally. First, it could be the core text and basis of a class in itself – indeed the structure of the course came from such a class that we run at the University of Sheffield. But it could also be used to supplement a more standard statistics course. In the main, individual chapters stand on their own, so chapters could be set readings that would help students understand why the concepts they are studying are relevant and important, rather than just abstract...