Survey Research and Sampling

Jan Eichhorn, Author

• Publisher: SAGE Publications Ltd
• Year: 2022
• Print ISBN: 9781526423801
• eBook ISBN: 9781529710878
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

This book is an ideal companion for students looking to undertake survey research. Anchored by lots of case studies of real research and expert interviews to strengthen your understanding, it provides guidance on:

• Selecting the best sampling method for your research
• Designing questionnaires to ensure you gather valuable data
• Collecting your data to represent populations well
• Piloting questionnaires to ensure quality research

Part of The SAGE Quantitative Research Kit, this book will give you the know-how and confidence needed to succeed on your quantitative research journey

1 Introduction

Chapter Overview

• Why do we need to worry about sampling and survey design? 2
• The approach of this volume 3
• Overview of the content of this volume 5
• Further Reading 8

Why do we need to worry about sampling and survey design?

According to a report by a committee of the Scottish Parliament issued in 2015, something extraordinary must have taken place in the aftermath of the 2014 Scottish independence referendum. Youth participation must have increased dramatically following the lowering of the voting age to 16 years. The headline of the news item covering the findings of a survey conducted for this age group by the Scottish Parliament read,

An overwhelming majority of 16- and 17-year-olds who were eligible to vote in last year’s historic referendum did so, with four out of five saying they want a vote in all future elections, a Scottish Parliament committee survey has found. More than 1200 of the eligible first-time voters responded to the online survey run by the Devolution (Further Powers) Committee. (Scottish Parliament, 2015b)

Indeed, these insights matched the results from other research that suggested that those young people who had recently been enfranchised largely made use of the opportunity to vote (especially compared to 18- to 24-year-olds, who participated much less; Electoral Commission, 2014, p. 64) and were also showing greater political engagement than their peers elsewhere in the UK (Eichhorn, 2018). At first sight, everything seems to be fine. A survey of 1200 respondents would usually be seen as a robust sample, and the results seemed to agree with findings from other research too. However, once we look at the details a little more, we might become more sceptical.

One of the most remarkable findings of the survey, the increase in young people joining political parties, made national news headlines. The Guardian, for example, titled ‘Quarter of youngest Scottish voters have joined a party since referendum’ (Brooks, 2015). The author cited the findings from the Scottish Parliament survey, which said that 26% of 16- and 17-year-olds had joined a political party since the independence referendum (Scottish Parliament, 2015a). Let us break those figures down for a moment. According to the Scottish Government’s records, there were roughly 122,000 16- and 17-year-olds living in Scotland in 2014 (Scottish Government, 2015). There was indeed a substantial increase in party membership immediately following the referendum, in particular for those parties that supported independence. The Scottish National Party’s (SNP’s) membership reached 93,000 in January 2015 (Keen & Apostolova, 2017) – up from about 26,000 on the eve of the referendum – while the Green Party’s membership increased from just under 2000 to around 9000 (Kennouche, 2015) during the same period. If we take those figures togetherⁱ, this would correspond roughly to a total increase of party members of 74,000. If indeed 26% of 16- and 17-year-olds had joined a party in this period, that would be equivalent to roughly 31,700. This would suggest that of all the people who joined political parties in the wake of the Scottish independence referendum, somewhere around 42% to 43% were 16- or 17-year olds. That already seems suspiciously high. If we then look at research into the actual structure of party membership, further doubts are cast, considering that the mean age of SNP members was approximately 49 years ahead of the 2015 general election (Bale & Webb, 2015, p. 6). This clearly could not be the case if so many 16- and 17-year-olds had joined. Something must have gone wrong here.

ⁱ Other Scottish political parties did not reveal membership figure changes in the same period and in informal interviews suggested that, while they also welcomed some new members, increases were in the hundreds rather than thousands.

A look into the background notes on the Scottish Parliament survey provides the main insight: ‘The survey was distributed to young people via schools, colleges and youth clubs. More than 1200 responses were received’. At the same time, the committee responsible ran a campaign to get young people in schools to engage with these issues and the Parliament. So the likely result of this was that the participants in this survey were composed in particular of school students who were either generally interested in politics or specifically engaged with related topics through classes in school or youth club activities. In other words, the young people surveyed were probably not very representative of their peers across Scotland more generally. The sample of 1200 respondents was skewed towards those who were more interested or engaged in politics already. This is often referred to as a self-selection bias in sampling. The news issued about the survey and the articles in the media reporting it, however, did not comment on this problem. So, quickly the image arose that young people in Scotland were even more politically engaged than they had indeed been found to be. The extent was massively exaggerated.

The approach of this volume

To avoid such mistakes, it is crucial to design a good sampling strategy when conducting any form of social survey. While we are, understandably, mostly interested in the results following the analysis of quantitative data, making sure that the data we get to work with are actually adequate is fundamental if we want to ensure that we can make statements that reflect what is really going on. It is also crucial for us to understand this in order to evaluate the quality of other research when assessing how meaningful the insights from it really are. This volume is about the part of the research that takes place before we get any quantitative data that we can actually analyse. It is about the stage at which we design the research, in terms of how we sample relevant groups of people for our studies, how we construct questionnaires that enable us to ask people about the issues we are interested in and how we check the quality of our data before we begin to analyse them properly. None of these tasks are easy, and they require a significant amount of critical thinking. The design of a survey project can look very different depending on the scope and aims – from quick polls that need to be administered within days to complex surveys that may take over a year from start to finish. Speed is one issue that can affect the process. Additionally, budget constraints often play an important role. In most situations, we have a good idea of how we could maximise the quality of a survey from a theoretical point of view; however, the costs are often prohibitive. Therefore, we need to find ways to achieve good quality outcomes even when our resources are constrained.

This volume aims to provide an overview of the most important issues that we need to take into account when designing surveys and developing sampling strategies to achieve high-quality outcomes. It will familiarise the reader with the variety of different approaches that can be chosen and discusses advantages and disadvantages of them. Crucially, this volume is meant to be applicable to the real world and the practice of carrying out this work. As the example above illustrates, the approaches we choose for collecting survey data heavily impact to what extent the analyses we conduct later on can be seen as valid and meaningful. Therefore, we will not only look at theoretical considerations – for example, with regard to what sampling strategies may be likely to provide the most representative types of samples – but also ask how we can achieve the best quality research while keeping in mind the real-life constraints that are often placed on researchers in practice (e.g. time, budget and topical limitations). Crucially, this volume is introductory and does not assume prior knowledge on the part of its readers. It should be accessible to any person who wants to learn about the topics covered. The goal is to provide the reader with access into the debates about questionnaire design and sampling and raise awareness about important aspects researchers should consider when they look at the data produced by others. Each chapter in this volume could have a whole volume dedicated to it in its own right, however. In order to capture a wide range of issues that we should consider when designing surveys, the depth we can go into regarding each individual topic has to be limited. On many occasions, other volumes in this series provide those deeper insights, and where that applies, we suggest such links. Also, in the chapters, we cite work by methodological experts exploring the respective topics and would encourage readers to follow up those references if they are interested in finding out more details about the specific themes discussed. The principal goal of the volume, however, is that readers will be able to engage with discussions about sampling and questionnaire design and the implications that stem from both for survey data in a meaningful way, both in terms of the academic background and in particular also the practical side of it.

To achieve this, we will not only discuss sampling and survey design in an abstract manner but apply those discussions to real-life examples. The chapters in this volume contain case studies about specific survey projects and discuss shortcomings in surveys and polls where results may have been distorted on the one hand and highlight examples of good practice and insights from high-quality approaches on the other hand. The surveys referred to range from smaller scale regional to large-scale comparative international surveys. Some focus on a very specific subgroup of the population, while others try their best to represent the general population of countries. Using those case studies will help to establish the relevance of the theoretical concerns raised in this volume and enable all readers to consider possible applications following some reflections. As survey research is carried out by people, many of whom specialise in this field and spend many years continuously improving their knowledge and abilities to undertake the best possible work, we will further complement our insights by hearing from some of those professionals. To gain an insight into the practical aspects of survey design and sampling work, this volume will present insights from interviews with professionals specifically carried out for this volume. To show the breadth of applications, we interview experts from academia, professional research institutes and the private sector, reflecting the wide range of contexts in which survey work is carried out. The interviewees for the volume are as follows:

• Paul Bradshaw, who is the Director of ScotCen Social Research and has previously been Group Head of Longitudinal Surveys for NatCen Social Research
• Prof. Sir John Curtice, who is an expert psephologist, Professor of Politics at the University of Strathclyde and the President of the British Polling Council
• Rachel Ormston, who is Associate Director at Ipsos MORI Scotland and who has previously worked for multiple survey and polling organisations
• Susan Reid, who is Research Director (Social Attitudes) at ScotCen Social Research and Project Manager of the Scottish Social Attitudes Survey
• Prof. Dr Christian Welzel, who is Professor for Political Culture Research at Leuphana Universität Lüneburg and Vice President of the World Values Survey Association

The interviews with the interviewees helped to prioritise key issues for discussion in this volume, but most importantly, their contributions, as insights from practitioners, are integrated directly into the context of this volume, most notably in the form of ‘Ask an Expert’ sections, where we present their accounts directly in relation to the topics of the respective sections.ⁱⁱ

ⁱⁱ The volume presents direct quotes from the interviews. They are taken verbatim, but have been corrected for grammar mistakes and filler words and phrases have been removed. The edited quotes used here have been shown to interviewees for confirmation of consent.

Overview of the content of this volume

In this volume, we engage extensively with questions related to the development stage and the data collection forming the foundation of social surveys. So, it is crucial to understand what the purpose of sampling is and how we go about it. In Chapter 2, we address this question.

We discuss in detail how we should approach the sampling process, so that the people we interview as respondents actually allow us to make statements about the population they are drawn from. To do this, we introduce the logic that underpins sampling for social surveys. Our main aim is to establish a sample that is representative, and we will discuss what that precisely means for a research design. The understanding will be aided by engaging with the concept of a sampling frame, which provides the set of possible respondents who could be interviewed. There are many different ways how this can be undertaken, and readers will learn about the difference between probability and non-probability sampling methods, their advantages and disadvantages and how they relate to questions of representativeness. In both instances, we consider multiple subtypes (e.g. stratified, multistage and cluster sampling in the case of the former, and quota, convenience and snowball sampling in the case of the latter). A classic case study into the 1936 US presidential elections illustrates the importance of understanding sampling methods well. Two ‘Ask an Expert’ sections provide some deep dives, first, to learn about the specific problems of sampling very small and specific population groups (e.g. young people or certain ethnic minorities), and second, to explore the distinction between surveys and polls and the special characteristics of exit polls, which we encounter on election nights.

After engaging with the theories behind sampling methods, Chapter 3 looks at the implementation in practice. Even within a particular sampling methodology, there are many ways of delivering the data collection. Interviews with respondents can be carried out face to face, over the telephone, online using a variety of platforms or via physical mail, for example. The mode of conducting survey interviews is important, as it can affect how people respond. People may give slightly different answers to the same question if asked in person or when entering their reply on a computer screen. In this chapter, we discuss the advantages and disadvantages of different ways of implementing sampling methods for the data collection, considering both quality-oriented concerns and very practical ones, such as budget and time constraints, and importantly, what to do when reaching particular groups of respondents is very difficult and why it can be helpful to combine multiple modes of data collection. We also briefly discuss why small deviations from perfect distributions in samples may not be a problem if we can make meaningful adjustments through a process called weighting. In a case study from the World Values Survey’s implementation in Germany, we look at how physically interviewers in face-to-face administration modes actually go about recruiting respondents, helping us to understand why such methods are so research intensive. We apply the knowledge gained in an ‘Ask an Expert’ section, where the misevaluations in polling ahead of recent UK general elections are discussed.

Getting the right sample of people to take part in the survey is crucial, but of course it is only the, albeit very important, starting point of a survey project. We also need to be able to ask people good questions in order to make sure that the information we receive actually enables us to conduct the analyses we are interested in. Chapter 4 therefore focuses on questionnaire design. We discuss different strategies for the development of questions and distinguish different types. Subsequently, we discuss how such a process can be undertaken and consider how we might think of questions not just individually but as grouped sets depending on the concepts we plan to study. Afterwards, we look at what makes a good question and what common pitfalls exist that might make a question less useful or even invalid. The chapter introduces readers to different types of questions and makes them aware of how these different varieties of question constructs correspond better or worse to certain types of analyses, which is why it is so important to think carefully not only about the content but also about the structure of the questions. Our expert interviewees reflect about what they deem to be the most important characteristics of good questions and what they think are the most common mistakes in questionnaire design that should be avoided. The case study in this chapter places us in the heated debate about Scottish independence in the lead-up to the referendum in 2014 and discusses how question order effects have been used to produce a particular polling result for a political campaign.

Even with the best planning and the application of existing theoretical and practical knowledge, it is often very difficult to know how exactly a survey question will work when it is used in the field. This particularly applies to new questions that have not been used before or those that are applied to contexts in which they have not been used previously. Chapter 5 discusses a very important stage in the development of surveys: piloting. Especially when we consider how costly and time-consuming surveys can be, it is important that we ensure that they will work in the way they are intended to before rolling them out fully. In this chapter, we therefore describe different options for testing how well a survey works through a number of piloting strategies, including the utilisation of internal checks, cognitive interviewing and small-scale tests. Furthermore, we discuss how interviewers can be prepared to minimise problems in implementing a survey and what limitations there may be to the feasibility of extensive piloting. Finally, we also look at what initial checks researchers can do during the pilots and when they receive the data sets after fieldwork to assess the quality – before any of the actual analyses can begin. The chapter also pays attention to the specific challenges of designing and administrati...

Table of contents

• Cover
• Half Title
• Acknowledgements
• Title Page
• Copyright Page
• Contents
• Table List
• Sidebar List
• About the Author
• 1 Introduction
• 2 Sampling Approaches: How to Achieve Representativeness
• 3 Sampling Mode: How We Actually Collect the Data
• 4 Questionnaire Design: Asking the Right Questions
• 5 Ensuring Survey Quality: Piloting, Checks and Cross-Cultural Comparability
• 6 Conclusion
• Glossary
• References
• Index