The SAGE Handbook of Survey Methodology

Christof Wolf, Dominique Joye, Tom W Smith, Yang-chih Fu, Christof Wolf, Dominique Joye, Tom W Smith, Yang-chih Fu

• Publisher: SAGE Publications Ltd
• Year: 2016
• Print ISBN: 9781446282663
• eBook ISBN: 9781473959040
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

Survey Methodology is becoming a more structured field of research, deserving of more and more academic attention. The SAGE Handbook of Survey Methodology explores both the increasingly scientific endeavour of surveys and their growing complexity, as different data collection modes and information sources are combined.

The handbook takes a global approach, with a team of international experts looking at local and national specificities, as well as problems of cross-national, comparative survey research. The chapters are organized into seven major sections, each of which represents a stage in the survey life-cycle:

• Surveys and Societies
• Planning a Survey
• Measurement
• Sampling
• Data Collection
• Preparing Data for Use
• Assessing and Improving Data Quality

The SAGE Handbook of Survey Methodology is a landmark and essential tool for any scholar within the social sciences.  

Part I Basic Principles

1 Survey Methodology: Challenges and Principles

Dominique Joye, Christof Wolf, Tom W. Smith and Yang-chih Fu

INTRODUCTION

There are a lot of reasons to publish a new handbook of survey methodology. Above all, the field of survey methodology is changing quickly in the era of the Internet and globalization. Furthermore, survey methodology is becoming an academic discipline with its own university programs, conferences, and journals. However, survey methodology could also be seen as a bridge between disciplines, resting on the shared methodological preoccupations between specialists of very different fields. These are some of the challenges we are addressing here.

Discussing the actual practices in many contexts is an invitation to think in a global perspective, along two directions. On the one hand, surveys are realized today all around the world in very different settings, which we call the globalization of surveys. But on the other hand, the ‘Total Survey Errors’ paradigm considers the complete survey life cycle and the interrelation of the different elements involved in the data collection process. That means that it would not be wise to pay too much attention to a single element at the risk of losing sight of the complete picture. This is of course valid for survey designers but also for estimating the quality and potential for use of existing surveys. A global perspective also requires a comparative frame. We even argue that integrating a comparative perspective from the beginning can enlighten many different aspects of survey design, even in a single national context.

These points will be developed throughout this handbook beginning in this introduction, with the idea of simultaneously providing a ‘state of the art’ and a perspective on the upcoming challenges. One important point in this respect is not to consider surveying as a technique but to consider it an integrated part of the scientific landscape and socio-political context. But first, we should explicate what we mean by a ‘survey'.

WHAT IS A SURVEY?

The Oxford Dictionary of Statistical Terms begins with a broad definition of surveys: ‘An examination of an aggregate of units, usually human beings or economic or social institutions’ (Dodge 2010 [2003]: 398). Although many authors, such as Ballou (2008), agree on the polysemy of the concept, a more precise definition is given by Groves et al. (2004): ‘A survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the large population of which the entities are members’ (p. 2). In this sense, the French word for survey, ‘enquête', the same term used for a criminal inquiry, denotes well this systematic quest for information.

Sometimes ‘surveying’ is defined as obtaining information through asking questions, in line with the German word for survey: ‘Befragung'. Dalenius (1985) recalls that observations are to be done according to a measurement process (measurement method and a prescription on its use) (Biemer and Lyberg 2003, see also Dodge 2010 [2003]: 399). That means that surveys defined in this sense share a lot of commonalities with other forms of data collection.

The idea of a systematic method for gathering observation includes for example exhaustive censuses as well as the use of a sample. In fact, some specialists explicitly limit surveys to data collection exercises conducted on samples (de Leeuw et al. 2008: 2). This handbook includes many chapters (Chapters 21, 22, 23) on the question of sampling, and the sample survey will be the first target, even though we see no reason to exclude by definition censuses which share a lot methodologically with surveys and are of great importance in the history of the quantitative observation of society.

‘Quantitative descriptors’ implies not only ‘numbers’ but also their interpretation, which in turn is placed in a broader interpretative frame. There is a process of operationalization that progresses from theory to measurement (Chapters 9, 14 and 34). In this sense, ‘descriptors', i.e., point estimates, can only be understood by taking into account the structure and functioning of a given society. In other words, how to build the measure of ‘items’ is also one of the main topics to address, and a full part of the handbook is dedicated to survey-based measurement (Chapters 14–20). One strength of this handbook is the attention it gives to measurement and survey quality.

The definition of survey nevertheless excludes a lot of approaches useful for social research that are outside the scope of this handbook (but see Bickman and Rog 2009). Generally, qualitative methods are not considered, as we focus on quantitative descriptors. However, in certain parts of the survey life cycle, qualitative methods are well established and important to consider, such as in pretesting (Chapter 24). Along the same lines, ‘big data', e.g., administrative data or data from the Internet, are not considered here because they are not organized a priori as a ‘systematic model for gathering information'. to rephrase Dalenius. Nevertheless, such data are becoming vital to understanding social life, and must be taken as complementary with surveys.1 In the last part of the handbook we will take into account the growing integration of surveys into a set of different sources of information (predominately Chapter 42, but also Chapters 40 and 41 in some aspects).

There are multiple ways of collecting information through surveys, and some distinctions between them are useful. Although a complete typology is outside the scope of this introduction, Stoop and Harrison (2012), for example, classify surveys based on the interrogatives who, what, by whom, how, when, where and why. Without mimicking their excellent chapter, we can discuss some lines.

In the ‘by whom’ category, different types of actors that sponsor activities can be mentioned:

• The scientific community tries to develop theory and analytical models in order to explain behavior, attitudes or values as well as the distribution of health, wealth and goods in given societies.
• The public administration; quantitative information is needed for governance – it is no coincidence that the words ‘statistic’ and ‘state’ have the same root – and an important task of a state is to assess the number of inhabitants or households it contains.
• Commercial enterprises need knowledge about their clients and their clients’ reactions to their products in order to be as efficient as possible in their markets.
• Mass media are a special actor in the survey field and were mentioned as a particular category already in the 1980s (Rossi et al. 1985). Sometimes, they use results of polls more as a spectacular result to gain an audience rather than as a piece of systematic knowledge about society. That is part of the debate about the concept of public opinion (Chapter 5).

This type of distinction is also of importance when thinking for example about ethical aspects of surveys (Chapter 7). Of course, the boundaries between these actors are not always clear and depend on the national context, at least for the relation between administrative organizations and academia, and this could be important for the definition of measurement tools (Chapters 5 and 20). Nevertheless, we can expect that these actors have different expectations of surveys, their quality and their precision. In fact, most of the examples used in this book are taken from the academic context, implying that we focus more on the link between theory and measurement (Chapters 9 and 14) than other indicators of quality used, for example, in official statistics.2 This example also reminds us that no absolute criterion for quality exists independently of the goals. This is clearly stated in the definition of quality given by Biemer and Lyberg, ‘Quality can be defined simply as ‘fitness for use.’ In the context of a survey, this translates to a requirement for survey data to be as accurate as necessary to achieve their intended purposes, be available at the time it is needed (timely) and be accessible to those for whom the survey was conducted’ (2003: 9). This handbook hasmany chapters assessing data quality and aspects that can jeopardize quality (Chapters 34 to 39), an important aspect that must also be taken into account in the design state of a survey (Chapter 16 for example). This is of prime importance when developing the total survey error frame (Chapter 3).

We can further distinguish between the different ways to acquire information, the ‘how’ mentioned by Stoop and Harrison (2012). The first distinction is between modes of data collection, even though the boundaries between them are blurring, and multi or mixed modes are more and more often utilized (Chapter 11). We will come back to this later in the introduction when considering the development of surveys during the last century.

Who (or what) are the units of analysis of the survey, is another question. In most of the chapters in this handbook it would be individuals or households, but this is clearly a choice: a ...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Contents
• Illustration List
• Illustration List
• Notes on the Editors and Contributors
• Preface
• Part I Basic Principles
• 1 Survey Methodology: Challenges and Principles
• 2 Survey Standards
• 3 Total Survey Error: A Paradigm for Survey Methodology
• 4 Challenges of Comparative Survey Research
• Part II Surveys and Societies
• 5 Surveys and Society
• 6 Defining and Assessing Survey Climate
• 7 The Ethical Issues of Survey and Market Research
• 8 Observations on the Historical Development of Polling
• Part III Planning a Survey
• 9 Research Question and Design for Survey Research
• 10 Total Survey Error Paradigm: Theory and Practice
• 11 Survey Mode or Survey Modes?
• 12 Surveying in Multicultural and Multinational Contexts
• 13 Surveys in Societies in Turmoil
• Part IV Measurement
• 14 What Does Measurement Mean in a Survey Context?
• 15 Cognitive Models of Answering Processes
• 16 Designing Questions and Questionnaires
• 17 Creating a Good Question: How to Use Cumulative Experience
• 18 Designing a Mixed-Mode Survey
• 19 The Translation of Measurement Instruments for Cross-Cultural Surveys
• 20 When Translation is not Enough: Background Variables in Comparative Surveys1
• Part V Sampling
• 21 Basics of Sampling for Survey Research
• 22 Non-probability Sampling
• 23 Special Challenges of Sampling for Comparative Surveys
• Part VI Data Collection
• 24 Questionnaire Pretesting
• 25 Survey Fieldwork
• 26 Responsive and Adaptive Designs
• 27 Unit Nonresponse
• 28 Incentives as a Possible Measure to Increase Response Rates
• Part VII Preparing Data for Use
• 29 Documenting Survey Data Across the Life Cycle
• 30 Weighting: Principles and Practicalities
• 31 Analysis of Data from Stratified and Clustered Surveys
• 32 Analytical Potential Versus Data Confidentiality – Finding the Optimal Balance
• 33 Harmonizing Survey Questions Between Cultures and Over Time
• Part VIII Assessing and Improving Data Quality
• 34 Survey Data Quality and Measurement Precision
• 35 Nonresponse Error: Detection and Correction
• 36 Response Styles in Surveys: Understanding their Causes and Mitigating their Impact on Data Quality
• 37 Dealing with Missing Values
• 38 Another Look at Survey Data Quality
• 39 Assessment of Cross-Cultural Comparability
• Part IX Further Issues
• 40 Data Preservation, Secondary Analysis, and Replication: Learning from Existing Data
• 41 Record Linkage
• 42 Supplementing Cross-National Survey Data with Contextual Data
• 43 The Globalization of Surveys
• Index