Advances in Comparative Survey Methods

Multinational, Multiregional, and Multicultural Contexts (3MC)

Timothy P. Johnson, Beth-Ellen Pennell, Ineke A. L. Stoop, Brita Dorer, Timothy P. Johnson, Beth-Ellen Pennell, Ineke A. L. Stoop, Brita Dorer

• Publisher: Wiley
• Year: 2018
• Print ISBN: 9781118884980
• eBook ISBN: 9781118885017
• Edition: 1
• Topic: Social Sciences
• Subtopic: Probability & Statistics
• Index: Social Sciences

Covers the latest methodologies and research on international comparative surveys with contributions from noted experts in the field

Advances in Comparative Survey Methodology examines the most recent advances in methodology and operations as well as the technical developments in international survey research. With contributions from a panel of international experts, the text includes information on the use of Big Data in concert with survey data, collecting biomarkers, the human subject regulatory environment, innovations in data collection methodology and sampling techniques, use of paradata across the survey lifecycle, metadata standards for dissemination, and new analytical techniques.

This important resource:

• Contains contributions from key experts in their respective fields of study from around the globe
• Highlights innovative approaches in resource poor settings, and innovative approaches to combining survey and other data
• Includes material that is organized within the total survey error framework
• Presents extensive and up-to-date references throughout the book

Written for students and academic survey researchers and market researchers engaged in comparative projects, this text represents a unique collaboration that features the latest methodologies and research on global comparative surveys.

Section III
Cross‐cultural Questionnaire Design and Testing

6
Overview of Questionnaire Design and Testing

Julie A.J. de Jong¹, Brita Dorer², Sunghee Lee¹, Ting Yan³, and Ana Villar⁴

¹ Survey Research Center, University of Michigan, Ann Arbor, MI, USA

² GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany

³ Westat, Rockville, MD, USA

⁴ European Social Survey Headquarters, City, University of London, London, UK

6.1 Introduction

As multinational, multicultural, or multiregional (3MC) surveys have become more prolific in recent decades, corresponding methodological research has enjoyed a similar surge in growth, although it remains far more limited than survey research centered on data from single‐country surveys. This is true for all stages of the survey lifecycle including questionnaire design and associated pretesting (see Figure 34.1).

In 2010, Harkness and her colleagues published a comprehensive overview of the state of questionnaire design in a comparative perspective and acknowledged that both questionnaire design and pretesting research were scarce in the 3MC context. For example, there was little research considering how aspects of the interview context may intersect with questionnaire design, what the cultural frameworks of different populations may signify for design and analysis, or how the idea of “question meaning” is neither standardized across populations nor static over time in the same population [1]. In the intervening years, researchers engaged in comparative research have continued rigorous methodological examination of questionnaire design and testing challenges specific to 3MC research, with attention toward the gaps in this research area.

In this chapter, we first briefly discuss the operational challenges and choices specific to questionnaire design in 3MC surveys. Like its predecessors (see Refs. [1, 2]), we do not attempt to discuss the concepts of questionnaire design. Rather, we only focus on the choices and challenges that pertain specifically to 3MC surveys. We then consider the conceptual challenges in comparative questionnaire design – specifically, the role that culture and context hold in comparative research – followed by a brief discussion of comparability and measurement error in comparative questionnaire design. We also discuss the implications for questionnaire design where English is the most used lingua franca for source questionnaires in comparative surveys. Question evaluation is an essential part of the question design process, and here we review the testing approaches most commonly used as well as several recent advances in several specific evaluation approaches. Lastly, we touch on some of the newest advances in research related to questionnaire design and testing in the comparative context, including brief summaries of the literature on questionnaire design and testing published in the seven following chapters of this volume.

6.2 Review of Questionnaire Design and Testing in a Comparative Context

6.2.1 Questionnaire Design Models

There are several approaches to questionnaire design in the 3MC context (see Ref. [1] for a thorough discussion of each method). The first, ask‐the‐same‐question (ASQ), is typically achieved by producing one (or more) source questionnaire(s) and translating the source questionnaire(s) relevant for the study population(s). ASQ tends to be a popular choice because the approach targets comparability on the basis of similar wording across questionnaires, permitting comparison during analysis of both geographical and temporal data. ASQ also facilitates replicability in both of these dimensions, although research has shown that even nominally “accurate” translations do not necessarily produce comparable data [3].

A second approach used in 3MC questionnaire design, ask‐different‐questions (ADQ), is to ask the most salient questions in each target population in order to address the same underlying concepts, even if different questions or different indicators are used across different locations. The ADQ approach relies on achieving comparable measurement by asking questions that may differ in wording across cultures and countries. Although this approach stresses the standardization of meanings and aims for functional equivalence, item‐by‐item analyses across populations may be a concern since the questions are not the same across different groups [4].

A third approach to design is that of “decentering,” where questions are developed simultaneously in more than one language by translating back and forth between two or more languages in an iterative process, with a focus on conveying the conceptual essence rather than an exact translation. As versions are produced and compared, problematic cultural and linguistic elements are removed, which can ultimately result in target language instruments not only with lower cultural and linguistic bias but also with lower specificity and saliency.

It is worth mentioning that not all design options are available for every study or question; the specific study design, the target population, and the mode of data collection required may impose constraints on available options. The choice of approach will depend on various factors including the number of contexts involved, the available budget, and the experience of the researchers, who must weigh the advantages and disadvantages of the comparative question design options available and make informed choices based on their consideration of practical design parameters. Comparative researchers may elect to use both the ASQ and ADQ approach to questionnaire design not only for overall instrument development but also for individual survey items. A 3MC survey using the former approach may elect to replicate some series of questions in all countries while permitting an optional series of country‐specific questions at the end of instrument. In the latter approach, an instrument would include questions that are a blend of a common part (ASQ) and a country‐specific part (ADQ)...

Table of contents

• Cover
• Table of Contents
• Preface
• Section I: Introduction
• Section II: Sampling Approaches
• Section III: Cross‐cultural Questionnaire Design and Testing
• Section IV: Languages, Translation, and Adaptation
• Section V: Mixed Mode and Mixed Methods
• Section VI: Response Styles
• Section VII: Data Collection Challenges and Approaches
• Section VIII: Quality Control and Monitoring
• Section IX: Nonresponse
• Section X: Multi‐group Analysis
• Section XI: Harmonization, Data Documentation, and Dissemination
• Section XII: Looking Forward
• Wiley Series In Survey Methodology
• Index
• End User License Agreement