Advances in Questionnaire Design, Development, Evaluation and Testing

Paul C. Beatty, Debbie Collins, Lyn Kaye, Jose-Luis Padilla, Gordon B. Willis, Amanda Wilmot, Paul C. Beatty, Debbie Collins, Lyn Kaye, Jose-Luis Padilla, Gordon B. Willis, Amanda Wilmot

• Publisher: Wiley
• Year: 2019
• Print ISBN: 9781119263623
• eBook ISBN: 9781119263647
• Edition: 1
• Topic: Mathematics
• Subtopic: Probability & Statistics
• Index: Mathematics

A new and updated definitive resource for survey questionnaire testing and evaluation

Building on the success of the first Questionnaire Development, Evaluation, and Testing (QDET) conference in 2002, this book brings together leading papers from the Second International Conference on Questionnaire Design, Development, Evaluation, and Testing (QDET2) held in 2016. The volume assesses the current state of the art and science of QDET; examines the importance of methodological attention to the questionnaire in the present world of information collection; and ponders how the QDET field can anticipate new trends and directions as information needs and data collection methods continue to evolve.

Featuring contributions from international experts in survey methodology, Advances in Questionnaire Design, Development, Evaluation and Testing includes latest insights on question characteristics, usability testing, web probing, and other pretesting approaches, as well as:

• Recent developments in the design and evaluation of digital and self-administered surveys
• Strategies for comparing and combining questionnaire evaluation methods
• Approaches for cross-cultural and cross-national questionnaire development
• New data sources and methodological innovations during the last 15 years
• Case studies and practical applications

Advances in Questionnaire Design, Development, Evaluation and Testing serves as a forum to prepare researchers to meet the next generation of challenges, making it an excellent resource for researchers and practitioners in government, academia, and the private sector.

Part I
Assessing the Current Methodology for Questionnaire Design, Development, Testing, and Evaluation

1
Questionnaire Design, Development, Evaluation, and Testing: Where Are We, and Where Are We Headed?

Gordon B. Willis

Division of Cancer Control and Population Sciences, National Cancer Institute, National Institutes of Health, Rockville, Maryland, USA

Although questionnaire development, evaluation, and testing (QDET) could be considered a fairly specific area of survey methods, it spans a wide range of activities and perspectives that the chapters in this book endeavor to describe and to further develop. This opening chapter – itself based on my keynote address delivered at the 2016 International Conference on Questionnaire Design, Development, Evaluation, and Testing (QDET 2) – outlines the contours of the QDET field and delineates the major trends, developments, and challenges that confront it. Further, I attempt to project into the future, both in terms of promising areas to pursue, as well as the types of methodological research that will further this goal. To this end, I consider three basic questions:

1. What is the current state of the art, and of the science, of QDET?
2. Whatever that may be, how important is methodological attention to the questionnaire, in the present world of information collection?
3. As information collection needs evolve, how can the QDET field anticipate new trends and directions?
   In addressing these issues, I make reference to chapters within this volume that relate to each of them.

1.1 Current State of the Art and Science of QDET

A half‐century ago, Oppenheim (1966, p. vii) stated that “The world is full of well‐meaning people who believe that anyone who can write plain English and has a modicum of common sense can produce a good questionnaire.” Although that viewpoint may to some extent persist, there has been considerable evolution in the status of questionnaire design within survey methods, and researchers in the social and health sciences especially have increasingly focused on question design as a scientifically backed endeavor, as opposed to a skill attained simply through familiarity with the use of language (Fowler 1995; Schaeffer and Dykema 2011b). Further, researchers and practitioners assert that many pitfalls in survey measurement derive from characteristics of survey questions that produce error – especially in the form of response error, e.g. the departure of a reported value from the true score on the measure of interest (Sudman and Bradburn 1982). Finally, as a vital component of the Questionnaire Design, Development, Evaluation, and Testing (QDET) approach, evaluation and testing – the “E” and “T” within the acronym – have become pronounced as means for assessing the performance of survey items in minimizing such error.

1.1.1 Milestones in QDET History: The Quest to Minimize Error

It was the challenge of controlling response error that largely motivated the organizers of the QDET2 conference, as this has been a consistent theme throughout the history of questionnaire design. To a considerable extent, efforts in this area have focused on understanding sources of error through modeling the survey response process, based on the supposition that humans need to think in order to answer survey questions, and on the observation that they can experience difficulty in doing so. These efforts have a relatively longer history than is sometimes acknowledged. Some highlights of this development are summarized in Table 1.1.

Table 1.1 Selected milestones in the history of questionnaire design, evaluation, and pretesting as a scientific endeavor.

Description of cognitive probing: Cantril and Fried (1944) A systematic set of rules of question design: Payne (1951) A psychological model of the survey response process: Lansing et al. (1961) The 1978 Royal Statistical Society and Social Science Research Council (U.K.) seminar on retrospective and recall data in social surveys (Moss and Goldstein 1979) A cognitive/motivational model of survey response: Cannell et al. (1981) The 1983 CASM Seminar (CNSTAT Advanced Research Seminar on Cognitive Aspects of Survey Methodology; Jabine et al. 1984) The four‐stage model of the survey response process: Tourangeau (1984) The Zentrum fur Umfragen, Methoden und Analysen (ZUMA)‐sponsored international conference on social information processing and survey methodology (Hippler et al. 1987) The Second Advanced Research Seminar in the Cognitive Aspects of Survey Methodology (CASM II, 1997) The International Conference on Question Development, Evaluation, and Testing (QDET 1, 2002) The International Conference on Question Design, Development, Evaluation, and Testing (QDET 2, 2016)

Within this series of events, a key turning point in QDET history, associated with the 1983 conference on the Cognitive Aspects of Survey Methodology (now referred to as CASM 1), was the introduction by Tourangeau (1984) of the Four‐Stage model of survey response, which infused survey methodology with key concepts of cognitive psychology by specifying the importance of Comprehension, Recall, Judgment/Estimation, and Response processes. This model was elegant in its simplicity and has served to capture much of what goes wrong with survey questions at each of these stages. The model was operationalized at the National Center for Health Statistics (NCHS) though the inception of the NCHS Cognitive Lab in the 1980s (Royston et al. 1986; Sirken and Fuchsberg 1984; Sirken et al. 1988); as well as by the development of the cognitive laboratories at the US Census Bureau (DeMaio and Rothgeb 1996) and Bureau of Labor Statistics; and also at Statistics Canada and several European survey organizations (Prüfer et al. 2003). The embedding of the Tourangeau Four‐Stage model into the larger rubric of CASM – along with the advent of the cognitive labs – in conjunction worked to propel the science of QDET further over the following decades (Jobe and Mingay 1991; Willis 2005).

1.1.2 Influences from Outside of Cognitive Psychology

There have, however, always been indications that the Four‐Stage model, within the CASM rubric, did not tell the entire story concerning question function. A complementary, if not alternative, strand that has existed throughout QDET history concerns not how to ask questions, but rather what to ask – are we asking the right question in the first place? This notion has persisted, under different guises – and can be thought of as a non‐statistical analog of specification error (Lazarsfeld 1959) – in simple form, where the investigator desires one thing, but measures another. In this vein, there was the associated recognition that the Four‐Stage cognitive model did not capture all problem types revealed through cognitive testing. In response, Willis et al. (1991) added Logical/Structural errors, as a fifth element, and where problems are more “in the question” than “in the respondent” – that is, in the interaction between the two, in a conceptual sense.

As a simple example, item testing revealed that physical activity questionnaires devised for university graduates were found to fail to function adequately for low‐income Hispanic women, but not because of failures that could be cleanly tied to the four proposed cognitive stages. Rather, the questions...

Table of contents

• Cover
• Table of Contents
• List of Contributors
• Preface
• Part I: Assessing the Current Methodology for Questionnaire Design, Development, Testing, and Evaluation
• Part II: Question Characteristics, Response Burden, and Data Quality
• Part III: Improving Questionnaires on the Web and Mobile Devices
• Part IV: Cross-Cultural and Cross-National Questionnaire Design and Evaluation
• Part V: Extensions and Applications
• Index
• End User License Agreement