Cognitive Interviewing

A Tool for Improving Questionnaire Design

Gordon B. Willis

• Publisher: SAGE Publications, Inc
• Year: 2004
• Print ISBN: 9780761928041, 9780761928034
• eBook ISBN: 9781483389301
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

"As both an academic instructor in questionnaire design and a research design methodologist for the federal government, I feel this book is very timely, useful for students and practitioners, and unique in its use of real world practical examples that most everyone can relate."
—Terry Richardson, General Accounting Office

"The combination of theory and practical application will make this a useful book for students as well as professionals who want to learn how to incorporate cognitive interviewing into the questionnaire design process."
—Rachel Caspar, RTI International

The design and evaluation of questionnaires—and of other written and oral materials—is a challenging endeavor, fraught with potential pitfalls. Cognitive Interviewing: A Tool for Improving Questionnaire Design describes a means of systematically developing survey questions through investigations that intensively probe the thought processes of individuals who are presented with those inquiries. The work provides general guidance about questionnaire design, development, and pre-testing sequence, with an emphasis on the cognitive interview. In particular, the book gives detailed instructions about the use of verbal probing techniques, and how one can elicit additional information from subjects about their thinking and about the manner in which they react to tested questions. These tools help researchers discover how well their questions are working, where they are failing, and determine what they can do to rectify the wide variety of problems that may surface while working with questionnaires.

Cognitive Interviewing is ideally suited as a course text for advanced undergraduate and graduate research courses across the social sciences. Professional researchers and faculty in the social sciences, as well as practice fields such as medicine, business, and education, will also find this an invaluable reference for survey research. There is no other book on the market that covers cognitive interviewing as applied to questionnaire design.

PART I

---

Orientation and Background

1

---

Introduction to Cognitive Interviewing

What Is (and Isn’t) Cognitive Interviewing?

---

Across a wide range of disciplines—including health research, market research, political science, and the social sciences in general—we endeavor to develop materials that effectively convey information, such as brochures, cover letters, statistical reports, oral presentations, and computer-based user applications. Very often, we also request information in return by administering data collection instruments, such as survey questionnaires, tax forms, medical forms, and so on. Cognitive interviewing is a general method that developers of such materials can use to critically evaluate this transfer of information. In particular, we use cognitive interviewing techniques to study the manner in which targeted audiences understand, mentally process, and respond to the materials we present—with a special emphasis on potential breakdowns in this process. This book focuses on the cognitive interview as it has most often been used, that is, to evaluate survey questionnaires. More generally, however, researchers can adapt cognitive interviewing to test virtually any type of material, whether delivered orally or in writing, to identify difficulties that may otherwise go unrecognized.

To demonstrate the generality of the cognitive interview, I first refer to an example having little to do directly with survey questionnaires (taken from Willis, Reeve, & Barofsky, in press). Consider an academic subject that is the bane of students everywhere: the math word problem, viewed by many as a particularly insidious form of verbal material. As part of an elementary school educational research project (Fuson & Willis, 1988), I evaluated a set of word problems that included the following:

The poodle has 9 puppies

The collie has 5 puppies

How many more puppies does the poodle have?

To study young children’s problem-solving behavior, I read this aloud to a number of first and second graders in a series of one-on-one interviews. To my initial surprise, several gave the same incorrect answer: “none” or “zero.” Because I was interested in delving further into the children’s thought processes, within each interview I asked additional questions that intensively probed the basis for their answers. This mainly consisted of asking them “How did you get that?”—a practice that is, by definition, cognitive interviewing, although I did not at the time know it by that name. Use of this follow-up probe did provide insights into the reason that this variety of word problem presents particular difficulties for some children. A representative child’s explanation was “You told me that she had nine puppies. But then she didn’t have any more, so it’s none!”

It seemed evident that, from the child’s point of view, although it was perhaps nice to hear about the collie, this was not especially relevant to solving the word problem, as it seemed to require only a consideration of the reproductive history of the poodle. Based on this observation, the investigators postulated that some children interpreted the term “more” strictly as an increase in quantity, rather than as a comparison (i.e., “more than something else”). Given the objective of making the comparison clearer, the indicated solution was to rewrite the problem to instead ask “How many more puppies does the poodle have than the collie?” Based on further interviews, it appeared that this modification did facilitate understanding and helped the children to learn to successfully solve this variety of problem even when stated in its original, more ambiguous form.

The “puppy word problem” example is, in a nutshell, cognitive interviewing. In application to survey questionnaire evaluation, we test a range of target questions that may pose difficulties that generally originate in the cognitive processing of those questions. The questionnaire designer may intend one interpretation yet find that individuals presented with the question adopt an alternate understanding that, in retrospect, appears quite reasonable. If cognitive interviewing leads us to appropriate findings or insights, we may then modify our materials to enhance clarity. This eases the cognitive processing demands for respondents and ultimately increases the likelihood that they will respond in a thoughtful manner and give accurate answers. When we bring about this result, we have achieved our goal of improving the question through cognitive techniques.

AN IMPORTANT EXCLUSION: A DIFFERENT TYPE OF COGNITIVE INTERVIEWING

I have claimed that a range of developmental and evaluative activities (such as the word problem study described above) implicitly involve cognitive interviewing, even when these activities are not termed as such. A potential source of confusion is that the converse is also true: The label “cognitive interviewing” is sometimes applied to an activity that is outside the realm covered in this book and that involves a somewhat different category of investigation. This type of cognitive interview, introduced by Fisher and Geiselman (1992), does not focus on the testing and development of verbal materials, but is mainly used in law enforcement to enhance the accuracy and reliability of information retrieval by eyewitnesses to crimes. This method has been applied widely, and a literature review of the “cognitive interview” will produce many results related to the Fisher and Geiselman application. However, their interrogation-oriented application has developed separately, involves somewhat different techniques, and serves a fundamental purpose that departs from the goal of evaluating materials such as survey questions. Further, to my knowledge, there is virtually no overlap—and very little communication—between practitioners of these alternate practices. As such, I will not review or otherwise consider the eyewitness-related application. Admittedly, survey-oriented cognitive researchers may “interrogate” people, but not generally for the purpose of solving a crime (notwithstanding the fact that we may consider deficient survey questions to represent a criminal violation of acceptable design practice!).

A Broader Perspective: The Cognitive Testing Process

---

My opening description of the puppy word problem is somewhat simplistic. In the world of survey questionnaire pretesting, the cognitive interview is usually conducted within a broader testing environment, as part of a sequence of activities that, in aggregate, are referred to as cognitive testing. Such testing generally incorporates the properties listed in Text Box 1.1.

Each of the general features in Text Box 1.1 has exceptions, if only because cognitive testing is very flexible and can be tailored to particular situations. However, considering the approach described to be fairly typical, I next turn to an example (Figure 1.1) that involves a hypothetical (but also typical) survey questionnaire, in order to illustrate how cognitive testing might be conducted to evaluate a set of questions or “items” in a real-world context.

Text Box 1.1

General Features of the Cognitive Testing Process

---

Cognitive Focus

We study the cognitive processes that respondents use to answer survey questions; in particular, their comprehension, recall, decisions and judgment, and response processes. However, our goal is to detect a wide range of problems in survey questionnaires.

Timing

Cognitive interviews are typically done between initial drafting and administration in the field, to pretest a survey questionnaire.

Interviewers

Specially trained cognitive interviewers conduct interviews of volunteer subjects.

Specialized Recruitment

Researchers often select subjects who have particular characteristics of interest (e.g., the elderly, those who have used illicit drugs in the past 12 months, teenagers who have used chewing tobacco).

Use of Verbal Report Procedures

To identify problems and limitations, cognitive interviewing involves both think-aloud and verbal probing procedures.

Reliance on a Range of Probing Techniques

Most interviewers rely heavily on verbal probe questions when interviewing subjects (see Appendix 1 for examples).

Emphasis on Both Overt and Covert Problems

Our objective is not only to obtain evidence of clear, overt problems but to identify hidden or covert ones that are not evident simply by analyzing the normal question-answer sequence.

Laboratory Environment

Interviews are often conducted in a dedicated cognitive laboratory, and cognitive interviewing is commonly referred to as “laboratory testing.” However, having a physical laboratory is not a critical requirement of cognitive interviewing.

Modest Sample Sizes

Small numbers of individuals are tested—generally between 5 and 15 in an interviewing round—before the findings are reviewed and interpreted.

Iterative Testing

Following a round of testing, review, and modification, the revised questionnaire is often tested in a further round; this is a major strength of cognitive testing.

Flexible Application

Although the focus of cognitive testing is limited to the questionnaire, rather than the entire survey process, it is flexible enough to apply to different survey environments (e.g., telephone, face-to-face, paper, or Internet administration). It can be used for factual or attitudinal surveys, and for surveys of individuals (population surveys) as well as those of businesses (establishment surveys).

Information Rather Than Validation

Cognitive interviewing is a useful means for testing survey questions but does not validate questions in a formal sense—we strive to produce better questions, but generally have no proof, statistical or otherwise.

Advisory Nature

The practice of cognitive interviewing does not in itself result in improved questionnaires. Cognitive testing serves as “question inspection” and identifies potential problems, but solving these problems requires a skilled questionnaire designer (see Appendix 2 for some sample testing results).

Cognitive testing of a questionnaire containing items in Figure 1.1 could be accomplished through the following steps:

1. Develop a testing plan. Do we need to recruit individuals with particular characteristics? How many interviews should we plan, and how many interviewers do we need? Where and when will the interviews be done?
2. Develop a cognitive testing protocol consisting of the tested (target) questions along with a variety of probe questions to supplement the targeted questions, such as “What time period are you thinking of here?” or “What did you go to the dentist for?” (see Appendix 1).
3. Recruit subjects and invite them into our cognitive lab at appointed times for their interviews.
4. Administer the questionnaire in private, one-on-one interviews, and apply cognitive interviewing techniques, in particular (a) have subjects think aloud as they answer questions, and (b) administer the cognitive probes.
5. With each subject’s consent, audiotape or videotape the interview, write notes during the interview, and review the notes and recordings after each interview.
6. If a team is involved, write a testing report (see Appendix 2) and meet with project staff to review the results.
7. Make modifications to the questions based on our findings.
8. If indicated and time permits, conduct a further testing round and then reevaluate the questionnaire.

Part A: Dental Visits

Q1.	How many times did you go to the dentist in the past year?
Q2.	The last time you visited the dentist, did you have your teeth cleaned?
Q3.	The last time you visited the dentist, did you receive any other treatment?
Q4.	The last time you visited the dentist, did you have any X-rays or other diagnostic tests?

---

Figure 1.1

Sample of Dental Questions Submitted to Cognitive Testing

After carrying out this sequence, the investigators might determine that the targeted questions pose no serious difficulties. This is itself a finding, and provides some assurance that one can go ahead and collect survey data with some confidence. More likely, however, the researchers will make the types of observations in Figure 1.2, which I have assembled from archival reports summarizing cognitive interviews of these target questions. (See Chapter 11 for specific ways to record and process cognitive interview results.)

(a)	Three of nine subjects interpreted “the dentist” to mean their main dentist but weren’t sure whether to include oral surgeons, dental hygienists, etc.
(b)	The term “past year” was interpreted variably. One person thought this meant the prior calendar year, as opposed to the past 12 months. Yet another subject reported visits since January 1 of this year.
(c)	Overall, subjects were not that sure of the correctness of their answers; five reported that they couldn’t be confident how many dental visits they had made, and three stated that they had assumed it was two times because that is their usual pattern. Overall, the answers given appeared to be “ballpark estimates.”

---

Figure 1.2

Sample Aggregate Findings Based on the Compilation of Interviewer Notes

Based on such results, the designers might consider a few changes to the questions:

1. Through question rewording, make clearer whether we mean “their main dentist,” “any dentist,” or “a dentist, oral surgeon, dental hygienist.”
2. Select “the past 12 months” as the timeframe, as opposed to “the past year.”
3. Review our measurement objectives. What level of precision is really needed? If the goal is to broadly classify people into levels of use of dental care (e.g., those who haven’t gone at all, those who went one to three times, and those going more than three times), an estimate may be sufficient. If ...

Table of contents

• Cover Page
• Title
• Copyright
• Contents
• Preface
• PART I: ORIENTATION AND BACKGROUND
• PART II: THE INTRICACIES OF VERBAL PROBING
• PART III: THE COGNITIVE TESTING PROCESS
• PART IV: OTHER ISSUES AND TOPICS
• Appendix 1: Example of Cognitive Testing Protocol
• Appendix 2: Examples of Findings From Cognitive Testing Reports
• References
• Index
• About the Author