Measuring Psychological Responses To Media Messages

Annie Lang

• Publisher: Routledge
• Year: 2014
• Print ISBN: 9780805807172
• eBook ISBN: 9781136695599
• Topic: Languages & Linguistics
• Subtopic: Communication Studies
• Index: Languages & Linguistics

Characterized by its multi-level interdisciplinary character, communication has become a variable field -- one in which the level of analysis varies. This has had important ramifications for the study of communication because, to some extent, the questions one asks are determined by the methods one has available to answer them. As a result, communication research is characterized by the plethora of both qualitative and quantitative approaches used by its practitioners. These include survey and experimental methods, and content, historical, and rhetorical analyses.

A variety of tools has been developed in cognitive psychology and psychophysiology which attempts to measure "thinking" without asking people how they do it. This book is devoted to exploring how these methods might be used to further knowledge about the process of communication. The methods chosen have all been used extensively in cognitive and experimental psychology. Each chapter in this book is designed to describe the history of the method being introduced, the theory behind it, how to go about using it, and how it has already been used to study some area of communication. The methods introduced here vary widely in terms of the amount of equipment and training needed to use them. Some require only theoretical knowledge and a paper and pencil; others require more elaborate hardware and software for implementation. These methods also vary widely in terms of what sorts of variables they can be used to measure. Some of them adapt quite readily to traditional communication variables like persuasion, attitude change, and knowledge; others are more applicable to process type variables such as attention, arousal, involvement, encoding, and retrieval.

1	Think-Aloud and Thought-List Procedures in Investigating Mental Processes

Michael A. Shapiro

Cornell University

Think-aloud and thought-list techniques ask a subject to report all his or her thoughts while doing some task. The goal is to “increase the temporal density of observations of [mental] behavior” (Ericsson & Simon, 1978, p. 1)—to capture what actually goes through a person’s mind as he or she performs a task. The investigator can then use these recorded thoughts to build a picture of and test hypotheses about the mental processes used in the task.

Communication researchers have used these techniques to investigate a variety of communication tasks, including what happens when a person processes a persuasive message (Cacioppo & Petty, 1981), what writers think about when constructing a news story (Pitts, 1982; Pitts, 1989), and what rules people follow in conversations (Daly, Weber, Vangelisti, Maxwell, & Neel, 1989).

Communication researchers are vitally concerned with the mental processes involved in the creation of meaning from various communications in both theoretical and applied communication research. Think-aloud techniques are often the only way to gather detailed information about those mental processes. However, these deceptively simple techniques are actually full of traps for the unwary. Even carefully designed studies can generate invalid data.

This chapter briefly traces the historical and theoretical underpinnings of the use of verbal reports. Next it provides some guidelines for designing and analyzing think-aloud studies.

HISTORY AND THEORY

Attempts to discover the nature of mental processes by examining one’s own thoughts date back at least to Descartes. A full discussion of the philosophical considerations is beyond the scope of this chapter. However, early in this century psychologists rejected the notion that trained observers could systematically observe and analyze their own thinking, a technique called introspection (Ericsson & Simon, 1981; Gardner, 1985).

In the 1970s, several investigators began using subjects’ verbal descriptions of thought processes to investigate cognition. Most prominent were Newell and Simon (1972) who used such techniques to gain insight into human problem solving as input into models of problem solving processes. These studies did not use introspection in the classical sense. Subjects were not asked to analyze thought processes. Instead, ordinary untrained people were asked to report everything they were thinking while solving a problem.

Some investigators wondered whether these think-aloud techniques were any more valid than introspection. For example, Nisbett and Wilson (1977) pointed to numerous studies in which subjects were either unable to report being influenced by an effective experimental treatment or wrong about the nature or direction of the treatment’s influence on their own thinking and behavior. Nisbett and Wilson believed that in many cases, when asked to report on their own mental processes, subjects used naive theories about thinking instead of observing their own thinking. (See also Nisbett & Bellows, 1977; Wilson & Nisbett, 1978.)

This prompted Ericsson and Simon to specify a model that predicted when verbal reports of mental processes would be valid observations and that distinguished their verbal report techniques from classical introspectionism (Ericsson & Simon, 1978, 1979, 1980, 1981). They said that the problems Nisbett and Wilson identified were by-and-large predicted by the model.

The basic tenant of the model is simple. As a person does a task, he or she can report relatively accurately on information that passes through short-term memory (STM).¹ However, that is the only information a subject can reliably report. Three types of problems arise in think-aloud procedures as a result.

1. Not all the information an investigator might want passes through STM.

2. A subject’s ability to report the contents of STM are limited.

3. Investigators must be cautious that subjects are indeed reporting on the contents of STM instead of taking mental shortcuts.

4. The process of reporting on mental contents may interfere with task performance.

The next section discusses these problems in more detail by exploring some of the basic techniques of think-aloud studies.

DESIGNING A STUDY

The basics of a think-aloud study in communication are simple. The subject is given some communication task—listening to a persuasive message, for example. He or she is given instructions to report everything that went through his or her mind while doing the task. The investigator then takes a transcript of those reports and analyzes them, usually in some form of content analysis. Each of these stages—task, instructions, reporting, and analysis—has its own set of problems.

The Task

Investigators may want to know what subjects are thinking in almost any communication task—listening to a persuasive message, watching television, deciding which newspaper stories to read, or carrying on a conversation. However, some tasks are not suitable for think-aloud studies. As discussed earlier, only thoughts that pass through some form of short-term memory can be accurately reported. But some communication tasks are so practiced by adulthood that under normal circumstances they are relatively automatic. To somewhat oversimplify, that means that little, if any, processing capacity is needed to accomplish these tasks, and subjects may not be consciously aware that they are doing them. Such tasks can include attentional (Schneider, Dumais, & Shiffrin, 1984; Shiffrin & Schneider, 1977) and memory processes (Birnbaum, Taylor, Johnson, & Raye, 1987; Hasher & Zacks, 1979; Sanders, Gonzalez, Murphy, Liddle, & Vitina, 1987).

Even new mental procedures may become more automatic with practice. Information involved in such automatic procedures does not seem to pass through short term memory. Therefore, any task that depends heavily on highly automatic processes is probably not a good candidate for a think-aloud study. For example, thought-list studies have been used extensively to show that fewer thoughts are generated during low involvement or heuristic responses to persuasive messages than during higher involvement or more thoughtful processing. However, because many investigators believe that low involvement and most heuristic processing is usually highly automatized, think-aloud studies are unlikely to reveal much about thoughts during those specific processes (Chaiken, Liberman, & Eagly, 1989; Petty & Cacioppo, 1986). In addition, any instructions that encourage thinking aloud are likely to change the subject’s mental strategy to some less automatic process.

Investigators should also keep in mind that experts in a particular area may have developed more automated procedures than novices. Although often valid, think-aloud differences between experts and novices should be viewed with caution. Such differences may, in part, be an artifact of increased automaticity in expert thinking processes.

Instructions and Probes

Instructions given to a subject and probes for additional information can have a dramatic effect on what a subject reports during a think-aloud session. Probably the most important thing to keep in mind is that, in general, mental procedures themselves are inaccessible to subjects. So, for example, it makes sense to ask a subject to report everything that goes through his or her mind while reading a newspaper headline. But it probably does not make sense to ask, “how did you decide the story was about pollution?” Questions and probes seeking information about the influence of motives, reasons for attending to a particular part of a message, and procedures for accessing a particular memory are all suspect. Ericsson and Simon concluded that “one shouldn’t ask subjects why, but should simply use verbalization instructions to discover what information is attended to or is stored.” (Ericsson & Simon, 1979, p. 43). “Proper protocols ask subjects to report their thoughts, not to explain them” (Russo, Johnson, & Stephens, 1989).

Another problem is that subjects seem to have a strong tendency to draw on other kinds of information when reporting on mental contents and processes. It may be mentally easier for a subject to draw on general knowledge about how these mental procedures are performed (Ericsson & Simon, 1978) or to use naive theories about mental processing—theories that are often wrong (Nisbett & Wilson, 1977). Such mental shortcuts may cause subjects to infer or deduce mental contents rather than just observe them. Obviously, investigators want to discourage these tendencies.

That can get tricky. For example, it seems natural to ask the subject to report everything, not to censor him or herself. That may work, or it may cause the subject to set up a monitoring procedure to make sure that everything is reported. That monitoring procedure may interfere with the subject’s natural thought processes (Ericsson & Simon, 1979). Both the instructions told to a subject and the probes given to encourage information in a particular area may cause a subject to report inferences instead of memories.

Without knowing anything else about natural thinking processes in a particular situation, the safest instruction is the most general—for example, “tell me everything that passed through your head while you were listening to the message.” But once such procedures establish that a particular kind of thought is common, more specific questions and probes can be useful in getting information about certain kinds of thoughts (e.g., evaluative thoughts in attitude change experiments.)

However, such specific questions may imply that the investigator expects a certain kind of information to be available. Cooperative subjects may use various mental processes to generate that information, even if it was not available during message processing.

On the other hand, subjects given general instructions to report “everything” may decide for themselves that certain thoughts do not interest the researcher. To combat this, some investigators have encouraged subjects not to censor themselves or have used “priming” instructions (Woodside, 1983; Wright, 1980). Priming instructions mention several thought categories that should be reported without emphasizing any one category. Wright (1980) admitted that the effect of such instructions is largely unknown.

Russo, Johnson, and Stephens (1989) maintained that it is probably better to lose some detail in the verbal reports than to risk the possibility that reporting will change the process of interest. They recommended that subjects should be instructed to preserve naturalness over completeness or speed. However, the effect of such instructions is largely untested.

Wright (1980) suggested four conditions that should encourage honest reporting: (a) the subject’s self-esteem is not at stake; (b) the subject does not think the report will be used to judge him or her; (c) valid reporting can be gauged and will be rewarded, usually by social approval; (d) subjects think the investigator has independent means to judge the validity of the report. Unfortunately, telling the subject that the validity of his or her report can and will be gauged is more likely to elicit responses aimed at impression management. In most cases, it is probably best to simply ensure anonymity and, if possible, to select situations in which self-esteem and social approval are not an issue.

Reporting Thoughts

An investigator must record each subject’s thoughts so that those thoughts can be analyzed later. There are several considerations here. To get the most accurate and valid information, the communication task should not interfere with reporting and reporting should not interfere with performing the task. In addition, memory failures are least likely to distort the results if the subject reports thoughts while doing the communication task and not at some later time. Unfortunately, most realistic communication situations make it difficult to report on thoughts without interfering with the task.

Interference with Task Performance. Whether talk-aloud reporting interferes with task performance depends in part on what aspect of performance interests the investigator. According to Ericsson and Simon (1980) the primary consideration is the number and type of intermediate processes between generating a thought and reporting that thought. For example, less recoding is necessary when the thought is verbal and needs no recoding to report verbally. However, a verbal description of response to a picture may require some recoding.

The probe itself may require the subject to set up additional mental procedures. For example, if the probe for an advertisement is “evaluate the advertisement,” the subject must first decide if a thought is an evaluation before reporting it. Some probes may additionally ask subjects to verbalize about aspects of the task that he or she would not ordinarily attend to. For example, at what part of a picture a subject is looking at any given moment.

In reviewing the literature, Ericsson and Simon (1979) found that thinking aloud interferes very little with task performance if the probe requires reporting on normally available verbal information. However, overt verbalization may slow down task performance and may facilitate memory retrieval and storage.

If there is some criterion for task performance available, it may be worthwhile to check and see how much thinking aloud influenced that criterion. In attitude change studies, for example, it may be worthwhile to compare attitude change for a group that did the thought-list procedure to a group that did not.

Some researchers are not optimistic about think-aloud studies. Using relatively simple problem-solving tasks, Russo, Johnson, & Stephens (1989) found evidence that think-aloud procedures changed the process of interest. Moreover, they found that Ericsson and Simon’s theoretical framework did not do a good job of predicting which tasks would be a problem.

Concurrent vs. Retrospective Reporting. Subjects can be asked to report thoughts while doing the task (concurrent reporting)—usually by speaking into a tape recorder—or subjects can be asked to report the contents of STM some time after completing the task (retrospective reporting). Both techniques have advantages and disadvantages. Whenever possible, concurrent reporting is preferred.

Concurrent procedures are most likely to result in full and accurate reporting of the contents of STM. For many tasks, however, concurrent reporting may be so unnatural or so mentally burdensome that reporting interferes with natural performance of the task. Viewing and understanding radio and television messages would seem to fall into that category. However, when the subject controls the pace at which he or she takes up each task, such as reading a printed news story, concurrent reporting is reasonable and preferred. Interestingly, when subjects were free to decide whether to give concurrent or retrospective reports while reading magazine text and ads, almost all did concurrent reporting (Woodside, 1983).

However, a little ingenuity may enable an investigator to use concurrent reporting where it would otherwise seem impossible. For example, Daly and his colleagues used special networked computers to explore people’s thoughts as they carried on a conversation and during various violations of conversational rules (Daly et al., 1989). The subjects were all excellent typists who carried on a conversation by typing. A split screen allowed them to type while being able to see the other end of the conversation on the other side of the screen. Because typing was highly automatic for these typists, they were able to think aloud while conversing.

Ericsson and Simon (1979) said that reminders for the subject to “keep talking” during concurrent procedures are not likely to interfere any more than the presence of the researcher in the first place. But Russo et al. (1989) worried that such prompts shift the focus ...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Preface
• 1. Think-Aloud and Thought-List Procedures in Investigating Mental Processes
• 2. Continuous Response Measurement (CRM): A Computerized Tool for Research on the Cognitive Processing of Communication Messages
• 3. Using Eyes on Screen as a Measure of Attention to Television
• 4. Secondary Reaction-Time Measures
• 5. What Can the Heart Tell Us About Thinking?
• 6. Electrodermal Measurement: Particularly Effective for Forecasting Message Influence on Sales Appeal
• 7. Signal Detection Measures of Recognition Memory
• 8. The Time Needed to Answer: Measurement of Memory Response Latency
• 9. Designing Experiments That Assess Psychological Responses to Media Messages
• 10. Detection and Modeling of Time-Sequenced Processes
• 11. Measuring Children’s Cognitive Processing of Television
• 12. Comments on Setting up a Laboratory
• Author index
• Subject Index