Verbal Protocols of Reading

The Nature of Constructively Responsive Reading

Michael Pressley, Peter Afflerbach

• Publisher: Routledge
• Year: 2012
• ISBN:
  9781136483028
• Edition: 1
• Topic: Education
• Subtopic: Education General

Researchers from a variety of disciplines have collected verbal protocols of reading as a window on conscious reading processes. Because such work has occurred in different disciplines, many who have conducted verbal protocol analyses have been unaware of the research of others. This volume brings together the existing literature from the various fields in which verbal protocols of reading have been generated. In so doing, the authors provide an organized catalog of all conscious verbal processes reported in studies to date -- the most complete analysis of conscious reading now available in the literature. When the results of all of the studies are considered, there is clear support for a number of models of reading comprehension including reader response theories, schema perspectives, executive processing models, and bottom-up approaches such as the one proposed by van Dijk and Kintsch. The summary of results also demonstrates that none of the existing models goes far enough. Thus, a new framework -- constructively responsive reading -- is described. This new model encompasses reader response, schematic and executive processing, and induction from word- and phrase-level comprehension to higher-order meaning. The important concept in this new model is that readers respond to bits and pieces of text as they are encountered, all as part of the overarching goal of constructing meaning from text. This volume also includes a critical review of the thinking aloud methodology as it has been used thus far. This examination suggests that it continues to be an immature methodology, and that much work is needed if a complete theory of conscious processing during reading is to be developed via verbal protocol analysis. Finally, after reviewing what has been accomplished to date, the authors provide extensive discussion of the work that remains to be done and the adequacy of the verbal protocol methodology for permitting telling conclusions about text processing.

1

An Introduction to Protocol Analysis of Reading

The understanding of human thoughts and actions continues as a goal of psychology and affiliated areas of inquiry. We begin this book by considering briefly the development of protocol analysis as a methodology for examining thought and action, the uses of protocol analysis in investigations of reading past and present, and the historic and ongoing concerns with verbal reports as data. The use of protocol analysis in reading is accompanied by a history of claims for and challenges to the methodology. This is a healthy situation, for the ongoing use of think-aloud protocols has provided information that can be used to refine the methodology. For some, the fact that protocol analysis is regularly used in the investigation of reading appears to be equated with the idea that it is a mature methodology. We consider protocol analysis to be a maturing methodology with much interesting work already accomplished and considerable work to be done.

In think-aloud studies, subjects report their thinking as they do a task. The use of think-aloud data in reading has occurred throughout the 20th century (e.g., Marbe, 1901; McCallister, 1930; Olshavsky, 1976–1977; Piekarz, 1954; Strang, 1970; Titchener, 1912a, 1912b), which is not surprising, because it has been used for thousands of years to reveal thinking. For example, Aristotle and Plato encouraged people to talk about what was on their minds (Boring, 1953; cited in Pritchard, 1990b). More recently, James (1890) used subjects’ reports of their thinking to develop psychological theory. The use of protocol analysis in the 20th century has been used to reveal processing of diverse tasks, including physics problem solving (Simon & Simon, 1978), student cognitions during instruction Peterson, Swing, Braverman, & Buss, 1982), and reading comprehension (Olshavsky, 1976–1977).

Investigations of reading have used protocol analysis both as an exploratory methodology (i.e., inductively) and as a means of testing hypotheses about reading that emanate from initial explorations (i.e., deductively). Protocol analysis of reading has served a varied set of research agendas, including investigations of readers using context to derive word meanings (Werner & Kaplan, 1950), reasoning to complete cloze sentences (Bridge & Winograd, 1982), answering comprehension questions (Kavale & Schreiner, 1979), summarizing texts (Brown & Day, 1983), and reacting to texts in a field of expertise (Wineberg, 1991).

Huey (1908) suggested that the human achievement of reading has few if any equals. Using protocol analysis data related to reading, we make the case that Huey was right. We develop a rich description and understanding of cognitive and affective processes during reading. In doing so, we come to the conclusion that reading is constructively responsive—that is, good readers are always changing their processing in response to the text they are reading. The result is complex processing. The elegant description of reading that emerges from protocol analysis is proof enough of the utility of the method, although throughout this book we attempt to increase awareness of the limitations of protocol analysis, although our perspective is that the limitations in no way diminish the promise of protocol analysis. The results summarized in this volume are only a beginning, for we believe a much more detailed understanding of reading can be developed through future, more analytical uses of protocol analysis.

Challenges to Protocol Analysis

Whatever the value of people’s reports of their thinking, there are important challenges to the validity of protocol analysis. Protocol analysis is all about the relation of peoples’ words to their thoughts, a troubling concern since Watson (1913, 1920). Spoken language is the data used in protocol analysis, and the richness and variability of language are the greatest assets and liabilities of the verbal reporting methodology. We consider the constructive nature of language comprehension to be one of the greatest ongoing challenges to protocol analysis. When a subject provides verbal reports, there is the built-in language variation that is part of the individual’s personality and way of interacting with the world. When a researcher attempts to analyze the verbal report, a separate worldview, vocabulary, and set of inferencing processes is put into action. Despite these constraints, a great deal has been achieved through protocol analysis.

Watson (1920) was concerned with how task parameters might affect verbal reporting. For example, protocol analysis has been used to examine how subjects solve the “Missionaries and Cannibals” problem. A problem-solving subject sitting in a laboratory, describing how missionaries are being ferried from one shore to another to avoid becoming the cannibals’ next meal, is working in a problem space with only a few possible actions or moves. Compare this with readers who are asked to report how they construct meaning from a text such as Martin Luther King, Jr.’s “Letter from a Birmingham Jail,” or an editorial on raising cigarette taxes. In contrast to the missionaries’ options, the range of possible actions during reading is very great. Is it reasonable to assume that self-reports can be as adequate in this more complex situation than in the simple problem-solving situation?

Many continuing concerns with protocol analysis have centered on the related questions of what is requested of subjects, and when it is requested. Watson (1913) directly addressed the first question when he raised concerns about introspection. Introspection occurs when subjects speculate about their actions, their reasons for carrying them out, and narratives of how they carry them out. As a result, subjects report not only the contents of short-term memory, but samples of their theories of mind. Introspective reports can be fascinating and can provide compelling accounts of subjects’ thoughts and reflections. Yet, they are generally considered too reflective and too prone to digression to provide firm ground for building theories of on-line cognitive processing, or response to reading. In general, protocol analysis for the purpose of building theory about cognition and response is best served by regular, on-line reports of the contents of short-term memory.

The second concern, when subjects report, has become increasingly informed by our understanding of human information processing and short-and long-term memory. As we explain later, the contents of short-term memory are the fertile ground for verbal reports (Ericsson & Simon, 1984/1993). Because of the limited capacity of short-term memory, and the movement of information and process through it, the recency of verbal reports of cognition and response to their actual occurrence is critical. Like a chance encounter with a stranger’s face flashing by in a train window and one’s memory of the face, the greater the temporal distance between the event and report, the greater the chance for embellishment or decay of the information. Although this may not mean a less interesting report, it will probably mean a less accurate one.

Relationship of Protocol Analysis to Reigning Paradigms

Throughout history, protocol analysis has had varied relationships with the reigning psychological paradigms, constructs, and models it was used to investigate. For example, James (1890) regularly relied on introspective reports to inform theories of mind, although perhaps Duncker (1926, 1945) conducted the most famous of the early think-aloud analyses.

Duncker asked people to think aloud as they solved problems. One of the most analyzed of these was the “radiation” problem: A human has an inoperable tumor that can be destroyed by radiation. A ray of sufficient intensity to destroy the tumor, however, would also destroy healthy tissue. How could the tumor be destroyed and yet the healthy tissue preserved? Many of Duncker’s subjects provided model think-alouds, in that they reported the contents of short-term memory, revealing their hypotheses and false starts toward a solution, as well as their good progress and eventual solutions. For example, here are the final two self-reports of a subject, after many other possibilities entered consciousness and were dismissed:

I see no more than two possibilities: either to protect the body or to make the rays harmless …

Somehow divert … diffuse rays … disperse … stop! Send a broad and a weak bundle of rays through a lens in such a way that the tumor lies at the local (sic, focal?) point and thus receives intensive radiation. (Duncker, 1945, p. 3)

This work with protocol analysis, however, seemed to be in need of a theory. Although the data generated in the late 19th and early 20th centuries was important in original form and analysis, there was no formal model of thought processes (e.g., where they come from, what they consist of, how they are generated) that permitted certain interpretation of such data.

The middle part of the 20th century witnessed a relative lull in the use of protocol analysis, as introspection was challenged by behaviorists, beginning with Watson and his contemporaries, whose theories dominated psychology. Behaviorist theory had little need of cognition, for its focus was more on “overt performance than mediating processes” (Ericsson & Simon, 1984/1993). The use of protocol analysis waned for it was fruitless to ask people to report on something that was not considered theoretically important … nor did it make sense to use a method that seemed so certain to produce data that could not be believed, data that were merely introspective.

In contrast to behaviorism, the tenets of cognitive psychology and reader response, which began to rise in prominence in the 1960s, can be considered a standing invitation to use protocol analysis. When readers are mindful information processors, who at least sometimes consciously mediate their understanding, it makes sense to expect self-reports that are veridical with ongoing cognitive processes and strategies, to expect that people can report their cognitive and affective responses to text.

From this view, there may be at least three advantages to using protocol analysis: first, it can provide data on cognitive processes and reader responses that otherwise could be investigated only indirectly; second, verbal reports sometimes can provide access to the reasoning processes underlying sophisticated cognition, response, and decision making; third, verbal reports allow for the analysis of affective processes of reading in addition to (or in relation to) cognitive processes (Afflerbach & Johnston, 1984). All three types of information are present in the verbal self-reports summarized in this volume.

Ericsson and Simon’s (1984/1993) “Protocol Analysis”

One of the most important events in the development and refinement of protocol analysis was the publication of Ericsson and Simon’s Protocol Analysis: Verbal Reports as Data in 1984 (revised in 1993). The book summarizes much of the work in protocol analysis and many of the challenges to the methodology, sampling across the domain of cognitive psychology from studies that employ protocol analysis. Ericsson and Simon interpreted protocol analyses with respect to information processing theory, describing conditions when think-aloud data might be expected to be believable and when think-aloud data should be suspect. The volume also includes a definitive review of the evidence (produced up until 1983) that validates the claims about thinking aloud on the basis of information processing theory. After its publication, Ericsson and Simon (1984/1993) became the standard reference on think-aloud methodology.

Theoretical Underpinnings of Thinking Aloud

Ericsson and Simon (1984/1993) identified strongly with information processing theory and focused on two constructs in that theory of special importance: longterm memory and short-term memory. Long-term memory contains knowledge of how to do things (i.e., procedural knowledge) as well as a great deal of factual knowledge (i.e., declarative knowledge). Some of the declarative knowledge is episodic knowledge, in that it is memory of some specific event in the thinker’s life (e.g., recollection of attending a particular concert). Other knowledge is more generalized, not tied to specific events, but rather representative of types of events in general (e.g., knowledge of what goes on at concerts and in what order). The most important characteristic of long-term memory is that it is vast. Fortunately, it also is organized so that information contained in it can be accessed. Of course, people differ in the degree of organization of the long-term store. For any individual, some types of information will be more organized than other types of information. Thus, a career dentist has extensive and well-organized knowledge of teeth but, typically, less extensive knowledge of automobile engines. The organization of a dentist’s knowledge of engines would not approach the sophistication of a mechanic’s organization of knowledge of engines. Consequently, when asked about cracks in fillings, the dentist can quickly relate vast knowledge of how such cracks develop, their consequences, and how and when they should be treated. If asked about cracks in engine blocks, the dentist’s knowledge is accessed more slowly and is much less complete. Exactly the opposite pattern occurs for mechanics, who can provide detailed commentary about the mechanical relationships in engines that can result in a cracked engine block, but whose memory of cracked fillings is limited to when he or she had one, resulting in the need for a root canal. Ordinarily, a mechanic would not think of a long-past root canal, but that episode can be activated and represented in short-term memory if the environment cues the event (e.g., someone asks the mechanic if he has had a root canal).

The second construct that figures largely in Ericsson and Simon’s (1984/1993) work is short-term memory, which is often thought of as information currently in consciousness. The information in short-term memory derives from two sources. One source is external stimulation. For example, if you see a person for the first time and close your eyes to imagine the individual in your mind’s eye, the contents of short-term memory reflect stimulation from external sensations. If you begin to think of other people who look like this individual, new names and faces will enter short-term memory. These associations come from your long-term memory, which is the second source of information in short-term memory. Recall the mechanic who had the root canal. The question about the root canal is external stimulation that is coded into short-term memory, with the short-term memory processing of the question cuing long-term memories of the root canal experience.

The consensus view is that short-term memory is extremely limited in capacity, with information leaving short-term memory if the thinker does not operate on it. If a person hears the phone number 725–2276, that number will remain active in short-term memory if the individual repeats it, but will be forgotten rapidly if there is no activity to hold the number sequence in mind. With sufficient rehearsal, or other coding activities, the number may be transferred to long-term memory where it is stored as declarative knowledge.

One especially important property of short-term memory is that people can quickly access the contents of short-term memory and report them. If what is being held is a verbal sequence, then the sequence simply can be verbalized (e.g., “7, 2, 5, 2, 2, 7, 6”). If what is being held is nonverbal (e.g., an image of a cherry red Cadillac convertible), a person can typically verbalize a description of the image. In addition, people are often aware of what they recently held in short-term memory, because some of the contents of short-term memory are translated to long-term memory before they exit short-term awareness. Thus, if a person is asked, “What were you thinking about when you dialed the phone number a minute ago?” often people can recollect what they were thinking about a short time ago. Ericsson and Simon (1984/1993) extensively developed the case that people’s reports of the current contents of short-term memory are often valid; they also argued that reports of recent memories (i.e., episodic recollections) are often veridical with what was being reflected on in the recent past. As time passes since the original cognition, the validity of recollections decreases, however. Moreover, the quality of recollections depends on retrieval cues. For example, if asked about your thinking as you made a phone call last night, you might be less likely to remember than if you are reminded that the phone call was to your mother.

In contrast to valid reports of the current contents of short-term memory and recollections of recent episodes, Ericsson and Simon (1984/1993) contended, based on substantial data, that some questions do not stimulate certainly accurate verbal reports. In particular, people typically cannot answer “why” questions, questions about their motives for their behaviors. Thus, if asked, “Why did you call your mother last night?” it is likely that the subject will generate an answer in response to the question, even, if when the call was actually made, the individual was not thinking about why he or she called. Here, Ericsson and Simon’s concerns are a clear refrain of those raised about introspection by Watson.

The very best verbal reports, from the Ericsson and Simon (1984/1993) perspective, are of exactly what is heeded in short-term memory. The more that the verbal reports vary from what was heeded, the less certainly believable they are. Thus, if a person is holding in mind, “7-2-5-2-2-7-6,” it is believable if they self-report, “7,” “2,” “5,” “2,” “2,” “7,” “6.” A report, “I’m holding Steve’s phone number in mind,” is more suspect, because it does not reflect the exact contents of short-term memory. Nonetheless, Ericsson and Simon (1984/1993) made the case that often such labels of processing represent the processing that occurred, and thus they viewed such self-reports as interpretable. A report, “I’m holding Steve’s number in mind because I would have tried so hard to get it,” is much less desirable from Ericsson and Simon’s (1984/1993) perspective, representing interpretations of thinking on the part of the person making the verbal report.

In short, Ericsson and Simon’s (1984/1993) most important conclusion is that people can self-report the contents of their short-term memory. The conclusion provides retrospective validation of think-aloud data gathered in the 1920s and 1930s, data that experienced cycles of acceptance and dismissal until it became more grounded when provided the anchor of the short-term memory model developed by Ericsson and Simon. This position was supported mostly by analysis of think-aloud data, generated in the context of problem solving. In particular, they argued that people can report the intermediate and final products of problem-solving processes with great accuracy, much more certainly than they can detail use of the processes per se. Thus, if a person is adding 1,045 and 2,764 in his or her head, a completely believable self-report would be, “9,” “0,” “8,” “3”—“3,809.” Less certainly believable would be labeling of the addition process, rather than reporting of the actual information heeded in short-term memory. Such a report might be, “I’m adding the columns from the right to the left, making certain to keep track of the digits being carried. I’m holding the digits of the solution that I have generated already in mind as I work through the problem from right to left.” Ericsson and Simon were explicit that such a description or explanation of process is not as convincing as the reports of the products of processing, arguing that the researcher should infer processes from reports of products rather than encourage their subjects to make such inferences about their processes as part of self-reports. Still, they accepted retrospective descriptions of processes as valid self-reports. Much less believable than descriptions, however, would be the report, “I’m adding column by column because that is how I learned to do it in grade school,” which is more an interpretation of one’s motivation for processing than a report of the contents of short-term memory. Ericsson and Simon (1984/1993) clearly believed that interpretive descriptions and explanations of cognitive processing were better left to the researcher, who was armed with subject reports of the contents of short-term memory.

One of the most impressive parts of Ericsson and Simon’s (1984/1993) book is the section in which they delineated and described introspective, concurrent, and retrospective think-alouds. The careful description of shared characteristics (e.g., all verbal reports are given by subjects related to performing particular tasks) and the important...