Methods and Tactics in Cognitive Science

W. Kintsch, J. R. Miller, P. G. Polson, W. Kintsch, J. R. Miller, P. G. Polson

• Publisher: Psychology Press
• Year: 2014
• ISBN:
  9781317769064
• Edition: 1
• Topic: Psychologie
• Subtopic: Geschichte & Theorie in der Psychologie

First published in 1984. The editors of this volume are three psychologists whose work has brought them into frequent contact with other disciplines within cognitive science, particularly linguistics and artificial intelligence. Cognitive science is based on the belief that crossing the boundaries of the traditional disciplines is not merely possible, but indeed essential in the study of cognition. The purpose of this volume is to facilitate this interaction among the disciplines that constitute cognitive science, and trying to do this not by an abstract discussion of methodological issues, but by concrete example.

1	Problems of Methodology in Cognitive Science

James R. Miller
University of Colorado

Peter G. Polson
University of Colorado

Walter Kintsch
University of Colorado

The conglomerate that makes up cognitive science includes portions of psychology, artificial intelligence, and linguistics, as well as anthropology, philosophy, and neuroscience. There can be no doubt that, in some of these disciplines, considerable progress has occurred during the last decade and that, therefore, cognitive science is a field of genuine promise today. But in what sense does cognitive science transcend these individual disciplines? At the moment, attitudinal differences in methodology sometimes preclude productive interactions. Of course, each discipline should have its own research style, and there can be no question of imposing some common methodology on cognitive science as a whole. But we must aim for a better understanding of the goals and methods employed by other disciplines and an appreciation of the strengths and limitations of these methods. After all, cognitive science is built on the premise that these methods complement each other, thereby permitting us to advance research in cognition beyond our own discipline’s boundaries.

But when, in fact, is cognitive science research really interdisciplinary? Suppose a linguist makes an excellent case for the proposition that anaphora rules should apply to indexed noun phrases at a quasi-syntactic level intermediate between the autonomous syntactic part of the grammar and the logical formalism; he then shows that this is not the case in several current AI language comprehension programs, and dismisses them. Is this cognitive science? Alternatively, consider a program for text comprehension that makes bridging inferences when there are gaps in the text. The author points out that people can make such inferences, too, and therefore asserts the “psychological reality” of the program. Is this cognitive science? Or, if a psychologist shows that reaction times to passive questions are longer than reaction times to active questions, does this establish the “linguistic reality” of the passive transformation? These examples are not real ones, but neither are they fanciful. Is this limited extension of research from one domain to another what we mean by interdisciplinary work?

Before addressing what we perceive as potentially serious methodological problems, we need to remark on the rather broad view of methodology taken here. We are concerned as much with methods of observation, data collection, and analysis, as with problems of theory construction and evaluation. Indeed, it seems that the latter are primary; once you have decided what sort of theory you want, the empirical methods simply follow.

We first make a few brief observations about the history of cognitive science, because there are some problems we have inherited and have not yet come to terms with. We then enumerate a number of issues that we think deserve attention and that seem to spring from long-standing, unresolved historical conflicts. These sections comprise our diagnosis of what is wrong with cognitive science. We then suggest some treatments to set things right again. We explore how cognitive science could make more effective use of experimental methods and we discuss the opportunities and dangers that stem from the heavy reliance on introspective reports within cognitive science. Most importantly, we take a look at the nature of theory construction in cognitive science, and consider some of its puzzles and contradictions. It is certainly not the case that we have all the answers, but we like to raise questions, even at times in an intentionally pejorative manner, in the hope that the discussions will help us to find some of the answers.

The Dual Origins of Cognitive Science

Cognitive science, as it is practiced today, has two distinct historical roots. It derives from one scientific tradition that emphasizes objectivity and the study of behavior from the outside, and from another that is subjectively oriented and that has proposed to study mental life from the inside (Aebli, 1980). The information processing tradition of cognitive psychology on the one hand, and action theory and purposive or intentional descriptions of behavior on the other, are representative examples of these incompatible trends with cognitive science. Essentially, it is the same conflict that was acted out in a more extreme form earlier in this century by behaviorism and phenomenology.

Some of cognitive science is based on self-observation. We observe a process in ourselves, analyze it, then recognize it again when we see it in others. We understand what happens in others because we know what happens with ourselves. Cognitive theory, thus, is an externalization of processes gained through self-knowledge. The data to be accounted for are direct, simple observations; complicated operationalizations and statistical manipulations would only obscure this straightforward characterization. It is cognitive science from the inside.

The antithesis is the black-box approach that characterizes much of current information-processing psychology. The organism is a black-box, whose input and output can be studied, and we must infer often complex information-processing mechanisms in order to relate the two. These mechanisms are conceived as fixed, interacting components, typically described in rather mechanistic terms. For example, our information-processor might have a short-term memory with a capacity of 7 ± 2 units, various pattern matching devices, a high sensory threshold, and so on.

This is very different from a person who has goals, purposes, and a conscious mind. The contrast is perhaps best expressed by their opposing views of verbal communication. Central to the information-processing approach is the information transmission metaphor. On one end, we have a sender, on the other the receiver, each of which may be decomposed into complex information-processing components; between the two, there is a channel with certain properties, through which information flows from the sender to the receiver. The alternative conceptualization holds that there are two conscious minds, each in a certain state; one is trying to influence the other, that is to change its state, by providing it with certain cues (spoken words, gestures), from which the receiver will infer what the sender meant. Here there is no flow of information anywhere (Hörmann, 1981).

We argue that the problem for cognitive science is to find the right synthesis of these approaches. Neither is satisfactory in itself, but their synthesis must be something other than thoughtless confusion.

METHODOLOGICAL QUESTIONS

The Rejection of Intuition

Despite the growth of cognitive science over the last 10 years, many experimental psychologists are still puzzled by some of the goals of cognitive science. Particularly bothersome is the fact that a theory in cognitive science often amounts to an explication and externalization of intuitions. An anecdote may help to focus this issue. Suppose, like some of us, you are working on a model of text comprehension. We are all able to comprehend text and we have fairly good intuitions about some aspects of the comprehension process. What is the function of a model under these circumstances? In part, it is to make these intuitions explicit, in part to fill out these components of the process where our intuitions fail. For instance, one component where intuitions have proven to be reliable is the following: If you read a text that you can comprehend well, you can predict quite accurately which portions of that text will be recalled later and which will be forgotten. One of the feats of the model we have been developing (Miller & Kintsch, 1980), is that it is also able to make such predictions, just about as successfully as people do. Recently, an experimental psychologist reviewing a paper describing such recall predictions rejected them as being trivial: All they show is that people recall the important passages and forget the irrelevant detail, as everyone knows and as alternative theories also predict. (Indeed, any theory that wouldn’t predict this trend should be rejected.)

A fundamental problem revealed by this criticism is the collision between an approach that views behavior from the outside with one whose goal is to specify and explicate behavior on the inside, including peoples’ intuitions about comprehension. However, such general intuitive knowledge about comprehension is implicit, vague, and cannot be used reliably to troubleshoot comprehension problems nor can it improve the readability of a textbook chapter. The model discussed above is not trivial simply because it makes some predictions anybody can make, just as the linguistic analysis of a sentence is not useless simply because it parses the sentence into constituents any naive speaker could identify. Explaining intuitions is not a bad goal for a theory.

On Mistaking Intuitions for Theory

Self-observation and intuition, and task analyses based on them, are important tools in cognitive science. But it is important to realize that they provide a starting point for research, not an ultimate goal. Intuitions are not in themselves a cognitive theory; rather, they need to be explained. Cognitive scientists do not always seem to appreciate this point. Let us invent an anecdote to illustrate this point. Imagine a psychologist delivering a paper entitled, “Everyday planning techniques: How to bake a cake.” He relates how he personally planned this procedure, and then, after reporting protocols of 15 other subjects, identifies three major cake-baking strategies, through which he claims 85% of the statements in the protocols can be explained. Among traditional psychologists, such a paper would be greeted with derision. Some cognitive scientists might applaud. Both reactions miss the point: The paper is inadequate because it offers no theory; it fails to explain, because it merely restates intuitions, and offers them as explanations. On the other hand, such a paper might provide a valuable task analysis, which is a necessary step in formulating a theory of planning strategies, that could account for the behaviors in question.

Describing intuitions is the first step toward explaining them, but no more. What does constitute an explanation? What is the difference between the model that explains prose recall and the planning strategies for baking a cake that merely describe what people do without explaining that process? The comprehension model is an information-processing model: It identifies certain processes and mechanisms (a short-term memory buffer, cyclical processing, memory retrieval) that interact to produce comprehension. As a result of this interaction, certain parts of the text will be recalled later while others will not be recalled. In a sense we are not directly explicating intuitions in this case at all. Instead, what we have are two ways of understanding a phenomenon like text recall: One is by intuitions that we as scientists share with every literate adult; the other is by explaining behavior as a result of particular information-processing operations that are specified in our model.

The point of this example is not that all explanations must involve information-processing notions. Any kind of general theoretical framework, not just that of information-processing psychology, could, in principle, provide an adequate scientific explanation. That is, our cake-baking strategies could have been related to any concrete, specific theory of planning—whatever the basic terms of that theory. In the cake-baking example it is the atheoretical treatment of the task that we object to, not the teleological terminology.

Are theories at different levels—a strategy explanation versus an information-processing account—real alternatives that can be compared and tested against each other, or are they merely different ways of talking about the same thing? A somewhat idealized description of a recent dissertation at Colorado illustrates the dilemma (Caccamise, 1981). When subjects are asked to generate ideas relevant to a given topic, they occasionally repeat ideas they have mentioned before—more frequently under some experimental conditions than under others. The question of interest concerns the nature of these repetitions. Consider first an information-processing model of idea generation from long-term memory, that gives a reasonably accurate account of these repetitions. In such a model, repetitions indicate an interference process: once an idea is generated, its accessibility in memory increases, and the retrieval process is more likely to follow this same route because of its increased accessibility. Thus, idea repetitions are explained as consequences of the retrieval dynamics.

An interesting alternative views the same phenomenon as a planning task. Ideas are repeated because the subject wants to do so, and not because of interference effects over which he has only partial control. The subject plans the repetition because it might serve as a starting point for a process of elaboration that did not occur when the idea was mentioned the first time. This is certainly a very reasonable intuition, namely, that a repeated idea may lead to a chain of associations that were not produced the first time.

Thus, the information-processing model accounts for precisely the same observations as the planning model. Which model is right? The appealing intuition or the mechanistic model? The mechanistic model clearly “explains” the data; does the strategy model? If so, how do we tell which explanation is better? Are they really alternatives, or is the one a description of processes, while the other describes our intuitions? It can’t be that easy, because there is a clear incompatibility between an interference process and a planned elaboration! Can you have both—one to describe the dynamics of an automatic retrieval process and the other to describe a consciously controlled planning strategy that makes use of that retrieval process? We have returned to the conflict between cognition as mechanistic information-processing and cognition as intentional behavior.

Conceptual Confusions, New Insights, and a Latter Day Tower of Babel

One thing that can enormously upset the more meticulous observers of the cognitive science scene is the terminological inflation and confusion. If only words were explanations, especially fuzzy words that can mean anything and everything! We have only to invent three more terms as all-encompassing as “schema” and cognitive science will be ripe for Nobel prizes. But young sciences are notoriously hard to pin down to terminological purity, and terms like “schema” obviously have their uses.

More interesting, and perhaps more serious, is the confusion between purposive and mechanistic language that characterizes much of the writing in cognitive science. As if it were the most natural thing in the world, purposive terminology has been imported into an information-processing framework: subgoals are stored in short-term memory; unconscious expectations are processed in parallel; opinions are represented propositionally; the mind contains schemata. Is it a sign of conceptual weakness, or merely an excusable sloppiness in the use of language? Or is it not confusion at all, but a new synthesis that cognitive science has achieved? Can the methodology of artificial intelligence overcome the distinctions between information-processing terms and purposive concepts? It is not the same to say that we now can talk about intentionality in information-processing terms (a claim we believe to be valid) and to simply disregard the distinction! We need to explore whether or not the distinction has any merit today. If the answer is no, we need to be able to tell people why; if the answer is yes, the current practice in cognitive science is badly in need of a change.

But even if we were much more careful about our language than we are, cognitive science has a serious communication problem. At one time theories were eminently describable, because they were simply described verbally. The spectre of fuzzy-headed thinking was always present, of course, and substantial amounts of handwaving could be (and often was) concealed by a carefully designed paragraph. The precision of mathematical models helped to remove some of this fuzziness. One needed more than a passing familiarity with mathematics to understand these models, but this was nothing that some time spent with a couple of books on probability theory and Markov chains could not cure. As the topics of interest to psychologists grew increasingly complex and knowledge-laden, these abstract mathematical techniques became less useful, and simulation models, based on the technology (if not the methodology) of artificial intelligence, have come to dominate this aspect of cognitive psychology. As powerful as these models are, and as hopeful as the outlook on understanding the true nature of cognitive processes has become as a result of these models, significant communication problems have appeared.

The communication problem that characterizes computer models cannot be solved as easily as that of mathematical models. It is very difficult to understand how a simulation model works without gaining experience in const...