Representations in Mind and World

Essays Inspired by Barbara Tversky

Jeffrey M. Zacks, Holly A. Taylor

• Publisher: Routledge
• Year: 2017
• Print ISBN: 9781138829695
• eBook ISBN: 9781351689953
• Topic: Psychology
• Subtopic: Cognitive Psychology & Cognition
• Index: Psychology

This volume pulls together interdisciplinary research on cognitive representations in the mind and in the world. The chapters—from cutting-edge researchers in psychology, philosophy, computer science, and the arts—explore how structured representations determine cognition in memory, spatial cognition information visualization, event comprehension, and gesture. It will appeal to graduate-level cognitive scientists, technologists, philosophers, linguists, and educators.

1
Insight in Mind

Holly A. Taylor and Jeffrey M. Zacks

Representations of Barbara Tversky

How do representations in our heads and representations out in the world interact to produce human behavior? This is one of the great questions of contemporary cognitive science and the theme of this book. These questions also reflect the career of Barbara Tversky. The chapters presented here all take aim at cognitive representations in complex cognition. They emerged from a symposium held in Chicago in November, 2015, to honor Barbara Tversky’s career. They extend beyond traditional definitions of cognitive science by representing a convergence of psychologists, artists, computer scientists, philosophers, and designers.

This volume is broad in scope, befitting the panoramic range of Barbara Tversky’s scientific career. In a recent conversation with one of us, Tversky referred to Yuval Noah Harari’s contrast between hunter-gatherers and farmers (Harari, 2014) and characterized herself as an intellectual hunter-gatherer. Indeed, her restless scientific mind has ranged over verbal memory, scenes, events, diagrams, and mental imagery. She has studied college students and children, architects and artists. With her collaborators, she has invented tasks, wrestled behaviors such as wayfinding, drawing, and graphing into the laboratory, and conjured methods to render the dynamic processes of free-range cognition accessible to quantitative analysis. At the same time, within this vast range of topics, she masterfully connects ideas both within and across topics, thus creating order and theory relevant for understanding and predicting complex cognition more generally.

This volume aims to reflect not only Barbara’s research but also her theoretical approach, which is unique in its ability to provoke insight and connection. Across the wide range of her work stretches a common focus on the format of representations. Representations can be in one’s mind or made explicit on a piece of paper or computer screen in the world. Barbara clearly also recognized that even considering both internal and external representations would be insufficient to understand complex cognition. The mind interacts with the world and it is important to understand fundamentals of how internal and external representations interact.

Another equally important factor to understanding complex cognition emerges in Barbara’s research strategy. Just as she examines internal and external representations and their interaction, she takes the real world into the lab and the lab into the real world. In this way, she has been able to explore how cognition actually functions in important real-world situations. In her work and the work she has inspired, people have navigated environments without ever leaving the lab room, creating mental representations from spatial language, maps, and virtual reality. People have also navigated cities of the world and many a college campus. People have been asked to understand or explain diagrams created as experimental stimuli, and Barbara has set out to understand and find commonalities amongst diagrams others have published to explain a concept. She has explored how artists create, how designers design, and how scientists explain.

This wide-ranging curiosity is evident in her collaborators and students—and in the chapters assembled here. We have arranged the book in three sections, moving from representations in the mind to representations in the world to the interaction of mind and world. The chapters themselves reflect research emanating from the lab, “in the wild” of the real world, and conceptualized in the mind of her students, post-docs, and collaborators based on their inspiring interactions with her.

Representations in Mind

Roberto Casati opens with a philosophical grounding of mental representations. Building on the popular distinction between fast/automatic and slow/effortful modes of processing (e.g., Kahneman, 2011), Casati proposes a new mode that offloads some of the slow and effortful components of reasoning onto the environment. The view that human cognition can only be understood in terms of how it is embedded in the environment of things is called “situated cognition,” and it is a view with which Tversky has long been engaged. One way to describe Casati’s project is to say that he “situates situated cognition.” Nancy Franklin and Michael Greenstein take ideas of mental representation to a highly constrained—and high-stakes—setting: legal testimony. Their chapter reviews how memory and storytelling interact during legal testimony. We anticipate that this chapter will be extremely valuable not just to cognitive scientists but also to legal practitioners. In the final chapter of this section, Tad Brunyé, Zach Haga, Lindsay Houck, and Holly Taylor take on another real-world application of memory: finding one’s way around. This work, inspired by Taylor’s early work with Barbara (Taylor & Tversky, 1992a, 1992b), explores how spatial mental representations impact knowledge of, and interactions with, the environment. All three of these chapters demonstrate the Tverskyan strategy of uncovering fundamental attributes of memory representations by looking at how they engage with complex, naturalistic tasks.

Representations in World

The second section focuses on the representational artifacts that humans create. It opens with a visual essay by Jane Nisselson, abstracting her short film based on Tversky’s paper Visualizing Thought (Tversky, 2011). The essay and the film vividly illustrate how the elements of diagrams function. Michel Denis’ chapter zooms in on one diagrammatic element, the arrow, wonderfully taking a common artifact that appears simple enough to take for granted and showing the actual cognitive complexity that underlies its functioning. Increasing the complexity of the representational artifacts, Mireille Bétrancourt takes on a particularly contemporary cognitive artifact: animation. It turns out to be surprisingly difficult to design effective animations; Bétrancourt’s chapter investigates why this is so and suggests what can be done. Representational artifacts exist for people to use, even if only for the artifact’s creator. Jeffrey Nickerson moves from the artifacts themselves to how practitioners really use them, showing how designers use artifacts to reason, to discover, to provoke themselves to new insights. Finally, Francesca Pazzaglia and Chiara Meneghetti show that there are dramatic differences in how people use cognitive artifacts to learn about spatial environments, and that these differences are systematic.

Interaction of Mind and World

We close, of course, with how representations in the mind and in the world interact. Elizabeth Marsh and Kathleen Arnold open this section by considering how using memory cycles back to influence memory representations. Much of Tversky’s early research investigated the operations used by children and adults when encoding and remembering verbal materials (e.g., Tversky, 1973; Tversky & Teiffer, 1976). When Marsh and Tversky began collaborating, they focused on how these mechanisms function in the sorts of things that people actually do with verbal memories (Tversky & Marsh, 2000). When people use verbal memories, they interact in interesting ways with the world and come back to influence the mental representation. One important thing people do is tell others about what happened—and as Marsh and Arnold demonstrate, this affects not only the listener but also the memory of the teller. Christian Freksa et al’s chapter applies this mind-world interaction analysis to spatial problem solving. Notably, the arrangement of objects in space, whether they be buildings in a city, rooms in a building, or matchsticks on a table, affects spatial problem solving related to those objects. Jeffrey Zacks’s chapter applies a similar analysis to event representations, showing how our mental and neural representations of events determine our media—and vice versa. Finally, Andrea Kantrowitz illustrates (literally!) a fine-grained description of how artists go back and forth with their marks, shaping their creative cognition in real time.

The Complete Picture

Many of the authors represented have been committed to the field of cognitive science for long years. Others are recent converts or fellow travelers. We hope that this book will be useful to both old hands and new recruits. For ourselves, we think that the work collected here renders vivid the power, breadth, and creativity of what contemporary cognitive science can be. This only makes sense, given that it was inspired by Barbara Tversky.

Authors’ Note

Preparation of this volume and the introductory chapter was a joint and equal effort. Order of authorship was determined by a coin flip.

References

Harari, Y. N. (2014). Sapiens: A brief history of humankind. Random House.

Kahneman, D. (2011). Thinking, fast and slow. Macmillan.

Taylor, H. A., & Tversky, B. (1992a). Descriptions and depictions of environments. Memory and Cognition, 20, 483–496.

Taylor, H. A., & Tversky, B. (1992b). Spatial mental models derived from survey and route descriptions. Journal of Memory and Language, 31, 261–282.

Tversky, B. (1973). Encoding processes in recognition and recall. Cognitive Psychology, 5, 275–287.

Tversky, B. (2011). Visualizing thought. Topics in Cognitive Science, 3(3), 499–535.

Tversky, B., & Marsh, E. (2000). Biased retellings of events yield biased memories. Cognitive Psychology, 40, 1–38.

Tversky, B., & Teiffer, E. (1976). Development of strategies for recall and recognition. Developmental Psychology, 12, 406–410.

Representations in Mind

2
Two, Then Four Modes of Functioning of the Mind

Towards a Unification of “Dual” Theories of Reasoning and Theories of Cognitive Artifacts

Roberto Casati

The main aim of this contribution is to stabilize and generalize the use of the conceptual labels originating from “dual” theories of reasoning, so as to provide a theoretical unification with theories of cognitive artifacts, and to describe in an abstract way the mechanics of cognitive artifacts. Psychological literature has by and large accepted the distinction between two “systems”, or – as I shall say – two modes of operation of the brain in certain tasks, mainly reasoning and decision-making tasks (Evans 2003, 2012, 2015 for reviews; Evans and Frankish 2009; Kahneman 2011). Mode 1 (M1, for brevity) is an automatic, autonomous, stimulus-driven, fast operating mode that delivers rough but locally acceptable results; M2 is modulated by will and attention, operates slowly and stepwise, intensely uses working memory, and is in general more accurate. I shall take the distinction for granted (with some caveats, in particular I shall argue that we do not need to endorse a substantive view of cognitive systems, as opposed to a more neutral talk of modes) and argue for an extension of the conceptualization to cover cases discussed in the literature of cognitive artifacts, with the goal of unifying the two fields. I’ll first introduce M4, an operating mode that completely outsources the computations typically run by M1 and/or M2 to external artifacts. The M4 mode fully delegates the relevant mental activity – what I shall dub “core” computational tasks – and only makes its user care about the input and output of the computation. Then I’ll vindicate the existence of a Mode 3, best understood as occupying an intermediate position between M2 and M4. In the third mode we interact with cognitive artifacts (such as maps, measuring instruments, written text) and this interaction is both essential to performing a certain task (as opposed to what happens in M1 and M2) and is not an instance of wholesale offload (as opposed to M4). Interactions with cognitive artifacts actually display proprietary computations, which give some hints about the architecture of cognition, and about its flexibility. Flexibility in turn creates room for the activity of designers of cognitive artifacts.

In this chapter I investigate the relationships between the M3, M1 and M2, and discuss some demarcation issues: whether M3 activities are a subclass of M2 activities, and whether we should postulate continuity between the four modes. More specifically, I look into some of the proprietary computations of M3 (such as shunting information, bridging cognitive modules, displacing search processes, or restructuring memory search). Other targets are popular metaphors such as the “extended mind” and “the world as external memory” that, by making the M3 look too much like M4, risk missing out on the specific properties of brain-artifact interaction.

M1 vs M2: An Example from Navigation

I grew up in a right-driving country. My parents taught me how to cross the street when I was very young. Over time I refined this practice. I moved from having no idea about crossing streets to being moderately skilled, to being an expert. When you acquire such a skill, the imperatives are to train and to aim at being error free. Over such a long, endless process of learning and perfecting, your responses inevitably become automatic: they involve a sense of your body, a sense of your target, time planning, quick decision making, and gut feelings. For instance, when you cross the street, you automatically expect and almost feel danger first on your left-hand side, then on your right-hand side. Navigation is computation (Hutchins 1995).¹ You assess time in relation to the space of travel, the contingencies of the street scene, your assessment of safe zones, the presence of other actors. You even break the law from time to time. You fac...

Table of contents

• Cover
• Title
• Copyright
• CONTENTS
• List of contributors
• 1 Insight in Mind
• Representations in Mind
• Representations in World
• Interaction of Mind and World
• Index