The Cognitive Psychology of Speech-Related Gesture

Pierre Feyereisen

• Publisher: Routledge
• Year: 2017
• Print ISBN: 9781138702738
• eBook ISBN: 9781351788274
• Topic: Psychology
• Subtopic: Cognitive Psychology & Cognition
• Index: Psychology

Why do we gesture when we speak? The Cognitive Psychology of Speech-Related Gesture offers answers to this question while introducing readers to the huge interdisciplinary field of gesture. Drawing on ideas from cognitive psychology, this book highlights key debates in gesture research alongside advocating new approaches to conventional thinking.

Beginning with the definition of the notion of communication, this book explores experimental approaches to gesture production and comprehension, the possible gestural origin of language and its implication for brain organization, and the development of gestural communication from infancy to childhood. Through these discussions the author presents the idea that speech-related gestures are not just peripheral phenomena, but rather a key function of the cognitive architecture, and should consequently be studied alongside traditional concepts in cognitive psychology.

The Cognitive Psychology of Speech Related Gesture offers a broad overview which will be essential reading for all students of gesture research and language, as well as speech therapists, teachers and communication practitioners. It will also be of interest to anybody who is curious about why we move our bodies when we talk.

1

Introduction

Matters of terminology and of philosophy

It has currently become commonplace to affirm that in order to communicate face to face, human beings use visible movements – mainly gaze, facial expressions and manual gestures – as well as speech sounds. In the eighteenth century, one could already read in the “Encyclopaedia” of Diderot and d’Alembert (1751/1969), in the entry for Expression: “One can express thoughts in three ways: through the tone of voice, as when one moans; through gesture, as one makes a sign for approaching or withdrawing; and through speech, either articulate or written [emphases in original].” Nonetheless, the number of scientists interested in body language, nonverbal communication or multimodality in discourse is nowadays so large that it has become impossible to present a coherent synthesis of this literature. Choices are necessary; this introductory chapter aims at making explicit the criteria used and briefly justifying them, also in order to explain the choice of title. The domain of gestural communication will be deliberately restricted as far as possible: it will be explored through the lens of cognitive psychology (see Box 1.1), by examining the scientific publications of the last 25 years, without neglecting the fact that they have much older roots.

Box 1.1Cognitive psychology: a basic definition

The term cognition refers to the set of ordinary pieces of knowledge that enable individuals to behave in their daily life: building relationships with other people, using objects, communicating through language and nonverbal means, orienting in space and time, imagining and planning actions, computing, etc. The subject matter of cognitive psychology is the set of mechanisms by which organisms process information in the goal-oriented activities that are determined by their biological structure, which has evolved in their phylogeny and ontogeny. These mechanisms underlying overt behaviour involve perceptual recognition, selective attention, memory functioning, decision making and problem solving, among others. In practice, cognitive psychology proceeds by empirical (experimental, clinical, comparative) and formal methods (mathematical modelling, computerized simulations).

In the 1950s, the first studies in gestural communication were mainly the work of linguists and anthropologists, and they also attracted the attention of psychiatrists and social psychologists. In recent years, with the development of cognitive sciences, the landscape has taken on new shades and gesture studies, previously neglected in psycholinguistics, now occupies a recognized place, as witnessed in the publication of major books such as those of David McNeill in 1992 and 2005, Susan Goldin-Meadow in 2003a and 2003b and Adam Kendon in 2004, all pioneer scientists who exerted a profound influence in the domain. These books anchored gestures in the cognitive sciences, while beforehand bodily communication had been mainly concerned with emotional expression and interpersonal attitudes. Admittedly, these monographs intended above all to present the personal views of the authors and their original contributions to the progress of knowledge. They did not propose an overview and did not attempt theoretical integration. Nor did they aim at an exhaustive coverage, an impossible task as already mentioned. Here, I propose more modestly to set out conflicting perspectives and to examine the empirical evidence that inspired them. The goal is to assess the state of the art and to identify some points of divergence.

To illustrate the point of view of a cognitive psychologist, let us examine a very simple example of an everyday situation. Some time ago, in an underground car park in my town, a woman asked me the way to the new shopping centre. Giving directions often entails body use such as torso orientation, gaze or manual pointing.¹ Why is this banal behaviour an object of scientific inquiry? What the cognitive psychologist is interested in, above all, is to understand the mechanisms allowing the speaker to coordinate several kinds of high-level mental representations: the mental map of the itinerary from the car park to the shopping centre, hypotheses about the mental dispositions of the addressee (her knowledge of the town, the information she desires), the linguistic devices used by the speaker to translate spatial and social representation into well-formed and understandable utterance, action and body schemas to control bodily movements that provide elements of answer to the request. Other aspects of this communicative behaviour, although essential ones, are less important in cognitive psychology than in other disciplines: anthropologists and sociologists will consider the cultural rules shaping interaction rituals (for instance, in Ghana, pointing with the left hand is taboo and prohibited: Kita & Essegbey, 2001), social psychologists will examine the factors influencing helping attitudes and impression management, neurophysiologists will study the links between cerebral activity and the guidance of pointing movements, etc.

In brief, the ultimate goal of a cognitive psychology of gestural communication would be to provide a detailed description of the steps going from understanding a question to giving a response, in order to enable engineers to conceive an artificial device as efficient as a human informant. For several years, computer scientists have become increasingly interested in conversational gestures, with various purposes. In the field of human–computer interactions, research has explored the possibility of creating alternative devices as a substitute for the mouse and keyboard, such as voice and gesture recognition devices to convey instructions. In the domain of computer-assisted animation software, realistic avatars are being developed with the hope of increasing their credibility by means of human-like nonverbal expressive behaviour (for example, Pélachaud, 2009). In that way, it should be possible to generate systems able to describe an itinerary or to replace museum guides in the repetitive aspects of their functions. The rationale of these studies is to reproduce by imitation the characteristics of human behaviour in a virtual agent (for example, Kipp, 2003/2004; Rehm & André, 2008). In so doing, the endeavour of the engineer differs from that of the psychologist. The former tries to obtain valuable technological results, the latter to understand the cognitive and cerebral mechanisms underlying gesture uses. Nonetheless, this ancient distinction must be qualified: one knows that planes do not fly like birds by wing flapping but, in many respects, the aeronautics industry has benefited from the biology of flying organisms.

Since the middle of the twentieth century, advances in artificial intelligence have deeply influenced the way of conceiving the human cognitive system. Yet the brain–computer analogy has some limits, partly because human communication involves various functions, whereas computational models are most often designed to serve only one of them in a well-defined context. Thus, it will be difficult to explain uses of speech-related gestures if one believes that they result from a single, simple programme. Multiple factors have to be considered to account for flexible adaptations to circumstances and unexpected events. The theme of the present book crosses multiple approaches and takes advantage of advances in several domains. Psycholinguistics is certainly essential for studying conversational behaviour and it will receive its due place. However, gesture studies also requires recourse to notions that do not strictly belong to the psychology of language, such as the notion of “theory of mind” (see Box 1.2) and of “motor imagery”, or mental representations of action (Jeannerod, 1994, 2006).

Box 1.2What is meant by theory of mind?

The notion of “theory of mind”, which has elicited much discussion in cognitive sciences (particularly in the philosophy of mind), refers to the ability of individuals to attribute to other beings mental states such as desires and beliefs. The term “theory” is used for two reasons: (1) mental states in themselves cannot be observed (as is the case for many scientific concepts) but they are inferred from their real effects; (2) attribution of mental states aims at explaining behaviour by means of folk psychology (when for instance a child thinks that the dog is scratching the door because it “wants” to go out).

Beliefs are not always true – there are false beliefs – or justified. Current debates concern a necessary distinction to be made between mind reading and behaviour reading, i.e. the ability to detect behavioural regularities in event sequences. Animals that do not have a theory of mind are nevertheless sensitive to temporal contingencies. In order to understand communicative signals and to perceive their underlying intentions, one must not only perceive them but also test hypotheses about unobservable mental states: wishing, thinking, meaning (or other mental verbs) that p (proposition). This theory of mind might rely either on introspection of our own mental states (the simulation or like-me theory) or on previous experience (the theory-theory), which allows children to progressively build, through trial and error and successive approximations, a plausible representation of the mental states of other people, like scientists who based their theories on available evidence (see for example Carruthers, 1996).

Since the advent of this notion in the cognitive sciences, at the end of the 1970s, a very large amount of research has been devoted to the study of normal and pathological development of the theory of mind in children, to the comparison of human and non-human primates and to the neuropsychology of mental state attribution (see for example Apperly, Samson, & Humphreys, 2005; Baron-Cohen, Tager-Flusberg, & Cohen, 2000; Saxe, Carey, & Kanwisher, 2004).

The orientation chosen for this book is radically different from that used in popular or entertainment press with titles such as What is your body language saying? or How to read hand gestures. In that literature, speculative interpretations are proposed for gestures, most often self-touching gestures such as hands making a steeple (a supposed sign of authority and expression of confidence) or rubbing the brow (to signify doubt or worry). Unfortunately, dictionaries that offer these alleged keys may amuse scholars who can look below the surface but they mislead a wide public audience who have no information about the scientific studies of gestures.

In cognitive psychology, research has followed several directions. Before trying to show the particular interest of investigations on speech-related gestures, let us immediately indicate that two topics will not be dealt with in this book: (1) studies on nonverbal communication in the general sense; and (2) studies on sign languages.²

Studies on nonverbal communication, which play an important role in social and clinical psychology, began with a postulate – that has now been given up – of a division of work in communication between language used to convey propositional information and nonverbal behaviour expressing affects and interpersonal attitudes. Admittedly, various body movements display transient feelings such as joy or anger episodes, more enduring moods such as anxiety or boredom and qualities of long-lasting social relationships such as attachment or hierarchical status. During many years, a dominant idea was that the face, called the mirror of the soul, was the main way to display discrete emotions, whereas the rest of the body simply revealed emotion intensity or arousal levels.

Today, these traditional conceptions have been revised in many respects (see Box 1.3). On the one hand, emotions are no longer considered as a separate domain, but relate to cognitive processes such as attention, memory and decision making (see the papers published in the journal Cognition and Emotion). For instance, in the analysis of the causation of emotion, an influential but still disputed proposal is that emotions result from a sequence of appraisals of upcoming events (or stimulus evaluation checks) on several dimensions: pleasantness, novelty, relevance, social desirability, etc. (on the theory of emotion as sequence of appraisals, see Scherer & Ellgring, 2007 as well as Scherer, 2009; for a discussion and alternative models, see Moors, 2009). Thus, feeling and knowing are strongly interconnected. On the other hand, increased attention is now devoted to the verbal expression of affects in diverse perspectives. Communication about emotional states involves a large amount of figurative language with expressions such as “seeing red” or “losing one’s head”. Sharing emotions has an important place in daily life conversations since emotional events often elicit recalls and comments. In cases of emotional disorders, many therapeutic interventions of different kinds involve verbalization. The term of “alexithymia” has been proposed to refer to the difficulty experienced by some people in speaking about their affects. Due to their social and theoretical importance, all these components of mental functioning a...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Contents
• Preface
• Acknowledgements
• 1 Introduction: Matters of terminology and of philosophy
• 2 Speaking hands: Inventory of forms and functions
• 3 Ghosts in the machine: Models of gesture processing
• 4 Production mechanisms
• 5 The impact of gestures on speech comprehension
• 6 Evolution and neuropsychology of gestures
• 7 The development of gestural communication
• 8 Final remarks
• Index