Integrating Gestures

The Dimension of Multimodality in Cognitive Grammar

Silva Ladewig

• Publisher: De Gruyter Mouton
• Year: 2020
• Print ISBN:
  9783110668414
• eBook ISBN:
  9783110668650
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

Gestures are now viewed as an integral part of spoken language. But little attention has been paid to the recipients' cognitive processes of integrating both gesture and speech. How do people understand a speaker's gestures when inserted into gaps in the flow of speech? What cognitive-semiotic mechanisms allow this integration to occur? And what linguistic and gestural properties do people draw on when construing multimodal meaning? This book offers answers by investigating multimodal utterances in which speech is replaced by gestures. Through fine-grained cognitive-linguistic and cognitive-semiotic analyses of multimodal utterances combined with naturalistic perception experiments, six chapters explore gestures' potential to realize grammatical notions of nouns and verbs and to integrate with speech by merging into multimodal syntactic constructions. Analyses of speech-replacing gestures and a range of related phenomena compel us to consider gestures as well as spoken and signed language as manifestations of the same conceptual system. An overarching framework is proposed for studying these different modalities together – a multimodal cognitive grammar.

1 Introduction: cognitive grammar and gesture studies?

This book takes a usage-based approach to the integration of gestures into speech. Based on linguistic and cognitive-semiotic analyses of multimodal utterances and on naturalistic perception studies, it argues that gestures integrate on a syntactic and semantic level with speech merging into multimodal syntactic constructions. Looking specifically at discontinued utterances and their perception, it becomes evident that gestures participate vitally in the dynamics of meaning construal and may take over the function of verbs and nouns in their respective syntactic slots contributing to the semantics of the sentence under construction. With this phenomenon under scrutiny, the book takes a unified perspective on the integration of gestures with speech, following the plea formulated recently by sign language linguists to elaborate an overarching framework for studying and understanding spoken and signed language and gestures. The integration of different modes of expression to form multimodal or “composite utterances” (Enfield 2009) serves as a sample domain, showing that language and gesture “are manifestations of the same underlying conceptual system that is the basis for the human expressive ability. Thus, we propose that the general principles of Cognitive Grammar can be applied to the study of gesture” (Wilcox and Xavier 2013: 95).

Integration of component structures into composite structures can take place at either the phonological or semantic pole of symbolic structures, or the integration can be of componential symbolic units themselves into more complex symbolic units. […] Integration of component structures into composite structures with greater complexity is a central aspect of grammar, and a topic well-studied by linguists describing spoken and signed languages. One question that could be posed is whether integration can take place across linguistic and gestural systems. We propose that such integration does indeed take place, and should be studied as such.

(Wilcox and Xavier 2013: 92)

This book aims to answer the question posed by both authors: “One question that could be posed is whether integration can take place across linguistic and gestural systems” (Wilcox and Xavier 2013: 92). For this purpose, cognitive-linguistic research on sign language (e.g., Wilcox 2004a) and gesture (e.g., Cienki 1998b, 2005; Müller 2008b, 2017b) will be united to investigate multimodal integration from a linguistic and usage-based perspective. By bringing together Cognitive Grammar and cognitive-semiotic/linguistic analyses of gestures (e.g., Ladewig and Bressem 2013a; Mittelberg 2006; Müller, Bressem and Ladewig 2013; Müller, Ladewig and Bressem 2013a), the book offers a framework to study multimodal utterance construction and, thus, contributes to a more general understanding of the syntactic and semantic roles of gestures in multimodal utterances. It will conclude with reflections upon the symbolic nature of grammatical classes of verbs and nouns as indicated by gestures, based on manual actions.

In order to study the syntactic and semantic integration systematically from a usage-based naturalistic point of view, the study focuses on instances of the integration of gestures into discontinued spoken utterances in which gestures replace linguistic units in utterance-final position (Figure 1).

Figure 1: Example of a gesture realizing a syntactic slot.

In the utterances under investigation, gestures realize syntactic slots in utterance-final position, replace the spoken constituents of nouns and verbs, and complete the utterance. According to perception analyses, these multimodal utterances do not cause problems in the understanding, but recipients treat them as meaningful for the proceeding discourse.

Although this phenomenon offers revealing insights into the integration processes of speech and gestures and many researchers are sensitive to this phenomenon (e.g., Clark 1996, 2016; Engle 1998; McNeill 2005, 2007; Slama-Cazacu 1976; Wilcox 2004a), it has not been studied in depth before (Keevallik’s [2013] study is an exception). This book fills this research gap, aiming to reveal the potential of gestures to realize the grammatical categories of nouns and verbs by embodying their conceptual schemas. By tracing the interactive processes between speech, particularly the grammar of spoken language, and gesture, a bridge from grammar to multimodal Cognitive Grammar is built.

The approach advocated in this book is considered as one possible framework to address the relation of spoken language and gesture. It allows the elucidation of the interrelation between speech and gesture on the level of grammar, taking into account the cognitive processes of meaning construal. Moreover, it aims at addressing symbolization processes in the gestural modality and, thus, gestures’ potential of becoming linguistic signs. However, sure enough, by zooming in on one particular relation of both modalities, others are moved out of the attentional space, which is why the approach developed here should be treated as one puzzle piece for illuminating the multimodality of language.

1.1 The cognitive grammar enterprise

Since 1976, I have been developing a linguistic theory that departs quite radically from the assumptions of the currently predominant paradigm. Called ‘cognitive grammar’ (alias ‘space grammar’), this model assumes that language is neither self-contained nor describable without essential reference to cognitive processing (regardless of whether one posits a special faculté de langage). Grammatical structures do not constitute an autonomous formal system or level of representation: They are claimed instead to be inherently symbolic, providing for the structuring and conventional symbolization of conceptual content. Lexicon, morphology, and syntax form a continuum of symbolic units, divided only arbitrarily into separate ‘components’ – it is ultimately as pointless to analyze grammatical units without reference to their semantic value as to write a dictionary which omits the meanings of its lexical items. Moreover, a formal semantics based on truth conditions is deemed inadequate for describing the meaning of linguistic expressions. One reason is that semantic structures are characterized relative to knowledge systems whose scope is essentially open-ended. A second is that their value reflects not only the content of a conceived situation, but also how this content is structured and construed.

(Langacker 1986: 1–2)

Cognitive Grammar was developed out of the disagreement with generative grammar which refuses to comprise meaning into the description of grammatical structures but argues instead for cognitively separated modules of syntax and semantics. Its central ideas were first spelled out in the seminal two-volume work Foundations of a cognitive grammar (Langacker 1987a, 1991b), establishing one research strand within the field of Cognitive Linguistics. Arguing that meaning is central to linguistic analysis, the cognitive linguists Wallace Chafe, Charles Fillmore, George Lakoff, Ronald Langacker, and Leonard Talmy started to examine language structure and language use regarding general cognitive principles and mechanisms such as perception, human categorization, iconicity attention, or memory. Each of them developed his own theory, such as conceptual metaphor theory (see Lakoff and Johnson 1980), frame semantics (see Fillmore 1985) or force dynamics (see Talmy 1988).

The theory of Cognitive Grammar “advances the controversial […] proposal that essential grammatical notions can be characterized semantically” (Langacker 2008a: 103). Moreover, by claiming that “grammar […] is inherently symbolic” (Langacker 1987a: 12) and that “all valid grammatical constructs have some kind of conceptual import” (Langacker 1987a: 282) meaning is equated with conceptualization. Hence, semantic and grammatical structures are conceived as having conceptual content. It is noteworthy that the term “grammar” is not used in a narrow sense, referring only to (morpho-)syntactic structures, but it is employed in a broad sense, referring to the language system as a whole, including sounds, meaning and (morpho-)syntax. Langacker further assumes that grammar is motivated by general cognitive capacities and processes, which is why he applies principles of gestalt psychology comprehensively to the analysis of language and, in doing so, draws analogies between linguistic structure and visual perception. As such, grammar is claimed to be grounded in conceptualization, which, on the other hand, is claimed to be grounded in the human’s interaction with the physical reality (embodiment theory). Analyzing language from this perspective allows conclusions to be drawn about not only the relation of human language and cognition but also, as will be argued in this book, gesture and cognition.

The major premises of Cognitive Grammar will be outlined in the following pages. However, given that encompassing and enlightening résumés of its main tenets have been provided by other scholars (see e.g., Kok 2016), the basic principles that are pivotal for the enterprise followed in this book are outlined in the following sections.

1.1.1 Grammar is symbolic in nature, profiling conceptual content

The first “[o]utrageous [p]roposal,” as Langacker (2008a: 5) puts it in his book Cognitive grammar: A basic introduction, is the well-known and fundamental claim that “grammar is symbolic in nature” (Langacker 2008a: 5) and that grammar and lexicon form a continuum, while both are claimed to be fully describable as assemblies of symbolic structures. This means that linguistic structures observable in use (usage events) are understood as merging phonological material with semantic content. This notion was inspired by de Saussure’s (1916) notion of a two-dyadic sign combining the sense of a sound (Lautbild) and an idea (Vorstellung).

The view that morphemes are inherently symbolic, associating phonological representations with semantic content, is not controversial. Consider the following example to illustrate his argument. If we take the morpheme gesture into account, we can describe the mapping of form and meaning as follows: The phonological structure (the word’s phonological pole) that comprises the smaller units of [ʤ], [e], [s], [t], [ʃ], and [ə], can be represented as [[ʤ]-[e]-[s]-[t]-[ʃ]-[ə]], where the hyphens indicate the syntagmatic combination of the components into higher-order phonological units represented by the outermost square brackets. The semantic...

Table of contents

• Title Page
• Copyright
• Contents
• 1 Introduction: cognitive grammar and gesture studies?
• 2 Multimodality of grammar and its cognitive foundations
• 3 How are gestures integrated into linguistic structures?
• 4 Semantic integration of gestures: constructing multimodal reference objects
• 5 Multimodal sentences and discourse contexts: salience, attention and foregrounding
• 6 Conclusion
• Index