New Perspectives on Narrative and Multimodality

Ruth Page, Ruth Page

• Publisher: Routledge
• Year: 2009
• ISBN:
  9781135254605
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

The contributors in this collection question what kinds of relationships hold between narrative studies and the recently established field of multimodality, evaluate how we might develop an analytical vocabulary which recognizes that stories do not consist of words alone, and demonstrate the ways in which multimodality brings into fresh focus the embodied nature of narrative production and processing. Engaging with a spectrum of multimodal storytelling, from 'low tech' examples encompassing face-to-face stories, comic books, printed literature, through to opera, film adaptation and television documentary, stretching beyond to narratives that employ new media such as hypertext, performance art, and interactive museum guides, this volume examines the interplay of semiotic codes (visual, oral, aural, haptic, physiological) within each case under scrutiny, thereby exposing both points of commonality and difference in the range of multimodal narrative experiences.

1 Introduction

Ruth Page

THE MULTIMODAL NATURE OF STORYTELLING

From conversational anecdotes told in face-to-face contexts through to film, digital storytelling, and beyond, narrative experiences employ a rich range of semiotic resources. The multifaceted nature of storytelling is nothing new, and is without doubt far more widespread, creative, and diverse than these initial examples signal. Stories might be spoken or written, and in their performance employ gesture, movement, facial expression, and prosodic elements such as voice quality, pitch, pace, and rhythm. Other narrative resources might include soundtracks, music, image, typeface, and hyperlinks, none of which are exceptional for their presence in stories of various kinds available at the outset of the twenty-first century. Put simply, stories do not consist of words alone. However, the multiple and integrated nature of semiotic resources used in storytelling is less simple to explain than to assert, and is long overdue for systematic and close attention in narrative theory. The dominant and interrelated trends that have shaped contemporary narrative studies in the last three decades provide the backdrop against which we can situate this now pressing need.

TRENDS IN CONTEMPORARY NARRATIVE STUDIES

The interdisciplinary expansion of narratology across the humanities and beyond means that the kinds of stories that now come under scrutiny extend much further than the literary texts typically prominent in classical narratology. Instead, audiovisual stories of various kinds (such as cinematic film or televised serials); stories that foreground image (including tableaux and cartoons of many kinds); and music (ranging from opera to hip-hop) have all been found to be of interest for their narrative function and potential. The stories that harness the rapid development of new technologies are part of this semiotic expansion, and in themselves are often characterized by multiple resources as they integrate words, image, sound, hyperlinks, and animation (for example, as seen on Web-based homepages, or creatively exploited in digital fiction). The innovative nature and social impact of recent technology developments mean that the question of digital media and its role in narrative processing has come to center stage, prioritizing the place of media in narrative studies more generally. The increasing diversity of narrative texts, combined with an openness to embrace methodology from other fields of inquiry, means that a narratology derived from the study of verbal resources alone can no longer be fully adequate to the task of interrogating storytelling in its broadest sense.

Narratologists have long recognized the limitations of early studies in the field. The contextualist rejection of structuralist abstraction led to an increased interest in the situated and process-oriented nature of narrative. The outcome of this was a shift away from earlier text-immanent analyses towards an attempt to account for the nonlinguistic factors that might be involved in the cover-all domain of “context.” Contextual sensitivity in narrative studies has been manifest in various ways. Within the sociolinguistic fields influenced by Labov and Waletzky’s (1967) groundbreaking work on personal narratives, the shaping force of variables such as the narrative participant’s gender, age, and ethnicity began to be given due attention. Likewise, discourse analysts debated the constraining and enabling nature of the immediate situation in which the narrative was elicited, for example, evaluating stories told in interview settings (Lambrou 2005), meal times (Blum Kulka 1993), and peer group conversation (Georgakopoulou 1997). Critically, studies influenced by sociolinguistics also sought to theorize the social function of narrative, for example, as a means of managing interpersonal relations or performing identity work. However, even in these more contextually oriented studies, the spoken stories investigated were often transcribed in such a way that paralinguistic features (including gaze or prosodic resources like intonation) appeared as annotations embellishing the verbal record of telling rather than being recognized as semiotic systems in their own right.

Literary studies of narrative have followed a similar move away from abstract, quasi-scientific formalist systems, but with a rather different focus. At least in some quarters, literary narratologists rejected empirical analysis of actual and immediate contexts of narration (Chatman 1990), instead conceptualizing “context” in broader, culturally oriented terms. A wealth of what might be termed “critical narratology” emerged, exposing connections between narrative and ideology. Again, the influence of gender, race, and ethnicity on storytelling is found in work that illustrates the move from “poetics to politics” (Currie 1998, 4), seen through recouping the value of previously marginalized writing. For example, feminist narratology sought to take account of literary texts by women (Lanser 1986), while others have used narratological tools to uncover postcolonial, and/or historicist perspectives in literary and cinematic texts (see Aldama 2003).

Cognitive Narrative Analysis developed as a separate but allied strand of contextualism in both literary and linguistic domains of study. Here the contextual focus was trained on the user and the cognitive processes employed in making sense of narratives. As Herman (2003) points out, Cognitive Narrative Analysis is inherently interdisciplinary (drawing on neurolinguistics, psycholinguistics, and anthropology, for example) and intermedial. Nonetheless, within narratology, analyses have tended to be devoted to literary texts or to fabricated examples, again given in verbal form (Herman 1997). In both cases, critical and cognitive trends in contextualist narratology rightly allude to the interactive nature of narrative production and reception. Nonetheless, the nonlinguistic resources of the contextual domain are extrapolated through discussion of given texts, and more specifically mediated through analysis of specific verbal choices from sample (often) literary texts to the neglect of other semiotic resources.

From the stance of the early twenty-first century, contemporary narrative analysis has clearly come a long way from its structuralist beginnings and focus on literary texts and analysis. Nonetheless, these origins have left significant legacies for narrative theory that are problematic if we are to take account of the full range of semiotic resources regularly found in storytelling. Most prominent are the assumption of monomodality and the privileging of verbal resources. Both in classical narratology and sociolinguistic accounts of narrative practice, the source material has predominately focused on verbal resources, either realized in the form of literary texts or written transcriptions of spoken data. Similarly, linguistics has functioned as a dominant paradigm in the development of narrative theory. From the initial use of Saussurean principles to distinguish between deep and surface structures, through to grammatical metaphors used to explain plot structure (Longacre 1983) or actantial relations (Greimas 1983) and still current in contemporary applications of systemic functional linguistics (Herman 2002) and corpus linguistics (Toolan 2008), tools from linguistics have been used to build evidence for narrative patterns and in turn underpin narrative concepts themselves. Both typical source data and conceptual bias in narratology leads to a situation not just of media-blindness, (the assumption that concepts derived from one format can be unproblematically transferred to another) but also mode-blindness. By this I mean that we should not assume that the dominance of the verbal mode thus far in narrative theorizing means that it is fully adequate to explicate the contribution of other modes (be they visual, verbal, kinaesthetic, or related to conventions such as dress codes). Instead, we need to reconfigure narrative theory and analysis in such a way that verbal resources are understood as only one of many semiotic elements integrated together in the process of storytelling.

MULTIMODAL THEORY: BACKGROUND AND DEFINITIONS

The chapters in this collection mark a paradigm shift away from mode-blindness. Instead, in various ways, the discussions develop and debate our understanding of narrative resources as multimodal phenomena. Although communication has always exhibited multimodal qualities, it is since the 1990s that conceptualizing multimodality has enjoyed renewed critical interest. Initially associated with the scholarly work of the New London Group, multimodality is rooted in semiotics but interfaces particularly with discourse analysis, systemic functional linguistics, and socially oriented work in Critical Discourse Analysis. As the prefix “multi-” suggests, work that might come under the heading of multimodality is a pluralistic enterprise, drawing on different perspectives and data. Baldry and Thibault thus describe multimodality as “a multipurpose toolkit, not a single tool for a single purpose” (2006, xv). The work in this volume is no exception to such pluralism, and the authors employ different kinds of analyses, survey different kinds of narratives, and take different stances towards multimodality itself. However, the diversity of multimodality is neither random nor ad hoc. Instead, multimodality is grounded in certain central claims, which I revisit now.

Above all else, multimodality insists on the multiple integration of semiotic resources in all communicative events. From this perspective, all texts are multimodal (Kress 2000, 187; Baldry and Thibault 2006, 19). Monomodality in comparison is not an actual quality of texts but rather a way of thinking about individual semiotic resources once abstracted from the communicative ensembles in which they occur. Multimodality’s insistence on the multiple resources used in communication is coupled with the democratic stance that all modes are equal. The extensive knowledge of verbal resources (and relative neglect of other systems) in certain quarters is not necessarily a result of the sole existence of language, but rather a continuing aftereffect of the centrality of linguistics (Kress 2000, 193). The assertion of modal democracy does not deny that when in use particular modes from the ensemble can relate to one another in various ways (by complementing or contrasting in the meanings they construct, or being arranged in hierarchies), and that in any given text, one mode may dominate. But even when one mode dominates, this does not mean that the other either less prominent or less well-recognized semiotic resources are not in play. Multimodality requires us to fundamentally rethink the position of verbal resources within semiotic configurations (here specifically within narrative theory) and to ask what the narrative system would look like if we examined other modes with equal priority.

If multimodality calls attention to the range of semiotic resources used in communication, it also seeks to draw connections between them. In some work this is expressed as a unifying tendency, where multimodality is viewed as “common semiotic principles [that] operate in and across different modes” (Kress and van Leeuwen 2001, 2). Typically, if not ironically, these semiotic principles thus far have borrowed and reworked concepts from various subfields of linguistics (for example, grammars, framing, modality, elaboration). The question of how far such metaphors are useful is contentious and not without debate (Kress and van Leeuwen 2001, 124–25), and indeed the wider issue of whether it is possible or not to create a single system of principles for all modes lies beyond the scope of this volume. Instead, applying multimodal principles to narrative analysis must negotiate the tensions between finding transferable frames of reference that enable comparisons to be made between different modal combinations and avoiding the levels of abstraction that negate the inevitable influence of localized narrative contexts.

The process-oriented nature of multimodal analysis forces us to return to the multifaceted nature of narrative context from a fresh perspective. Drawing on the work of Malinowski (1935), multimodal theory distinguishes between context of situation (the localized situation in which words are uttered) and context of culture. In narrative analysis, these two contextual facets have often been subject to separation across disciplinary divides. In narratological criticism of literary texts, discussion of context appears closer to the broader cultural concerns (for example, as treated in subfields such as feminist narratology). In sociolinguistic studies of narrative, the factors involved in “context of situation” are more strongly foregrounded through typical data (often narratives that occur in talk of various kinds) and methodological preoccupations. Multimodal narrative analysis reminds us that such separation is illusory and that not only does all language operate within cultural systems, so narratives (whether naturally or technologically produced) are received in local contexts by actual audiences.

The principles of multimodality provide a significant means of expanding the project of contemporary narratology. Like other recent narrative studies that have indicated a turn to media (Ryan 2004), multimodality brings into the frame a number of issues that have previously been overlooked in the verbal hegemony of classical narratology, namely questions of remediation (Bolter and Grusin 1999) and the narrative affordances of specific media. The distinct contribution of a multimodal analysis is to shift the focus from media to modes, and to focus less on a comparison of specific media but instead to reconceptualize all narrative communication as multimodal. Clearly the enormity of fully examining the semiotic resources used in narrative (in its broadest sense) falls far beyond a single framework, and must by necessity be diverse, interdisciplinary, and integrative. On the one hand, this opens up a wealth of possible connections between methodology, source material, and critical perspective. On the other, such diversity is also open to terminological confusion and controversy. Central to the debates and analyses that follow in this volume are the terms “medium” and “mode,” which are notoriously slippery and ambiguous. In order to anchor the chapters that follow within some kind of critical context, I move now to sketch out some definitions against which multimodal narrative analysis might position itself.

CRITICAL TERMINOLOGY

The modes described in multimodal analysis refer specifically to semiotic modes (as opposed to other, specialist uses of the term). Thus a mode is understood here as a system of choices used to communicate meaning. What might count as a mode is an open-ended set, ranging across a number of systems including but not limited to language, image, color, typography, music, voice quality, dress, gesture, spatial resources, perfume, and cuisine. The status of a mode is relative and may vary according to its instantiation within a given community. For example, the potential of particular scents to carry meaning may be high for perfume creators but less so for other individuals who are not trained to differentiate between them. Given the fluid nature of modes, central questions are how, why, and to what extent some modes become particularly privileged in certain contexts. The collection of work in this volume concentrates on narrative as a significant mode by which humans make sense of themselves and the world around them. However, the principles of multimodality remind us that narrative is only one mode amongst many, and a multimodal narrative analysis should not be taken to reinforce narrative imperialism but rather might serve to broaden our understanding of how and why narrative functions in relation to other modes in different contexts.

Semiotic modes are realized materially through particular media. Although closely intertwined with multimodality, the analysis of medium is a separable and independent issue. As Kress and van Leeuwen put it, “multimodality and multimediality are not quite the same thing” (2001, 67). In probing this issue further, it is worth distinguishing between the different uses of “medium.” Ryan (2004) rightly points out that two meanings are current in theoretical discussions. First, medium refers to the physical materials used when conveying communication (e.g., print, airwaves, radio). Second, it may also be defined as a channel of communication. Both meanings are important in multimodality, which recognizes the potential of materiality for meaning-making. However, modes and medium cannot be mapped univocally. As Kress and van Leeuwen (2001) argue, a given semiotic mode can appear in different media. For example, language can be spoken or written. Conversely, and fundamental to the project of multimodality, different modes can be realized in the same medium, as demonstrated through the use of image and words in comics or illustrated stories. As materiality may function as a source of difference and hence meaning, multimodal narrative analysis treats media as one element operating within the wider ensemble of semiotic modes used in storytelling. It is less concerned with media-specific affordances in isolation. Thus multimodal narrative analysis can be focused on a single medium (print literature) or may survey narratives from different media (audiovisual and written). However, regardless of whether one or more media are involved, the focus remains on the integration of semiotic resources, not the comparison of media alone.

The relationship between materiality and multimodality draws attention to the physical work involved in narrative processing, both in the use of tools and technology, and also by the human body and its sensory organs. Kress and van Leeuwen (2001, 66) argue that although semiotic modes are grounded in physiological experience, they are not naturalistically equated to sensory modes. The senses being addressed, the priority between sensory tracks and the various affordances they yield within a storytelling context thus contribute to but should not be taken as synonymous with semiotic mode in its widest sense. Rather, recognizing that narrative is not just a means of artistic expression but a funda...