Advancing Multimodal and Critical Discourse Studies

Interdisciplinary Research Inspired by Theo Van Leeuwen’s Social Semiotics

Sumin Zhao, Emilia Djonov, Anders Björkvall, Morten Boeriis, Sumin Zhao, Emilia Djonov, Anders Björkvall, Morten Boeriis

• Publisher: Routledge
• Year: 2017
• Print ISBN:
  9781138697638
• eBook ISBN:
  9781315520995
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

As a founder and leading figure in multimodality and social semiotics, Theo van Leuween has made significant contributions to a variety of research fields, including discourse analysis, sociolinguistics, communication and media studies, education, and design. In celebration of his illustrious research career, this volume brings together a group of leading and emerging scholars in these fields to review, explore and advance two central research agendas set out by van Leeuwen: the categorisation of the meaning potential of various semiotic resources and the examination of their uses in different forms of communication, and the critical analysis of the interaction between semiotic forms, norms and technology in discursive practices. Through 11 cutting-edge research papers and an experimental visual essay, the book investigates a broad range of semiotic resources including touch, sound, image, texture, and discursive practices such as community currency, fitness regime, film scoring, and commodity upcycling. The book showcases how social semiotics and multimodality can provide insights into the burning issues of the day, such as global neoliberalism, terrorism, consumerism, and immigration.

1 Social Semiotics

A Theorist and a Theory in Retrospect and Prospect

Emilia Djonov and Sumin Zhao

1. Theo Van Leeuwen’s Social Semiotics

When we look back at the body of work produced by Theo van Leeuwen, we cannot help but be struck by its breadth and evolving nature, its range in subjects and perspectives, and its transdisciplinary reach. For almost four decades, Van Leeuwen has examined phenomena as diverse as film (Van Leeuwen, 1985, 1991a, 2014; Van Leeuwen & Boeriis, 2017) and children’s toys (Caldas-Coulthard & Van Leeuwen, 2001, 2002, 2003; Van Leeuwen, 2009b), music (Van Leeuwen, 1991b, 1999) and school textbooks (Van Leeuwen, 1992, 2000; Van Leeuwen & Humphrey, 1996; Van Leeuwen & Kress, 1995), women’s magazines (Machin & Van Leeuwen, 2003, 2007) and kinetic art (Van Leeuwen, 2015a), news journalism (Van Leeuwen, 2006b; Van Leeuwen & Jaworski, 2002) and semiotic software such as PowerPoint (Djonov & Van Leeuwen, 2012, 2013; Van Leeuwen & Djonov, 2013; Zhao, Djonov, & Van Leeuwen, 2014; Zhao & Van Leeuwen, 2014). Van Leeuwen’s work also gives voice to wide-ranging theories and perspectives, as he draws inspiration from the Paris and Prague schools of semiotics, especially the work of Roland Barthes and Roman Jakobson; Foucault’s theory of discourse; the Bauhaus art and design movement; Rudolf Arnheim’s psychology of visual perception; Raymond Murray Schafer’s studies of music and sound; John Gage’s theory of colour; and the anthropologist Erving Goffmann, to name just a few. This richness has enabled Van Leeuwen’s own theories of legitimation in discourse (Van Leeuwen, 2007; see also Chapter 3 in this volume), the role of discourse in recontextualising social practice (Van Leeuwen, 2008a; see also Chapters 11 & 12 in this volume), new writing (Van Leeuwen, 2008b; see also Chapter 10 in this volume), and semiotic technologies (Djonov & Van Leeuwen, in press; Van Leeuwen, 2010; Zhao et al., 2014) to provide tools for understanding the seismic social, cultural, and political changes in the past two decades and thereby have influence beyond semiotics, communication studies, and applied linguistics, in fields such as education, arts, design, media, cultural, and management studies.

While diverse in scope and perspectives, Van Leeuwen’s work is grounded in social semiotic theory. Van Leeuwen’s key theoretical contributions to social semiotics are captured in his book Introducing Social Semiotics (Van Leeuwen, 2005), written in his staple accessible yet intellectually rich style, with intriguing examples from a wide array of semiotic practices. For Van Leeuwen, social semiotics is “not ‘pure’ theory, not a self-contained field” but “a form of enquiry” that “comes into its own when it is applied to specific instances and specific problems” (Van Leeuwen, 2005, p. 1). It is thus a theory that is both an ‘appliable’ (Halliday, 1985) and necessarily agile and interdisciplinary. Social semiotic enquiries pursue three central goals:

1. collect, document and systematically catalogue semiotic resources—including their history
2. investigate how these resources are used in specific historical, cultural and institutional contexts, and how people talk about them in these contexts—plan them, teach them, justify them, critique them, etc.
3. contribute to the discovery and development of new semiotic resources and new uses of existing semiotic resources. (Van Leeuwen, 2005, p. 3)

Central to Van Leeuwen’s theory of social semiotics is the notion of ‘semiotic resource’, which reflects Halliday’s (1978) model of language as a social semiotic resource whose meaning-making potential is dynamic, simultaneously shaped by and shaping the social contexts in which it is employed:

Semiotic resources have a meaning potential, based on their past uses, and a set of affordances based on their possible uses, and these will be actualised in concrete social contexts where their use is subject to some form of semiotic regime.

(Van Leeuwen, 2005, p. 285)

It also integrates a strong focus on materiality through Gibson’s (1979) concept of ‘affordances’, the perceptible, physical qualities of objects that, together with the needs and interests of users, define their possible uses.

Semiotic resources have a meaning potential, based on their past uses, and a set of affordances based on their possible uses, and these will be actualized in concrete social contexts where their use is subject to some form of semiotic regime.

(Van Leeuwen, 2005, p. 285)

Studying semiotic resources for Van Leeuwen entails examining their roles in specific social practices and cultural-historical contexts. The origin of such an approach can be traced back to Malinowski’s (1923) conceptualisation of ‘context of situation’ and ‘context of culture’. Van Leeuwen’s work also responds to Hodge and Kress’s (1988) pan semiotic ambition that “texts and contexts, agents and objects of meaning, social structures and forces and their complex interrelationships together constitute the minimal and irreducible object of semiotic analysis” (p. viii). In particular, he draws on ethnographic approaches in the social sciences and considers not only how people use semiotic resources in specific socio-historical contexts but also the ways in which they talk about and legitimate (aspects of) these practices. This approach underpins Van Leeuwen’s (2005, p. 47–68) inventory of semiotic regimes that govern people’s meaning-making. The inventory includes (i) rules of personal authority, such as those developed by observing and conforming to trends, emulating role models, and drawing on the opinion of experts as well as rules imposed by people in power, and (ii) rules of impersonal authority, which are imposed through writing (the law, religion, etc.), tradition, and the design of technologies (e.g., PowerPoint) and objects (e.g., furniture) used in communication. Awareness of the emergence and changes in such norms is key to understanding and contributing to semiotic change.

As Van Leeuwen’s work focuses on the relationship between meaning-making or semiotic resources, the interests/agency of meaning-makers, and the ways in which specific institutional and broader social contexts govern the use of semiotic resources, it has left enduring legacy in two strands of discourse studies—multimodal and critical discourse studies, and ultimately served as a catalyst for their merger (Djonov & Zhao, 2014; Machin & Mayr, 2012; Machin & Van Leeuwen, 2016; Van Leeuwen, 2013), as reflected in special issues on ‘Critical Analysis of Musical Discourse’ (2012) and ‘Multimodal Critical Discourse Studies’ (2013) of the journal Critical Discourse Studies; on ‘Multimodality, Politics and Ideology’ (2016) in Journal of Language and Politics; and on ‘Gender and Multimodality’ (2016) in Gender and Language.

2. Multimodality and Multimodal Discourse Studies

Van Leeuwen’s most ground-breaking contribution consists in co-founding, alongside Gunther Kress, multimodality as a transdisciplinary field of research concerned with the meaning-making potential, use, and development of different semiotic resources. In two seminal publications, Reading Images: The Grammar of Visual Design (2006 [1996]) and Multimodal Discourse: The Modes and Media of Contemporary Communication (2001), Kress and Van Leeuwen have laid the groundwork for the two main directions in multimodal studies:

1. exploring the use and mapping the meaning-making potential of individual semiotic resources,
2. studying how choices from various semiotic resources interact to create meaning multimodally.

Reading Images incorporates insights from iconography, structural semiotics, Gestalt psychology, film, and the fine arts and explores a rich variety of Western-culture visual texts from different historical periods (advertising and news images, maps and technical diagrams, pages from magazines, picture books, and textbooks, three-dimensional objects such as sculptures and toys, and web pages). The book presents an analytical framework based on two central tenets of Halliday’s systemic functional linguistics (Halliday & Matthiessen, 2004). The first is that every act of communication simultaneously constructs three broad types of meaning, or ‘metafunctions’:

• ideational/representational—representing patterns of experience (as configurations of processes, participants and circumstances) and the logico-semantic relations between them
• interpersonal/interactional—enacting social interactions, relations, attitudes, and values
• textual/compositional—interweaving ideational and interpersonal meanings into cohesive and coherent units, i.e. texts.

The second tenet is that the meaning potential of semiotic modes can be modelled as systems of interrelated choices, paradigmatically, where each option has a distinctive structural realisation.

While describing SFL as “a good source for thinking about all modes of representation” (Kress & Van Leeuwen, 2006 [1996], p. 20), Reading Images adds two interrelated caveats. The first is that although different modes may have the potential to make the same general types of meaning, their affordances and formal organisational principles differ (e.g., temporal organisation in spoken language vs. spatial organisation in images and spatio-temporal organisation in dance). To illustrate, as modelled in Halliday’s system of ‘modality’ for English grammar (Halliday & Matthiessen, 2004, p. 143–150), linguistic resources for representing different versions of reality and truth values for different communities include modal verbs and modal adjuncts that construct degrees of probability and obligation between the polarity values of ‘yes’ and ‘no’. In visual representations, as Kress and Van Leeuwen (2006 [1996]) show, modality relies on several cues such as colour saturation, colour differentiation, brightness, and detail, and their interaction may lead viewers to ‘read’ a picture as more or less naturalistic, abstract, sensory, or technical. The second is that the use of the term ‘grammar’ in the book’s title is not intended to suggest that the visual mode has organisational or grammatical structures similar to those found in language, but to emphasise the need for visual analysis to move beyond interpreting the meaning of individual elements (e.g., a particular colour or shape) and represented objects, beyond what Kress and Van Leeuwen see as analogous to ‘lexis’ in language, and to examine the structures such elements form within a visual composition such as a photograph or a webpage.

In contrast to Reading Images, which focuses on visual design as a distinct mode, Kress and Van Leeuwen’s Multimodal Discourse (2001, p. 2) presents “a view of multimodality in which common semiotic principles operate in and across different modes”. Their key argument for adopting this type of multimodal perspective is that shifts in the semiotic landscape, and particularly advances in digital technologies, mean that non-specialists are increasingly able to select from and combine semiotic resources (e.g., typography, sound, layout) previously associated with discrete and highly specialised domains and professions. Studying contemporary communication thus requires “a unified and unifying semiotics” (ibid., p. 2).

Multimodal Discourse presents several fundamental principles for a unified theory of multimodality. The first is that the study of multimodal communication should focus on identifying broad semiotic principles that apply across different semiotic resources (in accordance with their unique affordances) and semiotic practices. These principles can then be built into frameworks for analysing multimodal interaction. Modality is one such principle, as not only language and images, but sound, too, can represent different degrees and kinds of truth depending on the extent to which it appears authentic or manipulated with technologies (Van Leeuwen, 1999, see also Chapter 8 in this volume).

The second key idea is that multimodal analysis must always consider semiotic resources in relation to specific, situated social practices, and should engage with each of four layers, or strata, of communication:

1. Discourse, “socially constructed knowledge(s) of (some aspects of) reality” (p. 4);
2. Design, blueprints or conceptualisations of the ways one or more discourses can be materialised and embedded in particular interactions through semiotic objects or events that involve certain combinations of semiotic resources;
3. Production, the material articulation of a semiotic object or event;
4. Distribution, “the technical ‘re-coding’ of semiotic products and events, for purposes of recording […] and/or distribution” (p. 21).

The stratum of discourse invites investigation into the relationship between epistemology (see also Chapters 4 & 5 in this volume), social roles and meaning-making. Kress and Van Leeuwen (2001) also emphasise the fluid boundaries between the strata of design and production, pointing out that “at any moment the implementer of a design can become a designer in respect to a particular facet of the productive process” (p. 56), as in jazz improvisation. The concepts of production and distribution, on the other hand, draw attention to the role of materiality and technologies for producing/recording and distributing semiotic products and events in specific communicative practices.

One of Van Leeuwen’s distinctive contributions to multimodality lies in developing frameworks for studying material resources such as colour, texture, sound and (kinetic) typography, thereby drawing attention to semiotic resources that have generally been marginalised in linguistics, semiotics, and discourse analysis and providing tools for explicitly teaching and discussing them in semiotic theory as well as semiotic practice.

A strong focus on materiality in semiosis underpins Van Leeuwen’s unified framework for analysing sound. Presented in Speech, Music, Sound (1999), it reflects his background in film production (where speech, music and sound can all be part of a soundtrack), jazz music practice, and his earlier research (Van Leeuwen, 1982) on intonation and rhythm (rhythm is a key organising principle for time-based modes and media such as radio). The framework incorporates principles from phonology, musicology, the psychology of perception, and conceptual metaphor theory.

As in Reading Images, in Speech, Music, Sound, tools from systemic functional linguistics such as the metafunctions provide a springboard for examining sound. For Van Leeuwen, sound is better equipped for realising interpersonal and textual rather than ideational meanings. He also argues that a ‘bottom-up’ approach (see also Chapter 8 in this volume), starting from material qualities (e.g., timbre, tempo), rather than larger structures, is more suitable for mapping the meaning potential of sound and other material semiotic resources, because compared to language they construct meaning “quite differently, on the basis either of an experiential meaning potential, hence grounded […] in our bodily experience of [their] materiality, and/or provenance, hence grounded in intertextuality” (Van Leeuwen, 1999, p. 192). Experiential meaning is based on our ability to extend prior physical experiences metaphorically into knowledge, as Lakoff and Johnson (1980) have argued for linguistic metaphors (e.g., ‘things are looking up’).

In an earlier paper titled ‘Taste in the Framework of a Semiotics of Materiality’, for instance, Van Leeuwen (1998, p. 149) explains that such semiotics “must be placed in the still broader context of a semiotics of action, as it is only in our physical experience of materials, that the qualifies of those materials can be perceived”. For taste, he would begin by inventorising the physical actions involved in tasting—touching, linki...

Table of contents

• Cover
• Title
• Copyright
• Contents
• List of Figures
• List of Tables
• Preface
• Author Bios
• 1 Social Semiotics: A Theorist and a Theory in Retrospect and Prospect
• 2 Changing Academic Common Sense: A Personal Recollection of Collaborative Work
• 3 “Strangers in Europe”: A Discourse-Historical Approach to the Legitimation of Immigration Control 2015/16
• 4 The Limits of Semiotics—Epistemology and the Concept of ‘Race’
• 5 Can a Sign Reveal Its Meaning?: On the Question of Interpretation and Epistemic Contexts
• 6 Towards a Multimodal Social Semiotic Agenda for Touch
• 7 Reading That Which Should Not Be Signified: Community Currency in the UK
• 8 A Sound Semiotic Investigation of How Subjective Experiences Are Signified in Ex Machina (2014)
• 9 Unravelling the Myth of Multiple Endings and the Narrative Labyrinth in Mr. Nobody (2010)
• 10 New Codifications, New Practices: The Multimodal Communication of CrossFit
• 11 The ‘Semiotics of Value’ in Upcycling
• 12 Multimodal Recontextualisations of Images in Violent Extremist Discourse
• Revisiting the Family Silver: A Visual Essay on the Grammar of Visual Design
• Index