Multimodal Theory and Methodology

For the Analysis of (Inter)action and Identity

Sigrid Norris

• Publisher: Routledge
• Year: 2020
• eBook ISBN:
  9781000041842
• Topic: Sprachen & Linguistik
• Subtopic: Bibliotheks- & Informationswissenschaft

This concise guide outlines core theoretical and methodological developments of the growing field of Multimodal (Inter)action Analysis.

The volume unpacks the foundational relationship between multimodality and language and the key concepts which underpin the analysis of multimodal action and interaction and the study of multimodal identity. A focused overview of each concept charts its historical development, reviews the essential literature, and outlines its underlying theoretical frameworks and how it links to analytical tools. Norris illustrates the concept in practice via the inclusion of examples and an image-based transcript, table, or graph.

The book provides a succinct overview of the latest research developments in the field of Multimodal (Inter)action Analysis for early career scholars in the field as well as established researchers looking to stay up-to-date on core developments and learn more about a complementary approach to systemic functional and social semiotic frameworks.

1 Multimodal Theory and Methodology

Background and Definitions

Introduction

This chapter is not only an introduction to multimodal discourse analysis, but more specifically, an introduction to multimodal (inter)action analysis. The chapter is divided into two parts. The first part discusses how multimodality relates to language and outlines how and why multimodal theory and methodology are important for linguistics. The second part goes into some detail in relation to theory and methodology, briefly discussing literature in regard to relevant concepts that are integral to the explicated theory and methodology. The text that follows each textbox of concise information goes into more detail about the concept, adding a longer example and/or a table and/or a graph to illustrate the concept further. Examples, some real and some envisioned, are used throughout merely to enhance comprehension of the analytical tools.

The chapter is brief and can be used as a guide for understanding the theoretical background of how to analyze everyday (inter)action and identity production. The aim of the chapter is to explicate the theoretical and analytical bases of multimodal discourse analysis, or more specifically, multimodal (inter)action analysis.

This chapter clearly explicates five concepts needed for the analysis of actions and interactions, listing for each in a gray textbox:

1. The history of the concept, that is, when and where it was developed and when and where it was first published. Based on this information, the reader can then go to the reference list and find detailed bibliographical information for further reading.
2. The theory that is embedded in the concept.
3. How the concept is used as an analytical tool to analyze data.
4. Brief practical example(s), elucidating the concept.
5. Some analytical capabilities of the concept.

The following text after each textbox of concise information goes into more detail about the concept, adding a longer example and/or a table and/or a graph to illustrate the concept further.

Multimodal (Inter)action Analysis

History

Multimodal (inter)action analysis, a theory and methodology for the analysis of action, interaction and identity, was initially developed in Norris (2002) and first published in Norris (2004a) with a focus on the analysis of actions and interactions. A first publication on identity production is Norris (2005), followed by other writings and a first book outlining the theory and methodology for the study of identity (Norris 2011a).

Theoretical Essence

The essence of the theory is to analyze human action, interaction and identity holistically on micro, meso and macro levels, which are viewed as each consisting of multiple levels. The theory allows one to study not only distinct levels but also the multiplicity of levels.

Methodological Essence

The methodology is built upon a strong theoretical foundation. Clarity of units of analysis and diverse analytical tools allow for consistency of analysis without determining discoveries or findings. Clarity of methodology allows for replicability and therefore reliability.

Multimodality

Actions, interactions and identity are necessarily multimodal. By examining what people do, we realize that all modes together build one coherent system of communication.

Examples

1. The study of face-to-face interactions.
2. The study of online or mixed online and face-to-face interactions.
3. The study of YouTube, music videos, films, etc.

Some Analytical Capabilities

1. Actions, interactions and identity can be analyzed in their complexity. This means that the context of an (inter)action is analyzed as well as the multimodal (inter)action under scrutiny.
2. The multiplicity of (inter)action can be analyzed due to the various analytical tools that allow for such analysis (see below and Chapters 2 and 3 for detail).
3. The fact that social actors in (inter)action do not focus upon the same things, that is, that each social actor produces and experiences a co-produced (inter)action differently, can be analyzed.

As soon as we speak of multimodal interaction, we are moving beyond the most theorized and best understood mode, the mode of language. In order to do a multimodal discourse analysis, we use analytical tools that help us to make sense of the multimodal complexity. Of course, analytical tools are not unusual in linguistics. Just think of phonetics, here, we find analytical tools. As an example, we can take a look at the international phonetic alphabet (IPA), or specifically at the vowel chart (Figure 1.1). This vowel chart is an analytical tool that allows us to place vowels of the world’s languages in relation to one another.

Figure 1.1 Analytical tool: the IPA vowel chart (represented from www.ipachart.com).

With this tool, we can listen to vowels and then place them in relation to the frontness, backness, etc. on the chart. Everyone who has learned to listen to vowels and place them, will, when done correctly, place the same vowel on the chart in an identical place. This tool, the vowel chart, thus allows for reliability and replicability of findings.

But now, let us contemplate syntax as one of our linguistic theories and let us briefly look at Chomsky’s universal grammar (Chomsky 1982; Cook and Newson 1996). No matter if we agree with this theory or not, the theory can help us understand an aspect of languages. Just think of a simple sentence such as The child drew an elephant. With a universal grammar approach, we can draw a tree diagram and this tree diagram will reflect the grammatical structure of the sentence (Figure 1.2).

Figure 1.2 Analytical tool: a tree diagram (replicated from Cook and Newson 1996: 4).

Each student who has learned the universal grammar can draw such a tree diagram and, if done correctly, all tree diagrams of this sentence will be identical. Here, the tree diagram is the analytical tool that allows us to understand and portray the underlying structures in sentences.

In both of our examples, in phonetics as well as in syntax, theoretical principles allowed for the development of analytical tools. The analytical tools in turn allow for the analysis of language without determining where vowels of an unknown language would be positioned on the IPA chart, and without predetermining how a tree diagram of a sentence in an unknown language would need to be drawn. Similarly, we find analytical tools in phonology, morphology, semantics and so on, and because of a multitude of analytical tools, we can systematically analyze languages. Similarly, we find analytical tools in discourse analytical approaches. Just think of the turn at talk in conversation analysis (Sacks et al. 1974).

Each time a new branch of linguistics was developed, new theories were created and new analytical tools were devised. Similarly, when we moved beyond the study of language in interaction, a new theory was needed and new analytical tools had to be devised. Of course, just as the universal grammar was not developed based on the theoretical foundations or the analytical tools of the international phonetic alphabet, so the theory of multimodal (inter)action is not based on the theoretical foundation of spoken or written language. Rather, language, while being the most researched and well-understood mode, here becomes a (very important) part within the multimodal constellations when people act and interact, illuminating language from a new perspective, allowing us to gain new insight into linguistics.

(Inter)action

History

Beginning of the development in Norris (2002) with a first publication in Norris (2004a), where, however, the term “interaction” was still used in its more common understanding, further developed and clearly stated as (inter)action in Norris (2011a).

Theoretical Essence

(Inter)action is defined as social actor(s) acting or interacting with others, the environment and/or objects.

Methodological Essence

One person can (inter)act alone or multiple social actors may (inter)act together.

Examples

1. Peter plays a computer game.
2. Laura and Peter play a card game together.
3. Mary has a meeting with several of her coworkers.

Some Analytical Capabilities

1. Discovery of how a single individual (inter)acts with the environment and objects.
2. Discovery of how two or multiple people simultaneously (inter)act with each other, the environment and objects.

When speaking of (inter)action, we always take human action and interaction as our point of interest. With this, we are neither specifically interested in one or another mode. Ye...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedication
• Table of Contents
• List of Figures
• List of Tables
• Acknowledgments
• Preface and General Introduction
• 1 Multimodal Theory and Methodology: Background and Definitions
• 2 Examining Multimodal Action and Interaction: Theory and Analytical Tools
• 3 Examining Multimodal Identity: Theory and Analytical Tools
• Index