Introduction to Multimodal Analysis

Per Ledin, David Machin

• Publisher: Bloomsbury Academic
• Year: 2020
• ISBN:
  9781350069152
• Edition: 2
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

Introduction to Multimodal Analysis is a unique and accessible textbook that critically explains this ground-breaking approach to visual analysis. Now thoroughly revised and updated, the second edition reflects the most recent developments in theory and shifts in communication, outlining the tools for analysis and providing a clear model that students can follow. Chapters on colour, typography, framing and composition contain fresh, contemporary examples, ranging from product packaging and website layouts to film adverts and public spaces, showing how design elements make up a visual language that is used to communicate with the viewer. The book also includes two new chapters on texture and diagrams, as well as a helpful image index so students can clearly understand how images and multimodal texts can be analysed from different perspectives. Featuring chapter summaries, student activities and a companion website hosting all images in full colour, this new edition remains an essential guide for students studying multimodality within visual communication in linguistics, media and cultural studies, critical discourse analysis or journalism studies.

1

What is multimodal analysis?

This chapter presents the basic theoretical framework for the chapters that follow. Here we explain the nature of the approach and tools this book presents. What we present is a theory of visual communication which is linked to how society and power operate. It is about the way that visual materials come to house our ideas and values and shape and structure how we act and interact. We also place this into some of the shifts that have taken place in visual communication, particularly in the second decade of the twenty-first century. This is part of the key to revealing how the elements and features of visual communication have become used in new and more integrated configurations.

The idea of choice in communication

As we began to explain in the introduction to this book, the approach we use is inspired by a theory of communication in linguistics pioneered by Halliday (1978). Halliday himself called this a ‘social semiotic’ theory of communication. You may have come across the term ‘semiotics’ (Barthes, 1977) in other forms of visual analysis where it has been used to look at things like symbolism in advertisements. So we might look at the webpage for the plastic surgery company seen in Figure 1.1 and say that the body we see as the main element on the page connotes sleek, glowing, beauty. Barthes was interested in the way that such connotations could then be associated with the product or service being advertised. In this case plastic surgery is not about medical procedures, grasping at lost youth, body management or alteration. The surgery itself is omitted and the fantasy of the outcome foregrounded. The semiotics of Barthes may also then look at the meanings of other elements on the design, such as the abstract drawing of the figure to the top left, connoting something artistic, or ‘fine art’. But these are just two parts of the whole design.

Social semiotics is different from this however, as it wants a more systematic form of analysis. We need to look at the whole of the composition. And this is very much influenced by the tradition of linguistics where we do not try to grasp the meaning of a sentence only by attending to the connotations of one single word in it. We would need to understand how it was part of a whole, how its meaning was realized through its placement with other words and tenses and where it was positioned in that sentence.

Figure 1.1 Plastic surgery website, California Surgical institute (http://www.californiasurgicalinstitute.com/).

In the case of the plastic surgery webpage in Figure 1.1, it is clear that the fonts are important. They are formed by quite fine lines emphasizing lightness rather than wider lines to suggest weight and substantiality. They appear rather tall and elegant rather than short and squat. The colours emphasize brightness rather than darkness. There is a lot of empty space on the page, suggesting ease and room to breathe and simply the luxury of space. A state-run hospital may place less emphasis on space and luxury. While many state-run hospitals are also now heavily branded, foregrounded in some images may be machines used for medical procedures and more use of space to show these in order to indicate ‘resources’. The private clinic here does not represent these at all, but rather luxury, ease, lightness and optimism. The plastic surgery clinic also uses a particular kind of level of realism in its representation of the body and of the setting, which is an empty space.

Semiotic choices

The notion of choice is important in how a social semiotic approach deals with what is taking place here. The designer of the webpage (seen in Figure 1.1) had a choice as regards the thickness of the typeface, the kinds of colours (tones, saturation and purity levels), levels of brightness, where to put the borders on the page, how thick and what shape they should be and where to locate the cut-lines and the words. Such choices have been made here with a sense of the way that people will understand such a combination in this particular socio-historical present and in regard to what would be the target group of the clinic.

Social semiotics is a theory of language and communication based on the idea of choices between available options. So in order to carry our an analysis of the plastic surgery clinic webpage we would be interested in describing the choices and meaning potentials of semiotic resources such as fonts, colours and uses of borders. We could, for example, ask if the font was curved, angular, narrow or wide. Or what meanings dilute rather than saturated colours create. Choosing the thinner typefaces or the dilute colours on the webpage in Figure 1.1 can provide messages about a kind of experience of plastic surgery – certainly not one that foregrounds surgical instruments, cutting of actual flesh, removal of unwanted body parts.

By a ‘system of choices’ we mean that in, for example, language, when, as speakers, we want to describe a person, thing, place or idea, we have a range of options we can choose from. Our choice will depend on our needs in any particular context. For example, if we describe a person there are choices relating not only to size and shape but also to things like religion and ethnicity. Such words, in one sense, do not really ‘describe’ that person in any neutral sense but allow us to use culturally evolved terms to ‘evoke’ things about them from a particular point of view. And, of course, the choices we make will reflect what we want to achieve in that context – such as persuade someone to like or dislike a person.

In Halliday’s social semiotic model of language, which was a huge inspiration for multimodality, the analyst would be interested in making an inventory of the kinds of choices available to speakers. So, for example, we could create an inventory of the choices available for indicating levels of certainty.

I am doing that

I will do that

I must do that

I may do that

I would do that

I could do that

Making such an inventory, and understanding the meaning potential for each, allows us to better look for the instances of choices made in, say, speech or a text. If a politician comments that ‘we must act for the good of the nation’, it is not the same level of commitment as saying ‘we will act for the good of the nation’. As such, politicians often hedge their promises and commitments carefully.

This notion of choice was taken up in multimodality to enable a more systematic and predictive form of analysis. This approach based on creating inventories of choices was different than some earlier forms of visual analysis that had been based more on open forms of analysis of individual elements in visual communication. In media and communication and cultural studies, there had been great reliance on the approach of Roland Barthes (1977), for example, which we began to discuss in the introduction, who looked at things like the meaning of individual elements in advertisements rather than working from an inventory of meaning potentials. In multimodality, we would be interested in all of the design choices found on an advertisement. This would include fonts, colours, the layout of the design, its shape and texture. For example, Van Leeuwen (2005) has shown that fonts can be described as regards a limited range of qualities such as curvature versus angularity, narrow versus wide, heavy versus light. Each quality can communicate quite specific ideas. Lighter and more vertically oriented fonts might be used to communicate a diet food, for example.

Materials and affordances

In multimodality, as well as an interest in the smaller level choices, such as fonts, there is also an interest in the form of communication that we think of as the ‘semiotic materials’ (Ledin and Machin, 2018). When we encounter and interact with instances of communication, such as the plastic surgery website, or say the café interior pictured in Figure 1.2, we certainly do not attend to them as regards the individual smaller choices described above, even though these may be part of our experience of them. Rather, we encounter them as whole things which we can think of as semiotic materials. This is important for the process of analysis that this book allows. It allows us to be mindful of the reason the instance of communication is used and also how it is in a sense already meaningful to us. When we enter a trendy café in Stockholm or London to buy a Fairtrade cappuccino and organic croissant, like the Fairtrade coffee package we discussed in the introduction, we already know about things like cafés, global inequality, chic style, climate change and so on. And we know that shopping for ethical products can be a reasonable way to express moral and political opinions – even though many analysts would question whether this can actually ever bring about any significant changes.

Figure 1.2 Interior of Heritage Bicycles Café. Image from ‘Food & Wine – Hybrid Coffee Shops’ by Michael Salvatore, Heritage Bicycles Blog, 4 December 2014.

If, to begin with, we take the case of the Fairtrade food package seen in Figure 0.1, this is a form of communication that has evolved to do specific things as processes of commodification, standardization and consumerism have progressed since the end of the nineteenth century. Things like monuments, Facebook, cafés and food packages are all semiotic materials which have evolved to do very different kinds of things. This is because semiotic materials themselves also come loaded with ideas and assumptions, sometimes called affordances (Gibson, 1979), and shape communication and social behaviour. So packaging shapes how we communicate about food, and also here about ethical behaviour and fashion.

The café in Figure 1.2 is loaded with assumptions about style and ethics, and it also shapes how we behave. As we see, it has rough wooden benches and tables, communicating something authentic and unprocessed, as does the stripped back industrial look of other parts of the café. But the tables are also designed so that people can sit and work alone at their computer or in small groups. They are for flexible urban living. In the case of the package and the café, these build into our lives meanings and values regarding how we understand and relate to the world and other people in it. In this sense semiotic materials shape social organization and social interactions. They allow different things to be accomplished, and each shapes what can be done in any instance. A stone monument is good for showing something is deeply important. A webpage would not do the trick, nor would a café. But if you wanted to share information with people around the world, the stone monument would be less useful.

Discourse

In semiotics it is common to talk about the way that signs, or semiotic choices can connote particular ideas about the world, in other words, the different meanings they can transport. For example, a flag can connote ideas associated with nationalism and patriotism. A flag like the British Union Jack might connote unity and strength to some. A person might display it in their garden or on their T-shirt, to connote a nation that had a large, proud empire, which had a favourable outcome in two large-scale wars in the twentieth century. Of course, to others the same Union Jack might instead connote racism and intolerance as its use is often associated with far right groups or at least with conservative views of nationhood and retrospective imagined national pride rather than a more open-minded cosmopolitanism and internationalism. In either case, in the use of the flag as with the use of all national flags, there is a transport of meaning. Therefore, this material object is able to connote complex ideas about the nature of the world and the people in it – whether they are to be considered citizens under that flag, and exactly what associations that belonging brings with it.

Following Foucault (1977, 1980) we think of these complex ideas as ‘discourses’. Put simply, this means chunks of knowledge or ideas about the way the world works and the way people in that world behave. In media and communications studies, the term ‘discourse’ is often used for a confusing range of phenomena. For example, we might read about ‘television discourse’ or ‘news discourse’. It is hard to imagine what ‘discourse’ could have in common in these cases. Discourse is also sometimes used to mean talk itself or a piece of writing or text. But here we will use it to refer to socially constructed knowledge about the world. Union Jack is one semiotic material that is able to connote this glorious past. But more complex combinations of materials can be used to the same effect and to introduce more meanings, such as national monuments, buildings, advertisements for products which are made in that country. Hobsbawm and Ranger (1992) have described how national war monuments were constructed around Brita...