Multimodality and Aesthetics

Elise Seip Tønnessen, Frida Forsgren, Elise Seip Tønnessen, Frida Forsgren

• Publisher: Routledge
• Year: 2018
• Print ISBN: 9781138103511
• eBook ISBN: 9781351592758
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

This volume explores the relationship between aesthetics and traditional multimodal communication to show how all semiotic resources, not just those situated within fine arts, have an aesthetic function. Bringing together contributions from an interdisciplinary group of researchers, the book meditates on the role of aesthetics in a broader range of semiotic resources, including urban spaces, blogs, digital scrapbooks, children's literature, music, and online learning environments. The result is a comprehensive collection of new perspectives on how communication and aesthetics enrich and complement one another when meaning is made with semiotic resources, making this key reading for students and scholars in multimodality, fine arts, education studies, and visual culture.

1 Introduction

Elise Seip Tønnessen and Frida Forsgren

One Example—Two Approaches

Let us start by looking at a visual example: Per Kleiva’s American Butterflies (Amerikanske sommerfuglar), serigraph on paper, from 1971. The picture consists of a dark blue background. The lower section of the picture is bordered by a tight row of red and yellow flames. The centre of the picture is occupied by a file of military helicopters; they increase in size as they approach the viewer. The helicopters are skeletal and transparent; two of them have details that resemble eyes and a nose, giving them perhaps a threatening image. The helicopter rotors are mounted on butterflies in shades of purple, pink, green, dark red and blue, giving them a lightweight, fluttering feel. The juxtaposition of skeletal/threatening/military with fluttering/lightweight/butterfly creates the symbolic tension in the picture. The military helicopters are likened to swarms of butterflies that instead of warmth and summer bring flames and war.

The picture is a serigraph, a method where you create an image on paper (or fabric) by passing ink through a screen with areas blocked off by a stencil. This creates a graphic result seen in the stylized appearance of flames and helicopter/butterflies. American pop artist Andy Warhol is generally credited with popularizing serigraphy as an artistic technique. And Per Kleiva (1933–2017), who made American Butterflies, was one of the earliest and most important pop artists in Norway to use this technique. Kleiva was active in an artistic collective called GRAS, a group that took pop culture in an explicitly political direction, as opposed to Americans like Warhol and Roy Lichtenstein (Brandtzæg, 2016). The radically politically committed GRAS group wanted to communicate art to an ordinary public and used serigraphy as a cheap production method that enabled the easy spread of their works. Kleiva’s most famous pictures are the serigraphs in Pages From the Diary of Imperialism 1971 where the Vietnam War is criticized. His serigraph American Butterflies stands as a striking example of the political aesthetical activism of Norwegian artists in the 1970s. And his elegant composition of butterflies, military helicopters, and flames juxtaposed with the dark, looming backdrop constitutes an aesthetic, visual narrative. This interpretation is a standard art-historical analysis of a visual image. It follows the three-step method of description, interpretation, and observation of the wider historical/political/social context that Erwin Panofsky established in his Studies in Iconology (Panofsky, 1939). Panofsky’s concern was to locate function in the artwork’s historical context, rather than in style and aesthetic taste.

Figure 1.1 American Butterflies by Per Kleiva (1971). (See page 55 for the colour image.)

But what if a social semiotician should analyse the same image—would the analysis be very different? In regard to terms used, it would. The artwork might be called a text; the serigraph technique would be defined as the medium of expression, and the full repertoire of aesthetic compositional features would be called semiotic resources and carefully mapped to show how they make meaning: what is represented, what kind of relations are established, and how this text is assembled as a coherent whole. The artwork would be interpreted in connection with its modal affordances, what is possible to express and represent within the modes in play in the artistic visual image. And chiefly, the semiotician would observe this process of meaning-making as a social practice. He would most probably reach the same conclusions as the art historian, that this is an object made to communicate opposition to the Vietnam War. The main difference would be in how is American Butterflies experienced? The social semiotician might stress how Kleiva’s work is part of a social practice, in this case of a specific genre of arts, where the ability to access, interpret, and appreciate the artwork is socially distributed. To the art historian it would be imperative to stress how Kleiva’s work is first and foremost a unique experience made by a unique individual: the artist. To the art historian art can never be seen as merely a particular instantiation of a semiotic system, but as a unique object carrying a deeper kind of meaning experienced on a sensory, transcendental, bodily and fundamentally existential level. To the art historian, the artwork is what transports you out of the everyday routine, while the social semiotic sees semiotic resources as situated in networks of signifying practices shaped by their functions in society.

The distinction between fine arts and everyday communication has changed through history. The development of the fine art institution in the nineteenth century and Modernism’s establishment of the idea of l’art pour l’art contributed to the construction of what we may see as an artificial gap between art (aesthetics) and society (SNL, 2018). Traditionally aesthetics was an integrated, inextricable part of everyday life, whereas today, (fine) art is something one is educated to appreciate and something to be experienced at the art museum, in the concert hall, or the theatre. Yet, all human communication is fundamentally aesthetic. This volume is a probing into the aesthetics of multimodal communication from ‘both sides’ as it were—with a broad approach to aesthetics and meaning-making we wish to bridge the two positions outlined earlier. The core question of the present book is how communication is aesthetical and how aesthetics communicates. The book investigates aesthetics and multimodality from a broad perspective and across several domains: education, fine and popular art, and digital media.

Previously we have emphasized the differences—but we also see signs that the two approaches are coming closer together. In 1993, Norman Fairclough commented on how the ‘marketization’ of public discourse had consequences that included amalgamation of the private or personal and the public. Theo van Leeuwen (2015) reflects on this development as a forerunner in developing a new discourse aesthetics. He claims that time is running out for the division we have seen where on the one hand critical discourse analysis focuses on political discourse and on the other hand aesthetics is seen as more appropriate to the study of poetry:

Today, this view can no longer be maintained. As a result of the ubiquity of branding, aesthetics has made inroads into many important forms of public text and talk. Public discourse, today, must do more than inform, persuade, instruct, explain, it must also ‘look good’.

(van Leeuwen 2015, p. 191)

Multimodality and Aesthetics: Predecessors and New Approaches

The concept of multimodality has only been around for a few decades; the phenomenon of multimodal communication, however, is as old as mankind. This means that research into the field of multimodality can be found in various scholarly traditions. One tradition belongs within the arts, in analyses of visual arts, music, theatre, literature etc. Through the development of institutions of fine arts, these approaches have become more and more specialized as we mentioned earlier, refining the insights about one mode at a time. But a scholar such as Michael Baxandall refocused attention on the plurality of modes at stake in the process of interpretation. Baxandall (1988) examines how artists and their works functioned in their original social, commercial, and religious context. And his concept the period eye draws attention to the mental and visual equipment brought to bear on works of art in a particular place and time.

Another tradition is within semiotics. Here we also find approaches that focus on specific modes, such as “Rhetoric of the image” (Barthes, 1977 [1964]), “reading film music” (Brown, 1994) or The language of music (Cooke, 1959). These approaches tend to take a bottom-up perspective, emphasizing how specific modes contribute to aesthetic experience and meaning-making. In popular culture, on the other hand, the modes have been brought back together, as can be seen in film studies and in new digital media, and demanding a more multimodal approach. The unique contribution of the multimodal approach, as it emerged some 20 years ago in the social semiotic tradition, is how it includes the full multimodal orchestration. Hence it pays attention to the interplay between modes as well as to the total multimodal ensemble from a top-down perspective.

Even before scholars inspired by M.A.K. Halliday’s social semiotics introduced the concept of multimodality (Kress & van Leeuwen, 1996, 2001; van Leeuwen 2005; Kress, 2010; Jewitt, 2014), semiotics was introduced as a common approach to meaning-making across modes (as was proposed, but not put into practice by Ferdinand de Saussure). A particularly relevant example is the French semiotician Roland Barthes, who developed from a strictly structuralist semiotics in his early work, to a more context-sensitive approach in his later work. His combination of systemic and functional interest has inspired later approaches to multimodality. Theo van Leeuwen, who read Barthes as part of his film studies, honours his legacy in an interview:

I continue to admire Roland Barthes. He was trying to do things in a structuralist way but he was a critic as well, and always aware of the social and cultural context. Never ‘specializing’ in the semiotics of this or that, there is not a single mode, as we would now say, he has not tackled.

(van Leeuwen in Andersen, Boeriis, Maagerø, & Tønnessen, 2015)

Theo van Leeuwen (2015) also points to the Prague school as a legacy for scholars who are interested in developing multimodal discourse aesthetics. The Linguistic Circle in Prague in the interwar period combined linguistics with stylistics, poetics and aesthetics. Van Leeuwen reminds us that this group was an inspiration for M.A.K. Halliday’s work in social semiotics, and suggests their views as a possible source for an expanded perspective:

Building on the linguistics of Halliday, the study of multimodality has developed detailed accounts of the communicative functioning of images, music and other semiotic modes. But Halliday’s functional linguistics, though inspired by the Prague School, does not recognize the aesthetic function. Perhaps multimodal studies can build on the work of the Prague School and the linguistic stylisticians to develop methods for the analysis of multimodal discourse aesthetics.

(van Leeuwen, 2015, p. 196)

Jan Mukařovský (1970), one of the members of the Prague school, proposed to add the aesthetic function to the representative, expressive, and appellative ones already established by Karl Bühler. Van Leeuwen (2015) highlights Mukařovský’s concept of ‘de-automization’, which is a characteristic of poetic language, as particularly useful for understanding the aesthetization of modern communication. He claims that today form and function exist side by side, “each realizing a specific aspect of what has to be communicated” (2015, p. 198).

Meaning-Making Through Sensing, Feeling, and Thinking

As we have seen, opening up the field of multimodality leads to encounters with a great variety of texts experienced through various senses. In later years, scholarly attention to multimodality has been brought to bear on all aspects of communication, from textbooks to film and architecture, and on various functions from didactics to games and entertainment, to mention just a few areas. This is the result of cultural changes which call for new perspectives on how texts invite meaning-making, and consequently on what makes texts meaningful. In social semiotics the functional perspective on how we make meaning from texts in various modes has led to an emphasis on representation, ideas and social relations, rather than personal, emotional and sensory experience. The concept of meaning is mostly taken in a straightforward matter-of-fact way when social semioticians explore processes of meaning-making. However, the concept of meaning raises complex philosophical questions as to the ‘whats’ and ‘hows’ and ‘whos’ of meaning-making.

In an interview Jay Lemke questions the sharp division between affect and cognition and states that “aesthetics is fundamental to all aspects of meaning” (Lemke in Andersen et al., 2015, p. 132). Observing young players of computer games, he was struck by what he has called the “meaning-feeling cycle” (p. 134): “The choices that you make in the game are not based purely on the meanings that those choices have; they also are based on the feeling state that you are in at the time you make them”. From his background in the natural sciences he has been inspired by ideas from biosemiotics and embodied cognition (Thibault, 2011). In an attempt to figure out how meaning-making processes, including feeling processes, take place in material systems, he suggests:

Why can’t we do for our perspective on feelings and emotions what we’ve already done for our perspective on cognition? That is, to go outside of the head, to make it culturally specific, to make it situated in interactions in an environment in a context, and, in effect, to turn it into a social semiotics so that it becomes a social semiotics of feeling.

(Lemke in Andersen et al., 2015, p. 131)

Hence, Lemke calls for an inclusion of the aesthetic, emotional, and affective dimension as an important challenge for the future of multimodal analysis (p. 133). Seen through the lenses of fine arts and phenomenology the aesthetical experience communicates with and through our emotions, providing us with a sense that experiencing the aesthetical object transcends pragmatic communication and feels meaningful, beautiful, or fulfilling in its own right. The aesthetic field describes how our bodies and senses react to sensory stimuli, and how sensing, feeling, and interpreting are connected in the aesthetical experience.

Perspectives on Multimodality and Aesthetics in This Volume

In this volume we have included contributions that find inspiration from different traditions, but they all share an interest in multimodal communication. We are interested in discussing how all semiotic resources may have aesthetic functions. This volume brings together contributions by researchers to show how aesthetics plays a key role in traditional multimodal communication within arts, literature, and education, but importantly also, how it plays a role in the everyday aesthetics of our built environment, school books, social media, digital scrapbooks, and digital templates. The contributors have all been challenged to explicate their understanding of aesthetics within their field of research, and how this may relate to multimodal communication. Consequently, this volume does not offer one unified view of aesthetics and multimodality. Rather, we want to explore and discuss what a variety of views may contribute.

Roughly speaking we find three perspectives across the contributions in this volume. Since they are not mutually exclusive, various combinations can also be found. Firstly, we find an interest in everyday aesthetics, in what makes mundane expressions and practices stand out and catch our attention. One area where this perspective is prominent is within the analysis of digital discourse, where templates and filters invite aesthetic considerations on everyday communicational practices.

Secondly, we find several articles that refer to John Dewey’s (1934) work on aesthetic experience. Experience as Dewey understood it comes as a result of an interaction between the self and elements of the world. His views on aesthetic experience encompass individual as well as shared emotions and ideas. His focus is on action rather than on the artistic artefact, in line with the modern performative philosophy of art. Dewey (1950) argued for a broad view of aesthetics, as opposed to a sharp division between artistic and mundane communication. He sees aesthetics as part of any experience and outlines a continuous development from this primary phase of aesthetic experience to a more developed and cultivated experience of fine arts. Focusing on aesthetics as experience rather than as essential characteristics of objects of art directs the analytical interest towards the sensing subject. It also opens up to see aesthetic experience as something that can be found in every aspect of daily life.

Thirdly, we find perspectives inspired by Gunther Kress’s (2010, p. 28) statement that aesthetics is “the politics of style”, where style is seen as the “politics of choice”. This perspec...

Table of contents

• Cover
• Title
• Copyright
• Contents
• Preface
• List of Figures
• List of Tables
• List of Color Plates
• 1 Introduction
• Part I
• 2 Multimodality, Style, and the Aesthetic: The Case of the Digital Werewolf
• 3 A Phenomenological Approach to Multimodality and Aesthetic Experiences
• Part II
• 4 Memoria of a National Trauma
• 5 Reconstruction of Chilean Memories in the National Stadium of Chile: A Multimodal Critical Discourse Analysis of a Centre of Detention and Torture
• 6 Shaping the Social Through the Aesthetics of Public Places: The Renovation of Leeds Kirkgate Market
• Part III
• 7 A Multimodal Analysis of Aesthetics in Brazilian School Textbooks
• 8 Aesthetic Experience Through Students’ Production of Digital Books
• 9 Digital Argumentation Aesthetics
• 10 Visualizing Norms of Science in Early School Years: Visual Aesthetics and Content Formation in Students’ Multimodal Compositions
• 11 The Aesthetic Potential of Vocal Sound in Online Learning Situations
• Part IV
• 12 The Templatized Aesthetics of Wix: A Social Semiotic Technology Approach to Web Design
• 13 Digital Scrapbooks, Everyday Aesthetics, and the Curatorial Self: Social Photography in Female Visual Blogging
• 14 Multimodality, Moving Images, and Aesthetics
• 15 Filtered Aesthetics: A Study of Instagram’s Photo Filters From the Perspective of Semiotic Technology
• Part V
• 16 Sensory Experience and a Subjective Reading Position in The Lost Thing
• 17 Tears in Heaven: Eric Clapton Coping with Loss Through Music and Words
• 18 Multimodal Aesthetics and Gender in Beck’s Song Reader
• 19 Intermodal Contrast in Film: Looking for the Aesthetics of Intermodal Relations
• List of Contributors
• Index