The Multisensory Film Experience

A Cognitive Model of Experiental Film Aesthetics

Luis Rocha Antunes

• Publisher: Intellect Books
• Year: 2016
• Print ISBN: 9781783206285
• eBook ISBN: 9781783206308
• Edition: 1
• Topic: Art
• Subtopic: Art General

When the lights dim in a movie theatre and the projector begins to click and whir, the light and sounds of the motion picture become the gateway to a multisensory experience. Moving beyond the oft-discussed perceptual elements of vision and hearing, The Multisensory Film Experience analyses temperature, pain and balance in order to argue that it is the experience of film that's inherently multisensory, not the medium. Luis Rocha Antunes here explores the work of well-loved filmmakers Erik Jensen, Gus Van Sant and Ki-Duk Kim to offer new insights into how viewers experience films and understand their stories. This is an original contribution to an emerging field of research and will become essential reading for film scholars.

Chapter 1

The Multisensory Film Experience, Experiential Film Aesthetics, Cinema and the Senses

However much the spectator may be engaged by plot or genre, subject matter or thematic implication, the texture of the film experience depends centrally upon the moving images and the sound that accompanies them. The audience gains access to story or theme only through that tissue of the sensory materials. […] However unaware spectators may be of it, style is working at every moment to shape their experience.

(Bordwell, 1997: 7−8)

Sensations of smell, equilibrium, or touch are, of course, never conveyed in a film through direct stimuli, but are suggested indirectly through sight. […] [The filmmaker] eliminates entire areas of sensory perception, and thereby brings others into higher relief, ingeniously making them take the place of those that are missing.

(Arnheim, 1933: 34)

Let us assume that, unlike the other types of pictures, film images affect primarily the spectator’s senses, engaging him physiologically before he is in a position to respond intellectually.

(Kracauer, 1960: 158)

Although this statement may be an audacious way to open this book, I am convinced that there is no such thing as a purely visual—or purely audiovisual—experience of film. What we see, or what we call seeing, is multisensory in its nature and remains multisensory, even in the final stage of what we consider consciousness: that supposed moment when perception lightens and the marvel of awareness comes about. The perceptions resulting from much of the visual and auditory information in the external sensory world are multisensory. They are not visual, auditory or audiovisual, but multisensory.

When this idea first began to germinate in my mind, I had smaller ambitions and aimed only to show that it is possible to have a multisensory experience through an audiovisual medium such as film and that our brains can perceive an audiovisual medium in a multisensory manner. As my research advanced, though, I realized that not only can our brains perceive an audiovisual medium in a multisensory way, but they must do so because there is no other way for perception to occur. Our natural, not exceptional or synaesthetic, way of perceiving is multisensory. This form of perception remains multisensory when we experience a film, whatever the configuration of the apparatus used to watch the film: a dark theatre room, a cosy living room with a television set or even a small tablet. The natural way for the brain to operate is multisensory, and even if we desired purely visual experiences, we would be frustrated and incapable to force our brains to block out and inhibit some of the neural connections between the senses.

The idea of multisensoriality in film is often based on the intellectual and phenomenal capacity of spectators to make associations creatively through imagination and memory (Marks, 2000, 2002; Sobchack, 1992, 2004). I do not wish to refute such a capacity and possibility, but rather wish to show a level of multisensoriality that is perceptual—not intellectual, imagined or remembered—in principle. This multisensoriality presents itself even before our consciousness has the capacity to make any sense or intellectual and phenomenal associations in a synaesthetic manner. To reiterate Siegfried Kracauer’s words cited at the beginning of this chapter, a film engages the spectator “[…] physiologically before he is in a position to respond intellectually” (Kracauer, 1997: 158). In point of fact, a multisensory experience is the natural and common way for all of us to perceptually experience film instead of a mere synaesthetic, exceptional capacity to make high-order, intellectual and phenomenal associations among ideas of a multisensory nature.

My analysis of this issue is a matter of perception, not phenomenology. The multisensory film experience derives from hard-wired, low-level mechanisms of perception that are not within our conscious control. These take place in the milliseconds of the time window of sensory integration, that is, when the senses are integrated. We can divide the levels of our perceptual control of a film through the concepts of autonomic and somatic responses, where somatic response refers to a process that does not require conscious and effortful processing but can nevertheless be inhibited or controlled and autonomic response refers to a process on which we cannot exert control. This layout of a multisensory film experience contradicts the idea of a voyeur spectator who leans back comfortably and visually watches and enjoys in a detached way and in control of his private audiovisual cinematic experience.

Because the primary goal of this book is to support the idea that there are no visual, auditory or audiovisual experiences of film—only a multisensory experience—I am much more concerned with the autonomic level of our film perception than with the somatic level. Furthermore, I aim to add an examination of film aesthetics based on multisensoriality to this perceptual investigation of the senses. The resulting corollary of this combination of perception and aesthetics is materialized by what I specifically call the experiential film aesthetics of Gus Van Sant, Ki-Duk Kim and Knut Erik Jensen (see also Antunes, 2015a, 2015b, 2015c). My analysis of the work of these three directors within a multisensory frame of reference is tantamount to stylistic, narrative and emotional elements that together shape the concrete realm of the multisensory film experience using thermoception (perception of temperature), nociception (perception of pain) and the vestibular sense (perception of orientation and balance) as sense modalities not captured by the discussion of the classic five senses or of the traditional senses associated with film perception and film phenomenology.

The multisensory film experience is the conceptual formulation that evolved from my idea—and the apparent contradiction—that film is an audiovisual medium that is perceptually experienced by spectators in a multisensory fashion. This conceptual distinction between a medium source of stimuli and the resulting perceptual experience is paramount to my claims and aims to shift the burden of proof from having to show that it is possible to perceive an audiovisual medium in a multisensory manner to having to corroborate an orthodox view of film as an audiovisual medium and experience. My proposed shift of the burden of proof invites sceptics to investigate the notion that when we experience a film, we can consciously or unconsciously have exclusively visual or, at most, audiovisual experiences of film, not multisensory experiences, without the influence of the other senses.

In my model of the multisensory film experience, seeing, or what we call seeing, is not only a visual experience but also an experience that involves all of our senses, including hearing in a discriminate manner, specifically through the most relevant sensory modalities explored by certain experiential aesthetics, such as those of the three directors analysed in this book. Considering film a visual experience is not a perspective of specific film scholars but a truly common perspective of nearly all of them; thus, my arguments widely engage with a variety of studies both within and outside the film literature, without a critical focus on particular film scholarship.

In the multisensory film experience, I assume different degrees of sensitivity and individual and subjective perceptual contents of spectators’ experiences. Although the exact perceptual contents change from one individual to another and from one sensory context to another, the nature and general principles of our multisensory perception remain common to us all. In other words, the principles of multisensory processing and integration, or the idea that the brain works in a multisensory manner, are universal principles, but the exact perceptual contents of our experiences certainly vary by individual and sensory context. Thus, the multisensory film experience does not deal with the perceptual contents but with the perceptual principles of film perception.

The idea of experiential film aesthetics, as I define it, implies that film is not only the result of formal and compositional elements of style, narrative and themes but also the result of the intersection of those elements (especially film style created through camerawork, editing, light, colour and sound design) and our perceptual, multisensory nature as spectators. Our perceptual experience of a film is thus not a mechanistic method of receiving, processing and integrating a film’s sensory information, but rather an active, dynamic set of perceptual processes that are proactive and creative (although its autonomic levels are not within the reach of our conscious control). This is a constructivist frame of reference very much in line with many conceptions of cognitive film theory, however, with specific stress on the role of the senses, rather than on the more explored roles of emotion and empathy. The bottom line, however, is widely recognized, namely, that aesthetics and perception are not separate, but rather common and dialogical layers.

For instance, colour and lighting can have different expressive, formal roles, but in the realm of Jensen’s films, they are not merely sensory features of the visual domain; they also have the potential to evoke, provoke and elicit emotional, perceptual and thematic experiences associated with thermoception. Similarly, the same applies to Kim’s articulation of sound with nociception and Van Sant’s articulation of camerawork and editing with the vestibular sense. Colour and lighting, along with all other stylistic elements, can be connected to specific senses in ways that can be understood only in relation to the authorial use of each specific director. Experiential film aesthetics result, then, from a dialogical relationship between style, narrative, themes, the characters’ bodies and our perceptual experience because the elements of the style cue certain perceptual experiences and because the nature of our human perception simultaneously has the ability to relate the sensory information of each specific sense modality. A matrix of dialogical relationships finds different configurations across each of the sensory modalities that I examine in this book.

The films I have selected from the oeuvres of Van Sant, Kim and Jensen are based primarily on non-verbal communication. This criterion facilitates a focus on the experiential nature of their films, but it does not presuppose that verbal language is not part of our multisensory human perception. Verbal language may not be a sensory modality on its own, but it is a cognitive function with a perceptual basis and a strong capacity to generate multisensory imagery. Verbal language covers many aspects of human perception and cognition and also deserves an in-depth examination in future studies of the multisensory film experience. However, here, I focus on specific levels of sensory perception, and examining non-verbal cinema simplifies a methodological focus on thermoception, nociception and the vestibular sense. Otherwise, we could be led to believe that the verbal language of these films was the element prompting an associative experience of the senses. For example, in a scene from Darren Aronofsky’s The Fountain (2006), I failed to notice that the character Isabel/Izzi Creo (Rachel Weisz) sat shoeless on her rooftop while it snowed until Tomas/Tommy/Tom Creo (Hugh Jackman) joined her and made verbal mention of this fact. This is a case in which it is fair to say verbal language can play a significant role in inducing our perceptual experiences, as opposed to the almost-mute films of Van Sant, Kim and Jensen.

In addition to the minimal use of verbal language, these films display minimal narratives. There are still traces of narrative and storylines, but they are minimal or fragmented, and they are often conducted not through dialogue and verbal communication but rather through the characters’ actions, the film style and information from which we can make perceptual inferences. They are offered to spectators in forms that connect to the ineffable or to the idea that many of the contents of film—for example, thematic segments (Antunes, 2013), story events, character motivations and even some moral alignment between spectators and characters—belong to a non-verbal realm. Instead of being told what happens and hearing dialogue about the motivations behind the actions and events in the film’s story, these films resort to a level of experiential aesthetics.

Additionally, these films venture one step further from the motto Show, don’t tell (Lubbock, 1957) to, I would risk saying, Don’t just show, deliver to the senses. They make spectators active perceptual-participants, not mere observers of an unfolding narrative experience. In this sense, these films’ minimal narrative nature is still based on the serendipity principles of any other film narrative, but their perceptual nature is based on the active interaction with a participant’s perceptual experience and is constructivist, given the creative ways in which our perception can be engaged. Downplaying verbal language and narrative helps us to see the role and importance of the pure sensory experience, but verbal language is not, I stress, incompatible with any variable of the multisensory film experience.

The films of Van Sant, Kim and Jensen also have the advantage of reflecting cinematic approaches across different cultures and geographies in a sample of world cinema that includes films from North America (Van Sant), East Asia (Kim) and Northern Europe/the Arctic (Jensen). I do not intend to delve into universalist claims that experiential film aesthetics arises regardless of spectators’ cultural background or that the senses are universal. Certainly, there are culturally and individually shaped differences in perception. Instead, I note only that my claim of a multisensory film experience is not contingent upon a specific geography or culture but can be found not only across a range of aesthetics and thematic sensibilities but also across different cinematic and geographic cultures. I do, however, assume that despite these films’ textures and cultural nuances, they appeal to a wide range of audiences outside of their respective home cultures, as corroborated by their greater success overseas than that in their home countries. By using non-verbal communication and the senses, these films capture the interest of various audiences. The experiential appeal of these films is universal, but spectators’ actual perceptual experiences will certainly vary according to cultural specificities, individuals and sensory contexts (e.g., a small television set in living room versus a large, dark theatre with surround sound). Their films represent a universally appealing language of the senses with room for individual and subjective experiences.

Although the multisensory film experience is, in the aforementioned sense, a universal form of appeal offered by film, its cultural and geographic influences vary. It is given that any film can and will cue forms of experiential engagement across a number of senses: however, the following questions remain: (1) how? and (2) taking advantage mainly of which sensory modalities? Experiential immersion is present in all films with different degrees of intensity and quality. For instance, an extreme example opposed to the vestibular sense could be My Dinner With André (Louis Malle, 1981),¹ in which two men sit at a restaurant table, dining and talking throughout the film. In that case, there is not much room for salient responses to experiential immersion on the level of orientation and balance. The experience takes place more on the level of verbal language than anything else. That, however, does not mean that we do not have some spatial awareness of the characters and the room in which they are located: we need at least some sort of orientation. Even a film with such an extreme lack of movement constructs a spatial awareness through its editing, shot-scale, camera positions and even sound design. However, My Dinner With André is in opposition to, for instance, the roller coaster ride of the three-screen projection of This is Cinerama! (Various directors, 1952), where a visceral experience based on the vestibular sense is basically what is at stake and where words are replaced by screams, body contortion, mugging, vertigo and, if one has the full experience, some motion sickness.

Although all films have a foundation of experientiality (they are a form of sensory experience), some are more closely based on non-verbal forms of communication that more directly call for an immersive experience through the senses. This use of experiential film aesthetics within the realm of non-verbal communication seems to represent a strong tendency in contemporary filmmaking. In addition to the directors discussed in this book, directors as diverse as Andrea Arnold (Antunes, 2015d), Terrence Malick (Antunes, 2014), Ang Lee, Jim Jarmusch, Sofia Coppola, Jane Campion, David Cronenberg, Jacques Audiard, Jean-Pierre and Luc Dardenne, Nuri Ceylan, Steve McQueen, Kar-Wai Wong, Andrew Dominik, Ming-liang Tsai, Abdellatif Kechiche, Cristian Mungiu and Albert Serra are among those who I consider most representative of world cinema’s contemporary tendency towards experiential film aesthetics. Their cinematic projects are exceedingly challenging to our notions of film genre and even film narrative because although these films offer stories, they do so without the verbal communication to which we have become accustomed from the more common forms of film narrative based on verbal interaction, such as the screwball comedy or the film noir, as the most obvious examples.

This contemporary type of filmmaking across various geographical and cultural contexts and authorial voices, giving primacy to the experiential aesthetics of film, defies the general presupposition that Eastern arts have a more experiential nature that is opposed to the more narrative forms of the West:

[…] the Japanese aesthetic tradition opens possible areas for inquiry not often explored in modern Western aesthetics. They include the relationship between the aesthetic and its moral and spiritual considerations in the sense explained above, creative activity that is object/material-centered, and the appreciation of multisensory experiences and those qualities that are typically depreciated.

(Saito, 2009: 386)

In my view, the contemporary cinema of the senses and this contemporary search for experiential film aesthetics is not a product of Asian cinema, but represents a generalized tendency across European and North American cinema as much as in Asian cinema.

The broader idea of experiential aesthetics (applied not to film, but to the arts in general) has been used to describe art as an experience outside objectivism. According to experiential aesthetics, compositional aspects of an object of art matter only insomuch as they relate to how humans perceive and construct them; they are not immutable objective stylist features. Experiential aesthetics are then born from this dialogical relationship between the subjective and the objective, the style and perception. However, it has only been applied to film to a very limited extent because there exists what I think of as an orthodox view, which assumes that film is a visual (and at most, an audiovisual) experience, and its aesthetics are therefore audiovisual, as well. In this manner, it has been difficult to conceive how film can be experiential in the same way as an interactive art installation in a museum, which directly convokes haptic contact between the perceiver and the work of art.

This orthodox view accepts that our perceptual experience of film is experiential insofar as it is an audiovisual experience because the direct sensory link with the other sensory modalities is absent in the form of sensory stimuli. This orthodox view considers that this sensory link of a haptic nature is lacking in film. Film in its audiovisual form has not, then, been considered an experiential medium with the same immersive and interactive potential but instead one in which spectators do not actually participate perceptually in its full properties because some sensory modalities are supposedly shut off. Although the expression “experiential aesthetics” seems to have come to use only a few years later (by Beiswanger, 1937),² the foundations ...

Table of contents

• Cover Page
• Title Page
• Copyright
• Contents
• Foreword by Michael Grabowski
• Acknowledgements
• Chapter 1: The Multisensory Film Experience, Experiential Film Aesthetics, Cinema and the Senses
• Chapter 2: The Vestibular in Film: Orientation and Balance in Gus Van Sant’s Cinema of Walking
• Chapter 3: Nociception in Film: A Cinematic Account of Ki-Duk Kim’s Aesthetics of Pain
• Chapter 4: Thermoception in Film: Knut Erik Jensen’s Experiential Aesthetics of Cold
• Chapter 5: Conclusion
• Bibliography
• Filmography
• Back Cover