Foundations in Sound Design for Interactive Media

A Multidisciplinary Approach

Michael Filimowicz, Michael Filimowicz

• Publisher: Routledge
• Year: 2019
• ISBN:
  9781351603850
• Edition: 1
• Topic: Technology & Engineering
• Subtopic: Acoustical Engineering
• Index: Technology & Engineering

This volume provides a comprehensive introduction to foundational topics in sound design for interactive media, such as gaming and virtual reality; compositional techniques; new interfaces; sound spatialization; sonic cues and semiotics; performance and installations; music on the web; augmented reality applications; and sound producing software design.

The reader will gain a broad understanding of the key concepts and practices that define sound design for its use in computational media and design. The chapters are written by international authors from diverse backgrounds who provide multidisciplinary perspectives on sound in its interactive forms.

The volume is designed as a textbook for students and teachers, as a handbook for researchers in sound, design and media, and as a survey of key trends and ideas for practitioners interested in exploring the boundaries of their profession.

1

Cinematic Sound Design for Players

Tom A. Garner

1.1. Introduction

By definition, the term “cinematic” describes something possessing a quality that is “of the cinema.” Consequently, if the nature of cinema changes, so too should that which aims to evoke a cinematic aesthetic. Contemporary video games are most certainly a prime example of an artistic form with cinematic ambitions. There are numerous games that illustrate moments in which video games have appeared rather enamored with film, such as in the development of Fable (Molyneux et al. 2004). For this title, the studio enlisted Danny Elfman to produce the main theme, then passed the remainder of the scoring duties to Russell Shaw, instructing Shaw to directly imitate the cinematic template set by Elfman (Gibbons 2017).

While something of an obvious statement, it is important to observe that video games are not films. As we shall see throughout this chapter, the fact that films and games are substantially different creatures remains a persistent issue when attempting to adapt theory from the former to the latter. As Whalen (2007) observes, the primary distinction between film and video games is interactivity. This presents an immediate difficulty with regard to nonlinearity of games’ narrative structures. While a film follows a predetermined and singular timeline, a game must offer multiple-choice, freedom of exploration and the ability to stop, start and replay particular sections of game play at the player’s discretion. As a result, the creative process of game sound design cannot possibly utilize the same methods used for film. So how can game sound designers evoke a cinematic aesthetic with such dramatic practical differences?

1.2. Defining Cinematic Sound

To reach a convincing definition of cinematic sound it makes sense to employ a bit of divide and conquer. This section therefore begins with a very brief look at sound, specifically at a couple of issues with terminology. Next is a quick overview of some of the more generalized definitions of cinematic. Here the term’s alternative and developing meanings in film, television and video games are outlined. Finally we bring the two halves together to introduce the meaning of cinematic sound and consider its relevance to contemporary video games.

1.2.1. Defining “Sound”

As is often the case in the more creative disciplines, flexible terminology tends to be favored over that which is precise and consistent. When professionals discuss sound, they may be referring to speech, musical timbres, aesthetics or just music in general. They may also be describing the most stubbornly elusive of auditory elements, one so difficult to pin down it is often referred to as “non-music, non-speech sound.” Other possible monikers include “environmental sound,” “foley,” “ambient sound” and “sound effects.” For better clarity, this chapter brings together these various elements under “audio” as an umbrella term. The extensive range of auditory content types is acknowledged in section 2 where the intention is to provide a broad overview of how the notion of cinematic quality relates to the various aspects of sound design. While there is certainly much overlap between the various types of cinematic sound, this chapter does exclude speech. The main reason for this is that speech, while admittedly containing acoustic, timbral features, is commonly more associated with language than sound. This also frees up some space to explore musical and audio aspects of cinematic sound in greater depth.

1.2.2. What Do We Mean by “Cinematic”?

As you might expect, one of the broadest and most widely accepted meanings of “cinematic” is as a quality derived from cinema (Ebrahimian 2004). The term “quality” can of course encompass seemingly any aspect of an artifact. In film, this could extend to elements such as narrative, acting (even qualities of the actors themselves), dialogue, cinematography, environment/setting, visual effects and of course, musical scoring and sound design. Facilitated by rapidly increasing budgets, a great deal of contemporary television and video games have revealed their cinematic aspirations. A prominent example of the former would undoubtedly be the fantasy serial drama, Game of Thrones (Benioff and Weiss 2011–present). A review by Mills (2013) identifies several broader cinematic qualities relating to television that include: experimentation, greater use of special effects, cutting-edge computer-generated imagery, originality, richer audiovisual textures and more lingering scenes that encourage an audience to reflect and consider the meaning of the work. Referring back to Game of Thrones, Mills’ interpretation of cinematic television certainly rings true. From its almost photorealistic environment rendering to its pushing of boundaries, the series continues to epitomize the meaning of cinematic in television (Alexander 2017; Epstein 2016).

Within the video game industry, the term “cinematic” has been acknowledged to the extent that it has essentially become a genre (or at least, type of game) in its own right (Abdou 2015). Examples of “cinematic games” include Red Dead Redemption (Martin et al. 2010), Alan Wake (Ranki et al. 2010) and L.A. Noire (McNamara et al. 2011). Such games are often described in similar terms to Mills’ (2013) cinematic qualities of television. The opening scene of Alan Wake for example, is a lingering aerial shot, while L.A. Noire’s development experimented with Motion Scan technology as a means of rendering highly detailed character facial animations (see Chandler 2017). However, as Newman (2013) observes, “cinematic” has also become synonymous with “expensive,” and games development often plays a delicate balancing act between budget, player-expectation and cinematic quality. As the known budget increases, so does the player’s expectation. However, meeting such expectation by increasing the cinematic quality puts increased strain on the budget. In terms of musical scoring for instance, “[once] the consumer demand is high enough, developers of big-budget games will have to use live orchestras to remain competitive” (Newman 2013, p. 157).

In terms of video games’ history, the definition of cinematics has certainly changed over the years. Throughout the fifth generation of video games consoles (roughly 1993 to 2001, beginning with the lesser-known Fujitsu FM Towns Marty and including the much more well-known Sony PlayStation, Sega Saturn and Nintendo 64 consoles), cinematics typically referred to noninteractive “cutscenes.” These cutscenes would characteristically be scripted and linear, primarily serving as a means of churning out exposition and progressing the narrative. During the 1990s, games utilized cinematics for various functions. As Salen and Zimmerman (2004) state, such functions include: rhythm and pacing (a means of continuing the game’s narrative while also facilitating brief moments of relief from the game play), player reward, “game play catapult” (moving the player into a new situation quickly without breaking narrative continuity), scene/mood setting, and as a way of presenting the consequences of a player’s actions/choices. Making use of the fifth generation’s uptake of optical disks, cinematics were usually full motion videos (such as Command & Conquer [Sperry et al. 1995]) or pre-rendered computer-generated graphics (Final Fantasy VII [Kitase et al. 1997]). Push forward to the sixth console generation (Sony PlayStation 2, Microsoft Xbox and Nintendo Game Cube) and pioneering games such as Half-Life (Lombardi et al. 1998) began to respond to criticisms that were being raised against noninteractive cutscene cinematics. The primary problem of noninteractive cinematics was instigated by communities of more experienced players who did not require cutscenes in order to feel situated and comfortable within the game world and, instead, perceived such cinematics as both distractors and delayers of game play (Salen and Zimmerman 2004). The response to this complaint appeared in pioneering games such as Half-life. In this particular game, scripted dialogue and character animations provided the cinematic functions but player-control was retained and the cinematic itself was integrated into the diegetic and interactive game world.

As we shall observe later in this chapter, the above tension between interactivity and cinematic quality is one of the most significant issues for video game sound designers to consider.

1.2.3. Cinematic Sound in Film and Video Games

From the above we can observe that a cinematic aesthetic is synonymous with rich atmosphere, narrative exposition, experimentation, boundary pushing, high production costs and, overall, a finely crafted and controlled presentation. These points can be broadly carried over to inform cinematic sound design. For many games industry professionals, cinematic sound design typically refers to measures of fidelity. Such measures include high sample rates and resolution depths, implementation of a wide range of digital signal processing tools, and careful attention to postproduction. Cinematic video game sound is also expected to be realistic, vivid and dynamic in terms of spatialization, timbre and loudness (Newman 2013). For composer Bryce Jacobs (2017), cinematics relates heavily to the use of surround sound, echoing a commonplace assertion that cinematic sound is derived from the mechanical reproduction of sound within a theater that encircles the audience by way of multiple loudspeakers. Jacobs also refers to cinematics with regard to musical arrangement, asserting that the use of large ensembles (particularly those employing orchestral and/or choral elements) are akin to a cinematic score. Ultimately, cinematic game sound is generally expected to reflect the craft and financial investment of cinema (Bridgett 2007). We shall return to these points in greater detail in the next section of this chapter.

1.3. Approaches to a Cinematic Aesthetic in Video Game Sound

In this section, we will be exploring what are generally perceived to be the most current approaches to designing cinematic sound for video games. The discussion is structured into classifications of game sound as either creative (within which we look into musical scoring and implementation) or technical (audio processing effects). This is not to say that the development and use of audio processing is not creative, nor that scoring requires no technical skill. This distinction is only drawn as a way of helping compartmentalize the numerous aspects of game sound that must be considered when seeking to evoke a cinematic aesthetic.

1.3.1. Creative Approaches: Music Production and Implementation

1.3.1.1. Scores and Soundtracks

The compositional aspect of a musical score encompasses what we might call “classical” aspects of music production, including melodic/harmonic construction, texture, structure, rhythm, timbre, orchestration and dynamics. Precisely how these numerous elements can be utilized to evoke a cinematic aesthetic arguably encompasses a vast collection of techniques that are further compounded by matters of method and genre. As a result, a full review of contemporary compositional techniques and their relevance to cinematic game music is beyond the scope of this chapter. That said, several approaches appear more prominent than others as effective routes to cinematic sound.

A useful overview of the technical requirements of a cinematic score is presented by Davis (2010). These requirements include physical, structural and psychological functions. Physical function refers to the use of a score to reinforce or punctuate the physical environments and actions displayed on the screen. This could include the use of regional instrumentation to signify a particular place, or melodic conventions that typify a particular musical era as a means of establishing the time period in which the narrative is set. Structural functions, by contrast, relate primarily to issues of continuity and overarching identity. For Davis, cinematic scoring must support the fluidity and consistency of transitions between shots, enabling the visuals to jump to different angles and focus upon different subjects with an auditory continuation that gives the audience the experience of a single, unified scene. Regarding structural identity, Davis asserts that a cinematic score will often utilize leitmotifs (brief musical themes representative of a specific character, place or action) to add sensory weight to particular aspects of the narrative. Through attaching such leitmotifs to a character for example, a film can reintroduce the idea of that character at a later point in time without actually having to make the character appear on screen, or it can vary the tone of the music to apply an emotional overlay to our concept of the character. Film composer Michael Giacchino (Patches 2013) gives an example of this practice in describing his score for the opening scene in Up (Docter 2009). Giacchino describes his use of a brief, constantly repeating melodic theme intended to embed an idea in the audience’s minds, then retaining that melody while slowing the tempo and shifting the orchestration from a larger ensemble to a solo piano. The effect of this technique is a poignant tonal shift that applies a saddening event to a character we feel we know, despite them only being on screen very briefly and without uttering a word of dialogue. Giacchino’s process is a good illustration of a cinematic score that exploits Davis’ (2010) structural requirement (the use of a repeated motif) but also satisfies an emotional function (sadness at a character’s passing away). This technique is also brilliantly exploited in the horror-shooter Left 4 Dead (Booth et al. 2008), in which each major enemy type is accompanied by a unique leitmotif whenever it spawns in the map. This gives the player an indication of that enemy’s presence well in advance of them appearing on the screen as a particularly effective means of raising tension.

Psychological function, Davis’ (2010) third requirement of a cinematic score, is arguably the most important requirement for a cinematic aesthetic. It describes the power of music to be emotionally evocative, which is of course also a fundamental function of cinema in general. As Davis explains, the score is required to complement the emotional direction of the visuals, but not necessarily to directly reinforce them. For instance, a film’s narrative may be particularly dark and serious in tone to the extent that an equally solemn score would be too intense or aggressively one-dimensional. In such an instance, a good cinematic score would seek to counterbalance the tone (adding nuance and tonal layering) without undermining it. This technique may also be extended to enhance the narrative’s emotional color by encouraging an ironic interpretation from the music’s tone. Video games already display a keen awareness of this cinematic approach in games such as the narrative-driven science fiction shooter Bioshock (Levine et al. 2007), in which the relentlessly cheery 1930s soundtrack serves only to highlight the many blood-stained corpses that litter the sunken city of Rapture.

Sticking with Bioshock and the use of a soundtrack as opposed to musical score, Gibbons (2011) observes that a carefully constructed set of popular songs can be subtly woven into both the narrative and the actions of the player. As well as creating a jarring dichotomy by presenting fundamentally optimistic and upbeat songs within an otherwise bleak and decaying world, Bioshock’s soundtrack fulfills two further functions. Firstly, it effectively establishes a sense of the narrative’s time period. Secondly, several of the songs are used to subtly raise key narrative points that contextualize some of the player’s actions¹. As is something of a r...