Materializing Digital Futures

Touch, Movement, Sound and Vision

Toija Cinque, Jordan Beth Vincent, Toija Cinque, Jordan Beth Vincent

• Publisher: Bloomsbury Academic
• Year: 2022
• Print ISBN:
  9781501388088
• Edition: 1
• eBook ISBN:
  9781501361265
• Topic: Social Sciences
• Subtopic: Media Studies
• Index: Social Sciences

Digital, visual media are found in most aspects of everyday life, from workplaces to household devices - computer and digital television screens, appliances such as refrigerators and home assistants, and applications for social media and gaming. Each technologically enabled opportunity brings an increasingly sophisticated language with the act of pursuing the intrasensorial ways of perceiving the world around us - through touch, movement, sound and vision - that is the heart of screen media use and audience engagement with digital artifacts. Drawing on digital media's currently evolving transformation and transforming capacity this book builds a story of the multiple processes in robotics and AI, virtual reality, creative image and sound production, the representation of data and creative practice. Issues around commodification, identity, identification, and political economy are critically examined for the emerging and affecting encounters and perceptions that are brought to bear.

SECTION ONE Socio-aesthetics of sound and sight

Introduction

This section establishes digital media in a global context of interconnected and interrelated media ecologies by exploring digital media through sound composition, sound dissemination, electronic media art, the representation of data and creative process broadly conceived. It positions music as affectively vital.

For their insight into the affective impact of music we spoke to master audio engineer Ern Rose. Rose is predominantly a producer/engineer with a career in sound recording in the music industry stretching back to the late 1960s when he worked at Armstrong Studios – a commercial recording studio located in Melbourne, Australia. These studios grew out of Telefil Sound Recording and Film Studios, which was at the time housed in a converted cinema, subsequently becoming the largest studio complex in the southern hemisphere with five studios by 1972. After this time the company was renamed AAV (Armstrong Audio Visual). By the late 1980s and early 1990s, a staff consortium of AAV, headed by Rose, acquired the audio arm of operations of AAV, which recorded U2, Paul McCartney, Madonna, Bob Dylan and Crowded House. Rose worked directly with numerous artists including Renee Geyer Band (RCA Victor), Rick Springfield (Wizard Records), Little River Band (Capital Records), Split Enz (Mushroom), Deep Purple (Thames), Kiss (Sanctuary Visual Entertainment) among multiple others.

Ern Rose: You know I’m passionate about the role that music plays. Music is something that audiences hold in their memories. The recall associated with music I think is a massive aspect in remembering during our life or in our society. For example, when you relate music to film, the right piece of music can change how the audience feels or their reactions to what they see. It has the same impact in advertising or entertainment. It stirs emotions, our feelings and is a unique and powerful way to make a connection. Often you hear a song, and you can remember where you first heard it, who you were with. When an artist adds to that the value of the lyric, words about life experiences, the narrative coalesces the form.

Certain music preferences and style are often ingrained generationally for what has storied people’s lives; for what one is attached to – it accompanies nostalgia. The great songwriters write our lifetimes in their lyrics. I think that we are seeing this personal connection in older people, especially through those with dementia that listen to music from their earlier years. They will hear a tune and suddenly many that never rise out of a wheelchair will get up and start dancing.

Looking ahead, a form of artificial intelligence is entering the music market at the level of computer driven creation where it’s given a set of parameters – for emotions, style, timing – guidelines and data about music made over decades can be assessed for something ‘new’ that fits the exact formula. As much as a computer and artificial intelligence can look for patterns, however, it’s the human element of knowing the subtleties of culture, of society and human emotions, that is what inflects the music that cannot be fully re-produced by a machine. And there is something about experiencing music together. We are not going to move away from that, and I think seeing what Covid did to people when they were restricted during lockdown from going to live music underscores this – it was almost a form of starvation. We came up with ways to try to overcome this with home online performances and the like. Some were mediocre if not shocking, but audiences soaked them up. It is only coming back now to see and hear and just sense some live music that we appreciate this as what people have been craving. The collective experience of music is something special.

Chapter 1 is situated in the context of people connecting virtually, so prevalent during Covid-19 lockdowns. Angela Ndalianis undertakes a critical discussion of ‘being in the [virtual] world’ through exploration of the nature of exchange that occurs between the person in real life and their virtual self. This navigation is in the context of virtual reality (VR) platforms where Ndalianis argues there is a perceptual shift occurring in the VR experience one enters that offers an alternative experience of presence. They draw on the notion of a symbolic virtual body which consequently becomes a literal virtual body in this realm complicating the nature of embodiment as the body is materially present in real life while simultaneously being present in a virtual space. The case studies for close analysis are a VR experience of Hieronymus Bosch’s painting The Garden of Earthly Delights and the VR painting application Tilt Brush (created by Drew Skillman and Patrick Hackett). The author contends that emersion in fictional spaces affect our bodies haptically through their imaginative geographies, surfaces and textures, and the materiality they convey. As with all modes of communication, a perception of a doubled body also develops in the virtual space. Here, a body in motion becomes the means by which to understand and interpret the virtual world being inhabited.

In Chapter 2, Merlin Seller considers that nature of ludic gameplay that requires dedicated effort, skill and ingenuity and moves us to consider seriously how videogame aesthetics can articulate novel kinds of sense perception by taking a post-phenomenological understanding of atmospheres as ‘quasi-objective’ aesthetic objects. There is a continuum in gameplay along which all games can be placed, with ‘paidia play’ at one end and ‘ludus play’ at the other. According to this definition, paidia covers the spontaneous manifestations of the play instinct, or spontaneous or uncontrolled fantasy play – think of a child playing with an empty cardboard box and its seemingly endless possibilities for play options. This is not to say, however, that there are no rules. For example, a child pretending their cardboard box is a car will demonstrate actions of holding and turning a steering wheel and making the requisite brrrm brrrm noises. By contrast, ludus encompasses the various games which require concentrated industry and ability. Through two case studies of twenty-first-century videogames, Unfinished Swan (2012) and Scanner Sombre (2017), Seller argues that videogames offer users cyborg access to alien forms of experience, as well as new defamiliarized perspectives on sense perception, wherein each game challenges their respective players to sense hidden environments.

Turning to the examination of sound in detail for Chapter 3, co-authors Darrin Verhagan and Ben Byrne explore contemporary screen and digital media for the role that sound specifically plays in embodied audiovisual experiences. Drawing on sound design, sound studies and interaction design literature, Verhagen and Byrne argue that methods of sound production for VR, augmented reality (AR), mixed reality (MR) to 4D cinema go far beyond traditional understandings of sound design and score to instead employ sound as a crucial aspect of multisensory, interactive experiences that extend beyond the screen. The authors take examples of VR experiences via Collisions (2016) set in central Australia, 4D cinema and the superhero film Venom (2018) in 4DX, to the 360° documentary HOBO UFO (v. Chernobyl) (2019), an audiovisual release that employs Google Street View footage of Chernobyl. Each of these, the authors assert, employs digital technologies in aid of affective multisensory experiences in which ‘the viewer’ is explicitly spatially positioned, and the relationship of sound to movement is key such that the listener can see as much as hear sound.

Triton Mobley explicitly underscores the limitations of producing Blackness in image-making technologies in Chapter 4. As Mobley states, Blackness in America (as elsewhere) has always been rendered with a veil of opaqueness over it and this results in a kind of unseeing visibility. Mobley proposes a series of radical possibilities for visible Blackness – situating their central argument between film, formal architectural infrastructure(s) and speculative design. Here they position filmic space as ‘architectural space’ that has conserved the function of reconfiguring the imaginative desires of the audience. Mobley writes that Volumetric Black creatively reimagines possibilities for digital mirroring of Black bodies as a viewing experience without computational inadequacies. The author’s aim is achieved through a jugaad combination of image-processing computations, microcontroller programming and the inclusion of RGB light emitting diodes so that images of Black bodies are produced with an illumination that appears less visually compromised on visual displays. The aesthetic reconfiguration of visual Blackness is to be found when the filmic-architectural space and cultural collapse (i.e. between audience, (image) frame and infrastructure) so that the Volumetric Black might be revealed.

1 Virtual reality, the chiasm and the doubled body

Angela Ndalianis

Introduction

The year 2016 was hailed as the year of virtual reality (VR). The commercial release of the Oculus Rift, HTC Vive, Google Cardboard, PlaystationVR and other VR platforms signalled a perceptual shift in our mediated experiences of alternative realities by finally bringing VR into the private sphere. I am interested in the nature of embodiment in these new forms of VR. In this chapter, I focus on representational spaces and how they express their unique articulations of what phenomenologist Maurice Merleau-Ponty called ‘the flesh of the world’ (1968: 267). I begin with Merleau-Ponty’s definition of the virtual body, which he argues comes to the fore in all modes of communication, but, specifically, I examine how he understands his relationship to the paintings of Cézanne. While I begin with painting, the main focus of this chapter is to explore and extend Merleau-Ponty’s phenomenological reflections and consider their usefulness in understanding the user’s experience with VR. I contend that, when applied to VR, Merleau-Ponty’s symbolic virtual body becomes a literal virtual body that complicates the nature of embodiment. In particular, I am interested in Merleau-Ponty’s discussion of the virtual body as one that occupies a space ‘in between’ – a space he calls the intertwining or chiasm. Extending this idea, through an analysis of a VR experience of Hieronymus Bosch’s painting The Garden of Earthly Delights and the VR painting application Tilt Brush, I explore how the VR user occupies a space in between where an exchange occurs between the user and their virtual self; a new state of embodiment occurs, one that generates the perception of a doubled body. In examining this, I also turn to the writings of Maxine Sheets-Johnstone who elaborates on Merleau-Ponty’s ideas regarding thought and sensation being grounded in a body that moves. I argue that kinaesthetic perception becomes central to our experience of virtual space and that the body in motion becomes the means to making sense of the virtual world we inhabit.

The virtual and the virtual Bosch

In August of 2016, I visited the Prado Museum in Madrid. There were so many works by Ribera, Goya, Velasquez and others that I wanted to see, but it was in the Hieronymus Bosch rooms that I experienced something I had never experienced before. To this day I am not quite sure what happened to me. I found myself standing in front of the amazing triptych The Garden of Earthly Delights (ca.1490–1505) and becoming overwhelmed with intense emotion. I have become teary before when confronted with works of art, but never like this. A...

Table of contents

• Cover
• Half-title Page
• Dedication Page
• Title Page
• Contents
• List of Figures
• List of Contributors
• Acknowledgements
• Introduction Screen media, artefacts and intra-communication
• Section 1 Socio-aesthetics of sound and sight
• Section 2 Meaning-making in the data-driven era
• Section 3 Touch, body, metal, screen
• Section 4 Digital futures
• Index
• Copyright Page