In 1963, the modern concept of virtual reality was born. From then it took more than twenty years for us to give this concept a name and, since then, we continue to have trouble agreeing upon the meaning of (virtual reality)VR and how exactly the term should be defined. VR is a field of study, a synthetic experience, an interactive computer simulation, a source of synthetic feedback, a three-dimensional representation of both the concrete and the abstract, an immersive experience generated by a computer—the list goes on. The same difficulty applies to sound, but here the alternate perspectives are even less compatible with each other. Sound can be, amongst other things, physical perturbations in the air or electrical impulses triggered by displacements of the cochlea hair cells. Sound can be synonymous with a source object, an event or a listener. It can exist only with a listener and it can exist without a listener. This emphasises the great challenge in understanding the composite of the two, VR sound.
Virtual Reality Sound as an Emergent Phenomenon
The central position of this book builds upon several years of research into auditory perception, sound design for digital games and applications of VR. These themes that have been brought together by emergence theory. Inherently integrative and holistic, emergence theory encourages us to try and reconcile various notions and positions on a subject, in order to understand it from a broader, more ecological perspective. It discourages silos of thought and proclaims the huge complexity of things, based on the assertion that such things do not exist in a vacuum and are instead part of a huge and ever-changing ‘aggregate ’ of potentials. Therefore, VR sound cannot be separated from either sound or VR, nor can it be separated from the numerous wider related elements that contribute to its precise nature. Such a framework may initially seem plausible but incomprehensibly dense and without clear practical application, but fortunately this is not the case. When we centre the thing we wish to understand within an emergence model, it becomes possible to map out the likely elements of its aggregate . Once constructed, we are then able to conceptually step back and observe both the potential contributors to the subject, and also the potential interactions that exist between them.
Let us take sound as an example. From the various perspectives on this subject, we can position various properties upon our conceptual map. This could include the source object, the event that is occurring, the physiology of the listener, the acoustic properties of the physical environment and the affective and cognitive state of the listener. This is itself a very cut-down list and within each of these properties we are likely to find many further sub-properties. How much detail we wish to include is at our discretion, but to collect the various elements together and draw lines between them provides us with a platform from which to further explore. For instance, our map could draw a line between proprioceptive feelings of comfort and short-term memory, raising the question of how these elements may influence each other and collectively (along with many other things) determine our perception of sound. The same can be done for sound in VR, which brings us the central argument of this book; that to better understand and create sound material for VR (and, in turn, create better VR in general) requires VR sound to be recognised as an emergent phenomenon. The intention is to present a more integrated framework of sound in VR that emphasises the importance of perceptual effects and user experience from within a wider cultural and technological domain. It acknowledges and attempts to uncover some of the hidden properties that influence our experience of sound in VR with the hope that this will encourage readers to take a step back when approaching VR design—improving both sound in VR through greater awareness of wider interrelating factors, and VR in general by way of acknowledging the vital importance of sound.
Chapter 2
Throughout the second chapter, we begin to uncover some of the key components that will constitute the emergent framework of VR sound. This includes a consideration of some of the broader cultural and philosophical issues that shape our experience in VR, including a discussion regarding how our history of associated technologies has steadily prepared us for acceptance of contemporary VR. We discuss some of the technological precursors that first made virtual experience a ubiquitous phenomenon (television, telephony) alongside contemporary associated technology such as applications of the Internet (emails, online shopping, video calls, social media , etc.) and mobile systems (smartphones, tablets). The central assertion of this chapter is that through such technology, the broader landscape has shifted to make present day a period in which we are now highly open to VR as both a concept and a technology. This chapter also addresses some fundamental issues regarding how VR is presently defined, what its relationship to digital games is and what is meant by ‘mixed-reality ’. How things have changed since the infamous narrative of 1990s consumer VR, that fell from grace so spectacularly, is also considered as this chapter asserts how issues of wider application, increasing corporatisation , technological ubiquity and ever-greater integration have contributed towards the much greater levels of public acceptance the contemporary VR has so far acquired. Lastly, this chapter makes preparations for the following discussion on an emergent framework of sound by discussing Deleuzean philosophy and virtuality as an organisational theory. The central argument raised here presents the virtual as a component of reality , countering the widely held assumption that a virtual world is an alternative to a ‘real world’.
Chapter 3
The third chapter brings the focus promptly back to sound with an exploration of Sonic Virtuality and how sound can be conceptualised as an emergent phenomenon. As with definitions pertaining to VR, this chapter posits that conventional perspectives on sound lack comprehensive explanatory power. Perspectives such as ‘sound as object’, ‘sound as event’ and ‘sound as phenomenon’ are reviewed, and we question where sound is actually located (at the source, in the transmission space between the source and the listener, at the point of hearing, or within the auditory cortex—or even nowhere at all). This chapter asserts that conventional theories on sound incorrectly present its empirical components as dominant and neglect to fully acknowledge to role of the listener. We then observe some of the primary issues with such perspectives by revealing problems in reconciling them with observable phenomena such as non-cochlea sound and auditory hallucinations (tinnitus , exploding head syndrome , musical hallucinations, etc.). In response to this, some of the other existing concepts that are beginning to acknowledge sound in more holistic terms are presented. These include theories of acoustic ecology , auditory grouping, perceptual localisation effects and the Figure Ground phenomenon . This chapter then closes with a discussion on Sonic Virtuality and how our perception of sound is best understood from an emergent framework. The underlying theory upon which Sonic Virtuality is based is also discussed, and we directly explore the literature surrounding emergence theory and associated concepts including Plato’s Allegory of the Cave , embodied cognition and construal level theory .
Chapter 4
Bringing together the preceding discussions on sound and VR, this chapter reviews key literature pertaining to the entity at the centre of all this, the user, discussing prominent user-experience notions. The review begins with a brief outline of physiological issues, specifically adverse effects pertaining to cybersickness , before addressing psychological components of user experience including flow, diegesis and fun . As a more contentious point of discussion, immersion and presence are discussed in more detail as we review the differences between system-side and user-side explanations of immersion. Largely originating in theory pertaining to digital games, these points of user experience are posited to be of equal, if not greater relevance to VR. Throughout this chapter, the various components of user experience are revealed to be connected to one another and, in each instance, sound is presented as a significant element, whether it be increasing immersion by way of its inherently surrounding and dynamic properties, or expanding diegesis by seamlessly transitioning its state from the diegetic world to the non-diegetic world. This chapter closes with a discussion on the implications of VR for identity and our understanding of ourselves. Again, taking notions from digital games a step further, the central assertion raised here is that VR is dramatically changing our concept of the Self, by presenting us with seemingly limitless opportunities for customising our existing identity and forming entirely new ones. Including matters of depersonalisation and even out of body experience, the increasing fidelity and convincing realism of VR are shown to be emphasising new issues that were once considered purely science fiction .
Chapter 5
‘Representations’ considers the various ways in which fictional representations of VR in cinema, television and literature have dramatically influenced user expectation and experience in the past, and how they continue to do so today. From the utopias of ancient classicism to the Orwellian nightmares of the twentieth century, the nature of fictional worlds is revealed to be a substantial influence upon the nature of virtual worlds. From matters of aesthetics and scale to alternate forms of literary realism , both the classic conventions of world-building in fiction and the established methods of breaking them are shown to draw noteworthy parallels to the worlds in digital games and VR. Throughout Chap. 5, the discussion is centred upon the treatment of sound in such representations, examining both how the sounds of fictional worlds have relevance to those of virtual ones, and also how the quality of sound in fictional VR is described in literature and portrayed upon the screen. This chapter continues with a review of the more contemporary representations of VR in cinema and television, as we consider how such visions shaped the expectations for 1990s VR, setting the bar so incredibly high that actual VR had an insurmountable fantasy to live up to. The chapter then expands upon these expectations by looking into the foundational desires underpinning the fantasy of VR.
Chapter 6
‘Technological history’ adds another collection of components to the overarching emergent framework of VR sound by tracing the history of artistic and mechanical precursors to VR sound and reviewing how the functionality and aesthetics of the past continue to influence VR design and technology today. Key characteristics of contemporary VR (including panorama , stereoscopy , parallax and multimodality ) are documented and, throughout this discussion, the lesser-mentioned auditory aspects of these technologies are detailed, from cacophony of whirring film reels that sonically characterised the Kaiserpanorama to the use of binaural audio recordings in the Sensorama . The history of VR, up to and including the 1980s, reveals notably more points of interest with regards to VR research, points that are explored in this chapter before we turn our attention to the consumer VR of the 1990s. Here, the development of VR sound is reviewed, and it is revealed just how consistently underappreciated sound was throughout this period. This chapter comes to a close with a more detailed exploration of VR sound technology. Reaching back to the dawn of electronic sound and the earliest approaches to sound recording, this discussion makes two primary assertions. Firstly, that sound reproduction technology is the auditory equivalent of VR, with headphones and loudspeaker arrays essentially HMDs (head-mounted display) and CAVE systems for the ears. Secondly, because headphones and loudspeakers were not subject to the same hype as HMD/CAVE systems, their technology avoided the 1990s crash into the trough of disillusionment and instead have made consistent progress which continues to the present day—to the extent that certain properties of contemporary headphones may be indicative of future developments in VR-HMDs.
Chapter 7
‘Expectations, reality and digital games’ focus specifically upon VR’s turbulent narrative throughout the 1990s and early 2000s with a look at how VR sound continued to progress, largely through its implementation in digital games. This chapter begins with a response to the expectations set by fictional representations (discussed in Chap. 5), revealing precisely how the realities of 1990s VR inevitably failed to match such impossible standards. Throughout the majority of this chapter, digital games are posited to have been the primary incubator of consumer VR concepts and technology, both in general and with regard to VR sound. Points discussed include the development of the first-person perspective, multichannel audio technology and new approaches to positional/3D audio, intelligent non-player game characters that responded to sounds made by the player, progress in haptics and motion tracking technologies, new speech recognition and voice-command systems and refined affective frameworks of game sound. These issues reveal how the underlying concepts and technologies of VR and VR sound were successfully retained and advanced by digital games, meaning that as the time for contemporary consumer VR rolled around, the technology and design techniques were more refined and in a much better position to deliver a quality experience.
Chapter 8
‘Current status’ brings us up to now with a review of how sound is currently treated within VR, plus speculations on future developments. Commencing with a hypothetical exploration of the likely integrated future of VR, this chapter asserts that, by way of technology that includes the Internet of Things , social media and collaborative virtual environments, integration is a pivotal theme when considering the shape of things to come. This incorporates integration at the technological level, in which future VR is poised to be ...