Multisensory Perception

From Laboratory to Clinic

K. Sathian,V.S. Ramachandran

• Publisher: Academic Press
• Year: 2019
• ISBN:
  9780128125649
• Topic: Psychology
• Subtopic: Neuropsychology
• Index: Psychology

Multisensory Perception: From Laboratory to Clinic surveys the current state of knowledge on multisensory processes, synthesizing information from diverse streams of research and defining hypotheses and questions to direct future work. Reflecting the nature of the field, the book is interdisciplinary, comprising the findings and views of writers with diverse backgrounds and varied methods, including psychophysical, neuroanatomical, neurophysiological and neuroimaging approaches. Sections cover basic principles, specific interactions between the senses, the topic of crossmodal correspondences between particular sensory attributes, the related topic of synesthesia, and the clinic.

• Offers a comprehensive, up-to-date overview of the current state of knowledge on multisensory processes
• Coverage includes basic principles, specific interactions between the senses, crossmodal correspondences and the clinical aspects of multisensory processes
• Includes psychophysical, neuroanatomical, neurophysiological and neuroimaging approaches

Section II

Multisensory interactions

Outline

Chapter 7. Visuo-haptic object perception

Chapter 8. Multisensory processes in body ownership

Chapter 9. Visual–vestibular interactions

Chapter 10. Multisensory flavor perception: A cognitive neuroscience perspective

Chapter 11. Audiovisual crossmodal correspondences: behavioral consequences and neural underpinnings

Chapter 12. How do crossmodal correspondences and multisensory processes relate to synesthesia?

Chapter 13. Synesthesia: the current state of the field

Chapter 14. How synesthesia may lead to enhanced memory

Chapter 7

Visuo-haptic object perception

Simon Lacey ^{1 , 2} , and K. Sathian ^{1 , 2 1} Department of Neurology, Neural & Behavioral Sciences, and Psychology, Pennsylvania State University, Hershey, PA, United States ² Departments of Neurology and Psychology, Emory University, Atlanta, GA, United States

Abstract

Vision and touch have many similarities in their processing of information, manifested in multiple behavioral similarities in terms of categorization, recognition, and individual differences. This chapter reviews how these similarities contribute to multisensory object processing. For example, similar unisensory visual and haptic representations are integrated into a multisensory representation that supports both visuo-haptic crossmodal object recognition and view-independence. These behavioral similarities between vision and touch, and the evidence for integration of visual and haptic information into a multisensory representation, imply a shared neural basis for visuo-haptic object processing. We review the evidence that several brain regions, previously thought to be specialized for aspects of visual processing, are additionally active during analogous haptic tasks. Finally, we describe a model of visuo-haptic multisensory object recognition in which the object-selective lateral occipital complex is served by both top-down and bottom-up pathways depending on object familiarity and variable involvement of object and spatial imagery processes.

Keywords

Categorization; Crossmodal; Effective connectivity; fMRI; Metamodal; Multisensory; Object imagery; Recognition; Spatial imagery; View-dependence

Introduction

The field of object perception and recognition has been dominated by vision science even though in real life these are invariably multisensory processes. For a long time, haptics was the “Cinderella” modality, ¹ with the other senses left even further behind. ^2,3 Now, times have changed: following seminal work by Klatzky and Lederman in the 1980s ^4–6 ( ^7,8 for reviews), haptics has developed as a field in its own right, alongside renewed interest in the psychophysics and neurophysiology of tactile perception. ^9–12 The 1990s ushered in an explosion of interest in multisensory interactions at both behavioral and neural levels. The old orthodoxy of a brain organized around discrete unisensory inputs has yielded to a growing consensus around the concept of a “metamodal” brain whose regions are organized around specific tasks and receive multisensory inputs. ^13–15 In each subsequent section of this chapter, we review the interactions and commonalities between vision and touch for a particular aspect of multisensory object processing, firstly from a behavioral standpoint and then moving on to consider the neural basis. A review of basic within-modal haptic and crossmodal visuo-haptic object recognition sets the stage. Because grouping like objects together is one way to facilitate recognition, we then move to a consideration of recent work on the similarities between visual and haptic categorization and crossmodal transfer of category knowledge. Changes in object orientation pose a significant computational challenge to object recognition; we show that this challenge is solved by a multisensory representation that underlies both crossmodal recognition and view-independence. We next address individual differences in visuo-haptic representations and, finally, review the evidence for our working conceptual model of multisensory visuo-haptic object recognition.

Before proceeding to the main focus of this chapter on visuo-haptic object perception, we lay some groundwork with a few examples of what from a classical perspective is considered “crossmodal” recruitment of “visual” cortex during tactile tasks. Because this topic has been extensively reviewed recently, ¹¹ only a brief summary is given here. Our laboratory provided the first demonstration of activity of a visual cortical area during normal tactile perception, in a positron emission tomographic study in humans. ¹⁶ In this study, we found that tactile discrimination of the orientation of gratings on the fingerpad, relative to a control task requiring tactile discrimination of grating groove width, recruited a focus in extrastriate visual cortex, close to the parieto-occipital fissure. This focus, corresponding to the location of human V6, ¹⁷ had previously been shown to be active during visual discrimination of grating orientation. ¹⁸ Using transcranial magnetic stimulation (TMS) to transiently interfere with the function of this parieto-occipital region, we were able to disrupt tactile performance on grating orientation discrimination, ¹⁹ signifying that this area is truly involved functionally, rather than the activation merely being an epiphenomenon. Analogously, other neocortical regions originally described as exhibiting selectivity for specific visual attributes are also active when these attributes are sensed nonvisually. For instance, the human MT complex (hMT+), a visual cortical region well known to encode visual motion, is also active during tactile ^20–23 and auditory motion perception. ²⁴ Together, these findings suggest that hMT+ functions as a modality-independent motion processor. Similar findings have been reported for convergent processing of various other visuo-haptic object properties, as will be reviewed later in this chapter.

Haptic and visuo-haptic object recognition

Behavioral studies

The speed and accuracy of visual object recognition is well established. Haptic recognition is also highly accurate, at least for familiar objects, with 96% correctly named overall: 68% in less than 3 seconds and 94% within 5 seconds. ⁴ Although slower than visual recognition, longer haptic response times are likely due to the longer time required to explore larger items compared with essentially parallel visual processing across the retinal image ⁴ ; in fact, in some contexts, a “haptic glance” of less than 1 second is sufficient. ²⁵ Interestingly, haptic object recognition is possible with the feet as well as the hands, albeit more slowly and less accurately, with hand and foot performance being highly correlated across individuals. ²⁶ Haptic identification begins with a “grasp and lift” stage that provides low-level information about an object and proceeds to a series of hand movements, known as “exploratory procedures” (EPs), which extract more precise information. ²⁷ EPs are property-specific; for example, texture information is extracted by lateral motion while precise shape information is provided by contour following. ⁶ These object properties differ in their salience to haptic processing depending on the context. Under instructions that emphasize haptic processing, salience p...