Multimodal Studies

Exploring Issues and Domains

Kay O'Halloran, Bradley Smith, Kay O'Halloran, Bradley Smith, Kay O'Halloran, Bradley A. Smith

• Publisher: Taylor & Francis
• Year: 2012
• eBook ISBN: 9781136811166
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

The phenomenon of multimodality has, as Jewitt observes, generated interest "across many disciplines...against the backdrop of considerable social change." Contemporary societies are grappling with the social implications of the rapid increase in sophistication and range of multimodal practices, particularly within interactive digital media, so that the study of multimodality also becomes essential within an increasing range of practical domains. As a result of this increasing interest in multimodality, scholars, teachers and practitioners are on the one hand uncovering many different issues arising from its study, such as those of theory and methodology, while also exploring multimodality within an increasing range of domains.

Such an increase and range of interest in multimodality heralds the emergence of a distinct multimodal studies field: as both the mapping of a domain of enquiry, and as the site of the development of theories, descriptions and methodologies specific to and adapted for the study of multimodality. The present volume presents a range of works by an impressive international roster of contributors who both explore issues arising from the study of multimodality and explore the scope of this emerging field within specific domains of multimodal phenomena. Contributors aim to show that each individual work and works in general within multimodal studies represent a dialectic or complementarity between the exploration of issues of general significance to multimodal studies and the exploration of specific domains of multimodality; while characterizing specific works as tending to some degree towards one or other of these main areas of focus. Such a characterization is seen as part of a move towards the identification and thus development of a distinct field of multimodal studies.

1	Multimodal Studies Kay L. O’Halloran and Bradley A. Smith

The phenomenon of multimodality has, as Jewitt (2009: 3) observes, generated interest ‘across many disciplines . . . against the backdrop of considerable social change’. Contemporary societies are grappling with the social implications of the rapid increase in sophistication and range of multimodal practices, particularly within interactive digital media, so that the study of multimodality also becomes essential within an increasing range of practical domains. As a result of this increasing interest in multimodality, scholars, teachers and practitioners are on the one hand uncovering many different issues arising from its study, such as those of theory and methodology, while also exploring multimodality within an increasing range of domains, for example, corporate advertising, cartoons, museums etc (e.g. Bednarek and Martin 2010; Jewitt 2009; Ventola and Moya 2009).

Such an increase in interest in multimodality heralds the emergence of a distinct field of study within which a diverse range of scholars and practitioners converge to discuss phenomena and issues specific to multimodal study and develop theories and methodologies appropriate to such a field. Yet at the present state, as Jewitt suggests (2009: 2), ‘[m]ultimodality, it could be argued, strictly speaking, refers to a field of application rather than a theory’. Kress (2009: 54) similarly observes that ‘multimodality as such is not a theory even though it is often used as if it were. The term maps a domain of enquiry’. Nevertheless, it is a field that requires the (Jewitt 2009: 2) ‘ongoing development of theories that account for the multimodal’. Thus, as Jewitt argues (2009: 12), ‘multimodality can be understood as a theory, a perspective or a field of enquiry or a methodological application’.

While the term multimodality, as such, does ‘map a domain of enquiry’ (Kress 2009), we would like to draw an important distinction here between studies of multimodality, which focus on particular domains of enquiry (multimodal phenomena within specific contexts, media, etc), and multimodal studies as a field of expertise, distinct from linguistics or any other field of expertise. The reason for this distinction is that while multimodality has tended to be explored within specific sites and for specific disciplinary purposes, any study of multimodal phenomena inevitably raises issues that have wider relevance to the study of multimodality in general, while proposals for solutions to such issues have relevance and application to any particular study of multimodality. In this sense, although there can be no (single) theory of multimodality, certainly there can, should be and indeed are already emerging theories and descriptions of and methodologies for studying multimodal phenomena (semiotic resources, modes and their interactions in multimodal discourse)—in the same way as there are theories (not a theory) of language within the field of linguistics potentially applicable to any study (within or outside of linguistics) involving a consideration of language. The recognition of the need to develop and apply theories of multimodality is in fact a prerequisite for the emergence of a distinct field of multimodal studies, as a site for the development of such theories.

Although most scholars working within this emerging field do come from established disciplines (as identified in Kress 2009), with their own theoretical and descriptive orientations, styles and concerns, there has been a clear movement towards the development of generalisations applicable beyond the particular concerns of those studying within particular domains of reference or with particular academic backgrounds and with application to the study of multimodal phenomena in general. Scholars such as O’Toole (1994/2010), Kress and van Leeuwen (2006), O’Halloran (2005), Bateman (2008), Lemke (2009) and Baldry and Thibault (2006) have, for quite some time, devoted energy to developing theoretical, descriptive and methodological resources particularly adapted to and for the study of multimodality, in general, and in facing issues arising from such developments. Jewitt’s comprehensive (2009) volume is an exemplar of the diversity of interests and issues in and approaches to contemporary multimodal studies and offers itself a persuasive argument that while there is no single theory of multimodality as such, there are certainly distinct theoretical concepts and frameworks emerging from the study of multimodality as a field. Martin’s (2010: 1) contribution also is explicitly ‘concerned with developing the general theoretical framework informing’ research into the variety of semiotic resources used by humans to communicate meaning within that volume. The development of such theories, methodologies etc to account for multimodal phenomena across multiple domains of application is therefore evidence that such a field is emerging to take its place alongside other established fields such as linguistics.

Thus it is in two senses that we refer to the emergence of a distinct multimodal studies field: as both the mapping of a domain of enquiry and as a site for the development of theories, descriptions and methodologies specific to and adapted for the study of multimodality (although potentially applicable, at certain levels of generality, to other distinct fields of study, including linguistics). In the first sense, multimodal studies applies existing generalisations (of theory, description, methodology) to the exploration of specific multimodal phenomena, sets of texts or contexts in order to cast new light on those domains. Such domains might be more broadly defined areas of multimodality; e.g. ‘language of displayed art’ (O’Toole 1994), ‘grammar of visual design’ (Kress and van Leeuwen 2006), ‘speech, music, sound’ (van Leeuwen 1999), ‘mathematics discourse’ (O’Halloran 2005); particular (sets of) semiotic resources in interaction; e.g. images and text (e.g. Martinec 2005; Unsworth and Cleirigh 2009) and gesture and phonology (e.g. Zappavigna, Cleirigh, Dwyer and Martin, 2010); or sites where multimodal discourse is at issue, such as classroom discourse (e.g. Clarke 2001; Jewitt 2006), corporate advertising, video and interactive digital media such as games and the Internet, mobile media etc (e.g. Jewitt 2009; Ventola and Moya 2009).

In the second sense multimodal studies, such as those listed above, use texts or types of text to explore, illustrate, problematise or apply general issues in multimodal studies, such as those arising from the development of theoretical frameworks specific to the study of multimodal phenomena (e.g. modeling semiotic resources other than language, inter-semiosis and the integration of semiotic resources and resemiotisation of multimodal artefacts and events) or methodological issues (including challenges in transcription, analysis and representation within publications). In any particular work, of course, one may always find a complementarity of attention, to the specifics of a particular corpus or area of research, practice or teaching, and to the more general issues that inevitably arise when considering particular domains of research. Both O’Toole’s (1994) and Kress and van Leeuwen’s (2006) pioneering works, while extending the study of ‘language’ and ‘grammar’ into new domains, were also clearly foundation texts exploring issues of theory and methodology of general significance to multimodal studies.

This categorisation is therefore perhaps better seen as a continuum, the two different orientations—focus on general theoretical and methodological issues, or on specific domains of study—representing poles along which individual works range in terms of their major concerns. This approach acknowledges that most, if not all studies, no matter how focused on an issue of general relevance or a specific domain of application, contribute both to the development of our understanding of multi-modality in general as well as to the application of that understanding to the study of specific domains of multimodality. The main point to be made here is that the specific demands of multimodal phenomena motivate ways of approaching, conceptualising or doing multimodal studies distinct from those which have been standard practice within the study of language or other (particularly monomodal) semiotic phenomena, but that such demands only appear when analysts are confronted with specific multimodal phenomena in actual texts. Such considerations appear relevant if we are to begin considering multimodal studies as a field, not only of application but also as the site of emerging theories, descriptions and methodologies.

EXPLORING ISSUES AND DOMAINS

In this volume our aim is to present new explorations within the emerging field of multimodal studies by bringing together fourteen chapters that both discuss issues arising from the study of multimodality and explore the scope of this emerging field within specific domains of multimodal phenomena, to map out an exemplar of current multimodal studies work. The chapters in this volume are thus organised into these two broad categories according to the main focus within each: either on exploring general issues arising from multimodal studies or on extending multimodal studies into or focusing on specific domains of multimodality.

However, as discussed in the previous section, there is always a complementarity of attention within any work on multimodality, to the specifics of a particular corpus or area of research, practice or teaching, and to the more general issues that inevitably arise when exploring these domains. Generalities tend towards obscurity or irrelevance if developed without reference to the specifics of actual discourse, while insights obtained into specific texts or types of texts remain trivial or barren, or are indeed impossible, if uninformed by the consideration of more general perspectives on theory and methodology. We wish therefore to present the various studies in this volume as representing a dialectic and complementarity, between the exploration of issues of general significance to multimodal studies and the exploration of specific domains of multimodality, while, however, also acknowledging that some works do tend towards one or other of these main areas of focus.

We propose, therefore, following our observations in the previous section, to present this categorisation as a continuum, the two different orientations—focus on general issues, or on specific domains of study—representing poles of a cline, along which individual works range in terms of their major concerns. The aim of this approach is to characterise the works in terms of their main focuses, while at the same time highlighting the productiveness of this complementarity of focus both within individual chapters and across the volume as a whole.

The terms ‘issues’ and ‘domains’ are used here in a very general sense, in the present volume. First, issues may be those involving theoretical or methodological apparatus, the comparison of or application of different theoretical or analytical approaches or models to account for the integration of different semiotic modes. Thus Bateman’s Chapter 2 is concerned with the theory of semiotic mode, with the aim of developing a definition of a semiotic mode that supports the identification of more fine-grained semiotic modes than has hitherto been the case, and which has greater applicability and responsiveness to the needs of multimodal texts. Drawing on social semiotic approaches to multimodality, but also upon work within other fields such as cognitive science, computer science and film studies, the chapter begins by problematising the tendency within multimodal studies towards the application of a priori assumptions with respect to this important aspect of multimodal theory: in particular the assumptions that such categorisations are self-evident and/or aligned with sensory modalities (visual, auditory modalities etc). Bateman is very much focused on theory in his discussion, but he demonstrates through consideration of empirical analysis how such a characterisation of semiotic modes can be of assistance in analytical tasks.

In Chapter 3 Smith considers the consequences of taking different approaches to the study of multimodal phenomena, finding analogies in the history of intonation study. Exploring bottom-up (anatomistic) and top-down (functional) approaches to intonation description, he shows how two different theoretical approaches to the study of intonation can yield different results in terms of analysis and in terms of what gets analysed and argues that each approach thus has its own affordances and constraints in terms of its capacity for making statements of meaning about semiotic phenomena. He also explores variation in the interpretation of a written transcript into speech, the implications of this potential variation for literate cultures in general and concludes with some observations of relevance to theory and practice in multimodal studies.

Feng explores the calibration of a cognitivist perspective with aspects of social semiotic theory in Chapter 4: in particular, in relation to the construction and viewer interpretation of spatial orientations and page layout and the construction of persuasive ideology through such resources. The comparison of different theoretical perspectives is however firmly grounded in the analysis of a corpus of 100 static visual car advertisements from newspapers, magazines and the Internet.

Berry and Wyse discuss issues arising from the design of tangible interfaces for music composition in Chapter 5, in particular, the relations of the abstract and material planes in music and in computational interfaces. The discussion combines perspectives from the study of music, including discussions of the composition process, inspiration and execution with respect to instruments in their relations to and representations of musical abstraction, from computing, in particular the development of graphical user interfaces, and also from Piaget’s observations of child development of formal thinking through concrete operations. They observe that tangible interfaces generally offer advantages in terms of their physical accessibility and nature, features that are of benefit to adults as well as children, but are limited in what they can offer in terms of the large-scale abstractions that motivate and constitute a significant aspect of music composition.

Goebel draws upon ethnomethodology, linguistic anthropology and studies of embodied interaction to present an exploration of the concept of ‘enregisterment’ in Chapter 6. Via the analysis of an episode taken from a corpus of shows from an Indonesian television serial, Goebel explores how multimodal signs become emblems of identity leading to the formation of a ‘semiotic register’, by which interactants orient to one another with respect to characteristic signs of their personhood. Goebel teases out the distinctive affordances of the televisual medium within this genre for representing personhood in relation to ethnicity and social relations and discusses how the persistence of particular semiotic encounters over time, across a variety of multimodal signs such as facial expressions, gestures, prosodies and the like, creates an emergent semiotic register which then forms the context for interpretation of such emblematic signs. The chapter thus contains important suggestions towards the theory and analysis of multimodal signs, particularly the integrative analysis of multiple signs within multimodal communicative acts.

In Chapter 7, the final chapter in the ‘Issues’ section of the volume, Van Leeuwen is ostensibly concerned with an argument for and suggestions towards the study of a new domain, urging a new ‘semiotics of decoration’. But for van Leeuwen, this domain of application is very abstractly identified, ‘decoration’ representing here a particular philosophical approach to design within different modes, materials and eras—in dress, architecture, PowerPoint, language, music, typography etc in the nineteenth and twentieth centuries—which is contrasted with the bare functionalism of the Bauhaus and other similar approaches to design. However, in extending and thus defining the domain of multimodal studies this way, van Leeuwen in fact models an approach to multimodality, one which has important consequences for the way in which the emerging field of multimodal studies might develop. He shows that, as semioticians, we need attend to meaning-making activities beyond those traditionally thought of or attended to as such. Not for the first time, van Leeuwen may have sketched here not only a whole sub-field for multimodal studies but also an approach to doing multimodal semiotics.

In the same way that the term ‘issues’ in this volume encompasses a broad range of theoretical and methodological concerns, ‘domains’ of multimodal phenomena may be defined in a variety of ways, as evidenced in van Leeuwen’s chapter. Like van Leeuwen, Eisenlauer, although interested in a particular domain, ‘personal publishing texts’, in fact explores a broader domain of study in Chapter 8: social networking as a category, distinct from the old and new media through which such social action has historically been mediated. However, here the domain itself is clearly the primary concern, with the discussion being based upon the analysis of an interesting and valuable corpus of data: examples of the German Posieal-bum or ‘poetry album’, a site for social networking since the mid-sixteenth century, persisting into the twentieth century primarily amongst school pupils. Eisenlauer applies a diachronic perspective on a new media form, Web 2.0 social network sites, showing the continuities and also the differences between new and old forms of media (again, similar to van Leeuwen who explores decoration across different modes and media and thus shows the semiotic continuities between them). Nevertheless, in relating the analysis of higher-level social contexts and communicative structures to their expression in lower-level media resources, Eisenlauer thus models a useful approach to multimodal analysis.

Maier explores the domain of marketing discourse in Chapter 9, specifically the multimodal communication of knowledge within eco-business contexts. While clearly focused on this domain as an area of sustained interest, Maier seeks ‘to identify how the meaning-making potentials of language and images are integrated, and how this multimodal integration influences the persuasive communication of knowledge types’, with a ‘central focus on the model of analysis’. Maier explores the issue of interdependencies between different semiotic modes, with the verbal and visual modes of discourse being shown to subvert rather than complement one another, thereby identifying an important change in advertising discourse from persuasion to presenting eco-friendly credentials. The chapter also orie...

Table of contents

• Cover
• Halftitle
• Title
• Copyright
• Dedication
• Contents
• List of Figures
• List of Tables
• Copyright Permissions Acknowledgments
• Acknowledgments
• 1 Multimodal Studies
• PART I: Issues in Multimodal Studies
• PART II: Domains of Multimodal Studies
• Contributor
• Authors Index
• Subject Index