Translation and Multimodality

Beyond Words

Monica Boria, Ángeles Carreres, María Noriega-Sánchez, Marcus Tomalin, Monica Boria, Ángeles Carreres, María Noriega-Sánchez, Marcus Tomalin

• Publisher: Routledge
• Year: 2019
• eBook ISBN: 9781000682144
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

Translation and Multimodality: Beyond Words is one of the first books to explore how translation needs to be redefined and reconfigured in contexts where multiple modes of communication, such as writing, images, gesture, and music, occur simultaneously. Bringing together world-leading experts in translation theory and multimodality, each chapter explores important interconnections among these related, yet distinct, disciplines.

As communication becomes ever more multimodal, the need to consider translation in multimodal contexts is increasingly vital. The various forms of meaning-making that have become prominent in the twenty-first century are already destabilising certain time-honoured translation-theoretic paradigms, causing old definitions and assumptions to appear inadequate. This ground-breaking volume explores these important issues in relation to multimodal translation with examples from literature, dance, music, TV, film, and the visual arts.

Encouraging a greater convergence between these two significant disciplines, this text is essential for advanced students and researchers in Translation Studies, Linguistics, and Communication Studies.

1
Transposing meaning

Translation in a multimodal semiotic landscape

Gunther Kress

1. Preamble: the question of translation in the contemporary multimodal semiotic landscape

In everyday interactions, banal or profound, actors in the contemporary social and semiotic world draw on a wide range of material means for making meanings (materially) evident – the modes of the semiotic landscape. In this chapter the aim is to explore what issues arise in moving from the dominance of one means (“language”) to a recognition of the potential equality of many means of making meaning evident (“the modes of multimodality”) and to consider the effects of that change on present conceptions of translation. Unlike, say, the chapter by Matthew Reynolds in this volume, the present chapter is not written from within translation theory. Its focus is a sketch of a paradigm shift, in terms of the issue at its core, moving from the centrality of “language” to a focus on “meaning”; and in terms of disciplines, a move from a linguistic to a semiotic frame. In that, it is part of the much larger changes affecting the global political, social, economic, and cultural landscapes, in Europe as elsewhere, which are unmaking centuries of relative stability. That brings the problem of “naming”: the names which had lent some stability to that world no longer adequately describe the contemporary social and semiotic landscape. Clarity in “naming” is required and, with that, the development of a terminology coherent and apt for that change. Throughout the chapter, I introduce requisite (semiotic) terms, tying them in to essential elements of the discussion.

2. A slightly odd take on translation: “Chinese Whispers” with a twist

At a semi-formal meeting, about twenty people – in the main, craftspeople/artists of various kinds, a few academics and others sprinkled in – are gently cajoled by the Master of Ceremonies (MC) of the evening to participate in a game of “Chinese Whispers.” The MC has copied a range of different kinds of images – abstract, figurative, contemporary, traditional, “western” or, indeed, from anywhere on the globe. Members of the group are invited to select one of the images. Everyone has received an A4 piece of paper and a pencil. The MC tells the group to “produce a written description of ‘your’ image, in maximally thirty minutes” (see Figure 1.1).

Figure 1.1 The translation game

When the time is up, the images that had been handed out are gathered by the MC, together with the written descriptions. She then distributes these accounts randomly, one for everyone there. Around the room are tables with boxes of coloured pencils and crayons, to produce the drawings that are to be made on the basis of the written description each person has received. Everyone now tries to produce the image that corresponds with the written description they have. After maybe forty minutes of drawing, the MC calls a halt. She invites everyone to find the original image which had been the basis of the description. There is considerable interest in comparing the written descriptions with the original and with what has just been drawn from what had been written.

Although everyone in the room is familiar with the game of “Chinese Whispers,” there is, nevertheless, astonishment at just how badly our notions of “communication” fit with a reality marked by difference of all kinds and at all points: difference of personal “interest” of the participants, leading to difference in focus, leading to difference in selecting aspects from a message; based on differences in the “deeper” perspectives from which each of us selects elements to newly constitute the message.

The game, as it is traditionally played, makes use of a shared meaning resource: speech. The “twist,” here, lies in the fact that there is no shared meaning-resource – no shared “code” – for moving from image to writing and back to image. The building blocks of images are not words; and while there are relations between elements of the image, they are neither the elements nor the relations of the grammar of writing: no Subject-Verb-Object relations nor, for instance, indications of time as tense. The meanings at issue in the game – whether written or visual – do express the meanings of a particular community, yet they are made evident by entirely different material/semiotic means: more or less adequately, in line with the distinct affordances of writing and image. It raises the question: even though meanings of one social group are dealt with, the absence of a shared code for writing and image makes the activity, in many ways, quite similar to that of translation. (One difference being the absence, here, as yet, of relatively well understood assumptions of various kinds, which exist in language-to-language translation.)

The traditional version of Chinese Whispers relies on the fact that the meaning resource of speech is shared by all players: it is used by each in relation to their differing interests and perspectives in the moment of playing. In the “twisted game” there is no shared meaning resource for moving between speech and image, even though there may be a broadly settled, general sense of shared meanings in that community of players. The “twisted game” brings out a strongly unsettling sense that writing and image each enable meanings to be made which cannot, however, easily be “shifted” from the one mode to the other – nor, actually, made at all in the other mode.

Each mode offers profoundly distinct potentials for documenting meaning. Immediately the problem of “naming” emerges: in the move from one mode, writing, to a different mode, image, translation is not exactly what is the issue here. A term such as transduction, indicating a shift of a specific kind, suggests itself.

The traditional version of the game makes clear that notions of “en-coding” and “de-coding” barely work even when using the shared resource of speech; in the “twisted” version it makes no sense. At a somewhat abstract level, writing and image realise aspects of social functions; they do not, however, share anything resembling a commensurate “code.”

What questions – beyond those we know from the traditional version – arise from playing the “twisted version”? Any painter, sculptor, or composer will be familiar with the request “Could you give me some idea what this is about?” At times the response might be an exasperated “Have a good close look; do listen; reflect for a while; it’s all there.” In schools, in most contemporary textbooks and in most online resources, whether in Biology or Mathematics, in Geography or Physics, the majority of the page or the screen is given over to image, rather than to writing. Much the same – or even more so – is the case in an ever-increasing number of social domains. How can we expect this to work when as a society we have no real sense of the resources needed to reconstitute, to transpose a meaning which appears as an image – a diagram of a magnetic field, let’s say – with the resources of writing, of words in syntactic relations?

Writing is still the privileged mode in important social domains (in schools, for instance, in the case of most exams; in legal domains; etc.). Yet the game does highlight the shortcomings of an approach based on the existing commonsense: writing does particular kinds of semiotic work; it does not and cannot do all of it. Despite centuries-long certainties that “language” is fully capable of expressing all meaning that needs to be expressed, whether as writing or as speech, each is a partial means of making meaning. The same is true of image, as it is true of every means of making meaning. It is that which makes the case for a social semiotic multimodal approach.

In the everyday, most messages still occur in face-to-face interaction; yet more and more frequently, messages come via the small screens of the devices we carry with us everywhere. There, image and writing usually appear together; though ever more frequently image is dominant or appears by itself. The majority of our communicational life is beginning to resemble versions of Chinese Whispers, with multiple twists.

The huge project looming to be addressed is to bring to light the characteristics and the regularities of the wide range of meaning resources we now use every day, in still very new ways. That is an urgent issue already in just about every area of our lives. In relation to the interests of the group assembled that evening, the game highlighted, yet again, how difficult – impossible even – it is to describe meanings that inhere in “making” and in “things made”: that is, meaning beyond the presently recognised, canonical means. There is a need to find ways other than the limiting, only partially adequate, routes of writing or speech for making such meanings evident: giving attention and recognition to meaning-makers in all areas of social life, to making, with all kinds of materials and to all things made.

In the game played at the meeting, the MC kept the instructions simple: “Document the meanings made by your image as a written account, an account sufficient to allow the meaning of the original in your written account to be reconstituted as an image, by another member of this group.” Nothing could seem simpler, right? Yet what consequences loom if we do not develop relatively reliable means for such a simple task?

3. A brief sketch of a social semiotic theory and its tools for documenting and transposing meaning

Social semiotics (Hodge and Kress 1988; van Leeuwen 2005; Kress 2010; Bezemer and Kress 2016) offers a clear focus on meaning and meaning-making. It suggests both a frame and tools to account for the constitution of meaning. In that frame, it offers an account of translation seen as the transposition of meaning in the multimodal semiotic landscape of the contemporary social world. The term transposition offers a conceptual/theoretical frame capable of dealing with changes in the making of meaning in all areas. “Position” – as an encompassing metaphor – is the crucial issue in all instances of making meaning: whether position in social environments, of semiotic resources, of personal “interest,” or of other significant factors. In what is still called translation, “position” is the issue.

A rudimentary sketch of social semiotic theory is needed to make that case. Here, a brief outline of that theory is followed by an account of sign-making as the core of that theory. A further step suggests that translation/transposition – in the now expanded sense of the transposition of all and any meaning – is an instance of communication and is accounted for in that overarching theory.

Social semiotic theory takes it as given that meaning arises in social actions and interactions, communication being perhaps the most common of these. As discussed in the Introduction, the theory draws on the work of the semiotician-linguist Michael Halliday, and specifically on two assumptions: one (stated by him in slightly varying forms), that “language is as it is because of the functions it serves in society” (Halliday 1978, 18–19); the other, that language, as a full communicational resource, has to satisfy three (social) functions. One deals with meanings about “states of affairs in the world,” the ideational function; another deals with the meanings of the social relations of the participants in interaction, the interpersonal function. A third deals with the meanings produced by relations of elements within the text, as well as the relation of a text with its environment, the textual function.

The first assumption asserts that the semiotic resources of a society are the outcome of their shaping in social (inter-)actions. The second asserts that in order for any semiotic resource to be a fully functioning means for human communication, it has to deal with meanings which arise in each of these three distinct yet entirely integrated social domains.

By substituting the phrase “semiotic resource” for “language,” and by bringing human agency clearly into the frame, Halliday’s statement becomes an encompassing social semiotic theory: “semiotic resources are as they are because of the functions they are made to serve in society.” It provides a workable frame for a social semiotic multimodal theory of meaning and meaning-making. It asserts, among other things, that every community has a range of resources for making meanings evident: speech, gesture, gaze, writing, and others; that is, the modes of social semiotic multimodal theory. Further, the characteristics of the modes (their elements and relations) are the product of social semiotic action – of social and semiotic work, in a specific community – with and on the material “stuff” of each mode. Modes and their constituent semiotic elements mirror both the interests of the makers and the potentials and limitations of the materials chosen for making meaning: their affordances (Gibson 2014). Examples of “material stuff” are: movement of parts of the body against the (relatively) still upper torso in the case of gesture; air, in the case of speech; movement of the body in space, in the case of dance. Meanings are made constantly by every member of a community; hence, all meanings are the meanings of that specific community.

As a consequence, modes – together with their elements and the relations of elements – are repositories, traces of the history of meanings shaped in one community. The distinctly different affordances of each mode are the outcomes of ceaseless social semiotic work with and against the potentials and the resistances of the materials for making meaning. This work produces the semiotic/cultural resources of a social group. In these, each mo...

Table of contents

• Cover
• Half Title
• Title
• Copyright
• Dedication
• Contents
• List of figures
• List of contributors
• Acknowledgements
• Introduction
• 1 Transposing meaning: translation in a multimodal semiotic landscape
• 2 A theoretical framework for a multimodal conception of translation
• 3 Meaning-(re)making in a world of untranslated signs: towards a research agenda on multimodality, culture, and translation
• 4 From the “cinema of attractions” to danmu: a multimodal-theory analysis of changing subtitling aesthetics across media cultures
• 5 Translating “I”: Dante, literariness, and the inherent multimodality of language
• 6 The multimodal dimensions of literature in translation
• 7 Translations between music and dance: analysing the choreomusical gestural interplay in twentieth- and twenty-first-century dance works
• 8 Writing drawingly: a case study of multimodal translation between drawing and writing
• Beyond words: concluding remarks
• Index