The Language of New Media Design

Theory and Practice

Radan Martinec, Theo van Leeuwen

• Publisher: Routledge
• Year: 2020
• Print ISBN: 9780415372572
• eBook ISBN: 9781000155266
• Topic: Computer Science
• Subtopic: Digital Media
• Index: Computer Science

The Language of New Media Design is an innovative new textbook presenting methods on the design and analysis of a variety of non-linear texts, from websites to CD-Roms. Integrating theory and practice, the book explores a range of models for analyzing and constructing multimedia products. For each model the authors outline the theoretical background and demonstrate usage from students' coursework, commonly available websites and other multimedia products. Assuming no prior knowledge, the book adopts an accessible approach to the subject which has been trialled and tested on MA students at the London College of Communication. Written by experienced authors, this textbook will be an invaluable resource for students and teachers of new media design, information technology, linguistics and semiotics.

1 Introduction

Language and new media design

Language is essentially a resource for communicating through speech or writing. It provides us with ways of patterning sound or letter forms that are associated with meanings. These patterns and their associations with specific meanings have developed over time, in part through trial and error and gradual habituation, in part through deliberate and systematic efforts to regularize and regulate them. As a result language has become an immensely rich resource that allows us to communicate a vast range of content and to do so in formats that can perform a vast range of communicative functions – informing us, entertaining us, lecturing us, persuading us, and much more.

The new media are our most recent resource for communication. They differ from language in three ways. They are multimodal, combining language with visual communication and sound. They are non-linear, combining spatial and temporal patterns. And they are new, lacking the long history and the many years of systematic thought that have made language what it is.

But fundamentally, and in their essence, they are, or should be, like language. Like language they are resources for communication that provide us with ways of patterning text and image that users can, or should be able to, associate with meanings. It stands to reason that new media designers might have something to learn from the study of language, that it might pay to approach new media with the long-established and tried and proven concepts and methods of linguistics.

The fundamental idea of this book is that new media are, or should be, structured by invisible underlying patterns that connect image, sound and text into meaningful wholes. We call these patterns ‘non-linear models’. Non-linear models are semantic constructs that map out the relations between concepts in the semantic fields, or fields of meaning, that underlie new media products. Meaning itself is of course intangible. It needs to somehow be made concrete, perceivable by our senses. The way non-linear models meet the eye and ear of the user is through their realization, or translation, into navigation and layout patterns in the interface. These are the forms by means of which the meanings, i.e. the non-linear models, are accessed. What follows is that the more clearly and explicitly the non-linear models are translated into navigation and interface, the easier it is for the user to understand the non-linear semantics of new media products, and the more likely it is that the message will get across.

Systemic linguistics

Figure 1.1 Basic choices of process and entity

The linguistic model our thinking about new media takes inspiration from is systemic linguistics (e.g. Halliday 1994; Eggins 1994). Systemic linguistics describes language as a set of choices, extending from the most general to the most specific. These choices form the ‘meaning potential’ of (a) language. They are expressed as abstract features that specify among other things the processes and entities that make up the structure of the basic unit of language, which is the clause, or simple sentence. These entities and processes are then realized, or translated, by verbs and nouns and other such familiar grammatical categories. In the case of processes, for instance, the most general choice is that between ‘narrative processes’, which represent actions and events (for instance ‘walking’, ‘speaking’ and ‘desiring’), and ‘conceptual processes’, which represent more or less permanent and unchanging ‘states’ (for instance ‘having’ and ‘being’). As can be seen in Figure 1.1, each leads to more specific choices. Narrative processes, for instance, include ‘material processes’ (such as ‘walking’ or ‘building’), ‘verbal processes’ (such as ‘saying’ or ‘asking’) and ‘mental processes’ (such as ‘thinking’ or ‘fearing’). ‘Conceptual processes’ include ‘having’ and ‘being’. Needless to say, such choices lead to further, even more specific choices, until we arrive at very specific actions such as ‘walking’, ‘protesting’, ‘smelling’, and so on. The same applies to ‘entities’, as also shown in Figure 1.1. Different kinds of processes combine with different kinds of entities. ‘Mental processes’, for instance, combine with two kinds of entity, a human one, the entity whose mental process it is (it may also be any other entity to which we ascribe similar mental processes as we do to humans) and the object of the mental process (what it is that is being thought or feared or whatever). For instance in the sentence ‘He feared the worst’, ‘he’ is what systemic linguists call the ‘Senser’ (the human entity who does the ‘fearing’), ‘feared’ is the mental process, and ‘the worst’ is the object of the fear, the ‘Phenomenon’, as systemic linguists call it.

According to systemic linguists, language is always patterned to simultaneously communicate three broad types of meaning – ideational, interpersonal and textual. Linguistic messages such as sentences, and also larger stretches of text, always have an ideational, an interpersonal (we might also say, ‘interactive’) and a textual meaning. Ideational meanings communicate information, for instance the fact that there was a person who ‘feared the worst’. But by varying the interpersonal patterning, different interactions can be created while keeping the information the same, for instance the choice between providing information (‘He fears the worst’) and requesting information (‘Did he fear the worst?’) (Figure 1.2). Textual meanings allow different parts of the information to be emphasized or foregrounded in different ways, and to create coherence and cohesion in longer texts. They do this both by cohesive devices that hold a text together – such as anaphoric pronouns that refer back to entities that were mentioned earlier – and by more or less rhythmically summarizing and expanding a text’s content.

The linguistic choices of speakers and writers will be driven by their communicative purposes and by their target audiences. The same applies to new media designers. They also have different purposes to achieve and they must therefore also have different ideational, interpersonal and textual patterns to choose from. Internet designers refer to such communicative purposes as ‘strategies’ and subsume them under strategic web design; in systemic linguistics they have been referred to as ‘genres’. Just as the linguistic choices of speakers or writers are attuned to the audiences they are addressing, so in new media the target audience must also be an important part of the picture. In our model of new media design, the characteristics of the target audience, which in systemic linguistics have been called ‘coding orientations’, have two functions – they provide motivation for the choice of one strategy over another and they influence the choice of translations of the meanings into forms, i.e. of the non-linear models into navigation and interface.

Figure 1.2 Basic choices of interpersonal meaning (after Halliday 1994)

So we take six key elements from systemic-functional linguistics:

• Communication always involves ideational, interpersonal and textual meanings that run in parallel.
• There are systems of choices for all these aspects of meaning.
• These systems stipulate the available meanings as well as the ways in which they translate into visible or audible form.
• As shown in Figures 1.1 and 1.2, systems can be represented by what, below, we will refer to as classification trees.
• Linguistic choices are driven by communicative purposes, or genres.
• Different genres are ultimately motivated by different target audiences.

So our conception of the way new media work is based on the way language works, and indeed the way that other semiotic systems, such as gestures, images and music, work, too. At the same time, many of the concepts we will use have also been explored in related fields, such as artificial intelligence, cognitive science and visualization studies, and some of the ideas we use, such as the concept of strategy, are closely related to the way these terms are used in web design and in marketing. We will acknowledge this throughout the book.

The thesaurus

Language consists both of grammar and vocabulary (‘lexis’), and hence the language of new media design can also be seen from the perspective of the vocabulary. Roget’s Thesaurus is a better metaphor than the dictionary for looking at new media design from this complementary perspective. A dictionary’s organization is not based on the meanings of words, but on their alphabetical listing, which is semantically arbitrary. Our approach to new media design, on the other hand, is entirely based on semantic motivation – not only in the structuring of the content but also in interface design, and this has correspondences with how the thesaurus is organized.

Looking up an expression in the thesaurus involves something like the following. First you have an idea in your mind, a concept or meaning, for which you cannot find quite the right expression. So you look up the word that seems closest to expressing it in the thesaurus’ alphabetical index. The first step is thus similar to looking up an expression in a dictionary. The index then refers you to a paragraph where the same expression is surrounded by other expressions related to it in meaning. One or more of these are likely to fit the original idea better than the word you first looked up. If it is still not the right one, the reference that follows it will lead you to another paragraph containing it and other, related expressions. Now you are likely to find a word that expresses your original idea better. If you are still not satisfied, there will be a reference taking you to yet another paragraph with yet other related expressions, and so on.

The semantic connections between these thesaurus paragraphs closely resemble the navigation links that connect topic areas in the content of new media products based on the semantically motivated design principles we will be proposing in this book. The paragraphs themselves are like the topics that fill the product’s screens, and the process of looking up a suitable expression for one’s idea in the thesaurus is like browsing through a semantically motivated website or encyclopaedia. At each successive stage (or click), some content related in meaning is displayed. Some parts of it are flagged, either by a letter–number combination (the thesaurus) or by underlining and similar methods (the new media), as leading to other topic areas, or semantic fields, which are related in content. The whole process in both cases is thus a movement through semantic space.

But this is not where the analogy ends. Apart from being organized in terms of paragraphs of semantically related terms, the thesaurus also has an overarching hierarchical, tree-like classification structure that starts with the most general and ends with the most specific categories. Its whole universe of knowledge is first divided into basic semantic categories: ‘abstract relations’, ‘space’, ‘material world’, ‘intellect’, ‘volition’ and ‘sentient and moral powers’. Taking the volition path through the tree as an example, volition is further divided into ‘individual volition’ and ‘inter-social volition’. And following individual volition in more detail leads to ‘volition in general’, ‘prospective volition’, ‘voluntary action’, ‘antagonism’ and ‘results of action’. This general-to-specific structure is the same as that used by websites whose design successfully directs users to more and more specific topics of interest.

Non-linear models

The general-to-specific tree structure, like the sets of choices used to define processes and entities in systemic-functional grammar, is based on one of the non-linear models for organizing information we will present in the following chapters, namely the classification tree. It is a common model, but there are others as well. We will describe the Given–New and Ideal–Real models, which are both based on the principle of polarizing, or contrasting, different kinds of information. We will also discuss the star structure, based on the semantic principle of centre and periphery, with a central, or nuclear, item of information surrounded by other items that all depend on it in some way. We will discuss the table, or matrix, structured around the semantic principle of comparison, where information items are compared in terms of some attributes. And finally, we will describe the network model, which consists of nodes and relations of many different kinds.

The non-linear ways of organizing information we have identified are truly semantic in nature. They have to be visualized somehow to make them concrete, but we do not draw tree, star, network and other kinds of diagrams unless they have particular semantic relations behind them. This is a major distinguishing point between our approach and the way information is structured in non-linear storyboards or ‘wire frames’ and, based on them, non-linear products in the new media industry. There, the division of content between screens is, as a rule, only intuitive, and the links are simply devices to indicate how the screens should physically be connected in the actual electronic product. The storyboard diagrams are referred to as ‘trees’ in industry parlance, and use the same form to represent a whole variety of semantic relations which, if analysed by following our method, are revealed behind them. The same is true of the products that are based on the storyboards.

It is, for example, common for websites to have a ‘tree’ navigation on the left of the web page, but often this ‘tree’ is not a classification tree, because it is not based on the semantic principle of classification from general to specific. It is not even a componential tree, which is a variety of the same overarching principle of inclusion, in which the semantic relations between nodes are of the ‘part–whole’ kind, but one in which the ‘part–whole’ tree structure is based on the physical principle of inclusion, following the structure of a book’s table of contents, where chapters are part of the book, sections are part of chapters, etc.

Take the example of George Lakoff’s book Women, Fire and Dangerous Things: What Categories Reveal about the Mind (1990). It has a table of contents with four levels. At the highest level is the whole table of contents. The second level is formed by ‘Acknowledgements’, ‘Preface’, ‘Book 1’, ‘Book 2’, ‘Afterword’, ‘References’, ‘Name Index’ and ‘Subject Index’. At the third level, Book 1 is divided into ‘Part 1’ and ‘Part 2’, and Book 2 into ‘Introduction’, ‘Anger’, ‘Over’ and ‘There-Constructions'. Part 1 and Part 2 are then divided into chapters. In the text itself, the chapters are divided into sections, which thus form another level of the hierarchy. The tree can be represented as in Figure 1.3.

Figure 1.3 Part–whole tree based on physical inclusion in Lakoff’s Women, Fire and Dangerous Things (1990)

Like the table of contents of the Lonely Planet travel guide we will analyse in more detail in Chapter 2, only some of these relations are truly semantic part–whole relations. Due to the physical limitations of the book (an object which includes (the pages that make up) the chapters, which include (the pages that make up) the sections, etc.), the sections, chapters and the book as a whole are in a physical, progressively inclusive relationship. But the semantic relationship between the content that is in...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Contents
• List of figures
• Preface
• Acknowledgements
• Chapter 1 Introduction
• Chapter 2 Simple non-linear models
• Chapter 3 Complex non-linear models
• Chapter 4 The second translation
• Chapter 5 Generic structure
• Chapter 6 Case studies
• Chapter 7 Afterword
• References
• Index