Working with Multimodality

Rethinking Literacy in a Digital Age

Jennifer Rowsell

• Publisher: Routledge
• Year: 2013
• Print ISBN: 9780415676236
• eBook ISBN: 9781135097660
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

In today's digital world, we have multiple modes of meaning-making: sounds, images, hypertexts. Yet, within literacy education, even 'new' literacies, we know relatively little about how to work with and produce modally complex texts.

In Working with Multimodality, Jennifer Rowsell focuses on eight modes: words, images, sounds, movement, animation, hypertext, design and modal learning. Throughout the book each mode is illustrated by cases studies based on the author's interviews with thirty people, who have extensive experience working with a mode in their field. From a song writer to a well known ballet dancer, these people all discuss what it means to do multimodality well.

This accessible textbook brings the multiple modes together into an integrated theory of multimodality. Step-by-step, beginning with theory then exploring modes and how to work with them, before concluding with how to apply this in an investigation, each stage of working with multimodality is covered.

Working with Multimodality will help students and scholars to:
• Think about specific modes and how they function
• Consider the implications for multimodal meaning-making
• Become familiar with conventions and folk knowledge about given modes
• Apply this same knowledge to their own production of media texts in classrooms

Assuming no prior knowledge about multimodality and its properties, Working with Multimodality is designed to appeal to advanced undergraduate and postgraduate students interested in how learning and innovation is different in a digital and media age and is an essential textbook for courses in literacy, new media and multimodality within applied linguistics , education and communication studies.

1

FILM

In 1896, audiences first experienced moving-image media. Crowded into a theatre, an audience watched as a train pulled into a station. As the train inched towards the audience, people bolted out of their seats, screaming, trying to run out of the theatre in fear of their lives. Their ability to challenge viewers’ perceptions has set many films apart: Citizen Kain with its narrative structure and its visual and sound production breakthroughs; Star Wars with its special effects; Avatar with its three-dimensionality; Pan’s Labyrinth with its sheer beauty. Moving images push the boundaries of what modes can do. Take iconic films and their more subtle emotional evocations, such as sadness in the final scene of Cinema Paradiso, or humour in Some Like it Hot, or suspense in Hitchcock’s The Birds. Whether it is done through expressions, movements, images, sounds, film directors capitalize on the affordances of modes to produce moving-image media. In this chapter, I consider how three producers explore and exploit modes in their production of film.

Film incorporates several modes that are featured in the book – hence its position as the first of nine chapters on modes, material resources, and media and their production. Consider for example what Pixar has done for animation and for 3-D film productions. Aside from all of the technical strides – from texture mapping to 3-D to the sophistication of their colour saturation – since the 1990s, Pixar has elevated children’s films to mainstream, blockbuster film experiences enjoyed by both children and adults alike. With each film – Toy Story, A Bug’s Life, Monster’s Inc., Finding Nemo, The Incredibles, Cars, and the list goes on – Pixar continues to move film forward. And Pixar is just one company in a string of many production companies that have challenged and expanded audiences’ viewing experiences since the first moving-image of a train entering a station.

Good films pull audiences into their worlds through an artful combination of captivating stories and evocative effects. And, while a single chapter cannot cover all there is to know about how film makes immaterial things material, the case studies in this chapter examine how producers use film to convey how ideas, emotions, and themes materialize into visual productions.

A key concept in film production is differentiating between representation and communication. Gunther Kress claims that “representation focuses on my interest; communication focuses on the assumed interest of the recipient of the sign” (Kress, 2010: 77). When producers (of sound, of movement, of videogames) make a text, they mediate their own design agency and aesthetics with those of a certain audience. What is common to communication and representation, designer agency and audience agency, is human understandings of perceptual worlds. Bodily experience can in no way be universalized, yet there is folk wisdom within media and design industries about the kinds of modes that exude emotions. As Kress expresses it, “mode is a socially shaped and culturally given semiotic resource for making meaning. Image, writing, layout, music, gesture, speech, moving image, soundtrack, and 3D objects are examples of modes used in representation and communication” (Kress, 2010: 79).

In an article on animators and film producers, Suzie Hanna (2008) considers the role of the composer in creating animated films and identifies some historical and contemporary models of interpretation and collaboration between animation directors and music artists. Historically, a relationship between animator and composer was developed as soon as technology allowed it. Animations were seen as vehicles for the music and the performer. Over the last century, references to sound are most likely contained within the visual graphics and vice versa, suggesting that animators can perceive their own art as having musical parallels. The article refers to the work of Oskar Fischinger who saw music as an inspirational tool for his animation rather than viewing the composer as an equal creator of the work. Contemporary animators use music to reach new audiences by working closely with musicians and creating visual parallels to elements of the musical form. Animators are seen to embrace a shared vocabulary through inventive visual practices based on musical structures. Whereas in the past, animators used prerecorded music in expression, contemporary animators engage in a live discourse with the composer/sound designer.

There are also researchers who use video production in their classroom fieldwork to broaden the repertoires of modes children have access to when they engage in literacy practices. John Potter, for example, works with primary school teachers who use media and video production as a form of “curatorship” of children’s lives (2010). In an article, Potter (2010) focuses on two young girls and their video production showing how video allowed these “loners” to produce a complex film narrative about their lives.

In their book on moving-image media, Andrew Burn and David Parker (2003) examine in some detail the nuances of moving-image production. Analyzing a short video on skateboarding, Burn and Parker explore how three GCSE (General Certificate of Secondary Education) students in the UK concentrate on the function of representation. Burn and Parker detail the three young men’s “planned set of representational resources” (p. 14). The students have an idea about how they want to film, and they are aware of representational resources that they will access and exploit to do so. The shots that they film construct their identities as skateboarding heroes. The three film-makers are tacitly aware of the affordances of the kineikonic mode as a moving image mode that allows for framing, shot distance, camera angle, focus, light control, and camera movement (p. 16). Using a combination of stage movements and gestures with the spontaneity of everyday life, the young men create a complex film narrative. The skateboarders include footage of a grumpy old man whom they encounter every day and of how he disrupts the adventures and heroic feats of the three skateboarders. In the short video, the skateboarders perform an informal choreography alongside a subcultural role “complete with costume, gesture, spoken repertoires, and attitude” (p. 17). The film-makers add to the movement with sound techniques to create mood. The shot level of the documentary also creates mood and an ethos for the piece. Written language serves an intermodal purpose by labelling scenes such as “Enter Old Man.” Burn and Parker go on to analyze the film in greater detail, revealing modes accessed and how they afford particular kinds of meanings. Such research is a segue to the first case study, which takes a particular stance on how visual modes afford meanings in animated texts.

THE ANIMATOR: TOBIAS WIEGAND

Tobias Wiegand is a young executive and animator who works for Morro Images, a company he co-founded with German colleagues in Berlin in 2008. After training as a media designer at a large German publishing firm and completing a degree in film studies at “The German film school,” Wiegand transitioned into 3-D animation. Later, when he moved to Canada, Wiegand opened a Canadian arm of Morro Images, navigating both the business side of his company and the technical side of rendering 3-D films. Precise and detailed, Wiegand plans and storyboards films to a minute degree, and he has worked on such feature films as Happily N’Ever After, on children’s television, and on commercials, among other projects. During our two-hour interview, Wiegand spoke at length about technical processes, terms, and practices – such as humanizing animated figures through his animation work – and he let me watch show-reels of his animation.

At the beginning of our interview we talked about his process. Wiegand explained how he starts with a basic idea, and, from that idea, he writes a treatment: a half-page outline of the general concept of a text. This provides content and design directions, be it for a television show, commercial, or scene in a major film production. He then creates a script on a storytelling site, brainstorming possible moods and visuals for the text: “that can be images, that can be video files, that can be sketches, that can be the first test animations on another show.” During this discussion, Wiegand emphasized the importance of precision. In his words, “You’re creating something that was non-existent before based off of words because, and this is a very crucial step, there is often a disconnect between the spoken word and the seen image, so we have to be very precise with what we’re creating.” It is the subtle shift of meaning between the spoken word and the seen image that is one of the great challenges of producing moving-image texts.

Wiegand thinks in terms of being precise with colour and light in images as modes to imbue emotion. He introduced me to the notion of “mood treatments,” a concept that industry people use to create and develop the dominant emotions for an animation project. Whether a project is sombre, joyful, or somewhere in between, Wiegand seeks out modes to convey these emotions to audiences, often drawing on colour, palettes, shapes, illustrations to exude or mimic emotional content in scenes. For example, in our interview, he described a chewing gum commercial produced for a European company that wanted to project a fun, sporty feel for its product, and the ways he aimed to convey these images to consumers:

based off of the whole complex being sporty, I would take really vibrant colours, for example, if it comes to design of shapes, I would try to make them very . . . how do you say it in English . . . not aggressive, but very smooth shapes, very smooth designs. What comes to mind is the wind tunnel efficiency of an object in a wind tunnel – it needs to be smooth.

(January 2011)

Figure 1.1 is a still image from the commercial that Tobias talked about. It demonstrates the use of colours that are vibrant but, at the same time, “not aggressive.” In the commercial, colour palettes are being used, illustrations that exude emotional content in scenes. In Figure 1.1, Tobias shows how one can infuse fun and sophistication in the commercial through a softened colour scheme. Drawing on smooth shapes, bold colours, and action, one is able to project the desired image of the product to target audiences.

Another element or mode Wiegand discussed was animation and materiality and dimensionality in characters. He explained in great detail the technical processes involved in creating 3-D animation. For instance, Wiegand spent much of our interview describing how he animates characters through a labour-intensive animation process:

Every bone has a certain weight that is influencing the structure of the surface of that object. So, you can imagine, let’s say, I have the upper arm bone in here and the lower arm bone, if I told this bone to influence the skin of my upper arm by moving this, that would look very odd because I moved the lower bone and something would happen on my skin up here.

(January 2011)

Figure 1.1 Use of colour palette in a commercial

Through meticulous work focusing on the weight of bones, the authenticity of movements, the surface of objects, Wiegand brings this expertise into his productions of moving-image media.

The mode of animation and its attendant technical skills is important when thinking about moving-image texts created for children and how they function. During our conversation, Wiegand talked about attention to details and technical abilities, as well as his enthusiasm for the development of 3-D animation. All of the work behind the scenes to animate and to bring 3-D characters to life is central to animation as a mode of representation: “Everything we do is not really magic; we just really look at how nature works, and then we reproduce that in the virtual world.” To illustrate how far we have come, Wiegand talked about Harry Potter and the Deathly Hallows, and the realism of Dobby:

I have never seen a virtual character created that is that good, not even Gollum, that is as good as the elf creatures . . . the house elf, and Dobby, the way they have been designed. To my mind, we’re there. It’s done. We can now create human beings and no one will be able to say . . . we can create beings where people say, “Yeah, I have a house elf just like Dobby.” It’s so amazing . . .

(January 2011)

Wiegand explained the 3-D animation process with an emphasis on the affordances of particular modes that accentuate characterization and emotional properties. As with other producers in the book, such as Mendy, Benger, and Robins, light is a prominent mode that Wiegand considers a part of his production work:

half of the animation isn’t done because you put the character into the set with the props, and then you have to light the set as well, which is essential, because lighting, as you most likely know from television productions, is a very, very important creative element to make what you’re doing very interesting, and this is the same in the 3-D environment. So, we create the lighting and we set up the lights, and then we take the whole thing and we “render” it.

(January 2011)

From the excerpt, lighting was underscored as a central mode Wiegand uses in his work. Lighting strongly impacts characterization and setting. Once Wiegand enlivens characters by putting flesh on the bones, he then considers how light and lighting play a role in the film. The processes of establishing the light, setting up the props, selecting the right camera, and projecting a character’s gaze are fundamental to the emotional content of the scene. In speaking with a professional like Wiegand, I developed a strong sense of technical phases and the role of real environments as opposed to virtual environments. Placing physical objects into a physical space like a green room and enlivening them on film is very different from working in a virtual environment, because it is about humanizing characters, putting flesh on wired figures and thinking about skin pliability.

Film producers like Wiegand work fluidly in these material and immaterial worlds, and these kinds of thinking processes and practices need to filter into everyday thinking about production. Though Wiegand spoke of emotions and moods, they were a means to an end, which is in contrast to the next case study of a producer who uses emotion and mood as the tour de force of his films.

THE PROVOCATEUR: ROBIN BENGER

A South African who settled in Toronto, Robin Benger is a renegade film-maker who abides by the credo that you “use evocative details” to tell a story, “to nudge the world in the right direction.” Having made documentaries for decades on a range of topics, Benger is particularly animated about materializing ideas, emotions, and thoughts in his documentaries. Benger was generous and articulate when I interviewed him for another research study (Sheridan and Rowsell, 2010), so I decided to meet with him again to talk about his documentary film-making and how his experience and expertise contributes to an understanding of modal learning. Relying on his intuitions to figure out the best way to tell a story, he looks for cinematic truths that speak to his senses and uses modes to materialize those ideas for his audience.

Benger finds topics and then identifies the best professionals in the industry to draw out the emotion. In this instance, he is talking about a forensic photographer he knows who would capture precisely the kind of minute detail that he needs to produce a documentary about the Waterberg, which is a large valley in South Africa that Benger frequented in his youth. There are mental pictures in the interview excerpt – you can see the cave drawings that look like Modigliani paintings and the wildlife, the sunlight – and in this way, Benger visualizes worlds for his viewers. His nuanced understanding of the material, physical, concrete scenes, of objects, and camera angles makes his films strongly connected to perceptions and senses. Benger knew from the outset how he would express his dominant mode – the singular light in South Africa – and what practices he would use to materialize such light – a forensic visualization of insects and wildlife. As it turned out, the person who could best materialize the South Africa of his youth was a French-Canadian forensic veterinarian photographer:

he [the photographer] is such a great shooter, and he could shoot insects or I went on a long walk one morning and saw a fish eagle on a black tree, like a Scotch tree, above a beautiful little dam with geese and ducks that my bird watching, my Robert’s book on birds that I am fond of; so I just stood, you know how they move their head 358 degrees, and I know Maurice could just shoot that and that would be an image. But the other thing I need for that is that I need a modern day guy in that community and it has to be somebody who would do something really weird and interesting.

(October 2010)

To materialize physical beauty or an emotion that could be rendered immaterial in a medium such as film, Benger identifies the optic (i.e. detailed close-ups of insects and animals by an attuned eye) and the means of expression (i.e. sun, lightness, insects, etc.).

During our interview, we talked about several projects including a project about gender injustices within Islamic fundamentalism. On the topic of gender apartheid, Benger is particularly sensitive about the topic of how women are treated in places like Afghanistan and Iran. He talked about two documentaries that he produced on the topic of gender apartheid. One documentary, Daughters of Afghanistan, tells the story of five women as telling case ...

Table of contents

• Front Cover
• Half Title
• Title Page
• Copyright
• Contents
• LIST OF ILLUSTRATIONS
• ACKNOWLEDGEMENTS
• Introduction
• 1 Film
• 2 Sound
• 3 Visual
• 4 Interface
• 5 Videogames
• 6 Space
• 7 Movement
• 8 Word
• 9 Textile
• Conclusion
• References
• Index