1
Avatars, Apes and the âTestâ of Performance Capture
Ralf Remshardt
Kafka worries and Benjamin tests
In March 1913, Franz Kafka came across a poster in the lobby of a cinema. It advertised a new feature starring the notable German stage actor Albert Bassermann in his first film appearance. To say that Kafka, an avid and regular filmgoer in his native Prague as well as a dedicated patron of the theatre, was overcome by mixed feelings, is only to say that he was Kafka. In this instance, however, Kafkaâs ambivalence was excited by a philosophical problem. Like many middle-class filmgoers in the early days of cinema, Kafka was able to appreciate and reflect on literary theatre one day and attend a popular film the next without much cognitive distress. But here was his idol Bassermann, an actor whom Kafka had admired in classical roles in Berlin, clearly crossing the boundary between two incommensurable worlds, the stage and the screen. He wrote to his fiancĂ©e, Felice:
I found myself pitying B [i.e. Bassermann] as though he were the most unfortunate of men. This is how I imagine the situation: the satisfaction of acting is over; the film is made; B. himself can no longer influence it in any way; he need not even realize that he had allowed himself to be taken advantage of, and yet, when watching himself in the film, he may become aware of the utter futility of exerting all his considerable powers and â I am not exaggerating my sense of compassion â he grows older, weak, gets pushed aside in his armchair, and vanishes somewhere into the mists of time. How wrong! This is just where the error of my judgment lies. Even after the completion of the film Bassermann goes home as Bassermann, and no one else. If at any time he should withdraw [sich aufheben], he will withdraw completely, and be gone.
(Kafka 1973: 213, his emphasis)
Kafkaâs concern with the actorâs participation in what he views as a âwretched filmâ is not merely aesthetic but is in fact profoundly ontological. Having been robbed of his âinfluenceâ on the artefact, Bassermann, in Kafkaâs imagination, âvanishesâ (though he gets a reprieve in the last passage through the quasi-Hegelian phrase aufheben, which can paradoxically mean both to disappear and to retain). The performance is not so much given as taken â an exploitation of the actor in which he may or may not be complicit, but which results, as if it were some shamanic co-optation, in the diminution of his powers. Kafka uses the Bassermann example to discuss his own fragile sense of self, but he also implicitly formulates a theory of film acting, and a kind of new media theory. Attuned as no other writer to the exactions an alienating modernist environment made on the self, Kafka had grasped the actorâs dilemma intuitively, expressing the apparent paradox that as cinema had elevated Bassermann to the status of an imaginary and had achieved his apotheosis, it had also laid waste to his âconsiderable powersâ as an actor, diminishing him in his substance. The film is Der Andere (1913, dir. Max Mack), in which the character played by Bassermann suffers from a split personality following a shock. For Kafka, film was that shock.
In this cinematic revolution of the early twentieth century, camera acting first forced performers accustomed to the regimes of the stage to reconsider their craft. The director Joseph von Sternberg wrote openly about the camera as âa complicated machine [that] extracts an essence from the actor, over which the actor has no controlâ (Sternberg 1974: 86). This idea of the alienating quality of the camera is echoed in Walter Benjaminâs notion: âThe representation of human beings by means of an apparatus has made possible a highly productive use of the human beingâs self-alienationâ (Benjamin 2008: 32). Both regard the act of camera performance as a submission to control by outside forces, but to von Sternberg, it seems vaguely sinister that film coerces from the actors a loss of autonomy while simultaneously trading on the phantasm of their autonomous image. In Benjaminâs materialist analysis, however, the actor trades the auratic presence of the stage for a manufactured, commodified âmagic of the personalityâ (33), to be subjected to the critical gaze of the masses in the marketplace of images. Masses that are no less than the cameraâs alter ego and the final arbiter and critic of the performance in which Benjamin sees a âtest performanceâ [Testleistung] (30).
In Benjaminâs analogue film world, there is a clear positionality of the actor relative to the camera, which, in spite of all of the artifice engaged to reassemble the performance, is still merely a conduit to the actorâs ultimate confrontation with a mass audience who âwill control himâ: âThose who are not visible, not present while he executes his performance, are precisely the ones who will control it. This invisibility heightens the authority of their controlâ (33). In the following, I will discuss performance capture technology, a set of data acquisition processes which might well be regarded as the Benjaminian âtest performanceâ of the present. Performance capture has the potential to renegotiate the âpeculiar nature of the artistic performance of the film actorâ which, as Benjamin observed, is âcarried out before a group of specialists ⊠who are in a position to intervene in his performance at any timeâ. In Benjaminâs view, the film actor is subject to âexaminationsâ (PrĂŒfungen) by the apparatus that are analogous to the tests faced by workers in the mechanized industrial production process. But in capitalist production, Benjamin argues, these tests are hidden, invisible. Film exceeds these strictures by making the test âcapable of being exhibited by turning that ability itself into a testâ (30, his emphasis). The key to this test is the actorâs struggle âto preserve his humanity in the face of the apparatusâ, a heroic, even triumphal act of resistance where, as a proxy for the urban masses who congregate in the cinema to witness it, he takes ârevenge on their behalfâ (31).1
The nature of the âtestâ has changed with performance capture as it has emerged from ever-more complex and integrative iterations of motion capture. Performance capture offers the possibility of creating not just the digital representation of the actor but a comprehensive spatio-temporal simulacrum of a given filmic world, a shift which has redefined the physical presence of the performer, making him or her a potentially cyborgian performer in an increasingly fluid relationship to the utilized technology, within a post-human medial construct. The apparent power of the collective response has also been diminished in the performance capture process. First, by the interposition of invisible animators who splinter even that test of the actor by the apparatus-as-proxy which Benjamin still imagines as a legibly heroic confrontation between human and machine. Secondly, by the context of individualized image consumption (streaming, etc.) that never permits the masses to assemble as an authoritative perceiving body to absorb the actorâs performing body en masse.
One might expect the introduction of performance capture, computer-generated characters and other instances of digitally based performance to stir anxieties similar to those of the early film actors, but the present response has been curiously muted. Almost a century after Kafka, von Sternberg and Benjamin, the executive director of the American film actors union SAG-AFTRA, writing in the introduction to a theme issue on digital acting, sought to reassure performers that though this is âa time of sweeping changeâ, each previous evolution in technology, beginning with silent film, âhas produced a revolution in products, platforms, audience viewing habits and, fortunately, new projects for Screen Actors Guild members. Each period brought uncertainty but, with smart responses, we adapted to ensure that it also served to expand opportunities for our membersâ (White 2010: 12). This essay aims to push beyond such anodyne proclamations (and there are many) to enquire about the way in which digitally based acting not only expands opportunities but also extends and complicates the aesthetic and ethics of mediated performance as both an artistic and commercial discipline.
Acquiring bodies
Performance capture is situated on the terrain of two intersecting technical domains: motion capture (mocap) and computer-generated (CGI) character animation. Of the many succinct distinctions between motion and performance capture on offer, Matt Delbridgeâs is perhaps most useful: performance capture, he writes, âdescribes the total recording of a performance without cuts using an Optical MoCap system. [Performance capture] allows an entire performance to be captured in one take, significantly eliminating the need for multiple takes (of a single scene) to be recorded. It allows for the exploration and capture of a whole scene to be undertaken unhindered by device limitations (like the frame and physical environments)â (2014: 222). As a set of imaging techniques that enables the recording of animal or human movement sequences in order to analyse and repurpose the data, motion capture was arguably initiated with the experiments in chronophotography pioneered by Eadweard Muybridge and Ătienne-Jules Marey in the late nineteenth century. Artists of the 1930s and after widely used rotoscoping and frame-by-frame tracing as a way to translate natural motion into animated cartoons. In 1966, in â9 evenings: theatre & engineeringâ, Billy KlĂŒver, a former Bell Laboratories engineer, used electronic motion tracking with devices attached to suits of performers to generate soundscapes (Garwood 2007: 38) and Lee Harrison III deployed electronic motion tracking that could produce a virtual puppet avatar on a television screen in real time as early as 1967 in Scanimate (Auslander 2017: 117).
Full visualization, however, was dependent on much higher processing speeds, so that digital avatars in commercial video gaming or performance art â two forms that began to cross-fertilize across multiple mutual boundaries â became viable only in the 1990s. Contemporary mocap now makes use of technologies first developed in medical science for biomechanical analysis, encasing the performers in a body suit studded with dozens, sometimes hundreds of passive (optical) or active (LED or infrared) markers which are then triangulated by several cameras to yield a data set representing the performersâ movement in space. Increasingly, markerless systems that dispense with the cumbersome body glove have come into use. Dance choreography was in the vanguard of utilizing the biomechanical capture technology for purposes of artistic exploration. Beginning especially at the end of the last millennium, when processing speeds began to catch up with the moving human form, choreographers such as Marie-Claude Poulin, Susan Kozel and others, working in conjunction with media designers, began to make work that questioned the disposition of the digitized body as much as to celebrate its technological liberation. For instance, Poulinâs Montreal-based company kondition pluriel, as Chris Salter remarks, offered a âdarker reimagining of the body integrated within a sensor-occupied universe, while blurring the line between performer and participantâ (Salter 2010: 270).
In order to progress from motion to what director James Cameron soon renamed âperformanceâ capture, a convergent but more difficult aim was the credible representation of human or humanoid physiognomy through computer graphics. The first CGI character capable of registering emotion, long a particularly elusive goal, was the lounge singer Tony de Peltrie in an eponymous short film made by four University of Montreal graduates in 1985.2 The capturing and conversion of an actorâs varied facial expressions to access and manipulate the underlying emotional lexicon became the master objective of performance capture, but it remained vexed by the persistent problem roboticist Masahiro Mori in 1970 had famously termed the âuncanny valleyâ â the perceived revulsion of audiences against almost-but-not-quite lifelike virtual humans. The response is grounded in a high collective sensibility to any image judged not sufficiently human in appearance. Humanoid rather than explicitly human characters, which have been the main products of performance capture in mainstream cinema, sidestep the uncanniness problem to some degree, though the debacle over the live-action version of Cats (2019, dir. Tom Hooper) or the studioâs quick redesign of the title character in Sonic the Hedgehog (2020, dir. Jeff Fowler) following previews showed how narrow the line between rapture and revulsion can be.3 The often vitriolic responses by the public in such instances of failure, however, also illustrate that audiences, even if they can intervene in the cinematic process only via social media posts and the like, still maintain a kind of critical-ludic relationship to the creation of digitally based films. This is similar to the âsubversiveâ attitude towards datafication and dataveillance William W. Lewis notes in his adjacent essay in this volume.
The facial capture technology with a helmet-mounted camera now in use was principally brought to fruition at WETA digital in New Zealand on Peter Jacksonâs Lord of the Rings trilogy (2001â3) to allow for the generation of actor Andy Serkisâs performance as the creature Gollum. This was refined on James Cameronâs Avatar (2009) to render the alien Naâvi characters and has yielded impressive results in the performance of the sapient chimpanzee leader Caesar (Serkis again) in the Planet of the Apes reboot (2011â17). Gollumâs surreptitious jealousy and histrionic self-pity, Caesarâs smouldering anger and anxious tenderness are complexly layered, often contradictory emotions read by the audience as a direct and true reflection of the charactersâ inner state, even though there are multiple interventions by the cinematic apparatus, each one of which can alter the meaning. As Anders Langland of WETA remarked, âEven a fraction of a pixel change in the expression, particularly around the eyes, can completely change the way you perceive the characterâ (Langland 2017). It is on this granular, pixel-sized level of interpretive intervention that some of the most pertinent aesthetic and ethical questions about performance capture as performance arise.
The Serkis problem
While there are beginnings of an academic consideration of the questions associated with digital acting,4 there are few traces of a serious debate about performance capture in the film industry or in the media outlets of popular culture. This may have to do with the fact that audiences, while generally aware of the technical efforts involved, are indifferent to their implications for performers and certainly donât sweat the ontological minutiae. â[T]he audienceâ, Antonio Pizzo remarks, âseems not to care much about the authorship questionâ (2016: 8). âPerformance captureâ lacks its own Wikipedia entry, instead redirecting to âmotion captureâ, a sign that categorical distinctions have not yet received the imprimatur of the principal arbiter of the digital age.
A search for âperformance captureâ turns up primarily articles on the ubiquitous Serkis, who has become the face (or faces) of performance capture. As Gollum in Lord of the Rings or Caesar in Planet of the Apes series, Serkis is now identified with performance capture at the same time that he is, paradoxically, largely concealed by it. The actor, who owns his own performance capture outfit in London, The Imaginarium, and has most recently directed a photorealistic CGI version of The Jungle Book using the technology, is convinced that captured performances are close to being Oscar-worthy (Foster 2017) and argues tautologically that it is âno different than any process you go through to create a role, whether youâre on a stage, or in front of a screen in a more conventional sense. The actorâs performance is the actorâs performanceâ (Rafferty 2017). Academic critics like Sharon Marie Carnicke have lent support to this notion, noting that âacting is â at base â a discrete art form which has â over the centuries variously adapted to the changing technologiesâ and that the essential processes of acting âremain relatively stableâ (Carnicke 2014: 322). She argues that a performance such as Gollum âcan be subjected to the same kind of analysis and assessment as any other kind of screen performanceâ (332).
That question remains to be explored here, but it is plausible that an actor like Serkis, who is so linked to an emergent and ill-unde...