3D Video

From Capture to Diffusion

Laurent Lucas, Céline Loscos, Yannick Rémion, Laurent Lucas, Yannick Rémion, Céline Loscos, Laurent Lucas, Yannick Rémion, Céline Loscos

• Publisher: Wiley-ISTE
• Year: 2013
• Print ISBN: 9781848215078
• eBook ISBN: 9781118761861
• Edition: 1
• Topic: Tecnologia e ingegneria
• Subtopic: Elaborazione di segnali

While 3D vision has existed for many years, the use of 3D cameras and video-based modeling by the film industry has induced an explosion of interest for 3D acquisition technology, 3D content and 3D displays. As such, 3D video has become one of the new technology trends of this century.

The chapters in this book cover a large spectrum of areas connected to 3D video, which are presented both theoretically and technologically, while taking into account both physiological and perceptual aspects. Stepping away from traditional 3D vision, the authors, all currently involved in these areas, provide the necessary elements for understanding the underlying computer-based science of these technologies. They consider applications and perspectives previously unexplored due to technological limitations.

This book guides the reader through the production process of 3D videos; from acquisition, through data treatment and representation, to 3D diffusion. Several types of camera systems are considered (multiscopic or multiview) which lead to different acquisition, modeling and storage-rendering solutions. The application of these systems is also discussed to illustrate varying performance benefits, making this book suitable for students, academics, and also those involved in the film industry.

PART 1

3D Acquisition of Scenes

Chapter 1

Foundation

1.1. Introduction

Audiovisual production has, for a number of decades, used an increasing number of ever more sophisticated technologies to play 3D and 4D real and virtual content in long takes. Grouped under the term “3D video”, these technologies (motion capture (Mocap), augmented reality (AR) and free viewpoint TV (FTV) and 3DTV) complement one another and are jointly incorporated into modern productions. It is now common practice to propose AR scenes in FTV or 3DTV, either virtual or real, whether this relates to actors, sets or extras, giving virtual characters (both actors and extras) realistic movements and expressions obtained by Mocap, and even credible behavior managed by artificial intelligence.

With the success of films such as The Matrix in 1999 and Avatar in 2009 (see Figure 1.1), the acronym “3D” has become a major marketing tool for large audiovisual producers. The first, The Matrix, popularized a multiview sensor system containing 120 still cameras and two video cameras allowing slow motion virtual traveling, an effect known today as bullet time. This system has since been subject to various improvements which today not only allow the reproduction of this type of effect (FTV), but also for complete or parts of 3D reconstructions of scene content. The success of Avatar marked the renaissance of 3D cinema, a prelude to 3DTV even if it is not yet possible to free viewers from wearing 3D glasses. Glasses-free, or “autostereoscopic”, 3D display is undeniably advantageous in comparison to glasses-oriented technology due to its convincing immersive 3D vision, non-invasiveness and only slightly higher production costs in relation to 2D screens. Unfortunately, the need of multiple viewpoints (generally between five and nine) to yield immersion involves a spatial mix of these multiple images which limits their individual resolution. As a result, in contrast to stereoscopy with glasses, autostereoscopic visualization is not yet available in full HD. The induced loss of detail in relation to this current standard further limits its use. The principle challenge of autostereoscopy currently concerns the conversion of the overall dedicated tool chain into full HD.

Figure 1.1. Multiview system used to film The Matrix©Warner Bros. Entertainment Inc. a): 120 still cameras and two video cameras enabling time slicing (bullet time effect); b): stereoscopic filming; c): omnidirectional 3D capture for Avatar©20th Century Fox by James Cameron

This profusion of technologies, a veritable 3D race, is probably the result of the rapid banalizing of effects presented to the public, despite the fact that the technologies used have not yet been fully perfected. This race therefore evidently raises further challenges. All these techniques have a point in common. They rely on multiview capture of real scenes and more or less complex processing of the resulting recorded media. They also raise a series of problems relating to the volume of data, at each stage of the media chain: capture, coding [ALA 07], storage and transmission [SMO 07], concluding with its display. It is therefore essential to be able to synthesize the characteristics of this data as systems which mark their use in order to consolidate the bases of this technological explosion.

It is this point, which is the central proposal of this book, which examines two interrelated fields of this technological domain, as summarized by Kubota et al. [KUB 07]:

– 3D video technologies which aim to reconstruct varying scene characteristics (geometry, lighting and movement) for various uses;

– 3DTV/FTV technologies which focus on displaying in 3D, sometimes interactively; 3D scenes with less precise reconstruction requirements but which raise more clearly the challenges of transmitting and coding 3D or multiview medias.

The aim of this chapter is to introduce the fundamental principles of 3D videos and the techniques involved in this. In the following section, we will examine an overview of the different periods of history which have marked the development and formalization of 3D. Notably, we will detail the geometric principles related to central projection (pinhole cameras) without extending these developments to stereovision, the principles of epipolar geometry [HAR 04] exposed in Chapters 3, 4 and 5. We will then examine aspects relating to the physiology of human vision before concluding, with a more taxonomic perspective, by proposing a classification of 3D visual approaches.

1.2. A short history

The term “3D images” is the name given to what was known as “perspective” during the Renaissance period. While new developments concerning 3D arose during this period, with the appearance of the first 3D drawing machine (see Figure 1.2), consciousness of this sensation, as was its corollary–3D perception is far more ancient and founded during Antiquity.

Figure 1.2. a): the Dürer perspectograph; b): the ideal city (1475) from Piero della Francesca, c): Brunelleschi experiment

In this section, we present a brief overview of different periods which saw the development and theorization of 3D and its extension to stereoscopy using binocular vision. These two aspects mentioned in the following sections are independent of one another for practical reasons, as they need to be examined from a more global perspective, defining our relation to imaging.

1.2.1. The pinhole model

The pinhole camera, or camera obscura, was the precursor to the modern- day camera. It is composed of a dark room with a narrow hole, from which its name is derived, by which exterior lit objects are projected, in reverse, onto the opposite internal side of the dark room.

This principle was first described by the Mohists, a pacifist Chinese sect, in a collective work [MOH 00] written around 400 B.C. under the pseudonym Mo Zi. Aristotle also referred to it in the 4th Century B.C. [ARI 36]. Its first mathematical formulation was proposed by the Persian mathematician Alhazen (Ibn Al-Haytham) [ALH 21], one of the founders of optics, notably for his descriptions of vision. In 1515, Leonardo da Vinci detailed the principle and noted that, to produce a clear image, the hole must not exceed 0.5 mm in diameter [VIN 19]. In 1556, his Italian friend Girolamo Cardano placed a convex glass lens in front of the hole which provided images with hitherto unseen clarity [CAR 56]. This added the photographic lens to his long list of scientific and technical contributions¹.

1.2.1.1. A modern-day form of expression

As a result, the pinhole camera is, first and foremost, a simple yet antiquated imaging device. Its principle of central projection on a plane is illustrated in Figure 1.3 that shows the object/image inversion resulting from the central downward-projection through the hole.

Figure 1.3. A pinhole camera (camera obscura): a): illustration from The Encyclopedia of Diderot & d’Alembert; b): geometric model of the central projection involved

The geometric optical model of this device is shown in Figure 1.3. The center of projection O is the hole, located at a distance of fc from the back of the darkroom to which the optical axis is orthogonal while passing through O. It is usual to define a “viewer” orthonormal reference frame (O, x, y, z), with z being orthogonal to the back plane of the darkroom and directed, like the implicit viewer, toward the outside of the room: x, for example, is “horizontal”, directed toward the right of the presumed viewer and

This model gives the relation

which explains the observed inversion and characterizes the projection equation in (O, x, y, z) in Cartesian [1.1] as well as homogenous [1.2] coordinates:

[1.1]

[1.2]

1.2.1.2. From the pinhole to the camera

The pinhole camera, a relatively simple design, is occasionally used today despite several disadvantages that led to the common use of its success...

Table of contents

• Cover
• Contents
• Title Page
• Copyright
• Foreword
• Notations
• Acknowledgments
• Introduction
• PART 1. 3D ACQUISITION OF SCENES
• Chapter 6: Feature Points Detection and Image Matching
• PART 2. DESCRIPTION/RECONSTRUCTION OF 3D SCENES
• PART 3. STANDARDS AND COMPRESSION OF 3D VIDEO
• PART 4. RENDERING AND 3D DISPLAY
• PART 5. IMPLEMENTATION AND OUTLETS
• Conclusion
• List of Authors
• Index