Digital Image Sequence Processing, Compression, and Analysis

Todd R. Reed

• Publisher: CRC Press
• Year: 2004
• Print ISBN: 9780849315268
• eBook ISBN: 9781135504854
• Edition: 1
• Topic: Informatica
• Subtopic: Ingegneria informatica

Digital image sequences (including digital video) are increasingly common and important components in technical applications ranging from medical imaging and multimedia communications to autonomous vehicle navigation. The immense popularity of DVD video and the introduction of digital television make digital video ubiquitous in the consumer domain.

chapter 1
Introduction

Todd R. Reed

The use of image sequences to depict motion dates back nearly two centuries. One of the earlier approaches to motion picture “display” was invented in 1834 by the mathematician William George Horner. Originally called the Daedaleum (after Daedalus, who was supposed to have made figures of men that seemed to move), it was later called the Zoetrope (literally “life turning”) or the Wheel of Life. The Daedaleum works by presenting a series of images, one at a time, through slits in a circular drum as the drum is rotated.

Although this device is very simple, it illustrates some important concepts that also underlie modern image sequence displays:

1. The impression of motion is illusory. It is the result of a property of the visual system referred to as persistence of vision. An image is perceived to remain for a period of time after it has been removed from view. This illusion is the basis of all motion picture displays.
2. When the drum is rotated slowly, the images appear (as they are) a disjoint sequence of still images. As the speed of rotation increases and the images are displayed at a higher rate, a point is reached at which motion is perceived, even though the images appear to flicker.
3. Further increasing the speed of rotation, we reach a point at which flicker is no longer perceived (referred to as the critical fusion frequency).
4. Finally, the slits in the drum illustrate a vital aspect of this illusion. In order to perceive motion from a sequence of images, the stimulus the individual images represent must be removed for a period of time between each presentation. If not, the sequence of images simply merges into a blur. No motion is perceived.

The attempt to display image sequences substantially predates the ability to acquire them photographically. The first attempt to acquire a sequence of photographs from an object in motion is reputed to have been inspired by a wager of Leland Stanford circa 1872. The wager involved whether or not, at any time in its gait, a trotting horse has all four feet off the ground.

The apparatus that eventually resulted, built on Stanford’s estate in Palo Alto by Eadweard Muybridge, consisted of a linear array of cameras whose shutters are tripped in sequence as the subject passes each camera. This device was used in 1878 to capture the first photographically recorded (unposed) sequence. This is also the earliest known example of image sequence analysis.

Although effective, Muybridge’s apparatus was not very portable. The first portable motion picture camera was designed by E. J. Marey in 1882. His “photographic gun” used dry plate technology to capture a series of 12 images in 1 second on a single disk. In that same year, Marey modified Muybridge’s multicamera approach to use a single camera, repeatedly exposing a plate via a rotating disk shutter. This device was used for motion studies, utilizing white markers attached to key locations on a subject’s anatomy (the hands, joints, feet, etc.). This basic approach is widely used today for motion capture in animation.

Although of substantial technical and scientific interest, motion pictures had little commercial promise until the invention of film by Hannibal Goodwin in 1887, and in 1889 by Henry W. Reichenbach for Eastman. This flexible transparent substrate provided both a convenient carrier for the photographic emulsion and a means for viewing (or projecting) the sequence. A great deal of activity ensued, including work sponsored by Thomas Edison and conducted by his assistant, W. K. L. Dickson.

By 1895, a camera/projector system embodying key aspects of current film standards (35-mm width, 24-frame-per-second frame rate) was developed by Louis Lumiére. This device was named the Cinematographe (hence the cinéma).

The standardization of analog video in the early 1950s (NTSC) and late 1960s (SECAM and PAL) made motion pictures ubiquitous, with televisions appearing in virtually every home in developed countries. Although these systems were used primarily for entertainment purposes, systems for technical applications such as motion analysis continued to be developed. Although not commercially successful, early attempts at video communication systems (e.g., by AT&T) also appeared during this time.

The advent of digital video standards in the 1990s (H.261, MPEG, and those that followed), together with extremely inexpensive computing and display platforms, has resulted in explosive growth in conventional (entertainment) applications, in video communications, and in evolving areas such as video interpretation and understanding.

In this book, we seek both to establish the current state of the art in the utilization of digital image sequences and to indicate promising future directions for this field.

The choice of representation used in a video-processing, compression, or analysis task is fundamental. The proper representation makes features of interest apparent, significantly facilitating operations that follow. An inappropriate representation obscures such features, adding significantly to complexity (both conceptual and computational). In “Content-Based Image Sequence Representation” by Aguiar, Jasinschi, Moura, and Pluempitiwir-iyawej, video representations based on semantic content are examined. These representations promise to be very powerful, enabling model-based and object-based techniques in numerous applications. Examples include video compression, video editing, video indexing, and scene understanding.

Motion analysis has been a primary motivation from the earliest days of image sequence acquisition. More than 125 years later, the development of motion analysis techniques remains a vibrant research area. Numerous schools of thought can be identified. One useful classification is based on the domain in which the analysis is conducted.

In “The Computation of Motion” by Stiller, Kammel, Horn, and Dang, a survey and comparison of methods that could be classified as spatial domain techniques are presented. These methods can be further categorized as gradient-based, intensity-matching, and feature-matching algorithms. The relative strengths of some of these approaches are illustrated in representative real-world applications.

An alternative class of motion analysis techniques has been developed in the frequency (e.g., Fourier) domain. In addition to being analytically intriguing, these methods correlate well with visual motion perception models. They also have practical advantages, such as robustness in the presence of noise. In “Motion Analysis and Displacement Estimation in the Frequency Domain” by Lucchese and Cortelazzo, methods of this type are examined for planar rigid motion, planar affine motion, planar roto-translational displacements, and planar affine displacements.

Although there remain technical issues surrounding wireless video communications, economic considerations are of increasing importance. Quality of service assurance is a critical component in the cost-effective deployment of these systems. Customers should be guaranteed the quality of service for which they pay. In “Quality of Service Assessment in New Generation Wireless Video Communications,” Giunta presents a discussion of quality-of-service assessment methods for Third Generation (3G) wireless video communications. A novel technique based on embedded video watermarks is introduced.

Wireless communications channels are extremely error-prone. While error-correcting codes can be used, they impose computational overhead on the sender and receiver and introduce redundancy into the transmitted bitstream. However, in applications such as consumer-grade video communications, error-free transmission of all video data may be unnecessary if the errors can be made unobtrusive. “Error Concealment in Digital Video” by De Natale provides a survey and critical analysis of current techniques for obscuring transmission errors in digital video.

With the increase in applications for digital media, the demand for content far exceeds production capabilities. This makes archived material, particularly motion picture film archives, increasingly valuable. Unfortunately, film is a very unstable means of archiving images, subject to a variety of modes of degradation. The artifacts encountered in archived film, and algorithms for correcting these artifacts, are discussed in “Image Sequence Restoration: A Wider Perspective” by Kokaram.

As digital video archives continue to grow, accessing these archives in an efficient manner has become a critical issue. Concise condensations of video material provide an effective means for browsing archives and may also be useful for promoting the use of particular material. Approaches to generating concise representations of video are examined in “Video Summarization” by Taskiran and Delp.

Technological developments in video display have advanced very rapidly, to the point that affordable high-definition displays are widely available. High definition program material, although produced at a growing rate, has not kept pace. Furthermore, archival video may be available only at a fixed (relatively low) resolution. In the final chapter of this book, “High-Resolution Images from a Sequence of Low-Resolution Observations,” Alvarez, Molina, and Katsaggelos examine approaches to producing high-definition material from a low-definition source.

Bibliography

Gerald Mast. A Short History of Movies. The Bobbs-Merrill Company, Inc., New York, 1971.

Kenneth Macgowan. Behind the Screen – The History and Techniques of the Motion Picture. Delacorte Press, New York, 1965.

C.W. Ceram. Archaeology of the Cinema. Harcourt, Brace & World, Inc., New York, 1965.

John Wyver. The Moving Image – An International History of Film, Television, and Video. BFI Publishing, London, 1989.

chapter 2
Content-based image sequence representation

Pedro M. Q. Aguiar, Radu S. Jasinschi, José M. F. Moura, and Charnchai Pluempitiwiriyawej¹

Contents

2.1 Introduction

2.1.1 Mosaics for static 3-D scenes and large depth: single layer

2.1.2 Mosaics for static 3-D scenes and variable depth: multiple layers

2.1.3 Video representations with fully 3-D models

2.1.3.1 Structure from motion: factorization

2.2 Image segmentation

2.2.1 Calculus of variations

2.2.1.1 Adding constraints

2.2.1.2 Gradient descent flow

2.2.2 Overview of image segmentation methods

2.2.2.1 Edge-based approach

2.2.2.2 Region-based approach

2.2.3 Active contour methods

2.2.4 Parametric active contour

2.2.4.1 Variations of classical snakes

2.2.5 Curve evolution theory

2.2.6 Level set method

2.2.7 Geometric active contours

2.2.8 STACS: Stochastic active contour scheme

2.3 Mosaics: From 2-D to 3-D

2.3.1 Generative video

2.3.1.1 Figure and background mosaics generation

2.3.2 3-D Based mosaics

2.3.2.1 Structure from motion: generalized eight-point algorithm

2.3.2.2 Layered mosaics based on 3-D information

2.3.2.3 3-D mosaics

2.3.2.4 Summary

2.4 Three-dimensional object-based representation

2.4.1 3-D object modeling from video

2.4.1.1 Surface-based rank 1 factorization method

2.4.2 Framework

2.4.2.1 Image sequence representation

2.4.2.2 3-D motion representation

2.4.2.3 3-D shape representation

2.4.3 Video analysis

2.4.3.1 Image motion

2.4.3.2 3-D structure from 2-D motion

2.4.3.3 Translation estimation

2.4.3.4 Matrix of 2-D motion parameters

2.4.3.5 Rank 1 factorization

2.4.3.6 Decomposition stage

2.4.3.7 Normalization stage

2.4.3.8 Texture recovery

2.4.4 Video synthesis

2.4.5 Experiment

2.4.6 Applications

2.4.6.1 Video coding

2.4.6.2 Video content addressing

2.4.6.3 Virtualized reality

2.4.7 Summary

2.5 Conclusion

References

Abstract. In this chapter we overview methods that represent video sequences in terms of their content. These methods differ from those developed for MPEG/H.26X coding standards in that sequences are described in terms of extended images instead of collections of frames. We describe how these extended images, e.g., mosaics, are generated by basically the same principle: the incremental compositi...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Preface
• About the Editor
• Contributors
• Chapter 1: Introduction
• Chapter 2: Content-based image sequence representation
• Chapter 3: The computation of motion
• Chapter 4: Motion analysis and displacement estimation in the frequency domain
• Chapter 5: Quality of service assessment in new generation wireless video communications
• Chapter 6: Error concealment in digital video
• Chapter 7: Image sequence restoration: A wider perspective
• Chapter 8: Video summarization
• Chapter 9: High-resolution images from a sequence of low-resolution observations