Digital Video Processing for Engineers

A Foundation for Embedded Systems Design

Suhel Dhanani, Michael Parker

• Publisher: Newnes
• Year: 2012
• Print ISBN: 9780124157606
• eBook ISBN: 9780124157613
• Topic: Technology & Engineering
• Subtopic: Digital Media
• Index: Technology & Engineering

Any device or system with imaging functionality requires a digital video processing solution as part of its embedded system design. Engineers need a practical guide to technology basics and design fundamentals that enables them to deliver the video component of complex projects.This book introduces core video processing concepts and standards, and delivers practical how-to guidance for engineers embarking on digital video processing designs using FPGAs. It covers the basic topics of video processing in a pictorial, intuitive manner with minimal use of mathematics. Key outcomes and benefits of this book for users include: understanding the concepts and challenges of modern video systems; architect video systems at a system level; reference design examples to implement your own high definition video processing chain; understand implementation trade-offs in video system designs.- Video processing is a must-have skill for engineers working on products and solutions for rapidly growing markets such as video surveillance, video conferencing, medical imaging, military imaging, digital broadcast equipment, displays and countless consumer electronics applications- This book is for engineers who need to develop video systems in their designs but who do not have video processing experience. It introduces the fundamental video processing concepts and skills in enough detail to get the job done, supported by reference designs, step-by-step FPGA- examples, core standards and systems architecture maps- Written by lead engineers at Altera Corp, a top-three global developer of digital video chip (FPGA) technology

1

Video in the Modern World

Video began as a purely analog technology. Successive images were captured on film streaming through the camera. The movie was played by using flashes of light to illuminate each frame on the moving film, at rates sufficient to show continual motion. Flicker, however, was easily seen.

An improved system for early broadcast television utilized the luminance (or light intensity) information represented as an analog signal. To transmit an image, the luminance information was sent in successive horizontal scans. Sufficient horizontal scans built up a two-dimensional image. Televisions and monitors used cathode ray guns that shot a stream of electrons to excite a phosphorus-coated screen. The slowly fading phosphorus tended to eliminate flicker. The cathode gun scanned in successive rows, each row just below the previous row, guided by magnetic circuits. This happened so rapidly that images were “painted” at a rate of 25 to 30 frames per second (fps). The luminance signal was used to control the intensity of the electron stream.

A horizontal synchronization signal is used to separate the horizontal scan periods. The horizontal sync period is a short pulse at the end of each scan line. This has to be long enough to allow the electron gun to move back to the left side of the screen, in preparation for the next scan line. Similarly, the vertical synchronization signal occurs at the end of the last or bottom scan, and is used to separate each video frame. The vertical synchronization interval is much longer, and allows the electron gun to move back up from the lower right corner of the screen to the upper left corner, to begin a new frame.

Later, color information in the form of red and blue hues was added, known as chrominance information. This was superimposed on the luminance signal, so that the color system is backwards compatible to the black-and-white system.

Modern video signals are represented, stored and transmitted digitally. Digital representation has opened up all sorts of new usages of video. Digital video processing is of growing importance in a variety of markets such as video surveillance; video conferencing; medical imaging; military imaging including UAVs (Unmanned Aerial Vehicles), weapons sights and night vision; broadcast; digital cinema; industrial displays and consumer electronics. All these sectors are embarking on a massive decade-long upgrade cycle from standard definition to HD-and higher than HD-resolution video processing. In some cases the old analog video-processing equipment is being replaced by systems using digital techniques.

Figure 1.1 Video raster scan.

In many cases, industries that have not traditionally been involved in video processing must now integrate this technology into their products. Examples are rear cameras, entertainment centers, “lane departure” and “head-up” displays in automotives; video data handling in networking servers and routers; sharing and merging of video to provide situational awareness in military systems; surveillance and guidance in military and commercial airborne systems; robotic systems; facial and other features recognition (such as license plates) in security surveillance systems and myriad other applications. This trend is placing new requirements on system designers as well as implementation engineers to understand video technology. This book is designed for those individuals who need to understand basic concepts and applications, so that they can either build their own video systems, or integrate third-party video technology into their products.

Another target audience is those involved in technical marketing and sales and executives from the many industries requiring video technology. Again, the need is to understand basic concepts and applications, without being overwhelmed by the details and implementation complexity.

The market sizes of these new video applications are growing rapidly. For example, here are some publically available projections:

• ABI Research believes that the video surveillance market is poised for explosive growth, which the firm forecasts to expand from revenue of about $13.5 B in 2006 to a remarkable $46 B in 2013. Those figures include cameras, computers and storage, professional services, and hardware infrastructure: everything that goes into an end-to-end security system.

• According to Wainhouse Research, the overall endpoint market for video conferencing will grow from $1.3 B in 2007 to over $4.9 B in 2013. Videoconferencing infrastructure product revenues, including MCUs, gateways, and gatekeepers, are forecast to grow to $725 M over the same period.

• HD penetration rates: there is still a lot of work to be done to develop, store, edit and transmit HD signals within both the USA and Europe.

Digital cinema is ramping up within the next five years – 10,000 US theaters are to be upgraded in 2010–2011. Digital cinemas drive significant design activity in HD and 4K video processing.

The $16 B US medical-imaging product industry will grow six percent annually in the course of 2010 based on technological advances, aging demographics and changing health care approaches. Equipment will outpace consumables, led by CT scanners and by MRI and PET machines.

All of these trends and many more lead us to believe that there is a tremendous and growing demand for a book that demystifies video processing. Professional engineers, marketers, executives and students alike need to understand:

• What is video - in terms of colors, bits and resolutions?

• What are the typical ways that video is transported?

• What functions are typical in video processing - scaling, deinterlacing, mixing?

• What are the typical challenges involved in building video-processing designs - frame buffering, line buffering, memory bandwidth, embedded control, etc.?

• What is video compression?

• How is video modulated, encoded and transmitted?

These concepts provide a solid theoretical foundation upon which the reader can build their video knowledge. This book intends to be the first text on this subject for these engineers/students.

2

Introduction to Video Processing

Chapter Outline

2.1 Digital Video: Pixels and Resolution

2.2 Digital Video: Pixels and Bits

2.3 Digital Video: Color Spaces

2.4 Video Processing Performance

Video processing – the manipulation of video to resize, clarify or compress it – is increasingly done digitally and is rapidly becoming ubiquitous in both commercial and domestic settings.

This book looks at video in the digital form – so we will talk about pixels, color spaces, etc. We start with the assumption that video is made of pixels, that a row of pixels makes a line, and a collection of lines makes a video frame. In some chapters we will briefly discuss the older analog format but mainly in the context of displaying it on a digital display.

Since this is an introductory text, and is meant to serve as a first book that clarifies digital video concepts, digital video is explained primarily through pictures, with little mathematics.

2.1 Digital Video: Pixels and Resolution

Digital video is made of pixels – think of a pixel as a small dot on your television screen. There are many pixels in one frame of video and many frames within one second – commonly 60 fps.

When you look at your TV there are various resolutions such as standard definition (SD), high definition (HD) with 720p or high-definition with 1080p. The resolution determines how many pixels your TV shows you. Figure 2.1 shows the number of pixels for these different resolutions – as you can see the same video frame for a 1080p TV is represented by a little over two million pixels compared to only about 300,000 pixels for standard definition. No wonder HD looks so good.

Figure 2.1 Increasing number of pixels in each frame of video

It might be interesting to note that the old cathode ray tube (CRT) TVs had only half of the pixels of even SD resolution shown here – so going from a CRT TV to a new 1080p TV just gave your eyes a feast of 12 times more pixels for each video frame.

The number of pixels makes a huge difference.

Take another example – when Apple created the new ‘retina’ display on the iPhone 4 it proved extremely popular with consumers. The new iPhone 4 had a resolution of 940 × 640 pixels compared to the old iPhone 3, which had a resolution of 320 × 480. So Apple found a way to increase the number of pixels on the same size screen by a factor of four.

The number of pixels also determines the complexity of the hardware used to manipulate these pixels. Since all manipulation is in terms of bits, let’s see how pixels translate to bits.

2.2 Digital Video: Pixels and Bits

Each pixel has a unique color which is a combination of the primary colors: red, blue and green. How much of red, how much of blue and how much of green is the key. And this “how much” is described precisely by the value of the pixel. The value of the pixel is represented by bits and the more bits are available, the more accurate the representation. Bear in mind however, that bits are expensive to store, to manipulate and to transmit from one device to the other. So a happy balance must be realized.

Each pixel has a red (R), green (G) and blue (B) component. There are other ways to describe this as well, but we will look at red, blue and green first. Let’s say that you use eight bits to store the value of red, eight bits for blue and eight bits for green. With eight bits you can have 2⁸ or 256 different possible values for red, blue and green each. When this is the case, people refer to this as a color depth of eight, or an 8-bit color depth.

Some HD video will be encoded with 10-bit color ...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• 1. Video in the Modern World
• 2. Introduction to Video Processing
• 3. Sampling and Aliasing
• 4. Introduction to Digital Filtering
• 5. Video Scaling
• 6. Video Deinterlacing
• 7. Alpha Blending
• 8. Sensor Processing for Image Sensors
• 9. Video Interfaces
• 10. Video Rotation
• 11. Entropy, Predictive Coding and Quantization
• 12. Frequency Domain Representation
• 13. Image Compression Fundamentals
• 14. Video Compression Fundamentals
• 15. From MPEG to H.264 Video Compression
• 16. Video Noise and Compression Artifacts
• 17. Video Modulation and Transport
• 18. Video over IP
• 19. Segmentation and Focus
• 20. Memory Considerations When Building a Video Processing Design
• 21. Debugging FPGA-based Video Systems
• Index