Camera Image Quality Benchmarking

Jonathan B. Phillips, Henrik Eliasson

• Publisher: Wiley
• Year: 2017
• Print ISBN: 9781119054498
• eBook ISBN: 9781119054511
• Edition: 1
• Topic: Technology & Engineering
• Subtopic: Electrical Engineering & Telecommunications
• Index: Technology & Engineering

The essential guide to the entire process behind performing a complete characterization and benchmarking of cameras through image quality analysis

Camera Image Quality Benchmarking contains the basic information and approaches for the use of subjectively correlated image quality metrics and outlines a framework for camera benchmarking.  The authors show how to quantitatively compare image quality of cameras used for consumer photography. This book helps to fill a void in the literature by detailing the types of objective and subjective metrics that are fundamental to benchmarking still and video imaging devices. Specifically, the book provides an explanation of individual image quality attributes and how they manifest themselves to camera components and explores the key photographic still and video image quality metrics. The text also includes illustrative examples of benchmarking methods so that the practitioner can design a methodology appropriate to the photographic usage in consideration.

The authors outline the various techniques used to correlate the measurement results from the objective methods with subjective results. The text also contains a detailed description on how to set up an image quality characterization lab, with examples where the methodological benchmarking approach described has been implemented successfully. This vital resource:

• Explains in detail the entire process behind performing a complete characterization and benchmarking of cameras through image quality analysis
• Provides best practice measurement protocols and methodologies, so readers can develop and define their own camera benchmarking system to industry standards
• Includes many photographic images and diagrammatical illustrations to clearly convey image quality concepts
• Champions benchmarking approaches that value the importance of perceptually correlated image quality metrics 

Written for image scientists, engineers, or managers involved in image quality and evaluating camera performance, Camera Image Quality Benchmarking combines knowledge from many different engineering fields, correlating objective (perception-independent) image quality with subjective (perception-dependent) image quality metrics. 

Chapter 1
Introduction

Camera imaging technology has evolved from a time-consuming, multi-step chemical analog process to that of a nearly instantaneous digital process with a plethora of image sharing possibilities. Once only a single-purpose device, a camera is now most commonly part of a multifunctional device, for example, a mobile phone. As digital single lens reflex (DSLR) cameras become more sophisticated and advanced, so also mobile imaging in products such as smartphones and tablet computers continues to surge forward in technological capability. In addition, advances in image processing allow for localized automatic enhancements that were not possible in the past. New feature algorithms and the advent of computational photography, for example, sophisticated noise reduction algorithms and post-capture depth processing, continue to flood the market. This necessitates an ever expanding list of fundamental image quality metrics in order to assess and compare the state of imaging systems. There are standards available that describe image quality measurement techniques, but few if any describe how to perform a complete characterization and benchmarking of cameras that consider combined aspects of image quality. This book aims to describe a methodology for doing this for both still and video imaging applications by providing (1) a discourse and discussions on image quality and its evaluation (including practical aspects of setting up a laboratory to do so) and (2) benchmarking approaches, considerations, and example data.

To be most useful and relevant, benchmarking metrics for image quality should provide consistent, reproducible, and perceptually correlated results. Furthermore, they should also be standardized in order to be meaningful to the international community. These needs have led to initiatives such as CPIQ (Camera Phone Image Quality), originally managed by the I3A (International Image Industry Association) but now run as part of standards development within the IEEE (Institute of Electrical and Electronics Engineers). The overall goal of this specific CPIQ work is to develop an image quality rating system that can be applied to camera phones and that describes the quality delivered in a better way than just a megapixel number. In order to accomplish this, metrics that are well-correlated with the subjective experience of image quality have been developed. Other imaging standards development includes the metrics by Working Group 18 of Technical Committee 42 of the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU). Theses standards bodies have provided, and continue to develop, both objective and subjective image quality metrics. In this context, objective metrics are defined measurements for which the methodology and results are independent of human perception, while subjective metrics are defined measurements using human observers to quantify human response. In following chapters, the science behind these metrics will be described in detail and provide groundwork for exemplary benchmarking approaches.

1.1 Image Content and Image Quality

Before delving into the specifics related to objective and subjective image quality camera benchmarking, exploration of the essence of photography provides justification, motivation, and inspiration for the task. As the initial purpose for photography was to generate a permanent reproduction of a moment in time (or a series of moments in time for motion imaging), an understanding of what constitutes the quality of objects in a scene will necessitate what to measure to determine the level of image quality of that permanent reproduction. The more a photograph or video represents the elements of a physical scene, the higher the possible attainment of perceived quality can become.

The efforts to create the first permanent photograph succeeded in the mid-1820s when Nicéphore Niépce captured an image of the view from his dormer window—a commonplace scene with buildings, a tree, and some sky. The image, produced by a heliographic technique, is difficult to interpret when observing the developed chemicals in the original state on a pewter plate (see Figure 1.1). In fact, the enhancement of this “raw” image, analogous to the image processing step in a digital image rendering, produces a scene with more recognizable content (see Figure 1.2). But, even though key elements are still discernible, the image is blurry, noisy, and monochrome. The minimal sharpness and graininess of the image prevent discernment of the actual textures in the scene, leaving the basic shapes and densities as cues for object recognition. Of note is the fact that the west and east facing walls of his home, seen on the sides of the image, are simultaneously illuminated by sunlight. This is related to the fact that the exposure was eight hours in length, during which the sun's position moved across the sky and exposed opposing facades (Gernsheim and Gernsheim, 1969). Needless to say, the monochrome image is void of any chromatic information.

Figure 1.1 Image of first permanent photograph circa 1826 by N. Niépce on its original pewter plate.

Source: Courtesy of Gernsheim Collection, Harry Ransom Center, The University of Texas at Austin.

Figure 1.2 Enhanced version of first permanent photograph circa 1826 by N. Niépce.

Source: Courtesy of Gernsheim Collection, Harry Ransom Center, The University of Texas at Austin.

That we can recognize objects in the rustic, historic Niépce print is a comment on the fundamentals of perception. Simple visual cues can convey object information, lighting, and depth. For example, a series of abstract lines can be used to depict a viola as shown in Figure 1.3. However, the addition of color and shading increases the perceived realism of the musical instrument, as shown in the center image. A high quality photograph of a viola contains even more information, such as albedo and mesostructure of the object which constitute the fundamental elements of texture, as shown on the right. Imaging that aims for realism contains the fundamental, low level characteristics of color, shape, texture, depth, luminance range, and motion. Faithful reproduction of these physical properties results in an accurate, realistic image of scenes and objects. These properties will be described in general in the following sections and expanded upon in much greater detail in later chapters of the book, which define image quality attributes and their accompanying objective and subjective metrics.

Figure 1.3 Three renditions of a viola. Left: line sketch; middle: colored clip art (Papapishu, 2007); right: photograph. Each shows different aspects of object representation.

Source: Papapishu, https://openclipart.org/detail/4802/violin. CC0 1.0

1.1.1 Color

Color is the visual perception of the physical properties of an object when illuminated by light or when self-luminous. On a basic level, color can describe hues such as orange, blue, green, and yellow. We refer to objects such as yellow canaries, red apples, blue sky, and green leaves. These colors are examples of those within the visible wavelength spectrum of 380 nm to 720 nm for the human visual system (HVS). However, color is more complex than perception of primary hues: color includes the perception of lightness and brightness, which allows one to discriminate between red and light red (i.e., pink), for example, or to determine which side of a uniformly colored house is facing the sun based on the brightness of the walls. These are relative terms related to the contextualized perception of the physical properties of reflected, transmitted, or emitted light, including consideration of the most luminous object in the scene. Color perception is also impacted by the surrounding colors—even if two colors have the same hue, they can appear as different hues if surrounded by different colors. Figure 1.4 shows an example of this phenomenon called simultaneous contrast. Note in this example that the center squares are identical. However, the surrounding color changes the appearance of the squares such that they do not look like the same color despite the fact that they are measurably the same.

Figure 1.4 Example illustrating simultaneous contrast. The center squares are identical in hue, chroma, and lightness. However, they appear different when surrounded by backgrounds with different colors.

There are other aspects of the HVS that can influence our perception of...

Table of contents

• Cover
• Title Page
• Copyright
• About the Authors
• Series Preface
• Preface
• List of Abbreviations
• About the Companion Website
• Chapter 1: Introduction
• Chapter 2: Defining Image Quality
• Chapter 3: Image Quality Attributes
• Chapter 4: The Camera
• Chapter 5: Subjective Image Quality Assessment—Theory and Practice
• Chapter 6: Objective Image Quality Assessment—Theory and Practice
• Chapter 7: Perceptually Correlated Image Quality Metrics
• Chapter 8: Measurement Protocols—Building Up a Lab
• Chapter 9: The Camera Benchmarking Process
• Chapter 10: Summary and Conclusions
• Index
• End User License Agreement