A Practical Introduction to Computer Vision with OpenCV

Kenneth Dawson-Howe

• Publisher: Wiley
• Year: 2014
• ISBN:
  9781118848739
• Edition: 1
• Topic: Informatik
• Subtopic: Computer Vision & Mustererkennung

Explains the theory behind basic computer vision and provides a bridge from the theory to practical implementation using the industry standard OpenCV libraries

Computer Vision is a rapidly expanding area and it is becoming progressively easier for developers to make use of this field due to the ready availability of high quality libraries (such as OpenCV 2). This text is intended to facilitate the practical use of computer vision with the goal being to bridge the gap between the theory and the practical implementation of computer vision. The book will explain how to use the relevant OpenCV library routines and will be accompanied by a full working program including the code snippets from the text. This textbook is a heavily illustrated, practical introduction to an exciting field, the applications of which are becoming almost ubiquitous. We are now surrounded by cameras, for example cameras on computers & tablets/ cameras built into our mobile phones/ cameras in games consoles; cameras imaging difficult modalities (such as ultrasound, X-ray, MRI) in hospitals, and surveillance cameras. This book is concerned with helping the next generation of computer developers to make use of all these images in order to develop systems which are more intuitive and interact with us in more intelligent ways.

• Explains the theory behind basic computer vision and provides a bridge from the theory to practical implementation using the industry standard OpenCV libraries
• Offers an introduction to computer vision, with enough theory to make clear how the various algorithms work but with an emphasis on practical programming issues
• Provides enough material for a one semester course in computer vision at senior undergraduate and Masters levels
• Includes the basics of cameras and images and image processing to remove noise, before moving on to topics such as image histogramming; binary imaging; video processing to detect and model moving objects; geometric operations & camera models; edge detection; features detection; recognition in images
• Contains a large number of vision application problems to provide students with the opportunity to solve real problems. Images or videos for these problems are provided in the resources associated with this book which include an enhanced eBook

1
Introduction

Computer vision is the automatic analysis of images and videos by computers in order to gain some understanding of the world. Computer vision is inspired by the capabilities of the human vision system and, when initially addressed in the 1960s and 1970s, it was thought to be a relatively straightforward problem to solve. However, the reason we think/thought that vision is easy is that we have our own visual system which makes the task seem intuitive to our conscious minds. In fact, the human visual system is very complex and even the estimates of how much of the brain is involved with visual processing vary from 25% up to more than 50%.

1.1 A Difficult Problem

The first challenge facing anyone studying this subject is to convince themself that the problem is difficult. To try to illustrate the difficulty, we first show three different versions of the same image in Figure 1.1. For a computer, an image is just an array of values, such as the array shown in the left-hand image in Figure 1.1. For us, using our complex vision system, we can perceive this as a face image but only if we are shown it as a grey scale image (top right).

Figure 1.1 Different versions of an image. An array of numbers (left) which are the values of the grey scales in the low resolution image of a face (top right). The task of computer vision is most like understanding the array of numbers

Computer vision is quite like understanding the array of values shown in Figure 1.1, but is more complicated as the array is really much bigger (e.g. to be equivalent to the human eye a camera would need around 127 million elements), and more complex (i.e. with each point represented by three values in order to encode colour information). To make the task even more convoluted, the images are constantly changing, providing a stream of 50–60 images per second and, of course, there are two streams of data as we have two eyes/cameras.

Another illustration of the difficulty of vision was provided by psychologist John Wilding considering his own visual system:

As I look out of my window, I see grass and trees, gently swaying in the wind, with a lake beyond … An asphalt path leads down through the trees to the lake and two squirrels are chasing each other to and fro across it, ignoring the woman coming up the path …

This is the scene I experience, a world of objects with background, acted upon and sometimes acting and interacting in events. I have no problem seeing and hearing and smelling and feeling all these things because they affect my senses directly and they make up the real world.

Or do they? I can look again and notice things I missed before, or see the scene in new ways. There is a white wall framing the window I am looking through and the window in fact fills less of my field of view that the wall, but I did not even notice the wall at first, and my impression was that the scene through the window was a panorama right across in front of me. There are metal bars dividing the window into squares and the glass is obscured with dust and spots but for me the view seems complete and un-obscured. The ‘grass’ is patches of colour ranging from nearly white in the bright sun to nearly black in the shade but I ‘saw’ green grass in light and shade. Other changing greenish shapes were for me permanent leafy branches moved by a wind I neither saw nor felt, and two constantly varying grey shapes were squirrels moving with a purpose. Another shape increasing in size and changing in position was an approaching woman. (Wilding, 1983)

1.2 The Human Vision System

If we could duplicate the human visual system then the problem of developing a computer vision system would be solved. So why can’t we? The main difficulty is that we do not understand what the human vision system is doing most of the time.

If you consider your eyes, it is probably not clear to you that your colour vision (provided by the 6–7 million cones in the eye) is concentrated in the centre of the visual field of the eye (known as the macula). The rest of your retina is made up of around 120 million rods (cells that are sensitive to visible light of any wavelength/colour). In addition, each eye has a rather large blind spot where the optic nerve attaches to the retina. Somehow, we think we see a continuous image (i.e. no blind spot) with colour everywhere, but even at this lowest level of processing it is unclear as to how this impression occurs within the brain.

The visual cortex (at the back of the brain) has been studied and found to contain cells that perform a type of edge detection (see Chapter 6), but mostly we know what sections of the brain do based on localised brain damage to individuals. For example, a number of people with damage to a particular section of the brain can no longer recognise faces (a condition known as prosopagnosia). Other people have lost the ability to sense moving objects (a condition known as akinetopsia). These conditions inspire us to develop separate modules to recognise faces (e.g. see Section 8.4) and to detect object motion (e.g. see Chapter 9).

We can also look at the brain using functional MRI, which allows us to see the concentration of electrical activity in different parts of the brain as subjects perform various activities. Again, this may tell us what large parts of the brain are doing, but it cannot provide us with algorithms to solve the problem of interpreting the massive arrays of numbers that video cameras provide.

1.3 Practical Applications of Computer Vision

Computer vision has many applications in industry, particularly allowing the automatic inspection of manufactured goods at any stage in the production line. For example, it has been used to:

• Inspect printed circuits boards to ensure that tracks and components are placed correctly. See Figure 1.2.
• Inspect print quality of labels. See Figure 1.3.
• Inspect bottles to ensure they are properly filled. See Figure 1.3.
• Inspect apples to determine if there is any bruising.
• Locate chocolates on a production line so that a robot arm can pick them up and place them in the correct locations in the box.
• Guide robots when manufacturing complex products such as cars.

Figure 1.2 PCB inspection of pads (left) and images of some detected flaws in the surface mounting of components (right). Reproduced by permission of James Mahon

Figure 1.3 Checking print quality of best-before dates (right), and monitoring level to which bottles are filled (right). Reproduced by permission of Omron Electronics LLC

On the factory floor, the problem is a little simpler than in the real world as the lighting can be constrained and the possible variations of what we can see are quite limited. Computer vision is now solving problems outside the factory. Computer vision applications outside the factory include:

• The automatic reading of license plates as they pass through tollgates on major roads.
• Augmenting sports broadcasts by determining distances for penalties, along with a range of other statistics (such as how far each player has travelled during the game).
• Biometric security checks in airports using images of faces and images of fingerprints. See Figure 1.4.
• Augmenting movies by the insertion of virtual objects into video sequences, so that they appear as though they belong (e.g. the candles in the Great Hall in the Harry Potter movies).
• Assisting drivers by warning them when they are drifting out of lane.
• Creating 3D models of a destroyed building from multiple old photographs.
• Advanced interfaces for computer games allowing the real time detection of players or their hand-held controllers.
• Classification of plant types and anticipated yields based on multispectral satellite images.
• D...