In this chapter, we are going to learn the main techniques of face detection and recognition. Face detection is the process whereby faces are located in a whole image. In this chapter, we are going to cover different techniques to detect faces in images, from classic algorithms using cascade classifiers with Haar features to newer techniques using deep learning. Face recognition is the process of identifying a person that appears in an image. We are going to cover the following topics in this chapter:
Face recognition is the process of putting a label to a known face. Just like humans learn to recognize their family, friends, and celebrities just by seeing their face, there are many techniques for recognize a face in computer vision.
These generally involve four main steps, defined as follows:
- Face detection: This is the process of locating a face region in an image (the large rectangle near the center of the following screenshot). This step does not care who the person is, just that it is a human face.
- Face preprocessing: This is the process of adjusting the face image to look clearer and similar to other faces (a small grayscale face in the top center of the following screenshot).
- Collecting and learning faces: This is a process of saving many preprocessed faces (for each person that should be recognized), and then learning how to recognize them.
- Face recognition: This is the process that checks which of the collected people are most similar to the face in the camera (a small rectangle in the top right of the following screenshot).
Note that the phrase face recognition is often used by the general public to refer to finding the positions of faces (that is, face detection, as described in step 1), but this book will use the formal definition of face recognition referring to step 4, and face detection referring to Step 1.
The following screenshot shows the final WebcamFaceRec project, including a small rectangle at the top-right corner highlighting the recognized person. Also, notice the confidence bar that is next to the preprocessed face (a small face at the top center of the rectangle marking the face), which in this case shows roughly 70 percent confidence that it has recognized the correct person:
Current face detection techniques are quite reliable in real-world conditions, whereas current face recognition techniques are much less reliable when used in real-world conditions. For example, it is easy to find research papers showing face recognition accuracy rates above 95 percent, but when testing those same algorithms yourself, you may often find that accuracy is lower than 50 percent. This comes from the fact that current face recognition techniques are very sensitive to exact conditions in images, such as the type of lighting, direction of lighting and shadows, exact orientation of the face, expression of the face, and the current mood of the person. If they are all kept constant when training (collecting images), as well as when testing (from the camera image), then face recognition should work well, but if the person was standing to the left-hand side of the lights in a room when training, and then stood to the right-hand side while testing with the camera, it may give quite bad results. So, the dataset used for training is very important.
Face preprocessing aims to reduce these problems by making sure the face always appears to have similar brightness and contrast, and perhaps making sure the features of the face will always be in the same position (such as aligning the eyes and/or nose to certain positions). A good face preprocessing stage will help improve the reliability of the whole face recognition system, so this chapter will place some emphasis on face preprocessing methods.
Despite the big claims about using face recognition for security in the media, it is unlikely that the current face recognition methods alone are reliable enough for any true security system. However, they can be used for purposes that don't need high reliability, such as playing personalized music for different people entering a room, or a robot that says your name when it sees you. There are also various practical extensions to face recognition, such as gender recognition, age recognition, and emotion recognition.
![]()
Until the year 2000, there were many different techniques used for finding faces, but all of them were either very slow, very unreliable, or both. A major change came in 2001 when Viola and Jones invented the Haar-based cascade classifier for object detection, and in 2002 when it was improved by Lienhart and Maydt. The result is an object detector that is both fast (it can detect faces in real time on a typical desktop with a VGA webcam) and reliable (it detects approximately 95 percent of frontal faces correctly). This object detector revolutionized the field of face recognition (as well as that of robotics and computer vision in general), as it finally allowed real-time face detection and face recognition, especially as Lienhart himself wrote the object detector that comes free with OpenCV! It works not only for frontal faces but also side-view faces (referred to as profile faces), eyes, mouths, noses, company logos, and many other objects.
This object detector was extended in OpenCV v2.0 to also use LBP features for detection based on the work done by Ahonen, Hadid, and Pietikäinen in 2006, as LBP-based detectors are potentially several times faster than Haar-based detectors, and don't have the licensing issues that many Haar detectors have.
OpenCV has implemented deep learning from v3.4 and it's more stable in v4.0. In this chapter, we will show how to use Single Shot Multibox Detector (SSD) algorithm for face detection.
The basic idea of the Haar-based face detector is that if you look at most frontal faces, the region with the eyes should be darker than the forehead and cheeks, the region with the mouth should be darker than the cheeks, and so on. It typically performs about 20 stages of comparisons like this to decide whether it is a face or not, but it must do this at each possible position in the image, and for each possible size of the face, so in fact, it often does thousands of checks per image. The basic idea of the LBP-based face detector is similar to the Haar-based one, but it uses histograms of pixel intensity comparisons, such as edges, corners, and flat regions.
Rather than have a person decide which comparisons would best define a face, both Haar and LBP-based face detectors can be automatically trained to find faces from a large set of images, with the information stored as XML files to be used later. These cascade classifier detectors are typically trained using at least 1,000 unique face images and 10,000 non-face images (for example, photos of trees, cars, and text), and the training process can take a long time even on a multi-core desktop (typically a few hours for LBP, but one week for Haar!). Luckily, OpenCV comes with some pretrained Haar and LBP detectors for you to use! In fact, you can detect frontal faces, profile (side-view) faces, eyes, or noses just by loading different cascade classifier XML files into the object detector and choosing between the Haar and LBP detector, based on which XML file you choose.
![]()
As mentioned previously, OpenCV v2.4 comes with various pretrained XML detectors that you can use for different purposes. The following table lists some of the most popular XML files:
| Type of cascade classifier | XML filename |
| Face detector (default) | haarcascade_frontalface_default.xml |
| Face detector (fast Haar) | haarcascade_frontalface_alt2.xml |
| Face detector (fast LBP) | lbpcascade_frontalface.xml |
| Profile (side-looking) face detector | haarcascade_profileface.xml |
| Eye detector (separate for left and right) | haarcascade_lefteye_2splits.xml |
| Mouth detector | haarcascade_mcs_mouth.xml |
| Nose detector | haarcascade_mcs_nose.xml |
| Whole person detector | haarcascade... |
