Deep Learning in Computer Vision

Principles and Applications

Mahmoud Hassaballah, Ali Ismail Awad, Mahmoud Hassaballah, Ali Ismail Awad

• Publisher: CRC Press
• Year: 2020
• ISBN:
  9781351003803
• Edition: 1
• Topic: Computer Science
• Subtopic: Computer Science General
• Index: Computer Science

Deep learning algorithms have brought a revolution to the computer vision community by introducing non-traditional and efficient solutions to several image-related problems that had long remained unsolved or partially addressed. This book presents a collection of eleven chapters where each individual chapter explains the deep learning principles of a specific topic, introduces reviews of up-to-date techniques, and presents research findings to the computer vision community. The book covers a broad scope of topics in deep learning concepts and applications such as accelerating the convolutional neural network inference on field-programmable gate arrays, fire detection in surveillance applications, face recognition, action and activity recognition, semantic segmentation for autonomous driving, aerial imagery registration, robot vision, tumor detection, and skin lesion segmentation as well as skin melanoma classification. The content of this book has been organized such that each chapter can be read independently from the others. The book is a valuable companion for researchers, for postgraduate and possibly senior undergraduate students who are taking an advanced course in related topics, and for those who are interested in deep learning with applications in computer vision, image processing, and pattern recognition.

1 Accelerating the CNN Inference on FPGAs

Kamel Abdelouahab, Maxime Pelcat, and François Berry

Contents

1.1 Introduction

1.2 Background on CNNs and Their Computational Workload

1.2.1 General Overview

1.2.2 Inference versus Training

1.2.3 Inference, Layers, and CNN Models

1.2.4 Workloads and Computations

1.2.4.1 Computational Workload

1.2.4.2 Parallelism in CNNs

1.2.4.3 Memory Accesses

1.2.4.4 Hardware, Libraries, and Frameworks

1.3 FPGA-Based Deep Learning

1.4 Computational Transforms

1.4.1 The im2col Transformation

1.4.2 Winograd Transform

1.4.3 Fast Fourier Transform

1.5 Data-Path Optimizations

1.5.1 Systolic Arrays

1.5.2 Loop Optimization in Spatial Architectures

Loop Unrolling

Loop Tiling

1.5.3 Design Space Exploration

1.5.4 FPGA Implementations

1.6 Approximate Computing of CNN Models

1.6.1 Approximate Arithmetic for CNNs

1.6.1.1 Fixed-Point Arithmetic

1.6.1.2 Dynamic Fixed Point for CNNs

1.6.1.3 FPGA Implementations

1.6.1.4 Extreme Quantization and Binary Networks

1.6.2 Reduced Computations

1.6.2.1 Weight Pruning

1.6.2.2 Low Rank Approximation

1.6.2.3 FPGA Implementations

1.7 Conclusions

Bibliography

1.1 Introduction

The exponential growth of big data during the last decade motivates for innovative methods to extract high semantic information from raw sensor data such as videos, images, and speech sequences. Among the proposed methods, convolutional neural networks (CNNs) [1] have become the de facto standard by delivering near-human accuracy in many applications related to machine vision (e.g., classification [2], detection [3], segmentation [4]) and speech recognition [5].

This performance comes at the price of a large computational cost as CNNs require up to 38 GOPs to classify a single frame [6]. As a result, dedicated hardware is required to accelerate their execution. Graphics processing units GPUs are the most widely used platform to implement CNNs as they offer the best performance in terms of pure computational throughput, reaching up 11 TFLOPs [7]. Nevertheless, in terms of power consumption, field-programmable gate array (FPGA) solutions are known to be more energy efficient (vs. GPU). While GPU implementations have demonstrated state-of-the-art computational performance, CNN acceleration will soon be moving towards FPGAs for two reasons. First, recent improvements in FPGA technology put FPGA performance within striking distance of GPUs with a reported performance of 9.2 TFLOPs for the latter [8]. Second, recent trends in CNN development increase the sparsity of CNNs and use extremely compact data types. These trends favor FPGA devices, which are designed to handle irregular parallelism and custom data types. As a result, next-generation CNN accelerators are expected to deliver up to 5.4× better computational throughput than GPUs [7].

As an inflection point in the development of CNN accelerators might be near, we conduct a survey on FPGA-based CNN accelerators. While a similar survey can be found in [9], we focus in this chapter on the recent techniques that were not covered in the previous works. In addition to this chapter, we refer the reader to the works of Venieris et al. [10], which review the toolflows automating the CNN mapping process, and to the works of Sze et al., which focus on ASICs for deep learning acceleration.

The amount and diversity of research on the subject of CNN FPGA acceleration within the last 3 years demonstrate the tremendous industrial and academic interest. This chapter presents a state-of-the-art review of CNN inference accelerators over FPGAs. The computational workloads, their parallelism, and the involved memory accesses are analyzed. At the level of neurons, optimizations of the convolutional and fully connected (FC) layers are explained and the performances of the different methods compared. At the network level, approximate computing and data-path optimization methods are covered and state-of-the-art approaches compared. The methods and tools investigated in this survey represent the recent trends in FPGA CNN inference accelerators and will fuel the future advances on efficient hardware deep learning.

1.2 Background on CNNs and Their Computational Workload

In this first section, we overview the main features of CNNs, mainly focusing on ...