Deep Learning through Sparse and Low-Rank Modeling

Zhangyang Wang, Yun Fu, Thomas S. Huang

• Publisher: Academic Press
• Year: 2019
• Print ISBN: 9780128136591
• eBook ISBN: 9780128136607
• Topic: Computer Science
• Subtopic: Digital Media
• Index: Computer Science

Deep Learning through Sparse Representation and Low-Rank Modeling bridges classical sparse and low rank models—those that emphasize problem-specific Interpretability—with recent deep network models that have enabled a larger learning capacity and better utilization of Big Data. It shows how the toolkit of deep learning is closely tied with the sparse/low rank methods and algorithms, providing a rich variety of theoretical and analytic tools to guide the design and interpretation of deep learning models. The development of the theory and models is supported by a wide variety of applications in computer vision, machine learning, signal processing, and data mining.This book will be highly useful for researchers, graduate students and practitioners working in the fields of computer vision, machine learning, signal processing, optimization and statistics.- Combines classical sparse and low-rank models and algorithms with the latest advances in deep learning networks- Shows how the structure and algorithms of sparse and low-rank methods improves the performance and interpretability of Deep Learning models- Provides tactics on how to build and apply customized deep learning models for various applications

Chapter 1

Introduction

Zhangyang Wang^⁎; Ding Liu^{† ⁎}Department of Computer Science and Engineering, Texas A&M University, College Station, TX, United States
^†Beckman Institute for Advanced Science and Technology, Urbana, IL, United States

Abstract

Deep learning has achieved prevailing success in a wide domain of machine learning and computer vision fields. On the other hand, sparsity and low-rankness have been popular regularizations in classical machine learning. This section is intended as a brief introduction to the basics if deep learning, and then focuses on its inherent connections to the concepts of sparsity and low-rankness.

Keywords

Sparsity; Low rank; Deep learning

Chapter Outline

• 1.1 Basics of Deep Learning
• 1.2 Basics of Sparsity and Low-Rankness
• 1.3 Connecting Deep Learning to Sparsity and Low-Rankness
• 1.4 Organization
• References

1.1 Basics of Deep Learning

Machine learning makes computers learn from data without explicitly programming them. However, classical machine learning algorithms often find it challenging to extract semantic features directly from raw data, e.g., due to the well-known “semantic gap” [1], which calls for the assistance from domain experts to hand-craft many well-engineered feature representations, on which the machine learning models operate more effectively. In contrast, the recently popular deep learning relies on multilayer neural networks to derive semantically meaningful representations, by building multiple simple features to represent a sophisticated concept. Deep learning requires less hand-engineered features and expert knowledge. Taking image classification as an example [2], a deep learning-based image classification system represents an object by gradually extracting edges, textures, and structures, from lower to middle-level hidden layers, which becomes more and more associated with the target semantic concept as the model grows deeper. Driven by the emergence of big data and hardware acceleration, the intricacy of data can be extracted with higher and more abstract level representation from raw inputs, gaining more power for deep learning to solve complicated, even traditionally intractable problems. Deep learning has achieved tremendous success in visual object recognition [2–5], face recognition and verification [6,7], object detection [8–11], image restoration and enhancement [12–17], clustering [18], emotion recognition [19], aesthetics and style recognition [20–23], scene understanding [24,25], speech recognition [26], machine translation [27], image synthesis [28], and even playing Go [29] and poker [30].

A basic neural network is composed of a set of perceptrons (artificial neurons), each of which maps inputs to output values with a simple activation function. Among recent deep neural network architectures, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are the two main streams, differing in their connectivity patterns. CNNs deploy convolution operations on hidden layers for weight sharing and parameter reduction. CNNs can extract local information from grid-like input data, and have mainly shown successes in computer vision and image processing, with many popular instances such as LeNet [31], AlexNet [2], VGG [32], GoogLeNet [33], and ResNet [34]. RNNs are dedicated to processing sequential input data with variable length. RNNs produce an output at each time step. The hidden neuron at each time step is calculated based on input data and hidden neurons at the previous time step. To av...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• Contributors
• About the Editors
• Preface
• Acknowledgments
• Chapter 1: Introduction
• Chapter 2: Bi-Level Sparse Coding: A Hyperspectral Image Classification Example
• Chapter 3: Deep ℓ0 Encoders: A Model Unfolding Example
• Chapter 4: Single Image Super-Resolution: From Sparse Coding to Deep Learning
• Chapter 5: From Bi-Level Sparse Clustering to Deep Clustering
• Chapter 6: Signal Processing
• Chapter 7: Dimensionality Reduction
• Chapter 8: Action Recognition
• Chapter 9: Style Recognition and Kinship Understanding
• Chapter 10: Image Dehazing: Improved Techniques
• Chapter 11: Biomedical Image Analytics: Automated Lung Cancer Diagnosis
• Index