Advanced Deep Learning with Python

Design and implement advanced next-generation AI solutions using TensorFlow and PyTorch

Ivan Vasilev

• Publisher: Packt Publishing
• Year: 2019
• ISBN:
  9781789952711
• Edition: 1
• Topic: Informatique
• Subtopic: Réseaux de neurones

Gain expertise in advanced deep learning domains such as neural networks, meta-learning, graph neural networks, and memory augmented neural networks using the Python ecosystem

Key Features

• Get to grips with building faster and more robust deep learning architectures
• Investigate and train convolutional neural network (CNN) models with GPU-accelerated libraries such as TensorFlow and PyTorch
• Apply deep neural networks (DNNs) to computer vision problems, NLP, and GANs

Book Description

In order to build robust deep learning systems, you'll need to understand everything from how neural networks work to training CNN models. In this book, you'll discover newly developed deep learning models, methodologies used in the domain, and their implementation based on areas of application.

You'll start by understanding the building blocks and the math behind neural networks, and then move on to CNNs and their advanced applications in computer vision. You'll also learn to apply the most popular CNN architectures in object detection and image segmentation. Further on, you'll focus on variational autoencoders and GANs. You'll then use neural networks to extract sophisticated vector representations of words, before going on to cover various types of recurrent networks, such as LSTM and GRU. You'll even explore the attention mechanism to process sequential data without the help of recurrent neural networks (RNNs). Later, you'll use graph neural networks for processing structured data, along with covering meta-learning, which allows you to train neural networks with fewer training samples. Finally, you'll understand how to apply deep learning to autonomous vehicles.

By the end of this book, you'll have mastered key deep learning concepts and the different applications of deep learning models in the real world.

What you will learn

• Cover advanced and state-of-the-art neural network architectures
• Understand the theory and math behind neural networks
• Train DNNs and apply them to modern deep learning problems
• Use CNNs for object detection and image segmentation
• Implement generative adversarial networks (GANs) and variational autoencoders to generate new images
• Solve natural language processing (NLP) tasks, such as machine translation, using sequence-to-sequence models
• Understand DL techniques, such as meta-learning and graph neural networks

Who this book is for

This book is for data scientists, deep learning engineers and researchers, and AI developers who want to further their knowledge of deep learning and build innovative and unique deep learning projects. Anyone looking to get to grips with advanced use cases and methodologies adopted in the deep learning domain using real-world examples will also find this book useful. Basic understanding of deep learning concepts and working knowledge of the Python programming language is assumed.

Section 1: Core Concepts

This section will discuss some core Deep Learning (DL) concepts: what exactly DL is, the mathematical underpinnings of DL algorithms, and the libraries and tools that make it possible to develop DL algorithms rapidly.

This section contains the following chapter:

• Chapter 1, The Nuts and Bolts of Neural Networks

The Nuts and Bolts of Neural Networks

In this chapter, we'll discuss some of the intricacies of neural networks (NNs)—the cornerstone of deep learning (DL). We'll talk about their mathematical apparatus, structure, and training. Our main goal is to provide you with a systematic understanding of NNs. Often, we approach them from a computer science perspective—as a machine learning (ML) algorithm (or even a special entity) composed of a number of different steps/components. We gain our intuition by thinking in terms of neurons, layers, and so on (at least I did this when I first learned about this field). This is a perfectly valid way to do things and we can still do impressive things at this level of understanding. Perhaps this is not the correct approach, though.

NNs have solid mathematical foundations and if we approach them from this point of view, we'll be able to define and understand them in a more fundamental and elegant way. Therefore, in this chapter, we'll try to underscore the analogy between NNs from mathematical and computer science points of view. If you are already familiar with these topics, you can skip this chapter. Still, I hope that you'll find some interesting bits you didn't know about already (we'll do our best to keep this chapter interesting!).

In this chapter, we will cover the following topics:

• The mathematical apparatus of NNs
• A short introduction to NNs
• Training NNs

The mathematical apparatus of NNs

In the next few sections, we'll discuss the mathematical branches related to NNs. Once we've done this, we'll connect them to NNs themselves.

Linear algebra

Linear algebra deals with linear equations such as

and linear transformations (or linear functions) and their representations, such as matrices and vectors.

Linear algebra identifies the following mathematical objects:

• Scalars: A single number.
• Vectors: A one-dimensional array of numbers (or components). Each component of the array has an index. In literature, we will see vectors denoted either with a superscript arrow (
  
  ) or in bold (x). The following is an example of a vector:

Throughout this book, we'll mostly use the bold (x) graph notations. But in some instances, we'll use formulas from different sources and we'll try to retain their original notation.

We can visually represent an n-dimensional vector as the coordinates of a point in an n-dimensional Euclidean space,

(equivalent to a coordinate system). In this case, the vector is referred to as Euclidean and each vector component represents the coordinate along the corresponding axis, as shown in the following diagram:

Vector representation in

space

However, the Euclidean vector is more than just a point and we can also represent it with the following two properties:

• Magnitude (or length) is a generalization of the Pythagorean theorem for an n-dimensional space:

• Direction is the angle of the vector along each axis of the vector space.

• Matrices: This is a two-dimensional array of numbers. Each element is identified by two indices (row and column). A matrix is usually denoted with a bold capital letter; for example...