Deep Learning Patterns and Practices

Andrew Ferlitsch

• Publisher: Manning
• Year: 2021
• ISBN:
  9781638356677
• Topic: Informatique
• Subtopic: Réseaux de neurones

Discover best practices, reproducible architectures, and design patterns to help guide deep learning models from the lab into production. In Deep Learning Patterns and Practices you will learn: Internal functioning of modern convolutional neural networks
Procedural reuse design pattern for CNN architectures
Models for mobile and IoT devices
Assembling large-scale model deployments
Optimizing hyperparameter tuning
Migrating a model to a production environment The big challenge of deep learning lies in taking cutting-edge technologies from R&D labs through to production. Deep Learning Patterns and Practices is here to help. This unique guide lays out the latest deep learning insights from author Andrew Ferlitsch's work with Google Cloud AI. In it, you'll find deep learning models presented in a unique new way: as extendable design patterns you can easily plug-and-play into your software projects. Each valuable technique is presented in a way that's easy to understand and filled with accessible diagrams and code samples.Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.About the technology
Discover best practices, design patterns, and reproducible architectures that will guide your deep learning projects from the lab into production. This awesome book collects and illuminates the most relevant insights from a decade of real world deep learning experience. You'll build your skills and confidence with each interesting example.About the book
Deep Learning Patterns and Practices is a deep dive into building successful deep learning applications. You'll save hours of trial-and-error by applying proven patterns and practices to your own projects. Tested code samples, real-world examples, and a brilliant narrative style make even complex concepts simple and engaging. Along the way, you'll get tips for deploying, testing, and maintaining your projects.
What's inside Modern convolutional neural networks
Design pattern for CNN architectures
Models for mobile and IoT devices
Large-scale model deployments
Examples for computer visionAbout the reader
For machine learning engineers familiar with Python and deep learning. About the author
Andrew Ferlitsch is an expert on computer vision, deep learning, and operationalizing ML in production at Google Cloud AI Developer Relations.Table of Contents PART 1 DEEP LEARNING FUNDAMENTALS
1 Designing modern machine learning
2 Deep neural networks
3 Convolutional and residual neural networks
4 Training fundamentals
PART 2 BASIC DESIGN PATTERN
5 Procedural design pattern
6 Wide convolutional neural networks
7 Alternative connectivity patterns
8 Mobile convolutional neural networks
9 Autoencoders
PART 3 WORKING WITH PIPELINES
10 Hyperparameter tuning
11 Transfer learning
12 Data distributions
13 Data pipeline
14 Training and deployment pipeline

Part 1. Deep learning fundamentals

In this first part, you’ll learn the fundamentals to get started building deep learning models. We start with the basic principles and steps of deep neural networks (DNN), with lots of diagrams illustrating these steps along with code snippets to implement the steps. I’ll describe each step and then walk you through the code. Next, we cover the principles and steps of convolutional neural networks (CNN). I’ll walk you through the seminal design patterns behind early state-of-the-art ConvNets, VGGs, and ResNets. You’ll learn how to code each one of these model architectures, with complete code accessible from our public GitHub repository.

Once you are coding CNNs, what’s next? You train them. We wrap up this part by learning fundamentals of training CNN models.

  

1 Designing modern machine learning

This chapter covers

• Evolving from classical AI to cutting-edge approaches
• Applying design patterns to deep learning
• Introducing the procedural reuse design pattern for modeling neural networks

The latest revolution in deep learning is at the macro level rather than the micro level, with the introduction of an approach that I coined while working at Google Cloud AI as model amalgamation. In this approach, models are broken into composable units that share and adapt components to achieve different objectives with the same initial data. The components are interconnected in a variety of connectivity patterns, in which each component learns communication interfaces between the models through design, without the necessity of a backend application.

In addition, model amalgamation can be used to train Internet of Things (IoT) devices for data enrichment, turning IoT sensors from static to dynamically learning devices—a technique called model fusion. Amalgamation is providing the means for putting AI into production at a scale and operational complexity not conceivable in 2017, when the push into production first started to emerge.

Think, for example, of the operational complexity of visual real estate data on a variety of aspects of the rental market, such as pricing, property condition, and amenities. Using the model amalgamation approach, you could create a vision analysis pipeline that connects individual models’ components, each working on one of those aspects. In the end, you’d have a system in place to automatically learn to determine the condition, amenities, and general market appeal with the corresponding appropriate rental pricing.

The model amalgamation approach encourages engineers to view models as design patterns or templates that can be adapted to create individual components. So if you hope to use this approach, you’ll need to understand the designs of the key models and systems that other engineers have developed to solve problems similar to the ones you will encounter.

The goal of this book is to aid you in that deep understanding by introducing you to the design patterns of seminal deep learning models, as well as the design or system architecture that puts those components together to develop, train, deploy, and serve larger deep learning systems. Even if you never work with huge enterprise amalgamations, becoming fluent in the underlying designs of these models and architectures will improve the engineering of any deep learning system you create.

1.1 A focus on adaptability

Because this book is aimed at less-experienced deep learning engineers and data scientists, part 1 starts with the designs of the basic deep neural networks (DNNs), convolutional neural networks (CNNs), and residual neural networks (ResNets). Part 1 also looks at the architecture of simple training pipelines. Whole books are written about just these networks and architectures, so here you’ll get more of a reminder of how they work, with an emphasis on design patterns and principles. The point here is to lay out the design of basic deep learning components that all of the models you’ll see in part 2 will fit into.

That said, if you are well versed in the fundamentals, you can go directly to part 2, which looks at the seminal models in the development of deep learning. My approach is to provide enough information about each model design so that you can play around with them and come up with solutions to the AI challenges you may encounter. The models are introduced more or less chronologically, so part 2 also serves as a kind of history of deep learning, with an emphasis on the evolution from one model to the next.

Now, if enterprise production is moving toward automatic learning for model development, you may wonder about the value of examining these manually designed, formerly state-of-the-art (SOTA) models. Many of these models, however, continue to be used as stock models, particularly for transfer learning. Some of the others never made it into production at all, but were responsible for discoveries that continue to be used today.

Model development for production continues to be a combination of automatic and hand-designed learning—which is often crucial for proprietary needs or advantages. But designing by hand does not mean starting from scratch; typically, you would start with a stock model and make tweaks and adjustments. To do this effectively, you need to know how the model works and why it works that way, the concepts that underlie its design, and the pros and cons of alternative building blocks you will learn from other SOTA models.

The final part of the book takes a deep dive into the design patterns for training and deployment for production. While not all readers will be deploying the kinds of enterprise systems that are my focus, I feel this information is relevant to all. Becoming familiar with many types—and sizes—of systems addressing a variety of problems can help you when you need to think outside the box to solve a problem. The more you know about the underlying concepts and designs, the more able and adaptable you become.

This adaptability is probably the most valuable takeaway from this book. Production involves a vast number of moving parts, and an endless flow of “monkey wrenches” being tossed into the mix. If engineers or data scientists simply rote-memorize sets of reproducible steps in a framework, how will they handle the diversity of tasks they’ll encounter and resolve the monkey wrenches thrown at them? Employers look for more than just skill and experience; they want to know how technically adaptive you are.

Imagine yourself in an interview: you score high on skill and work experience and nail the stock machine learning (ML) coding challenge. Then the interviewers throw you a monkey wrench, an unexpected or unusual problem. They do this to observe how you think through a challenge, which concepts you apply and the reasoning behind them, how you evaluate the pros and cons of the various solutions, and your ability to debug. That’s adaptability. And that’s what I hope deep learning developers and data scientists will get from this book.

1.1.1 Computer vision leading the way

I teach all of these concepts primarily in the context of computer vision, because design patterns evolved first in computer vision. But they are applicable to natural-language processing (NLP), structured data, signal processing, and other fields. If we roll the clock back to prior to 2012, ML in all fields was mostly using classical statistics-based methods.

Various academic researchers, such as Fei-Fei Liu at Stanford University and Geoffrey Hinton of the University of Toronto, began to pioneer applying neural networks to computer vision. Liu, along with her students, compiled a computer vision dataset, now known as ImageNet, to advance the research into computer vision. ImageNet, along with the PASCAL dataset, became the basis for the annual ImageNet Large Scale Vision Recognition Challenge (ILSVRC) competition in 2010. Early entries used traditional image recognition/signal processing methods.

Then, in 2012, Alex Krizhevsky, also of the University of Toronto, entered a deep learning model, AlexNet, using convolution layers. This model won the ILSVRC contest and by a sizable margin. The AlexNet model, jointly designed with Hinton and Ilya Sutskever, kicked off deep learning. In their corresponding paper, “ImageNet Classification with Deep Convolutional Neural Networks” (http://mng.bz/1ApV), they showed how neural networks can be designed.

In 2013, Matthew Zeiler and Rob Fergus of New York University won the competition by fine-tuning AlexNet into what they called ZFNet. This pattern of building on each other’s success continued. The Visual Geometry Group at Oxford expanded on the AlexNet design principles and won the 2014 contest. In 2015, Kaiming He and others at Microsoft Research further expanded on the AlexNet/VGG design principles and introduced new design patterns, winning the competition. Their model, ResNet, and their “Deep Residual Learning for Image Recognition” paper (https://arxiv.org/abs/ 1512.03385), set off a surge in discovering and exploring the design space of CNNs.

1.1.2 Beyond computer vision: NLP, NLU, structured data

In these early years of developing design principles and design patterns using deep learning for computer vision, developments in natural-language understanding (NLU) and structured data models lagged behind and continued to focus on classical approaches. They used classical ML frameworks, like the Natural Language Toolkit (NLTK) for text input, and classical algorithms based on decision trees, like random forest, for structured data input.

In the NLU field, progress was made with the introduction of RNNs and long-short-term-memory (LSTM) and gated recurrent unit (GRU) layers. That progress took a leap in 2017 with the introduction of the Transformer design pattern for natural language, and the corresponding paper “Attention Is All You Need” by Ashish Vaswani et. al (https://arxiv.org/abs/1706.03762). Google Brain, a deep learning research organization within Google AI, adopted early a comparable attention mechanism in ResNet. Likewise, advancements in design patterns for structured data evolved with the introduction of the wide-and-deep model pattern, outlined in “Wide & Deep Learning for Recommender Systems” (https://arxiv.org/abs/1606.07792) by Heng-Tze Cheng et. al, at the technology-agnostic research group Google Research, in 2016.

While I focus on compute...