Grokking Deep Learning

Andrew W. Trask

• Publisher: Manning
• Year: 2019
• Print ISBN: 9781617293702
• eBook ISBN: 9781638357209
• Topic: Computer Science
• Subtopic: Computer Science General
• Index: Computer Science

Summary Grokking Deep Learning teaches you to build deep learning neural networks from scratch! In his engaging style, seasoned deep learning expert Andrew Trask shows you the science under the hood, so you grok for yourself every detail of training neural networks.Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the Technology Deep learning, a branch of artificial intelligence, teaches computers to learn by using neural networks, technology inspired by the human brain. Online text translation, self-driving cars, personalized product recommendations, and virtual voice assistants are just a few of the exciting modern advancements possible thanks to deep learning. About the Book Grokking Deep Learning teaches you to build deep learning neural networks from scratch! In his engaging style, seasoned deep learning expert Andrew Trask shows you the science under the hood, so you grok for yourself every detail of training neural networks. Using only Python and its math-supporting library, NumPy, you'll train your own neural networks to see and understand images, translate text into different languages, and even write like Shakespeare! When you're done, you'll be fully prepared to move on to mastering deep learning frameworks. What's inside

• The science behind deep learning
• Building and training your own neural networks
• Privacy concepts, including federated learning
• Tips for continuing your pursuit of deep learning

About the Reader For readers with high school-level math and intermediate programming skills. About the Author Andrew Trask is a PhD student at Oxford University and a research scientist at DeepMind. Previously, Andrew was a researcher and analytics product manager at Digital Reasoning, where he trained the world's largest artificial neural network and helped guide the analytics roadmap for the Synthesys cognitive computing platform. Table of Contents

• Introducing deep learning: why you should learn it
• Fundamental concepts: how do machines learn?
• Introduction to neural prediction: forward propagation
• Introduction to neural learning: gradient descent
• Learning multiple weights at a time: generalizing gradient descent
• Building your first deep neural network: introduction to backpropagation
• How to picture neural networks: in your head and on paper
• Learning signal and ignoring noise: introduction to regularization and batching
• Modeling probabilities and nonlinearities: activation functions
• Neural learning about edges and corners: intro to convolutional neural networks
• Neural networks that understand language: king - man + woman ==?
• Neural networks that write like Shakespeare: recurrent layers for variable-length data
• Introducing automatic optimization: let's build a deep learning framework
• Learning to write like Shakespeare: long short-term memory
• Deep learning on unseen data: introducing federated learning
• Where to go from here: a brief guide

Chapter 1. Introducing deep learning: why you should learn it

In this chapter

• Why you should learn deep learning
• Why you should read this book
• What you need to get started

“Do not worry about your difficulties in Mathematics. I can assure you mine are still greater.”

Albert Einstein

Welcome to Grokking Deep Learning

You’re about to learn some of the most valuable skills of the century!

I’m very excited that you’re here! You should be, too! Deep learning represents an exciting intersection of machine learning and artificial intelligence, and a very significant disruption to society and industry. The methods discussed in this book are changing the world all around you. From optimizing the engine of your car to deciding which content you view on social media, it’s everywhere, it’s powerful, and, fortunately, it’s fun!

Why you should learn deep learning

It’s a powerful tool for the incremental automation of intelligence

From the beginning of time, humans have been building better and better tools to understand and control the environment around us. Deep learning is today’s chapter in this story of innovation.

Perhaps what makes this chapter so compelling is that this field is more of a mental innovation than a mechanical one. Much like its sister fields in machine learning, deep learning seeks to automate intelligence bit by bit. In the past few years, it has achieved enormous success and progress in this endeavor, exceeding previous records in computer vision, speech recognition, machine translation, and many other tasks.

This is particularly extraordinary given that deep learning seems to use largely the same brain-inspired algorithm (neural networks) for achieving these accomplishments across a vast number of fields. Even though deep learning is still an actively developing field with many challenges, recent developments have lead to tremendous excitement: perhaps we’ve discovered not just a great tool, but a window into our own minds.

Deep learning has the potential for significant automation of skilled labor

There’s a substantial amount of hype around the potential impacts of deep learning if the current trend of progress is extrapolated at varying speeds. Although many of these predictions are overzealous, I believe one merits your consideration: job displacement. I think this claim stands out from the rest because even if deep learning’s innovations stopped today, there would already be an incredible impact on skilled labor around the globe. Call-center operators, taxi drivers, and low-level business analysts are compelling examples where deep learning can provide a low-cost alternative.

Fortunately, the economy doesn’t turn on a dime; but in many ways we’re already past the point of concern, given the current power of the technology. It’s my hope that you (and people you know) will be enabled by this book to transition from perhaps one of the industries facing disruption into an industry ripe with growth and prosperity: deep learning.

It’s fun and creative. You’ll discover much about what it is to b- be human by trying to simulate intelligence and creativity

Personally, I got into deep learning because it’s fascinating. It’s an amazing intersection between human and machine. Unpacking exactly what it means to think, to reason, and to create is enlightening, engaging, and, for me, inspiring. Consider having a dataset filled with every painting ever painted, and then using that to teach a machine how to paint like Monet. Insanely, it’s possible, and it’s mind-bogglingly cool to see how it works.

Will this be difficult to learn?

How hard will you have to work before there’s a “fun” payoff?

This is my favorite question. My definition of a “fun” payoff is the experience of witnessing something that I built learning. There’s something amazing about seeing a creation of your hands do something like that. If you also feel this way, then the answer is simple. A few pages into chapter 3, you’ll create your first neural network. The only work involved until then is reading the pages between here and there.

After chapter 3, you may be interested to know that the next fun payoff occurs after you’ve memorized a small snippet of code and proceeded to read to the midway of chapter 4. Each chapter will work this way: memorize a small code segment from the previous chapter, read the next chapter, and then experience the payoff of a new learning neural network.

Why you should read this book

It has a uniquely low barrier to entry

The reason you should read this book is the same reason I’m writing it. I don’t know of another resource (book, course, large blog series) that teaches deep learning without assuming advanced knowledge of mathematics (a college degree in a mathy field).

Don’t get me wrong: there are really good reasons for teaching it using math. Math is, after all, a language. It’s certainly more efficient to teach deep learning using this language, but I don’t think it’s absolutely necessary to assume advanced knowledge of math in order to become a skilled, knowledgeable practitioner who has a firm understanding of the “how” behind deep learning.

So, why should you learn deep learning using this book? Because I’m going to assume you have a high school–level background in math (and that it’s rusty) and explain everything else you need to know as we go along. Remember multiplication? Remember x-y graphs (the squares with lines on them)? Awesome! You’ll be fine.

It will help you understand what’s inside a framework (Torch, Ten- nsorFlow, and so on)

There are two major groups of deep learning educational material (such as books and courses). One group is focused around how to use popular frameworks and code libraries like Torch, TensorFlow, Keras, and others. The other group is focused around teaching deep learning itself, otherwise known as the science under the hood of these major frameworks. Ultimately, learning about both is important. It’s like if you want to be a NASCAR driver: you need to learn both about the particular model of car you’re driving (the framework) and about driving (the science/skill). But just learning about a framework is like learning about the pros and cons of a Generation 6 Chevrolet SS before you know what a stick shift is. This book is about teaching you what deep learning is so you can then be prepared to learn a framework.

All math-related material will be backed by intuitive analogies

Whenever I encounter a math formula in the wild, I take a two-step approach. The first is to translate its methods into an intuitive analogy to the real world. I almost never take a formula at face value: I break it into parts, each with a story of its own. That will be the approach of this book, as well. Anytime we encounter a math concept, I’ll offer an alternative analogy for what the formula is actually doing.

“Everything should be made as simple as possible, but not simpler.”

Attributed to Albert Einstein

Everything after the introduction chapters is “project” based

If there’s one thing I hate when learning something new, it’s having to question whether what I’m learning is useful or relevant. If someone is teaching me everything there is to know about a hammer without actually taking my hand and helping me drive in a nail, then they’re not really teaching me how to use a hammer. I know there will be dots that aren’t connected, and if I’m thrown out into the real world with a hammer, a box of nails, and a bunch of two-by-fours, I’ll have to do some guesswork.

This book is about giving you the wood, nails, and hammer before telling you what they do. Each lesson is about picking up the tools and building stuff with them, explaining how things work as we go. This way, you won’t leave with a list of facts about the various deep learning tools you’ll work with; you’ll have the ability to use them to solve problems. Furthermore, you’ll understand the most important part: when and why each tool is appropriate for each problem you want to solve. It is with this knowledge that you’ll be empowered to pursue a career in research and/or industry.

What you need to get started

Install Jupyter Notebook and the NumPy Python library

My absolute favorite place to work is in Jupyter Notebook. One of the most important parts of learning deep learning (for me) is the ability to stop a network while it’s training and tear apart absolutely every piece to see what it looks like. This is something Jupyter Notebook is incredibly useful for.

As for NumPy, perhaps the most compelling case for why this book leaves nothing out is that we’ll be using only a single matrix library. In this way, you’ll understand how everything works, not just how to call a framework. This book teaches deep learning from absolute scratch, soup to nuts.

Installation instructions for these two tools can be found at http://jupyter.org for Jupyter and http://numpy.org for NumPy. I’ll build the examples in Python 2.7, but I’ve tested them for Python 3 as well. For easy installation, I also recommend the Anaconda framework: https://docs.continuum.io/anaconda/install.

Pass high school mathematics

Some mathematical assumptions are out of depth for this book, but my ...

Table of contents

• Copyright
• Brief Table of Contents
• Table of Contents
• Preface
• Acknowledgments
• About this book
• About the author
• Chapter 1. Introducing deep learning: why you should learn it
• Chapter 2. Fundamental concepts: how do machines learn?
• Chapter 3. Introduction to neural prediction: forward propagation
• Chapter 4. Introduction to neural learning: gradient descent
• Chapter 5. Learning multiple weights at a time: generalizing gradient descent
• Chapter 6. Building your first deep neural network: introduction to backpropagation
• Chapter 7. How to picture neural networks: in your head and on paper
• Chapter 8. Learning signal and ignoring noise: introduction to regularization and batching
• Chapter 9. Modeling probabilities and nonlinearities: activation functions
• Chapter 10. Neural learning about edges and corners: intro to convolutional neural networks
• Chapter 11. Neural networks that understand language: king – man + woman == ?
• Chapter 12. Neural networks that write like Shakespeare: recurrent layers for variable-length data
• Chapter 13. Introducing automatic optimization: let’s build a deep learning framework
• Chapter 14. Learning to write like Shakespeare: long short-term memory
• Chapter 15. Deep learning on unseen data: introducing federated learning
• Chapter 16. Where to go from here: a brief guide
• Index