Deep Learning Quick Reference

Mike Bernico

• Publisher: Packt Publishing
• Year: 2018
• eBook ISBN: 9781788838917
• Edition: 1
• Topic: Computer Science
• Subtopic: Artificial Intelligence (AI) & Semantics
• Index: Computer Science

Dive deeper into neural networks and get your models trained, optimized with this quick reference guideAbout This Book• A quick reference to all important deep learning concepts and their implementations• Essential tips, tricks, and hacks to train a variety of deep learning models such as CNNs, RNNs, LSTMs, and more• Supplemented with essential mathematics and theory, every chapter provides best practices and safe choices for training and fine-tuning your models in Keras and Tensorflow.Who This Book Is ForIf you are a Data Scientist or a Machine Learning expert, then this book is a very useful read in training your advanced machine learning and deep learning models. You can also refer this book if you are stuck in-between the neural network modeling and need immediate assistance in getting accomplishing the task smoothly. Some prior knowledge of Python and tight hold on the basics of machine learning is required.What You Will Learn• Solve regression and classification challenges with TensorFlow and Keras• Learn to use Tensor Board for monitoring neural networks and its training• Optimize hyperparameters and safe choices/best practices• Build CNN's, RNN's, and LSTM's and using word embedding from scratch• Build and train seq2seq models for machine translation and chat applications.• Understanding Deep Q networks and how to use one to solve an autonomous agent problem.• Explore Deep Q Network and address autonomous agent challenges.In DetailDeep learning has become an essential necessity to enter the world of artificial intelligence. With this book deep learning techniques will become more accessible, practical, and relevant to practicing data scientists. It moves deep learning from academia to the real world through practical examples.You will learn how Tensor Board is used to monitor the training of deep neural networks and solve binary classification problems using deep learning. Readers will then learn to optimize hyperparameters in their deep learning models. The book then takes the readers through the practical implementation of training CNN's, RNN's, and LSTM's with word embeddings and seq2seq models from scratch. Later the book explores advanced topics such as Deep Q Network to solve an autonomous agent problem and how to use two adversarial networks to generate artificial images that appear real. For implementation purposes, we look at popular Python-based deep learning frameworks such as Keras and Tensorflow, Each chapter provides best practices and safe choices to help readers make the right decision while training deep neural networks.By the end of this book, you will be able to solve real-world problems quickly with deep neural networks.Style and approachAn easy-to-follow, step-by-step guide to help you get to grips with real-world applications of training deep neural networks.

Training LSTMs with Word Embeddings from Scratch

So far, we've seen examples of the application of deep learning in structured data, image data, and even time series data. It seems only right to move on to natural language processing (NLP) as the next stop on our tour. The connection between machine learning and human language is a fascinating one. Deep learning has exponentially accelerated the pace at which this field is moving, as it has with computer vision. Let's start with a brief overview of NLP and some of the tasks we'll be taking on in this chapter.

We will also cover the following topics in this chapter:

• An introduction to natural language processing
• Vectorizing text
• Word embedding
• Keras embedding layer
• 1D CNNs for natural language processing
• Case studies for document classifications

An introduction to natural language processing

The field of NLP is vast and complex. Any interaction between human language and computer science might technically fall into this category. For the sake of this discussion though, I'll confine NLP to analyzing, understanding, and, sometimes, generating human language.

From the beginnings of computer science, we've been fascinated by NLP as a gateway to strong artificial intelligence. In 1950, Alan Turing proposed the Turing test, which involves a computer impersonating a human so well that it's indistinguishable from another human, as a metric for machine intelligence. Ever since, we've worked to find clever ways to help machines understand human language. Along the way, we've developed speech-to-text transcription, automatic translation between human languages, the automatic summation of documents, topic modeling, named entity identification, and a variety of other use cases.

As our understanding of NLP continues to grow, we find AI applications becoming common in everyday life. Chatbots have become commonplace as customer service applications and, more recently, they have become our personal digital assistants. As I write this, I'm able to ask Alexa to add something to my shopping list or play some smooth jazz. Natural language processing connects humans to computers in a very interesting and powerful way.

In this chapter, I'm going to focus on understanding human language and then using that understanding to classify. I will actually have two classification case studies, one that covers semantic analysis and another that covers document classification. Both case studies provide great opportunities for the application of deep learning, and they're really very similar.

Semantic analysis

Semantic analysis is technically the analysis of the meaning of language, but usually when we say semantic analysis, we are talking about understanding the feelings of the author. Semantic classifiers are typically trying to classify some utterance as positive, negative, happy, sad, neutral, and so on.

One of my favorite features of human language, sarcasm, makes this a challenging problem to solve. There are many subtle patterns in human language that are very challenging for computers to learn. But challenging doesn't mean impossible. Given a good dataset, this task is very possible.

Success for this type of problem requires a good dataset. While we can most certainly find ample amounts of human conversation all over the internet, most of it isn't labeled. Finding labeled cases is more challenging. An early attempt at solving this problem was to gather twitter data that contained emoticons. If a tweet contained a :), it was considered a positive tweet. This became the well-named emoticon trick referenced in Large-Scale Machine Learning at Twitter by Jimmy Lin and Alek Kolcz.

Most business applications of this type of classifier are binary, where we attempt to predict if the customer is happy or not. That's certainly not the limit to this type of language model, however. We can model other tones as long as we have labels for that sort of thing. We might even attempt to measure anxiety or distress in someone's voice or language; however, addressing audio input is outside the scope of this chapter.

Further attempts to mine data have included using the language associated with positive and negative movie reviews and language related to online shopping product reviews. These are all great approaches; however, great care should be used when using these types of data sources to classify text from a different domain. As you might imagine, the language used in a movie review or an online purchase might be very different from the language used in an IT helpdesk customer support call.

Of course, we can certainly classify more than just sentiment. We will talk about the more general application of document classification in the following section.

Document classification

Document classification is closely related to sentiment analysis. In both cases, we're classifying documents into categories using their text. It's really only the why that changes. Document classification is all about classifying a document based on its type. The world's most obvious and common document classification system is a spam filter, but that has many other uses.

One of my favorite uses of document classification is in settling the debate around the original authors of The Federalist Papers. Alexander Hamilton, James Madison, and John Jay published 85 essays under the pseudonym Publius in 1787 and 1788 supporting the ratification of the United States Constitution. Later, Hamilton provided a list detailing the author of each paper before his fatal duel with Aaron Burr in 1804. Madison provided his own list in 1818 that created a dispute in authorship that scholars have been attempting to solve ever since. While it's mostly agreed upon that the disputed works belonged to Madison, there remain some theories as to a collaborative effort between the two. Classifying these 12 disputed documents as either Madison or Hamilton has been fodder for many a data science blog. Most formally, the paper, The Disputed Federalist Papers: SVM Feature Selection via Concave Minimization, by Glenn Fung covers the topic with quite a bit of rigor.

A final example of document classification might be around understanding the content of the document and prescribing action. Imagine a classifier that might read some information about a legal case, for example, the petition/complaint and summons, and then make a recommendation to the defendant. Our imaginary system might then say, given my experience with other cases like this one, you probably want to settle.

Sentiment analysis and documentation classification ...

Table of contents

• Title Page
• Copyright and Credits
• Dedication
• Packt Upsell
• Foreword
• Contributors
• Preface
• The Building Blocks of Deep Learning
• Using Deep Learning to Solve Regression Problems
• Monitoring Network Training Using TensorBoard
• Using Deep Learning to Solve Binary Classification Problems
• Using Keras to Solve Multiclass Classification Problems
• Hyperparameter Optimization
• Training a CNN from Scratch
• Transfer Learning with Pretrained CNNs
• Training an RNN from scratch
• Training LSTMs with Word Embeddings from Scratch
• Training Seq2Seq Models
• Using Deep Reinforcement Learning
• Generative Adversarial Networks
• Other Books You May Enjoy