Recurrent Neural Networks with Python Quick Start Guide

Sequential learning and language modeling with TensorFlow

Simeon Kostadinov

• Publisher: Packt Publishing
• Year: 2018
• Print ISBN: 9781789132335
• eBook ISBN: 9781789133660
• Edition: 1
• Topic: Computer Science
• Subtopic: Artificial Intelligence (AI) & Semantics
• Index: Computer Science

Learn how to develop intelligent applications with sequential learning and apply modern methods for language modeling with neural network architectures for deep learning with Python's most popular TensorFlow framework.

Key Features

• Train and deploy Recurrent Neural Networks using the popular TensorFlow library
• Apply long short-term memory units
• Expand your skills in complex neural network and deep learning topics

Book Description

Developers struggle to find an easy-to-follow learning resource for implementing Recurrent Neural Network (RNN) models. RNNs are the state-of-the-art model in deep learning for dealing with sequential data. From language translation to generating captions for an image, RNNs are used to continuously improve results. This book will teach you the fundamentals of RNNs, with example applications in Python and the TensorFlow library. The examples are accompanied by the right combination of theoretical knowledge and real-world implementations of concepts to build a solid foundation of neural network modeling.

Your journey starts with the simplest RNN model, where you can grasp the fundamentals. The book then builds on this by proposing more advanced and complex algorithms. We use them to explain how a typical state-of-the-art RNN model works. From generating text to building a language translator, we show how some of today's most powerful AI applications work under the hood.

After reading the book, you will be confident with the fundamentals of RNNs, and be ready to pursue further study, along with developing skills in this exciting field.

What you will learn

• Use TensorFlow to build RNN models
• Use the correct RNN architecture for a particular machine learning task
• Collect and clear the training data for your models
• Use the correct Python libraries for any task during the building phase of your model
• Optimize your model for higher accuracy
• Identify the differences between multiple models and how you can substitute them
• Learn the core deep learning fundamentals applicable to any machine learning model

Who this book is for

This book is for Machine Learning engineers and data scientists who want to learn about Recurrent Neural Network models with practical use-cases. Exposure to Python programming is required. Previous experience with TensorFlow will be helpful, but not mandatory.

Creating a Spanish-to-English Translator

This chapter will push your neural network knowledge even further by introducing state-of-the-art concepts at the core of today's most powerful language translation systems. You will build a simple version of a Spanish-to-English translator, which accepts a sentence in Spanish and outputs its English equivalent.

This chapter includes the following sections:

• Understanding the translation model: This section is entirely focused on the theory behind this system.
• What an LSTM network is: We'll be understanding what sits behind this advanced version of recurrent neural networks.
• Understanding sequence-to-sequence network with attention: You will grasp the theory behind this powerful model, get to know what it actually does, and why it is so widely used for different problems.
• Building the Spanish-to-English translator: This section is entirely focused on implementing the knowledge acquired up to this point in a working program. It includes the following:
  • Training the model
  • Predicting English translations
  • Evaluating the accuracy of the model

Understanding the translation model

Machine translation is often done using so-called statistical machine translation, based on statistical models. This approach works very well, but a key issue is that, for every pair of languages, we need to rebuild the architecture. Thankfully, in 2014, Cho et al. (https://arxiv.org/pdf/1406.1078.pdf) came out with a paper that aims to solve this, and other problems, using the increasingly popular recurrent neural networks. The model is called sequence-to-sequence, and has the ability to be trained on any pair of languages by just providing the right amount of data. In addition, its power lies in its ability to match sequences of different lengths, such as in machine translation, where a sentence in English may have a different size when compared to a sentence in Spanish. Let's examine how these tasks are achieved.

First, we will introduce the following diagram and explain what it consists of:

The architecture has three major parts: the encoder RNN network (on the left side), the intermediate state (marked by the middle arrow), and the decoder RNN network (on the right side). The flow of actions for translating the sentence Como te llamas? (Spanish) into What is your name? (English) is as follows:

• Encode the Spanish sentence, using the encoder RNN, into the intermediate state
• Using that state and the decoder RNN, generate the output sentence in English

This simple approach works with short and simple sentences, but, in practice, the true use of translation models lies in longer and more complicated sequences. That is why we are going to extend our basic approach using the powerful LSTM network and an attention mechanism. Let's explore these techniques in the following sections.

What is an LSTM network?

LSTM (long short-term memory) network is an advanced RNN network that aims to solve the vanishing gradient problem and yield excellent results on longer sequences. In the previous chapter, we introduced the GRU network, which is a simpler version of LSTM. Both include memory states that determine what information should be propagated further at each timestep. The LSTM cell looks as follows:

Let's introduce the main equations that will clarify the preceding diagram. They are similar to the ones for gated recurrent units (see Chapter 3, Generating Your Own Book Chapter). Here is what happens at every given timestep, t:

is the output gate, which determines what exactly is important for the current prediction and what information should be kept around for the future.

is called the input gate, and determines how much we concern ourselves about the current vector (cell).

is the value for the new memory cell.

is the forget gate, which determines how much to forget from the current vector (if the forget gate is 0, we are entirely forgetting the past). All four,

, have the same equation insight (with its corresponding weights), but

uses tanh and the others use sigmoid.

Finally, we have the final memory cell

and final hidden state

:

The final memory cell separates the input and forget gate, and decides how much of the previous output

should be kept and how much of the current output

should ...

Table of contents

• Title Page
• Copyright and Credits
• About Packt
• Contributors
• Preface
• Introducing Recurrent Neural Networks
• Building Your First RNN with TensorFlow
• Generating Your Own Book Chapter
• Creating a Spanish-to-English Translator
• Building Your Personal Assistant
• Improving Your RNN Performance
• Other Books You May Enjoy