Deep Learning with R for Beginners

Design neural network models in R 3.5 using TensorFlow, Keras, and MXNet

Mark Hodnett, Joshua F. Wiley, Yuxi (Hayden) Liu, Pablo Maldonado

• Publisher: Packt Publishing
• Year: 2019
• ISBN:
  9781838647223
• Edition: 1
• Topic: Computer Science
• Subtopic: Neural Networks
• Index: Computer Science

Explore the world of neural networks by building powerful deep learning models using the R ecosystem

Key Features

• Get to grips with the fundamentals of deep learning and neural networks
• Use R 3.5 and its libraries and APIs to build deep learning models for computer vision and text processing
• Implement effective deep learning systems in R with the help of end-to-end projects

Book Description

Deep learning finds practical applications in several domains, while R is the preferred language for designing and deploying deep learning models.

This Learning Path introduces you to the basics of deep learning and even teaches you to build a neural network model from scratch. As you make your way through the chapters, you'll explore deep learning libraries and understand how to create deep learning models for a variety of challenges, right from anomaly detection to recommendation systems. The book will then help you cover advanced topics, such as generative adversarial networks (GANs), transfer learning, and large-scale deep learning in the cloud, in addition to model optimization, overfitting, and data augmentation. Through real-world projects, you'll also get up to speed with training convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) in R.

By the end of this Learning Path, you'll be well versed with deep learning and have the skills you need to implement a number of deep learning concepts in your research work or projects.

This Learning Path includes content from the following Packt products:

• R Deep Learning Essentials - Second Edition by Joshua F. Wiley and Mark Hodnett
• R Deep Learning Projects by Yuxi (Hayden) Liu and Pablo Maldonado

What you will learn

• Implement credit card fraud detection with autoencoders
• Train neural networks to perform handwritten digit recognition using MXNet
• Reconstruct images using variational autoencoders
• Explore the applications of autoencoder neural networks in clustering and dimensionality reduction
• Create natural language processing (NLP) models using Keras and TensorFlow in R
• Prevent models from overfitting the data to improve generalizability
• Build shallow neural network prediction models

Who this book is for

This Learning Path is for aspiring data scientists, data analysts, machine learning developers, and deep learning enthusiasts who are well versed in machine learning concepts and are looking to explore the deep learning paradigm using R. A fundamental understanding of R programming and familiarity with the basic concepts of deep learning are necessary to get the most out of this Learning Path.

Text Generation using Recurrent Neural Networks

In this chapter, we will describe some of the most exciting techniques in modern (at the time of writing—late 2017) machine learning, recurrent neural networks. They are, however, not new; they have been around since the 1980s, but they have become popular due to the numerous records in language-related tasks in recent years.

Why do we need a different type of architecture for text? Consider the following example:

"I live in Prague since 2015"

and

"Since 2015 I live in Prague"

If we would like to teach a traditional feed-forward network such as a perceptron or a multi-layer perceptron to identify the date I moved to Prague, then this network would have to learn separate parameters for each input feature, which in particular implies that it would have to learn grammar to answer this simple question! This is undesirable in many applications. Similar issues motivated machine learning researchers and statisticians in the 1980s to introduce the idea of sharing parameters across different parts of the model. This idea is the secret sauce of recurrent neural networks, our next deep learning architecture.

By design, recurrent neural networks are well-suited for processing sequential data. In general, machine learning applied to sequential data can be roughly divided into four main areas:

• Sequence prediction: Given
  
  , predict the next element of the sequence,
  
• Sequence classification: Given
  
  , predict a category or label for it
• Sequence generation: Given
  
  , generate a new element of the sequence,
  
• Sequence to sequence prediction: Given
  
  , generate an equivalent sequence,
  

Applications of sequence prediction include weather forecasting and stock market prediction. For classification, we can think, for example, of sentiment analysis and document classification. Automatic image captioning or text generation are part of the sequence generation family of problems, whereas machine translation might be the most familiar example of sequence to sequence prediction we see in our everyday lives.

Our focus for this chapter is on applications of recurrent neural networks for text generation. Since, as we saw previously, text generation is part of a much larger set of problems, many of our algorithms are portable to other contexts.

Training deep learning models is often time-consuming, and recurrent neural networks are not the exception. Our focus is on the ideas over the data, which we will illustrate with smaller datasets than those that you might encounter later on in the wild. This is for the purpose of clarity: We want to make it easier for you to get started on any standard laptop. Once you grasp the basics, you can spin off your own cluster in your favorite cloud provider.

What is so exciting about recurrent neural networks?

Coming from a mathematics background, in my rather hectic career I have seen many different trends, particularly during the last few years, which all sound very similar to me: "you have a problem? wavelets can save you!", "finite elements are the solution to everything", and similar over-enthusiastic claims.

Of course, each tool has its time and place and, more importantly, an application domain where it excels. I find recurrent neural networks quite interesting for the many features they can achieve:

• Produce consistent markup text (opening and closing tags, recognizing timestamp-like data)
• Write Wikipedia articles with references, and create URLs from non-existing addresses, by learning what a URL should look like
• Create credible-looking scientific papers from LaTeX

All these amazing features are possible without the network having any context information or metadata. In particular, without knowing English, nor what a URL or a bit of LaTeX syntax looks like.

These and even more interesting capabilities of neural networks are superbly described by Andrej Karpathy in The Unreasonable Effectiveness of Recurrent Neural Networks: http://karpathy.github.io/2015/05/21/rnn-effectiveness/.

What makes recurrent neural networks exciting? Instead of a constrained fixed-input size to fixed-output size, we can operate over sequences of vectors instead.

A limitation of many machine learning algorithms, including standard feed-forward neural networks, is that they accept a fixed size vector as input and produce a fixed size vector as output. For instance, if we want to classify text, we receive a corpus of documents from which we create a vocabulary to vectorize each document and the output is a vector with class probabilities. Recurrent neural networks instead allow us to take sequences of vectors as input. So, from a one-to-one correspondence between fixed input size and fixed output size, we have a much richer landscape, one-to-one, one-to-many, many-to-one, many-to-many.

Why is that desirable? Let's look at a few examples:

• One-to-one: Supervised learning, for instance, text classification
• One-to-many: Given an input text, generate a summary (a sequence of words with important information)
• Many-to-one: Sentiment analysis in text
• Many-to-many: Machine translation

Moreover, as recurrent neural networks maintain an internal state which gets updated according to new information, we can view RNNs as a description of a program. In fact, a paper by Siegelman in 1995 shows that recurrent neural networks are Turing complete, they can simulate arbitrary programs.

But what is a recurrent neural network, really?

How does the network keep track of the previous states? To put it in the context of text generation, think of our training data as a list of character sequences (tokenized words). For each word, from the first character, we will predict the following:

Formally, let's denote a sequence of t+1 characters as x = [x₀, x₁, x₂, ... , x_t]. Let s_-1 =0.

For k=0,2,...t, we construct the following sequence:

This is summarized in the following diagram, when input x is received, the internal state, s, of the network is modified, and then used to generate an output, o:

Figure 1...