Natural Language Processing with TensorFlow

Teach language to machines using Python's deep learning library

Thushan Ganegedara

• Éditeur: Packt Publishing
• Année: 2018
• ISBN:
  9781788477758
• Édition: 1
• Sujet: Computer Science
• Sous-sujet: Artificial Intelligence (AI) & Semantics

Write modern natural language processing applications using deep learning algorithms and TensorFlowAbout This Book• Focuses on more efficient natural language processing using TensorFlow• Covers NLP as a field in its own right to improve understanding for choosing TensorFlow tools and other deep learning approaches• Provides choices for how to process and evaluate large unstructured text datasets• Learn to apply the TensorFlow toolbox to specific tasks in the most interesting field in artificial intelligenceWho This Book Is ForThis book is for Python developers with a strong interest in deep learning, who want to learn how to leverage TensorFlow to simplify NLP tasks. Fundamental Python skills are assumed, as well as some knowledge of machine learning and undergraduate-level calculus and linear algebra. No previous natural language processing experience required, although some background in NLP or computational linguistics will be helpful.What You Will Learn• Core concepts of NLP and various approaches to natural language processing• How to solve NLP tasks by applying TensorFlow functions to create neural networks• Strategies to process large amounts of data into word representations that can be used by deep learning applications• Techniques for performing sentence classification and language generation using CNNs and RNNs• About employing state-of-the art advanced RNNs, like long short-term memory, to solve complex text generation tasks• How to write automatic translation programs and implement an actual neural machine translator from scratch• The trends and innovations that are paving the future in NLPIn DetailNatural language processing (NLP) supplies the majority of data available to deep learning applications, while TensorFlow is the most important deep learning framework currently available. Natural Language Processing with TensorFlow brings TensorFlow and NLP together to give you invaluable tools to work with the immense volume of unstructured data in today's data streams, and apply these tools to specific NLP tasks.Thushan Ganegedara starts by giving you a grounding in NLP and TensorFlow basics. You'll then learn how to use Word2vec, including advanced extensions, to create word embeddings that turn sequences of words into vectors accessible to deep learning algorithms. Chapters on classical deep learning algorithms, like convolutional neural networks (CNN) and recurrent neural networks (RNN), demonstrate important NLP tasks as sentence classification and language generation. You will learn how to apply high-performance RNN models, like long short-term memory (LSTM) cells, to NLP tasks. You will also explore neural machine translation and implement a neural machine translator.After reading this book, you will gain an understanding of NLP and you'll have the skills to apply TensorFlow in deep learning NLP applications, and how to perform specific NLP tasks.Style and approachThe book provides an emphasis on both the theory and practice of natural language processing. It introduces the reader to existing TensorFlow functions and explains how to apply them while writing NLP algorithms. The popular Word2vec method is used to teach the essential process of learning word representations. The book focuses on how to apply classical deep learning to NLP, as well as exploring cutting edge and emerging approaches. Specific examples are used to make the concepts and techniques concrete.

Natural Language Processing with TensorFlow

---

Table of Contents

Natural Language Processing with TensorFlow

Why subscribe?

PacktPub.com

Contributors

About the author

About the reviewers

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

1. Introduction to Natural Language Processing

What is Natural Language Processing?

Tasks of Natural Language Processing

The traditional approach to Natural Language Processing

Understanding the traditional approach

Example – generating football game summaries

Drawbacks of the traditional approach

The deep learning approach to Natural Language Processing

History of deep learning

The current state of deep learning and NLP

Understanding a simple deep model – a Fully-Connected Neural Network

The roadmap – beyond this chapter

Introduction to the technical tools

Description of the tools

Installing Python and scikit-learn

Installing Jupyter Notebook

Installing TensorFlow

Summary

2. Understanding TensorFlow

What is TensorFlow?

Getting started with TensorFlow

TensorFlow client in detail

TensorFlow architecture – what happens when you execute the client?

Cafe Le TensorFlow – understanding TensorFlow with an analogy

Inputs, variables, outputs, and operations

Defining inputs in TensorFlow

Feeding data with Python code

Preloading and storing data as tensors

Building an input pipeline

Defining variables in TensorFlow

Defining TensorFlow outputs

Defining TensorFlow operations

Comparison operations

Mathematical operations

Scatter and gather operations

Neural network-related operations

Nonlinear activations used by neural networks

The convolution operation

The pooling operation

Defining loss

Optimization of neural networks

The control flow operations

Reusing variables with scoping

Implementing our first neural network

Preparing the data

Defining the TensorFlow graph

Running the neural network

Summary

3. Word2vec – Learning Word Embeddings

What is a word representation or meaning?

Classical approaches to learning word representation

WordNet – using an external lexical knowledge base for learning word representations

Tour of WordNet

Problems with WordNet

One-hot encoded representation

The TF-IDF method

Co-occurrence matrix

Word2vec – a neural network-based approach to learning word representation

Exercise: is queen = king – he + she?

Designing a loss function for learning word embeddings

The skip-gram algorithm

From raw text to structured data

Learning the word embeddings with a neural network

Formulating a practical loss function

Efficiently approximating the loss function

Negative sampling of the softmax layer

Hierarchical softmax

Learning the hierarchy

Optimizing the learning model

Implementing skip-gram with TensorFlow

The Continuous Bag-of-Words algorithm

Implementing CBOW in TensorFlow

Summary

4. Advanced Word2vec

The original skip-gram algorithm

Implementing the original skip-gram algorithm

Comparing the original skip-gram with the improved skip-gram

Comparing skip-gram with CBOW

Performance comparison

Which is the winner, skip-gram or CBOW?

Extensions to the word embeddings algorithms

Using the unigram distribution for negative sampling

Implementing unigram-based negative sampling

Subsampling – probabilistically ignoring the common words

Implementing subsampling

Comparing the CBOW and its extensions

More recent algorithms extending skip-gram and CBOW

A limitation of the skip-gram algorithm

The structured skip-gram algorithm

The loss function

The continuous window model

GloVe – Global Vectors representation

Understanding GloVe

Implementing GloVe

Document classification with Word2vec

Dataset

Classifying documents with word embeddings

Implementation – learning word embeddings

Implementation – word embeddings to document embeddings

Document clustering and t-SNE visualization of embedded documents

Inspecting several outliers

Implementation – clustering/classification of documents with K-means

Summary

5. Sentence Classification with Convolutional Neural Networks

Introducing Convolution Neural Networks

CNN fundamentals

The power of Convolution Neural Networks

Understanding Convolution Neural Networks

Convolution operation

Standard convolution operation

Convolving with stride

Convolving with padding

Transposed convolution

Pooling operation

Max pooling

Max pooling with stride

Average pooling

Fully connected layers

Putting everything together

Exercise – image classification on MNIST with CNN

About the data

Implementing the CNN

Analyzing the predictions produced with a CNN

Using CNNs for sentence classification

CNN structure

Data transformation

The convolution operation

Pooling over time

Implementation – sentence classification with CNNs

Summary

6. Recurrent Neural Networks

Understanding Recurrent Neural Networks

The problem with feed-forward neural networks

Modeling with Recurrent Neural Networks

Technical description of a Recurrent Neural Network

Backpropagation Through Time

How backpropagation works

Why we cannot use BP directly for RNNs

Backpropagation Through Time – training RNNs

Truncated BPTT – training RNNs efficiently

Limitations of BPTT – vanishing and exploding gradients

Applications of RNNs

One-to-one RNNs

One-to-many RNNs

Many-to-one RNNs

Many-to-many RNNs

Generating text with RNNs

Defining hyperparameters

Unrolling the inputs over time for Truncated BPTT

Defining the validation dataset

Defining weights and biases

Defining state persisting variables

Calculating the hidden...