Getting Started with Google BERT

Build and train state-of-the-art natural language processing models using BERT

Sudharsan Ravichandiran

• Publisher: Packt Publishing
• Year: 2021
• Print ISBN: 9781838821593
• eBook ISBN: 9781838826239
• Edition: 1
• Topic: Computer Science
• Subtopic: Computer Science General
• Index: Computer Science

Kickstart your NLP journey by exploring BERT and its variants such as ALBERT, RoBERTa, DistilBERT, VideoBERT, and more with Hugging Face's transformers library

Key Features

• Explore the encoder and decoder of the transformer model
• Become well-versed with BERT along with ALBERT, RoBERTa, and DistilBERT
• Discover how to pre-train and fine-tune BERT models for several NLP tasks

Book Description

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer's encoder and decoder work.

You'll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you'll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT.

By the end of this BERT book, you'll be well-versed with using BERT and its variants for performing practical NLP tasks.

What you will learn

• Understand the transformer model from the ground up
• Find out how BERT works and pre-train it using masked language model (MLM) and next sentence prediction (NSP) tasks
• Get hands-on with BERT by learning to generate contextual word and sentence embeddings
• Fine-tune BERT for downstream tasks
• Get to grips with ALBERT, RoBERTa, ELECTRA, and SpanBERT models
• Get the hang of the BERT models based on knowledge distillation
• Understand cross-lingual models such as XLM and XLM-R
• Explore Sentence-BERT, VideoBERT, and BART

Who this book is for

This book is for NLP professionals and data scientists looking to simplify NLP tasks to enable efficient language understanding using BERT. A basic understanding of NLP concepts and deep learning is required to get the best out of this book.

Section 1 - Starting Off with BERT

In this section, we will familiarize ourselves with BERT. First, we will understand how the transformer works, and then we will explore BERT in detail. We will also get hands-on with BERT and learn how to use the pre-trained BERT model.

The following chapters are included in this section:

• Chapter 1, A Primer on Transformers
• Chapter 2, Understanding the BERT Model
• Chapter 3, Getting Hands–On with BERT

A Primer on Transformers

The transformer is one of the most popular state-of-the-art deep learning architectures that is mostly used for natural language processing (NLP) tasks. Ever since the advent of the transformer, it has replaced RNN and LSTM for various tasks. Several new NLP models, such as BERT, GPT, and T5, are based on the transformer architecture. In this chapter, we will look into the transformer in detail and understand how it works.

We will begin the chapter by getting a basic idea of the transformer. Then, we will learn how the transformer uses encoder-decoder architecture for a language translation task. Following this, we will inspect how the encoder of the transformer works in detail by exploring each of the encoder components. After understanding the encoder, we will deep dive into the decoder and look into each of the decoder components in detail. At the end of the chapter, we will put the encoder and decoder together and see how the transformer works as a whole.

In this chapter, we will learn the following topics:

• Introduction to the transformer
• Understanding the encoder of the transformer
• Understanding the decoder of the transformer
• Putting the encoder and decoder together
• Training the transformer

Introduction to the transformer

RNN and LSTM networks are widely used in sequential tasks such as next word prediction, machine translation, text generation, and more. However one of the major challenges with the recurrent model is capturing the long-term dependency.

To overcome this limitation of RNNs, a new architecture called Transformer was introduced in the paper Attention Is All You Need. The transformer is currently the state-of-the-art model for several NLP tasks. The advent of the transformer created a major breakthrough in the field of NLP and also paved the way for new revolutionary architectures such as BERT, GPT-3, T5, and more.

The transformer model...

Table of contents

• Title Page
• Copyright and Credits
• Dedication
• About Packt
• Contributors
• Preface
• Section 1 - Starting Off with BERT
• A Primer on Transformers
• Understanding the BERT Model
• Getting Hands-On with BERT
• Section 2 - Exploring BERT Variants
• BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT
• BERT Variants II - Based on Knowledge Distillation
• Section 3 - Applications of BERT
• Exploring BERTSUM for Text Summarization
• Applying BERT to Other Languages
• Exploring Sentence and Domain-Specific BERT
• Working with VideoBERT, BART, and More
• Assessments
• Other Books You May Enjoy