Learning Objectives
By the end of this chapter, you will be able to:
- Describe what natural language processing (NLP) is all about
- Describe the history of NLP
- Differentiate between NLP and Text Analytics
- Implement various preprocessing tasks
- Describe the various phases of an NLP project
In this chapter, you will learn about the basics of natural language processing and various preprocessing steps that are required to clean and analyze the data.
Introduction
To start with looking at NLP, let's understand what natural language is. In simple terms, it's the language we use to express ourselves. It's a basic means of communication. To define more specifically, language is a mutually agreed set of protocols involving words/sounds we use to communicate with each other.
In this era of digitization and computation, we tend to comprehend language scientifically. This is because we are constantly trying to make inanimate objects understand us. Thus, it has become essential to develop mechanisms by which language can be fed to inanimate objects such as computers. NLP helps us do this.
Let's look at an example. You must have some emails in your mailbox that have been automatically labeled as spam. This is done with the help of NLP. Here, an inanimate object – the email service – analyzes the content of the emails, comprehends it, and then further decides whether these emails need to be marked as spam or not.
History of NLP
NLP is an area that overlaps with others. It has emerged from fields such as artificial intelligence, linguistics, formal languages, and compilers. With the advancement of computing technologies and the increased availability of data, the way natural language is being processed has changed. Previously, a traditional rule-based system was used for computations. Today, computations on natural language are being done using machine learning and deep learning techniques.
The major work on machine learning-based NLP started during the 1980s. During the 1980s, developments across various disciplines such as artificial intelligence, linguistics, formal languages, and computations led to the emergence of an interdisciplinary subject called NLP. In the next section, we'll look at text analytics and how it differs from NLP.
Text Analytics and NLP
Text analytics is the method of extracting meaningful insights and answering questions from text data. This text data need not be a human language. Let's understand this with an example. Suppose you have a text file that contains your outgoing phone calls and SMS log data in the following format:
Figure 1.1: Format of call data
In the preceding figure, the first two fields represent the date and time at which the call was made or the SMS was sent. The third field represents the type of data. If the data is of the call type, then the value for this field will be set as voice_call. If the type of data is sms, the value of this field will be set to sms. The fourth field is for the phone number and name of the contact. If the number of the person is not in the contact list, then the name value will be left blank. The last field is for the duration of the call or text message. If the type of the data is voice_call, then the value in this field will be the duration of that call. If the type of data is sms, then the value in this field will be the text message.
The following figure shows records of call data stored in a text file:
Figure 1.2: Call records in a text file
Now, the data shown in the preceding figure is not exactly a human language. But it contains various information that can be extracted by analyzing it. A couple of questions that can be answered by looking at this data are as follows:
- How many New Year greetings were sent by SMS on 1st January?
- How many people were contacted whose name is not in the contact list?
The art of extracting useful insights from any given text data can be referred to as text analytics. NLP, on the other hand, is not just restricted to text data. Voice (speech) recognition and analysis also come under the domain of NLP. NLP can be broadly categorized into two types: Natural Language Understanding (NLU) and Natural Language Generation (NLG). A proper explanation of these terms is provided as follows:
- NLU: NLU refers to a process by which an inanimate object with computing power is able to comprehend spoken language.
- NLG: NLG refers to a process by which an inanimate object with computing power is able to manifest its thoughts in a language that humans are able to understand.
For example, when a human speaks to a machine, the machine interprets the human language with the help of the NLU process. Also, by using the NLG process, the machine generates an appropriate response and shares that with the human, thus making it easier for humans to understand. These tasks, which are part of NLP, are not part of text analytics. Now we will look at an exercise that will give us a better understanding of text analytics.
Exercise 1: Basic Text Analytics
In this exercise, we will perform some basic text analytics on the given text data. Follow these steps to implement this exercise:
- Open a Jupyter notebook.
- Insert a new cell. Assign a sentence variable with 'The quick brown fox jumps over the lazy dog'. Insert a new cell and add the following code to implement this:
sentence = 'The quick brown fox jumps over the lazy dog'
- Check whether the word 'quick' belongs to that text using the following code:
'quick' in sentence
The preceding code will return the output 'True'.
- Find out the index value of the word 'fox' using the following code:
sentence.index('fox')
The code will return the output 16.
- To find out the rank of the word 'lazy', use the following code:
sentence.split().index('lazy')
The code generates the output 7.
- For printing the third word of the given text, use the following code:
sentence.split()[2]
This will return the output 'brown'.
- To print the third word of the given sentence in reverse order, use the following code:
sentence.split()[2][::-1]
This will return the output 'nworb'.
- To concatenate the first and last words of the given sentence, use the following cod...
