Data Lake for Enterprises

Tomcy John, Pankaj Misra

• Publisher: Packt Publishing
• Year: 2017
• eBook ISBN: 9781787282650
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Modelling & Design
• Index: Computer Science

A practical guide to implementing your enterprise data lake using Lambda Architecture as the baseAbout This Book• Build a full-fledged data lake for your organization with popular big data technologies using the Lambda architecture as the base• Delve into the big data technologies required to meet modern day business strategies• A highly practical guide to implementing enterprise data lakes with lots of examples and real-world use-casesWho This Book Is ForJava developers and architects who would like to implement a data lake for their enterprise will find this book useful. If you want to get hands-on experience with the Lambda Architecture and big data technologies by implementing a practical solution using these technologies, this book will also help you.What You Will Learn• Build an enterprise-level data lake using the relevant big data technologies• Understand the core of the Lambda architecture and how to apply it in an enterprise• Learn the technical details around Sqoop and its functionalities• Integrate Kafka with Hadoop components to acquire enterprise data• Use flume with streaming technologies for stream-based processing• Understand stream- based processing with reference to Apache Spark Streaming• Incorporate Hadoop components and know the advantages they provide for enterprise data lakes• Build fast, streaming, and high-performance applications using ElasticSearch• Make your data ingestion process consistent across various data formats with configurability• Process your data to derive intelligence using machine learning algorithmsIn DetailThe term "Data Lake" has recently emerged as a prominent term in the big data industry. Data scientists can make use of it in deriving meaningful insights that can be used by businesses to redefine or transform the way they operate. Lambda architecture is also emerging as one of the very eminent patterns in the big data landscape, as it not only helps to derive useful information from historical data but also correlates real-time data to enable business to take critical decisions. This book tries to bring these two important aspects — data lake and lambda architecture—together.This book is divided into three main sections. The first introduces you to the concept of data lakes, the importance of data lakes in enterprises, and getting you up-to-speed with the Lambda architecture. The second section delves into the principal components of building a data lake using the Lambda architecture. It introduces you to popular big data technologies such as Apache Hadoop, Spark, Sqoop, Flume, and ElasticSearch. The third section is a highly practical demonstration of putting it all together, and shows you how an enterprise data lake can be implemented, along with several real-world use-cases. It also shows you how other peripheral components can be added to the lake to make it more efficient.By the end of this book, you will be able to choose the right big data technologies using the lambda architectural patterns to build your enterprise data lake.Style and approachThe book takes a pragmatic approach, showing ways to leverage big data technologies and lambda architecture to build an enterprise-level data lake.

Indexed Data Store using Elasticsearch

In the previous chapter on Hadoop, we persisted the data in hand onto Hadoop (HDFS). Reading/querying data from Hadoop at a fast pace is an issue, and that's when an indexed data store such as Elasticsearch and its significance come forth in our Data Lake implementation.

As in other chapters in this part of the book, we will start off the chapter by explaining the layer where this technology will be used. We will then explain the reason for choosing this technology for this capability and start diving deep into Elasticsearch and its working. We will cover enough details on Elasticsearch so that you have adequate details to understand this technology. As always we will only give enough details and full deep dive is beyond the scope of this book.
We would then take you through a hands-on coding session, where you will first learn to install this technology and then see it in action. We will also make sure to connect you to the SCV use case that we are trying to implement using Data Lake.
Finally we will see scenarios where we should and should not use this indexed data store. We will wrap the chapter by looking into other technology options that could be used in place of Elasticsearch.

Let's dig into it!

Context in Data Lake: data storage and lambda speed layer

The Data Storage Layer is where data is persisted. Our core persistent store is HDFS but because of its inherent slowness in querying, we need to have a technology on top of it for fast reading/querying. After indexing, the appropriate speed views are created and kept in the Lambda Speed Layer. The Lambda Speed Layer is entrusted with indexing the data stored in HDFS for high performance and scalable querying of required data.

Elasticsearch is our choice of technology capable of doing this effectively. Elasticsearch is one of the de-facto choices for such a capability, and with not much deliberation this technology choice was made.

The following sections of this chapter aim at covering Elasticsearch in detail so that you get a clear picture of this technology as well as get to know the data storage layer in detail.

Data Storage and Lambda Speed Layer

In Chapter 2, Comprehensive Concepts of a Data Lake, you would have got a glimpse of Data Storage and Lambda Speed layer. The Data Storage layer’s responsibility is to persist the gathered data into a permanent place in our data lake. The Lambda Speed Layer’s responsibility is to index the data stored in Data Storage layer and create appropriate speed views. The following figure would refresh your memory and give you a good pictorial view of this layer:

Figure 01: Data Lake: Data Storage and Lambda Speed Layer

We have given enough details in a previous chapter on the Data Storage Layer so we don't want to repeat ourselves. The Lambda Speed layer stores the various speed views created from the data storage layer. Since it was just the previous chapter, your memory would also be quite fresh.

Data Storage and Lambda Speed Layer: technology mapping

This chapter aims at covering the storage mechanism for an indexed data store. This chapter dives deep into Elasticsearch, which is our choice for this capability.

The following figure (Figure 02), brings in the technology aspect to the conceptual architecture that we will be the following throughout this book. We will keep explaining each technology and its relevance in the overall architecture before we bring all the technologies together in the final part of this book (Part 3).

Figure 02: Technology mapping for Data Storage and Lambda Speed Layer

For our use case, SCV, we have already gathered data from various source system data stores and persisted in HDFS. Elasticsearch is a layer on top of HDFS, which does data indexing and makes sure that querying required data is very fast.

The next sections of this chapter give more details on Elasticsearch and will make you conversant with many important aspects and working of this technology.

What is Elasticsearch?

The Wikipedia description of Elasticsearch is as follows:

Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. Official clients are available in Java, .NET (C#), Python, Groovy and many other languages.

The given description does cover every aspect of what Elasticsearch is and also gives some details on how exactly it works and its core capabilities. It is distributed, scalable, enterprise-grade, and offers high performance querying and it does this by indexing the data available in HDFS. It also has rich API’s and also supports a variety of languages. Let's keep this section crisp as we have this entire chapter explaining Elasticsearch and its working.

Why Elasticsearch

For the capability that we are looking for, Elasticsearch is the leading technology and that's the main reason for this choice.

Some of the prominent reasons why Elasticsearch has been chosen as the technology of choice for the technical capability that we are looking for in our Data Lake implementation:

• Compatibility with Hadoop (as this is our persistent store)
• Distributed
• Scalable
• Capability of indexing data
• Highly performant (fast query and search)
• Battle hardened technology (enterprise-grade having all the capabilities required by an enterprise)
• Capability of handling a huge volume and variety of data
• Failover and data redundancy capability

History of Elasticsearch

Compared to other technologies in the Big Data arena, Elasticsearch is more recent with a short history. This doesn't mean that the technol...

Table of contents

• Title Page
• Copyright
• Credits
• Foreword
• About the Authors
• About the Reviewers
• www.PacktPub.com
• Customer Feedback
• Preface
• Introduction to Data
• Comprehensive Concepts of a Data Lake
• Lambda Architecture as a Pattern for Data Lake
• Applied Lambda for Data Lake
• Data Acquisition of Batch Data using Apache Sqoop
• Data Acquisition of Stream Data using Apache Flume
• Messaging Layer using Apache Kafka
• Data Processing using Apache Flink
• Data Store Using Apache Hadoop
• Indexed Data Store using Elasticsearch
• Data Lake Components Working Together
• Data Lake Use Case Suggestions