HDInsight Essentials - Second Edition

Rajesh Nadipalli

• Verlag: Packt Publishing
• Jahr: 2015
• ISBN:
  9781784399429
• Thema: Computer Science
• Thema: Data Warehousing

HDInsight Essentials Second Edition

---

Table of Contents

HDInsight Essentials Second Edition

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Instant updates on new Packt books

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Hadoop and HDInsight in a Heartbeat

Data is everywhere

Business value of big data

Hadoop concepts

Brief history of Hadoop

Core components

Hadoop cluster layout

HDFS overview

Writing a file to HDFS

Reading a file from HDFS

HDFS basic commands

YARN overview

YARN application life cycle

YARN workloads

Hadoop distributions

HDInsight overview

HDInsight and Hadoop relationship

Hadoop on Windows deployment options

Microsoft Azure HDInsight Service

HDInsight Emulator

Hortonworks Data Platform (HDP) for Windows

Summary

2. Enterprise Data Lake using HDInsight

Enterprise Data Warehouse architecture

Source systems

Data warehouse

Storage

Processing

User access

Provisioning and monitoring

Data governance and security

Pain points of EDW

The next generation Hadoop-based Enterprise data architecture

Source systems

Data Lake

Storage

Processing

User access

Provisioning and monitoring

Data governance, security, and metadata

Journey to your Data Lake dream

Ingestion and organization

Transformation (rules driven)

Access, analyze, and report

Tools and technology for Hadoop ecosystem

Use case powered by Microsoft HDInsight

Problem statement

Solution

Source systems

Storage

Processing

User access

Benefits

Summary

3. HDInsight Service on Azure

Registering for an Azure account

Azure storage

Provisioning an HDInsight cluster

Cluster topology

Provisioning using Azure PowerShell

Creating a storage container

Provisioning a new HDInsight cluster

HDInsight management dashboard

Dashboard

Monitor

Configuration

Exploring clusters using the remote desktop

Running a sample MapReduce

Deleting the cluster

HDInsight Emulator for the development

Installing HDInsight Emulator

Installation verification

Using HDInsight Emulator

Summary

4. Administering Your HDInsight Cluster

Monitoring cluster health

Name Node status

The Name Node Overview page

Datanode Status

Utilities and logs

Hadoop Service Availability

YARN Application Status

Azure storage management

Configuring your storage account

Monitoring your storage account

Managing access keys

Deleting your storage account

Azure PowerShell

Access Azure Blob storage using Azure PowerShell

Summary

5. Ingest and Organize Data Lake

End-to-end Data Lake solution

Ingesting to Data Lake using HDFS command

Connecting to a Hadoop client

Getting your files on the local storage

Transferring to HDFS

Loading data to Azure Blob storage using Azure PowerShell

Loading files to Data Lake using GUI tools

Storage access keys

Storage tools

CloudXplorer

Key benefits

Registering your storage account

Uploading files to your Blob storage

Using Sqoop to move data from RDBMS to Data Lake

Key benefits

Two modes of using Sqoop

Using Sqoop to import data (SQL to Hadoop)

Organizing your Data Lake in HDFS

Managing file metadata using HCatalog

Key benefits

Using HCatalog Command Line to create tables

Summary

6. Transform Data in the Data Lake

Transformation overview

Tools for transforming data in Data Lake

HCatalog

Persisting HCatalog metastore in a SQL database

Apache Hive

Hive architecture

Starting Hive in HDInsight

Basic Hive commands

Apache Pig

Pig architecture

Starting Pig in HDInsight node

Basic Pig commands

Pig or Hive

MapReduce

The mapper code

The reducer code

The driver code

Executing MapReduce on HDInsight

Azure PowerShell for execution of Hadoop jobs

Transformation for the OTP project

Cleaning data using Pig

Executing Pig script

Registering a refined and aggregate table using Hive

Executing Hive script

Reviewing results

Other tools used for transformation

Oozie

Spark

Summary

7. Analyze and Report from Data Lake

Data access overview

Analysis using Excel and Microsoft Hive ODBC driver

Prerequisites

Step 1 – installing the Microsoft Hive ODBC driver

Step 2 – creating Hive ODBC Data Source

Step 3 – importing data to Excel

Analysis using Excel Power Query

Prerequisites

Step 1 – installing the Microsoft Power Query for Excel

Step 2 – i...