Hadoop Essentials
eBook - ePub

Hadoop Essentials

Shiva Achari

Share book
  1. 194 pages
  2. English
  3. ePUB (mobile friendly)
  4. Available on iOS & Android
eBook - ePub

Hadoop Essentials

Shiva Achari

Book details
Book preview
Table of contents
Citations

Frequently asked questions

How do I cancel my subscription?
Simply head over to the account section in settings and click on “Cancel Subscription” - it’s as simple as that. After you cancel, your membership will stay active for the remainder of the time you’ve paid for. Learn more here.
Can/how do I download books?
At the moment all of our mobile-responsive ePub books are available to download via the app. Most of our PDFs are also available to download and we're working on making the final remaining ones downloadable now. Learn more here.
What is the difference between the pricing plans?
Both plans give you full access to the library and all of Perlego’s features. The only differences are the price and subscription period: With the annual plan you’ll save around 30% compared to 12 months on the monthly plan.
What is Perlego?
We are an online textbook subscription service, where you can get access to an entire online library for less than the price of a single book per month. With over 1 million books across 1000+ topics, we’ve got you covered! Learn more here.
Do you support text-to-speech?
Look out for the read-aloud symbol on your next book to see if you can listen to it. The read-aloud tool reads text aloud for you, highlighting the text as it is being read. You can pause it, speed it up and slow it down. Learn more here.
Is Hadoop Essentials an online PDF/ePUB?
Yes, you can access Hadoop Essentials by Shiva Achari in PDF and/or ePUB format, as well as other popular books in Computer Science & Databases. We have over one million books available in our catalogue for you to explore.

Information

Year
2015
ISBN
9781784396688

Hadoop Essentials


Table of Contents

Hadoop Essentials
Credits
About the Author
Acknowledgments
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
1. Introduction to Big Data and Hadoop
V's of big data
Volume
Velocity
Variety
Understanding big data
NoSQL
Types of NoSQL databases
Analytical database
Who is creating big data?
Big data use cases
Big data use case patterns
Big data as a storage pattern
Big data as a data transformation pattern
Big data for a data analysis pattern
Big data for data in a real-time pattern
Big data for a low latency caching pattern
Hadoop
Hadoop history
Description
Advantages of Hadoop
Uses of Hadoop
Hadoop ecosystem
Apache Hadoop
Hadoop distributions
Pillars of Hadoop
Data access components
Data storage component
Data ingestion in Hadoop
Streaming and real-time analysis
Summary
2. Hadoop Ecosystem
Traditional systems
Database trend
The Hadoop use cases
Hadoop's basic data flow
Hadoop integration
The Hadoop ecosystem
Distributed filesystem
HDFS
Distributed programming
NoSQL databases
Apache HBase
Data ingestion
Service programming
Apache YARN
Apache Zookeeper
Scheduling
Data analytics and machine learning
System management
Apache Ambari
Summary
3. Pillars of Hadoop – HDFS, MapReduce, and YARN
HDFS
Features of HDFS
HDFS architecture
NameNode
DataNode
Checkpoint NameNode or Secondary NameNode
BackupNode
Data storage in HDFS
Read pipeline
Write pipeline
Rack awareness
Advantages of rack awareness in HDFS
HDFS federation
Limitations of HDFS 1.0
The benefit of HDFS federation
HDFS ports
HDFS commands
MapReduce
The MapReduce architecture
JobTracker
TaskTracker
Serialization data types
The Writable interface
WritableComparable interface
The MapReduce example
The MapReduce process
Mapper
Shuffle and sorting
Reducer
Speculative execution
FileFormats
InputFormats
RecordReader
OutputFormats
RecordWriter
Writing a MapReduce program
Mapper code
Reducer code
Driver code
Auxiliary steps
Combiner
Partitioner
Custom partitioner
YARN
YARN architecture
ResourceManager
NodeManager
ApplicationMaster
Applications powered by YARN
Summary
4. Data Access Components – Hive and Pig
Need of a data processing tool on Hadoop
Pig
Pig data types
The Pig architecture
The logical plan
The physical plan
The MapReduce plan
Pig modes
Grunt shell
Input data
Loading data
Dump
Store
FOREACH generate
Filter
Group By
Limit
Aggregation
Cogroup
DESCRIBE
EXPLAIN
ILLUSTRATE
Hive
The Hive architecture
Metastore
The Query compiler
The Execution engine
Data types and schemas
Installing Hive
Starting Hive shell
HiveQL
DDL (Data Definition Language) operations
DML (Data Manipulation Language) operations
The SQL operation
Joins
Aggregations
Built-in functions
Custom UDF (User Defined Functions)
Managing tables – external versus managed
SerDe
Partitioning
Bucketing
Summary
5. Storage Component – HBase
An Overview of HBase
Advantages of HBase
The Architecture of HBase
MasterServer
RegionServer
WAL
BlockCache
LRUBlockCache
SlabCache
BucketCache
Regions
MemStore
Zookeeper
The HBase data model
Logical components of a data model
ACID properties
The CAP theorem
The Schema design
The Write pipeline
The Read pipeline
Compaction
The Compaction policy
Minor compaction
Major compaction
Splitting
Pre-Splitting
Auto Splitting
Forced Splitting
Commands
help
Create
List
Put
Scan
Get
Disable
Drop
HBase Hive integration
Performance tuning
Compression
Filters
Counters
HBase coprocessors
Summary
6. Data Ingestion in Hadoop – Sqoop and Flume
Data sources
Challenges in data ingestion
Sqoop
Connectors and drivers
Sqoop 1 architecture
Limitation of Sqoop 1
Sqoop 2 architecture
Imports
Exports
Apache Flume
Reliability
Flume architecture
Multitier topology
Flume master
Flume nodes
Components in Agent
Source
Sink
Channels
Memory channel
File Channel
JDBC Channel
Examples of configuring Flume
The Single agent example
Multiple flows in an agent
Configuring a multiagent setup
Summary
7. Streaming and Real-time Analysis – Storm and Spark
An introduction to Storm
Features of Storm
Physical architecture of Storm
Data architecture of Storm
Storm topology
Storm on YARN
Topology configuration example
Spouts
Bolts
Topology
An introduction to Spark
Features of Spark
Spark framework
Spark SQL
GraphX
MLib
Spark streaming
Spark architecture
Directed Acyclic Graph engine
Resilient Distributed Dataset
Physical architecture
Operat...

Table of contents