Monitoring Hadoop

Gurmukh Singh

• Verlag: Packt Publishing
• Jahr: 2015
• ISBN:
  9781783281558
• Thema: Ciencia de la computación
• Thema: Tratamiento de datos

Monitoring Hadoop

---

Table of Contents

Monitoring Hadoop

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Errata

Piracy

Questions

1. Introduction to Monitoring

The need for monitoring

The monitoring tools available in the market

Nagios

Nagios architecture

Prerequisites for installing and configuring Nagios

Prerequisites

Installing Nagios

Web interface configuration

Nagios plugins

Verification

Configuration files

Setting up monitoring for clients

Ganglia

Ganglia components

Ganglia installation

System logging

Collection

Transportation

Storage

Alerting and analysis

The syslogd and rsyslogd daemons

Summary

2. Hadoop Daemons and Services

Hadoop daemons

NameNode

DataNode and TaskTracker

Secondary NameNode

JobTracker and YARN daemons

The communication between daemons

YARN framework

Common issues faced on Hadoop cluster

Host-level checks

Nagios server

Configuring Hadoop nodes for monitoring

Summary

3. Hadoop Logging

The need for logging events

System logging

Logging levels

Logging in Hadoop

Hadoop logs

Hadoop log level

Hadoop audit

Summary

4. HDFS Checks

HDFS overview

Nagios master configuration

The Nagios client configuration

Summary

5. MapReduce Checks

MapReduce overview

MapReduce control commands

MapReduce health checks

Nagios master configuration

Nagios client configuration

Summary

6. Hadoop Metrics and Visualization Using Ganglia

Hadoop metrics

Metrics contexts

Named contexts

Metrics system design

Metrics configuration

Configuring Metrics2

Exploring the metrics contexts

Hadoop Ganglia integration

Hadoop metrics configuration for Ganglia

Setting up Ganglia nodes

Hadoop configuration

Metrics1

Metrics2

Ganglia graphs

Metrics APIs

The org.apache.hadoop.metrics package

The org.apache.hadoop.metrics2 package

Summary

7. Hive, HBase, and Monitoring Best Practices

Hive monitoring

Hive metrics

HBase monitoring

HBase Nagios monitoring

HBase metrics

Monitoring best practices

The Filter class

Nagios and Ganglia best practices

Summary

Index

Monitoring Hadoop

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: April 2015

Production reference: 1240415

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78328-155-8

www.packtpub.com

Credits

Author

Gurmukh Singh

Reviewers

David Greco

Randal Scott King

Yousuf Qureshi

Acquisition Editor

Meeta Rajani

Content Development Editor

Siddhesh Salvi

Technical Editor

Parag Topre

Copy Editors

Hiral Bhat

Sarang Chari

Tani Kothari

Trishla Singh

Project Coordinator

Nidhi Joshi

Proofreaders

Safis Editing

Paul Hindle

Indexer

Hemangini Bari

Graphics

Disha Haria

Production Coordinator

Melwyn D'sa

Cover Work

Melwyn D'sa

About the Author

Gurmukh Singh has been an infrastructure engineer for over 10 years and has worked on big data platforms in the past 5 years. He started his career as a field engineer, setting up lease lines and radio links. He has vast experience in enterprise servers and network design and in scaling infrastructures and tuning them for performance. He is the founder of a small start-up called Netxillon Technologies, which is into big data training and consultancy. He talks at various technical meetings and is an active participant in the open source community's activities. He writes at http://linuxaddict.org and maintains his Github account at https://github.com/gdhillon.

About the Reviewers

David Greco is a software architect with more than 27 years of experience. He started his career as ...