Apache Flume: Distributed Log Collection for Hadoop - Second Edition

Steve Hoffman

• Verlag: Packt Publishing
• Jahr: 2015
• ISBN:
  9781784392178
• Thema: Ciencia de la computación
• Thema: Programación de código abierto

Apache Flume: Distributed Log Collection for Hadoop Second Edition

---

Table of Contents

Apache Flume: Distributed Log Collection for Hadoop Second Edition

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Overview and Architecture

Flume 0.9

Flume 1.X (Flume-NG)

The problem with HDFS and streaming data/logs

Sources, channels, and sinks

Flume events

Interceptors, channel selectors, and sink processors

Tiered data collection (multiple flows and/or agents)

The Kite SDK

Summary

2. A Quick Start Guide to Flume

Downloading Flume

Flume in Hadoop distributions

An overview of the Flume configuration file

Starting up with "Hello, World!"

Summary

3. Channels

The memory channel

The file channel

Spillable Memory Channel

Summary

4. Sinks and Sink Processors

HDFS sink

Path and filename

File rotation

Compression codecs

Event Serializers

Text output

Text with headers

Apache Avro

User-provided Avro schema

File type

SequenceFile

DataStream

CompressedStream

Timeouts and workers

Sink groups

Load balancing

Failover

MorphlineSolrSink

Morphline configuration files

Typical SolrSink configuration

Sink configuration

ElasticSearchSink

LogStash Serializer

Dynamic Serializer

Summary

5. Sources and Channel Selectors

The problem with using tail

The Exec source

Spooling Directory Source

Syslog sources

The syslog UDP source

The syslog TCP source

The multiport syslog TCP source

JMS source

Channel selectors

Replicating

Multiplexing

Summary

6. Interceptors, ETL, and Routing

Interceptors

Timestamp

Host

Static

Regular expression filtering

Regular expression extractor

Morphline interceptor

Custom interceptors

The plugins directory

Tiering flows

The Avro source/sink

Compressing Avro

SSL Avro flows

The Thrift source/sink

Using command-line Avro

The Log4J appender

The Log4J load-balancing appender

The embedded agent

Configuration and startup

Sending data

Shutdown

Routing

Summary

7. Putting It All Together

Web logs to searchable UI

Setting up the web server

Configuring log rotation to the spool directory

Setting up the target – Elasticsearch

Setting up Flume on collector/relay

Setting up Flume on the client

Creating more search fields with an interceptor

Setting up a better user interface – Kibana

Archiving to HDFS

Summary

8. Monitoring Flume

Monitoring the agent process

Monit

Nagios

Monitoring performance metrics

Ganglia

Internal HTTP server

Custom monitoring hooks

Summary

9. There Is No Spoon – the Realities of Real-time Distributed Data Collection

Transport time versus log time

Time zones are evil

Capacity planning

Considerations for multiple data centers

Compliance and data expiry

Summary

Index

Apache Flume: Distributed Log Collection for Hadoop Second Edition

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: July 2013

Second edition: February 2015

Production reference: 2250315

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78439-217-8

www.packtpub.com

Credits

Author

Steve Hoffman

Reviewers

Sachin Handiekar

Michael Keane

Stefan Will

Commissioning Editor

Dipika Gaonkar

Acquisition Editor

Reshma Raman

Content Development Editor

Neetu Ann Mathew

Technical Editor

Menza Mathew

Copy Editors

Vikrant Phadke

Stuti Srivastava

Project Coordinator

Mary Alex

Proofreader

Simran Bhogal

Safis Editing

Indexer

Rekha Nair

Graphics

Sheetal Aute

Abhinash Sahu

Production Coordinator

Komal Ramchandani

Cover Work

Komal Ramchandani

About the Author

Steve Hoffman has 32 years of experience in software development, ranging from embedded software development ...