![]()
Apache Flume: Distributed Log Collection for Hadoop
![]()
Table of Contents
Apache Flume: Distributed Log Collection for Hadoop
Credits
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers and more
Why Subscribe?
Free Access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Errata
Piracy
Questions
1. Overview and Architecture
Flume 0.9
Flume 1.X (Flume-NG)
The problem with HDFS and streaming data/logs
Sources, channels, and sinks
Flume events
Interceptors, channel selectors, and sink processors
Tiered data collection (multiple flows and/or agents)
Summary
2. Flume Quick Start
Downloading Flume
Flume in Hadoop distributions
Flume configuration file overview
Starting up with "Hello World"
Summary
3. Channels
Memory channel
File channel
Summary
4. Sinks and Sink Processors
HDFS sink
Path and filename
File rotation
Compression codecs
Event serializers
Text output
Text with headers
Apache Avro
File type
Sequence file
Data stream
Compressed stream
Timeouts and workers
Sink groups
Load balancing
Failover
Summary
5. Sources and Channel Selectors
The problem with using tail
The exec source
The spooling directory source
Syslog sources
The syslog UDP source
The syslog TCP source
The multiport syslog TCP source
Channel selectors
Replicating
Multiplexing
Summary
6. Interceptors, ETL, and Routing
Interceptors
Timestamp
Host
Static
Regular expression filtering
Regular expression extractor
Custom interceptors
Tiering data flows
Avro Source/Sink
Command-line Avro
Log4J Appender
The Load Balancing Log4J Appender
Routing
Summary
7. Monitoring Flume
Monitoring the agent process
Monit
Nagios
Monitoring performance metrics
Ganglia
The internal HTTP server
Custom monitoring hooks
Summary
8. There Is No Spoon – The Realities of Real-time Distributed Data Collection
Transport time versus log time
Time zones are evil
Capacity planning
Considerations for multiple data centers
Compliance and data expiry
Summary
Index
![]()
Apache Flume: Distributed Log Collection for Hadoop
Copyright © 2013 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: July 2013
Production Reference: 1090713
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78216-791-4
www.packtpub.com
![]()
Author
Steve Hoffman
Reviewers
Subash D'Souza
Stefan Will
Acquisition Editor
Kunal Parikh
Commissioning Editor
Sharvari Tawde
Technical Editors
Jalasha D'costa
Mausam Kothari
Project Coordinator
Sherin Padayatty
Proofreader
Aaron Nash
Indexer
Monica Ajmera Mehta
Graphics
Valentina D'silva
Abhinash Sahu
Production Coordinator
Kirtee Shingan
Cover Work
Kirtee Shingan
![]()
Steve Hoffman has 30 years of software development experience and holds a B.S. in computer engineering from the University of Illinois Urbana-Champaign and a M.S. in computer science from the DePaul University. He is currently a Principal Engineer at Orbitz Worldwide.
More information on Steve can be found at http://bit.ly/bacoboy or on Twitter @bacoboy.
This is Steve's first book.
