- 178 pages
- English
- ePUB (mobile friendly)
- Available on iOS & Android
eBook - ePub
Apache Flume: Distributed Log Collection for Hadoop - Second Edition
Tools to learn more effectively
Saving Books
Keyword Search
Annotating Text
Listen to it instead
Information
Apache Flume: Distributed Log Collection for Hadoop Second Edition
Table of Contents
Apache Flume: Distributed Log Collection for Hadoop Second Edition
Credits
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
1. Overview and Architecture
Flume 0.9
Flume 1.X (Flume-NG)
The problem with HDFS and streaming data/logs
Sources, channels, and sinks
Flume events
Interceptors, channel selectors, and sink processors
Tiered data collection (multiple flows and/or agents)
The Kite SDK
Summary
2. A Quick Start Guide to Flume
Downloading Flume
Flume in Hadoop distributions
An overview of the Flume configuration file
Starting up with "Hello, World!"
Summary
3. Channels
The memory channel
The file channel
Spillable Memory Channel
Summary
4. Sinks and Sink Processors
HDFS sink
Path and filename
File rotation
Compression codecs
Event Serializers
Text output
Text with headers
Apache Avro
User-provided Avro schema
File type
SequenceFile
DataStream
CompressedStream
Timeouts and workers
Sink groups
Load balancing
Failover
MorphlineSolrSink
Morphline configuration files
Typical SolrSink configuration
Sink configuration
ElasticSearchSink
LogStash Serializer
Dynamic Serializer
Summary
5. Sources and Channel Selectors
The problem with using tail
The Exec source
Spooling Directory Source
Syslog sources
The syslog UDP source
The syslog TCP source
The multiport syslog TCP source
JMS source
Channel selectors
Replicating
Multiplexing
Summary
6. Interceptors, ETL, and Routing
Interceptors
Timestamp
Host
Static
Regular expression filtering
Regular expression extractor
Morphline interceptor
Custom interceptors
The plugins directory
Tiering flows
The Avro source/sink
Compressing Avro
SSL Avro flows
The Thrift source/sink
Using command-line Avro
The Log4J appender
The Log4J load-balancing appender
The embedded agent
Configuration and startup
Sending data
Shutdown
Routing
Summary
7. Putting It All Together
Web logs to searchable UI
Setting up the web server
Configuring log rotation to the spool directory
Setting up the target – Elasticsearch
Setting up Flume on collector/relay
Setting up Flume on the client
Creating more search fields with an interceptor
Setting up a better user interface – Kibana
Archiving to HDFS
Summary
8. Monitoring Flume
Monitoring the agent process
Monit
Nagios
Monitoring performance metrics
Ganglia
Internal HTTP server
Custom monitoring hooks
Summary
9. There Is No Spoon – the Realities of Real-time Distributed Data Collection
Transport time versus log time
Time zones are evil
Capacity planning
Considerations for multiple data centers
Compliance and data expiry
Summary
Index
Apache Flume: Distributed Log Collection for Hadoop Second Edition
Copyright © 2015 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: July 2013
Second edition: February 2015
Production reference: 2250315
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78439-217-8
www.packtpub.com
Credits
Author
Steve Hoffman
Reviewers
Sachin Handiekar
Michael Keane
Stefan Will
Commissioning Editor
Dipika Gaonkar
Acquisition Editor
Reshma Raman
Content Development Editor
Neetu Ann Mathew
Technical Editor
Menza Mathew
Copy Editors
Vikrant Phadke
Stuti Srivastava
Project Coordinator
Mary Alex
Proofreader
Simran Bhogal
Safis Editing
Indexer
Rekha Nair
Graphics
Sheetal Aute
Abhinash Sahu
Production Coordinator
Komal Ramchandani
Cover Work
Komal Ramchandani
About the Author
Steve Hoffman has 32 years of experience in software development, ranging from embedded software development ...
Table of contents
- Apache Flume: Distributed Log Collection for Hadoop Second Edition
Frequently asked questions
Yes, you can cancel anytime from the Subscription tab in your account settings on the Perlego website. Your subscription will stay active until the end of your current billing period. Learn how to cancel your subscription
No, books cannot be downloaded as external files, such as PDFs, for use outside of Perlego. However, you can download books within the Perlego app for offline reading on mobile or tablet. Learn how to download books offline
Perlego offers two plans: Essential and Complete
- Essential is ideal for learners and professionals who enjoy exploring a wide range of subjects. Access the Essential Library with 800,000+ trusted titles and best-sellers across business, personal growth, and the humanities. Includes unlimited reading time and Standard Read Aloud voice.
- Complete: Perfect for advanced learners and researchers needing full, unrestricted access. Unlock 1.4M+ books across hundreds of subjects, including academic and specialized titles. The Complete Plan also includes advanced features like Premium Read Aloud and Research Assistant.
We are an online textbook subscription service, where you can get access to an entire online library for less than the price of a single book per month. With over 1 million books across 990+ topics, we’ve got you covered! Learn about our mission
Look out for the read-aloud symbol on your next book to see if you can listen to it. The read-aloud tool reads text aloud for you, highlighting the text as it is being read. You can pause it, speed it up and slow it down. Learn more about Read Aloud
Yes! You can use the Perlego app on both iOS and Android devices to read anytime, anywhere — even offline. Perfect for commutes or when you’re on the go.
Please note we cannot support devices running on iOS 13 and Android 7 or earlier. Learn more about using the app
Please note we cannot support devices running on iOS 13 and Android 7 or earlier. Learn more about using the app
Yes, you can access Apache Flume: Distributed Log Collection for Hadoop - Second Edition by Steve Hoffman in PDF and/or ePUB format, as well as other popular books in Computer Science & Open Source Programming. We have over one million books available in our catalogue for you to explore.