Learning ELK Stack
eBook - ePub

Learning ELK Stack

  1. 206 pages
  2. English
  3. ePUB (mobile friendly)
  4. Available on iOS & Android
eBook - ePub

Learning ELK Stack

About this book

Build mesmerizing visualizations, analytics, and logs from your data using Elasticsearch, Logstash, and Kibana

About This Book

  • Solve all your data analytics problems with the ELK stack
  • Explore the power of Kibana4 search and visualizations built over Elasticsearch queries and learn about the features and plugins of Logstash
  • Develop a complete data pipeline using the ELK stack

Who This Book Is For

If you are a developer or DevOps engineer interested in building a system that provides amazing insights and business metrics out of data sources, of various formats and types, using the open source technology stack that ELK provides, then this book is for you. Basic knowledge of Unix or any programming language will be helpful to make the most out of this book.

What You Will Learn

  • Install, configure, and run Elasticsearch, Logstash, and Kibana
  • Understand the need for log analytics and the current challenges in log analysis
  • Build your own data pipeline using the ELK stack
  • Familiarize yourself with the key features of Logstash and the variety of input, filter, and output plugins it provides
  • Build your own custom Logstash plugin
  • Create actionable insights using charts, histograms, and quick search features in Kibana4
  • Understand the role of Elasticsearch in the ELK stack

In Detail

The ELK stack—Elasticsearch, Logstash, and Kibana, is a powerful combination of open source tools. Elasticsearch is for deep search and data analytics. Logstash is for centralized logging, log enrichment, and parsing. Kibana is for powerful and beautiful data visualizations. In short, the Elasticsearch ELK stack makes searching and analyzing data easier than ever before.

This book will introduce you to the ELK (Elasticsearch, Logstash, and Kibana) stack, starting by showing you how to set up the stack by installing the tools, and basic configuration. You'll move on to building a basic data pipeline using the ELK stack.

Next, you'll explore the key features of Logstash and its role in the ELK stack, including creating Logstash plugins, which will enable you to use your own customized plugins. The importance of Elasticsearch and Kibana in the ELK stack is also covered, along with various types of advanced data analysis, and a variety of charts, tables, and maps.

Finally, by the end of the book you will be able to develop full-fledged data pipeline using the ELK stack and have a solid understanding of the role of each of the components.

Style and approach

This book is a step-by-step guide, complete with various examples to solve your data analytics problems by using the ELK stack to explore and visualize data.

Tools to learn more effectively

Saving Books

Saving Books

Keyword Search

Keyword Search

Annotating Text

Annotating Text

Listen to it instead

Listen to it instead

Information

Learning ELK Stack


Table of Contents

Learning ELK Stack
Credits
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
1. Introduction to ELK Stack
The need for log analysis
Issue debugging
Performance analysis
Security analysis
Predictive analysis
Internet of things and logging
Challenges in log analysis
Non-consistent log format
Tomcat logs
Apache access logs – combined log format
IIS logs
Variety of time formats
Decentralized logs
Expert knowledge requirement
The ELK Stack
Elasticsearch
Logstash
Kibana
ELK data pipeline
ELK Stack installation
Installing Elasticsearch
Running Elasticsearch
Elasticsearch configuration
Network Address
Paths
The cluster name
The node name
Elasticsearch plugins
Installing Logstash
Running Logstash
Logstash with file input
Logstash with Elasticsearch output
Configuring Logstash
Installing Logstash forwarder
Logstash plugins
Input plugin
Filters plugin
Output plugin
Installing Kibana
Configuring Kibana
Running Kibana
Kibana interface
Discover
Visualize
Dashboard
Settings
Summary
2. Building Your First Data Pipeline with ELK
Input dataset
Data format for input dataset
Configuring Logstash input
Filtering and processing input
Putting data to Elasticsearch
Visualizing with Kibana
Running Kibana
Kibana visualizations
Building a line chart
Building a bar chart
Building a Metric
Building a data table
Summary
3. Collect, Parse and Transform Data with Logstash
Configuring Logstash
Logstash plugins
Listing all plugins in Logstash
Data types for plugin properties
Array
Boolean
Codec
Hash
String
Comments
Field references
Logstash conditionals
Types of Logstash plugins
Input plugins
file
Configuration options
add_field
codec
delimiter
exclude
path
sincedb_path
sincedb_write_interval
start_position
tags
type
stdin
Configuration options
add_field
codec
tags
type
twitter
Configuration options
add_field
codec
consumer_key
consumer_secret
full_tweet
keywords
oauth_token
oauth_token_secret
tags
type
lumberjack
Configuration options
add_field
codec
host
port
ssl_certificate
ssl_key
ssl_key_passphrase
tags
type
redis
Configuration options
add_field
codec
data_type
host
key
password
port
Output plugins
csv
Configuration options
codec
csv_options
fields
gzip
path
file
Configuration options
email
Configuration options
attachments
body
cc
from
to
htmlbody
replyto
subject
elasticsearch
Configuration options
ganglia
Configuration options
metric
unit
value
jira
Configuration options
kafka
Configuration options
topic_id
lumberjack
Configuration options
hosts
port
ssl_certificate
redis
Configuration options
rabbitmq
stdout
mongodb
Configuration options
collection
database
uri
Filter plugins
csv
Configuration options
date
Configuration options
drop
Configuration options
geoip
Configuration options
source
grok
Custom grok patterns
mutate
Configuration options
sleep
Codec plugins
json
line
multiline
plain
rubydebug
Summary
4. Creating Custom Logstash Plug...

Table of contents

  1. Learning ELK Stack

Frequently asked questions

Yes, you can cancel anytime from the Subscription tab in your account settings on the Perlego website. Your subscription will stay active until the end of your current billing period. Learn how to cancel your subscription
No, books cannot be downloaded as external files, such as PDFs, for use outside of Perlego. However, you can download books within the Perlego app for offline reading on mobile or tablet. Learn how to download books offline
Perlego offers two plans: Essential and Complete
  • Essential is ideal for learners and professionals who enjoy exploring a wide range of subjects. Access the Essential Library with 800,000+ trusted titles and best-sellers across business, personal growth, and the humanities. Includes unlimited reading time and Standard Read Aloud voice.
  • Complete: Perfect for advanced learners and researchers needing full, unrestricted access. Unlock 1.4M+ books across hundreds of subjects, including academic and specialized titles. The Complete Plan also includes advanced features like Premium Read Aloud and Research Assistant.
Both plans are available with monthly, semester, or annual billing cycles.
We are an online textbook subscription service, where you can get access to an entire online library for less than the price of a single book per month. With over 1 million books across 990+ topics, we’ve got you covered! Learn about our mission
Look out for the read-aloud symbol on your next book to see if you can listen to it. The read-aloud tool reads text aloud for you, highlighting the text as it is being read. You can pause it, speed it up and slow it down. Learn more about Read Aloud
Yes! You can use the Perlego app on both iOS and Android devices to read anytime, anywhere — even offline. Perfect for commutes or when you’re on the go.
Please note we cannot support devices running on iOS 13 and Android 7 or earlier. Learn more about using the app
Yes, you can access Learning ELK Stack by Saurabh Chhajed in PDF and/or ePUB format, as well as other popular books in Computer Science & Databases. We have over one million books available in our catalogue for you to explore.