Monitoring Elasticsearch

Dan Noble

• Publisher: Packt Publishing
• Year: 2016
• eBook ISBN: 9781784397807
• Edition: 1
• Topic: Computer Science
• Subtopic: System Administration
• Index: Computer Science

Monitor your Elasticsearch cluster's health, and diagnose and solve its performance and reliability issues

About This Book

• Understand common performance and reliability pitfalls in ElasticSearch
• Use popular monitoring tools such as ElasticSearch-head, BigDesk, Marvel, Kibana, and more
• This is a step-by-step guide with lots of case studies on solving real-world ElasticSearch cluster issues

Who This Book Is For

This book is for developers and system administrators who use ElasticSearch in a wide range of capacities. Prior knowledge of ElasticSearch and related technologies would be helpful, but is not necessary.

What You Will Learn

• Explore your cluster with ElasticSearch-head and BigDesk
• Access the underlying data of the ElasticSearch monitoring plugins using the ElasticSearch API
• Analyze your cluster's performance with Marvel
• Troubleshoot some of the common performance and reliability issues that come up when using ElasticSearch
• Analyze a cluster's historical performance, and get to the bottom of and recover from system failures
• Use and install various other tools and plugins such as Kibana and Kopf, which is helpful to monitor ElasticSearch

In Detail

ElasticSearch is a distributed search server similar to Apache Solr with a focus on large datasets, a schema-less setup, and high availability. This schema-free architecture allows ElasticSearch to index and search unstructured content, making it perfectly suited for both small projects and large big data warehouses with petabytes of unstructured data.

This book is your toolkit to teach you how to keep your cluster in good health, and show you how to diagnose and treat unexpected issues along the way. You will start by getting introduced to ElasticSearch, and look at some common performance issues that pop up when using the system. You will then see how to install and configure ElasticSearch and the ElasticSearch monitoring plugins. Then, you will proceed to install and use the Marvel dashboard to monitor ElasticSearch. You will find out how to troubleshoot some of the common performance and reliability issues that come up when using ElasticSearch. Finally, you will analyze your cluster's historical performance, and get to know how to get to the bottom of and recover from system failures.

This book will guide you through several monitoring tools, and utilizes real-world cases and dilemmas faced when using ElasticSearch, showing you how to solve them simply, quickly, and cleanly.

Style and approach

This is a step-by-step guide to monitoring your ElasticSearch cluster and correcting performance issues. It is filled with lots of in-depth, real-world use-cases on solving different ElasticSearch cluster issues.

Monitoring Elasticsearch

---

Table of Contents

Monitoring Elasticsearch

Credits

About the Author

About the Reviewers

www.PacktPub.com

eBooks, discount offers, and more

Why subscribe?

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Introduction to Monitoring Elasticsearch

An overview of Elasticsearch

Learning more about Elasticsearch

Data distribution, redundancy, and fault tolerance

Full-text search

Similar technologies

Apache Lucene

Solr

Ferret

Monitoring Elasticsearch

Resourcefulness and problem solving

Summary

2. Installation and the Requirements for Elasticsearch

Installing Elasticsearch

DEB/RPM installation

The yum and apt-get repositories

Ubuntu/Debian and apt-get

CentOS/RHEL and yum

Verification

Configuration files

Configuring an Elasticsearch cluster

Cluster name

Memory configuration

Open file limit

The maximum file limit

Updating max file descriptors on Ubuntu Linux

Enabling pluggable authentication modules

Verifying the open file limit

Disabling swapping

Understanding your cluster

Installing Elasticsearch-head

Installing Bigdesk

Marvel

Cluster requirements

Summary

3. Elasticsearch-head and Bigdesk

Cluster setup

Cluster configuration

Sample data

Elasticsearch-head

The Overview tab

Cluster states

Node and index actions

The Indices tab

The Browser tab

The Structured Query tab

The Any Request tab

The official website

Bigdesk

The Elasticsearch cat API

Background

Count

Health

Indices

Shards

Summary

4. Marvel Dashboard

Setting up Marvel

Upgrading Marvel

Configuring Marvel

Marvel agent configuration settings

Marvel index configuration

Understanding the Marvel dashboard

Overview dashboard

Indices dashboard

Nodes dashboard

Monitoring node failures

Summary

5. System Monitoring

Working with Kopf

Installing Kopf

The cluster page

The nodes page

The rest page

The more dropdown

Working with Logstash and Kibana

ELK

Installation

Installing Logstash

Loading NGINX logs

Installing Kibana

Working with Nagios

Installing Nagios

Command line tools for system and process management

top

tail

grep

ps

kill

free

du and df

Summary

6. Troubleshooting Performance and Reliability Issues

System configuration

The fielddata cache

Analyzing queries

Slow log

Improving query performance

High-cardinality fields

Querying smaller indices

Cold indices

The shard query cache

Script queries

Testing meticulously

System and data architecting

Hot-Warm architecture

Master nodes

Hot nodes

Warm nodes

Reducing disk size

Compression

Storing the _source and analyzed fields

Optimizing data ingestion

Bulk indexing operations

Drive configuration

Case studies

Node configuration

Query optimization

Web application performance

Summary

7. Node Failure and Post-Mortem Analysis

Diagnosing problems

OutOfMemoryError exceptions

Shard failures

Slow queries

Resolving OutOfMemoryError exceptions

Elasticsearch process crashes

Disk space

Resolving the issue

Reviewing some case studies

The ES process quits unexpectedly

Query requests slow and timing out

Summary

8. Looking Forward

Elasticsearch 5 overview

Performance and reliability

Data loss

Upgrading to Elasticsearch 5.0

When to upgrade

Monitoring Elasticsearch 5

Summary

Index

Monitoring Elasticsearch

Copyright © 2016 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the infor...

Table of contents

• Monitoring Elasticsearch