Getting Started with Elastic Stack 8.0

Asjad Athick, Shay Banon

• Publisher: Packt Publishing
• Year: 2022
• eBook ISBN: 9781800564107
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Modelling & Design
• Index: Computer Science

Use the Elastic Stack for search, security, and observability-related use cases while working with large amounts of data on-premise and on the cloudKey Features• Learn the core components of the Elastic Stack and how they work together• Build search experiences, monitor and observe your environments, and defend your organization from cyber attacks• Get to grips with common architecture patterns and best practices for successfully deploying the Elastic StackBook DescriptionThe Elastic Stack helps you work with massive volumes of data to power use cases in the search, observability, and security solution areas.This three-part book starts with an introduction to the Elastic Stack with high-level commentary on the solutions the stack can be leveraged for. The second section focuses on each core component, giving you a detailed understanding of the component and the role it plays. You'll start by working with Elasticsearch to ingest, search, analyze, and store data for your use cases. Next, you'll look at Logstash, Beats, and Elastic Agent as components that can collect, transform, and load data. Later chapters help you use Kibana as an interface to consume Elastic solutions and interact with data on Elasticsearch. The last section explores the three main use cases offered on top of the Elastic Stack. You'll start with a full-text search and look at real-world outcomes powered by search capabilities. Furthermore, you'll learn how the stack can be used to monitor and observe large and complex IT environments. Finally, you'll understand how to detect, prevent, and respond to security threats across your environment. The book ends by highlighting architecture best practices for successful Elastic Stack deployments.By the end of this book, you'll be able to implement the Elastic Stack and derive value from it.What you will learn• Configure Elasticsearch clusters with different node types for various architecture patterns• Ingest different data sources into Elasticsearch using Logstash, Beats, and Elastic Agent• Build use cases on Kibana including data visualizations, dashboards, machine learning jobs, and alerts• Design powerful search experiences on top of your data using the Elastic Stack• Secure your organization and learn how the Elastic SIEM and Endpoint Security capabilities can help• Explore common architectural considerations for accommodating more complex requirementsWho this book is forDevelopers and solutions architects looking to get hands-on experience with search, security, and observability-related use cases on the Elastic Stack will find this book useful. This book will also help tech leads and product owners looking to understand the value and outcomes they can derive for their organizations using Elastic technology. No prior knowledge of the Elastic Stack is required.

Section 1: Core Components

This section offers a quick introduction to the core components of the Elastic Stack: Elasticsearch, Kibana, Logstash, and Beats.

This section includes the following chapters:

• Chapter 1, Introduction to the Elastic Stack
• Chapter 2, Installing and Running the Elastic Stack

Chapter 1: Introduction to the Elastic Stack

Welcome to Getting Started with Elastic Stack 8.0. The Elastic Stack has exploded in popularity over the last couple of years, becoming the de facto standard for centralized logging and "big data"-related use cases. The stack is leveraged by organizations, both big and small, across the world to solve a range of data-related problems. Hunting for adversaries in your network, looking for fraudulent transactions, real-time monitoring and alerting in systems, and searching for relevant products in catalogs are some of the real-world applications of the Elastic Stack.

The Elastic Stack is a bundle of multiple core products that integrate with each other. We will look at each product briefly in this chapter, and then dive into each one in later chapters in this book. The Elastic Stack attracts a great deal of interest from developers and architects that are working for organizations of all sizes. This book aims to serve as the go-to guide for those looking to get started with the best practices when it comes to building real-world search, security, and observability platforms using this technology.

In this chapter, you will learn a little bit about each component that makes up the Elastic Stack, and how they can be leveraged for your use cases. Those of you who are beginners or intermediary learners of this subject will benefit from this content to gain useful context for Chapter 3, Indexing and Searching for Data, to Chapter 13, Architecting Workloads on the Elastic Stack, of this book.

Specifically, we will cover the following topics:

• An overview of the Elastic Stack
• An introduction to Elasticsearch
• Visualizing and interacting with data on Kibana
• Ingesting various data sources using Logstash and Beats
• End-to-end solutions on the Elastic Stack

An overview of the Elastic Stack

The Elastic Stack is made up of four core products:

• Elasticsearch is a full-text search engine and a versatile data store. It can store and allow you to search and compute aggregations on large volumes of data quickly.
• Kibana provides a user interface for Elasticsearch. Users can search for and create visualizations, and then administer Elasticsearch, using this tool. Kibana also offers out-of-the-box solutions (in the form of apps) for use cases such as search, security, and observability.
• Beats can be used to collect and ship data directly from a range of source systems (such as different types of endpoints, network and infrastructure appliances, or cloud-based API sources) into Logstash or Elasticsearch.
• Logstash is an Extract, Transform, and Load (ETL) tool that's used to process and ingest data from various sources (such as log files on servers, Beats agents in your environment, or message queues and streaming platforms) into Elasticsearch.

This diagram shows how the core components of the Elastic Stack work together to ingest, store, and search on data:

Figure 1.1 – Components of the Elastic Stack

Each core component solves a single, common data-related problem. This genericity makes the stack flexible and domain-agnostic, allowing it to be adopted in multiple solution areas. Most users start with a simple logging use case where data is collected, parsed, and stored in Elasticsearch to create dashboards and alerts. Others might create more sophisticated capabilities, such as a workplace search to make information across a range of data sources accessible to your team; leveraging SIEM and machine learning to look for anomalous user/machine behavior and hunt for adversaries on your company network; understanding performance bottlenecks in applications; and monitoring infrastructure logs/metrics to respond to issues on critical systems.

The evolution of the Elastic Stack

Multiple independent projects have evolved over the years to create the present-day version of the Elastic Stack. Knowing how these components evolved indicates some of the functional gaps that existed in the big data space and how the Elastic Stack components come together to solve these challenges. Let's take a look:

1. An open source transactional Object/Search Engine Mapping (OSEM) framework for Java called Compass was released. Compass leveraged Lucene, an open source search engine library for implementing high-performance full-text search and indexing functionality.
2. To address scalability concerns in Compass, it was rewritten as a distributed search engine called Elasticsearch. Elasticsearch implemented RESTful APIs over HTTP using JSON, allowing programming languages other than Java to interact with it. Elasticsearch quickly gained popularity in the open source community.
3. As Elasticsearch was adopted by the community, a modular tool called Logstash was being developed to collect, transform, and send logs to a range of target systems. Elasticsearch was one of the target systems supported by Logstash.
4. Kibana was written to act as a user interface for using the REST APIs on Elasticsearch to search for and visualize data. Elasticsearch, Logstash, and Kibana were commonly referred to as the ELK Stack.
5. Elastic started providing managed Elasticsearch clusters on the cloud. Elastic Cloud Enterprise (ECE) was offered for customers to orchestrate and manage Elasticsearch deployments on-premises or on private cloud infrastructure.
6. An open source tool called Packetbeat was created to collect and ship network packet data to Elasticsearch. This later evolved into the Beats project, a collection of lightweight agents designed to collect and ship several types of data into Elasticsearch.
7. Machine learning capabilities were added to Elasticsearch and Kibana to support anomaly detection use cases on data residing on Elasticsearch.
8. Application Performance Monitoring (APM) capabilities were added to the Elastic Stack. The APM app on Kibana, together with the Logs, Metrics, and Uptime apps, formed the Observability solution.
9. Kibana added security analytics functionality as part of the Security Information and Event Management (SIEM) app.
10. A collection of proprietary features known as X-Pack was made open source under the Elastic licensing model.
11. Endpoint Detection and Response (EDR) capabilities were added to the Elastic Stack. EDR and SIEM capabilities formed the Security solution.
12. Out-of-the-box website, application, and content search functionality was offered as part of the Enterprise Search solution.

A note about licensing

The core components of the stack are open source software projects, licensed under a mix of the Apache 2, Elastic License version 2 (ELv2), and Server Side Public License (SSPL) licensing agreements. The LICENSE.txt file in the root of each product's GitHub repository should explain how the code is licensed.

A...

Table of contents

• Getting Started with Elastic Stack 8.0
• Foreword
• Preface
• Section 1: Core Components
• Chapter 1: Introduction to the Elastic Stack
• Chapter 2: Installing and Running the Elastic Stack
• Section 2: Working with the Elastic Stack
• Chapter 3: Indexing and Searching for Data
• Chapter 4: Leveraging Insights and Managing Data on Elasticsearch
• Chapter 5: Running Machine Learning Jobs on Elasticsearch
• Chapter 6: Collecting and Shipping Data with Beats
• Chapter 7: Using Logstash to Extract, Transform, and Load Data
• Chapter 8: Interacting with Your Data on Kibana
• Chapter 9: Managing Data Onboarding with Elastic Agent
• Section 3: Building Solutions with the Elastic Stack
• Chapter 10: Building Search Experiences Using the Elastic Stack
• Chapter 11: Observing Applications and Infrastructure Using the Elastic Stack
• Chapter 12: Security Threat Detection and Response Using the Elastic Stack
• Chapter 13: Architecting Workloads on the Elastic Stack
• Other Books You May Enjoy