Logging in Action

With Fluentd, Kubernetes and more

Phil Wilkins

• Publisher: Manning
• Year: 2022
• ISBN:
  9781638355670
• Topic: Computer Science
• Subtopic: Software Development
• Index: Computer Science

Make log processing a real asset to your organization with powerful and free open source tools. In Logging in Action you will learn how to: Deploy Fluentd and Fluent Bit into traditional on-premises, IoT, hybrid, cloud, and multi-cloud environments, both small and hyperscaled
Configure Fluentd and Fluent Bit to solve common log management problems
Use Fluentd within Kubernetes and Docker services
Connect a custom log source or destination with Fluentd's extensible plugin framework
Logging best practices and common pitfalls Logging in Action is a guide to optimize and organize logging using the CNCF Fluentd and Fluent Bit projects. You'll use the powerful log management tool Fluentd to solve common log management, and learn how proper log management can improve performance and make management of software and infrastructure solutions easier. Through useful examples like sending log-driven events to Slack, you'll get hands-on experience applying structure to your unstructured data. About the technology
Don't fly blind! An effective logging system can help you see and correct problems before they cripple your software. With the Fluentd log management tool, it's a snap to monitor the behavior and health of your software and infrastructure in real time. Designed to collect and process log data from multiple sources using the industry-standard JSON format, Fluentd delivers a truly unified logging layer across all your systems. About the book
Logging in Action teaches you to record and analyze application and infrastructure data using Fluentd. Using clear, relevant examples, it shows you exactly how to transform raw system data into a unified stream of actionable information. You'll discover how logging configuration impacts the way your system functions and set up Fluentd to handle data from legacy IT environments, local data centers, and massive Kubernetes-driven distributed systems. You'll even learn how to implement complex log parsing with RegEx and output events to MongoDB and Slack. What's inside Capture log events from a wide range of systems and software, including Kubernetes and Docker
Connect to custom log sources and destinations
Employ Fluentd's extensible plugin framework
Create a custom plugin for niche problemsAbout the reader
For developers, architects, and operations professionals familiar with the basics of monitoring and logging. About the author
Phil Wilkins has spent over 30 years in the software industry. Has worked for small startups through to international brands.Table of Contents
PART 1 FROM ZERO TO "HELLO WORLD"
1 Introduction to Fluentd
2 Concepts, architecture, and deployment of Fluentd
PART 2 FLUENTD IN DEPTH
3 Using Fluentd to capture log events
4 Using Fluentd to output log events
5 Routing log events
6 Filtering and extrapolation
PART 3 BEYOND THE BASICS
7 Performance and scaling
8 Driving logs with Docker and Kubernetes
9 Creating custom plugins
PART 4 GOOD LOGGING PRACTICES AND FRAMEWORKS TO MAXIMIZE LOG VALUE
10 Logging best practices
11 Logging frameworks

Part 1. From zero to “Hello World”

Any good thriller starts by introducing its primary protagonists. Their motivations, backgrounds, and strengths and weaknesses are presented. The environment in which the key players operate is shown in the first 20 minutes.

This is what the first part of the book is about. The first chapter introduces our hero, Fluentd (and sibling Fluent Bit); we set the scene with the context, the use cases, and so on. If you are still in the process of discovering what Fluentd is about or thinking about the things that will help you make a case to your colleagues for adopting Fluentd, there is plenty of fuel for thought here.

If chapter 1 is about our principal player, then chapter 2 looks at the environments in which Fluentd can operate. We will progress through the first practical steps by installing Fluentd and keep with the time-honored tradition established by Brian Kernighan, with the first solution being “Hello World.”

1 Introduction to Fluentd

This chapter covers

• Examining use cases for logs and log events
• Identifying the value of log unification
• Differentiating between log analytics and unified logging
• Understanding monitoring concepts
• Understanding Fluentd and Fluent Bit

Before getting into the details of Fluentd, we should first focus on the motivations for using a tool such as Fluentd. How can logging help us? What are log analytics, and why is log unification necessary? These are among the questions we will work to answer in this chapter. We’ll highlight the kinds of activities logging can help or enable us to achieve.

Let’s also take a step back and understand some contemporary thinking around how systems are measured and monitored; understanding these ideas will mean we can use our tools more effectively. After all, a tool is only as good as the user creating the configuration or generating log events to be used.

As we do this, it is worth exploring how Fluentd has evolved and understanding why it holds its position within the industry. If you are considering Fluentd as a possible tool or looking to make a case for its adoption, then it is helpful to understand its “origin story,” as this will inform how Fluentd may be perceived.

1.1 Elevator pitch for Fluentd

Given that you’re looking at this book, we presume you have at least heard of Fluentd and probably have a vague sense of what it is. Let’s start with the “elevator pitch” as to what Fluentd and Fluent Bit are.

The primary purpose of Fluentd and its sibling Fluent Bit is to capture log events from a diverse range of possible sources (infrastructure such as network switches, OS, custom applications, and prebuilt applications, including Platform as a Service and Software as a Service). It then gets those events to an appropriate tool where the log events can be processed to extract meaning and insight, and possibly trigger actions. Fluentd’s primary job is not to perform detailed log analytics itself, although it can derive meaning, and deeper analysis could be incorporated into its configuration if needed.

By unifying the log events from all the sources of logs impacting the operation of our solution, we have the opportunity to see the big picture. For example, was the error in the database the cause of an error returned to a user by the application, or was the database error a symptom of the operating system not being able to write to storage?

1.1.1 What is a log event?

We’ve described Fluentd in terms of log events, so what qualifies as a log event? A log event is best described as the following:

• Log events are humanly readable information that is primarily textual in nature. The textual information can range from unstructured to highly structured.
• Each log event has a place in time, defined with a timestamp (usually absolute 01:00:00 1 Jan 1970, but could be relative +0.60), or time can be inferred by the log event’s position in a series of events.
• Each event also has an explicit or implicit association to a location that can be associated with a component running in a location that may be physical or logical.

Let’s illustrate the point. Anyone with some coding experience will probably recognize the screenshot shown in figure 1.1 as an extract of log output. In this case, the output is generated by Fluentd. As you can see, there is a timestamp for the event; a location, which comes from the host the events are occurring on; and some additional semistructured content.

Figure 1.1 Log output from Fluentd

1.1.2 Fluentd compared to middleware

Those who have worked with middleware (e.g., Apache Camel, MuleSoft, Oracle SOA Suite) will appreciate the idea of describing Fluentd as an enterprise service bus specialized in logs. Figure 1.2 suggests this, with the concept of input and output and capabilities to route and transform the log events. This will become ever more apparent as the book progresses.

Figure 1.2 Illustration showing different types of Fluentd plugins and their relationship to the core

NOTE If you’d like to explore this analogy further, you might consider reading the liveBook version of Open-Source ESBs in Action by Tijs Rademakers and Jos Dirksen (Manning, 2008) at http://mng.bz/Nx6n.

Definition Middleware is a generic term covering software that provides services to software applications beyond those available from the operating system. Often this entails connecting different pieces of software. It can sometimes be described as “software glue.”

Definition An enterprise service bus is a specific category of middleware for passing data in a near-real-time manner between pieces of software. This usually includes the sequencing of the execution of the different software components as well.

1.2 Why do we produce logs?

We create log entries for a wide range of reasons. Some of the use cases for logs are only needed a fraction of the time but are invaluable when needed. Nearly every use case we can think of will fall into one of the following categories:

• Debugging —Knowing which parts of the code are being executed in a scenario makes it easy to isolate a bug. Yes, we have debuggers, and so on, but often it’s just as easy to drop a few log lines in to help. Some of these log messages will be left in to provide assurance that things are running fine during production. Other lines of log messages may be disabled while we’re not developing and testing software. Note that we would never recommend trying to connect to a production environment with a debugger. Allowing a production system to log information intended for debugging should be done with an understanding of the possible consequences (later in the book, we’ll explore why this is so).
• Unexpected data values or abnormal conditions occurring —...