Mastering Distributed Tracing

Analyzing performance in microservices and complex systems

Yuri Shkuro

• Publisher: Packt Publishing
• Year: 2019
• ISBN:
  9781788627597
• Edition: 1
• Topic: Computer Science
• Subtopic: Computer Science General
• Index: Computer Science

Understand how to apply distributed tracing to microservices-based architectures

Key Features

• A thorough conceptual introduction to distributed tracing
• An exploration of the most important open standards in the space
• A how-to guide for code instrumentation and operating a tracing infrastructure

Book Description

Mastering Distributed Tracing will equip you to operate and enhance your own tracing infrastructure. Through practical exercises and code examples, you will learn how end-to-end tracing can be used as a powerful application performance management and comprehension tool.

The rise of Internet-scale companies, like Google and Amazon, ushered in a new era of distributed systems operating on thousands of nodes across multiple data centers. Microservices increased that complexity, often exponentially. It is harder to debug these systems, track down failures, detect bottlenecks, or even simply understand what is going on. Distributed tracing focuses on solving these problems for complex distributed systems. Today, tracing standards have developed and we have much faster systems, making instrumentation less intrusive and data more valuable.

Yuri Shkuro, the creator of Jaeger, a popular open-source distributed tracing system, delivers end-to-end coverage of the field in Mastering Distributed Tracing. Review the history and theoretical foundations of tracing; solve the data gathering problem through code instrumentation, with open standards like OpenTracing, W3C Trace Context, and OpenCensus; and discuss the benefits and applications of a distributed tracing infrastructure for understanding, and profiling, complex systems.

What you will learn

• How to get started with using a distributed tracing system
• How to get the most value out of end-to-end tracing
• Learn about open standards in the space
• Learn about code instrumentation and operating a tracing infrastructure
• Learn where distributed tracing fits into microservices as a core function

Who this book is for

Any developer interested in testing large systems will find this book very revealing and in places, surprising. Every microservice architect and developer should have an insight into distributed tracing, and the book will help them on their way. System administrators with some development skills will also benefit. No particular programming language skills are required, although an ability to read Java, while non-essential, will help with the core chapters.

Mastering Distributed Tracing

---

Table of Contents

Mastering Distributed Tracing

Why subscribe?

Packt.com

Contributors

About the author

About the reviewer

About the illustrator

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

I. Introduction

1. Why Distributed Tracing?

Microservices and cloud-native applications

What is observability?

The observability challenge of microservices

Traditional monitoring tools

Metrics

Logs

Distributed tracing

My experience with tracing

Why this book?

Summary

References

2. Take Tracing for a HotROD Ride

Prerequisites

Running from prepackaged binaries

Running from Docker images

Running from the source code

Go language development environment

Jaeger source code

Start Jaeger

Meet the HotROD

The architecture

The data flow

Contextualized logs

Span tags versus logs

Identifying sources of latency

Resource usage attribution

Summary

References

3. Distributed Tracing Fundamentals

The idea

Request correlation

Black-box inference

Schema-based

Metadata propagation

Anatomy of distributed tracing

Sampling

Preserving causality

Inter-request causality

Trace models

Event model

Span model

Clock skew adjustment

Trace analysis

Summary

References

II. Data Gathering Problem

4. Instrumentation Basics with OpenTracing

Prerequisites

Project source code

Go development environment

Java development environment

Python development environment

MySQL database

Query tools (curl or wget)

Tracing backend (Jaeger)

OpenTracing

Exercise 1 – the Hello application

Hello application in Go

Hello application in Java

Hello application in Python

Exercise summary

Exercise 2 – the first trace

Step 1 – create a tracer instance

Create a tracer in Go

Create a tracer in Java

Create a tracer in Python

Step 2 – start a span

Start a span in Go

Start a span in Java

Start a span in Python

Step 3 – annotate the span

Annotate the span in Go

Annotate the span in Java

Annotate the span in Python

Exercise summary

Exercise 3 – tracing functions and passing context

Step 1 – trace individual functions

Trace individual functions in Go

Trace individual functions in Java

Trace individual functions in Python

Step 2 – combine multiple spans into a single trace

Combine multiple spans into a single trace in Go

Combine multiple spans into a single trace in Java

Combine multiple spans into a single trace in Python

Step 3 – propagate the in-process context

In-process context propagation in Python

In-process context propagation in Java

In-process context propagation in Go

Exercise summary

Exercise 4 – tracing RPC requests

Step 1 – break up the monolith

Microservices in Go

Microservices in Java

Microservices in Python

Step 2 – pass the context between processes

Passing context between processes in Go

Passing context between processes in Java

Passing context between processes in Python

Step 3 – apply OpenTracing-recommended tags

Standard tags in Go

Standard tags in Java

Standard tags in Python

Exercise summary

Exercise 5 – using baggage

Using baggage in Go

Using baggage in Java

Using baggage in Python

Exercise summary

Exercise 6 – auto-instrumentation

Open source instrumentation in Go

Auto-instrumentation in Java

Auto-instrumentation in Python

Exercise 7 – extra credit

Summary

References

5. Instrumentation of Asynchronous Applications

Prerequisites

Project source code

Java development environment

Kafka, Zookeeper, Redis, and Jaeger

The Tracing Talk chat application

Implementation

The lib module

AppId

Message

KafkaConfig and KafkaService

RedisConfig and RedisService

GiphyService

The chat-api service

The storage-service microservice

The giphy-service microservice

Running the application

Observing traces

Instrumenting with OpenTracing

Spring instrumentation

Tracer resolver

Redis instrumentation

Kafka instrumentation

Producing messages

Consuming messages

Instrumenting asynchronous code

Summary

References

6. Tracing Standards and Ecosystem

Styles of instrumentation

Anatomy of tracing deployment and interoperability

Five shades of tracing

Know your audience

The ecosystem

Tracing systems

Zipkin and OpenZipkin

Jaeger

SkyWalking

X-Ray, Stackdriver, and more

Standards projects

W3C Trace Context

W3C "Data Interchange Format"

OpenCensus

OpenTracing

Summary

References

7. Tracing with Service Meshes

Service meshes

Observability via a service mesh

Prerequisites

Project source code

Java development environment

Kubernetes

Istio

The Hello application

Distributed tracing with Istio

Using Istio to generate a service graph

Distributed context and routing

Summary

References

8. All About Sampling

Head-based consistent sampling

Probabilistic sampling

Rate limiting sampling

Guaranteed-throughput probabilistic sampling

Adaptive sampling

Local adaptive sampling

Global adaptive sampling

Goals

Theory

Architecture

Calculating sampling probability

Implications of adaptive sampling

Extensions

Context-sensitive sampling

Ad-hoc or debug sampling

How to deal with oversampling

Post-collection down-sampling

Throttling

Tail-based consistent sampling

Partial sampling

Summary

References

III. Getting Value from Tracing

9. Turning the Lights On

Tracing as a knowledge base

Service graphs

Deep, path-aware service graphs

Detecting architectural problems

Performance analysis

Critical path analysis

Recognizing trace patterns

Look for error markers

Look for the longest span on the critical path

Look out for missing details

Avoid sequential execution or "staircase"

Be wary when things finish at exactly the same time

Exemplars

Latency histograms

Long-term profiling

Summary

References

10. Distributed Context Propagation

Brown Tracing Plane

Pivot tracing

Chaos engineering

Traffic labeling

Testing in production

Debugging in production

Developing in production

Summary

References

11. Integration with Metrics and Logs

Three pillars of observability

Prerequisites

Project source code

Java development environment

Running the servers in Docker

Declaring index pattern in Kibana

Running the clients

The Hello application

Integration with metrics

Standard metrics via tracing instrumentation

Adding context to metrics

Context-aware metrics APIs

Integration with logs

Structured logging

Correlating logs with trace context

Context-aware logging APIs

Capturing logs in the tracing system

Do we need separate logging and tracing backends?

Summary

References

12. Gathering Insights with Data Mining

Feature extraction

Components of a data mining pipeline

Tracing backend

Trace completion trigger

Feature extractor

Aggregator

Feature extraction exercise

Prerequisites

Project source code

Running the servers in Docker

Defining index mapping in Elasticsearch

Java development environment

Microservices simulator

Running as a Docker image

Running from source

Verify

Define an index pattern in Kibana

The Span Count job

Trace completion trigger

Feature extractor

...