Event Streams in Action

Real-time event systems with Kafka and Kinesis

Valentin Crettaz, Alexander Dean

• Publisher: Manning
• Year: 2019
• ISBN:
  9781638355830
• Topic: Computer Science
• Subtopic: Data Processing
• Index: Computer Science

Summary Event Streams in Action is a foundational book introducing the ULP paradigm and presenting techniques to use it effectively in data-rich environments.Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the Technology Many high-profile applications, like LinkedIn and Netflix, deliver nimble, responsive performance by reacting to user and system events as they occur. In large-scale systems, this requires efficiently monitoring, managing, and reacting to multiple event streams. Tools like Kafka, along with innovative patterns like unified log processing, help create a coherent data processing architecture for event-based applications. About the Book Event Streams in Action teaches you techniques for aggregating, storing, and processing event streams using the unified log processing pattern. In this hands-on guide, you'll discover important application designs like the lambda architecture, stream aggregation, and event reprocessing. You'll also explore scaling, resiliency, advanced stream patterns, and much more! By the time you're finished, you'll be designing large-scale data-driven applications that are easier to build, deploy, and maintain. What's inside

• Validating and monitoring event streams
• Event analytics
• Methods for event modeling
• Examples using Apache Kafka and Amazon Kinesis

About the Reader For readers with experience coding in Java, Scala, or Python. About the Author Alexander Dean developed Snowplow, an open source event processing and analytics platform. Valentin Crettaz is an independent IT consultant with 25 years of experience. Table of Contents

PART 1 - EVENT STREAMS AND UNIFIED LOGS

• Introducing event streams
• The unified log 24
• Event stream processing with Apache Kafka
• Event stream processing with Amazon Kinesis
• Stateful stream processing

PART 2- DATA ENGINEERING WITH STREAMS

• Schemas
• Archiving events
• Railway-oriented processing
• Commands

PART 3 - EVENT ANALYTICS

• Analytics-on-read
• Analytics-on-write

Part 1. Event streams and unified logs

In this first part, we’ll introduce the basics of event streaming and explain what a unified log is. We’ll also show how to use technologies such as Apache Kafka, Amazon Kinesis, and Apache Samza in order to process event streams.

Chapter 1. Introducing event streams

This chapter covers

• Defining events and continuous event streams
• Exploring familiar event streams
• Unifying event streams with a unified log
• Introducing use cases for a unified log

Believe it or not, a continuous stream of real-world and digital events already powers the company where you work. But it’s unlikely that many of your coworkers think in those terms. Instead, they probably think about their work in terms of the following:

• The people or things that they interact with on a daily basis—for example, customers, the Marketing team, code commits, or new product releases
• The software and hardware that they use to get stuff done
• Their own daily inbox of tasks to accomplish

People think and work in these terms because people are not computers. It is easy to get up in the morning and come to work because Sue in QA really needs those reports for her boss by lunchtime. If we stopped and started to think about our work as creating and responding to a continuous stream of events, we would probably go a little crazy—or at least call in to the office for a duvet day.

Computers don’t have this problem. They would be comfortable with this definition of a business:

A company is an organization that generates and responds to a continuous stream of events.

This definition is not going to win any awards from economists, but we, the authors, believe that reframing your business in terms of a continuous stream of events offers huge benefits. Specifically, event streams enable the following:

• Fresher insights— A continuous stream of events represents the “pulse” of a business and makes a conventional batch-loaded data warehouse look stale in comparison.
• A single version of the truth— Ask several coworkers the same question and you may well get different answers, because they are working from different “pots” of data. Well-modeled event streams replace this confusion with a single version of the truth.
• Faster reactions— Automated near-real-time processing of continuous event streams allows a business to respond to those events within minutes or even seconds.
• Simpler architectures— Most businesses have built up a bird’s nest of bespoke point-to-point connections between their various transactional systems. Event streams can help to unravel these messy architectures.

Some of these benefits may not seem obvious now, but don’t worry: in this chapter, we will go back to first principles, starting with what we mean by events. We will introduce some simple examples of events, and then explain what a continuous event stream really is. There’s a good chance you will find that you are pretty comfortable working with event streams already—you just haven’t thought of them in those terms.

Once we have presented some familiar event streams, we will zoom out a level and explain how businesses’ handling of events has evolved over the past 20 years. You will see that successive waves of technology have made things much more complex than they should be, but that a new architectural pattern called the unified log promises to simplify things again.

For these new approaches to reach the mainstream, they must be backed up with compelling use cases. We will make the benefits of continuous event streams and the unified log significantly more real with a set of tangible real-world use cases, across a variety of industries.

1.1. Defining our terms

If you work in any kind of modern-day business, chances are that you have already worked with event streams in various forms but have not been introduced to them as such. This section presents a simple definition for an event and then explains how events combine into a continuous event stream.

1.1.1. Events

Before we can define a continuous event stream, we need to break out of Synonym City and concretely define a single event. Fortunately, the definition is simple: an event is anything that we can observe occurring at a particular point in time. That’s it, fin. Figure 1.1 sets out four example events from four different business sectors.

Figure 1.1. The precision on the timestamps varies a little, but you can see that all four of these events are discrete, recordable occurrences that take place in the physical or digital worlds (or both).

It is easy to get carried away with the simplicity of this definition of an event, so before we go any further, let’s clarify what is not an event. This is by no means an exhaustive list, but these are some of the more common mistakes to avoid. An event is not any of the following:

• A description of the ongoing state of something— The day was warm; the car was black; the API client was broken. But “the API client broke at noon on Tuesday” is an event.
• A recurring occurrence— The NASDAQ opened at 09:30 every day in 2018. But each individual opening of the NASDAQ in 2018 is an event.
• A collection of individual events— The Franco-Prussian war involved the Battle of Spicheren, the Siege of Metz, and the Battle of Sedan. But “war was declared between France and Prussia on 19 July 1870” is an event.
• A happening that spans a time frame— The 2018 Black Friday sale ran from 00:00:00 to 23:59:59 on November 23, 2018. But the beginning of the sale and the end of the sale are events.

Here’s a general rule of thumb: if the thing you are describing can be tied to a specific point in time, chances are that you are describing an event of some kind, even if it needs some verbal gymnastics to represent it.

1.1.2. Continuous event streams

Now that we have defined what an event is, what is a continuous event stream? Simply put, a continuous event stream is an unterminated succession of individual events, ordered by the point in time at which each event occurred. Figure 1.2 sketches out what a continuous event stream looks like at a high level: you can see a succession of individual events, stepping forward in time.

Figure 1.2. Anatomy of a continuous event stream: time is progressing left to right, and individual events are ordered within this time frame. Note that the event stream is unterminated; it can extend in both directions beyond our ability to process it.

We say that the succession of events is unterminated, because of these facts:

• The start of the stream may predate our observing of the stream.
• The end of the stream is at some unknown point in the future.

To illustrate this, let’s consider guests checking into the Hoshi Ryokan hotel in Japan. Hoshi Ryokan is one of the oldest businesses in the world, having been founded in AD 718. Whatever stream of guest check-in events we could analyze for Hoshi Ryokan, we would know that the oldest guest check-ins are lost in the mists of time, and that future check-in events will continue to occur long after we have retired.

1.2. Exploring familiar event streams

If you read the previous section and thought that events and continuous event streams seemed familiar, then chances are that you have already worked with event streams, although they were likely not labeled as such. A huge number of software systems are heavily influenced by the idea of generating and responding to a continuous stream of events, including these:

• Transactional systems— Many of these respond to external events, such as customers placing orders or suppliers delivering parts.
• Data warehouses— These collec...