Apache Kafka 1.0 Cookbook

Raúl Estrada, Sandeep Khurana, Brian Gatt, Alexey Zinoviev

• Publisher: Packt Publishing
• Year: 2017
• ISBN:
  9781787282186
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Processing
• Index: Computer Science

Simplify real-time data processing by leveraging the power of Apache Kafka 1.0

Key Features

• Use Kafka 1.0 features such as Confluent platforms and Kafka streams to build efficient streaming data applications to handle and process your data
• Integrate Kafka with other Big Data tools such as Apache Hadoop, Apache Spark, and more
• Hands-on recipes to help you design, operate, maintain, and secure your Apache Kafka cluster with ease

Book Description

Apache Kafka provides a unified, high-throughput, low-latency platform to handle real-time data feeds. This book will show you how to use Kafka efficiently, and contains practical solutions to the common problems that developers and administrators usually face while working with it.

This practical guide contains easy-to-follow recipes to help you set up, configure, and use Apache Kafka in the best possible manner. You will use Apache Kafka Consumers and Producers to build effective real-time streaming applications. The book covers the recently released Kafka version 1.0, the Confluent Platform and Kafka Streams. The programming aspect covered in the book will teach you how to perform important tasks such as message validation, enrichment and composition.Recipes focusing on optimizing the performance of your Kafka cluster, and integrate Kafka with a variety of third-party tools such as Apache Hadoop, Apache Spark, and Elasticsearch will help ease your day to day collaboration with Kafka greatly. Finally, we cover tasks related to monitoring and securing your Apache Kafka cluster using tools such as Ganglia and Graphite.

If you're looking to become the go-to person in your organization when it comes to working with Apache Kafka, this book is the only resource you need to have.

What you will learn

• -Install and configure Apache Kafka 1.0 to get optimal performance
• -Create and configure Kafka Producers and Consumers
• -Operate your Kafka clusters efficiently by implementing the mirroring technique
• -Work with the new Confluent platform and Kafka streams, and achieve high availability with Kafka
• -Monitor Kafka using tools such as Graphite and Ganglia
• -Integrate Kafka with third-party tools such as Elasticsearch, Logstash, Apache Hadoop, Apache Spark, and more

Who this book is for

This book is for developers and Kafka administrators who are looking for quick, practical solutions to problems encountered while operating, managing or monitoring Apache Kafka. If you are a developer, some knowledge of Scala or Java will help, while for administrators, some working knowledge of Kafka will be useful.

Configuring Kafka

In this chapter, we will cover the following topics:

• Installing Kafka
• Running Kafka
• Configuring Kafka brokers
• Configuring Kafka topics
• Creating a message console producer
• Creating a message console consumer
• Configuring the broker settings
• Configuring threads and performance
• Configuring the log settings
• Configuring the replica settings
• Configuring the Zookeeper settings
• Configuring other miscellaneous parameters

Introduction

This chapter explains the basic recipes to get started with Apache Kafka. It discusses how to install, configure, and run Kafka. It also discusses how to make basic operations with a Kafka broker.

Kafka can run in several operating systems: Mac, Linux, and even Windows. As it usually runs in production on Linux servers, the recipes in this book are designed to run in Linux environments. This book also considers the bash environment usage.

Kafka scales very well in a horizontal fashion without compromising speed and efficiency.

This chapter explains how to install, configure, and run Kafka. As this is a practical recipes book, it does not cover the theoretical details of Kafka. These three things are enough theory for the moment:

1. To connect heterogeneous applications, we need to implement a mechanism for message publishing by sending and receiving messages among them. A message router is known as a message broker. Kafka is a software solution to deal with routing messages among consumers in a quick way.
2. The message broker has two directives: the first is to not block the producers, and the second is to isolate producers and consumers (the producers should not know who their consumers are).
3. Kafka is two things: a real-time, publish-subscribe solution, and a messaging system. Moreover, it is a solution: open source, distributed, partitioned, replicated, commit-log based, with a publish-subscribe schema.

Before we begin it is relevant to mention some concepts and nomenclature in Kafka:

• Broker: A server process
• Cluster: A set of brokers
• Topic: A queue (that has log partitions)
• Offset: A message identifier
• Partition: An ordered and immutable sequence of records continually appended to a structured commit log
• Producer: Those who publish data to topics
• Consumer: Those who process the feed
• ZooKeeper: The coordinator
• Retention period: The time to keep messages available for consumption

In Kafka, there are three types of clusters:

• Single node: Single broker
• Single node: Multiple Broker
• Multiple node: Multiple Broker

There are three ways to deliver messages:

• Never redelivered: The messages may be lost
• May be redelivered: The messages are never lost
• Delivered once: The message is delivered exactly once

There are two types of log compaction:

• Coarse grained: By time
• Finer grained: By message

The next six recipes contain the necessary steps to make a full Kafka test from zero.

Installing Kafka

This is the first step. This recipe shows how to install Apache Kafka.

Getting ready

Ensure that you have at least 4 GB of RAM in your machine; the installation directory will be /usr/local/kafka/ for Mac users and /opt/kafka/ for Linux users. Create these directories.

How to do it...

Go to the Apache Kafka home page at http://kafka.apache.org/downloads, as in Figure 1-1, Apache Kafka download page:

Figure 1-1. Apache Kafka download page

The current available version of Apache Kafka is 0.10.2.1, as a stable release. A major limitation with Kafka since 0.8.x is that it is not backward-compatible. So, we cannot replace this version for one prior to 0.8. Once you've downloaded the latest available release, let's proceed with the installation.

Remember, for Mac users, replace the directory /opt/ for /usr/local/ in the examples.

Installing Java in Linux

We need Java 1.7 or later. Download and install the latest JDK from Oracle's website: http://www.oracle.com/technetwork/java/javase/downloads/index.html

1. Change the file mode:

> chmod +x jdk-8u131-linux-x64.rpm

2. Go to the directory in which you want to perform the installation:

> cd <directory path name>

3. Run the rpm installer with the command:

> rpm -ivh jdk-8u131-linux-x64.rpm

4. Finally, add the environment variable JAVA_HOME. This command will write the JAVA_HOME environment variable to the file /etc/profile:

> echo "export JAVA_HOME=/usr/java/jdk1.8.0_131" >> /etc/profile

Installing Scala in Linux

The following are the steps to install Scala in Linux:

1. Download the latest Scala binary from: http://www.scala-lang.org/download
2. Extract the downloaded file scala-2.12.2.tgz:

> tar xzf scala-2.12.2.tgz

3. Most tutorials agree that the best place to set environment variables is in the /etc/profile file.
4. Create the SCALA_HOME environment variable:

> export SCALA_HOME=/opt/scala

5. Add the Scala bin directory to the PATH variable:

> export PATH=$PATH:$SCALA_HOME/bin

Installing Kafka in Linux

The following are the steps to install Kafka in Linux:

1. Extract the downloaded file kafka_2.10-0.10.2.1.tgz:

> tar xzf kafka_2.10-0.10.2.1.tgz

2. Create the KAFKA_HOME environment variable:

> export KAFKA_HOME=/opt/kafka_2.10-0.10.2.1

3. Add the Kafka bin directory to the PATH variable:

> export PATH=$PATH:$KAFKA_HOME/bin

Now Java, Scala, and Kafka are installed.

There's more...

To do all these steps in command-line mode, there is a powerful tool for Mac users called brew (the equivalent in Linux would be yum).

To install from the command line, we use the following steps:

1. With brew, install sbt (Scala build tool):

> brew install sbt

If you already have it (downloaded in the past), upgrade it:

> brew upgrade sbt

The output is similar to:

> brew upgrade sbt
==> Upgrading 1 outdated package, with result:
sbt 0.13.15
==> Upgrading sbt
==> Using the sandbox
==> Downloading https://github.com/sbt/sbt/releases/download/v0.13.15/sbt-0.13.15.tgz
==> Downloading from https://github-cloud.s3.amazonaws.com/releases/279553/09838df4-23c6-11e7-9276-14
######################################################################## 100.0%
==> Caveats
You can use $SBT_OPTS to pass additional JVM options to SBT:
 SBT_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
This formula is now using the standard lightbend sbt launcher script.
Project specific options should be placed in .sbtopts in the root of your project.
Global settings should be placed in /usr/local/etc/sbtopts
==> Summary
/usr/local/Cellar/sbt/0.13.15: 378 files, 63.3MB, built in 1 minute 5 seconds

2. With brew, install Scala:

> brew install scala

If you a...