The DevOps 2.2 Toolkit

Viktor Farcic

• Publisher: Packt Publishing
• Year: 2018
• Print ISBN: 9781788991278
• eBook ISBN: 9781788996778
• Edition: 1
• Topic: Computer Science
• Subtopic: Project Management
• Index: Computer Science

Learn from an expert on how use self-adapting and self-healing systems within Docker.

Key Features

• Viktor Farcic shows you all aspects in the creation of self-adapting and self-healing systems in both a practical and hands-on approach.
• Learn how to choose a successful solution for metrics storage and query, including InfluxDB, Nagios and Sensu, Prometheus and Graphite.
• Discover how to integrate Docker Flow Monitor with Docker Flow Proxy.
• How to apply Docker self-healing and self-adaptive to both services and infrastructure.

Book Description

Building on The DevOps 2.0 Toolkit and The DevOps 2.1 Toolkit: Docker Swarm, Viktor Farcic brings his latest exploration of the Docker technology as he records his journey to explore two new programs, self-adaptive and self-healing systems within Docker.

The DevOps 2.2 Toolkit: Self-Sufficient Docker Clusters is the latest book in Viktor Farcic's series that helps you build a full DevOps Toolkit. This book in the series looks at Docker, the tool designed to make it easier in the creation and running of applications using containers. In this latest entry, Viktor combines theory with a hands-on approach to guide you through the process of creating self-adaptive and self-healing systems.

Within this book, Viktor will cover a wide-range of emerging topics, including what exactly self-adaptive and self-healing systems are, how to choose a solution for metrics storage and query, the creation of cluster-wide alerts and what a successful self-sufficient system blueprint looks like.

Work with Viktor and dive into the creation of self-adaptive and self-healing systems within Docker.

What you will learn

• Let Viktor Farcic show you all aspects in the creation of self-adapting and self-healing systems in both a practical and hands-on approach.
• Learn how to choose a successful solution for metrics storage and query, including InfluxDB, Nagios and Sensu, Prometheus and Graphite.
• Understand how to integrate Docker Flow Monitor with Docker Flow Proxy.
• The creation of cluster-wide alerts by creating alerts based on metrics.
• How to apply self-healing and self-adaptive to both services and infrastructure.

Who this book is for

This book is for professionals experienced with Docker looking to create both self-adapting and self-healing systems using the software.

Self-Adaptation Applied to Infrastructure

Our goal is within reach. We adopted schedulers (Docker Swarm in this case) that provide self-healing applied to services. We saw how Docker For AWS accomplishes a similar goal but on the infrastructure level. We used Prometheus, Alertmanager, and Jenkins to build a system that automatically adapts services to ever-changing conditions. The metrics we're storing in Prometheus are a combination of those gathered through exporters and those we added to our services through instrumentation. The only thing we're missing is self-adaptation applied to infrastructure. If we manage to build it, we'll close the circle and witness a self-sufficient system capable of running without (almost) any human intervention.

The logic behind self-adaptation applied to infrastructure is not much different from the one we used with services. We need metrics, alerts, and scripts that will adapt cluster capacity whenever conditions change.

We already have all the tools we need. Prometheus will continue gathering metrics and firing alerts. Alertmanager is still an excellent choice to receive those alerts and resend them to different system components. We'll keep using Jenkins as a tool that allows us to quickly write scripts that can interact with the system. Since we're using AWS to host our cluster, Jenkins will have to interact with its API.

We are so close to the final objective that I feel we should skip the theory and jump straight into practical hands-on parts of the chapter. So, without further ado, we'll create our cluster one more time.

Creating a cluster

In the previous chapter, we already explored how to create a cluster without UI. The commands that follow should be familiar and, hopefully, should not require much explanation.

All the commands from this chapter are available in the 15-self-adaptation-infra.sh Gist at https://gist.github.com/vfarcic/7f49e5d1565b2234b84d8fe01e5c2356.

Please replace [...] with your keys before executing the commands that follow.

export AWS_ACCESS_KEY_ID=[...]

export AWS_SECRET_ACCESS_KEY=[...]

export AWS_DEFAULT_REGION=us-east-1

export STACK_NAME=devops22

export KEY_NAME=devops22

aws cloudformation create-stack \
 --template-url https://editions-us-east-1.s3.amazonaws.com/aws\
/stable/Docker.tmpl
 --capabilities CAPABILITY_IAM \
 --stack-name $STACK_NAME \
 --parameters \
ParameterKey=ManagerSize,ParameterValue=3 \
ParameterKey=ClusterSize,ParameterValue=0 \
ParameterKey=KeyName,ParameterValue=$KEY_NAME \
ParameterKey=EnableSystemPrune,ParameterValue=yes \
ParameterKey=EnableCloudWatchLogs,ParameterValue=no \
ParameterKey=EnableCloudStorEfs,ParameterValue=yes \
ParameterKey=ManagerInstanceType,ParameterValue=t2.small \
ParameterKey=InstanceType,ParameterValue=t2.small

We defined a few environment variables and executed the aws cloudformation create-stack command that initiated creation of a cluster. It should take around five to ten minutes until it is finished.

aws cloudformation describe-stacks \
--stack-name $STACK_NAME | \
jq -r ".Stacks[0].StackStatus"

If the output of the describe-stacks command is CREATE_COMPLETE, our cluster is fully operational, and we can continue. Otherwise, please wait for a while longer and recheck the stack status.

Next, we'll retrieve cluster DNS and public IP of one of the manager nodes and store those values as environment variables CLUSTER_DNS and CLUSTER_IP.

CLUSTER_DNS=$(aws cloudformation \
 describe-stacks \
 --stack-name $STACK_NAME | \
 jq -r ".Stacks[0].Outputs[] | \
 select(.OutputKey==\"DefaultDNSTarget\")\
 .OutputValue")

CLUSTER_IP=$(aws ec2 describe-instances \
 | jq -r ".Reservations[] \
 .Instances[] \
 | select(.SecurityGroups[].GroupName \
 | contains("$STACK_NAME-ManagerVpcSG"))\
 .PublicIpAddress" \
 | tail -n 1)

Once we enter the cluster, we'll create a file that will hold the environment variables we'll need inside the cluster. Those are the same variables we already defined on our host. We'll output them so that we can easily copy and paste them when we enter one of the nodes.

echo "
export CLUSTER_DNS=$CLUSTER_DNS
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
"

Please copy the output of the echo command. We'll use it soon.

Now that we got all the cluster information we'll need, we can ssh into one of the manager nodes.

ssh -i $KEY_NAME.pem docker@$CLUSTER_IP

Next, we'll create a file that will hold all the information we'll need. That way we'll be able to get in and out of the cluster without losing the ability to retrieve that data quickly.

echo "
export CLUSTER_DNS=[...]
expo...

Table of contents

• Title Page
• Copyright
• Dedication
• Contributor
• Packt Upsell
• Preface
• Introduction to Self-Adapting and Self-Healing Systems
• Choosing a Solution for Metrics Storage and Query
• Deploying and Configuring Prometheus
• Scraping Metrics
• Defining Cluster-Wide Alerts
• Alerting Humans
• Alerting the System
• Self-Healing Applied to Services
• Self-Adaptation Applied to Services
• Painting the Big Picture – The Self-Sufficient System Thus Far
• Instrumenting Services
• Self-Adaptation Applied to Instrumented Services
• Setting Up a Production Cluster
• Self-Healing Applied to Infrastructure
• Self-Adaptation Applied to Infrastructure
• Blueprint of a Self-Sufficient System
• Other Books You May Enjoy