The DevOps 2.3 Toolkit

Viktor Farcic

• Publisher: Packt Publishing
• Year: 2018
• ISBN:
  9781789133295
• Edition: 1
• Topic: Computer Science
• Subtopic: Software Development
• Index: Computer Science

Learn from an expert on how to use Kubernetes, the most adopted container orchestration platform.About This Book• Get a detailed, hands-on exploration of everything from the basic to the most advanced aspects of Kubernetes• Explore the tools behind not only the official project but also the third-party add-ons• Learn how to create a wide range of tools, including clusters, Role Bindings, and Ingress Resources with default backends, among many applicable, real-word creations• Discover how to deploy and manage highly available and fault-tolerant applications at scale with zero downtimeWho This Book Is ForThis book is for professionals experienced with Docker, looking to get a detailed overview from the basics to the advanced features of Kubernetes.What You Will Learn• Let Viktor show you the wide range of features available in Kubernetes—from the basic to the most advanced features• Learn how to use the tools not only from the official project but also from the wide range of third-party add-ons• Understand how to create a pod, how to Scale Bids with Replica Sets, and how to install both Kubectl and Minikube• Explore the meaning of terms such as container scheduler and Kubernetes• Discover how to create a local Kubernetes cluster and what to do with itIn DetailBuilding on The DevOps 2.0 Toolkit, The DevOps 2.1 Toolkit: Docker Swarm, and The DevOps 2.2 Toolkit: Self-Sufficient Docker Clusters, Viktor Farcic brings his latest exploration of the DevOps Toolkit as he takes you on a journey to explore the features of Kubernetes.The DevOps 2.3 Toolkit: Kubernetes is a book in the series that helps you build a full DevOps Toolkit. This book in the series looks at Kubernetes, the tool designed to, among other roles, make it easier in the creation and deployment of highly available and fault-tolerant applications at scale, with zero downtime.Within this book, Viktor will cover a wide range of emerging topics, including what exactly Kubernetes is, how to use both first and third-party add-ons for projects, and how to get the skills to be able to call yourself a "Kubernetes ninja." Work with Viktor and dive into the creation and exploration of Kubernetes with a series of hands-on guides.Style and approachReaders join Viktor Farcic as he continues his exploration of DevOps and begins to explore the opportunities presented by Kubernetes.

Creating a Production-Ready Kubernetes Cluster

Creating a Kubernetes cluster is not trivial. We have to make many choices, and we can easily get lost in the myriad of options. The number of permutations is getting close to infinite and, yet, our clusters need to be configured consistently. Experience from the first attempt to set up a cluster can easily convert into a nightmare that will haunt you for the rest of your life.

Unlike Docker Swarm that packs almost everything into a single binary, Kubernetes clusters require quite a few separate components running across the nodes. Setting them up can be very easy, or it can become a challenge. It all depends on the choices we make initially. One of the first things we need to do is choose a tool that we'll use to create a Kubernetes cluster.

If we'd decide to install a Docker Swarm cluster, all we'd need to do is to install Docker engine on all the servers, and execute docker swarm init or docker swarm join command on each of the nodes. That's it. Docker packs everything into a single binary. Docker Swarm setup process is as simple as it can get. The same cannot be said for Kubernetes. Unlike Swarm that is highly opinionated, Kubernetes provides much higher freedom of choice. It is designed around extensibility. We need to choose among many different components. Some of them are maintained by the core Kubernetes project, while others are provided by third-parties. Extensibility is probably one of the main reasons behind Kubernetes' rapid growth. Almost every software vendor today is either building components for Kubernetes or providing a service that sits on top of it.

Besides the intelligent design and the fact that it solves problems related to distributed, scalable, fault-tolerant, and highly available systems, Kubernetes' power comes from adoption and support from a myriad of individuals and companies. You can use that power, as long as you understand that it comes with responsibilities. It's up to you, dear reader, to choose how will your Kubernetes cluster look like, and which components it'll host. You can decide to build it from scratch, or you can use one of the hosted solutions like Google Cloud Platform (GCE) Kubernetes Engine. There is a third option though. We can choose to use one of the installation tools. Most of them are highly opinionated with a limited amount of arguments we can use to tweak the outcome.

You might be thinking that creating a cluster from scratch using kubeadm cannot be that hard. You'd be right if running Kubernetes is all we need. But, it isn't. We need to make it fault tolerant and highly available. It needs to stand the test of time. Constructing a robust solution would require a combination of Kubernetes core and third-party components, AWS know-how, and quite a lot of custom scripts that would tie the two together. We won't go down that road. At least, not now.

We'll use Kubernetes Operations (kops) to create a cluster. It is somewhere in the middle between do-it-yourself-from-scratch and hosted solutions (for example, GCE). It's an excellent fit for both newbies and veterans. You'll learn which components are required for running a Kubernetes cluster. You'll be able to make some choices. And, yet, we won't go down the rabbit hole of setting up the cluster from scratch. Believe me, that hole is very deep, and it might take us a very long time to get out of it.

Typically, this would be a great place to explain the most significant components of a Kubernetes cluster. Heck, you were probably wondering why we didn't do that early on when we began the journey. Still, we'll postpone the discussion for a while longer. I believe it'll be better to create a cluster first and discuss the components through live examples. I feel that it's easier to understand something we can see and touch, instead of keeping it purely on the theoretical level.

All in all, we'll create a cluster first, and discuss its components later.

Since I already mentioned that we'll use kops to create a cluster, we'll start with a very brief introduction to the project behind it.

What is kubernetes operations (kops) project?

If you visit Kubernetes Operations (kops) (https://github.com/kubernetes/kops) project, the first sentence you'll read is that it is "the easiest way to get a production-grade Kubernetes cluster up and running." In my humble opinion, that sentence is accurate only if we exclude Google Kubernetes Engine (GKE). Today (February 2018), other hosting vendors did not yet release their Kubernetes-as-a-service solutions. Amazon's Elastic Container Service for Kubernetes (EKS) (https://aws.amazon.com/eks/) is still not open to the public. Azure Container Service (AKS) (https://azure.microsoft.com/en-us/services/kubernetes-service/) is also a new addition that still has a few pain points. By the time you read this, all major hosting providers might have their solutions. Still, I prefer kops since it provides almost the same level of simplicity without taking away the control of the process. It allows us to tweak the cluster more than we would be permitted with hosted solutions. It is entirely open source, it can be stored in version control, and it is not designed to lock you into a vendor.

If your hosting vendor is AWS, kops is, in my opinion, the best way to create a Kubernetes cluster. Whether that's true for GCE, is open for debate since GKE works great. We can expect kops to be extended in the future to other vendors. For example, at the time of this writing, VMWare is in alpha and should be stable soon. Azure and Digital Ocean support are being added as I write this.

We'll use kops to create a Kubernetes cluster in AWS. This is the part of the story that might get you disappointed. You might have chosen to run Kubernetes somewhere else. Don't be depressed. Almost all Kubernetes clusters follow the same principles even though the method of setting them up might differ. The principles are what truly matters, and I'm confident that, once you set it up successfully on AWS, you'll be able to transfer that knowledge anywhere else.

The reason for choosing AWS lies in its adoption. It is the hosting vendor with, by far, the biggest user-base. If I'd have to place a blind bet on your choice, it would be AWS solely because that is statistically the most likely choice. I could not explore all the options in a single chapter. If I am to go through all hosting vendors and different projects that might help with the installation, we'd need to dedicate a whole book to that. Instead, I invite you to explore the subject further once you're finished with installing Kubernetes in AWS with kops. As an alternative, ping me on slack.devops20toolkit.com or send me an email to [email protected] and I'll give you a hand. If I receive enough messages, I might even dedicate a whole book to Kubernetes installations.

I went astray from kops...

Kops lets us create a production-grade Kubernetes cluster. That means that we can use it not only to create a cluster, but also to upgrade it (without downtime), update it, or destroy it if we don't need it anymore. A cluster cannot be called "production grade" unless it is highly available and fault tolerant. We should be able to execute it entirely from the command line if we'd like it to be automated. Those and quite a few other things are what kops provides, and what makes it great.

Kops follows the same philosophy as Kubernetes. We create a set of JSON or YAML objects which are sent to controllers that create a cluster.

We'll discuss what kops can and cannot do in more detail soon. For now, we'll jump into the hands-on part of this chapter and ensure that all the prerequisites for the installation are set.

Preparing for the cluster setup

We'll continue using the specifications from the vfarcic/k8s-specs repository, so the first thing we'll do is to go inside the directory where we cloned it, and pull the latest version.

All the commands from this chapter are available in the 14-aws.sh (https://gist.github.com/vfarcic/04af9efcd1c972e8199fc014b030b134) Gist.

cd k8s-specs git pull 

I will assume that you already have an AWS account. If that's not the case, please head over to Amazon Web Services (https://aws.amazon.com/) and sign-up.

If you are already proficient with AWS, you might want to skim through the tex...