CoreOS in Action

Running Applications on Container Linux

Matt Bailey

• Publisher: Manning
• Year: 2017
• ISBN:
  9781638351672
• Topic: Ciencia de la computación
• Subtopic: Sistemas operativos

Summary CoreOS in Action is a clear tutorial for deploying container-based systems on CoreOS Container Linux.Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the Technology Traditional Linux server distributions include every component required for anything you might be hosting, most of which you don't need if you've containerized your apps and services. CoreOS Container Linux is a bare-bones distro with only the essential bits needed to run containers like Docker. Container Linux is a fast, secure base layer for any container-centric distributed application, including microservices. And say goodbye to patch scheduling; when Container Linux needs an update, you just hot-swap the whole OS. About the Book CoreOS in Action is a clear tutorial for deploying container-based systems on CoreOS Container Linux. Inside, you'll follow along with examples that teach you to set up CoreOS on both private and cloud systems, and to practice common sense monitoring and upgrade techniques with real code. You'll also explore important container-aware application designs, including microservices, web, and Big Data examples with real-world use cases to put your learning into perspective. Summary

• Handling scaling and failures gracefully
• Container-driven application designs
• Cloud, on-premises, and hybrid deployment
• Smart logging and backup practices

About the Reader Written for readers familiar with Linux and the basics of Docker. About the Author Matt Bailey is currently a technical lead at ZeniMax. He has worked in higher education and with scientific computing, medical, and networking technology companies, as well as a few startups. You can find him online via https://mdb.io. Table of Contents

PART 1 - GETTING TO KNOW COREOS

• Introduction to the CoreOS family
• Getting started on your workstation
• Expecting failure: fault tolerance in CoreOS

PART 2 - APPLICATION ARCHITECTURE

• CoreOS in production
• Application architecture and workflow
• Web stack application example
• Big Data stack

PART 3 - COREOS IN PRODUCTION

• CoreOS on AWS
• Bringing it together: deployment
• System administration

Part 1. Getting to know CoreOS

In these first three chapters, you’ll get to know what CoreOS is all about. I’ll cover some terminology and the systems that form CoreOS and get you up and running with a sandbox environment. You’ll also start working on an application stack that you’ll build on throughout the book.

Chapter 1. Introduction to the CoreOS family

This chapter covers

• Overview of CoreOS systems and concepts
• Understanding common workflow patterns for CoreOS
• Introducing fleet and etcd, and systemd units

Suppose you’ve been hired by a new company that wants you to build out a modern infrastructure and operational architecture for its developers. The company has a wide range of application stacks, and you have strong requirements around horizontal scalability and high availability. You know you want Linux, but the idea of maintaining endless operating system updates and changes or setting up complex configuration-management systems is unappealing. You recognize that containerization can make this far easier—you can separate the operational configuration from the applications’—but you’re still left with how to manage all those containers at scale. Plenty of distributions today support Docker, but not in a way that seems designed for large-scale production use.

Enter CoreOS: an OS designed from the ground up to facilitate container operationalization at any scale. It’s highly fault tolerant and extremely lightweight, and it appears performant, but how do you get started? You know the goal: you want to provide your engineers with a container-based platform as a service, and you know CoreOS can be the hammer to hit that nail. But how do you get it running? How do you adapt or design application architectures to best take advantage of this system?

Note

If you want to know more about where the ideas in CoreOS came from, be sure to read the “Background” section in the “About this book” portion of this book’s front matter.

In this chapter, we’ll go over the various parts that make up the CoreOS family of systems, and we’ll look a little at how they can solve infrastructure and architecture problems like those just described. By the end of this chapter, you’ll have a clear understanding of CoreOS and how its core components fit together, along with some ideas about its utility that you can take into chapter 2, when we discuss building out a local cluster.

1.1. Meet CoreOS

CoreOS is here to solve your scale, availability, and deployment workflow problems. In this chapter, we’ll go through a simple application deployment of NGINX (a popular HTTP server) to illustrate how CoreOS achieves some of these solutions, and review some essential systems. With CoreOS, you won’t have to manage packages, initiate lengthy upgrade processes, plan out complex configuration files, fiddle with permissions, plan significant maintenance windows (for the OS), or deal with complicated configuration schema changes. If you fully embrace CoreOS’s features, your cluster of nodes will always have the latest version of the OS, and you won’t have any downtime.

These ideas can be a little difficult to grasp when you’re first getting started with CoreOS, but they embody the philosophy of an immutable OS after boot, which creates an experience with the OS that you probably aren’t used to. CoreOS’s distributed scheduler, fleet, manages the state of your application stack, and CoreOS provides the platform on which those systems orchestrate your services. If you have a computer science background, you can consider traditional configuration-management systems as relying heavily on side effects to constantly manipulate the state of the OS, whereas in CoreOS, the state of the OS is created once on boot, never changes, and is lost on shutdown. This is a powerful concept, and it forces architectures with high degrees of idempotence and no hidden side effects, the results of which are vastly improved certainty about the reliability of your systems and drastically reduced need for layers of complex tooling to monitor and manage OSs. In this section, I provide an overview of the parts that make CoreOS tick and how they complement each other.

CoreOS background

CoreOS is a Linux distribution based, in a way, on Gentoo Linux. Similar to how Google’s Chrome OS is based on Gentoo, this only matters for those interested in hacking on CoreOS itself, which isn’t covered in this book (although this book would certainly be an excellent guide to understanding what you’re working on).

The reason this probably doesn’t matter to you is a bit more complicated. CoreOS is designed to present a small number of services that act as a lightweight, distributed system; the point of CoreOS is that it mostly stays out of your way, and it’s immutably configured on boot, much as a container is. This is very different from virtually all other Linux distributions or OSs as a whole. In chapter 8, we’ll dive deeper into cloud-config, which describes the state of the OS, most of which is concerned with cluster discovery and initializing core services that you may want to manage outside of fleet.

On containerization

We’ll go into how you can tune your containers to best function with CoreOS, but you should have some experience with Docker and the concepts of containerization to get the most out of this book. You can also check out Docker in Action by Jeff Nickoloff (Manning, 2016, www.manning.com/books/docker-in-action).

1.1.1. The CoreOS family

CoreOS consists of a few critical systems and services that manage all the scalability and fault tolerance it claims to facilitate. Figure 1.1 provides a high-level idea of how the cluster layout looks.

Figure 1.1. CoreOS layout

We’ll go into each of these components in some detail in the next section, and significantly more detail later in the book, but this represents the key systems that make up CoreOS:

• etcd acts as your cluster’s persistent configuration state (see section 1.1.2).
• fleetd acts as your cluster’s distributed runtime scheduler (see section 1.1.3).
• systemd unit files are the mechanism by which fleetd executes the runtime (see section 1.1.4).
• Docker and rkt are the common container platforms that your unit files will run. CoreOS intends all of your runtime to happen in containers, and you can choose from these two platforms (or a combination of both; see section 1.1.5).

The one essential system missing from figure 1.1 is cloud-config, which is used to set the initial configuration state of a machine. It’s more a detail of infrastructure configuration than a requirement to understand CoreOS’s concepts; section 1.1.6 covers it in detail.

1.1.2. etcd and the distributed configuration state

etcd is a high-reliability distributed key/value store. If you’re familiar with memcached or redis, it’s similar, but with more focus on (distributed) consistency and availability over performance. It’s accessible via custom command-line tools and is fully RESTful and JSON based. As its name implies, etcd is designed to distribute your system and services configuration. It’s the data store for fleet (CoreOS’s distributed scheduler).

fleet and CoreOS use etcd to find peers, distribute locks for various purposes, and orchestrate running systemd units across the cluster. Although it’s useful in that regard alone, it’s also designed to be your place to persist configuration in the cluster. In the example later in this chapter, you’ll use it to register NGINX instances for a load balancer to discover.

etcd isn’t designed for large-object storage or massive performance; its primary purpose is to make the state of the cluster singular. Beyond cloud-config, which sets initial state, no other s...