Pipeline as Code

Continuous Delivery with Jenkins, Kubernetes, and Terraform

Mohamed Labouardy

• Publisher: Manning
• Year: 2021
• Print ISBN: 9781617297540
• eBook ISBN: 9781638350378
• Topic: Computer Science
• Subtopic: Cloud Computing
• Index: Computer Science

Start thinking about your development pipeline as a mission-critical application. Discover techniques for implementing code-driven infrastructure and CI/CD workflows using Jenkins, Docker, Terraform, and cloud-native services.

In Pipeline as Code, you will master:

    Building and deploying a Jenkins cluster from scratch
    Writing pipeline as code for cloud-native applications
    Automating the deployment of Dockerized and Serverless applications
    Containerizing applications with Docker and Kubernetes
    Deploying Jenkins on AWS, GCP and Azure
    Managing, securing and monitoring a Jenkins cluster in production
    Key principles for a successful DevOps culture

Pipeline as Code is a practical guide to automating your development pipeline in a cloud-native, service-driven world. You’ll use the latest infrastructure-as-code tools like Packer and Terraform to develop reliable CI/CD pipelines for numerous cloud-native applications. Follow this book's insightful best practices, and you’ll soon be delivering software that’s quicker to market, faster to deploy, and with less last-minute production bugs.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Treat your CI/CD pipeline like the real application it is. With the Pipeline as Code approach, you create a collection of scripts that replace the tedious web UI wrapped around most CI/CD systems. Code-driven pipelines are easy to use, modify, and maintain, and your entire CI pipeline becomes more efficient because you directly interact with core components like Jenkins, Terraform, and Docker.

About the book
In Pipeline as Code you’ll learn to build reliable CI/CD pipelines for cloud-native applications. With Jenkins as the backbone, you’ll programmatically control all the pieces of your pipeline via modern APIs. Hands-on examples include building CI/CD workflows for distributed Kubernetes applications, and serverless functions. By the time you’re finished, you’ll be able to swap manual UI-based adjustments with a fully automated approach!

What's inside

    Build and deploy a Jenkins cluster on scale
    Write pipeline as code for cloud-native applications
    Automate the deployment of Dockerized and serverless applications
    Deploy Jenkins on AWS, GCP, and Azure
    Grasp key principles of a successful DevOps culture

About the reader
For developers familiar with Jenkins and Docker. Examples in Go.

About the author
Mohamed Labouardy is the CTO and co-founder of Crew.work, a Jenkins contributor, and a DevSecOps evangelist.

Table of Contents

PART 1 GETTING STARTED WITH JENKINS
1 What’s CI/CD?
2 Pipeline as code with Jenkins
PART 2 OPERATING A SELF-HEALING JENKINS CLUSTER
3 Defining Jenkins architecture
4 Baking machine images with Packer
5 Discovering Jenkins as code with Terraform
6 Deploying HA Jenkins on multiple cloud providers
PART 3 HANDS-ON CI/CD PIPELINES
7 Defining a pipeline as code for microservices
8 Running automated tests with Jenkins
9 Building Docker images within a CI pipeline
10 Cloud-native applications on Docker Swarm
11 Dockerized microservices on K8s
12 Lambda-based serverless functions
PART 4 MANAGING, SCALING, AND MONITORING JENKINS
13 Collecting continuous delivery metrics
14 Jenkins administration and best practices

Part 1. Getting started with Jenkins

This first part of this book takes you through the DevOps essential concepts. You’ll learn about CI/CD practices and how they allow you to integrate small pieces of code at one time and ease technical debt. After that, I’ll introduce the new approach of building CI/CD pipelines, pipeline as code, and how it can be implemented with Jenkins. Finally, I’ll lay the groundwork for a well-designed CI/CD workflow by introducing the GitFlow branching model.

1 What’s CI/CD?

This chapter covers

• The path organizations have taken to evolve from monolith to cloud-native applications
• The challenges of implementing CI/CD practices for cloud-native architectures
• An overview of continuous integration, deployment, and delivery
• How CI/CD tools like Jenkins can bring business value to organizations that undertake the journey of continuous everything

Software development and operations have experienced several paradigm shifts recently. These shifts have presented the industry with innovative approaches for building and deploying applications. More importantly, two significant paradigm shifts have consolidated capabilities for developing, deploying, and managing scalable applications: cloud-native architecture and DevOps.

Cloud-native architecture emerged with cloud adoption, with cloud providers like Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure taking ownership of the infrastructure. Open source tools like Kubernetes, Docker, and Istio offer horizontal scaling ability, letting developers build and run modern scalable applications without worrying about the underlying infrastructure. As a result, operational overhead is reduced, and the development velocity of applications is increased.

DevOps bridged the divide between developers and ops teams, and brought back harmony through collaboration, automated tools, and iterative and Agile development and deployment.

With these two significant, powerful approaches combined, organizations now have the capability to create scalable, robust, and reliable applications with a high level of collaboration and information sharing among small teams. However, to build, test, and safely deploy cloud-native applications, two essential DevOps practices must be implemented in a cloud-native manner: continuous integration (CI) and continuous deployment/delivery (CD).

The first part of this book takes you through the evolution of cloud-native applications. You’ll learn about the main principles of CI/CD and how automation invented the way those principles are implemented through the pipeline-as-code approach. This first chapter lays the foundation. It introduces basic principles of DevOps and cloud-native approaches, in addition to selecting the tools for implementing CI/CD pipelines.

1.1 Going cloud native

Before exploring the essential characteristics of cloud-native applications and how CI/CD practices contribute to standardizing feedback loops for developers and enabling fast product iterations, we will cover the changes the software development model went through and the challenges associated with each model, starting with the monolithic approach.

1.1.1 Monolithic

In the past, organizations used to build their software in a monolithic way: all functionalities were packaged in a single artifact and deployed in a single server running one process. This architecture comes with many drawbacks and limitations:

• Development velocity—Adding new features on top of an existing application is next to impossible. Application modules are tightly coupled and, most of the time, not documented. As a result, adding new features is often slow, expensive, and requires extra synchronization when working with multiple developers within distributed teams on a large codebase. Moreover, the release cycle can take months, if not several years, because of the application’s large codebase. This delay puts companies at risk of being surpassed by new competitors and ultimately undercuts the company’s profits.
• Maintainability—Modules in a monolithic architecture are frequently tightly coupled, which makes them hard to maintain and test. Plus, upgrading to new technology is limited to the framework used to develop the application (no polyglot programming).
• Scaling and resiliency—Applications are designed with no scalability in mind, and the application may face downtime if traffic increases. The monolithic application works as a single unit and is developed in a single programming language using a single tech stack. As a result, to achieve partial horizontal scaling, the whole application needs to be scaled (inefficient usage of server resources).
• Cost-effectiveness—The application is expensive to maintain in the long run (for example, finding an experienced COBOL developer is time-consuming and expensive).

In the late 2000s, many web giants (including Facebook, Netflix, Twitter, and Amazon) came onto the tech scene with innovative ideas, aggressive strategies, and a “move fast” approach that led to the exponential growth of their platforms. These companies introduced a new architecture pattern that is known today as microservices. So, what exactly is microservices architecture?

1.1.2 Microservices

James Lewis and Martin Fowler defined microservices architecture as follows in 2014:

In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies.

This architecture uses the same technique of “divide and conquer” to tackle the complexity of an application. An application is split into smaller, independent, and composable services/fragments, each responsible for a specific functionality or task of the application (organized around business capabilities).

Those microservices communicate using an application programming interface (API), typically over HTTP or HTTP/2 (for example, gRPC, RESTful APIs, Google Protocol Buffers, or Apache Thrift), or through message brokers (such as Apache ActiveMQ or Kafka). Each microservice can be implemented in a different programming language running on a different OS platform.

In contrast to microservices, the monolithic architecture means the code’s components are designed to work together as one cohesive unit, sharing the same server resources (memory, CPU, disk, and so forth). Figure 1.1 illustrates the differences between monolith and microservices architectures.

Figure 1.1 Comparing monolith and microservices architectures

Microservices architecture is an extension of service-oriented architecture...

Table of contents

• inside front cover
• Pipeline as Code
• Copyright
• brief contents
• contents
• front matter
• Part 1. Getting started with Jenkins
• 1 What’s CI/CD?
• 2 Pipeline as code with Jenkins
• Part 2. Operating a self-healing Jenkins cluster
• 3 Defining Jenkins architecture
• 4 Baking machine images with Packer
• 5 Discovering Jenkins as code with Terraform
• 6 Deploying HA Jenkins on multiple cloud providers
• Part 3. Hands-on CI/CD pipelines
• 7 Defining a pipeline as code for microservices
• 8 Running automated tests with Jenkins
• 9 Building Docker images within a CI pipeline
• 10 Cloud-native applications on Docker Swarm
• 11 Dockerized microservices on K8s
• 12 Lambda-based serverless functions
• Part 4. Managing, scaling, and monitoring Jenkins
• 13 Collecting continuous delivery metrics
• 14 Jenkins administration and best practices
• index