Mastering Mesos

Dipa Dubhashi, Akhil Das

• Publisher: Packt Publishing
• Year: 2016
• eBook ISBN: 9781785886249
• Edition: 1
• Topic: Computer Science
• Subtopic: System Administration
• Index: Computer Science

The ultimate guide to managing, building, and deploying large-scale clusters with Apache Mesos

About This Book

• Master the architecture of Mesos and intelligently distribute your task across clusters of machines
• Explore a wide range of tools and platforms that Mesos works with
• This real-world comprehensive and robust tutorial will help you become an expert

Who This Book Is For

The book aims to serve DevOps engineers and system administrators who are familiar with the basics of managing a Linux system and its tools

What You Will Learn

• Understand the Mesos architecture
• Manually spin up a Mesos cluster on a distributed infrastructure
• Deploy a multi-node Mesos cluster using your favorite DevOps
• See the nuts and bolts of scheduling, service discovery, failure handling, security, monitoring, and debugging in an enterprise-grade, production cluster deployment
• Use Mesos to deploy big data frameworks, containerized applications, or even custom build your own applications effortlessly

In Detail

Apache Mesos is open source cluster management software that provides efficient resource isolations and resource sharing distributed applications or frameworks.

This book will take you on a journey to enhance your knowledge from amateur to master level, showing you how to improve the efficiency, management, and development of Mesos clusters. The architecture is quite complex and this book will explore the difficulties and complexities of working with Mesos.

We begin by introducing Mesos, explaining its architecture and functionality. Next, we provide a comprehensive overview of Mesos features and advanced topics such as high availability, fault tolerance, scaling, and efficiency. Furthermore, you will learn to set up multi-node Mesos clusters on private and public clouds.

We will also introduce several Mesos-based scheduling and management frameworks or applications to enable the easy deployment, discovery, load balancing, and failure handling of long-running services. Next, you will find out how a Mesos cluster can be easily set up and monitored using the standard deployment and configuration management tools.

This advanced guide will show you how to deploy important big data processing frameworks such as Hadoop, Spark, and Storm on Mesos and big data storage frameworks such as Cassandra, Elasticsearch, and Kafka.

Style and approach

This advanced guide provides a detailed step-by-step account of deploying a Mesos cluster. It will demystify the concepts behind Mesos.

Mastering Mesos

---

Table of Contents

Mastering Mesos

Credits

About the Authors

About the Reviewer

www.PacktPub.com

eBooks, discount offers, and more

Why subscribe?

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Introducing Mesos

Introduction to the datacenter OS and architecture of Mesos

The architecture of Mesos

Introduction to frameworks

Frameworks built on Mesos

Long-running services

Big data processing

Batch scheduling

Data storage

The attributes and resources of Mesos

Attributes

Resources

Examples

Two-level scheduling

Resource allocation

Max-min fair share algorithm

Resource isolation

Monitoring in Mesos

Monitoring provided by Mesos

Types of metrics

The Mesos API

Messages

API details

Executor API

The Executor Driver API

The Scheduler API

The Scheduler Driver API

Mesos in production

Case study on HubSpot

The cluster environment

Benefits

Challenges

Looking ahead

Summary

2. Mesos Internals

Scaling and efficiency

Resource allocation

The Dominant Resource Fairness algorithm (DRF)

Weighted DRF

Configuring resource offers on Mesos

Reservation

Static reservation

Role definition

Framework assignment

Role resource policy setting

Dynamic reservation

Offer::Operation::Reserve

Offer::Operation::Unreserve

/reserve

/unreserve

Oversubscription

Revocable resource offers

Registering with the revocable resources capability

An example offer with a mix of revocable and standard resources

Resource estimator

The QoS controller

Configuring oversubscription

Extendibility

Mesos modules

Module invocation

Building a module

Hooks

The currently supported modules

The allocator module

Implementing a custom allocator module

High availability and fault tolerance

Mastering high availability

Framework scheduler fault tolerance

Slave fault tolerance

Executor/task

Slave recovery

Enabling slave checkpointing

Enabling framework checkpointing

Reconciliation

Task reconciliation

Offer reconciliation

Persistent Volumes

Offer::Operation::Create

Offer::Operation::Destroy

Summary

3. Getting Started with Mesos

Virtual Machine (VM) instances

Setting up a multi-node Mesos cluster on Amazon Web Services (AWS)

Instance types

Launching instances

Installing Mesos

Downloading Mesos

Building Mesos

Using mesos-ec2 script to launch many machines at once

Setting up a multi-node Mesos cluster on Google Compute Engine (GCE)

Introduction to instance types

Launching machines

Set up a Google Cloud Platform project

Create the network and firewall rules

Create the instances

Installing Mesos

Downloading Mesos

Building Mesos

Setting up a multi-node Mesos cluster on Microsoft Azure

Introduction to instance types

Launching machines

Create a cloud service

Create the instances

Configuring the network

Installing Mesos

Downloading Mesos

Building Mesos

Starting mesos-master

Start mesos-slaves

Mesos commands

Testing the installation

Setting up a multi-node Mesos cluster on your private datacenter

Installing Mesos

Preparing the environment

Downloading Mesos

Building Mesos

Starting mesos-master

Starting mesos-slaves

Automating the process when you have many machines

Debugging and troubleshooting

Handling missing library dependencies

Issues with directory permissions

Missing Mesos library (libmesos*.so not found)

Debugging a failed framework

Understanding the Mesos directory structure

Mesos slaves are not connecting with Mesos masters

Launching multiple slave instances on the same machine

Summary

4. Service Scheduling and Management Frameworks

Using Marathon to launch and manage long-running applications on Mesos

Installing Marathon

Installing ZooKeeper to store the state

Launching Marathon in local mode

Multi-node Marathon cluster setup

Launching a test application from the UI

Scaling the application

Terminating the application

Chronos as a cluster scheduler

Installing Chronos

Scheduling a new job

Chronos plus Marathon

The Chronos REST API endpoint

Listing the running jobs

Manually starting a job

Adding a scheduled job

Deleting a job

Deleting all the tasks of a job

The Marathon REST API endpoint

Listing the running applications

Adding an application

Changing the configuration of an application

Deleting the application

Introduction to Apache Aurora

Installing Aurora

Introduction to Singularity

Installing Singularity

Creating a Singularity configuration file

Service discovery using Marathoner

Service discovery using Consul

Running Consul

Load balancing with HAProxy

Creating the bridge between HAProxy and Marathon

Bamboo - Automatically configuring HAProxy for Mesos plus Marathon

Introduction to Netflix Fenzo

Introduction to PaaSTA

A comparative analysis of different Scheduling/Management frameworks

Summary

5. Mesos Cluster Deployment

Deploying and configuring a Mesos cluster using Ansible

Installing Ansible

Installing the control machine

Creating an ansible-mesos setup

Deploying and configuring Mesos cluster using Puppet

Deploying and configuring a Mesos cluster using SaltStack

SaltStack installation

Deploying and configuring a Mesos cluster using Chef

Recipes

Configuring mesos-master

Configuring mesos-slave

Deployin...

Table of contents

• Mastering Mesos