Apache Spark 2.x Cookbook

Rishi Yadav

• Publisher: Packt Publishing
• Year: 2017
• ISBN:
  9781787127517
• Edition: 2
• Topic: Informatica
• Subtopic: Elaborazione di dati

Over 70 recipes to help you use Apache Spark as your single big data computing platform and master its librariesAbout This Book• This book contains recipes on how to use Apache Spark as a unified compute engine• Cover how to connect various source systems to Apache Spark• Covers various parts of machine learning including supervised/unsupervised learning & recommendation enginesWho This Book Is ForThis book is for data engineers, data scientists, and those who want to implement Spark for real-time data processing. Anyone who is using Spark (or is planning to) will benefit from this book. The book assumes you have a basic knowledge of Scala as a programming language.What You Will Learn• Install and configure Apache Spark with various cluster managers & on AWS• Set up a development environment for Apache Spark including Databricks Cloud notebook• Find out how to operate on data in Spark with schemas• Get to grips with real-time streaming analytics using Spark Streaming & Structured Streaming• Master supervised learning and unsupervised learning using MLlib• Build a recommendation engine using MLlib• Graph processing using GraphX and GraphFrames libraries• Develop a set of common applications or project types, and solutions that solve complex big data problemsIn DetailWhile Apache Spark 1.x gained a lot of traction and adoption in the early years, Spark 2.x delivers notable improvements in the areas of API, schema awareness, Performance, Structured Streaming, and simplifying building blocks to build better, faster, smarter, and more accessible big data applications. This book uncovers all these features in the form of structured recipes to analyze and mature large and complex sets of data.Starting with installing and configuring Apache Spark with various cluster managers, you will learn to set up development environments. Further on, you will be introduced to working with RDDs, DataFrames and Datasets to operate on schema aware data, and real-time streaming with various sources such as Twitter Stream and Apache Kafka. You will also work through recipes on machine learning, including supervised learning, unsupervised learning & recommendation engines in Spark.Last but not least, the final few chapters delve deeper into the concepts of graph processing using GraphX, securing your implementations, cluster optimization, and troubleshooting.Style and approachThis book is packed with intuitive recipes supported with line-by-line explanations to help you understand Spark 2.x's real-time processing capabilities and deploy scalable big data solutions. This is a valuable resource for data scientists and those working on large-scale data projects.

Getting Started with Apache Spark

In this chapter, we will set up Spark and configure it. This chapter contains the following recipes:

• Leveraging Databricks Cloud
• Deploying Spark using Amazon EMR
• Installing Spark from binaries
• Building the Spark source code with Maven
• Launching Spark on Amazon EC2
• Deploying Spark on a cluster in standalone mode
• Deploying Spark on a cluster with Mesos
• Deploying Spark on a cluster with YARN
• Understanding SparkContext and SparkSession
• Understanding Resilient Distributed Datasets (RDD)

Introduction

Apache Spark is a general-purpose cluster computing system to process big data workloads. What sets Spark apart from its predecessors, such as Hadoop MapReduce, is its speed, ease of use, and sophisticated analytics.

It was originally developed at AMPLab, UC Berkeley, in 2009. It was made open source in 2010 under the BSD license and switched to the Apache 2.0 license in 2013. Toward the later part of 2013, the creators of Spark founded Databricks to focus on Spark's development and future releases.

Databricks offers Spark as a service in the Amazon Web Services(AWS) Cloud, called Databricks Cloud. In this book, we are going to maximize the use of AWS as a data storage layer.

Talking about speed, Spark can achieve subsecond latency on big data workloads. To achieve such low latency, Spark makes use of memory for storage. In MapReduce, memory is primarily used for the actual computation. Spark uses memory both to compute and store objects.

Spark also provides a unified runtime connecting to various big data storage sources, such as HDFS, Cassandra, and S3. It also provides a rich set of high-level libraries for different big data compute tasks, such as machine learning, SQL processing, graph processing, and real-time streaming. These libraries make development faster and can be combined in an arbitrary fashion.

Though Spark is written in Scala--and this book only focuses on recipes on Scala--it also supports Java, Python, and R.

Spark is an open source community project, and everyone uses the pure open source Apache distributions for deployments, unlike Hadoop, which has multiple distributions available with vendor enhancements.

The following figure shows the Spark ecosystem:

Spark's runtime runs on top of a variety of cluster managers, including YARN (Hadoop's compute framework), Mesos, and Spark's own cluster manager called Standalone mode. Alluxio is a memory-centric distributed file system that enables reliable file sharing at memory speed across cluster frameworks. In short, it is an off-heap storage layer in memory that helps share data across jobs and users. Mesos is a cluster manager, which is evolving into a data center operating system. YARN is Hadoop's compute framework and has a robust resource management feature that Spark can seamlessly use.

Apache Spark, initially devised as a replacement of MapReduce, had a good proportion of workloads running in an on-premises manner. Now, most of the workloads have been moved to public clouds (AWS, Azure, and GCP). In a public cloud, we see two types of applications:

• Outcome-driven applications
• Data transformation pipelines

For outcome-driven applications, where the goal is to derive a predefined signal/outcome from the given data, Databricks Cloud fits the bill perfectly. For traditional data transformation pipelines, Amazon's Elastic MapReduce (EMR) does a great job.

Leveraging Databricks Cloud

Databricks is the company behind Spark. It has a cloud platform that takes out all of the complexity of deploying Spark and provides you with a ready-to-go environment with notebooks for various languages. Databricks Cloud also has a community edition that provides one node instance with 6 GB of RAM for free. It is a great starting place f...

Table of contents

• Title Page
• Copyright
• Credits
• About the Author
• About the Reviewer
• www.PacktPub.com
• Customer Feedback
• Preface
• Getting Started with Apache Spark
• Developing Applications with Spark
• Spark SQL
• Working with External Data Sources
• Spark Streaming
• Getting Started with Machine Learning
• Supervised Learning with MLlib — Regression
• Supervised Learning with MLlib — Classification
• Unsupervised Learning
• Recommendations Using Collaborative Filtering
• Graph Processing Using GraphX and GraphFrames
• Optimizations and Performance Tuning