Scala Data Analysis Cookbook

Arun Manivannan

• Publisher: Packt Publishing
• Year: 2015
• eBook ISBN: 9781784394998
• Edition: 1
• Topic: Computer Science
• Subtopic: Object Oriented Programming
• Index: Computer Science

Navigate the world of data analysis, visualization, and machine learning with over 100 hands-on Scala recipesAbout This Book• Implement Scala in your data analysis using features from Spark, Breeze, and Zeppelin• Scale up your data anlytics infrastructure with practical recipes for Scala machine learning• Recipes for every stage of the data analysis process, from reading and collecting data to distributed analyticsWho This Book Is ForThis book shows data scientists and analysts how to leverage their existing knowledge of Scala for quality and scalable data analysis.What You Will Learn• Familiarize and set up the Breeze and Spark libraries and use data structures• Import data from a host of possible sources and create dataframes from CSV• Clean, validate and transform data using Scala to pre-process numerical and string data• Integrate quintessential machine learning algorithms using Scala stack• Bundle and scale up Spark jobs by deploying them into a variety of cluster managers• Run streaming and graph analytics in Spark to visualize data, enabling exploratory analysisIn DetailThis book will introduce you to the most popular Scala tools, libraries, and frameworks through practical recipes around loading, manipulating, and preparing your data. It will also help you explore and make sense of your data using stunning and insightfulvisualizations, and machine learning toolkits.Starting with introductory recipes on utilizing the Breeze and Spark libraries, get to grips withhow to import data from a host of possible sources and how to pre-process numerical, string, and date data. Next, you'll get an understanding of concepts that will help you visualize data using the Apache Zeppelin and Bokeh bindings in Scala, enabling exploratory data analysis. iscover how to program quintessential machine learning algorithms using Spark ML library. Work through steps to scale your machine learning models and deploy them into a standalone cluster, EC2, YARN, and Mesos. Finally dip into the powerful options presented by Spark Streaming, and machine learning for streaming data, as well as utilizing Spark GraphX.Style and approachThis book contains a rich set of recipes that covers the full spectrum of interesting data analysis tasks and will help you revolutionize your data analysis skills using Scala and Spark.

Scala Data Analysis Cookbook

---

Table of Contents

Scala Data Analysis Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why Subscribe?

Free Access for Packt account holders

Preface

Apache Flink

Scalding

Saddle

Spire

Akka

Accord

What this book covers

What you need for this book

Who this book is for

Sections

Getting ready

How to do it…

How it works…

There's more…

See also

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Getting Started with Breeze

Introduction

Getting Breeze – the linear algebra library

How to do it...

There's more...

The org.scalanlp.breeze dependency

The org.scalanlp.breeze-natives package

Working with vectors

Getting ready

How to do it...

Creating vectors

Constructing a vector from values

Creating a zero vector

Creating a vector out of a function

Creating a vector of linearly spaced values

Creating a vector with values in a specific range

Creating an entire vector with a single value

Slicing a sub-vector from a bigger vector

Creating a Breeze Vector from a Scala Vector

Vector arithmetic

Scalar operations

Calculating the dot product of two vectors

Creating a new vector by adding two vectors together

Appending vectors and converting a vector of one type to another

Concatenating two vectors

Converting a vector of Int to a vector of Double

Computing basic statistics

Mean and variance

Standard deviation

Find the largest value in a vector

Finding the sum, square root and log of all the values in the vector

The Sqrt function

The Log function

Working with matrices

How to do it...

Creating matrices

Creating a matrix from values

Creating a zero matrix

Creating a matrix out of a function

Creating an identity matrix

Creating a matrix from random numbers

Creating from a Scala collection

Matrix arithmetic

Addition

Multiplication

Appending and conversion

Concatenating matrices – vertically

Concatenating matrices – horizontally

Converting a matrix of Int to a matrix of Double

Data manipulation operations

Getting column vectors out of the matrix

Getting row vectors out of the matrix

Getting values inside the matrix

Getting the inverse and transpose of a matrix

Computing basic statistics

Mean and variance

Standard deviation

Finding the largest value in a matrix

Finding the sum, square root and log of all the values in the matrix

Sqrt

Log

Calculating the eigenvectors and eigenvalues of a matrix

How it works...

Vectors and matrices with randomly distributed values

How it works...

Creating vectors with uniformly distributed random values

Creating vectors with normally distributed random values

Creating vectors with random values that have a Poisson distribution

Creating a matrix with uniformly random values

Creating a matrix with normally distributed random values

Creating a matrix with random values that has a Poisson distribution

Reading and writing CSV files

How it works...

2. Getting Started with Apache Spark DataFrames

Introduction

Getting Apache Spark

How to do it...

Creating a DataFrame from CSV

How to do it...

How it works...

There's more…

Manipulating DataFrames

How to do it...

Printing the schema of the DataFrame

Sampling the data in the DataFrame

Selecting DataFrame columns

Filtering data by condition

Sorting data in the frame

Renaming columns

Treating the DataFrame as a relational table

Joining two DataFrames

Inner join

Right outer join

Left outer join

Saving the DataFrame as a file

Creating a DataFrame from Scala case classes

How to do it...

How it works...

3. Loading and Preparing Data – DataFrame

Introduction

Loading more than 22 features into classes

How to do it...

How it works...

There's more…

Loading JSON into DataFrames

How to do it…

Reading a JSON file using SQLContext.jsonFile

Reading a text file and converting it to JSON RDD

Explicitly specifying your schema

There's more…

Storing data as Parquet files

How to do it…

Load a simple CSV file, convert it to case classes, and create a DataFrame from it

Save it as a Parquet file

Install Parquet tools

Using the tools to inspect the Parquet file

Enable compression for the Parquet file

Using the Avro data model in Parquet

How to do it…

Creation of the Avro model

Generation of Avro objects using the sbt-avro plugin

Constructing an RDD of our generated object from Students.csv

Saving RDD[StudentAvro] in a Parquet file

Reading the file back for verification

Using Parquet tools for verification

Loading from RDBMS

How to do it…

Preparing data in Dataframes

How to do it...

4. Data Visualization

Introduction

Visualizing using Zeppelin

How to do it...

Installing Zeppelin

Customizing Zeppelin's server and websocket port

Visualizing data on HDFS – parameterizing inputs

Running custom functions

Adding external dependencies t...

Table of contents

• Scala Data Analysis Cookbook