Hands-On Data Science with the Command Line

Automate everyday data science tasks using command-line tools

Jason Morris, Chris McCubbin, Raymond Page

• Publisher: Packt Publishing
• Year: 2019
• Print ISBN: 9781789132984
• eBook ISBN: 9781788991919
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Modelling & Design
• Index: Computer Science

Big data processing and analytics at speed and scale using command line tools.

Key Features

• Perform string processing, numerical computations, and more using CLI tools
• Understand the essential components of data science development workflow
• Automate data pipeline scripts and visualization with the command line

Book Description

The Command Line has been in existence on UNIX-based OSes in the form of Bash shell for over 3 decades. However, very little is known to developers as to how command-line tools can be OSEMN (pronounced as awesome and standing for Obtaining, Scrubbing, Exploring, Modeling, and iNterpreting data) for carrying out simple-to-advanced data science tasks at speed.

This book will start with the requisite concepts and installation steps for carrying out data science tasks using the command line. You will learn to create a data pipeline to solve the problem of working with small-to medium-sized files on a single machine. You will understand the power of the command line, learn how to edit files using a text-based and an. You will not only learn how to automate jobs and scripts, but also learn how to visualize data using the command line.

By the end of this book, you will learn how to speed up the process and perform automated tasks using command-line tools.

What you will learn

• Understand how to set up the command line for data science
• Use AWK programming language commands to search quickly in large datasets.
• Work with files and APIs using the command line
• Share and collect data with CLI tools
• Perform visualization with commands and functions
• Uncover machine-level programming practices with a modern approach to data science

Who this book is for

This book is for data scientists and data analysts with little to no knowledge of the command line but has an understanding of data science. Perform everyday data science tasks using the power of command line tools.

Loops, Functions, and String Processing

Sometimes, magic one-liners are insufficient for manipulating data. Loops and conditionals enable us to iterate over data in interesting ways without sticking to default behavior.

Bash views non-binary files and streams as collections of characters. We commonly think of these characters as groups of strings separated by some kind of whitespace. It makes sense that some of the most useful and common tools in the command-line universe are the ones that search and manipulate these strings.

The following topics will be covered in this chapter:

• for loops
• while loops
• File test conditionals
• Numeric comparisons
• String case statements
• Using regular expressions and grep to search and filter
• String transformations using awk, sed, and tr
• Sorting lists of strings with sort and uniq

Along the way, we'll see how we can pipe the results of one program into another to get the results we want.

Once, twice, three times a lady loops

Few command-line tools have implicit looping and conditionals built into them. Often, tasks will only operate on each line of an input stream and then terminate. The shell provides just enough control flow and conditionals to solve many complex problems, making up for any deficiencies that command-line tools have for operating on data.

The almighty for loop is a common loop idiom, however bash's for loop might feel a little unfamiliar to users of more traditional languages. The for loop allows you to iterate over a list of words, and assign each one to a variable for processing. For example, (pun intended):

Often, we want a more traditional range of numbers in our for loops. The POSIX method of generating a number range is to use the seq command, as in seq -- $(seq 1 1 5), which will generate numbers from 1 (the first argument) to 5 (the third argument) in steps of 1 increment (the second argument).

In the following examples, you'll notice we are using bracket expansions, {}, and parentheses, (). For more information about both, check out https://ss64.com/bash/syntax-brackets.html.

Modern versions of bash provide an easy shorthand for this:

We can also set the amount that the sequence is incremented by:

Alternatively, we can use the bash supported C-like syntax:

Looping for a specified number of times may be what we need, but we can also pass in the result of a sub-command to generate the list of things to loop over. For example, we may want to do something to each file in the current directory:

Often, we may want to test one or more conditionals, especially in loops. Bash has an if-then construct, like most languages:

The statement inside the brackets is a test, and bash contains a set of special tests, such as -f for common tasks. Here's a list of some of the most common ones:

Test type	Parameter	Description
Filesystem	-O	True if file exists and is owned by the effective user ID
Filesystem	-f	True if file exists and is a regular file
Filesystem	-G	True if file exists and is owned by the effective group ID
Filesystem	-r	True if file exists and is readable
Filesystem	-w	True if file exists and is writable
Filesystem	-x	True if file exists and is executable
Filesystem	-s	True if file exists and has a size greater than ...

Table of contents

• Title Page
• Copyright and Credits
• About Packt
• Contributors
• Preface
• Data Science at the Command Line and Setting It Up
• Essential Commands
• Shell Workflows, and Data Acquisition and Massaging
• Bash Functions and Data Visualization
• Loops, Functions, and String Processing
• SQL, Math, and Wrapping it up
• Other Books You May Enjoy