Bash Cookbook

Leverage Bash scripting to automate daily tasks and improve productivity

Ron Brash, Ganesh Naik

• Publisher: Packt Publishing
• Year: 2018
• Print ISBN: 9781788629362
• eBook ISBN: 9781788839174
• Edition: 1
• Topic: Computer Science
• Subtopic: Operating Systems
• Index: Computer Science

Create simple to advanced shell scripts and enhance your system functionality with effective recipes

Key Features

• Automate tedious and repetitive tasks
• Create several novel applications ranging from a simple IRC logger to a Web Scraper
• Manage your system efficiently by becoming a seasoned Bash user

Book Description

In Linux, one of the most commonly used and most powerful tools is the Bash shell. With its collection of engaging recipes, Bash Cookbook takes you through a series of exercises designed to teach you how to effectively use the Bash shell in order to create and execute your own scripts.

The book starts by introducing you to the basics of using the Bash shell, also teaching you the fundamentals of generating any input from a command. With the help of a number of exercises, you will get to grips with the automation of daily tasks for sysadmins and power users. Once you have a hands-on understanding of the subject, you will move on to exploring more advanced projects that can solve real-world problems comprehensively on a Linux system. In addition to this, you will discover projects such as creating an application with a menu, beginning scripts on startup, parsing and displaying human-readable information, and executing remote commands with authentication using self-generated Secure Shell (SSH) keys.

By the end of this book, you will have gained significant experience of solving real-world problems, from automating routine tasks to managing your systems and creating your own scripts.

What you will learn

• Understand the basics of Bash shell scripting on a Linux system
• Gain working knowledge of how redirections and pipes interact
• Retrieve and parse input or output of any command
• Automate tasks such as data collection and creating and applying a patch
• Create a script that acts like a program with different features
• Customize your Bash shell and discover neat tricks to extend your programs
• Compile and install shell and log commands on your system's console using Syslog

Who this book is for

The Bash Cookbook is for you if you are a power user or system administrator involved in writing Bash scripts in order to automate tasks. This book is also ideal if you are interested in learning how to automate complex daily tasks.

Acting Like a Typewriter and File Explorer

In this chapter, we will introduce the following:

• Basic searching for strings and files
• Using wildcards and regexes
• Math and calculations in script
• Striping/altering/sorting/deleting/searching strings with Bash only
• Using SED and AWK to remove/replace substrings
• Formatting your data/output using echo and printf
• Readying your script for different languages with internationalization
• Calculating statistics and reducing duplicates based on file contents
• Using file attributes with conditional logic
• Reading delimited data and altered output format

Introduction

Hopefully, the previous Bash crash course chapter provided more than a hint of the utility and power of Bash. On the other hand, this chapter introduces several bolt-on technologies to make Bash even more extensive when searching for items and text, or automating file explorer/file system operations.

By itself, Bash is merely a powerful scripting language, but much of Bash's flexibility comes from being able to "glue" other technologies (tools or languages) together to make the output more useful. In other words, Bash is a base platform similar to how some auto/car lovers choose a particular platform before making their modifications. Will a modified car do everything, even with enhancements? Certainly not, but it can make it more powerful or useful in specific cases, and at least provides four wheels for movement.

Not only do common scripts contain a series of commands for automation, they often include logic to modify strings such as the following:

• Removing trailing characters
• Replacing sections of words (substrings)
• Searching for strings in files
• Finding files
• Testing file types (directory, file, empty, and so on)
• Performing small calculations
• Limiting the scope of searches or data (filtering)
• Modifying the contents of variables (strings inside of string variables)

This logic that modifies, limits, and even replaces input/output data can be very powerful when you need to execute broad searches for a specific string or when you have copious amounts of data. Terminals chock; full of output or massive data files can be very daunting to explore!

However, there is one very important concept that still needs to be discussed, and that is recursive functionality. Recursive functionality can apply to script functions, logic, and even a command operation. For example, you can use grep to recursively crawl an entire directory until no more files remain, or you can recursively execute a function inside of itself until a condition is met (for example, printing a single character at a time within a string):

# e.g. File system
# / (start here)
# /home (oh we found home)
# /home/user (neat there is a directory inside it called user)
# /home/user/.. (even better, user has files - lets look in them too)
# /etc/ # We are done with /home and its "children" so lets look in /etc
# ... # Until we are done

Be careful with recursion (especially with functions), as it can sometimes be really slow depending on the complexity of the structure (for example, file system or size of files). Also if there is a logic error, you can keep executing functions recursively forever!

This chapter is all about limiting data, utilizing it, modifying it, internationalizing it, replacing it, and even searching for it in the first place.

Basic searching for strings and files

Imagine searching for a four leaf clover in a big garden. It would be really hard (and it is still really hard for computers). Thankfully, words are not images and text on a computer is easily searchable depending on the format. The term format has to be used because if your tool cannot understand a given type of text (encoding), then you might have trouble recognizing a pattern or even detecting that there is text at all!

Typically, when you are looking at the console, text files, source code (C, C++, Bash, HTML), spreadsheets, XML, and other types, you are looking at it in ASCII or UTF. ASCII is a commonly used format in the *NIX world on the console. There is also the UTF encoding scheme, which is an improvement upon ASCII and can support a variety of extended characters that were not present in computing originally. It comes in a number of formats such as UTF-8, UTF-16, and UTF32.

When you hear the words encoding and decoding, it is similar to encryption and decryption. The purpose is not to hide something, but rather to transform some data into something appropriate for the use case. For example, transmission, usage with languages, and compression.
ASCII and UTF are not the only types your target data might be in. In various types of files, you may encounter different types of encoding of data. This is a different problem that's specific to your data and will need additional considerations.

In this recipe, we will begin the process of searching for strings and a couple of ways to search for some of your own needles in a massive haystack of data. Let's dig in.

Getting ready

Besides having a terminal open (and your favorite text editor, if necessary), we only need a couple of core commands such as grep, ls, mkdir, touch, traceroute, strings, wget, xargs, and find.

Assuming that your user already has the correct permissions for your usage (and authorized, of course), we will need to generate data to begin searching:

$ ~/
$ wget --recursive --no-parent https://www.packtpub.com www.packtpub.com # Takes awhile
$ traceroute packtpub.com > traceroute.txt
$ mkdir -p www.packtpub.com/filedir www.packtpub.com/emptydir
$ touch www.packtpub.com/filedir/empty.txt
$ touch www.packtpub.com/findme.xml; echo "<xml>" www.packtpub.com/findme.xml

How to do it...

Using the data obtained by recursively crawling the Packt Publishing website, we can see that inside of www.packtpub.com the entire website is available. Wow! We also created some test data directories and files.

1. Next, open up a terminal and create the following script:

#!/bin/bash

# Let's find all the files with the string "Packt"
DIRECTORY="www.packtpub.com/"
SEARCH_TERM="Packt"

# Can we use grep?
grep "${SEARCH_TERM}" ~/* > result1.txt 2&> /dev/null

# Recursive check
grep -r "${SEARCH_TERM}" "${DIRECTORY}" > result2.txt

# What if we want to check for multiple terms?
grep -r -e "${SEARCH_TERM}" -e "Publishing" "${DIRECTORY}" > result3.txt

# What about find?
find "${DIRECTORY}" -type f -print | xargs grep "${SEARCH_TERM}" > result4.txt

# What about find and looking for the string inside of a specific type of content?
find "${DIRECTORY}" -type f -name "*.xml" ! -name "*.css" -print | xargs grep "${SEARCH_TERM}" > result5.txt

# Can this also be achieved with wildcards and subshell?
grep "${SEARCH_TERM}" $(ls -R "${DIRECTORY}"*.{html,txt}) > result6.txt
RES=$?

if [ ${RES} -eq 0 ]; then
 echo "We found results!"
else
 echo "It broke - it shouldn't happen (Packt is everywhere)!"
fi

# Or for bonus points - a personal favorite
history | grep "ls" # This is really handy to find commands you ran yesterday!

# Aaaannnd the lesson is:
echo "We can do a lot with grep!"
exit 0

Notice in the script the use of ~/* ?. This refers to our home directory and introduces the * wildcard, which allows us to specify anything from that point on. There will be more on the concept of wildcards and regexes later in this chapter.

2. If you remain in your home directory (~/) and run the script, the output should be similar to the following:

$ bash search.sh; ls -lah result*.txt
We found results!
We can do a lot with grep!
-rw-rw-r-- 1 rbrash rbrash 0 Nov 14 14:33 result1.txt
-rw-rw-r-- 1 rbrash rbrash 1.2M Nov 14 14:33 result2.txt
-rw-rw-r-- 1 rbrash rbrash 1.2M Nov 14 14:33 result3.txt
-rw-rw-r-- 1 rbrash rbrash 1.2M Nov 14 14:33 result4.txt
-rw-rw-r-- 1 rbrash rbrash 33 Nov 14 14:33 result5.txt
-rw-rw-r-- 1 rbrash rbrash 14K Nov 14 14:33 result6.txt

How it works...

This section is a bit of a doozy because we are leading up to another ...

Table of contents

• Title Page
• Copyright and Credits
• Packt Upsell
• Contributors
• Preface
• Crash Course in Bash
• Acting Like a Typewriter and File Explorer
• Understanding and Gaining File System Mastery
• Making a Script Behave Like a Daemon
• Scripts for System Administration Tasks
• Scripts for Power Users
• Writing Bash to Win and Profit
• Advanced Scripting Techniques
• Other Books You May Enjoy