Text Processing with Ruby

Extract Value from the Data That Surrounds You

Rob Miller

• Publisher: Pragmatic Bookshelf
• Year: 2015
• eBook ISBN: 9781680504927
• Topic: Computer Science
• Subtopic: Data Processing
• Index: Computer Science

Most information in the world is in text format, and programmers often find themselves needing to make sense of the data hiding within. You want to do this efficiently, avoiding labor-intensive, manual work—and Ruby is ideally suited to this task.

Text Processing with Ruby takes a practical approach to working with text:

• First, Acquire: Explore Ruby's core and standard library, and what's possible with IO and its derived classes like File. Extract text into your Ruby programs from the file system and standard input. Process delimited files such as CSVs, and write utilities that interact with other programs in text-processing pipelines. Process web pages with Nokogiri to pull out information from even the messiest of HTML, and decipher character encoding mysteries.
• Second, Transform: Use regular expressions to match, extract, and replace patterns in text. Write a parser using Ruby's StringScanner library. Use Natural Language Processing techniques to extract keywords and implement fuzzy searching.
• Finally, Load: Write the transformed text and data to standard output, files and other processes. Serialize text into JSON, XML, and CVS, and use ERB to create more complex formats.

You'll soon be able to tackle even the most enormous and entangled text with ease, scything through gigabytes of data and effortlessly extracting the bits that matter.

Top Five Text Processing Tips
by Rob Miller, author of Text Processing with Ruby

Clean up your data first
Data in the real world is messy. It almost always pays off to take some
time to normalize different sources of data and to get them into the
same format before you begin whatever actual processing you need to do.
You'll have less exceptions and special cases in your code, and it'll be
a lot more resilient.

Master regular expressions
There are definitely some text processing problems that can't be solved
with regular expressions, but not that many. While they're not always
the best or more readable option, knowing regular expressions well will
get you out of many tight spots, and even more often than that will be
the first step towards a more robust solution.

Break your problem into discrete steps
Almost all text processing tasks, no matter how complicated they seem on the face of it, are really a series of small transformations. Figuring out how to frame your problem in this way will make it easy to take a pipeline approach, where your text flows through a series of small,
discrete steps, each of which transform the data in a particular way and
then passes it on. Such programs are both easier to reason about and
easier to modify and extend.

Figure out a strategy for missing data
Data in the real world, as well as being messy, also frequently has gaps. Decide early on how you're going to cope with that — how you'll represent the absence of particular fields or properties — and you'll
avoid messiness later on.

Make the most of existing tools
There are hundreds of command-line tools that exist solely to process
textual data. Each of them is capable of performing a particular
transformation, which means you don't need to reinvent the wheel. If you
use existing tools for the parts of your problem that have already been
solved, all that remains is to solve the unique problem that you have.

Table of contents

• Text Processing with Ruby