Raspberry Pi Super Cluster

Andrew K. Dennis

• Publisher: Packt Publishing
• Year: 2013
• eBook ISBN: 9781783286195
• Topic: Computer Science
• Subtopic: Hardware
• Index: Computer Science

In Detail

A cluster is a type of parallel/distributed processing system which consists of a collection of interconnected stand-alone computers cooperatively working together. Using Raspberry Pi computers, you can build a two-node parallel computing cluster which enhances performance and availability.

This practical, example-oriented guide will teach you how to set up the hardware and operating systems of multiple Raspberry Pi computers to create your own cluster. It will then navigate you through how to install the necessary software to write your own programs such as Hadoop and MPICH before moving on to cover topics such as MapReduce. Throughout this book, you will explore the technology with the help of practical examples and tutorials to help you learn quickly and efficiently.

Starting from a pile of hardware, with this book, you will be guided through exciting tutorials that will help you turn your hardware into your own super-computing cluster. Youll start out by learning how to set up your Raspberry Pi clusters hardware. Following this, you will be taken through how to install the operating system, and you will also be given a taste of what parallel computing is about. With your Raspberry Pi cluster successfully set up, you will then install software such as MPI and Hadoop. Having reviewed some examples and written some programs that explore these two technologies, you will then wrap up with some fun ancillary projects. Finally, you will be provided with useful links to help take your projects to the next step.

Approach

This book follows a step-by-step, tutorial-based approach which will teach you how to develop your own super cluster using Raspberry Pi computers quickly and efficiently.

Who this book is for

Raspberry Pi Super Cluster is an introductory guide for those interested in experimenting with parallel computing at home. Aimed at Raspberry Pi enthusiasts, this book is a primer for getting your first cluster up and running.

Basic knowledge of C or Java would be helpful but no prior knowledge of parallel computing is necessary.

Raspberry Pi Super Cluster

---

Table of Contents

Raspberry Pi Super Cluster

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers and more

Why Subscribe?

Free Access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Clusters, Parallel Computing, and Raspberry Pi – A Brief Background

A very short history of parallel computing

Supercomputers

Multi-core and multiprocessor machines

Commodity hardware clusters

Cloud computing

Big data

Raspberry Pi and parallel computing

Programming languages and frameworks

Summary

2. Setting Up your Raspberry Pi Software and Hardware for Parallel Computing

Setting up our work environment

HDMI-capable monitor or VGA/DVI monitor and adapter

USB keyboard and mouse

Two micro-USB power units

A desk-mounted power strip with both USB and mains outlets (optional)

Three Ethernet/RJ45 network cables

A small network switch

An existing Internet connection

Two SD cards that are compatible with the Raspberry Pi

Housing units for the Raspberry Pi boards and Lego (optional)

USB hard drives (optional)

Future expansion and a scalable setup

Completing the initial setup

Using an SD card as our Raspberry Pi's storage device

SD card setup

Formatting our card

Mac OS X SD card formatting instructions

Windows 8 SD card formatting instructions

Linux instructions for SD card formatting

BerryBoot version 2

Downloading the BerryBoot version 2 ZIP file

Mac OS X

Windows 8

Linux

Starting up the Raspberry Pi

The installation process

Installation complete

Testing SSH and setting up keys

Connecting via SSH

Mac OS X and Linux users

Windows 8 users with PuTTY

SSH running successfully

Setting up your SSH RSA keys

The ssh-agent and ssh-add tools

SSH setup complete

Wrapping up

Editing text files on Raspbian

Installing Fortran

Terminal multiplexing with Screen

Summary

3. Parallel Computing – MPI on the Raspberry Pi

MPI – Message Passing Interface

MPI implementations – MPICH and OpenMPI

Creating an environment and downloading MPICH

Building and installing MPICH

Configuring your Raspberry Pi to run with MPICH

Testing our MPICH installation

Building our second Raspberry Pi

Windows 8

Mac OS X

Linux

Powering up the second Raspberry Pi

RSA key setup for SSH

Writing an MPI-based application

MPI – point-to-point communication

Summary

4. Hadoop – Distributed Applications on the Raspberry Pi

A brief introduction to Apache Hadoop

Installing Java

Installing Apache Hadoop

Hadoop configuration

Testing our Hadoop server

Setting up our second Raspberry Pi

Summary

5. MapReduce Applications with Hadoop and Java

MapReduce

MapReduce in Hadoop

HDFS – The Hadoop distributed file system

The WordCount MapReduce program

Testing our application

Summary

6. Calculate Pi with Hadoop and MPI

Monte Carlo simulators

A Hadoop application to calculate Pi

Pi with C language and MPI

Summary

7. Going Further

Booting from an external USB HDD

Building a Lego enclosure

Experimenting with MPI and Fortran

Power for multiple devices

USB wall plates

Battery power

Using a PC power supply

Power over Ethernet

Summary

A. Appendix

Fortran and C/C++

MPI, Hadoop, and parallel computing

Raspberry Pi cases and clusters

Index

Raspberry Pi Super Cluster

Copyright © 2013 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: November 2013

Production Reference: 1131113

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78328-619-5

www.packtpub.com

Cover Image by Aniket Sawant (<[email protected]>)

Credits

Author

Andrew K. Dennis

Reviewers

Prasanna Gautam

Sungjin Han

Claes Jakobsson

Acquisition Editors

Anthony Albuquerque

Edward Gordon

Commissioning Editor

Amit Ghodake

Technical Editors

Faisal Siddiqui

Sonali S. Vernekar

Project Coordinator

Aboli Ambardekar

Proofreader

Stephen Copestake

Indexer

Monica Ajmera Mehta

Graphics

Abhinash Sahu

Production Coordinator

Alwin Roy

Cover Work

Alwin Roy

About the Author

Andrew K. Dennis is the Manager of Application Development at Prometheus Research. Prometheus Research is a leading provider of integrated data management for research and the home of HTSQL, an open source navigational query language for RDMS.

Andrew has a Diploma in Computing and a BS in Software Engineering; he is currently studying a second BS in Creative Computing in his spare time.

He has over 10 years of experience working in the software industry in the UK, Canada, and USA. This experience includes e-Learning, CMS and LMS development, SCORM consultancy, web development in a variety of languages, open source application development, and running a blog dedicated to maker culture and home automation.

His interests include web development, e-Learning, 3D printing, Linux, the Raspberry Pi and Arduino, open source projects, parallel computing, home automation, amateur electronics, home networ...

Table of contents

• Raspberry Pi Super Cluster