Python Parallel Programming Cookbook

Over 70 recipes to solve challenges in multithreading and distributed system with Python 3, 2nd Edition

Giancarlo Zaccone

• Publisher: Packt Publishing
• Year: 2019
• Print ISBN: 9781789533736
• eBook ISBN: 9781789530063
• Edition: 2
• Topic: Computer Science
• Subtopic: Bioinformatics
• Index: Computer Science

Implement effective programming techniques in Python to build scalable software that saves time and memory

Key Features

• Design distributed computing systems and massive computational tasks coherently
• Learn practical recipes with concise explanations that address development pain points encountered while coding parallel programs
• Understand how to host your parallelized applications on the cloud

Book Description

Nowadays, it has become extremely important for programmers to understand the link between the software and the parallel nature of their hardware so that their programs run efficiently on computer architectures. Applications based on parallel programming are fast, robust, and easily scalable.

This updated edition features cutting-edge techniques for building effective concurrent applications in Python 3.7. The book introduces parallel programming architectures and covers the fundamental recipes for thread-based and process-based parallelism. You'll learn about mutex, semaphores, locks, queues exploiting the threading, and multiprocessing modules, all of which are basic tools to build parallel applications. Recipes on MPI programming will help you to synchronize processes using the fundamental message passing techniques with mpi4py. Furthermore, you'll get to grips with asynchronous programming and how to use the power of the GPU with PyCUDA and PyOpenCL frameworks. Finally, you'll explore how to design distributed computing systems with Celery and architect Python apps on the cloud using PythonAnywhere, Docker, and serverless applications.

By the end of this book, you will be confident in building concurrent and high-performing applications in Python.

What you will learn

• Synchronize multiple threads and processes to manage parallel tasks
• Use message passing techniques to establish communication between processes to build parallel applications
• Program your own GPU cards to address complex problems
• Manage computing entities to execute distributed computational task
• Write efficient programs by adopting the event-driven programming model
• Explore cloud technology with Django and Google App Engine
• Apply parallel programming techniques that can lead to performance improvements

Who this book is for

The Python Parallel Programming Cookbook is for software developers who are well-versed with Python and want to use parallel programming techniques to write powerful and efficient code. This book will help you master the basics and the advanced of parallel computing.

Heterogeneous Computing

This chapter will help us to explore the Graphics Processing Unit (GPU) programming techniques through the Python language. The continuous evolution of GPUs is revealing how these architectures can bring great benefits to performing complex calculations.

GPUs certainly cannot replace CPUs. However, they are a well-structured and heterogeneous code that is able to exploit the strengths of both types of processors that can, in fact, bring considerable advantages.

We will examine the main development environments for heterogeneous programming, namely, the PyCUDA and Numba environments for Compute Unified Device Architecture (CUDA) and PyOpenCL environments, which are for Open Computing Language (OpenCL) frameworks in their Python version.

In this chapter, we will cover the following recipes:

• Understanding heterogeneous computing
• Understanding the GPU architecture
• Understanding GPU programming
• Dealing with PyCUDA
• Heterogeneous programming with PyCUDA
• Implementing memory management with PyCUDA
• Introducing PyOpenCL
• Building applications with PyOpenCL
• Element-wise expressions with PyOpenCL
• Evaluating PyOpenCL applications
• GPU programming with Numba

Let's start with understanding heterogeneous computing in detail.

Understanding heterogeneous computing

Over the years, the search for better performance for increasingly complex calculations has led to the adoption of new techniques in the use of computers. One of these techniques is called heterogeneous computing, which aims to cooperate with different (or heterogeneous) processors in such a way as to have advantages (in particular) in terms of temporal computational efficiency.

In this context, the processor on which the main program is run (generally the CPU) is called the host, while the coprocessors (for example, the GPUs) are called devices. The latter are generally physically separated from the host and manage their own memory space, which is also separated from the host's memory.

In particular, following significant market demand, the GPU has evolved into a highly parallel processor, transforming the GPU from devices for graphics rendering to devices for parallelizable and computationally intensive general-purpose calculations.

In fact, the use of GPU for tasks other than rendering graphics on the screen is called heterogeneous computing.

Finally, the task of good GPU programming is to make the most of the great level of parallelism and mathematical capabilities offered by the graphics card, minimizing all the disadvantages presented by it, such as the delay of the physical connection between the host and device.

Understanding the GPU architecture

A GPU is a specialized CPU/core for vector processing of graphical data to render images from polygonal primitives. The task of a good GPU program is to make the most of the great level of parallelism and mathematical capabilities offered by the graphics card and minimize all the disadvantages presented by it, such as the delay in the physical connection between the host and device.

GPUs are characterized by a highly parallel structure that allows you to manipulate large datasets in an efficient manner. This feature is combined with rapid improvements in hardware performance programs, bringing the attention of the scientific world to the possibility of using GPUs for purposes other than just rendering images.

A GPU (refer to the following diagram) is composed of several processing units called Streaming Multiprocessors (SMs), which represent the first logic level of parallelism. In fact, each SM works simultaneously and independently from the others:

GPU architecture

Each SM is divided into a group of Streaming Processors (SPs), which have a core that can run a thread sequentially. The SP represents the smallest unit of execution logic and the level of finer parallelism.

In order to best program this type of architecture, we need to introduce GPU programming, which is described in the next section.

Understanding GPU programming

GPUs have become increasingly programmable. In fact, their set of instructions has been extended to allow the execution of a greater number of tasks.

Today, on a GPU, it is possible to execute classic CPU programming instructions, such as cycles and conditions, memory access, and floating-point calculations. The two major discrete video card manufacturers—NVIDIA and AMD—have developed their GPU architectures, providing developers with related development environments that allow programming in different programming languages, including Python.

At present, developers have valuable tools for programming software that uses GPUs in contexts that aren't purely graphics-related. Among the main development environments for heterogeneous computing, we have CUDA and OpenCL.

Let's now have a look at them in detail.

CUDA

CUDA is a proprietary hardware architecture of NVIDIA, which also gives its name to the related development environment. Currently, CUDA has a pool of hundreds of thousands of active developers, which demonstrates the growing interest that is developing around this technology in the parallel programming environment.

CUDA offers extensions for the most commonly used programming languages, including Python. The most well known CUDA Python extensions are as follows:

• PyCUDA (https://mathema.tician.de/software/PyCUDA/)
• Numba (http://numba.pydata.org)

We'll use these extensions in the coming sections.

OpenCL

The second protagonist in parallel computing is OpenCL, which (unlike its NVIDIA counterpart) is open standard and can be used not only with GPUs of different manufacturers but also with microprocessors of different types.

However, OpenCL is a more complete and versatile solution as it does not boast the maturity and simplicity of use that CUDA has.

The OpenCL Python extension is PyOpenCL (https://mathema.tician.de/software/pyopencl/).

In the following sections, the CUDA and OpenCL programming models will be analyzed in their Python extension and will be accompanied by some interesting application examples.

Dealing with PyCUDA

PyCUDA is a binding library that provides access to CUDA's Python API by Andreas Klöckner. The main features include automatic cleanup, which is tied to an object's lifetime, thus preve...

Table of contents

• Title Page
• Copyright and Credits
• Dedication
• About Packt
• Contributors
• Preface
• Getting Started with Parallel Computing and Python
• Thread-Based Parallelism
• Process-Based Parallelism
• Message Passing
• Asynchronous Programming
• Distributed Python
• Cloud Computing
• Heterogeneous Computing
• Python Debugging and Testing
• Other Books You May Enjoy