Mastering Python High Performance

Fernando Doglio

• Publisher: Packt Publishing
• Year: 2015
• eBook ISBN: 9781783989300
• Edition: 1
• Topic: Ciencia de la computación
• Subtopic: Programación en Python

Measure, optimize, and improve the performance of your Python code with this easy-to-follow guide

About This Book

• Master the do's and don'ts of Python performance programming
• Learn how to use exiting new tools that will help you improve your scripts
• A step-by-step, conceptual guide to teach you how to optimize and fine-tune your critical pieces of code

Who This Book Is For

If you're a Python developer looking to improve the speed of your scripts or simply wanting to take your skills to the next level, then this book is perfect for you.

What You Will Learn

• Master code optimization step-by-step and learn how to use different tools
• Understand what a profiler is and how to read its output
• Interpret visual output from profiling tools and improve the performance of your script
• Use Cython to create fast applications using Python and C
• Take advantage of PyPy to improve performance of Python code
• Optimize number-crunching code with NumPy, Numba, Parakeet, and Pandas

In Detail

Simply knowing how to code is not enough; on mission-critical pieces of code, every bit of memory and every CPU cycle counts, and knowing how to squish every bit of processing power out of your code is a crucial and sought-after skill. Nowadays, Python is used for many scientific projects, and sometimes the calculations done in those projects require some serious fine-tuning. Profilers are tools designed to help you measure the performance of your code and help you during the optimization process, so knowing how to use them and read their output is very handy.

This book starts from the basics and progressively moves on to more advanced topics. You'll learn everything from profiling all the way up to writing a real-life application and applying a full set of tools designed to improve it in different ways. In the middle, you'll stop to learn about the major profilers used in Python and about some graphic tools to help you make sense of their output. You'll then move from generic optimization techniques onto Python-specific ones, going over the main constructs of the language that will help you improve your speed without much of a change. Finally, the book covers some number-crunching-specific libraries and how to use them properly to get the best speed out of them.

After reading this book, you will know how to take any Python code, profile it, find out where the bottlenecks are, and apply different techniques to remove them.

Style and approach

This easy-to-follow, practical guide will help you enhance your optimization skills by improving real-world code.

Mastering Python High Performance

---

Table of Contents

Mastering Python High Performance

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Profiling 101

What is profiling?

Event-based profiling

Statistical profiling

The importance of profiling

What can we profile?

Execution time

Where are the bottlenecks?

Memory consumption and memory leaks

The risk of premature optimization

Running time complexity

Constant time – O(1)

Linear time – O(n)

Logarithmic time – O(log n)

Linearithmic time – O(nlog n)

Factorial time – O(n!)

Quadratic time – O(n^)

Profiling best practices

Build a regression-test suite

Mind your code

Be patient

Gather as much data as you can

Preprocess your data

Visualize your data

Summary

2. The Profilers

Getting to know our new best friends: the profilers

cProfile

A note about limitations

The API provided

The Stats class

Profiling examples

Fibonacci again

Tweet stats

line_profiler

kernprof

Some things to consider about kernprof

Profiling examples

Back to Fibonacci

Inverted index

getOffsetUpToWord

getWords

list2dict

readFileContent

saveIndex

__start__

getOffsetUpToWord

getWords

list2dict

saveIndex

Summary

3. Going Visual – GUIs to Help Understand Profiler Output

KCacheGrind – pyprof2calltree

Installation

Usage

A profiling example – TweetStats

A profiling example – Inverted Index

RunSnakeRun

Installation

Usage

Profiling examples – the lowest common multiplier

A profiling example – search using the inverted index

Summary

4. Optimize Everything

Memoization / lookup tables

Performing a lookup on a list or linked list

Simple lookup on a dictionary

Binary search

Use cases for lookup tables

Usage of default arguments

List comprehension and generators

ctypes

Loading your own custom C library

Loading a system library

String concatenation

Other tips and tricks

Summary

5. Multithreading versus Multiprocessing

Parallelism versus concurrency

Multithreading

Threads

Creating a thread with the thread module

Working with the threading module

Interthread communication with events

Multiprocessing

Multiprocessing with Python

Exit status

Process pooling

Interprocess communication

Pipes

Events

Summary

6. Generic Optimization Options

PyPy

Installing PyPy

A Just-in-time compiler

Sandboxing

Optimizing for the JIT

Think of functions

Consider using cStringIO to concatenate strings

Actions that disable the JIT

Code sample

Cython

Installing Cython

Building a Cython module

Calling C functions

Solving naming conflicts

Defining types

Defining types during function definitions

A Cython example

When to define a type

Limitations

Generator expressions

Comparison of char* literals

Tuples as function arguments

Stack frames

How to choose the right option

When to go with Cython

When to go with PyPy

Summary

7. Lightning Fast Number Crunching with Numba, Parakeet, and pandas

Numba

Installation

Using Numba

Numba's code generation

Eager compilation

Other configuration settings

No GIL

NoPython mode

Running your code on the GPU

The pandas tool

Installing pandas

Using pandas for data analysis

Parakeet

Installing Parakeet

How does Parakeet work?

Summary

8. Putting It All into Practice

The problem to solve

Getting data from the Web

Postprocessing the data

The initial code base

Analyzing the code

Scraper

Analyzer

Summary

Index

Mastering Python High Performance

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: September 2015

Production reference: 1030915

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78398-930-0

www.packtpub.com

Credits

Author

Fernando Doglio

Reviewers

Erik Allik

Mike Driscoll

Enrique Escribano

Mosudi Isiaka

Commissioning Editor

Kunal Parikh

Acquisition Editors

Vivek Anantharaman

Richard Brookes-Bland

Content Development Editors

Akashdeep Kundu

Rashmi Suvarna

Technical Editor

Vijin Boricha

Copy Editors

Relin Hedly

Karuna Narayanan

Project Coordinator

Milton Dsouza

Proofreader

Safis Editing

Indexer

Mariammal Chettiyar

Graphics

Sheetal Aute

Production Coordinator

Arvindkumar Gupta

Cover Work

Arvindkumar Gupta

About the Author

Fernando Doglio has been working as a web developer for the past 10 years.

During that time, he shifted his focus to the Web and grabbed the opportunity of working with most of the leading technologies, such as PHP, Ruby on Rails, MySQL, Python, Node.js, AngularJS, AJAX, REST APIs, and so on.

In his spare time, Fernando likes to tinker and learn new things. This is why his GitHub account keeps getting new repos every month. He's also a big open source supporter and tries to win the support of new people with the help of his website, lookingforpullrequests.com.

You can reach him on Twitter at @deleteman123.

When he is not programming, he spends time with his family.

I'd like to thank my lovely wife for putting up with me and the long hours I spent writing this book; this book would not have been possible without her continued support. I would also like to tha...

Table of contents

• Mastering Python High Performance