Advanced Python Programming

Quan Nguyen

• Publisher: Packt Publishing
• Year: 2022
• eBook ISBN: 9781801817776
• Edition: 2
• Topic: Computer Science
• Subtopic: Client-Server Computing
• Index: Computer Science

Write fast, robust, and highly reusable applications using Python's internal optimization, state-of-the-art performance-benchmarking tools, and cutting-edge librariesKey Features• Benchmark, profile, and accelerate Python programs using optimization tools• Scale applications to multiple processors with concurrent programming• Make applications robust and reusable using effective design patternsBook DescriptionPython's powerful capabilities for implementing robust and efficient programs make it one of the most sought-after programming languages.In this book, you'll explore the tools that allow you to improve performance and take your Python programs to the next level.This book starts by examining the built-in as well as external libraries that streamline tasks in the development cycle, such as benchmarking, profiling, and optimizing. You'll then get to grips with using specialized tools such as dedicated libraries and compilers to increase your performance at number-crunching tasks, including training machine learning models.The book covers concurrency, a major solution to making programs more efficient and scalable, and various concurrent programming techniques such as multithreading, multiprocessing, and asynchronous programming.You'll also understand the common problems that cause undesirable behavior in concurrent programs.Finally, you'll work with a wide range of design patterns, including creational, structural, and behavioral patterns that enable you to tackle complex design and architecture challenges, making your programs more robust and maintainable.By the end of the book, you'll be exposed to a wide range of advanced functionalities in Python and be equipped with the practical knowledge needed to apply them to your use cases.What you will learn• Write efficient numerical code with NumPy, pandas, and Xarray• Use Cython and Numba to achieve native performance• Find bottlenecks in your Python code using profilers• Optimize your machine learning models with JAX• Implement multithreaded, multiprocessing, and asynchronous programs• Solve common problems in concurrent programming, such as deadlocks• Tackle architecture challenges with design patternsWho this book is forThis book is for intermediate to experienced Python programmers who are looking to scale up their applications in a systematic and robust manner. Programmers from a range of backgrounds will find this book useful, including software engineers, scientific programmers, and software architects.

Section 1: Python-Native and Specialized Optimization

Python, along with its many libraries and packages, offers specialized data structures and classes that facilitate highly optimized operations. You will learn how to navigate through these different tools and use them appropriately to accelerate your Python program.

This section contains the following chapters:

• Chapter 1, Benchmarking and Profiling
• Chapter 2, Pure Python Optimizations
• Chapter 3, Fast Array Operations with NumPy, Pandas, and Xarray
• Chapter 4, C Performance with Cython
• Chapter 5, Exploring Compilers
• Chapter 6, Automatic Differentiation and Accelerated Linear Algebra for Machine Learning

Chapter 1: Benchmarking and Profiling

Recognizing the slow parts of your program is the single most important task when it comes to speeding up your code. In most cases, the code that causes the application to slow down is a very small fraction of the program. By identifying these critical sections, you can focus on the parts that need the most improvement without wasting time in micro-optimization.

Profiling is a technique that allows us to pinpoint the most resource-intensive parts of an application. A profiler is a program that runs an application and monitors how long each function takes to execute, thus detecting the functions on which your application spends most of its time.

Python provides several tools to help us find these bottlenecks and measure important performance metrics. In this chapter, we will learn how to use the standard cProfile module and the line_profiler third-party package. We will also learn how to profile the memory consumption of an application through the memory_profiler tool. Another useful tool that we will cover is KCachegrind, which can be used to graphically display the data produced by various profilers.

Finally, benchmarks are small scripts used to assess the total execution time of your application. We will learn how to write benchmarks and use them to accurately time your programs.

The topics we will cover in this chapter are listed here:

• Designing your application
• Writing tests and benchmarks
• Writing better tests and benchmarks with pytest-benchmark
• Finding bottlenecks with cProfile
• Optimizing our code
• Using the dis module
• Profiling memory usage with memory_profiler

By the end of the chapter, you will have gained a solid understanding of how to optimize a Python program and will be armed with practical tools that facilitate the optimization process.

Technical requirements

To follow the content of this chapter, you should have a basic understanding of Python programming and be familiar with core concepts such as variables, classes, and functions. You should also be comfortable with working with the command line to run Python programs. Finally, the code for this chapter can be found in the following GitHub repository: https://github.com/PacktPublishing/Advanced-Python-Programming-Second-Edition/tree/main/Chapter01.

Designing your application

In the early development stages, the design of a program can change quickly and may require large rewrites and reorganizations of the code base. By testing different prototypes without the burden of optimization, you are free to devote your time and energy to ensure that the program produces correct results and that the design is flexible. After all, who needs an application that runs fast but gives the wrong answer?

The mantras that you should remember when optimizing your code are outlined here:

• Make it run: We have to get the software in a working state and ensure that it produces the correct results. This exploratory phase serves to better understand the application and to spot major design issues in the early stages.
• Make it right: We want to ensure that the design of the program is solid. Refactoring should be done before attempting any performance optimization. This really helps separate the application into independent and cohesive units that are easy to maintain.
• Make it fast: Once our program is working and well structured, we can focus on performance optimization. We may also want to optimize memory usage if that constitutes an issue.

In this section, we will write and profile a particle simulator test application. A simulator is a program that considers some particles and simulates their mov...

Table of contents

• Advanced Python Programming
• Second Edition
• Preface
• Section 1: Python-Native and Specialized Optimization
• Chapter 1: Benchmarking and Profiling
• Chapter 2: Pure Python Optimizations
• Chapter 3: Fast Array Operations with NumPy, Pandas, and Xarray
• Chapter 4: C Performance with Cython
• Chapter 5: Exploring Compilers
• Chapter 6: Automatic Differentiation and Accelerated Linear Algebra for Machine Learning
• Section 2: Concurrency and Parallelism
• Chapter 7: Implementing Concurrency
• Chapter 8: Parallel Processing
• Chapter 9: Concurrent Web Requests
• Chapter 10: Concurrent Image Processing
• Chapter 11: Building Communication Channels with asyncio
• Chapter 12: Deadlocks
• Chapter 13: Starvation
• Chapter 14: Race Conditions
• Chapter 15: The Global Interpreter Lock
• Section 3: Design Patterns in Python
• Chapter 16: The Factory Pattern
• Chapter 17: The Builder Pattern
• Chapter 18: Other Creational Patterns
• Chapter 19: The Adapter Pattern
• Chapter 20: The Decorator Pattern
• Chapter 21: The Bridge Pattern
• Chapter 22: The Façade Pattern
• Chapter 23: Other Structural Patterns
• Chapter 24: The Chain of Responsibility Pattern
• Chapter 25: The Command Pattern
• Chapter 26: The Observer Pattern
• Assessments
• Other Books You May Enjoy