Learn Python Programming

A beginner's guide to learning the fundamentals of Python language to write efficient, high-quality code, 2nd Edition

Fabrizio Romano

• Publisher: Packt Publishing
• Year: 2018
• ISBN:
  9781788991650
• Topic: Computer Science
• Subtopic: Programming in Python
• Index: Computer Science

Learn the fundamentals of Python (3.7) and how to apply it to data science, programming, and web development. Fully updated to include hands-on tutorials and projects.

Key Features

• Learn the fundamentals of Python programming with interactive projects
• Apply Python to data science with tools such as IPython and Jupyter
• Utilize Python for web development and build a real-world app using Django

Book Description

Learn Python Programming is a quick, thorough, and practical introduction to Python - an extremely flexible and powerful programming language that can be applied to many disciplines.Unlike other books, it doesn't bore you with elaborate explanations of the basics but gets you up-and-running, using the language. You will begin by learning the fundamentals of Python so that you have a rock-solid foundation to build upon.You will explore the foundations of Python programming and learn how Python can be manipulated to achieve results. Explore different programming paradigms and find the best approach to a situation; understand how to carry out performance optimization and effective debugging; control the flow of a program; and utilize an interchange format to exchange data. You'll also walk through cryptographic services in Python and understand secure tokens.Learn Python Programming will give you a thorough understanding of the Python language. You'll learn how to write programs, build websites, and work with data by harnessing Python's renowned data science libraries. Filled with real-world examples and projects, the book covers various types of applications, and concludes by building real-world projects based on the concepts you have learned.

What you will learn

• Get Python up and running on Windows, Mac, and Linux
• Explore fundamental concepts of coding using data structures and control flow
• Write elegant, reusable, and efficient code in any situation
• Understand when to use the functional or OOP approach
• Cover the basics of security and concurrent/asynchronous programming
• Create bulletproof, reliable software by writing tests
• Build a simple website in Django
• Fetch, clean, and manipulate data

Who this book is for

Learn Python Programming is for individuals with relatively little experience in coding or Python. It's also ideal for aspiring programmers who need to write scripts or programs to accomplish tasks. The book shows you how to create a full-fledged application.

]]>

Concurrent Execution

"What do we want? Now! When do we want it? Fewer race conditions!"

– Anna Melzer

In this chapter, I'm going to up the game a little bit, both in terms of the concepts I'll present, and in the complexity of the code snippets I'll show you. If you don't feel up to the task, or as you are reading through you realize it is getting too difficult, feel free to skip it. You can always come back to it when you feel ready.

The plan is to take a detour from the familiar single-threaded execution paradigm, and deep dive into what can be described as concurrent execution. I will only be able to scratch the surface of this complex topic, so I won't expect you to be a master of concurrency by the time you're done reading, but I will, as usual, try to give you enough information so that you can then proceed by walking the path, so to speak.

We will learn about all the important concepts that apply to this area of programming, and I will try to show you examples coded in different styles, to give you a solid understanding of the basics of these topics. To dig deep into this challenging and interesting branch of programming, you will have to refer to the Concurrent Execution section in the Python documentation (https://docs.python.org/3.7/library/concurrency.html), and maybe supplement your knowledge by studying books on the subject.

In particular, we are going to explore the following:

• The theory behind threads and processes
• Writing multithreaded code
• Writing multiprocessing code
• Using executors to spawn threads and processes
• A brief example of programming with asyncio

Let's start by getting the theory out of the way.

Concurrency versus parallelism

Concurrency and parallelism are often mistaken for the same thing, but there is a distinction between them. Concurrency is the ability to run multiple things at the same time, not necessarily in parallel. Parallelism is the ability to do a number of things at the same time.

Imagine you take your other half to the theater. There are two lines: that is, for VIP and regular tickets. There is only one functionary checking tickets and so, in order to avoid blocking either of the two queues, they check one ticket from the VIP line, then one from the regular line. Over time, both queues are processed. This is an example of concurrency.

Now imagine that another functionary joins, so now we have one functionary per queue. This way, both queues will be processed each by its own functionary. This is an example of parallelism.

Modern laptop processors feature multiple cores (normally two to four). A core is an independent processing unit that belongs to a processor. Having more than one core means that the CPU in question has the physical ability to actually execute tasks in parallel. Within each core, normally there is a constant alternation of streams of work, which is concurrent execution.

Bear in mind that I'm keeping the discussion generic on purpose here. According to which system you are using, there will be differences in how execution is handled, so I will concentrate on the concepts that are common to all, or at least most, systems.

Threads and processes – an overview

A thread can be defined as a sequence of instructions that can be run by a scheduler, which is that part of the operating system that decides which chunk of work will receive the necessary resources to be carried out. Typically, a thread lives within a process. A process can be defined as an instance of a computer program that is being executed.

In previous chapters, we have run our own modules and scripts with commands similar to $ python my_script.py. What happens when a command like that is run, is that a Python process is created. Within it, a main thread of execution is spawned. The instructions in the script are what will be run within that thread.

This is just one way of working though, and Python can actually use more than one thread within the same process, and can even spawn multiple processes. Unsurprisingly, these branches of computer science are called multithreading and multiprocessing.

In order to understand the difference, let's take a moment to explore threads and processes in slightly more depth.

Quick anatomy of a thread

Generally speaking, there are two different types of threads:

• User-level threads: Threads that we can create and manage in order to perform a task
• Kernel-level threads: Low-level threads that run in kernel mode and act on behalf of the operating system

Given that Python works at the user level, we're not going to deep dive into kernel threads at this time. Instead, we will explore several examples of user-level threads in this chapter's examples.

A thread can be in any of the following states:

• New thread: A thread that hasn't started yet, and hasn't been allocated any resources.
• Runnable: The thread is waiting to run. It has all the resources needed to run, and as soon as the scheduler gives it the green light, it will be run.
• Running: A thread whose stream of instructions is being executed. From this state, it can go back to a non-running state, or die.
• Not-running: A thread that has been paused. This could be due to another thread taking precedence over it, or simply because the thread is waiting for a long-running IO operation to finish.
• Dead: A thread that has died because it has reached the natural end of its stream of execution, or it has been killed.

Transitions between states are provoked either by our actions or by the scheduler. There is one thing to bear in mind, though; it is best not to interfere with the death of a thread.

Killing threads

Killing threads is not considered to be good practice. Python doesn't provide the ability to kill a thread by calling a method or function, and this should be a hint that killing threads isn't something you want to be doing.

One reason is that a thread might have children—threads spawned from within the thread itself—which would be orphaned when their parent dies. Another reason could be that if the thread you're killing is holding a resource that needs to be closed properly, you might prevent that from happening and that could potentially lead to problems.

Later, we will see an example of how we can work around these issues.

Context-switching

We have said that the scheduler can decide when a thread can run, or is paused, and so on. Any time a running thread needs to be suspended so that another can be run, the scheduler saves the state of the running thread in a way that it will be possible, at a later time, to resume execution exactly where it was paused.

This act is called context-switching. People do that all the time too. We are doing some paperwork, and we hear bing! on our phone. We stop the paperwork and check our phone. When we're done dealing with what was probably the umpteenth picture of a funny cat, we go back to our paperwork. We don't start the paperwork from the beginning, though; we simply continue where we had left off.

Context-switching is a marvelous ability of modern computers, but it can become troublesome if you generate too many threads. The scheduler then will try to give each of them a chance to run for a little time, and there will be a lot of time spent saving and recovering the state of the threads that are respectively paused and restarted.

In order to avoid this problem, it is quite common to limit the amount of threads (the same consideration applies to processes) that can be run at any given point in time. This is achieved by using a structure called a pool, the size of which can be decided by the programmer. In a nutshell, we create a pool and then assign tasks to its threads. When all the threads of the pool are busy, the program won't be able to spawn a new thread until one of them terminates (and goes back to the pool). Pools are also great for saving resources, in that they provide recycling features to the thread ecosystem.

When you write multithreaded code, it is useful to have information about the machine our software is going to run on. That information, coupled with some profiling (we'll learn about it in Chapter 11, Debugging and Troubleshooting), should enable us to calibrate the size of our pools correctly.

The Global Interpreter Lock

In July 2015, I attended the EuroPython conference in Bilbao, where I gave a talk about test-driven development. The camera operator unfortunately lost the first half of it, but I've since been able to give that talk another couple of times, so you can find a complete version of it on the web. At the conference, I had the great pleasure of meeting Guido van Rossum and talking to him, and I also attended his keynote speech.

One of the topics he addressed was the infamous Global Interpreter Lock (GIL). The GIL is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once. This means that even though you can write multithreaded code in Python, there is only one thread running at any point in time (per process, of course).

In computer programming, a mutual exclusion object (mutex) is a program object that allows multiple program threads to share the same resource, such as file access, but not simultaneously.

This is normally seen as an undesired limitation of the language, and many developers take pride in cursing this great villain. The truth lies somewhere else though, as was beautifully explained by Raymond Hettinger in his Keynote on Concurrency, at PyBay 2017 (https://bit.ly/2KcijOB). About 10 minutes in, Raymond explains that it is actually quite simple to remove the GIL from Python. It takes about a day of work. The price you pay for this GIL-ectomy though, is that you then ...