Learn Web Development with Python

Get hands-on with Python Programming and Django web development

Fabrizio Romano, Gaston C. Hillar, Arun Ravindran

• Publisher: Packt Publishing
• Year: 2018
• ISBN:
  9781789950885
• Edition: 1
• Topic: Computer Science
• Subtopic: Programming in Python
• Index: Computer Science

A comprehensive guide to Python programming for web development using the most popular Python web framework - Django

Key Features

• Learn the fundamentals of programming with Python and building web apps
• Build web applications from scratch with Django
• Create real-world RESTful web services with the latest Django framework

Book Description

If you want to develop complete Python web apps with Django, this Learning Path is for you. It will walk you through Python programming techniques and guide you in implementing them when creating 4 professional Django projects, teaching you how to solve common problems and develop RESTful web services with Django and Python. You will learn how to build a blog application, a social image bookmarking website, an online shop, and an e-learning platform.

Learn Web Development with Python will get you started with Python programming techniques, show you how to enhance your applications with AJAX, create RESTful APIs, and set up a production environment for your Django projects. Last but not least, you'll learn the best practices for creating real-world applications.

By the end of this Learning Path, you will have a full understanding of how Django works and how to use it to build web applications from scratch.

This Learning Path includes content from the following Packt products:

• Learn Python Programming by Fabrizio Romano
• Django RESTful Web Services by Gastón C. Hillar
• Django Design Patterns and Best Practices by Arun Ravindran

What you will learn

• Explore the fundamentals of Python programming with interactive projects
• Grasp essential coding concepts along with the basics of data structures and control flow
• Develop RESTful APIs from scratch with Django and the Django REST Framework
• Create automated tests for RESTful web services
• Debug, test, and profile RESTful web services with Django and the Django REST Framework
• Use Django with other technologies such as Redis and Celery

Who this book is for

If you have little experience in coding or Python and want to learn how to build full-fledged web apps, this Learning Path is for you. No prior experience with RESTful web services, Python, or Django is required, but basic Python programming experience is needed to understand the concepts covered.

Concurrent Execution

"What do we want? Now! When do we want it? Fewer race conditions!"

– Anna Melzer

In this chapter, I'm going to up the game a little bit, both in terms of the concepts I'll present, and in the complexity of the code snippets I'll show you. If you don't feel up to the task, or as you are reading through you realize it is getting too difficult, feel free to skip it. You can always come back to it when you feel ready.

The plan is to take a detour from the familiar single-threaded execution paradigm, and deep dive into what can be described as concurrent execution. I will only be able to scratch the surface of this complex topic, so I won't expect you to be a master of concurrency by the time you're done reading, but I will, as usual, try to give you enough information so that you can then proceed by walking the path, so to speak.

We will learn about all the important concepts that apply to this area of programming, and I will try to show you examples coded in different styles, to give you a solid understanding of the basics of these topics. To dig deep into this challenging and interesting branch of programming, you will have to refer to the Concurrent Execution section in the Python documentation (https://docs.python.org/3.7/library/concurrency.html), and maybe supplement your knowledge by studying books on the subject.

In particular, we are going to explore the following:

• The theory behind threads and processes
• Writing multithreaded code
• Writing multiprocessing code
• Using executors to spawn threads and processes
• A brief example of programming with asyncio

Let's start by getting the theory out of the way.

Concurrency versus parallelism

Concurrency and parallelism are often mistaken for the same thing, but there is a distinction between them. Concurrency is the ability to run multiple things at the same time, not necessarily in parallel. Parallelism is the ability to do a number of things at the same time.

Imagine you take your other half to the theater. There are two lines: that is, for VIP and regular tickets. There is only one functionary checking tickets and so, in order to avoid blocking either of the two queues, they check one ticket from the VIP line, then one from the regular line. Over time, both queues are processed. This is an example of concurrency.

Now imagine that another functionary joins, so now we have one functionary per queue. This way, both queues will be processed each by its own functionary. This is an example of parallelism.

Modern laptop processors feature multiple cores (normally two to four). A core is an independent processing unit that belongs to a processor. Having more than one core means that the CPU in question has the physical ability to actually execute tasks in parallel. Within each core, normally there is a constant alternation of streams of work, which is concurrent execution.

Bear in mind that I'm keeping the discussion generic on purpose here. According to which system you are using, there will be differences in how execution is handled, so I will concentrate on the concepts that are common to all, or at least most, systems.

Threads and processes – an overview

A thread can be defined as a sequence of instructions that can be run by a scheduler, which is that part of the operating system that decides which chunk of work will receive the necessary resources to be carried out. Typically, a thread lives within a process. A process can be defined as an instance of a computer program that is being executed.

In previous chapters, we have run our own modules and scripts with commands similar to $ python my_script.py. What happens when a command like that is run, is that a Python process is created. Within it, a main thread of execution is spawned. The instructions in the script are what will be run within that thread.

This is just one way of working though, and Python can actually use more than one thread within the same process, and can even spawn multiple processes. Unsurprisingly, these branches of computer science are called multithreading and multiprocessing.

In order to understand the difference, let's take a moment to explore threads and processes in slightly more depth.

Quick anatomy of a thread

Generally speaking, there are two different types of threads:

• User-level threads: Threads that we can create and manage in order to perform a task
• Kernel-level threads: Low-level threads that run in kernel mode and act on behalf of the operating system

Given that Python works at the user level, we're not going to deep dive into kernel threads at this time. Instead, we will explore several examples of user-level threads in this chapter's examples.

A thread can be in any of the following states:

• New thread: A thread that hasn't started yet, and hasn't been allocated any resources.
• Runnable: The thread is waiting to run. It has all the resources needed to run, and as soon as the scheduler gives it the green light, it will be run.
• Running: A thread whose stream of instructions is being executed. From this state, it can go back to a non-running state, or die.
• Not-running: A thread that has been paused. This could be due to another thread taking precedence over it, or simply because the thread is waiting for a long-running IO operation to finish.
• Dead: A thread that has died because it has reached the natural end of its stream of execution, or it has been killed.

Transitions between states are provoked either by our actions or by the scheduler. There is one thing to bear in mind, though; it is best not to interfere with the death of a thread.

Killing threads

Killing threads is not considered to be good practice. Python doesn't provide the ability to kill a thread by calling a method or function, and this should be a hint that killing threads isn't something you want to be doing.

One reason is that a thread might have children—threads spawned from within the thread itself—which would be orphaned when their parent dies. Another reason could be that if the thread you're killing is holding a resource that needs to be closed properly, you might prevent that from happening and that could potentially lead to problems.

Later, we will see an example of how we can work around these issues.

Context-switching

We have said that the scheduler can decide when a thread can run, or is paused, and so on. Any time a running thread needs to be suspended so that another can be run, the scheduler saves the state of the running thread in a way that it will be possible, at a later time, to resume execution exactly where it was paused.

This act is called context-switching. People do that all the time too. We are doing some paperwork, and we hear bing! on our phone. We stop the paperwork and check our phone. When we're done dealing with what was probably the umpteenth picture of a funny cat, we go back to our paperwork. We don't start the paperwork from the beginning, though; we simply continue where we had left off.

Context-switching is a marvelous ability of modern computers, but it can become troublesome if you generate too many threads. The scheduler then will try to give each of them a chance to run for a little time, and there will be a lot of time spent saving and recovering the state of the threads that are respectively paused and restarted.

In order to avoid this problem, it is quite common to limit the amount of threads (the same consideration applies to processes) that can be run at any given point in time. This is achieved by using a structure called a pool, the size of which can be decided by the programmer. In a nutshell, we create a pool and then assign tasks to its threads. When all the threads of the pool are busy, the program won't be able to spawn a new thread until one of them terminates (and goes back to the pool). Pools are also great for saving resources, in that they provide recycling features to the thread ecosystem.

When you write multithreaded code, it is useful to have information about the machine our software is going to run on. That information, coupled with some profiling (we'll learn about it in Chapter 10, Debugging and Troubleshooting), should enable us to calibrate the size of our pools correctly.

The Global Interpreter Lock

In July 2015, I attended the EuroPython conference in Bilbao, where I gave a talk about test-driven development. The camera operator unfortunately lost the first half of it, but I've since been able to give that talk another couple of times, so you can find a complete version of it on the web. At the conference, I had the great pleasure of meeting Guido van Rossum and talking to him, and I also attended his keynote speech.

One of the topics he addressed was the infamous Global Interpreter Lock (GIL). The GIL is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once. This means that even though you can write multithreaded code in Python, there is only one thread running at any point in time (per process, of course).

In computer programming, a mutual exclusion object (mutex) is a program object that allows multiple program threads to share the same resource, such as file access, but not simultaneously.

This is normally seen as an undesired limitation of the language, and many developers take pride in cursing this great villain. The truth lies somewhere else though, as was beautifully explained by Raymond Hettinger in his Keynote on Concurrency, at PyBay 2017 (https://bit.ly/2KcijOB). About 10 minutes in, Raymond explains that it is actually quite sim...