Internals of Python 3.x

Derive Maximum Code Performance and Delve Further into Iterations, Objects, GIL, Memory management, and various Internals

Prashanth Raghu

• Editorial: BPB Publications
• Año: 2021
• ISBN:
  9789391030940
• Categoría: Informatique
• Categoría: Programmation en Python

Deroute the syntactical way and start exploring the language from the source

Key Features
? In-depth practical understanding of CPython's internal workings.
? Step-by-step source code walkthrough utilizing descriptors on source code lines.
? Cutting-edge coverage of the interpreter, GIL, compilation, and memory allocations to help you develop better systems.

Description
Internals of Python 3.x transform a programmer's learning path by emphasizing the source code over the syntax to teach things from the ground up in nearly the same amount of time and effort.The book delves into the structure and distinctions between the primary Python object and iterable objects. The iterable types, namely, lists and tuples, have been thoroughly defined in the structure and operations. The internals of sets and dictionaries, which are data structures that provide O(1) insertion and search, have been thoroughly discussed. Memory allocation explains how Python handles memory for tiny and large objects. The chapter on GIL explains how the GIL works, which is halted by a semaphore and a conditional variable. The chapter on Async Python describes how the async module generates coroutines and async functions that can be executed on an event loop and interact through events.After reading this book, you will be more confident to create high-performance code on a day-to-day basis.

What you will learn
? Utilize data structures effectively for a variety of application functions.
? Discover how to optimize Python code performance.
? Develop an understanding of memory optimization and how to design programs accordingly.
? Investigate the inner working of GIL and Interpreter in detail.
? Recognize the internals of the garbage collection and reference counting processes.

Who this book is for
This book is intended for Python practitioners, new coding aspirants, and experienced Python developers who want to construct their frameworks and libraries by investigating tokenizers, parsers, code compilers, interpreters, memory management, GIL, and garbage collection. Prior programming skills in C may help you get the most out of this book.

Table of Contents
1. Design of Generic Objects
2. Basic Python Types
3. Iterable Sequence Objects
4. Set and Dictionary
5. Functions and Generators
6. Memory Management
7. Interpreter and Opcodes
8. GIL and Multithreading
9. Async Python
10. Source Code Layout and the Compiler Stages

CHAPTER 1

Design of Generic Objects

The components of a generic Python object contain the memory layout, operations, and memory management. This chapter covers the similarities and differences between Python types. The chapter begins with the PyObject and explains how it encapsulates the type, reference count, and bookkeeping pointers. The PyVarObject, which is a PyObject encapsulation with the size of the data structure, is covered along with the different functional attributes of the type such as numerical, sub stringing, and so on.

Structure

In this chapter, we will cover the following topics:

• The PyObject
  • Understanding _PyObject_HEAD_EXTRA
  • Reference counting
• The PyVarObject
• The PyTypeObject
  • Generic type function prototypes
  • Specific type function prototypes
  • The Type object substructures
    • The PyNumberMethods substructure
    • The PySequenceMethods substructure
    • The PyMappingMethods substructure
• The type object
  • Name and sizes of types
  • Allocator, deallocator, and initialization functions
  • Iterator functions
  • Methods and attributes

Objective

After studying this chapter, you will be able to understand the basic components, the object and the type, which compromise both the data types and data structures. You will also learn about the TypeObject that contains the operations implemented in the data types.

The PyObject

The PyObject contains the details of all Python objects and understanding its structure is crucial before we delve into the implementation of reference counting and Python types, which are key elements of this generic object:

Include/object.h Line no 104

typedef struct _object {

_PyObject_HEAD_EXTRA

Py_ssize_t ob_refcnt;

struct _typeobject *ob_type;

} PyObject;

The PyObject contains three elements, which are as follows:

• _PyObject_HEAD_EXTRA: A macro that expands to track all objects in the Python heap in debug mode.
• ob_refcnt: An integer value that holds the reference count of the object.
• _typeobject: A pointer to the type of the object such as integer/list/dictionary, which contains the operations that can be performed on an instance of the type.

The upcoming chapters contain both the implementation and structure of each of these elements.

Understanding _PyObject_HEAD_EXTRA

The _PyObject_HEAD_EXTRA adds a forward and reverse pointer to every object used to construct a doubly linked list that tracks live objects in the Python heap at runtime. This flag has to be enabled when building the Python executable using the following command:

../configure CFLAGS='-DPy_DEBUG -DPy_TRACE_REFS' --with-pydebug

The code block explaining the definition of the _PyObject_HEAD_EXTRA macro is as follows:

Include/object.h Line no 67

#ifdef Py_TRACE_REFS -> 1

/* Define pointers to support a doubly-linked list of all live heap objects. */

#define _PyObject_HEAD_EXTRA \

struct _object *_ob_next; -> 2 \

struct _object *_ob_prev; -> 3

#define _PyObject_EXTRA_INIT 0, 0,

#else

#define _PyObject_HEAD_EXTRA

#define _PyObject_EXTRA_INIT

#endif

Code insights are as follows:

1. The _PyObject_HEAD_EXTRA adds the pointers only when Python is compiled with the Py_TRACE_REFS flag enabled.
2. The _ob_next pointer points to the next created object in the linked list.
3. The _ob_prev pointer points to the previously created object in the linked list.

The following code block explains the construction of the live object heap as a doubly-linked list:

Objects/object.c (Line no 81)

static PyObject refchain = {&refchain, &refchain};

void _Py_AddToAllObjects(PyObject *op, int force) {

….

if (force || op->_ob_prev == NULL) {

op->_ob_next = refchain._ob_next;

op->_ob_prev = &refchain;

refchain._ob_next->_ob_prev = op;

refchain._ob_next = op;

}

…..

}

The highlighted part explains the construction of the doubly-linked list with reference to the current pointer to the end of the list that is pointed to by refchain.

This reference chain can be accessed using the getobjects method of the sys module. The sample Python code demonstrates how the objects can be fetched using the method:

>>> import sys

>>> objs = sys.getobjects(1) # 1 returns the first object from the linked list.

>>> more_objs = sys.getobjects(20) # Increase the count to return more objects.

>>> type_objs = sys.getobjects(20, str) # A second argument

The following code ...