C++ Template Metaprogramming in Practice

A Deep Learning Framework

Li Wei

• Publisher: Auerbach Publications
• Year: 2020
• ISBN:
  9781000219777
• Edition: 1
• Topic: Computer Science
• Subtopic: Programming in C++
• Index: Computer Science

Using the implementation of a deep learning framework as an example, C++ Template Metaprogramming in Practice: A Deep Learning Framework explains the application of metaprogramming in a relatively large project and emphasizes ways to optimize systems performance. The book is suitable for developers with a basic knowledge of C++. Developers familiar with mainstream deep learning frameworks can also refer to this book to compare the differences between the deep learning framework implemented with metaprogramming and compile-time computing with deep learning frameworks using object-oriented methods.

Consisting of eight chapters, the book starts with two chapters discussing basic techniques of metaprogramming and compile-time computing. The rest of the book's chapters focus on the practical application of metaprogramming in a deep learning framework. It examines rich types and systems, expression templates, and writing complex meta-functions, as well as such topics as:

• 
  
  

• Heterogeneous dictionaries and policy templates

• 
  
  

• An introduction to deep learning

• 
  
  

• Type system and basic data types

• 
  
  

• Operations and expression templates

• 
  
  

• Basic layers

• 
  
  

• Composite and recurrent layers

• 
  
  

• Evaluation and its optimization

Metaprogramming can construct flexible and efficient code. For C++ developers who are familiar with object-oriented programming, the main difficulty in learning and mastering C++ metaprogramming is establishing the thinking mode of functional programming. The meta-programming approach involved at compile time is functional, which means that the intermediate results of the construction cannot be changed, and the impact may be greater than expected. This book enables C++ programmers to develop a functional mindset and metaprogramming skills. The book also discusses the development cost and use cost of metaprogramming and provides workarounds for minimizing these costs.

INTRODUCTIONI

Chapter 1

Basic Tips

This chapter discusses the basic methods involved in metaprogramming and compile-time computing. We'll start with metafunctions and introduce the similarities and differences of functions used at compile time and runtime through some simple examples. We'll also discuss the basic writing rules of sequences, branches, and looping codes. Finally, we'll introduce a classic technique—the Curiously Recurring Template Pattern (CRTP).

The aforementioned content can be regarded as the basic metaprogramming technologies. Subsequent chapters of this book will also be considered as applications of these technologies. Mastering the techniques, as discussed in this chapter, is a prerequisite for skilled use of C++ template metaprogramming and compile-time calculation.

1.1Metafunction and type_traits

1.1.1Introduction to Metafunctions

C++ metaprogramming is typically functional programming in which functions have a pivotal figure in the entire programming system. The functions here are different from those defined and used in general C++ programs and are closer to functions in the mathematical sense—mapping or transformation without side effects: with the same input, the results remain the same no matter how many times the function is called.

If there are side effects in a function, it is usually caused by the presence of certain variables that maintain the state of the system. Each time a function is called, even if the input is the same, the difference in the state of the system will lead the output to be different; such a function is called a function with side effects. Metafunctions are called and executed at compile time. During the compilation phase, the compiler can only construct constants as intermediate results, unable to construct and maintain variables that can record the state of the system and change it hereafter, so functions (which are metafunctions) used at compile time can only be functions without side effects.

The following code snippet defines a function that meets the limitations of no side effects and can be used as a metafunction.

constexpr is a keyword in C++ 11, indicating that the function can be called at compile time and is a metafunction. If this keyword is removed, the function fun will only be used at runtime. Although it has no side effects, it cannot be called at compile time.

Consider the following procedure as a counter-example:

This code snippet cannot be compiled—it is wrong. The reason is that the function loses the “no side effects” property—the same input produces different outputs, while the keyword “constexpr” attempts to maintain the function's “no side effects” property, which leads to conflict. Compiling it will result in a corresponding compilation error. If the keyword constexpr declared in the function is removed, the program can be compiled. But fun2 cannot be called at compile time because it is no longer a metafunction.

Hopefully, the previous example gives a basic impression of metafunctions. In C++, we use the keyword constexpr to represent numeric metafunctions, which is a kind of metafunction involved in C++, but far from all. In fact, type metafunctions are more common in C++, which are metafunctions using types as input and/or output.

1.1.2Type Metafunction

From a mathematical point of view, functions can usually be written in the following form:

y=f(x)

The three symbols represent input (x), output (y), and mapping (f), respectively.¹ Generally speaking, the input and output of a function are numeric values. But we don't have to limit it; in probability theory, for example, there is a function map from event to probability value, where the corresponding input is the description of an event, not necessarily expressed as a numeric value.

Back to the discussion of metaprogramming, the core of metaprogramming is metafunctions. The input and output of metafunctions can also be of various forms: numeric value is just one kind of them. The metafunction in which the input and output are numeric values is the numeric metafunction mentioned in the previous section. Data types in C++ can also be the input and output of functions. Consider the following scenario: we want to map an integer type to the appropriate unsigned type. For example, when type int is the input, the mapping result is unsigned int; when unsigned long is the input, we want the result of the mapping to be the same as the input. This mapping can also be considered as a function, except that the input of the function is a type like int, unsigned long, etc., and the output is another type.

________________________

¹ Functions in C++ can be considered an extension of the above definition, allowing input or output to be empty.

The following code snippet can implement the metafunction as described earlier:

Where is the definition of the function? Programmers who are getting to know metafunctions tend to ask such a question. In fact, lines 1–8 of the code snippet already define a function Fun_, and line 10 uses Fun_<int>::type to return unsigned int, so line 10 is equivalent to defining an unsigned integer variable h and initialized with 3.

Fun_ looks completely different from C++ functions in the general sense, but according to the previous definition of a function, it is not difficult to find that Fun_ has all the properties required for a metafunction:

• The input is the information of a type T and passes to the Fun_ template as a template parameter;
• The output is an internal type of the Fun_ template, which is Fun_<T>:: type;
• The mapping is embodied in the transformation logic implemented by the template through specialization: if the input type is int, the output type is unsigned int, and so on.

Before the release of C++ 11, there were already some works discussing C++ metafunctions. In “C++ Template Metaprogramming,”² the author considers Fun_ declared in lines 1–8 in the preceding segment as a metafunction: when the function input is X, the output is Fun_<X>:: type. At the same time, that book specifies that the input and output of metafunctions discussed should be types. There is nothing wrong with defining a class template that contains a type declaration a metafunction: it fully satisfies the requirement that metafunctions have no side effects. But, the author of this book thinks its definition is too narrow. Of course, introducing restrictions like this would be equivalent to unifying the interface to some extent, which would lead to some convenience for programming. But, the author argues that the convenience comes at the expense of the flexibility of code writing and is too ...