LLVM Techniques, Tips, and Best Practices Clang and Middle-End Libraries

Design powerful and reliable compilers using the latest libraries and tools from LLVM

Min-Yih Hsu

• Publisher: Packt Publishing
• Year: 2021
• ISBN:
  9781838829728
• Edition: 1
• Topic: Ciencia de la computación
• Subtopic: Lenguajes de programación

Learn how you can build the next big programming language, compiler, or source code analyzer using LLVM and Clang

Key Features

• Explore Clang, LLVM's middle-end and backend, in a pragmatic way
• Develop your LLVM skillset and get to grips with a variety of common use cases
• Engage with real-world LLVM development through various coding examples

Book Description

Every programmer or engineer, at some point in their career, works with compilers to optimize their applications. Compilers convert a high-level programming language into low-level machine-executable code. LLVM provides the infrastructure, reusable libraries, and tools needed for developers to build their own compilers. With LLVM's extensive set of tooling, you can effectively generate code for different backends as well as optimize them.

In this book, you'll explore the LLVM compiler infrastructure and understand how to use it to solve different problems. You'll start by looking at the structure and design philosophy of important components of LLVM and gradually move on to using Clang libraries to build tools that help you analyze high-level source code. As you advance, the book will show you how to process LLVM IR – a powerful way to transform and optimize the source program for various purposes. Equipped with this knowledge, you'll be able to leverage LLVM and Clang to create a wide range of useful programming language tools, including compilers, interpreters, IDEs, and source code analyzers.

By the end of this LLVM book, you'll have developed the skills to create powerful tools using the LLVM framework to overcome different real-world challenges.

What you will learn

• Find out how LLVM's build system works and how to reduce the building resource
• Get to grips with running custom testing with LLVM's LIT framework
• Build different types of plugins and extensions for Clang
• Customize Clang's toolchain and compiler flags
• Write LLVM passes for the new PassManager
• Discover how to inspect and modify LLVM IR
• Understand how to use LLVM's profile-guided optimizations (PGO) framework
• Create custom compiler sanitizers

Who this book is for

This book is for software engineers of all experience levels who work with LLVM. If you are an academic researcher, this book will help you learn useful LLVM skills in a short time and enable you to build your prototypes and projects quickly. Programming language enthusiasts will also find this book useful for building a new programming language with the help of LLVM.

Section 1: Build System and LLVM-Specific Tooling

You will learn the advanced skills of developing LLVM's build system for both in-tree and out-of-tree scenarios. This section includes the following chapters:

• Chapter 1, Saving Resources When Building LLVM
• Chapter 2, Exploring LLVM's Build System Features
• Chapter 3, Testing with LLVM LIT
• Chapter 4, TableGen Development

Chapter 1: Saving Resources When Building LLVM

LLVM is the state-of-the-art compiler optimization and code generation framework adopted by many amazing industrial and academic projects, such as the Just-In-Time (JIT) compiler in JavaScript engines and machine learning (ML) frameworks. It is a useful toolbox for building programming languages and binary file tools. However, despite the project's robustness, its learning resources are scattered, and it doesn't have the best documentation either. Due to this, it has a pretty steep learning curve, even for developers with some LLVM experience. This book aims to tackle these issues by providing you with knowledge of common and important domains in LLVM in a pragmatic fashion – showing you some useful engineering tips, pointing out lesser-known but handy features, and illustrating useful examples.

As an LLVM developer, building LLVM from source has always been the first thing you should do. Given the scale of LLVM nowadays, this task can take hours to finish. Even worse, rebuilding the project to reflect changes might also take a long time and hinder your productivity. Therefore, it's crucial to know how to use the right tools and how to find the best build configurations for your project for the sake of saving various resources, especially your precious time.

In this chapter, we are going to cover the following topics:

• Cutting down building resources with better tooling
• Saving building resources by tweaking CMake arguments
• Learning how to use GN, an alternative LLVM build system, and its pros and cons

Technical requirements

At the time of writing this book, LLVM only has a few software requirements:

• A C/C++ compiler that supports C++14
• CMake
• One of the build systems supported by CMake, such as GNU Make or Ninja
• Python (2.7 is fine too, but I strongly recommend using 3.x)
• zlib

The exact versions of these items change from time to time. Check out https://llvm.org/docs/GettingStarted.html#software for more details.

This chapter assumes you have built an LLVM before. If that's not the case, perform the following steps:

1. Grab a copy of the LLVM source tree from GitHub:
   $ git clone https://github.com/llvm/llvm-project
2. Usually, the default branch should build without errors. If you want to use release versions that are more stable, such as release version 10.x, use the following command:
   $ git clone -b release/10.x https://github.com/llvm/llvm-project
3. Finally, you should create a build folder where you're going to invoke the CMake command. All the building artifacts will also be placed inside this folder. This can be done using the following command:
   $ mkdir .my_build
   $ cd .my_build

Cutting down building resources with better tooling

As we mentioned at the beginning of this chapter, if you build LLVM with the default (CMake) configurations, by invoking CMake and building the project in the following way, there is a high chance that the whole process will take hours to finish:

$ cmake ../llvm

$ make all

This can be avoided by simply using better tools and changing some environments. In this section, we will cover some guidelines to help you choose the right tools and configurations that can both speed up your building time and improve memory footprints.

Replacing GNU Make with Ninja

The first improvement we can do is using the Ninja build tool (https://ninja-build.org) rather than GNU Make, which is the default build system generated by CMake on major Linux/Unix platforms.

Here are the steps you can use to set up Ninja on your system:

1. On Ubuntu, for example, you can install Ninja by using this command:
   $ sudo apt install ninja-build
   Ninja is also available in most Linux distributions.
2. Then, when you're invoking CMake for your LLVM build, add an extra argument:
   $ cmake -G "Ninja" ../llvm
3. Finally, use the following build command instead:
   $ ninja all

Ninja runs significantly faster than GNU Make on large code bases such as LLVM. One of the secrets behind Ninja's blazing fast running speed is that while the majority of build scripts such as Makefile are designed to be written manually, the syntax of Ninja's build script, build.ninja, is more similar to assembly code, which should not be edited by developers but generated by other higher-level build systems such as CMake. The fact that Ninja uses an assembly-like build script allows it to do many optimizations under the hood and get rid of many redundancies, such as slower parsing speeds, when invoking the build. Ninja also has a good reputation for generating better dependencies among build targets.

Ninja makes clever decisions in terms of its degree of parallelization; that is, how many jobs you want to execute in parallel. So, usually, you don't need to worry about this. If you want to explicitly assign the number of worker threads, the same command-line option used by GNU Make still works here:

$ ninja -j8 all

Let's now see how you can avoid using the BFD linker.

Avoiding the use of the BFD linker

The second improvement we can do is using linkers other than the BFD linker, which is the default linker used in most Linux systems. The BFD linker, despite being the most mature linker on Unix/Linux systems, is not optimized for speed or memory consumption. This would create a performance bottleneck, especially for large projects such as LLVM. This is because, unlike the compiling phase, it's pretty hard for the linking phase to do file-level parallelization. Not to mention the fact that the BFD linker's...