Section 1: Build System and LLVM-Specific Tooling
You will learn the advanced skills of developing LLVM's build system for both in-tree and out-of-tree scenarios. This section includes the following chapters:
- Chapter 1, Saving Resources When Building LLVM
- Chapter 2, Exploring LLVM's Build System Features
- Chapter 3, Testing with LLVM LIT
- Chapter 4, TableGen Development
Chapter 1: Saving Resources When Building LLVM
LLVM is the state-of-the-art compiler optimization and code generation framework adopted by many amazing industrial and academic projects, such as the Just-In-Time (JIT) compiler in JavaScript engines and machine learning (ML) frameworks. It is a useful toolbox for building programming languages and binary file tools. However, despite the project's robustness, its learning resources are scattered, and it doesn't have the best documentation either. Due to this, it has a pretty steep learning curve, even for developers with some LLVM experience. This book aims to tackle these issues by providing you with knowledge of common and important domains in LLVM in a pragmatic fashion – showing you some useful engineering tips, pointing out lesser-known but handy features, and illustrating useful examples.
As an LLVM developer, building LLVM from source has always been the first thing you should do. Given the scale of LLVM nowadays, this task can take hours to finish. Even worse, rebuilding the project to reflect changes might also take a long time and hinder your productivity. Therefore, it's crucial to know how to use the right tools and how to find the best build configurations for your project for the sake of saving various resources, especially your precious time.
In this chapter, we are going to cover the following topics:
- Cutting down building resources with better tooling
- Saving building resources by tweaking CMake arguments
- Learning how to use GN, an alternative LLVM build system, and its pros and cons
Technical requirements
At the time of writing this book, LLVM only has a few software requirements:
- A C/C++ compiler that supports C++14
- CMake
- One of the build systems supported by CMake, such as GNU Make or Ninja
- Python (2.7 is fine too, but I strongly recommend using 3.x)
- zlib
The exact versions of these items change from time to time. Check out https://llvm.org/docs/GettingStarted.html#software for more details.
This chapter assumes you have built an LLVM before. If that's not the case, perform the following steps:
- Grab a copy of the LLVM source tree from GitHub:
$ git clone https://github.com/llvm/llvm-project
- Usually, the default branch should build without errors. If you want to use release versions that are more stable, such as release version 10.x, use the following command:
$ git clone -b release/10.x https://github.com/llvm/llvm-project
- Finally, you should create a build folder where you're going to invoke the CMake command. All the building artifacts will also be placed inside this folder. This can be done using the following command:
$ mkdir .my_build
$ cd .my_build
Cutting down building resources with better tooling
As we mentioned at the beginning of this chapter, if you build LLVM with the default (CMake) configurations, by invoking CMake and building the project in the following way, there is a high chance that the whole process will take hours to finish:
$ cmake ../llvm
$ make all
This can be avoided by simply using better tools and changing some environments. In this section, we will cover some guidelines to help you choose the right tools and configurations that can both speed up your building time and improve memory footprints.
Replacing GNU Make with Ninja
The first improvement we can do is using the Ninja build tool (https://ninja-build.org) rather than GNU Make, which is the default build system generated by CMake on major Linux/Unix platforms.
Here are the steps you can use to set up Ninja on your system:
- On Ubuntu, for example, you can install Ninja by using this command:
$ sudo apt install ninja-build
Ninja is also available in most Linux distributions.
- Then, when you're invoking CMake for your LLVM build, add an extra argument:
$ cmake -G "Ninja" ../llvm
- Finally, use the following build command instead:
$ ninja all
Ninja runs significantly faster than GNU Make on large code bases such as LLVM. One of the secrets behind Ninja's blazing fast running speed is that while the majority of build scripts such as Makefile are designed to be written manually, the syntax of Ninja's build script, build.ninja, is more similar to assembly code, which should not be edited by developers but generated by other higher-level build systems such as CMake. The fact that Ninja uses an assembly-like build script allows it to do many optimizations under the hood and get rid of many redundancies, such as slower parsing speeds, when invoking the build. Ninja also has a good reputation for generating better dependencies among build targets.
Ninja makes clever decisions in terms of its degree of parallelization; that is, how many jobs you want to execute in parallel. So, usually, you don't need to worry about this. If you want to explicitly assign the number of worker threads, the same command-line option used by GNU Make still works here:
$ ninja -j8 all
Let's now see how you can avoid using the BFD linker.
Avoiding the use of the BFD linker
The second improvement we can do is using linkers other than the BFD linker, which is the default linker used in most Linux systems. The BFD linker, despite being the most mature linker on Unix/Linux systems, is not optimized for speed or memory consumption. This would create a performance bottleneck, especially for large projects such as LLVM. This is because, unlike the compiling phase, it's pretty hard for the linking phase to do file-level parallelization. Not to mention the fact that the BFD linker's...
