Unix System V was one of the first versions of Unix available, and largely defined Unix for years. Under the hood, System V leveraged the System V ABI. As Linux and BSD (Unix-like operating systems) became more widely used, the popularity of System V declined. However, the System V ABI remained popular, as operating systems such as Linux adopted this specification for Intel-based PCs.
In this chapter, we will focus on the System V ABI for Intel platforms on the Linux operating system. It should be noted, however, that other architectures and operating systems might use different ABIs. For example, ARM has its own ABI, which is largely based on System V (and, oddly, the Itanium 64 specification), but has several key differences.
The goal of this section is to expose the inner workings of a single Unix ABI, which in turn should make learning other ABIs easier, if needed.
Most of the specifications discussed in this chapter can be found at the following link: https://refspecs.linuxfoundation.org/.
The System V ABI defines most of the low-level details of a program (which in turn define the interfaces for system programming), including:
The following is a brief description of the remaining details of the System V specification, with respect to the Intel 64-bit architecture.
For the purpose of keeping this topic simple, we will focus on Intel 64-bit. A whole book could be written on the different register layouts for each ABI, operating system, and architecture combination.
The Intel 64-bit architecture (which is usually referred to as AMD64, as AMD actually wrote it) defines several registers, of which a few have defined meanings within the instruction set.
The instruction pointer rip defines a program's current location in executable memory. Specifically, as a program executes, it executes from the location stored in rip, and each time an instruction is retired, rip advances to the next instruction.
The stack pointer and the base pointer (rsp and rbp respectively) are used to define the current location in the stack, as well as the location of the beginning of a stack frame (we will provide more information on this later).
The following are the remaining general-purpose registers. These have different meanings, which will be discussed in the rest of this section: rax, rbx, rcx, rdx, rdi, rsi, r8, r9, r10, r11, r12, r13, r14, and r15.
It should be noted before we continue that there are several other registers defined on the system that have very specific purposes, including floating-point registers and wide registers (which are used by special instructions designed to speed up certain types of calculations; for example, SSE and AVX). These are out of scope for the purpose of this discussion.
Finally, some of the registers end with letters, while others end with numbers, because versions of Intel's x86 processors only had letter-based registers, and the only true, general-purpose registers were AX, BX, CX, and DX.
When 64-bit was introduced by AMD, the number of general-purpose registers doubled, and to keep things simple, the register names were given numbers.
![]()
The stack frame is used to store the return address of each function, and to store function parameters and stack-based variables. It is a resource used heavily by all program, and it takes the following form:
high |----------| <- top of stack
| |
| Used |
| |
|----------| <- Current frame (rbp)
| | <- Stack pointer (rsp)
|----------|
| |
| Unused |
| |
low |----------|
The stack frame is nothing more than an array of memory that grows from top to bottom. That is to say, on an Intel PC, pushing to the stack subtracts from the stack pointer, while popping from the stack adds to the stack pointer—which means that memory actually grows down (assuming your view is that memory grows upward as an address increases, as in the previous diagram).
The System V ABI states that the stack is made up of stack frames. Each frame looks like the following:
high |----------|
| .... |
|----------|
| arg8 |
|----------|
| arg7 |
|----------|
| ret addr |
|----------| <- Stack pointer (rbp)
| |
low |----------|
Each frame represents a function call, and starts with any arguments to a function beyond the first six being called (the first six arguments are passed as registers—this will be discussed in more detail later). Finally, the return address is pushed to the stack, and the function is called.
Memory after the return address belongs to variables that are scoped to the function itself. This is why we call variables defined in a function stack-based variables. The remaining stack is used by functions that will be called in the future. Each time one function calls another, the stack grows, while each time a function returns, the stack shrinks.
It is the job of the operating system to manage the size of the stack, ensuring that it always has enough memory. For example, if an application is trying to use too much memory, the operating system will kill the program.
Finally, it should be noted that on most CPU architectures, special instructions are provided that return from a function call and automatically pop the return address of the stack. In the case of Intel, the call instruction will jump to a function and push the current rip to the stack as the return address, and then ret will pop the return address from the stack and jump the address that was popped.
![]()
Each function comes with a st...
