Practical Reverse Engineering

x86, x64, ARM, Windows Kernel, Reversing Tools, and Obfuscation

Bruce Dang, Alexandre Gazet, Elias Bachaalany

• Publisher: Wiley
• Year: 2014
• Print ISBN: 9781118787311
• eBook ISBN: 9781118787397
• Edition: 1
• Topic: Computer Science
• Subtopic: Cyber Security
• Index: Computer Science

Analyzing how hacks are done, so as to stop them in the future

Reverse engineering is the process of analyzing hardware or software and understanding it, without having access to the source code or design documents. Hackers are able to reverse engineer systems and exploit what they find with scary results. Now the good guys can use the same tools to thwart these threats. Practical Reverse Engineering goes under the hood of reverse engineering for security analysts, security engineers, and system programmers, so they can learn how to use these same processes to stop hackers in their tracks.

The book covers x86, x64, and ARM (the first book to cover all three); Windows kernel-mode code rootkits and drivers; virtual machine protection techniques; and much more. Best of all, it offers a systematic approach to the material, with plenty of hands-on exercises and real-world examples.

• Offers a systematic approach to understanding reverse engineering, with hands-on exercises and real-world examples
• Covers x86, x64, and advanced RISC machine (ARM) architectures as well as deobfuscation and virtual machine protection techniques
• Provides special coverage of Windows kernel-mode code (rootkits/drivers), a topic not often covered elsewhere, and explains how to analyze drivers step by step
• Demystifies topics that have a steep learning curve
• Includes a bonus chapter on reverse engineering tools

Practical Reverse Engineering: Using x86, x64, ARM, Windows Kernel, and Reversing Tools provides crucial, up-to-date guidance for a broad range of IT professionals.

Chapter 1

x86 and x64

The x86 is little-endian architecture based on the Intel 8086 processor. For the purpose of our chapter, x86 is the 32-bit implementation of the Intel architecture (IA-32) as defined in the Intel Software Development Manual. Generally speaking, it can operate in two modes: real and protected. Real mode is the processor state when it is first powered on and only supports a 16-bit instruction set. Protected mode is the processor state supporting virtual memory, paging, and other features; it is the state in which modern operating systems execute. The 64-bit extension of the architecture is called x64 or x86-64. This chapter discusses the x86 architecture operating in protected mode.

x86 supports the concept of privilege separation through an abstraction called ring level. The processor supports four ring levels, numbered from 0 to 3. (Rings 1 and 2 are not commonly used so they are not discussed here.) Ring 0 is the highest privilege level and can modify all system settings. Ring 3 is the lowest privileged level and can only read/modify a subset of system settings. Hence, modern operating systems typically implement user/kernel privilege separation by having user-mode applications run in ring 3, and the kernel in ring 0. The ring level is encoded in the CS register and sometimes referred to as the current privilege level (CPL) in official documentation.

This chapter discusses the x86/IA-32 architecture as defined in the Intel 64 and IA-32 Architectures Software Developer's Manual, Volumes 1–3 (www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html).

Register Set and Data Types

When operating in protected mode, the x86 architecture has eight 32-bit general-purpose registers (GPRs): EAX, EBX, ECX, EDX, EDI, ESI, EBP, and ESP. Some of them can be further divided into 8- and 16-bit registers. The instruction pointer is stored in the EIP register. The register set is illustrated in Figure 1.1. Table 1.1 describes some of these GPRs and how they are used.

Figure 1.1

Table 1.1 Some GPRs and Their Usage

Register	Purpose
ECX	Counter in loops
ESI	Source in string/memory operations
EDI	Destination in string/memory operations
EBP	Base frame pointer
ESP	Stack pointer

The common data types are as follows:

• Bytes—8 bits. Examples: AL, BL, CL
• Word—16 bits. Examples: AX, BX, CX
• Double word—32 bits. Examples: EAX, EBX, ECX
• Quad word—64 bits. While x86 does not have 64-bit GPRs, it can combine two registers, usually EDX:EAX, and treat them as 64-bit values in some scenarios. For example, the RDTSC instruction writes a 64-bit value to EDX:EAX.

The 32-bit EFLAGS register is used to store the status of arithmetic operations and other execution states (e.g., trap flag). For example, if the previous “add” operation resulted in a zero, the ZF flag will be set to 1. The flags in EFLAGS are primarily used to implement conditional branching.

In addition to the GPRs, EIP, and EFLAGS, there are also registers that control important low-level system mechanisms such as virtual memory, interrupts, and debugging. For example, CR0 controls whether paging is on or off, CR2 contains the linear address that caused a page fault, CR3 is the base address of a paging data structure, and CR4 controls the hardware virtualization settings. DR0–DR7 are used to set memory breakpoints. We will come back to these registers later in the “System Mechanism” section.

---

Note

Although there are eight debug registers, the system allows only four memory breakpoints (DR0–DR3). The remaining registers are used for status.

---

There are also model-specific registers (MSRs). As the name implies, these registers may vary between different processors by Intel and AMD. Each MSR is identified by name and a 32-bit number, and read/written to through the RDMSR/WRMSR instructions. They are accessible only to code running in ring 0 and typically used to store special counters and implement low-level functionality. For example, the SYSENTER instruction transfers execution to the address stored in the IA32_SYSENTER_EIP MSR (0x176), which is usually the operating system's system call handler. MSRs are discussed throughout the book as they come up.

Instruction Set

The x86 instruction set allows a high level of flexibility in terms of data movement between registers and memory. The movement can be classified into five general methods:

• Immediate to register
• Register to register
• Immediate to memory
• Register to memory and vice versa
• Memory to memory

The first four methods are supported by all modern architectures, but the last one is specific to x86. A classical RISC architecture like ARM can only read/write data from/to memory with load/store instructions (LDR and STR, respectively); for example, a simple operation like incrementing a value in memory requires three instructions:

1. Read the data from memory to a register (LDR).

2. Add one to the register (ADD).

3. Write the register to memory (STR).

On x86, such an operation would require only one instruction (either INC or ADD) because it can directly access memory. The MOVS instruction can read and write memory at the same time.

ARM

01: 1A 68 LDR R2, [R3] ; read the value at address R3 and save it in R2 02: 52 1C ADDS R2, R2, #1 ; add 1 to it 03: 1A 60 STR R2, [R3] ; write updated value back to address R3

x86

01: FF 00 inc dword ptr [eax] ; directly increment value at address EAX

Another important characteristic is that x86 uses variable-length instruction size: the instruction length can range from 1 to 15 bytes. On ARM, instructions are either 2 or 4 bytes in length.

Syntax

Depending on the assembler/disassembler, there are two syntax notations for x86 assembly code, Intel and AT&T:

Intel

mov ecx, AABBCCDDh mov ecx, [eax] mov ecx, eax

AT&T

movl $0xAABBCCDD, %ecx movl (%eax), %ecx movl %eax, %ecx

It is important to note that these are the same instructions but written differently. There are several differences between Intel and AT&T notation, but the most notable ones are as follows:

• AT&T prefixes the register with %, and immediates with $. Intel does not do this.
• AT&T adds a suffix to the instruction to indicate operation width. For example, MOVL (long), MOVB (byte), etc. Intel does not do this.
• AT&T puts the source operand before the destination. Intel reverses the order.

Disassemblers/assemblers and other reverse-engineering tools (IDA Pro, OllyDbg, MASM, etc.) on Windows typically use Intel notation, whereas those on UNIX frequently follow AT&T notation (GCC). In practice, Intel notation is the dominant form and is used throughout this book.

Data Movement

Instructions operate on values that come from registers or main memory. The most common instruction for moving data is MOV. The simplest usage is to move a register or immediate to register. For example:

01: BE 3F 00 0F 00 mov esi, 0F003Fh ; set ESI = 0xF003 02: 8B F1 mov esi, ecx ; set ESI = ECX

The next common usage is t...

Table of contents

• Cover
• Chapter 1: x86 and x64
• Chapter 2: ARM
• Chapter 3: The Windows Kernel
• Chapter 4: Debugging and Automation
• Chapter 5: Obfuscation
• Appendix: Sample Names and Corresponding SHA1 Hashes
• Introduction
• End User License Agreement