Learning Malware Analysis

Explore the concepts, tools, and techniques to analyze and investigate Windows malware

Monnappa K A

• Publisher: Packt Publishing
• Year: 2018
• ISBN:
  9781788397520
• Edition: 1
• Topic: Informatique
• Subtopic: Cybersécurité

Understand malware analysis and its practical implementationAbout This Book• Explore the key concepts of malware analysis and memory forensics using real-world examples• Learn the art of detecting, analyzing, and investigating malware threats• Understand adversary tactics and techniquesWho This Book Is ForThis book is for incident responders, cyber-security investigators, system administrators, malware analyst, forensic practitioners, student, or curious security professionals interested in learning malware analysis and memory forensics. Knowledge of programming languages such as C and Python is helpful but is not mandatory. If you have written few lines of code and have a basic understanding of programming concepts, you'll be able to get most out of this book.What You Will Learn• Create a safe and isolated lab environment for malware analysis• Extract the metadata associated with malware• Determine malware's interaction with the system• Perform code analysis using IDA Pro and x64dbg• Reverse-engineer various malware functionalities• Reverse engineer and decode common encoding/encryption algorithms• Perform different code injection and hooking techniques• Investigate and hunt malware using memory forensicsIn DetailMalware analysis and memory forensics are powerful analysis and investigation techniques used in reverse engineering, digital forensics, and incident response. With adversaries becoming sophisticated and carrying out advanced malware attacks on critical infrastructures, data centers, and private and public organizations, detecting, responding to, and investigating such intrusions is critical to information security professionals. Malware analysis and memory forensics have become must-have skills to fight advanced malware, targeted attacks, and security breaches.This book teaches you the concepts, techniques, and tools to understand the behavior and characteristics of malware through malware analysis. It also teaches you techniques to investigate and hunt malware using memory forensics. This book introduces you to the basics of malware analysis, and then gradually progresses into the more advanced concepts of code analysis and memory forensics. It uses real-world malware samples, infected memory images, and visual diagrams to help you gain a better understanding of the subject and to equip you with the skills required to analyze, investigate, and respond to malware-related incidents.Style and approachThe book takes the reader through all the concepts, techniques and tools to understand the behavior and characteristics of malware by using malware analysis and it also teaches the techniques to investigate and hunt malware using memory forensics.

Debugging Malicious Binaries

Debugging is a technique in which malicious code is executed in a controlled manner. A debugger is a program that gives you the ability to inspect malicious code at a more granular level. It provides full control over the malware's runtime behavior and allows you to execute a single instruction, multiple instructions, or select functions (instead of executing the entire program), while studying the malware's every action.

In this chapter, you will mainly learn the debugging features offered by IDA Pro (commercial disassembler/debugger) and x64dbg (open source x32/x64 debugger). You will learn about the features offered by these debuggers, and how to use them to inspect the runtime behavior of a program. Depending on the resources available, you will be free to choose either of these debuggers or both, for debugging the malicious binary. When you are debugging a malware, proper care needs to be taken, as you will be running the malicious code on a system. It is highly recommended that you perform any malware debugging in an isolated environment (as covered in Chapter 1, Introduction to Malware Analysis). At the end of this chapter, you will also see how to debug a .NET application using a .NET decompiler/debugger, dnSpy (https://github.com/0xd4d/dnSpy).

Other popular disassemblers/debuggers include radare2 (http://rada.re/r/index.html), the WinDbg part of debugging tools for Windows (https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/), Ollydbg (http://www.ollydbg.de/version2.html), Immunity Debugger (https://www.immunityinc.com/products/debugger/), Hopper (https://www.hopperapp.com/), and Binary Ninja (https://binary.ninja/).

1. General Debugging Concepts

Before we delve into the features offered by these debuggers (IDA Pro, x64dbg, and DnSpy), It is essential to understand some of the common features that most debuggers provide. In this section, you will mainly see the general debugging concepts; in the subsequent sections, we will focus on the essential features of IDA Pro, x64dbg, and dnSpy.

1.1 Launching And Attaching To Processes

Debugging normally begins by selecting the program to debug. There are two ways to debug a program: (a) attach the debugger to a running process, and (b) launch a new process. When you attach the debugger to a running process, you will not be able to control or monitor the process's initial actions, because by the time you have a chance to attach to the process, all of its startup and initialization code will have already been executed. When you attach the debugger to a process, the debugger suspends the process, giving you a chance to inspect the process's resources or set a breakpoint before resuming the process.

On the other hand, launching a new process allows you to monitor or debug every action the process takes, and you will also be able to monitor the process's initial actions. When you start the debugger, the original binary will be executed with the privileges of the user running the debugger. When the process is launched under a debugger, the execution will pause at the program's entry point. A program's entry point is the address of the first instruction that will be executed. In later sections, you will learn how to launch and attach to a process using IDA Pro, x64dbg, and dnSpy.

A program's entry point is not necessarily the main or WinMain function; before transferring control to main or WinMain, the initialization routine (startup routine) is executed. The purpose of the startup routine is to initialize the program's environment before passing control to the main function. This initialization is designated, by the debuggers, as the entry point of the program.

1.2 Controlling Process Execution

A debugger gives you the ability to control/modify the behavior of the process while it is executing. The two important capabilities offered by a debugger are: (a) the ability to control execution, and (b) the ability to interrupt execution (using breakpoints). Using a debugger, you can execute one or more instructions (or select functions) before returning control to the debugger. During analysis, you will combine both the debugger's controlled execution and the interruption (breakpoint) feature to monitor a malware's behavior. In this section, you will learn about the common execution control functionalities offered by the debuggers; in later sections, you will learn how to use these features in IDA Pro, x64dbg, and dnSpy.

The following are some of the common execution control options provided by the debuggers:

• Continue (Run): This executes all of the instructions, until a breakpoint is reached or an exception occurs. When you load a malware into a debugger and use the continue (Run) option without setting the breakpoint, it will execute all of the instructions without giving you any control; so, you normally use this option along with breakpoint, to interrupt the program at the breakpoint location.
• Step into and Step over: Using step into and step over, you can execute a single instruction. After executing the single instruction, the debugger stops, giving you a chance to inspect the process's resources. The difference between step into and step over occurs when you are executing an instruction that calls a function. For example, in the following code, at ➊, there is a call to the function sub_401000. When you use the step into option on this instruction, the debugger will stop at the start of the function (at the address 0x401000), whereas when you use step over, the entire function will be executed, and the debugger will pause at the next instruction, ➋ (that is, the address 0x00401018). You will normally use step into to get inside a function, to understand its inner workings. Step over is used when you already know what a function does (suc...