Windows API calls are often hooked by AV and EDR systems by using inline patching approaches to find strange behaviors or malicious artifacts.
Windows API hooking
Windows API hooking is a popular technique used to instrument and modifies the behavior and execution flow of syscalls. For instance, this technique is largely used by AV and EDR systems to determine if a piece of code is suspicious.
More technically, a hook can be compared to a proxy — where all or a specific group of syscalls such as CreateFile(), ReadFile(), OpenProcess(), VirtualAlloc() and so on, could be intercepted and inspected to validate if the intent of the behavior is suspicious or not. [CLICK IMAGES TO ENLARGE]
Figure 1: High-level diagram of a hooked Win API call (source).
In detail, AV and EDR vendors are taking advantage of userland APIs by hijacking the definitions of the functions in Windows DLLs, such as kernel32/kernel base and ntdll (source). The process is quite simple: a jmp instruction is added to the execution flow when a specific and predefined syscall is invoked, such as CreateFileA().
The workflow is then changed and redirected to the proxy DLL (EDR.DLL) that will perform validation tasks and return the