How to build a hook syscall detector

Windows API calls are often hooked by AV and EDR systems by using inline patching approaches to find strange behaviors or malicious artifacts

Windows API hooking

Windows API hooking is a popular technique used to instrument and modifies the behavior and execution flow of syscalls. For instance, this technique is largely used by AV and EDR systems to determine if a piece of code is suspicious.

More technically, a hook can be compared to a proxy — where all or a specific group of syscalls such as CreateFile(), ReadFile(), OpenProcess(), VirtualAlloc() and so on, could be intercepted and inspected to validate if the intent of the behavior is suspicious or not. [CLICK IMAGES TO ENLARGE]

Figure 1: High-level diagram of a hooked Win API call (source).

In detail, AV and EDR vendors are taking advantage of userland APIs by hijacking the definitions of the functions in Windows DLLs, such as kernel32/kernel base and ntdll (source). The process is quite simple: a jmp instruction is added to the execution flow when a specific and predefined syscall is invoked, such as CreateFileA(). 

The workflow is then changed and redirected to the proxy DLL (EDR.DLL) that will perform validation tasks and return the

Read More: https://resources.infosecinstitute.com/topic/how-to-build-a-hook-syscall-detector/