x86 API Hooking Demystified (2012)

I used this technique once to very useful effect in an automated dependency-checking tool.<p>Essentially given an idempotent data-transformation process, you can inject a dll into it and hook its filesystem calls, maintaining a list of all files it touches. Upon process termination, serialize out the filesystem state (size, timestamp, MD5) of each of those files, saving the data in a dependency file whose filename is the hash of the command line.<p>When you run the same command line again, the dll can first look up the dependency file from the previous run, and if all files on the filesystem are in the same state as previously, you can short-circuit the process execution entirely.<p>This is orders of magnitude faster in many cases for highly-parallel data build jobs where only some small percentage of the source data changes each time you run the build job, and has the advantage that you don&#x27;t need manually maintain a list of dependencies for each process type (no new dependency can be added without changing one of the existing dependencies).

I wrote a hooking library that implements the methods talked about, plus a bunch more it doesn&#x27;t. As well as handle a bunch of edge cases not mentioned: <a href="https:&#x2F;&#x2F;github.com&#x2F;stevemk14ebr&#x2F;PolyHook_2_0" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;stevemk14ebr&#x2F;PolyHook_2_0</a>

We used hooking to &quot;isolate&quot; the audio output from a certain application on Windows:<p>The application would ask the operating system for the default audio device. By intercepting this request, we were able to re-route it to our own, virtual audio device. Our program would then fetch the audio data from the virtual device, and replay it to the &quot;real&quot; audio device. At the same time, the audio gets saved to ram, and finally to disk.<p>The benefit of this method was that we were able to actually isolate the audio from all other sources on the computer. So you could, in theory, mute the playback, while still being able to let the recording run. Ultimately, we abandoned this method, as it proved quite unreliable. But it was fun to come up with, and finally implement.

Oh hey! This is a favorite topic of mine. I wrote myself a hooking library for a project where I didn’t want to use libc (and it was Win32 by design so I could just use the equivalent Win32 calls). The hardest part was definitely the LDE (length disassembly engine) and in the end I found a small header-only open source library that did it perfectly. The rest was very easy, especially on x86.<p>API hooking sometimes even works in hostile environments, like on software that tries to guard against patches and modification, simply because it can be challenging to detect, and you can do it early in (like inside a DLL entrypoint). So if you can do all of your work at API boundaries, you can get away with a whole lot, even if an app is packed with a strong VM packer.<p>Worth noting that for many less difficult use cases on Linux, you can use LD_PRELOAD to somewhat similar effect.

On Linux (and probably other Unices), another way to hook is via the interesting default behaviour of the dynamic linker to only use symbol names instead of qualifying them with the library&#x27;s filename from which they are to be imported and prefer symbols in already-loaded libraries; I suspect that more often than not, this happens by accident instead of deliberation.

Renderdoc uses hooking to instrument your 3D API calls to provide excellent graphics debugging. Good example to dive into since it&#x27;s also cross platform.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;baldurk&#x2F;renderdoc" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;baldurk&#x2F;renderdoc</a><p>I used graphics API hooking in a Source Engine game where I didn&#x27;t have the source for the engine but needed to display the 2D Flash based GUI (Iggy) at the correct time. It was fun to get it working.<p>Valve&#x27;s Steam also does something similar since it has the ability to superimpose it&#x27;s GUI over a running game.

API hooking used to be super easy and very common under DOS. All the API calls used to work using traps, so you&#x27;d just have to 1) store the original interrupt vector 2) change it to point at your own interrupt service routine, 3) do whatever when the trap was invoked.<p>Point 3 might include logging, passing the request through to the original ISR (possibly with changed parameter values), changing the return values, anything really. Easy &amp; fun. =)

I&#x27;ve used MS Detours to hook my own C++ interceptor functions into dispatch from Excel to XLL extensions via the internal Excel4V interface. It worked very well, showing me the XLOPER values. But that was for a 32 bit Excel. I didn&#x27;t appreciate how much trickier it is with the amd64 instruction set.

If I remember correctly the old Mac operating systems used this a lot to ship patches or to change system behavior.