Category Archives: call/ret Monitor

Posts relating the call/ret monitor reverse engineering experiment

call/ret, Without The ret

I resurrected my old call/ret experiments recently. I wish I had published more about how I did what I did 2 years ago — that’s why I started this blog, as a research journal, for my own benefit, for when I eventually return to a project after a lengthy hiatus. Fortunately, I emailed pretty thorough details to a fellow reverse engineer during the original project and still have the old email in the archives.

To refresh, the call/ret monitor watches a piece of code run and logs when function call or return instructions are executed, and where. After the fact, and combined with executable code that comes with meaningful symbol names, it generates a call graph to demonstrate where code really flows and gets around the problem of resolving indirect jumps in static disassembly listings. The original experiment assumes that for every call, there is an equal and opposite return (formally stated: num_calls == num_rets). The target I’m after right now behaves… differently. I am seeing way more calls than returns.

Obviously, there is a lot about compilers that I don’t understand. I think what I’m looking at in this situation is an artifact of C++ or perhaps its linkage from C. When I examine functions in static disassembly, there will be 7 NOP instructions after the stack frame initialization. Seems pretty innocuous. However, when I analyze the code more carefully while it’s running, I see that something — I suspect the dynamic loader — is using those 7 NOP bytes for something else. I imagine this is some common convention, and I would love to know more about it someday. But for the time being, I just want to get past this problem. In this particular situation, all that the patched code does is perform a call 5 bytes farther in the address space where a pop instruction is waiting to clear the return address that just got pushed on the stack by the call. So that’s where the massive call surplus comes from. Why the unknown entity couldn’t patch in an absolute jump is unclear.

Eventually, it dawned on me that the /ret half of the call/ret experiment was entirely unnecessary given the other data I already have. I have retooled the experiment with that in mind. It’s annoying to try to remember enough Perl to make this fly but it’s worth it for the resulting data visualization.

Pretty Call Graphs Based On The call/ret Monitor

Thanks for Benjamin Larsson for advising me on how to make a proper graph using Graphviz (specify a graph of type ‘prof’ vs. ‘unix’). Thus, here are some nice (and big) graphs based on the call/ret experiment at this point:

See this post (“Refining The call/ret Monitor”) for details on how the graph data is generated.

And as a bonus, I put the toolset to work analyzing Linspire’s WMV3/WMV9 decoding module. Here are some call graphs (I used the teaser trailer from Halo 2, hence the “halo2” in the filenames):

Refining The call/ret Monitor

Reverse engineering is, of course, a tedious, time-consuming, and error-prone task. It requires a lot of concentration that I either do not have or do not care to invest in the RE task. That is why so many of my RE experiments are geared toward automating the task as much as possible. To that end, I am optimistic about this call/ret monitoring experiment since it yields such a good high-level overview of an algorithm contained in a binary (with debug symbols). But it can use some improvement(s):

Continue reading

Foiling The call/ret Monitor

In the grand tradition of arms races, I like to give equal time to counter-reverse engineering techniques. Colin Hill suggested an interesting way to cause trouble for my call/ret monitoring solution. Make functions that start with a long sequence of NOPs. Instead of calling the actual start address, load the starting address into a register and add a random number to the base address that still points into the NOP range. This would pollute the address space range in the output.

For die-hard, old school reverse engineers, this would also have the effect of creating paranoia of self-modifying code.