[vbox-dev] CSAM triggering mechanism

[Martim Carbone]

Hello,

It is not clear to me on what occasions CSAM's main scanning routine
(csamAnalyzeCodeStream) is triggered. From studying the source code,
it seems that csamAnalyzeCodeStream does a recursive disassembly of
the code, starting at the point where it was invoked, and calling the
patch manager whenever necessary to insert breakpoints in sensitive
instructions and recompile code fragments delimited by CLI/STI
PUSHF/POPF. But this recursive disassembly leaves out, for example,
functions that are invoked only through indirect calls and are thus
not reached by the disassembly.

My question is: what mechanism is used to re-invoke CSAM when such a
function (or generally speaking, new, unscanned code) is executed,
since it also has to be analyzed/patched before the guest is allowed
to execute it? My guess is that some sort of page-fault (either
non-present or NX-bit based) exception would used for this, but I was
not able to identify it in the source code.

I would appreciate any feedback.

Thanks,

Martim