[vbox-dev] CSAM triggering mechanism

[Sander van Leeuwen]

On 22-4-2010 19:49, Martim Carbone wrote:
> Hello,
>
> It is not clear to me on what occasions CSAM's main scanning routine
> (csamAnalyzeCodeStream) is triggered. From studying the source code,
> it seems that csamAnalyzeCodeStream does a recursive disassembly of
> the code, starting at the point where it was invoked, and calling the
> patch manager whenever necessary to insert breakpoints in sensitive
> instructions and recompile code fragments delimited by CLI/STI
> PUSHF/POPF. But this recursive disassembly leaves out, for example,
> functions that are invoked only through indirect calls and are thus
> not reached by the disassembly.
>    
It's called whenever we are about to execute code and we haven't scanned 
that instruction before.
We also invoke the code scanner when getting faults on pages that aren't 
in our shadow page tables yet
(when cause by an instruction fetch)

> My question is: what mechanism is used to re-invoke CSAM when such a
> function (or generally speaking, new, unscanned code) is executed,
> since it also has to be analyzed/patched before the guest is allowed
> to execute it? My guess is that some sort of page-fault (either
> non-present or NX-bit based) exception would used for this, but I was
> not able to identify it in the source code.
>    
CSAM's goal isn't to scan the entire guest kernel; just to find problem 
areas. If you want to extend that
goal you will have to insert breakpoints for call/jump instructions for 
which you can't statically calculate
the target address. When you hit the breakpoint, you can check if 
further code scanning is required.
Of course this strategy comes with a performance penalty.

-- 
Kind regards / Mit freundlichen Gruessen / Met vriendelijke groet

-- 

Sun Microsystems GmbH        Sander van Leeuwen
Werkstrasse 24               Senior Staff Engineer, VirtualBox
71384 Weinstadt, Germany     mailto:Sander.Vanleeuwen at sun.com


================================================
Sitz der Gesellschaft: Sun Microsystems GmbH,
Sonnenallee 1, 85551 Kirchheim-Heimstetten
Amtsgericht Muenchen: HRB 161028
Geschaeftsfuehrer: Thomas Schroeder,
Wolfgang Engels, Dr. Roland Boehmer
Vorsitzender des Aufsichtsrates: Martin Haering
================================================