[vbox-dev] Modifying CSAM/PATM

[Sander van Leeuwen]

Martim,

The software virtualization code scanner was not designed to catch all 
occurrences of certain instructions, but
rather as a helper for static scanning upfront and dynamic scanning 
after hitting certain faulting instructions.

You can modify CSAM to inject int 3 patches (0xcc instructions)  for all 
call and ret instructions it encounters.
When you hit such an int3 patch, then you can do whatever it is you wish 
and invoke the scanner for a target address
you haven't encountered before. It shouldn't be very hard to adjust the 
code to do just that and to emulate all ret and
call variants.

The only tricky thing is dealing with self-modifying code. We already 
invalidate patches when unsupported modifications occur, but
you'll have to take this into account for further scanning too.

Sander

On 15-4-2010 23:21, Martim Carbone wrote:
> Hello,
>
> Thank you for your replies to my previous question "VBox CSAM/PATM in VMX mode".
>
> I think I should probably make my intentions clearer. My ultimate goal
> is to be able to intercept, from inside the hypervisor, all
> kernel-level executions of CALL and RET instructions performed by a
> guest OS. I understand the performance implications of this.
> Considering the role that CSAM/PATM already play, it seemed natural to
> me that these could be extended so that kernel CALL and RET
> instructions could be dynamically "patched" (as already done with
> sensitive operations) with code which would trap to the hypervisor
> upon execution. If CSAM/PATM are not used in VMX mode, I can live
> without VMX. The question really is to what extent would CSAM/PATM
> need to be modified to implement this kind of CALL/RET tracing.
>
> I'm not expecting a step-by-step guide, of course; rather an estimate
> from more experienced Vbox developers (as I'm not myself very
> experienced) as to what amount of effort would be involved in this
> addition and any possible pitfalls that I should watch out for when
> modifying the code.
>
> Thanks again,
>
> -- Martim
>
> _______________________________________________
> vbox-dev mailing list
> vbox-dev at virtualbox.org
> http://vbox.innotek.de/mailman/listinfo/vbox-dev
>    


-- 
Kind regards / Mit freundlichen Gruessen / Met vriendelijke groet

-- 

Sun Microsystems GmbH        Sander van Leeuwen
Werkstrasse 24               Senior Staff Engineer, VirtualBox
71384 Weinstadt, Germany     mailto:Sander.Vanleeuwen at sun.com


================================================
Sitz der Gesellschaft: Sun Microsystems GmbH,
Sonnenallee 1, 85551 Kirchheim-Heimstetten
Amtsgericht Muenchen: HRB 161028
Geschaeftsfuehrer: Thomas Schroeder,
Wolfgang Engels, Dr. Roland Boehmer
Vorsitzender des Aufsichtsrates: Martin Haering
================================================