[vbox-dev] Strange problem (vbox bug?) with page tables and int 3

[Timo Kreuzer]

Hi,

First sorry for posting on the developers list without being a 
developer, but since vbox-user didn't help as well as irc channels, I 
try it here :)

I'm a ReactOS developer working on the x64 port of ReactOS. I just 
stumbled upon something that seems to be a very strange VBox bug:

Under some circumstances an int 3 will cause the RIP value that is 
pushed on the stack to point *at* the 0xCC instead of pointing *after* it.
I verified this by putting a HLT instruction at the int 3 handler and 
then checked with the vbox debugger (log attached at the end)

I also tried this with int 2c, there the problem doesn't appear.

This behaviour occurred as soon as I was accessing a certain range of 
memory. Higher level page directories were already set up and I could 
set up a PTE without any problem, the bug occured after I accessed the 
mapped page.
I could map any physical page to the PTE without changing the behaviour. 
Also reloading cr3 before and after mapping the PTE didn't change 
anything. The VA range where it happened is the range of one PDE.
Changing the physical page for that PDE could "fix" the problem. Some 
pages worked, while others didn't.
The appearance of the bug is also sensitive to other things:
- putting the int 3 at different locations
- excluding a range of pages from being used by the OS loader (marking 
as firmware)
- double mapping of page tables

The problem also happened directly after mapping the PTE.
As soon as the behaviour starts it will continue this way, ie when 
continuing with a fixed rip the next int 3 will also push an rip on the 
stack that points at the 0xCC

I currently added an ugly hack to the trap dispatcher code that checks 
for this condition and corrects the RIP value in the trap frame.

I would accept that it is a bug in my code, but I would like to 
understand what I did wrong. And I also wonder how it could lead to 
something as strange as a wrong rip pushed on the stack after an int 3.
So any hints what the reason might be are very appreciated.  I can 
provide any additional information requested.

Btw, this is on vbox 4.1.8.r75467 with and without PAE/NX / nested 
paging enabled. Host CPU is an AMD Athlon 64 x2.

WBR,
Timo

Vbox debug log:

    Welcome to the VirtualBox Debugger!
    Current VM is 00920000, CPU #0
    VBoxDbg> stop

    dbgf event: VM 0000000000920000 is halted! (other)
    eax=0000002f ebx=00000000 ecx=ffc00000 edx=00000002 esi=fee0fae5
    edi=00005060
    eip=0052b23a esp=00584058 ebp=00000000 iopl=0 nv up di pl zr na po nc
    cs=0010 ds=002b es=002b fs=0053 gs=002b ss=0018              
    eflags=00000046
    0010:0052b23a 48 c7 c1 45 23 01 00    mov rcx, 00000000000012345h
    VBoxDbg> rg64
    rax=000000000000002f rbx=0000000000000000 rcx=ffffffffffc00000
    rdx=0000000000000002
    rsi=00000000fee0fae5 rdi=0000000000005060 r8 =0000000000000000 r9
    =fffff80000583fa0
    r10=0000000000000000 r11=fffff800005a1080 r12=0000000000000000
    r13=0000000000000000
    r14=0000000000000000 r15=0000000000000000 iopl=0 nv up di pl zr na po nc
    rip=fffff8000052b23a rsp=*fffff80000584058* rbp=0000000000000000
    cs=0010 ds=002b es=002b fs=0053 gs=002b ss=0018                    
    rflags=00000046
    %fffff8000052b23a 48 c7 c1 45 23 01 00    mov rcx, 00000000000012345h
    VBoxDbg> dq *fffff80000584058*
    %fffff80000584058: *fffff8000059aeb4* 0000000000000010
    %fffff80000584068: 0000000000000046 fffff80000584080
    %fffff80000584078: 0000000000000018 fffff80000598000
    %fffff80000584088: fffff800005a1080 fffff800005840f0
    %fffff80000584098: fffff800005840a4 000000b600000080
    %fffff800005840a8: 0000000000001003 fffff6ffffffe000
    VBoxDbg> u *fffff8000059aeb4*
    %*fffff8000059aeb4 cc                      int3*
    %fffff8000059aeb5 48 8b 05 0c 43 00 00    mov rax, qword [00000430ch
    wrt rip]
    %fffff8000059aebc 8b 40 04                mov eax, dword [rax+004h]
    %fffff8000059aebf 2d 00 10 00 00          sub eax, 000001000h
    %fffff8000059aec4 89 44 24 20             mov dword [rsp+020h], eax
    %fffff8000059aec8 8b 44 24 20             mov eax, dword [rsp+020h]
    %fffff8000059aecc 33 d2                   xor edx, edx
    %fffff8000059aece b9 04 00 00 00          mov ecx, 000000004h
    %fffff8000059aed3 48 f7 f1                div rcx
    %fffff8000059aed6 48 8b c8                mov rcx, rax
    VBoxDbg>



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20111222/2243f27f/attachment.html>