Win7 VM's crash when a debugger is connected via virtualized COM port pipe

[matthew.robben@gmail.com]

I've got a 64 bit win7 box running latest version of VirtualBox. I have a Win7 x64 VM that I'd like to use for development of kernel drivers. The VM has a virtualized COM port that creates a host pipe called VMCOM. Whenever I connect a kernel debugger (in this case WINDBG) to the VM over that pipe, after breaking and continuing execution a few times I see the VM crash in HALHandleNMI after a secondaryclockinterrupt occurs. The bugcheck details given via !analyze are also given.

###DEBUGGER SESSION COPIED BELOW Opened \.\pipe\vmcom Waiting to reconnect... Connected to Windows 7 7600 x64 target at (Wed Dec 28 10:26:18.444 2011 (UTC - 5:00)), ptr64 TRUE Kernel Debugger connection established. Symbol search path is: * Invalid *

• Symbol loading may be unreliable without a symbol search path. *
• Use .symfix to have the debugger choose a symbol path. *
• After setting your symbol path, use .reload to refresh symbol locations. *

Executable search path is: *

• Symbols can not be loaded because symbol path is not initialized. *
• *
• The Symbol Path can be set by: *
• using the _NT_SYMBOL_PATH environment variable. *
• using the -y <symbol_path> argument when starting the debugger. *
• using .sympath and .sympath+ *

* * ERROR: Symbol file could not be found. Defaulted to export symbols for ntkrnlmp.exe - Windows 7 Kernel Version 7600 MP (4 procs) Free x64 Product: WinNt, suite: TerminalServer SingleUserTS Built by: 7600.16385.amd64fre.win7_rtm.090713-1255 Machine Name: Kernel base = 0xfffff8000284a000 PsLoadedModuleList = 0xfffff80002a87e50 Debug session time: Wed Dec 28 10:26:12.029 2011 (UTC - 5:00) System Uptime: 0 days 5:12:49.358 Break instruction exception - code 80000003 (first chance) *

• *
• You are seeing this message because you pressed either *
• CTRL+C (if you run console kernel debugger) or, *
• CTRL+BREAK (if you run GUI kernel debugger), *
• on your debugger machine's keyboard. *
• *
• THIS IS NOT A BUG OR A SYSTEM CRASH *
• *
• If you did not intend to break into the debugger, press the "g" key, then *
• press the "Enter" key now. This message might immediately reappear. If it *
• does, press "g" and "Enter" again. *
• *

* ntDbgBreakPointWithStatus: fffff800`028b3f60 cc int 3 1: kd> .symfix 1: kd> .reload Connected to Windows 7 7600 x64 target at (Wed Dec 28 10:33:42.784 2011 (UTC - 5:00)), ptr64 TRUE Loading Kernel Symbols . -- User interrupt 1: kd> g Break instruction exception - code 80000003 (first chance) *

• *
• You are seeing this message because you pressed either *
• CTRL+C (if you run console kernel debugger) or, *
• CTRL+BREAK (if you run GUI kernel debugger), *
• on your debugger machine's keyboard. *
• *
• THIS IS NOT A BUG OR A SYSTEM CRASH *
• *
• If you did not intend to break into the debugger, press the "g" key, then *
• press the "Enter" key now. This message might immediately reappear. If it *
• does, press "g" and "Enter" again. *
• *

* ntRtlpBreakWithStatusInstruction: fffff800`028b3f60 cc int 3 3: kd> !stack No export stack found 3: kd> !ps No export ps found 3: kd> !eip No export eip found 3: kd> !thread THREAD fffff88002f1dfc0 Cid 0000.0000 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 3 Not impersonating DeviceMap fffff8a0000060c0 Owning Process fffff80002a43140 Image: Idle Attached Process fffffa8000c9e040 Image: System Wait Start TickCount 0 Ticks: 1203174 (0:05:12:49.634) Context Switch Count 1255366 IdealProcessor: 3 UserTime 00:00:00.000 KernelTime 05:11:32.149 Win32 Start Address ntKiIdleLoop (0xfffff800028c46a0) Stack Init fffff88002f3bdb0 Current fffff88002f3bd40 Base fffff88002f3c000 Limit fffff88002f36000 Call 0 Priority 16 BasePriority 0 UnusualBoost 0 ForegroundBoost 0 IoPriority 0 PagePriority 0 Child-SP RetAddr : Args to Child : Call Site fffff88002f3bac8 fffff80002882d73 : 0000000000000000 fffff88002f13180 0000000000000000 0000000000026161 : ntRtlpBreakWithStatusInstruction fffff88002f3bad0 fffff800028c8ba1 : 0000000000000000 fffff88002f3bb80 fffff88002f13180 0000000000000001 : nt! ?? ::FNODOBFM::`string'+0x5dd4 fffff88002f3bb00 fffff8800450f9c2 : fffff800028c9a3a 00000000ffffffed fffffa800203b2b8 fffff88002f1dfc0 : ntKiSecondaryClockInterrupt+0x131 (TrapFrame @ fffff880`02f3bb00) fffff88002f3bc98 fffff800028c9a3a : 00000000ffffffed fffffa800203b2b8 fffff88002f1dfc0 0000000000000001 : 0xfffff880`0450f9c2 fffff88002f3bca0 fffff800028c46cc : fffff88002f13180 fffff88000000000 0000000000000000 fffff80002950cf0 : ntPoIdle+0x53a fffff88002f3bd80 0000000000000000 : fffff88002f3c000 0000000000000000 0000000000000000 0000000000000000 : ntKiIdleLoop+0x2c

3: kd> g Break instruction exception - code 80000003 (first chance) *

• *
• You are seeing this message because you pressed either *
• CTRL+C (if you run console kernel debugger) or, *
• CTRL+BREAK (if you run GUI kernel debugger), *
• on your debugger machine's keyboard. *
• *
• THIS IS NOT A BUG OR A SYSTEM CRASH *
• *
• If you did not intend to break into the debugger, press the "g" key, then *
• press the "Enter" key now. This message might immediately reappear. If it *
• does, press "g" and "Enter" again. *
• *

* ntRtlpBreakWithStatusInstruction: fffff800`028b3f60 cc int 3 2: kd> g Break instruction exception - code 80000003 (first chance) ntRtlpBreakWithStatusInstruction: fffff800`028b3f60 cc int 3 3: kd> !thread THREAD fffff88002f1dfc0 Cid 0000.0000 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 3 Not impersonating DeviceMap fffff8a0000060c0 Owning Process fffff80002a43140 Image: Idle Attached Process fffffa8000c9e040 Image: System Wait Start TickCount 0 Ticks: 1204506 (0:05:13:10.414) Context Switch Count 1260552 IdealProcessor: 3 UserTime 00:00:00.000 KernelTime 05:11:52.335 Win32 Start Address ntKiIdleLoop (0xfffff800028c46a0) Stack Init fffff88002f3bdb0 Current fffff88002f3bd40 Base fffff88002f3c000 Limit fffff88002f36000 Call 0 Priority 16 BasePriority 0 UnusualBoost 0 ForegroundBoost 0 IoPriority 0 PagePriority 0 Child-SP RetAddr : Args to Child : Call Site fffff88002f1db58 fffff800029b16d2 : fffff80000000010 fffffa8001be2a20 0000000000000000 fffff800029b17e2 : ntRtlpBreakWithStatusInstruction fffff88002f1db60 fffff800028138da : fffff80000000005 0000002800000025 000000000000027f fffff800028292b0 : ntKiBugCheckDebugBreak+0x12 fffff88002f1dbc0 fffff800029d0513 : 0000000000000001 fffff800028292b0 0000000000000000 000000000000005c : halHalBugCheckSystem+0x1ba fffff88002f1dc00 fffff8000280d6c1 : fffffa80000006c0 fffff88002f1de20 fffff88002f1dcf0 fffff800028292b0 : ntWheaReportHwError+0x263 fffff88002f1dc60 fffff80002974311 : fffff88002f1de30 0000000000000001 0000000000000001 fffffa800203b200 : hal!HalHandleNMI+0x149 fffff88002f1dc90 fffff800028b9202 : 0000000000000000 0000000000000000 0000000000000000 0000000000000003 : nt!KiProcessNMI+0x131 fffff88002f1dcf0 fffff800028b9063 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntKxNmiInterrupt+0x82 fffff88002f1de30 fffff800028c8acf : 0000000000000000 fffff88002f3bb80 fffff88002f13180 0000000000000001 : ntKiNmiInterrupt+0x163 (TrapFrame @ fffff880`02f1de30) fffff88002f3bb00 fffff8800450f9c2 : fffff800028c9a3a 00000000ffffffed fffffa800203b2b8 fffff88002f1dfc0 : ntKiSecondaryClockInterrupt+0x5f (TrapFrame @ fffff880`02f3bb00) fffff88002f3bc98 fffff800028c9a3a : 00000000ffffffed fffffa800203b2b8 fffff88002f1dfc0 0000000000000001 : 0xfffff880`0450f9c2 fffff88002f3bca0 fffff800028c46cc : fffff88002f13180 fffff88000000000 0000000000000000 fffff80002950cf0 : ntPoIdle+0x53a fffff88002f3bd80 0000000000000000 : fffff88002f3c000 0000000000000000 0000000000000000 0000000000000000 : ntKiIdleLoop+0x2c

3: kd> !analyze -v TRIAGER: Could not open triage file : C:\Program Files\Windows Kits\8.0\Debuggers\x64\triage\oca.ini, error 2 TRIAGER: Could not open triage file : C:\Program Files\Windows Kits\8.0\Debuggers\x64\winxp\triage.ini, error 2 TRIAGER: Could not open triage file : C:\Program Files\Windows Kits\8.0\Debuggers\x64\triage\user.ini, error 2 Connected to Windows 7 7600 x64 target at (Wed Dec 28 10:49:24.239 2011 (UTC - 5:00)), ptr64 TRUE Loading Kernel Symbols ............................................................... ................................................................ ............. Loading User Symbols

Loading unloaded module list ................ *

• *
• Bugcheck Analysis *
• *

*

Unknown bugcheck code (111) Unknown bugcheck description Arguments: Arg1: 0000000000000000 Arg2: 0000000000000000 Arg3: 0000000000000000 Arg4: 0000000000000000

Debugging Details:

---

* ERROR: Module load completed but symbols could not be loaded for intelppm.sys TRIAGER: Could not open triage file : C:\Program Files\Windows Kits\8.0\Debuggers\x64\triage\modclass.ini, error 2

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0x111

PROCESS_NAME: System

CURRENT_IRQL: f

LAST_CONTROL_TRANSFER: from fffff800029b16d2 to fffff800028b3f60

STACK_TEXT: fffff88002f1db58 fffff800029b16d2 : fffff80000000010 fffffa8001be2a20 0000000000000000 fffff800029b17e2 : ntRtlpBreakWithStatusInstruction fffff88002f1db60 fffff800028138da : fffff80000000005 0000002800000025 000000000000027f fffff800028292b0 : ntKiBugCheckDebugBreak+0x12 fffff88002f1dbc0 fffff800029d0513 : 0000000000000001 fffff800028292b0 0000000000000000 000000000000005c : halHalBugCheckSystem+0x1ba fffff88002f1dc00 fffff8000280d6c1 : fffffa80000006c0 fffff88002f1de20 fffff88002f1dcf0 fffff800028292b0 : ntWheaReportHwError+0x263 fffff88002f1dc60 fffff80002974311 : fffff88002f1de30 0000000000000001 0000000000000001 fffffa800203b200 : hal!HalHandleNMI+0x149 fffff88002f1dc90 fffff800028b9202 : 0000000000000000 0000000000000000 0000000000000000 0000000000000003 : nt!KiProcessNMI+0x131 fffff88002f1dcf0 fffff800028b9063 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntKxNmiInterrupt+0x82 fffff88002f1de30 fffff800028c8acf : 0000000000000000 fffff88002f3bb80 fffff88002f13180 0000000000000001 : ntKiNmiInterrupt+0x163 fffff88002f3bb00 fffff8800450f9c2 : fffff800028c9a3a 00000000ffffffed fffffa800203b2b8 fffff88002f1dfc0 : ntKiSecondaryClockInterrupt+0x5f fffff88002f3bc98 fffff800028c9a3a : 00000000ffffffed fffffa800203b2b8 fffff88002f1dfc0 0000000000000001 : intelppm+0x39c2 fffff88002f3bca0 fffff800028c46cc : fffff88002f13180 fffff88000000000 0000000000000000 fffff80002950cf0 : ntPoIdle+0x53a fffff88002f3bd80 0000000000000000 : fffff88002f3c000 0000000000000000 0000000000000000 0000000000000000 : ntKiIdleLoop+0x2c

STACK_COMMAND: kb

FOLLOWUP_IP: intelppm+39c2 fffff880`0450f9c2 c3 ret

SYMBOL_STACK_INDEX: 9

SYMBOL_NAME: intelppm+39c2

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: intelppm

IMAGE_NAME: intelppm.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bc0fd

FAILURE_BUCKET_ID: X64_0x111_intelppm+39c2

BUCKET_ID: X64_0x111_intelppm+39c2

Followup: MachineOwner

---

[matthew.robben@gmail.com]

vbox log

[matthew.robben@gmail.com]

pretty printed text of the debugger

[matthew.robben@gmail.com]

Another crash occurred with a different stack:

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 111, {0, 0, 0, 0}

TRIAGER: Could not open triage file : C:\Program Files\Windows Kits\8.0\Debuggers\x64\triage\modclass.ini, error 2
Probably caused by : ntkrnlmp.exe ( nt!KiNmiInterruptEnd+15 )

Followup: MachineOwner
---------

nt!RtlpBreakWithStatusInstruction:
fffff800`028b3f60 cc              int     3
3: kd> !thread
THREAD fffff88002f1dfc0  Cid 0000.0000  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 3
Not impersonating
DeviceMap                 fffff8a0000060c0
Owning Process            fffff80002a43140       Image:         Idle
Attached Process          fffffa8000c9e040       Image:         System
Wait Start TickCount      0              Ticks: 917455 (0:03:58:32.389)
Context Switch Count      1161932        IdealProcessor: 3             
UserTime                  00:00:00.000
KernelTime                03:57:26.900
Win32 Start Address nt!KiIdleLoop (0xfffff800028c46a0)
Stack Init fffff88002f3bdb0 Current fffff88002f3bd40
Base fffff88002f3c000 Limit fffff88002f36000 Call 0
Priority 16 BasePriority 0 UnusualBoost 0 ForegroundBoost 0 IoPriority 0 PagePriority 0
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`02f1d578 fffff800`029b16d2 : 00000000`00000000 fffff880`02f1dfc0 00000000`00000065 fffff800`028fa314 : nt!RtlpBreakWithStatusInstruction
fffff880`02f1d580 fffff800`029b24be : 00000000`00000003 00000000`00000000 fffff800`028f6ee0 00000000`00000111 : nt!KiBugCheckDebugBreak+0x12
fffff880`02f1d5e0 fffff800`028bc004 : fffffa80`01bde9a0 fffff800`02960aaf fffffa80`0203b200 fffffa80`01bde9a0 : nt!KeBugCheck2+0x71e
fffff880`02f1dcb0 fffff800`028bb469 : 00000000`00000111 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KeBugCheckEx+0x104
fffff880`02f1dcf0 fffff800`028b914f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiBugCheckDispatch+0x69
fffff880`02f1de30 fffff800`0281a41c : 00000000`00000000 00000000`00000000 fffffa80`02c59b70 00000000`00000000 : nt!KiNmiInterruptEnd+0x15 (TrapFrame @ fffff880`02f1de30)
fffff880`02f1da58 00000000`00000000 : 00000000`00000000 fffffa80`02c59b70 00000000`00000000 00000000`00000002 : hal!XmOpcodeRegister+0x28

[matthew.robben@gmail.com]

another bugcheck (pretty print)

[matthew.robben@gmail.com]

I was able to resolve this by changing the number of CPU's from 4 to 1. This loads the single processor HAL and doesn't crash due to NMI issues. The bug looks like it's in the multiprocessor HAL or handling of NMI's in the presence of more than one cpu.

[Frank Mehnert]

You should also change the chipset emulation from ICH9 to PIIX3 since the former has known bugs and is marked as experimental.