[vbox-dev] Assertion in vmmdevHGCMSaveLinPtr()

[David P. Reese, Jr.]

Hi,
I'm trying to get my own HGCM module to work on a Mac OS X host and I'm occasionally getting an assertion just after my module's Connect() routine completes.  Here's the message and stack trace:

   !!Assertion Failed!!
   Expression: iPage == cPages
   Location  : /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/Devices/VMMDev/VMMDevHGCM.cpp(278) int vmmdevHGCMSaveLinPtr(PDMDEVINS*, uint32_t, RTGCPTR, uint32_t, uint32_t, VBOXHGCMLINPTR*, RTGCPHYS**)

   Program received signal SIGTRAP, Trace/breakpoint trap.
   [Switching to process 21798]
   0x05dd530f in vmmdevHGCMSaveLinPtr (pDevIns=0x4b1c7a0, iParm=1, GCPtr=3520618496, u32Size=120, iLinPtr=0, paLinPtrs=0x1100410, ppPages=0xb03cbabc) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/Devices/VMMDev/VMMDevHGCM.cpp:278
   278	    AssertRelease (iPage == cPages);
   (gdb) backtrace 
   #0  0x05dd530f in vmmdevHGCMSaveLinPtr (pDevIns=0x4b1c7a0, iParm=1, GCPtr=3520618496, u32Size=120, iLinPtr=0, paLinPtrs=0x1100410, ppPages=0xb03cbabc) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/Devices/VMMDev/VMMDevHGCM.cpp:278
   #1  0x05dd66d3 in vmmdevHGCMCall (pVMMDevState=0x4b1c820, pHGCMCall=0x1100490, cbHGCMCall=68, GCPhys=255787112, f64Bits=false) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/Devices/VMMDev/VMMDevHGCM.cpp:986
   #2  0x05dd3835 in vmmdevRequestHandler (pDevIns=0x4b1c7a0, pvUser=0x4b1c820, Port=53312, u32=255787112, cb=4) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/Devices/VMMDev/VMMDev.cpp:1334
   #3  0x0329c710 in IOMIOPortWrite (pVM=0x7bc000, Port=53312, u32Value=255787112, cbValue=4) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/VMM/VMMAll/IOMAll.cpp:707
   #4  0x0328517f in HWACCMR3RestartPendingIOInstr (pVM=0x7bc000, pVCpu=0x7d7000, pCtx=0x7d7900) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/VMM/VMMR3/HWACCM.cpp:2536
   #5  0x03212c94 in emR3ExecuteIOInstruction [inlined] () at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/VMM/VMMR3/EMHwaccm.cpp:298
   #6  0x03212c94 in emR3HwaccmHandleRC (pVM=0x7bc000, pVCpu=0x7d7000, pCtx=0x7d7900, rc=2621) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/VMM/VMMR3/EMHwaccm.cpp:208
   #7  0x0321308e in emR3HwAccExecute (pVM=0x7bc000, pVCpu=0x7d7000, pfFFDone=0xb03cee9f) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/VMM/VMMR3/EMHwaccm.cpp:578
   #8  0x0320fbf8 in EMR3ExecuteVM (pVM=0x7bc000, pVCpu=0x7d7000) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/VMM/VMMR3/EM.cpp:1995
   #9  0x03277d55 in vmR3EmulationThreadWithId (ThreadSelf=0x180a600, pUVCpu=0x38472c0, idCpu=0) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/VMM/VMMR3/VMEmt.cpp:229
   #10 0x0327809f in vmR3EmulationThread (ThreadSelf=0x1, pvArgs=0x3e3c31) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/VMM/VMMR3/VMEmt.cpp:60
   #11 0x0027a070 in rtThreadMain (pThread=0x180a600, NativeThread=1, pszThreadName=0x180ab88 "EMT") at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/Runtime/common/misc/thread.cpp:680
   #12 0x002c982e in rtThreadNativeMain (pvArgs=0x180a600) at /Users/daver/Software/VirtualBox-4.0.4_OSE/src/VBox/Runtime/r3/posix/thread-posix.cpp:258
   #13 0x90c7d85d in _pthread_start ()
   #14 0x90c7d6e2 in thread_start ()

Taking a look at vmmdevHGCMSaveLinPtr(), it looks like the VMM has failed to map the guest pages required to make a HGCM call.  This assertion is only triggered occasionally and, rather disturbingly, I cannot get the assertion to happen with a debug build of VirtualBox.

I'm running a Linux guest with a kernel module that uses vboxguest.ko to interface with HGCM.  My kernel module simply connects and spawns a kernel thread that makes a service call once every second.

This is using the 4.0.4_OSE sources from the web site.

Has anyone seen this before?

--
David P. Reese, Jr.
daver at siginfo.org