Null pointer dereference in vmsvgaR3PortReportMonitorPositions when using VboxVGA graphics device => fixed in svn/6.1.x x>10

[abacabadabacaba]

I'm using VirtualBox 6.1.8 with 64-bit Debian both as a host and as a guest OS.

Under specific conditions, VirtualBox crashes during guest OS boot. I found the following necessary conditions for the crash to occur:

• Graphics device: VboxVGA.
• Guest additions version 6.1.8 must be installed in the guest. With earlier versions (such as 6.1.6), the bug doesn't reproduce.

Here is a stack trace of the crash:

#0 vmsvgaR3PortReportMonitorPositions (pInterface=0x7fb40404a198, cPositions=<optimized out>, pPosition=0x7fb404da601c)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/Devices/Graphics/DevVGA-SVGA.cpp:857

#1 0x00007fb424276fd3 in Display::i_handleUpdateMonitorPositions (this=0x7fb4000023c0, cPositions=1, pPosition=0x7fb404da601c)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/Main/src-client/DisplayImpl.cpp:1296

#2 0x00007fb424296e98 in vmmdevUpdateMonitorPositions (pInterface=<optimized out>, cPositions=<optimized out>, pPositions=<optimized out>)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/Main/include/ConsoleImpl.h:146

#3 0x00007fb3eb043d69 in vmmdevRequestHandler (pDevIns=<optimized out>, pvUser=<optimized out>, offPort=<optimized out>, u32=<optimized out>, cb=<optimized out>)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/Devices/VMMDev/VMMDev.cpp:2022

#4 0x00007fb4246a46b0 in IOMIOPortWrite (pVM=0x7fb404c3e000, pVCpu=pVCpu@entry=0x7fb404c29000, Port=<optimized out>, u32Value=1037959912, cbValue=4)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/VMM/VMMAll/IOMAll.cpp:414

#5 0x00007fb4245a4172 in IOMR3ProcessForceFlag (pVM=pVM@entry=0x7fb404c3e000, pVCpu=pVCpu@entry=0x7fb404c29000, rcStrict=2622)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/VMM/VMMR3/IOM.cpp:386

#6 0x00007fb42459ab55 in emR3HighPriorityPostForcedActions (pVM=pVM@entry=0x7fb404c3e000, pVCpu=pVCpu@entry=0x7fb404c29000, rc=<optimized out>)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/VMM/VMMR3/EM.cpp:1469

#7 0x00007fb42459e4e2 in emR3HmExecute (pVM=pVM@entry=0x7fb404c3e000, pVCpu=pVCpu@entry=0x7fb404c29000, pfFFDone=pfFFDone@entry=0x7fb404d4bdf7)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/VMM/VMMR3/EMHM.cpp:437

#8 0x00007fb42459cbc9 in EMR3ExecuteVM (pVM=<optimized out>, pVCpu=<optimized out>) at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/VMM/VMMR3/EM.cpp:2658 #9 0x00007fb4245fd730 in vmR3EmulationThreadWithId (hThreadSelf=<optimized out>, pUVCpu=0x7fb404d536a0, idCpu=0)

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/VMM/VMMR3/VMEmt.cpp:237

#10 0x00007fb4361b5494 in rtThreadMain (pThread=pThread@entry=0x7fb3e4006370, NativeThread=NativeThread@entry=140411151894272, pszThreadName=pszThreadName@entry=0x7fb3e4006c50 "EMT")

at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/Runtime/common/misc/thread.cpp:727

#11 0x00007fb43626d1be in rtThreadNativeMain (pvArgs=0x7fb3e4006370) at /build/virtualbox-wbVDsa/virtualbox-6.1.8-dfsg/src/VBox/Runtime/r3/posix/thread-posix.cpp:362 #12 0x00007fb43654df27 in start_thread (arg=<optimized out>) at pthread_create.c:479 #13 0x00007fb43647a31f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

During the execution of vmsvgaR3PortReportMonitorPositions function, the value of pSVGAState variable is a null pointer. The crash happens when the pointer is dereferenced.

[paulson]

This is a regression in the VBox 6.1.8 Guest Additions which affects guests using X11 and the VBoxVGA graphics controller. If you change your VM to use the 'VMSVGA' graphics controller (the recommended default for Linux guests) then that will allow the VM to work with the 6.1.8 GAs.

[switch7derek]

Replying to paulson:

This is a regression in the VBox 6.1.8 Guest Additions which affects guests using X11 and the VBoxVGA graphics controller. If you change your VM to use the 'VMSVGA' graphics controller (the recommended default for Linux guests) then that will allow the VM to work with the 6.1.8 GAs.

running slackware/linux guest using VMSVGA controller and getting same error ( well, diff mem address, but, same general msg )

[paulson]

Replying to switch7derek:

Replying to paulson:

This is a regression in the VBox 6.1.8 Guest Additions which affects guests using X11 and the VBoxVGA graphics controller. If you change your VM to use the 'VMSVGA' graphics controller (the recommended default for Linux guests) then that will allow the VM to work with the 6.1.8 GAs.

running slackware/linux guest using VMSVGA controller and getting same error ( well, diff mem address, but, same general msg )

To be more specific about the root cause, the command 'VBoxClient --vmsvga-x11' command (which runs at X login via /usr/bin/VBoxClient-all) checks for the VMWARE_CTRL X11 extension (visible via 'xdpyinfo -queryExtensions' in a guest VM running X11) and if not present it normally returns and does nothing. My understanding is that this extension is present with the VBoxSVGA and VMSVGA graphics controllers but not the VBoxVGA graphics controller. The regression is that 'VBoxClient --vmsvga-x11' doesn't return right away in the 6.1.8 GAs when the VMWARE_CTRL X11 extension isn't present. This then leads to the code path seen in this bug report where it queries the monitor position and it expects to find data available from the VMWARE_CTRL X11 extension which isn't there and then crashes dereferencing it. If you are seeing this same crash and stack trace then it stems from using a graphics controller which doesn't have the VMWARE_CTRL X11 extension. Workarounds include commenting out the 'VBoxClient --vmsvga-x11' line in /usr/bin/VBoxClient-all, rolling back to the 6.1.6 GAs, or using a graphics controller which includes the VMWARE_CTRL X11 extension which in the VMs I have access to are VBoxSVGA and VMSVGA.

[paulson]

This has been addressed in revision r138504 of the development branch of VirtualBox and is available now in the development snapshot Testbuilds. The fix has also been backported to the 6.1 branch of VirtualBox as of revision r138505 and is available now in the 6.1.x Testbuilds. The fix will also be part of the next maintenance release of VirtualBox 6.1.x.