Segmentation Fault on Enabling VNC while VM is Running (Patch availabe)

[Low power]

I has set up VNC as the VRDE. I just found that if I enable the VRDE to a VM that is currently running, for example

VBoxManage controlvm 7c4412d9-abea-4c68-9e35-b235ab6c4367 vrde on

then the VM process crashes as soon as the VM display content changes. The versions I tested are 5.0.26 and 5.0.36 on FreeBSD, and 5.1.10 and Debian GNU/Linux.

Example

[whr@ZONGYI-TMACH]:[44]:[~]:$ VBoxManage modifyvm 7c4412d9-abea-4c68-9e35-b235ab6c4367 --vrde off
[whr@ZONGYI-TMACH]:[45]:[~]:$ VBoxHeadless --startvm 7c4412d9-abea-4c68-9e35-b235ab6c4367 &
[2] 5472
[whr@ZONGYI-TMACH]:[46]:[~]:$ Oracle VM VirtualBox Headless Interface 5.1.10_Debian
(C) 2008-2016 Oracle Corporation
All rights reserved.


[whr@ZONGYI-TMACH]:[46]:[~]:$ VBoxManage controlvm 7c4412d9-abea-4c68-9e35-b235ab6c4367 vrde on
24/03/2017 10:11:12 Listening for VNC connections on TCP port 9202
24/03/2017 10:11:12 rfbListenOnTCP6Port: error in bind IPv6 socket: Address already in use
VRDE server is listening on port 9202.
[whr@ZONGYI-TMACH]:[47]:[~]:$ 
[2]+  Segmentation fault      VBoxHeadless --startvm 7c4412d9-abea-4c68-9e35-b235ab6c4367
[whr@ZONGYI-TMACH]:[47]:[~]:$ dmesg | tail -n 1
[865080.247106] EMT[5488]: segfault at a5605 ip 00007f5c0874ca97 sp 00007f5c385a49f8 error 4 in VBoxVNC.so[7f5c0874b000+4000]

Analysis

A null pointer 'instance->mScreenBuffer' was dereferenced in function VNCServerImpl::VRDEUpdate; but the only possible place the pointer was set non-null is VNCServerImpl::VRDEResize, the screen resize handler. If VNC is enabled in the middle in a VM session, the screen is probably won't get resized, and the pointer will always be null.
A easy fix could be to call VNCServerImpl::VRDEResize from VNCServerImpl::VRDEUpdate, if the pointer is currently null.

[Klaus Espenlaub]

Integrated both in trunk and 5.2. See the corresponding testbuilds (revision 126965 or later) as soon as they're available. The change will be part of the tarballs of all future vbox releases, so the distributors of build which use the VNC extpack will also pick it up on the next version bump.