[vbox-dev] Fix 12611: VNC crash when client connects while restoring

[Yonathan]

We would like to contribute this patch for bug
https://www.virtualbox.org/ticket/12611 under the MIT license. It
fixes a crash in the VNC Extension of VirtualBoxVM when a VNC client
connects while VirtualBox is still restoring saved state. The fix is
to allocate a frame buffer of arbitrary size within
VRDEEnableConnections that the VNC thread will use until VRDEResize.

Background: This is the standard sequence of events at VM startup:

1. VirtualBox calls extension entry point VRDECreateServer.

2. VirtualBox calls extension entry point VRDEEnableConnections, which
listens for connections and starts VNC server thread.

3. VirtualBox calls extension entry point VRDEResize. Extension calls
the callback VRDECallbackFramebufferQuery to obtain the new size and
screen buffer, and allocates a frame buffer according to this size.

4. VirtualBox calls extension entry point VRDEUpdate whenever the
pixels in the screen buffer change, and extension will copy the pixels
from screen buffer to frame buffer.

There can be a significant time between steps 2 and 3 (e.g., when
VirtualBox has to restore saved state). If a VNC client happens to
connect during that gap, then the VNC server tries to copy pixel data
from the frame buffer to the client. Since the frame buffer was NULL
at this point, this resulted in SEGV in the VNC event loop thread.
Sample stack trace:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000

Thread 41 Crashed:
0 libsystem_c.dylib 0x00007fff8bc9c9e7 memmove$VARIANT$sse42 + 31
1 libvncserver.0.dylib 0x000000010d3b5c73 rfbTranslateNone + 59
2 libvncserver.0.dylib 0x000000010d3b497d rfbSendRectEncodingHextile + 4630
3 libvncserver.0.dylib 0x000000010d3ae07e rfbSendFramebufferUpdate + 3314
4 libvncserver.0.dylib 0x000000010d3a92fb clientOutput + 320
5 libsystem_c.dylib 0x00007fff8bcaf742 _pthread_start + 327
6 libsystem_c.dylib 0x00007fff8bc9c181 thread_start + 13

Yonathan Randolph
Acuitus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Fix-bug-NULL-dereference-if-a-VNC-client-connects-wh.patch
Type: application/octet-stream
Size: 3599 bytes
Desc: not available
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20150213/21c022c2/attachment.obj>