[vbox-dev] Fix 12611: VNC crash when client connects while restoring

[Frank Mehnert]

Hi Yonathan,

On Friday 13 February 2015 15:22:55 Yonathan wrote:
> We would like to contribute this patch for bug
> https://www.virtualbox.org/ticket/12611 under the MIT license. It
> fixes a crash in the VNC Extension of VirtualBoxVM when a VNC client
> connects while VirtualBox is still restoring saved state. The fix is
> to allocate a frame buffer of arbitrary size within
> VRDEEnableConnections that the VNC thread will use until VRDEResize.
> 
> Background: This is the standard sequence of events at VM startup:
> 
> 1. VirtualBox calls extension entry point VRDECreateServer.
> 
> 2. VirtualBox calls extension entry point VRDEEnableConnections, which
> listens for connections and starts VNC server thread.
> 
> 3. VirtualBox calls extension entry point VRDEResize. Extension calls
> the callback VRDECallbackFramebufferQuery to obtain the new size and
> screen buffer, and allocates a frame buffer according to this size.
> 
> 4. VirtualBox calls extension entry point VRDEUpdate whenever the
> pixels in the screen buffer change, and extension will copy the pixels
> from screen buffer to frame buffer.
> 
> There can be a significant time between steps 2 and 3 (e.g., when
> VirtualBox has to restore saved state). If a VNC client happens to
> connect during that gap, then the VNC server tries to copy pixel data
> from the frame buffer to the client. Since the frame buffer was NULL
> at this point, this resulted in SEGV in the VNC event loop thread.
> Sample stack trace:
> 
> Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
> Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
> 
> Thread 41 Crashed:
> 0 libsystem_c.dylib 0x00007fff8bc9c9e7 memmove$VARIANT$sse42 + 31
> 1 libvncserver.0.dylib 0x000000010d3b5c73 rfbTranslateNone + 59
> 2 libvncserver.0.dylib 0x000000010d3b497d rfbSendRectEncodingHextile + 4630
> 3 libvncserver.0.dylib 0x000000010d3ae07e rfbSendFramebufferUpdate + 3314
> 4 libvncserver.0.dylib 0x000000010d3a92fb clientOutput + 320
> 5 libsystem_c.dylib 0x00007fff8bcaf742 _pthread_start + 327
> 6 libsystem_c.dylib 0x00007fff8bc9c181 thread_start + 13

thank you for this patch. I've just applied a slightly modified version
of your patch to the repository. The next 4.3.x maintenance release will
contain the fix as well.

Kind regards,

Frank
-- 
Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox
ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany

Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Geschäftsführer: Jürgen Kunz

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher