[vbox-dev] Fwd: VNC Extension: make Windows text cursors visible

[Yonathan]

Dear VBox,

I hope you will enjoy these patches for the cursor in Windows. I have
a question about the third patch (to ConsoleVRDPServer.cpp) though.

1) Fix a memory leak caused by incorrect use of rfbFreeCursor.
rfbFreeCursor only frees the fields that are marked cleanup* = TRUE.
Furthermore, VirtualBox should not even call rfbFreeCursor; it is a
function called by libvncserver.

2) The second patch fixes text selection cursors in Windows (which
have AND=1, XOR=white). This is similar to the OSX and Linux
implementations of UISession::setPointerShape within
src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp. But instead
of always setting the pixel to black, I calculate the inversion.

It also fixes color cursors because both RDP and our VNC extension
settings use BGR, but the VNC extension was swapping R and B.

3) Explicitly zero the pixels of the color mask for semitransparent
pixels that fall under the 50% threshold, so that we don’t confuse
them for inverting pixels. I am not sure whether it is safe to modify
the ComSafeArrayIn parameter here; do I have to copy it first?

How to test these changes:
1. Compile vbox with VNC extension pack enabled. I use the following
LocalConfig.kmk:
VBOX_WITH_EXTPACK_VNC = 1
INCS += /path/to/libvncserver-git
LIBPATH += /path/to/libvncserver-git/libvncserver/.libs

2. Take a snapshot, then configure the VM to use the VNC extension:
VBoxManage modifyvm "windows 8" --vrdeextpack VNC --vrdeproperty
VNCPassword=12345 --vrdeport 5900 --vrde on

3. Start a Windows VM with VirtualBox guest additions installed. Make
sure hardware acceleration is enabled, so that VNC will draw the
cursor.

4. Connect to the VM with a VNC viewer.

I would like to contribute this under the MIT license.

Yonathan Randolph
Acuitus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-VNC-extension-Fix-memory-leak-make-libvncserver-own-.patch
Type: application/octet-stream
Size: 2634 bytes
Desc: not available
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20141017/b408c79c/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-VNC-extension-Make-XOR-cursors-from-Windows-visible-.patch
Type: application/octet-stream
Size: 5134 bytes
Desc: not available
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20141017/b408c79c/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Fully-zero-pixels-that-are-too-transparent.patch
Type: application/octet-stream
Size: 2613 bytes
Desc: not available
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20141017/b408c79c/attachment-0002.obj>