[vbox-dev] VNC Extension: make Windows text cursors visible

[Yonathan]

To Virtualbox maintainers,

You may replace the third patch that I mentioned with the attached
one. This new patch does not modify the input ComSafeArrayIn
parameter.

Please let me know what else I can do to rework any of these patches
so they can be merged more cleanly.

Yonathan Randolph
Acuitus

On Fri, Oct 17, 2014 at 12:15 PM, Yonathan <yonathan at gmail.com> wrote:
> Dear VBox,
>
> I hope you will enjoy these patches for the cursor in Windows. I have
> a question about the third patch (to ConsoleVRDPServer.cpp) though.
>
> 1) Fix a memory leak caused by incorrect use of rfbFreeCursor.
> rfbFreeCursor only frees the fields that are marked cleanup* = TRUE.
> Furthermore, VirtualBox should not even call rfbFreeCursor; it is a
> function called by libvncserver.
>
> 2) The second patch fixes text selection cursors in Windows (which
> have AND=1, XOR=white). This is similar to the OSX and Linux
> implementations of UISession::setPointerShape within
> src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp. But instead
> of always setting the pixel to black, I calculate the inversion.
>
> It also fixes color cursors because both RDP and our VNC extension
> settings use BGR, but the VNC extension was swapping R and B.
>
> 3) Explicitly zero the pixels of the color mask for semitransparent
> pixels that fall under the 50% threshold, so that we don’t confuse
> them for inverting pixels. I am not sure whether it is safe to modify
> the ComSafeArrayIn parameter here; do I have to copy it first?
>
> How to test these changes:
> 1. Compile vbox with VNC extension pack enabled. I use the following
> LocalConfig.kmk:
> VBOX_WITH_EXTPACK_VNC = 1
> INCS += /path/to/libvncserver-git
> LIBPATH += /path/to/libvncserver-git/libvncserver/.libs
>
> 2. Take a snapshot, then configure the VM to use the VNC extension:
> VBoxManage modifyvm "windows 8" --vrdeextpack VNC --vrdeproperty
> VNCPassword=12345 --vrdeport 5900 --vrde on
>
> 3. Start a Windows VM with VirtualBox guest additions installed. Make
> sure hardware acceleration is enabled, so that VNC will draw the
> cursor.
>
> 4. Connect to the VM with a VNC viewer.
>
> I would like to contribute this under the MIT license.
>
> Yonathan Randolph
> Acuitus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Fully-zero-pixels-that-are-too-transparent-revised.patch
Type: application/octet-stream
Size: 1982 bytes
Desc: not available
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20141028/be2b8496/attachment.obj>