[vbox-dev] VNC Extension: make Windows text cursors visible

Yonathan yonathan at gmail.com
Tue Oct 28 08:24:10 UTC 2014

To Virtualbox maintainers,

You may replace the third patch that I mentioned with the attached
one. This new patch does not modify the input ComSafeArrayIn

Please let me know what else I can do to rework any of these patches
so they can be merged more cleanly.

Yonathan Randolph

On Fri, Oct 17, 2014 at 12:15 PM, Yonathan <yonathan at gmail.com> wrote:
> Dear VBox,
> I hope you will enjoy these patches for the cursor in Windows. I have
> a question about the third patch (to ConsoleVRDPServer.cpp) though.
> 1) Fix a memory leak caused by incorrect use of rfbFreeCursor.
> rfbFreeCursor only frees the fields that are marked cleanup* = TRUE.
> Furthermore, VirtualBox should not even call rfbFreeCursor; it is a
> function called by libvncserver.
> 2) The second patch fixes text selection cursors in Windows (which
> have AND=1, XOR=white). This is similar to the OSX and Linux
> implementations of UISession::setPointerShape within
> src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp. But instead
> of always setting the pixel to black, I calculate the inversion.
> It also fixes color cursors because both RDP and our VNC extension
> settings use BGR, but the VNC extension was swapping R and B.
> 3) Explicitly zero the pixels of the color mask for semitransparent
> pixels that fall under the 50% threshold, so that we don’t confuse
> them for inverting pixels. I am not sure whether it is safe to modify
> the ComSafeArrayIn parameter here; do I have to copy it first?
> How to test these changes:
> 1. Compile vbox with VNC extension pack enabled. I use the following
> LocalConfig.kmk:
> INCS += /path/to/libvncserver-git
> LIBPATH += /path/to/libvncserver-git/libvncserver/.libs
> 2. Take a snapshot, then configure the VM to use the VNC extension:
> VBoxManage modifyvm "windows 8" --vrdeextpack VNC --vrdeproperty
> VNCPassword=12345 --vrdeport 5900 --vrde on
> 3. Start a Windows VM with VirtualBox guest additions installed. Make
> sure hardware acceleration is enabled, so that VNC will draw the
> cursor.
> 4. Connect to the VM with a VNC viewer.
> I would like to contribute this under the MIT license.
> Yonathan Randolph
> Acuitus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Fully-zero-pixels-that-are-too-transparent-revised.patch
Type: application/octet-stream
Size: 1981 bytes
Desc: not available
Url : http://www.virtualbox.org/pipermail/vbox-dev/attachments/20141028/be2b8496/attachment.obj 

More information about the vbox-dev mailing list