[vbox-dev] VNC Extension: make Windows text cursors visible

[Vitali Pelenjow]

Hi Yonathan,

the code in the third (ConsoleVRDPServer.cpp)  patch should go into the 
VNC server.
ConsoleVRDPServer.cpp must not modify the XOR mask for alpha cursors, 
because it will break them.
It the the task of the VRDE (VNC or VRDP) to figure out what to do with 
the XOR mask
if the client does not support the alpha channel.

Thanks,
Vitali.


Yonathan wrote:
> To Virtualbox maintainers,
>
> You may replace the third patch that I mentioned with the attached
> one. This new patch does not modify the input ComSafeArrayIn
> parameter.
>
> Please let me know what else I can do to rework any of these patches
> so they can be merged more cleanly.
>
> Yonathan Randolph
> Acuitus
>
> On Fri, Oct 17, 2014 at 12:15 PM, Yonathan <yonathan at gmail.com> wrote:
>> Dear VBox,
>>
>> I hope you will enjoy these patches for the cursor in Windows. I have
>> a question about the third patch (to ConsoleVRDPServer.cpp) though.
>>
>> 1) Fix a memory leak caused by incorrect use of rfbFreeCursor.
>> rfbFreeCursor only frees the fields that are marked cleanup* = TRUE.
>> Furthermore, VirtualBox should not even call rfbFreeCursor; it is a
>> function called by libvncserver.
>>
>> 2) The second patch fixes text selection cursors in Windows (which
>> have AND=1, XOR=white). This is similar to the OSX and Linux
>> implementations of UISession::setPointerShape within
>> src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp. But instead
>> of always setting the pixel to black, I calculate the inversion.
>>
>> It also fixes color cursors because both RDP and our VNC extension
>> settings use BGR, but the VNC extension was swapping R and B.
>>
>> 3) Explicitly zero the pixels of the color mask for semitransparent
>> pixels that fall under the 50% threshold, so that we don’t confuse
>> them for inverting pixels. I am not sure whether it is safe to modify
>> the ComSafeArrayIn parameter here; do I have to copy it first?
>>
>> How to test these changes:
>> 1. Compile vbox with VNC extension pack enabled. I use the following
>> LocalConfig.kmk:
>> VBOX_WITH_EXTPACK_VNC = 1
>> INCS += /path/to/libvncserver-git
>> LIBPATH += /path/to/libvncserver-git/libvncserver/.libs
>>
>> 2. Take a snapshot, then configure the VM to use the VNC extension:
>> VBoxManage modifyvm "windows 8" --vrdeextpack VNC --vrdeproperty
>> VNCPassword=12345 --vrdeport 5900 --vrde on
>>
>> 3. Start a Windows VM with VirtualBox guest additions installed. Make
>> sure hardware acceleration is enabled, so that VNC will draw the
>> cursor.
>>
>> 4. Connect to the VM with a VNC viewer.
>>
>> I would like to contribute this under the MIT license.
>>
>> Yonathan Randolph
>> Acuitus
>>
>>
>> _______________________________________________
>> vbox-dev mailing list
>> vbox-dev at virtualbox.org
>> https://www.virtualbox.org/mailman/listinfo/vbox-dev