[vbox-dev] VNC Extension: make Windows text cursors visible

[Yonathan]

Hi Vitali,

An extension that implements VRDEMousePtr will receive the AND mask
and 32-bit XOR image as you say. But an extension such as VNC that
only implements the old VRDEColorPointer will get a 24-bit color mask
with no alpha channel. (These callbacks correspond to RDP’s New
Pointer Update vs. Color Pointer Update.) The VNC extension cannot
tell whether there used to be an alpha channel, but if it wants to use
those pixels with AND=1 (patches 0001 and 0002) then it must ignore
the semitransparent pixels somehow.

I have attached the cursors that I have been testing with. Without
patch 0003, the color on the transparent half gets inverted (becomes
pinkish white) in VNC. With patch 0003, they just become invisible.
Granted, it never looks good given the alpha threshold and blending
with black. But at least it should not do something too unexpected.

Cheers,
Yonathan

On Tue, Oct 28, 2014 at 4:14 AM, Vitali Pelenjow
<Vitali.Pelenjow at oracle.com> wrote:
> Hi Yonathan,
>
> the code in the third (ConsoleVRDPServer.cpp)  patch should go into the VNC
> server.
> ConsoleVRDPServer.cpp must not modify the XOR mask for alpha cursors,
> because it will break them.
> It the the task of the VRDE (VNC or VRDP) to figure out what to do with the
> XOR mask
> if the client does not support the alpha channel.
>
> Thanks,
> Vitali.
>
>
> Yonathan wrote:
>>
>> To Virtualbox maintainers,
>>
>> You may replace the third patch that I mentioned with the attached
>> one. This new patch does not modify the input ComSafeArrayIn
>> parameter.
>>
>> Please let me know what else I can do to rework any of these patches
>> so they can be merged more cleanly.
>>
>> Yonathan Randolph
>> Acuitus
>>
>> On Fri, Oct 17, 2014 at 12:15 PM, Yonathan <yonathan at gmail.com> wrote:
>>>
>>> Dear VBox,
>>>
>>> I hope you will enjoy these patches for the cursor in Windows. I have
>>> a question about the third patch (to ConsoleVRDPServer.cpp) though.
>>>
>>> 1) Fix a memory leak caused by incorrect use of rfbFreeCursor.
>>> rfbFreeCursor only frees the fields that are marked cleanup* = TRUE.
>>> Furthermore, VirtualBox should not even call rfbFreeCursor; it is a
>>> function called by libvncserver.
>>>
>>> 2) The second patch fixes text selection cursors in Windows (which
>>> have AND=1, XOR=white). This is similar to the OSX and Linux
>>> implementations of UISession::setPointerShape within
>>> src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp. But instead
>>> of always setting the pixel to black, I calculate the inversion.
>>>
>>> It also fixes color cursors because both RDP and our VNC extension
>>> settings use BGR, but the VNC extension was swapping R and B.
>>>
>>> 3) Explicitly zero the pixels of the color mask for semitransparent
>>> pixels that fall under the 50% threshold, so that we don’t confuse
>>> them for inverting pixels. I am not sure whether it is safe to modify
>>> the ComSafeArrayIn parameter here; do I have to copy it first?
>>>
>>> How to test these changes:
>>> 1. Compile vbox with VNC extension pack enabled. I use the following
>>> LocalConfig.kmk:
>>> VBOX_WITH_EXTPACK_VNC = 1
>>> INCS += /path/to/libvncserver-git
>>> LIBPATH += /path/to/libvncserver-git/libvncserver/.libs
>>>
>>> 2. Take a snapshot, then configure the VM to use the VNC extension:
>>> VBoxManage modifyvm "windows 8" --vrdeextpack VNC --vrdeproperty
>>> VNCPassword=12345 --vrdeport 5900 --vrde on
>>>
>>> 3. Start a Windows VM with VirtualBox guest additions installed. Make
>>> sure hardware acceleration is enabled, so that VNC will draw the
>>> cursor.
>>>
>>> 4. Connect to the VM with a VNC viewer.
>>>
>>> I would like to contribute this under the MIT license.
>>>
>>> Yonathan Randolph
>>> Acuitus
>>>
>>>
>>> _______________________________________________
>>> vbox-dev mailing list
>>> vbox-dev at virtualbox.org
>>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20141028/924a7271/attachment.html>