[vbox-dev] VNC Extension: make Windows text cursors visible

[Vitali Pelenjow]

Hi Yonathan,

you are right, this is the old code path with 24-bit XOR mask.
In this case the patch is fine.

Thanks,
Vitali.


On 10/28/2014 6:27 PM, Yonathan wrote:
> Hi Vitali,
>
> An extension that implements VRDEMousePtr will receive the AND mask
> and 32-bit XOR image as you say. But an extension such as VNC that
> only implements the old VRDEColorPointer will get a 24-bit color mask
> with no alpha channel. (These callbacks correspond to RDP’s New
> Pointer Update vs. Color Pointer Update.) The VNC extension cannot
> tell whether there used to be an alpha channel, but if it wants to use
> those pixels with AND=1 (patches 0001 and 0002) then it must ignore
> the semitransparent pixels somehow.
>
> I have attached the cursors that I have been testing with. Without
> patch 0003, the color on the transparent half gets inverted (becomes
> pinkish white) in VNC. With patch 0003, they just become invisible.
> Granted, it never looks good given the alpha threshold and blending
> with black. But at least it should not do something too unexpected.
>
> Cheers,
> Yonathan
>
> On Tue, Oct 28, 2014 at 4:14 AM, Vitali Pelenjow
> <Vitali.Pelenjow at oracle.com> wrote:
>> Hi Yonathan,
>>
>> the code in the third (ConsoleVRDPServer.cpp)  patch should go into the VNC
>> server.
>> ConsoleVRDPServer.cpp must not modify the XOR mask for alpha cursors,
>> because it will break them.
>> It the the task of the VRDE (VNC or VRDP) to figure out what to do with the
>> XOR mask
>> if the client does not support the alpha channel.
>>
>> Thanks,
>> Vitali.
>>
>>
>> Yonathan wrote:
>>> To Virtualbox maintainers,
>>>
>>> You may replace the third patch that I mentioned with the attached
>>> one. This new patch does not modify the input ComSafeArrayIn
>>> parameter.
>>>
>>> Please let me know what else I can do to rework any of these patches
>>> so they can be merged more cleanly.
>>>
>>> Yonathan Randolph
>>> Acuitus
>>>
>>> On Fri, Oct 17, 2014 at 12:15 PM, Yonathan <yonathan at gmail.com> wrote:
>>>> Dear VBox,
>>>>
>>>> I hope you will enjoy these patches for the cursor in Windows. I have
>>>> a question about the third patch (to ConsoleVRDPServer.cpp) though.
>>>>
>>>> 1) Fix a memory leak caused by incorrect use of rfbFreeCursor.
>>>> rfbFreeCursor only frees the fields that are marked cleanup* = TRUE.
>>>> Furthermore, VirtualBox should not even call rfbFreeCursor; it is a
>>>> function called by libvncserver.
>>>>
>>>> 2) The second patch fixes text selection cursors in Windows (which
>>>> have AND=1, XOR=white). This is similar to the OSX and Linux
>>>> implementations of UISession::setPointerShape within
>>>> src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp. But instead
>>>> of always setting the pixel to black, I calculate the inversion.
>>>>
>>>> It also fixes color cursors because both RDP and our VNC extension
>>>> settings use BGR, but the VNC extension was swapping R and B.
>>>>
>>>> 3) Explicitly zero the pixels of the color mask for semitransparent
>>>> pixels that fall under the 50% threshold, so that we don’t confuse
>>>> them for inverting pixels. I am not sure whether it is safe to modify
>>>> the ComSafeArrayIn parameter here; do I have to copy it first?
>>>>
>>>> How to test these changes:
>>>> 1. Compile vbox with VNC extension pack enabled. I use the following
>>>> LocalConfig.kmk:
>>>> VBOX_WITH_EXTPACK_VNC = 1
>>>> INCS += /path/to/libvncserver-git
>>>> LIBPATH += /path/to/libvncserver-git/libvncserver/.libs
>>>>
>>>> 2. Take a snapshot, then configure the VM to use the VNC extension:
>>>> VBoxManage modifyvm "windows 8" --vrdeextpack VNC --vrdeproperty
>>>> VNCPassword=12345 --vrdeport 5900 --vrde on
>>>>
>>>> 3. Start a Windows VM with VirtualBox guest additions installed. Make
>>>> sure hardware acceleration is enabled, so that VNC will draw the
>>>> cursor.
>>>>
>>>> 4. Connect to the VM with a VNC viewer.
>>>>
>>>> I would like to contribute this under the MIT license.
>>>>
>>>> Yonathan Randolph
>>>> Acuitus
>>>>
>>>>
>>>> _______________________________________________
>>>> vbox-dev mailing list
>>>> vbox-dev at virtualbox.org
>>>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
>>