[vbox-dev] [ PATCH] Fix crash on startup under Wayland

[Hans de Goede]

Hi,

On 4/7/20 2:29 PM, Frank Batschulat wrote:
> Isn't that just working around the following bug?
> 
> Ticket #19357 (accepted defect)
> VBoxClient crashes on Luuntu 14.04 after updating from 6.1.2 to 6.1.4
> https://www.virtualbox.org/ticket/19357

No that is a different bug, this is about "VirtualBox" (the
hypervisor managing UI) crashing on startup when started
from a Fedora 32 GNOME3 (Wayland) session.

That bug is about VBoxClient, which is part of the guest additions
crashing.

IOW this fixes a hypervisor (UI) crash, the ticket you are
pointing to fixes a guest (additions) crash. So very much a
different thing.

Also note that I have provided a patch fixing the
guest (additions) crash. in the ticket you linked to
5 weeks ago.

As for this being a workaround, VirtualBox makes calls
to QX11Info functions in various places, a quick grep shows:

VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UIFrameBuffer.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UIMachineLogic.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UIKeyboardHandler.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UIMouseHandler.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/globals/UICommon.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/platform/x11/VBoxX11Helper.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/widgets/UIHostComboEditor.cpp

And the most called function is QX11Info::display() and the
resulting X11 Display is then used to make a whole lot of
direct Xlib calls in various places.

IOW the assumption that the X11 QT backend / X11 windows are used
under Linux is deeply embedded inside the VirtualBox code. So yes this
is a workaround but properly fixing this will take a significant
(huge even) amount of effort.

Regards,

Hans




> 
> 
> On Tue, 07 Apr 2020 14:00:47 +0200, Hans de Goede <hdegoede at redhat.com> wrote:
> 
>> I noticed that with a default Fedora 32 workstation install (default GNOME3 on
>> Wayland session) VirtualBox crashes immediately at startup. The problem is that
>> QT5 now defaults to the Wayland backend and VirtualBox assumes the X11 backend
>> is used and makes a bunch of unchecked X11 calls which cause NULL pointer derefs.
>>
>> The attached patch is a simple fix which fixes this by setting the QT backend
>> back to X11.
>>
>> As usual, the patch is released under the MIT license.
>>
>> Regards,
>>
>> Hans
> 
>