Segmentation fault on startup up of `VirtualBox` under sway-wm using Wayland

[tormen]

Hi,

ENVIRONMENT & VERSIONS:

I am using sway as wayland compositor on an archlinux system. sway is started with the following environment variables:

QT_QPA_PLATFORM=wayland QT_WAYLAND_DISABLE_WINDOWDECORATION="1" QT_WAYLAND_FORCE_DPI=100 QT_PLUGIN_PATH=/usr/lib/kde4/plugins/ QT_AUTO_SCREEN_SCALE_FACTOR=0 QT_QPA_PLATFORMTHEME=gtk3

community/virtualbox 6.0.8-2 [installed] community/virtualbox-host-dkms 6.0.8-2 [installed] extra/qt5-wayland 5.12.3-1 (qt qt5) [installed]

PROBLEM

VirtualBox segfaults everytime:

 VirtualBox [1] 8016 segmentation fault (core dumped) VirtualBox

Please let me know if I can provide anything else.

Tormen

[tormen]

coredumpctl gdb VirtualBox

[Michael Thayer]

Please see if this happens when you use an Oracle build of VirtualBox. Otherwise you should open a report with Arch Linux. The "extra/qt5-wayland 5.12.3-1" does not look good to me though. We expect Qt to be using X11.

[Kostiantyn P]

I have the same issue.

My environment: Arch Linux + sway-wm (on Wayland), no xwayland.

I have a HiDPI setup and I intentionally don't want to use xwayland because of scalability problems with it.

I start VirtualBox this way:

$ QT_QPA_PLATFORM=wayland VirtualBox 
Segmentation fault (core dumped)

journalctl shows me this:

Jun 28 17:45:43 eve systemd-coredump[2898]: Process 2862 (VirtualBox) of user 1000 dumped core.
                                            
                                            Stack trace of thread 2862:
                                            #0  0x00007fa778d2163c n/a (libX11.so.6)
                                            #1  0x00007fa778d21b4a XInternAtom (libX11.so.6)
                                            #2  0x00007fa77956159d _Z30X11IsCompositingManagerRunningv (VBoxGlobal.so)
                                            #3  0x00007fa7793e60ef _ZN10VBoxGlobal7prepareEv (VBoxGlobal.so)
                                            #4  0x000055816de3ff66 n/a (VirtualBox)
                                            #5  0x000055816de3d900 n/a (VirtualBox)
                                            #6  0x00007fa7775cfee3 __libc_start_main (libc.so.6)
                                            #7  0x000055816de3f32e n/a (VirtualBox)
                                            
                                            Stack trace of thread 2879:
                                            #0  0x00007fa777493415 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                                            #1  0x00007fa778c844a5 VBoxNsprPR_WaitCondVar (VBoxXPCOM.so)
                                            #2  0x00007fa778c846e7 VBoxNsprPR_Wait (VBoxXPCOM.so)
                                            #3  0x00007fa7711075fc n/a (VBoxXPCOMIPCC.so)
                                            #4  0x00007fa778c71565 n/a (VBoxXPCOM.so)
                                            #5  0x00007fa778c8303f n/a (VBoxXPCOM.so)
                                            #6  0x00007fa778fb03fc n/a (VBoxRT.so)
                                            #7  0x00007fa77906667d n/a (VBoxRT.so)
                                            #8  0x00007fa77748d57f start_thread (libpthread.so.0)
                                            #9  0x00007fa7776a50e3 __clone (libc.so.6)
                                            
                                            Stack trace of thread 2894:
                                            #0  0x00007fa7774937ca pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                                            #1  0x00007fa7790695a0 RTSemEventWait (VBoxRT.so)
                                            #2  0x00007fa7702ee8a8 n/a (VBoxC.so)
                                            #3  0x00007fa778fb03fc n/a (VBoxRT.so)
                                            #4  0x00007fa77906667d n/a (VBoxRT.so)
                                            #5  0x00007fa77748d57f start_thread (libpthread.so.0)
                                            #6  0x00007fa7776a50e3 __clone (libc.so.6)
                                            
                                            Stack trace of thread 2874:
                                            #0  0x00007fa777493415 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                                            #1  0x00007fa771f26f7c n/a (i965_dri.so)
                                            #2  0x00007fa771f26b78 n/a (i965_dri.so)
                                            #3  0x00007fa77748d57f start_thread (libpthread.so.0)
                                            #4  0x00007fa7776a50e3 __clone (libc.so.6)
                                            
                                            Stack trace of thread 2896:
                                            #0  0x00007fa7774937ca pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                                            #1  0x00007fa7790695a0 RTSemEventWait (VBoxRT.so)
                                            #2  0x00007fa77025805f n/a (VBoxC.so)
                                            #3  0x00007fa77025821c n/a (VBoxC.so)
                                            #4  0x00007fa7703b5702 n/a (VBoxC.so)
                                            #5  0x00007fa7795afb8c _ZN12CEventSource8GetEventERK14CEventListeneri (VBoxGlobal.so)
                                            #6  0x00007fa7793a6db8 n/a (VBoxGlobal.so)
                                            #7  0x00007fa778753bf0 n/a (libQt5Core.so.5)
                                            #8  0x00007fa77748d57f start_thread (libpthread.so.0)
                                            #9  0x00007fa7776a50e3 __clone (libc.so.6)
                                            
                                            Stack trace of thread 2895:
                                            #0  0x00007fa7774937ca pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                                            #1  0x00007fa778c84039 n/a (VBoxXPCOM.so)
                                            #2  0x00007fa778c8444a VBoxNsprPR_WaitCondVar (VBoxXPCOM.so)
                                            #3  0x00007fa778c846e7 VBoxNsprPR_Wait (VBoxXPCOM.so)
                                            #4  0x00007fa7710fe34d n/a (VBoxXPCOMIPCC.so)
                                            #5  0x00007fa7710ff39f n/a (VBoxXPCOMIPCC.so)
                                            #6  0x00007fa771105213 n/a (VBoxXPCOMIPCC.so)
                                            #7  0x00007fa778c7f0f5 n/a (VBoxXPCOM.so)
                                            #8  0x00007fa778c7e53d n/a (VBoxXPCOM.so)
                                            #9  0x00007fa7795afb8c _ZN12CEventSource8GetEventERK14CEventListeneri (VBoxGlobal.so)
                                            #10 0x00007fa7793a6db8 n/a (VBoxGlobal.so)
                                            #11 0x00007fa778753bf0 n/a (libQt5Core.so.5)
                                            #12 0x00007fa77748d57f start_thread (libpthread.so.0)
                                            #13 0x00007fa7776a50e3 __clone (libc.so.6)
                                            
                                            Stack trace of thread 2878:
                                            #0  0x00007fa77769a667 __poll (libc.so.6)
                                            #1  0x00007fa778c8675e n/a (VBoxXPCOM.so)
                                            #2  0x00007fa778c88248 VBoxNsprPR_Poll (VBoxXPCOM.so)
                                            #3  0x00007fa77110a247 n/a (VBoxXPCOMIPCC.so)
                                            #4  0x00007fa778c8303f n/a (VBoxXPCOM.so)
                                            #5  0x00007fa778fb03fc n/a (VBoxRT.so)
                                            #6  0x00007fa77906667d n/a (VBoxRT.so)
                                            #7  0x00007fa77748d57f start_thread (libpthread.so.0)
                                            #8  0x00007fa7776a50e3 __clone (libc.so.6)

I can say that VirtualBox works on this setup with xwayland. But it has no value to me, due to scalability issues with xwayland apps (e.g. blurriness).

As for the packages which I have installed:

• qt5-wayland 5.12.4-1
• virtualbox 6.0.8-2
• virtualbox-host-dkms 6.0.8-2

Here is what Arch Linux devs are saying: ​https://bugs.archlinux.org/task/58761

Particularly this comment:

--- This is a virtualbox bug, not a Qt one. VB is unconditionally using x11 specific code. The reason why this became an issue now is because Qt 5.11 defaults to the Wayland backend on Wayland sessions (eg. Gnome). Either the x11 specific code needs to be properly ifdef'd or, if that's not possible, VB should enforce the xcb backend. ---

I would really appreciate if this gets fixed. Until then, I've lost ability to use VirtualBox =(.

[Michael Thayer]

We will probably have to support Wayland at some point, but it will not happen just yet, for lack of developer time. I would like to say "patches welcome", but even integrating the patches would be more work than we can currently handle. That said, if anyone does want to speed up the process, looking at what is needed (the vbox-dev mailing list would be a good place) and giving us feedback would be a good way.

For now we will look to see how to enforce the xcb plugin.

[adrelanos]

I suppose this ticket could be considered a feature request for VirtualBox host software Wayland support.

For now we will look to see how to enforce the xcb plugin.

Has this been this done?

In other words, what is the status of VirtualBox host software Wayland support?

[tekstryder]

Is this the primary tracking bug for native Wayland support by Virtualbox?

Just ran into a segfault when launching virtualBox -platform wayland

Backtrace:

#0  0x00007f1d986ea4ef in XInternAtom (dpy=0x563ef0fe5c40, name=0x7f1d9923db55 "_NET_WM_CM_S0", onlyIfExists=1) at /usr/src/debug/libx11/libX11-1.8.7/src/IntAtom.c:175
        atom = <optimized out>
        sig = 11715082358241691136
        idx = 22078
        n = -247931360
        rep = {type = 48 '0', pad1 = 217 '\331', sequenceNumber = 61693, length = 22078, atom = 4043201600, pad2 = 22078, pad3 = 1353909600, pad4 = 32764, pad5 = 2592181872, pad6 = 32541}
#1  0x00007f1d99120ead in NativeWindowSubsystem::X11IsCompositingManagerRunning() () at /usr/src/debug/virtualbox/VirtualBox-7.0.10/src/VBox/Frontends/VirtualBox/src/platform/x11/VBoxUtils-x11.cpp:66
        pDisplay = 0x563ef0fe5c40
        atom_property_name = <optimized out>
#2  0x00007f1d98eece2e in UICommon::prepare() (this=0x563ef1256160) at /usr/src/debug/virtualbox/VirtualBox-7.0.10/src/VBox/Frontends/VirtualBox/src/globals/UICommon.cpp:327
        rc = <optimized out>
        strLanguageId = {d = 0x7f1d983960a0 <QArrayData::shared_null>}
        visualStateType = UIVisualStateType_Invalid
        startVM = <optimized out>
        fSeparateProcess = <optimized out>
        vmNameOrUuid = {d = 0x0}
        arguments = <optimized out>
        argc = <optimized out>
        i = <optimized out>
#3  0x00007f1d98eee5d9 in UICommon::create(UICommon::UIType) (enmType=enmType@entry=UICommon::UIType_SelectorUI) at /usr/src/debug/virtualbox/VirtualBox-7.0.10/src/VBox/Frontends/VirtualBox/src/globals/UICommon.cpp:165
#4  0x0000563eef0cc174 in TrustedMain(int, char**, char**) (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/virtualbox/VirtualBox-7.0.10/src/VBox/Frontends/VirtualBox/src/main.cpp:527
        fHelpShown = false
        a = {<QGuiApplication> = {<QCoreApplication> = {<QObject> = {_vptr.QObject = 0x7f1d97685620 <vtable for QApplication+16>, static staticMetaObject = {d = {superdata = {direct = 0x0}, stringdata = 0x7f1d98394b00 <_ZL26qt_meta_stringdata_QObject.lto_priv.0>, data = 0x7f1d983949e0 <_ZL20qt_meta_data_QObject.lto_priv.0>, static_metacall = 0x7f1d982d0970 <QObject::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)>, relatedMetaObjects = 0x0, extradata = 0x0}}, d_ptr = {d = 0x563ef0fdc840}, static staticQtMetaObject = {d = {superdata = {direct = 0x0}, stringdata = 0x7f1d983c5720 <_ZL21qt_meta_stringdata_Qt.lto_priv.0>, data = 0x7f1d983c28a0 <_ZL15qt_meta_data_Qt.lto_priv.0>, static_metacall = 0x0, relatedMetaObjects = 0x0, extradata = 0x0}}}, static staticMetaObject = {d = {superdata = {direct = 0x7f1d9851f5a0 <QObject::staticMetaObject>}, stringdata = 0x7f1d98390240 <_ZL35qt_meta_stringdata_QCoreApplication.lto_priv.0>, data = 0x7f1d98390120 <_ZL29qt_meta_data_QCoreApplication.lto_priv.0>, static_metacall = 0x7f1d982a2b70 <QCoreApplication::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)>, relatedMetaObjects = 0x0, extradata = 0x0}}, static self = 0x7ffc50b32370}, static staticMetaObject = {d = {superdata = {direct = 0x7f1d985243a0 <QCoreApplication::staticMetaObject>}, stringdata = 0x7f1d97d89b00 <_ZL34qt_meta_stringdata_QGuiApplication.lto_priv.0>, data = 0x7f1d97d89880 <_ZL28qt_meta_data_QGuiApplication.lto_priv.0>, static_metacall = 0x7f1d9794b290 <QGuiApplication::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)>, relatedMetaObjects = 0x0, extradata = 0x0}}}, static staticMetaObject = {d = {superdata = {direct = 0x7f1d97e5ce40 <QGuiApplication::staticMetaObject>}, stringdata = 0x7f1d97558820 <_ZL31qt_meta_stringdata_QApplication.lto_priv.0>, data = 0x7f1d975586a0 <_ZL25qt_meta_data_QApplication.lto_priv.0>, static_metacall = 0x7f1d9717a180 <QApplication::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)>, relatedMetaObjects = 0x0, extradata = 0x0}}}
        fontDataBase = {d = 0x7f1d97e72200 <_ZZN12_GLOBAL__N_115Q_QGS_privateDb13innerFunctionEvE6holder.lto_priv.0>}
        currentFamily = {d = 0x563ef1189ec0}
        isSubScaleable = true
        isCurrentScaleable = true
        subFamily = {d = 0x563ef1189ec0}
        iResultCode = 1
#5  0x0000563eef0c79d7 in main(int, char**, char**) (argc=<optimized out>, argv=<optimized out>, envp=0x7ffc50b325a8) at /usr/src/debug/virtualbox/VirtualBox-7.0.10/src/VBox/Frontends/VirtualBox/src/main.cpp:760
        fFlags = 0
        rc = 0

[tekstryder]

Unfortunately, the same crash and coredump occurs with Virtualbox 7.0.12.