Guru meditation (VERR_VMX_INVALID_VMXON_PTR) when using RT kernel

[Rainmaker]

When a host is running Debian Sid, and using the rt-kernel patchset, VirtualBox is unstable. Especially my Windows 2003 guest often crashes in a "guru meditation". Other guests, running Scientific Linux 6.1 (basically Redhat Enterprise Linux 6.1), do crash, but less often.

In dmesg on the host, a lot of these errors are visible, but the timing does not exactly match guru meditiations. Therefore, I'm unsure if these are the cause, a side effect, or harmless, but I'll mention them anyway :)

Full VBox.log is attached.

[12558.408017] BUG: scheduling while atomic: 
...
[12558.408076] Call Trace:
[12558.408082]  [<ffffffff8103bbe7>] ? __schedule_bug+0x3e/0x52
[12558.408086]  [<ffffffff81347616>] ? __schedule+0x96/0x537
[12558.408092]  [<ffffffff810434db>] ? schedule+0x6a/0x85
[12558.408094]  [<ffffffff81348c34>] ? rt_spin_lock_slowlock+0x12f/0x1c3
[12558.408104]  [<ffffffffa0fc5c41>] ? VBoxHost_RTHandleTableLookupWithCtx+0x61/0xcc [vboxdrv]
[12558.408110]  [<ffffffffa0fbb8ab>] ? SUPSemEventSignal+0x4e/0x83 [vboxdrv]
[12558.408121]  [<ffffffff810748bf>] ? do_futex+0x726/0x7f1
[12558.408127]  [<ffffffffa0fb80ed>] ? supdrvIOCtlFast+0x77/0x7e [vboxdrv]
[12558.408131]  [<ffffffffa0fb528f>] ? VBoxDrvLinuxIOCtl+0x3c/0x19d [vboxdrv]
[12558.408135]  [<ffffffff81110de3>] ? do_vfs_ioctl+0x41e/0x45f
[12558.408137]  [<ffffffff81347aa0>] ? __schedule+0x520/0x537
[12558.408140]  [<ffffffff811040c5>] ? fput+0x1a/0x194
[12558.408142]  [<ffffffff81110e6f>] ? sys_ioctl+0x4b/0x70
[12558.408146]  [<ffffffff8134da12>] ? system_call_fastpath+0x16/0x1b

[Rainmaker]

VBox.log

[Rainmaker]

dmesg error. For readibility, in an attachement.

[Rainmaker]

The same problem is unfortunately still present in 4.1.6.

[SynrG]

Same problem, Debian sid, linux-image-3.2.0-1-rt-amd64 kernel (Debian version 3.2.6-1, i.e. upstream 3.2.6 kernel release) and virtualbox-4.1 (Debian version 4.1.8-75467~Debian~wheezy). I have fallen back to non-rt kernel as a workaround.

[Technologov]

+1.

-Technologov

[rcs-34]

I have had the same (or at least a similar) problem. I have "solved" it by disabling PAE/NX, VT-x/AMD-V and Nested paging in the machine settings. I'm not sure which of them have done the trick, but now everything works fine.

The host is Debian Wheezy with RT-patched linux, guests are two gentoos with RT-patched kernels and one Windows XP.

[aeichner]

Please reopen if still relevant with a recent VirtualBox release.