vbox 3.2.12 causes kernel panic on Hardened Gentoo host

[Anthony G. Basile]

Using the latest stable hardened kernel from Gentoo (hardened-sources-2.6.36-r6) starting a virtual machine causes a kernel panic on amd64. The kernel is configured with UDEREF off (as discussed in Ticket #2652), but with KERNEX on. The problem occurs because the vbox code tries to modify read-only kernel pages which it should not do.

For the panic, see: ​http://bugs.gentoo.org/show_bug.cgi?id=350503

[Anthony G. Basile]

Sorry, minor error in my original description. Full kernel option name is CONFIG_PAX_KERNEXEC.

[Frank Mehnert]

Is this related to #941? Some information is missing from your VBox.log file. You could also try to start VirtualBox 4.0. It will not fix your issue but the error message about the missing symbol should be more verbose.

[PaX Team]

#941 is about a different conflict that i believe got resolved over time. in this case the problem is that certain kernel data structures are enforced to be read-only under the PaX feature called KERNEXEC, in particular the GDTs and the TSSs which i believe are being modified by vbox here (looked like resetting a bit, possibly the busy bit?). the way to get such code to work under PaX/KERNEXEC is to encompass the write between pax_open_kernel()/pax_close_kernel() calls (or get rid of the write, e.g., vmware manages without it somehow). KERNEXEC poses other challenges though, especially on i386 where the kernel no longer runs in a flat code segment and i'm not sure you guys want to go all the way to support this feature in which case we'll just have to declare KERNEXEC incompatible with vbox.

[Frank Mehnert]

Probably only incompatible in non-VT-x/AMD-V mode. I assume that the original reporter did not have VT-x/AMD-V enabled (not clear from his partly submitted log file).

[PaX Team]

err, this problem is not about guests, but the host and the problem does occur with VT-x for me ;). so to be clear: it doesn't matter what would run in the guest because the host/vbox doesn't get that far, it runs afoul of KERNEXEC early on some GDT/TSS/whatever manipulation attempt (that's probably part of the guest state setup, but you can probably tell better from the code dump in the oops).

[Frank Mehnert]

I know that this is a host problem but we do things differently if VT-x/AMD-V is enabled. Are you 100% sure that this is the case for your host?

Note that on 64-bit Linux hosts we load executable code into a static array. This might be the reason for the incompatibility with PaX. Ubuntu 11.04 kernels have RONX set and are affected by the same problem. 4.0 contains a workaround for these kernels, I wonder if it would help Gentoo as well.

[PaX Team]

yes, i'm sure it's with VT-x on (i explicitly enable it on my VMs). the method you use to get an rwx area on 64 bit linux has been supported by recent versions of PaX, the oops reported here happens afterwards in code that is loaded into that region (you can see it from the oops too, RIP=g_abExecMemory+0xa297/0x18490c[vboxdrv]). if you were only using the linux provided methods to load modules... we'd have proper symbolic backtrace for the offending code as well and wouldn't have to hunt down insns sequences by hand (something i personally didn't feel like spending my time on, hence this report ;). as for 4.0, i have yet to try it but as i said the issue is not with how you load your own code but what that code does later (on i386 there's an extra issue with logical/linear address translation for function pointers). and i believe despite the new ubuntu kernel's use of ro/nx separation you won't find the same issue there because unlike PaX/KERNEXEC, they don't then go out of their way to actually make important kernel structures such as the GDT/etc read-only.

[Frank Mehnert]

Be aware that enabling VT-x does not necessarily means that it is really enabled. Only the log will show this (or the blue icon of the VM window).

[PaX Team]

my logs has this:

HWACCM: VMX enabled
HWACCM: TPR Patching enabled.
HWACCM: VT-x/AMD-V init method: GLOBAL

[Frank Mehnert]

Right.

[aeichner]

Please reopen if still relevant with a recent VirtualBox release.