Linux hosts with kvm enabled are unable to use virtualization properly

[Patrick Lauer]

On linux hosts with kvm loaded virtualbox is unable to use the VT-x/AMD-V extensions. This prevents 64bit guests from booting and causes some issues for 32bit guests.

For 64bit it properly gives a warning "Could not enable VT-x/AMD-V" and aborts, for 32bit there is no warning. Trying to boot OpenSolaris (in 32bit mode) fails with:

Unexpected trap
instruction pointer  0x10
error code, optional 0x7
code segment         0x2
flags register       0xfe8c82d8
Attempting stack backtrace:
Stack traceback:
  0x10(0x2,0xfe8c82d8,0xfe80ae8c,0x0,0x18,0x2)
  0x10(0xff53f0,0xff53f0,0xff53f0,0xff53f0,0xfea5f0,0xe987f0)
  0xff53f0(0x6c746365,0x20200a79,0x20202020,0x20202020,0x69772020,0x6e696874)
  0x72696420Unexpected trap
instruction pointer  0xfe8155e0
error code, optional 0x0
code segment         0x10
flags register       0x6
Attempting stack backtrace:
Stack traceback:
  bop_traceback+74(0x10,0x6,0x2,0xfec0be1b,0xfec3e498,0xffffffff)
  bop_traceback+74
unexpected trap in early boot
Press any key to reboot.

This is quite unexpected and tricky to debug. Removing the kvm modules (or building a kernel without it) allows OpenSolaris to boot properly.

[Sander van Leeuwen]

That's a KVM bug. After many years they've finally changed this apparently.