Bug in the vboxdrv module (rtR0MemAllocEx) on Linux 4.8.8+ (grsecurity)

[vboxuser123]

Hi, I'm a Gentoo user, I've a custom kernel. Both with 4.8.8 and 4.8.10 I get the following error when I try to start a virtual machine. I'm using VirtualBox 5.1.10.

Here's the VirtualBox error message:

Failed to open a session for the virtual machine '...'.

The virtual machine '...' has terminated unexpectedly during startup because of signal 11.

Result Code: NS_ERROR_FAILURE (0x80004005)
Component: MachineWrap
Interface: IMachine {b2547866-a0a1-4391-8b86-6952d82efaa0}

From dmesg:

[  367.948576] ------------[ cut here ]------------
[  367.948664] kernel BUG at ./arch/x86/include/asm/irqflags.h:26!
[  367.948752] invalid opcode: 0000 [#1] SMP
[  367.948794] Modules linked in:  vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O)
[  367.949080] CPU: 4 PID: 4724 Comm: VirtualBox Tainted: G           O    4.8.10 #1
[  367.949156] Hardware name: 
[  367.949225] task: ffff880818dc6400 task.stack: ffff880818ed4000
[  367.949292] RIP: 0010:[<ffffffff820409e5>]  [<ffffffff820409e5>] _raw_spin_lock_irqsave+0x25/0x30
[  367.949382] RSP: 0018:ffff880818ed7a90  EFLAGS: 00050206
[  367.949431] RAX: ffff8808823fba00 RBX: 0000000000040246 RCX: ffff8808823fbb50
[  367.949494] RDX: ffff8808823fba90 RSI: 0000000000000002 RDI: ffff8808823fbde0
[  367.949557] RBP: ffff880818ed7a98 R08: ffffea001f3c6220 R09: 0000000000706231
[  367.949635] R10: 0000000000000000 R11: 0000000000000002 R12: ffff8808823fc400
[  367.949698] R13: ffff880818ed7c38 R14: ffff8808823fba00 R15: 0000000000003360
[  367.949762] FS:  00007f9e1b0bb700(0000) GS:ffff880882100000(0000) knlGS:0000000000000000
[  367.949834] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  367.949885] CR2: 00005576006e5000 CR3: 00000008059b9000 CR4: 00000000003606f0
[  367.949963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  367.950026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  367.950088] Stack:
[  367.950110]  0000000000000002 ffff880818ed7b70 ffffffff812475b4 ffffea001fb99380
[  367.950192]  00000001024280ca ffffea002138a780 ffff880818ed7b80 ffffffff8129e1bf
[  367.950286]  fffffffffffffffc 024080c0024280ca 0000001100400190 0000000000000100
[  367.950373] Call Trace:
[  367.950405]  [<ffffffff812475b4>] get_page_from_freelist+0x4d4/0xaa0
[  367.950465]  [<ffffffff8129e1bf>] ? get_partial_node.isra.71+0xdf/0x210
[  367.950541]  [<ffffffff8124807e>] __alloc_pages_nodemask+0x15e/0xe20
[  367.950601]  [<ffffffff812beee4>] ? lookup_fast+0xe4/0x2f0
[  367.950652]  [<ffffffff812473bc>] ? get_page_from_freelist+0x2dc/0xaa0
[  367.950714]  [<ffffffff81293738>] alloc_pages_current+0x98/0x140
[  367.950781]  [<ffffffff81266f79>] kmalloc_order_trace+0x29/0xe0
[  367.950841]  [<ffffffff8129ff23>] __kmalloc+0x1d3/0x220
[  367.950908]  [<ffffffffa002bfc0>] ? rtR0MemAllocEx+0x170/0x240 [vboxdrv]
[  367.950984]  [<ffffffffa002bfc0>] rtR0MemAllocEx+0x170/0x240 [vboxdrv]
[  367.951057]  [<ffffffffa0029d32>] VBoxHost_RTMemAllocTag+0x22/0x50 [vboxdrv]
[  367.951144]  [<ffffffffa0019549>] SUPR0Printf+0x1c9/0x310 [vboxdrv]
[  367.951205]  [<ffffffff812c8c81>] do_vfs_ioctl+0xa1/0x950
[  367.951257]  [<ffffffff812c95a4>] sys_ioctl+0x74/0x80
[  367.951308]  [<ffffffff82040d1f>] entry_SYSCALL_64_fastpath+0x13/0x93
[  367.951375] Code: 84 00 00 00 00 00 55 48 89 e5 53 9c 5b f7 c3 00 00 04 00 75 16 fa 31 c0 ba 01 00 00 00 f0 0f b1 17 85 c0 75 08 48 89 d8 5b 5d c3 <0f> 0b 89 c6 e8 d2 cf 15 ff eb ef 55 48 89 e5 48 83 ec 10 65 48 
[  367.951822] RIP  [<ffffffff820409e5>] _raw_spin_lock_irqsave+0x25/0x30
[  367.951889]  RSP <ffff880818ed7a90>
[  367.976384] ---[ end trace 0f2758158db10a18 ]---

[Frank Mehnert]

Could you post the lines around arch/x86/include/asm/irqflags.h:26? For vanilla Linux 4.8.11 I have

 11 static inline unsigned long native_save_fl(void)
 12 {
 13     unsigned long flags;
 14 
 15     /*
 16      * "=rm" is safe here, because "pop" adjusts the stack before
 17      * it evaluates its effective address -- this is part of the
 18      * documented behavior of the "pop" instruction.
 19      */
 20     asm volatile("# __raw_save_flags\n\t"
 21              "pushf ; pop %0"
 22              : "=rm" (flags)
 23              : /* no input */
 24              : "memory");
 25 
 26     return flags;
 27 }

which does not contain any WARN(), BUG() or similar macro.

[vboxuser123]

Before

return flags;

I have

BUG_ON(flags & X86_EFLAGS_AC);

This is part of the grsecurity patch, apparently they don't like to have alignment check enabled there.

[Frank Mehnert]

Thanks. It would have been helpful if you had mentioned grsecurity from the beginning. Which Linux sources are you using? Vanilla 4.8.8 or later plus the grsecurity patch? Any additional patches? Could you pst the URL of the grsecurity patch you are using?

[Frank Mehnert]

Btw, I don't think that we can make our code compatible to grsecurity. The AC flag has to be set to disable SMAP within the VirtualBox kernel code as it needs to access userland memory from kernel context. This will be eventually changed but not in the near-distant future.

[vboxuser123]

I'm using the hardened-sources Gentoo kernel, here should be the patches:

​http://dev.gentoo.org/~blueness/hardened-sources/hardened-patches/

Here you can find the ebuild:

​https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-kernel/hardened-sources/hardened-sources-4.8.12.ebuild