Freeze on VM start with hardened Gentoo

[René Rhéaume]

When I try to start a virtual machine in VirtualBox 1.5.2 OSE and the host kernel is Gentoo hardened-sources-2.6.23 (any revision), the virtual machine window freezes and I find the following in dmesg :

kernel BUG at mm/mmap.c:1695!
invalid opcode: 0000 [#1]
PREEMPT SMP
Modules linked in: bttv video_buf firmware_class ir_common
compat_ioctl32 i2c_algo_bit btcx_risc tveeprom videodev v4l2_common
v4l1_compat ipv6 af_packet nf_conntrack_ipv4 xt_state nf_conntrack
nfnetlink xt_limit xt_tcpudp iptable_filter ip_tables x_tables
nls_iso8859_1 nls_cp850 vfat fat xfs usb_storage snd_hda_intel snd_pcm
snd_timer snd soundcore snd_page_alloc ohci1394 ieee1394 ohci_hcd
i2c_nforce2 forcedeth ehci_hcd amd74xx amd64_agp agpgart 8139too mii
cpufreq_conservative powernow_k8 freq_table processor vboxdrv usbhid
hid usbcore ff_memless ide_scsi
CPU:    0
EIP:    0060:[<00069322>]    Not tainted VLI
EFLAGS: 00210206   (2.6.23-hardened-r2 #2)
eax: 00000000   ebx: 0001a000   ecx: d5cdb4d0   edx: d5cdb528
esi: 4bea5000   edi: 00084400   ebp: 00000000   esp: d5cf3ec4
ds: 0068   es: 0068   fs: 00d8  gs: 0033  ss: 0068
Process VirtualBox (pid: 5865, ti=d5cf2000 task=d5cf0000 task.ti=d5cf2000)
Stack: d5cdb4d0 d5cdb3c8 d5cdb4d0 0006a5c0 0000025a 083b9200 4be24000 d5c8ec00
      00000000 d5cf3f40 00000000 4bea5000 4be24000 d5c8ec00 00000000 0006b2ea
      4bea5000 00100073 00000000 00000000 0004be24 00000000 00081000 ac98bab0
Call Trace:
 [<0006a5c0>] <0> [<0006b2ea>] <0> [<00100073>] <0> [<0004be24>] <0>
[<00081000>] <0> [<00081000>] <0> [<0006babe>] <0> [<00100073>] <0>
[<0004be24>] <0> [<0000acac>] <0> [<0004be24>] <0> [<00005532>] <0>
[<00081000>] <0> [<00200256>] <0> [<001385ff>] <0> [<00138600>] <0>
=======================
Code: 39 c3 75 2b 8b 41 44 3b 42 44 74 0c 0f 0b eb fe 0f 0b eb fe 0f
0b eb fe 8b 41 3c 3b 42 3c 75 ec 33 7a 14 81 e7 dd df ef df 74 9e <0f>
0b eb fe 0f 0b eb fe 0f 0b eb fe 66 90 83 ec 14 89 74 24 0c
EIP: [<00069322>]  SS:ESP 0068:d5cf3ec4

I disabled MPROTECT on the VirtualBox binary, otherwise PaX simply kills VirtualBox (and is normal as VirtualBox allocates writable and executable memory for the virtual machine).

According to the PaX maintainer, the remap_pfn_range usage is not correct, it should be called implicitly by do_mmap when it goes through the to-be mapped file's f_op->mmap method. Currently, anonymous mapping is faked (i.e., something that is not file-based) and later remap_pfn_range is explicitly called and that is racy. The mmap semaphore is released/reacquired between the mmap and this call, therefore a page fault could come in and instantiate a page that remap_pfn_range would replace, something a PaX BUG check catches.

The normal solution for such cases is to create a pseudo file system and create/map pseudo files from it as needed. There are plenty of examples of this in the kernel.

Also, what is the reason for using a custom kernel module loader? It does not play well with the KERNEXEC feature of PaX.

[Frank Mehnert]

Regarding the incorrect usage of remap_pfn_range() and do_mmap(): We will try to make it the right way. However, I don't see that the mmap semaphore is released+reacquired between those both function calls. And I just checked Linux 2.6.22 to find out under which circumstances do_mmap_pgoff() releases the semaphore itself. It does it indeed but only if MAP_POPULATE was specified as mmap flags (and we don't do this). Of course we want to make our kernel module as compatible to PaX as possible. I'm NOT arguing against your proposal but I want to fully understand if we indeed did it wrong.

Regarding the custom module loader contained in vboxdrv: I think the main reason is that we share quite unique R0 modules between different operating systems (Linux, Windows, MacOS-X, ...).

[TranquilityBase]

Any news on this issue? 1.6.2 is still affected by this one :X

[Pavel Labushev]

You can use VirtualBox with PaX-(semi)enabled kernels, but:

1. CONFIG_PAX_KERNEXEC and CONFIG_PAX_MEMORY_UDEREF must be unset to run VirtualBox.
2. You can't compile VirtualBox' using Hardened Gentoo's GCC 3.4.6 with PIE and SSP support. For now you should use GCC 4.x with no PIE support.
3. CONFIG_PAX_NOELFRELOCS must be unset in working kernel to compile VirtualBox (though you can run VirtualBox with PAX_NOELFRELOCS-enabled kernel).
4. AMD-V/Intel VT-x is not for PaX users for now.

PS And all that is so only if you define "use" as "after a year since the bugs has been well known, my OpenBSD guests still are not usable at all, with or without PaX on the host."

[René Rhéaume]

Replying to Pavel Labushev:

1. CONFIG_PAX_KERNEXEC and CONFIG_PAX_MEMORY_UDEREF must be unset to run VirtualBox.

CONFIG_PAX_KERNEXEC was already disabled, because enabling it makes the mouse pointer stick to the left side of the screen in Xorg. Then, I disabled CONFIG_PAX_MEMORY_UDEREF, even though from what I read in mailing lists UDEREF cause problems in guest kernels, not host kernels.

2. You can't compile VirtualBox' using Hardened Gentoo's GCC 3.4.6 with PIE and SSP support. For now you should use GCC 4.x with no PIE support.

VirtualBox was compiled with the non-hardened 4.1.2 GCC.

3. CONFIG_PAX_NOELFRELOCS must be unset in working kernel to compile VirtualBox (though you can run VirtualBox with PAX_NOELFRELOCS-enabled kernel).

CONFIG_PAX_NOELFRELOCS was unset.

4. AMD-V/Intel VT-x is not for PaX users for now.

AMD-V is disabled in VirtualBox.

With this said, I still get the mmap BUG message and my Windows XP guest does not start.

[Pavel Labushev]

Replying to repzilon:

CONFIG_PAX_KERNEXEC was already disabled, because enabling it makes the mouse pointer stick to the left side of the screen in Xorg.

I use Xorg (no proprietary drivers here) and experience no problems with KERNEXEC. I doubt that KERNEXEC protection by itself cause your mouse pointer bug. Maybe some of your executables and/or shared libs are not ET_DYN binaries, and that's the problem.

Then, I disabled CONFIG_PAX_MEMORY_UDEREF, even though from what I read in mailing lists UDEREF cause problems in guest kernels, not host kernels.

I get "PaX: suspicious general protection fault" with UDEREF immediately after I try to start any VM. And due to how UDEREF works, this behavior is expected.

With this said, I still get the mmap BUG message and my Windows XP guest does not start.

Remove vboxrdv.ko from your kernel modules dir and rebuild virtualbox-modules (I assume your host is Gentoo). And try VirtualBox 1.6.2 instead of 1.5.x. I have no success with 1.5.x (didn't try much though).

[Frank Mehnert]

I could reproduce the freeze if software virtualization is used for a VM (on 2.6.28-hardened-r3). If VT-x is available and used, the VM runs fine. A bug related to hardened Gentoo was just fixed (see #3732).

[Sadako]

I'd love to see this "fixed", as I'm running hardened gentoo on older (ie no AMD-V/VT-x) hardware, and currently boot into a non-hardened kernel when I want/need to start a virtual machine.

So, here are some more info/observations which hopefully may help a bit;

Definitely related to the pax patches, if I "roll my own" kernel including the grsecurity patch from ​http://www.grsecurity.net/test.php, but without the pax patch, there's no issue, however with the pax patch applied the problem occurs, even with ALL pax options disabled within menuconfig.

I do remember it working under hardened kernels prior to 2.6.23, fwiw.

As has already been mentioned, it's only an issue with software virtualization, however it also only appears to occur under 64-bit hosts, with 32-bit hosts the VM boots fine even with software virtualization. From some conversations in #gentoo-hardened on freenode I've picked up that there are some substantial differences between how pax is implemented on 32- and 64-bit x86 hosts, so this isn't a huge surprise.

Lastly, I experience the exact same problem with kqemu, which isn't really surprising either, but I don't think it's been mentioned before.

HTH.

[Frank Mehnert]

I will close this as duplicate of #2652 even if this ticket is older. grsecurity-enabled hosts are not supported by VirtualBox.