Freeze on VM start with gentoo hardened-sources-2.6.23 as host kernel

[René Rhéaume]

When I try to start a virtual machine in VirtualBox 1.5.2 OSE and the host kernel is Gentoo hardened-sources-2.6.23 (any revision), the virtual machine window freezes and I find the following in dmesg :

kernel BUG at mm/mmap.c:1695!
invalid opcode: 0000 [#1]
PREEMPT SMP
Modules linked in: bttv video_buf firmware_class ir_common
compat_ioctl32 i2c_algo_bit btcx_risc tveeprom videodev v4l2_common
v4l1_compat ipv6 af_packet nf_conntrack_ipv4 xt_state nf_conntrack
nfnetlink xt_limit xt_tcpudp iptable_filter ip_tables x_tables
nls_iso8859_1 nls_cp850 vfat fat xfs usb_storage snd_hda_intel snd_pcm
snd_timer snd soundcore snd_page_alloc ohci1394 ieee1394 ohci_hcd
i2c_nforce2 forcedeth ehci_hcd amd74xx amd64_agp agpgart 8139too mii
cpufreq_conservative powernow_k8 freq_table processor vboxdrv usbhid
hid usbcore ff_memless ide_scsi
CPU:    0
EIP:    0060:[<00069322>]    Not tainted VLI
EFLAGS: 00210206   (2.6.23-hardened-r2 #2)
eax: 00000000   ebx: 0001a000   ecx: d5cdb4d0   edx: d5cdb528
esi: 4bea5000   edi: 00084400   ebp: 00000000   esp: d5cf3ec4
ds: 0068   es: 0068   fs: 00d8  gs: 0033  ss: 0068
Process VirtualBox (pid: 5865, ti=d5cf2000 task=d5cf0000 task.ti=d5cf2000)
Stack: d5cdb4d0 d5cdb3c8 d5cdb4d0 0006a5c0 0000025a 083b9200 4be24000 d5c8ec00
      00000000 d5cf3f40 00000000 4bea5000 4be24000 d5c8ec00 00000000 0006b2ea
      4bea5000 00100073 00000000 00000000 0004be24 00000000 00081000 ac98bab0
Call Trace:
 [<0006a5c0>] <0> [<0006b2ea>] <0> [<00100073>] <0> [<0004be24>] <0>
[<00081000>] <0> [<00081000>] <0> [<0006babe>] <0> [<00100073>] <0>
[<0004be24>] <0> [<0000acac>] <0> [<0004be24>] <0> [<00005532>] <0>
[<00081000>] <0> [<00200256>] <0> [<001385ff>] <0> [<00138600>] <0>
=======================
Code: 39 c3 75 2b 8b 41 44 3b 42 44 74 0c 0f 0b eb fe 0f 0b eb fe 0f
0b eb fe 8b 41 3c 3b 42 3c 75 ec 33 7a 14 81 e7 dd df ef df 74 9e <0f>
0b eb fe 0f 0b eb fe 0f 0b eb fe 66 90 83 ec 14 89 74 24 0c
EIP: [<00069322>]  SS:ESP 0068:d5cf3ec4

I disabled MPROTECT on the VirtualBox binary, otherwise PaX simply kills VirtualBox (and is normal as VirtualBox allocates writable and executable memory for the virtual machine).

According to the PaX maintainer, the remap_pfn_range usage is not correct, it should be called implicitly by do_mmap when it goes through the to-be mapped file's f_op->mmap method. Currently, anonymous mapping is faked (i.e., something that is not file-based) and later remap_pfn_range is explicitly called and that is racy. The mmap semaphore is released/reacquired between the mmap and this call, therefore a page fault could come in and instantiate a page that remap_pfn_range would replace, something a PaX BUG check catches.

The normal solution for such cases is to create a pseudo file system and create/map pseudo files from it as needed. There are plenty of examples of this in the kernel.

Also, what is the reason for using a custom kernel module loader? It does not play well with the KERNEXEC feature of PaX.