vbox kernel module load fail

[UlfR]

After installing VirutalBox 4.1.6 on Ubuntu 11.10 i received kernel bug during module loading

[   73.675447] vboxdrv: Found 2 processor cores.
[   73.675848] vboxdrv: fAsync=0 offMin=0x20a offMax=0x1b3f
[   73.675934] vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'.
[   73.675937] vboxdrv: Successfully loaded version 4.1.6 (interface 0x00190000).
[   74.066370] BUG: unable to handle kernel paging request at 33986000
[   74.066449] IP: [<f8065101>] VBoxPciLinuxInit+0x101/0x1000 [vboxpci]
[   74.066522] *pde = 00000000 
[   74.066556] Oops: 0002 [#1] SMP 
[   74.066598] Modules linked in: pci_stub vboxpci(+) vboxnetadp vboxnetflt vboxdrv rfcomm bnep parport_pc ppdev binfmt_misc snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device joydev btusb bluetooth snd i915 drm_kms_helper drm hp_wmi psmouse serio_raw arc4 sparse_keymap soundcore iwl3945 iwl_legacy wmi i2c_algo_bit video mac80211 cfg80211 snd_page_alloc lp parport usbhid hid ahci libahci e1000e
[   74.067303] 
[   74.067321] Pid: 1918, comm: modprobe Not tainted 3.0.6-030006-generic #201110050043 Hewlett-Packard HP 550/3618
[   74.067418] EIP: 0060:[<f8065101>] EFLAGS: 00010293 CPU: 1
[   74.067471] EIP is at VBoxPciLinuxInit+0x101/0x1000 [vboxpci]
[   74.067520] EAX: 00000000 EBX: 00000000 ECX: 00000001 EDX: f871468f
[   74.067574] ESI: f8452080 EDI: f8715214 EBP: e9e9bf68 ESP: e9e9bf54
[   74.067628]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[   74.067675] Process modprobe (pid: 1918, ti=e9e9a000 task=e9896580 task.ti=e9e9a000)
[   74.067741] Stack:
[   74.067759]  00000001 f8714686 00002460 f8065000 00000000 e9e9bf88 c100123a c107f63b
[   74.067851]  e9e9bf88 c1081aea 00002460 f8715020 099b2070 e9e9bfac c10847af 00002460
[   74.067944]  e9e9bfac c112abfe ec04d180 099ba270 b7732000 00000000 e9e9a000 c1531adf
[   74.069933] Call Trace:
[   74.069933]  [<f8065000>] ? 0xf8064fff
[   74.069933]  [<c100123a>] do_one_initcall+0xba/0x100
[   74.069933]  [<c107f63b>] ? set_page_attributes+0x1b/0x30
[   74.069933]  [<c1081aea>] ? set_section_ro_nx+0x5a/0x70
[   74.069933]  [<c10847af>] sys_init_module+0xaf/0x200
[   74.069933]  [<c112abfe>] ? sys_close+0x6e/0xc0
[   74.069933]  [<c1531adf>] sysenter_do_call+0x12/0x28
[   74.069933] Code: 00 00 e8 63 b4 ff c8 85 c0 75 5d b8 8f 46 71 f8 e8 45 b4 01 c9 85 c0 89 c6 a3 10 52 71 f8 74 32 83 38 02 74 57 8b 80 78 01 00 00 
[   74.069933]  ff 00 e9 00 00 00 00 eb 14 8b 43 04 b9 fb 50 06 f8 89 f2 ff 
[   74.069933] EIP: [<f8065101>] VBoxPciLinuxInit+0x101/0x1000 [vboxpci] SS:ESP 0068:e9e9bf54
[   74.069933] CR2: 0000000033986000
[   74.139013] ---[ end trace 0796c5dc3ec474a0 ]---

VirtualBox version: 4.1.6-74713~Ubuntu~oneiric

Kernel version: 3.0.6-030006-generic

Ubuntu 11.10

---

After starting VM, there is another bug:

[ 1959.190025] BUG: unable to handle kernel paging request at 33986000
[ 1959.192822] IP: [<f8702768>] vboxNetFltOsInitInstance+0x48/0x110 [vboxnetflt]
[ 1959.193461] *pde = 00000000 
[ 1959.193461] Oops: 0002 [#2] SMP 
[ 1959.193461] Modules linked in: hidp vmnet vsock vmci vmmon pci_stub vboxpci(+) vboxnetadp vboxnetflt vboxdrv rfcomm bnep parport_pc ppdev binfmt_misc snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device joydev btusb bluetooth snd i915 drm_kms_helper drm hp_wmi psmouse serio_raw arc4 sparse_keymap soundcore iwl3945 iwl_legacy wmi i2c_algo_bit video mac80211 cfg80211 snd_page_alloc lp parport usbhid hid ahci libahci e1000e
[ 1959.193461] 
[ 1959.193461] Pid: 3555, comm: VirtualBox Tainted: G      D     3.0.6-030006-generic #201110050043 Hewlett-Packard HP 550/3618
[ 1959.193461] EIP: 0060:[<f8702768>] EFLAGS: 00210293 CPU: 1
[ 1959.193461] EIP is at vboxNetFltOsInitInstance+0x48/0x110 [vboxnetflt]
[ 1959.193461] EAX: 00000000 EBX: e8c97410 ECX: e8702800 EDX: fffff1ee
[ 1959.193461] ESI: e8c97478 EDI: f87061a0 EBP: e990fd64 ESP: e990fd48
[ 1959.193461]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 1959.193461] Process VirtualBox (pid: 3555, ti=e990e000 task=e7a52610 task.ti=e990e000)
[ 1959.193461] Stack:
[ 1959.193461]  f70b3154 e990fd5c c106d070 e8c97410 e8c97410 f87061a8 f87061a0 e990fd84
[ 1959.193461]  f8702c5c f70b3150 f87052ec 00000005 e8772110 00014001 00000002 e990fdf4
[ 1959.193461]  f8c913b4 f87061a8 ebd910e0 e8772110 00000001 e877213c e990fdcc c10e06c1
[ 1959.193461] Call Trace:
[ 1959.193461]  [<c106d070>] ? up+0x30/0x50
[ 1959.193461]  [<f8702c5c>] vboxNetFltFactoryCreateAndConnect+0x20c/0x2c0 [vboxnetflt]
[ 1959.193461]  [<c10e06c1>] ? generic_file_buffered_write+0x51/0x80
[ 1959.193461]  [<c10e2eb4>] ? __generic_file_aio_write+0x284/0x520
[ 1959.193461]  [<f8b72a09>] ? rtR0MemAllocEx+0x89/0xd0 [vboxdrv]
[ 1959.193461]  [<c152a7fd>] ? _raw_spin_lock+0xd/0x10
[ 1959.193461]  [<f8b758a3>] ? VBoxHost_RTSpinlockRelease+0x13/0x20 [vboxdrv]
[ 1959.193461]  [<f8b70342>] ? SUPSemEventCreate+0xa2/0x100 [vboxdrv]
[ 1959.273170]  [<c111b6b5>] ? __kmalloc+0xe5/0x180
[ 1959.273170]  [<f8b6d19f>] ? supdrvIOCtl+0x23f/0x2930 [vboxdrv]
[ 1959.273170]  [<f8b72a09>] ? rtR0MemAllocEx+0x89/0xd0 [vboxdrv]
[ 1959.273170]  [<c128d6e1>] ? _copy_from_user+0x41/0x70
[ 1959.273170]  [<c10c5607>] ? ftrace_traceoff+0x17/0x30
[ 1959.273170]  [<f8b693ab>] ? VBoxDrvLinuxIOCtl_4_1_6+0xfb/0x1c0 [vboxdrv]
[ 1959.273170]  [<c10c5607>] ? ftrace_traceoff+0x17/0x30
[ 1959.273170]  [<f8b692b0>] ? SUPR0Printf+0x70/0x70 [vboxdrv]
[ 1959.273170]  [<c113a94b>] ? vfs_ioctl+0x3b/0x50
[ 1959.273170]  [<c113b580>] ? do_vfs_ioctl+0x70/0x1b0
[ 1959.273170]  [<c113b727>] ? sys_ioctl+0x67/0x70
[ 1959.273170]  [<c1531adf>] ? sysenter_do_call+0x12/0x28
[ 1959.273170]  [<c10c5607>] ? ftrace_traceoff+0x17/0x30
[ 1959.273170] Code: ff 2d d4 c8 ba ee f1 ff ff 85 c0 75 28 0f b6 43 61 84 c0 74 6e 0f b6 43 45 84 c0 75 18 83 3d 20 60 70 f8 02 74 0f a1 98 61 70 f8 
[ 1959.273170]  ff 00 e9 00 00 00 00 31 d2 83 c4 10 89 d0 5b 5e 5f 5d c3 8d 
[ 1959.273170] EIP: [<f8702768>] vboxNetFltOsInitInstance+0x48/0x110 [vboxnetflt] SS:ESP 0068:e990fd48
[ 1959.273170] CR2: 0000000033986000
[ 1959.338496] ---[ end trace 0796c5dc3ec474a1 ]---

and vbox hands

[Frank Mehnert]

Could you attach the vboxpci.ko module to this ticket?

[UlfR]

are any other logs/files needed?

[Frank Mehnert]

Please can you check two things:

1. Make sure that your currently running kernel matches the installed sources in /lib/modules/$(uname -r)/build and that the kernel modules were build against exactly these sources using the very same compiler.
2. What is the output of lsmod | grep pci_stub?

[UlfR]

[ulfrden ~]$uname -a
Linux ulfrden 3.0.6-030006-generic #201110050043 SMP Wed Oct 5 02:07:23 UTC 2011 i686 i686 i386 GNU/Linux

[ulfrden ~]$ls -lah /lib/modules/$(uname -r)/build
lrwxrwxrwx 1 root root 43 2011-10-05 09:38 /lib/modules/3.0.6-030006-generic/build -> /usr/src/linux-headers-3.0.6-030006-generic

[ulfrden ~]$lsmod | grep pci_stub
pci_stub               12550  0 

[ulfrden ~]$modinfo pci_stub
filename:       /lib/modules/3.0.6-030006-generic/kernel/drivers/pci/pci-stub.ko
author:         Chris Wright <chrisw@sous-sol.org>
license:        GPL
srcversion:     4C69260E3BB45AAFEC34BB8
depends:        
vermagic:       3.0.6-030006-generic SMP mod_unload modversions 686 
parm:           ids:Initial PCI IDs to add to the stub driver, format is "vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]" and multiple comma separated entries can be specified (string)

[Frank Mehnert]

Thanks! Oh, one more thing: Could you also attach the corresponding configuration of that kernel? Thank you!

[Frank Mehnert]

Still no really an idea. The mod->refptr seems to be NULL in your case but I don't know why. Do you see this kernel panic always or only in rare cases?

[UlfR]

Always, when VirtualBox service starts

[Frank Mehnert]

Actually this is a bug in the Ubuntu kernel headers. Please check the file /lib/modules/$(uname -r)/build/include/generated/utsrelease.h and compare it with the output of $(uname -r). They differ. Compare also the vermagic string of pci_stub with the vermagic string of vboxpci. They differ exactly because of this wrong header files.

[UlfR]

Checking /lib/modules/$(uname -r)/build/include/generated/utsrelease.h and $(uname -r):

$cat  /lib/modules/$(uname -r)/build/include/generated/utsrelease.h
#define UTS_RELEASE "3.0.6-030006-generic"
$uname -r
3.0.6-030006-generic

They are the same

Checking vermagic strings of pci_stub and vboxpci:

$modinfo pci_stub | grep vermagic
vermagic:       3.0.6-030006-generic SMP mod_unload modversions 686 
$modinfo vboxpci | grep vermagic
vermagic:       3.0.6-030006-generic SMP mod_unload modversions 686

The same also.

[UlfR]

Remain only linux-headers-3.0.6-030006, linux-headers-3.0.6-030006-generic and linux-image-3.0.6-030006-generic packages installed.

ls /lib/modules/ output is '3.0.6-030006-generic', uname -r also

Do dpkg-reconfigure virtualbox-4.1 and receive same kernel oops.

[Frank Mehnert]

Right, the UTS release is correct (my mistake, I did something wrong here). But still, the headers don't match the kernel. Do the following check:

$ readelf -r vboxpci.ko | grep cleanup_module
00000174  00005601 R_386_32          00000000   cleanup_module
$ readelf -r /lib/modules/3.0.6-0360006-generic/kernel/drivers/pci/pci-stub.o | grep cleanup_module
0000016c  00002401 R_386_32          00000000   cleanup_module

This displays the relocation entries of both modules. There is only one relocation entry for cleanup_module() contained in the .rel.gnu.linkonce.this_module section. This section is used by the kernel to load the module and to find certain information, for instance the offset of the module initialization function, the offset of the module cleanup function and some more information. As the patching offset for the field where the cleanup_module() function is stored is different, the both modules are compiled against different kernel headers. The kernel will just not find certain information in the vboxpci module. That is also the reason for the [permanent] mark for all VBox modules in the lsmod listing.

Please understand, this kernel headers package is broken and I already wasted enough time to figure this out.

[basix]

I am reopening this ticket. The reasons and explanation follows:

1. This is a bug on Virtualbox's end as well as Ubuntu's
2. The FIX for this bug for the end user is to recompile Virtualbox's kernel modules in the SAME chroot that Ubuntu's build farm uses to build the mainline kernel - for the current 3.x line it seems they use lucid and lucid uses a different compiler hence this bug shows on natty but not on lucid.
3. The bug from Virtualbox's end is that your kernel module is compiler sensitive and the pre-install / post-install script *should* detect the target system's compiler and if it's different / incompatible, you should fail the installation otherwise this bug will surface. In addition, you should let the user know that VB cannot run on his system reliably.
4. The bug from Ubuntu's end is that they should provide pre-build mainline kernels for each distribution. From talking to folks over at #ubuntu-kernel on freenode this seems that it's an issue of resources.

Suggestions:

1. If Oracle / Virtualbox team can work with the kernel team to get Virtualbox's modules merged into the mainline tree, it'll prevent such headaches for everyone.
2. Please consider donating resources to ubuntu / debian projects so that they can afford more resources to build mainline kernels.

The "FIX":

1. You can find chroot creation instructions here: ​https://help.ubuntu.com/community/DebootstrapChroot
2. sudo apt-get install debootstrap schroot
3. Create a chroot config (refer below for my actual config): sudo emacs -nw /etc/schroot/chroot.d/lucid_x86_64
4. Create the actual chroot: sudo debootstrap --variant=buildd --arch amd64 lucid /srv/chroot/lucid_x86_64 ​http://archive.ubuntu.com/ubuntu
5. enter the chroot: schroot -c lucid_x86_64 -u root
6. Install your kernel headers (note: these might be different for you): dpkg -i linux-headers-3.1.2-030102*deb
7. Install Virtualbox inside the chroot (note: this will 'fail'): dpkg -i virtualbox-4.1_4.1.8-75467~Ubuntu~lucid_amd64.deb
8. Fix the deps: apt-get -f install
9. At this point you should have the modules in /lib/modules/$(uname -r)/misc (they might also be modprobed already - check lsmod)
10. Copy these into your home directory and install them in the actual system's root - /lib/modules/$(uname -r)/misc

Please note for me dkms had those exact modules lying around in /lib/modules/3.1.2-030102-generic/updates/dkms which caused me to believe that this process had failed while infact it had succeeded. Just ensure that you don't have any modules lying around!

The modules that you need to copy are:

1. vboxdrv
2. vboxpci
3. vboxnetflt
4. vboxnetadp

Chroot config:

[lucid_x86_64]
description=Ubuntu Lucid Lynx 64 bit
directory=/srv/chroot/lucid_x86_64
root-users=basix
type=directory
users=basix

In case anybody is interested in the debugging session: ​http://irclogs.ubuntu.com/2011/12/27/%23ubuntu-kernel.html (although it doesn't seem to have the latest chat logs.)

[Frank Mehnert]

You cannot expect that the VirtualBox install script detects that the compiler / Linux headers is different. We tried so in the past and found a lot of issues. The usual way is to use a stock Ubuntu kernel, actually we cannot spend more time supporting strange custom kernel packages.