Kernel NULL pointer dereference in vboxnetadp on 2.6.35

[Cedric Vivier]

It seems VirtualBox cannot laod vboxnetadp module on 2.6.35 due to a kernel NULL poitner dereference.

Logs below :

vboxdrv: Trying to deactivate the NMI watchdog permanently...
vboxdrv: Warning: 2.6.31+ kernel detected. Most likely the hardware performance
vboxdrv: counter framework which can generate NMIs is active. You have to prevent
vboxdrv: the usage of hardware performance counters by
vboxdrv:   echo 2 > /proc/sys/kernel/perf_counter_paranoid
vboxdrv: Found 4 processor cores.
VBoxDrv: dbg - g_abExecMemory=ffffffffa0042240
vboxdrv: fAsync=0 offMin=0x278 offMax=0x236d
vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'.
vboxdrv: Successfully loaded version 3.2.8 (interface 0x00140001).
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffffa000d1ae>] vboxNetAdpOsCreate+0x3c/0x82 [vboxnetadp]
PGD 83b13067 PUD 5ec50067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/module/vboxdrv/initstate
CPU 0 
Modules linked in: vboxnetadp(+) vboxnetflt vboxdrv uvcvideo videodev v4l1_compat v4l2_compat_ioctl32 [last unloaded: scsi_wait_scan]

Pid: 8522, comm: modprobe Tainted: G        W   2.6.35+ #19 U30Jc/U30Jc
RIP: 0010:[<ffffffffa000d1ae>]  [<ffffffffa000d1ae>] vboxNetAdpOsCreate+0x3c/0x82 [vboxnetadp]
RSP: 0018:ffff88005ee0dea8  EFLAGS: 00010286
RAX: 0000000000000000 RBX: ffffffffa000d9f0 RCX: 0000000000000000
RDX: 000000000027000a RSI: ffffffffa000d65e RDI: ffff880142d56800
RBP: ffff88005ee0dec8 R08: ffff880001e10518 R09: ffff880142d56a38
R10: 0000000000000000 R11: 0000000000013080 R12: ffff880142d56800
R13: 00000000023764d0 R14: 0000000000000000 R15: 0000000002377430
FS:  00007fef14d94700(0000) GS:ffff880001e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000005ec52000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process modprobe (pid: 8522, threadinfo ffff88005ee0c000, task ffff880142dac500)
Stack:
 ffffffffa000d9f0 ffff88005ee0df00 000000000237e210 ffffffffa000d34f
<0> 000000000027000a ffffffff8104f5fe 0000000000000000 ffffffffa000dd30
<0> 0000000000000008 ffffffffa000d3b2 ffffffff81a54390 000000000237e210
Call Trace:
 [<ffffffffa000d34f>] ? vboxNetAdpCreate+0x50/0x7e [vboxnetadp]
 [<ffffffff8104f5fe>] notifier_call_chain+0x2e/0x5b
 [<ffffffffa000d3b2>] ? vboxNetAdpInit+0x35/0x3f [vboxnetadp]
 [<ffffffffa01e5013>] ? VBoxNetAdpLinuxInit+0x13/0xb1 [vboxnetadp]
 [<ffffffffa01e5000>] ? VBoxNetAdpLinuxInit+0x0/0xb1 [vboxnetadp]
 [<ffffffff810001e0>] ? do_one_initcall+0x4f/0x13e
 [<ffffffff8105feaf>] ? sys_init_module+0x97/0x1d4
 [<ffffffff81001fab>] ? system_call_fastpath+0x16/0x1b
Code: 89 f5 48 89 fb 48 c7 c6 54 d6 00 a0 bf b8 00 00 00 e8 30 d9 3d e1 49 89 c4 48 85 c0 74 4c 8b 55 00 48 8b 80 58 02 00 00 4c 89 e7 <89> 10 66 8b 55 04 66 89 50 04 e8 c3 d8 3d e1 89 c5 85 c0 75 17 
RIP  [<ffffffffa000d1ae>] vboxNetAdpOsCreate+0x3c/0x82 [vboxnetadp]
 RSP <ffff88005ee0dea8>
CR2: 0000000000000000
---[ end trace 6a1e41a6c0839587 ]---

[Frank Mehnert]

Unable to reproduce. Did you use a self-compiled 2.6.35 kernel? Please could you attach your vboxnetadp.ko module which fits the panic you posted?

[Cedric Vivier]

Sure, uploaded above.

Yes, self-compiled 2.6.35 kernel, with zcache patches (not sure it is related as iirc it did same thing with stock 2.6.35).

Interestingly, though there is this NULL dereference at some point, VirtualBox runs fine and guests have network access.

[Cedric Vivier]

(all of this meaning the bug priority can surely be lowered, but I can't do it myself it seems ;)

[Frank Mehnert]

Thanks for the additional information. For the record: It crashes for you at VBoxNetAdp-linux.c line 179, pNetDev->dev_addr is NULL.

[Aleksey Ilyushin]

Could you please apply the attached patch and reproduce the problem? Judging from the kernel source code dev_addr cannot be null when alloc_netdev returns non-null pointer. The patch merely prints the return code along with dev_addr pointer at initialization.

[Cedric Vivier]

Sure :)

~$ dmesg | grep -C 2 dev_addr_init
reserve RAM buffer: 000000000009bc00 - 000000000009ffff 
reserve RAM buffer: 00000000aace0000 - 00000000abffffff 
dev_addr_init: return 0 (dev->dev_addr=ffff8801470c1a90).
cfg80211: Calling CRDA to update world regulatory domain
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
--
atl1c 0000:04:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
atl1c 0000:04:00.0: setting latency timer to 64
dev_addr_init: return 0 (dev->dev_addr=ffff880145ef18b0).
atl1c 0000:04:00.0: version 1.0.0.2-NAPI
PPP generic driver version 2.4.2
--
ath: Regpair used: 0x60
phy0: Selected rate control algorithm 'ath9k_rate_control'
dev_addr_init: return 0 (dev->dev_addr=ffff880145ef1c10).
Registered led device: ath9k-phy0::radio
Registered led device: ath9k-phy0::assoc
--
sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
IPv4 over IPv4 tunneling driver
dev_addr_init: return 0 (dev->dev_addr=ffff880144cd5850).
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
--
tunl0: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
dev_addr_init: return 0 (dev->dev_addr=ffff880144cd5eb0).
sit0: Disabled Privacy Extensions
NET: Registered protocol family 17
--
vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'.
vboxdrv: Successfully loaded version 3.2.8 (interface 0x00140001).
dev_addr_init: return 0 (dev->dev_addr=ffff880143eb2190).
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffffa01ee1ae>] vboxNetAdpOsCreate+0x3c/0x82 [vboxnetadp]

[Frank Mehnert]

That looks more and more like a bug in your current Linux kernel. Have a look at net/core/dev.c in the function alloc_netdev_mq(). dev_addr_init() is called from there and when this function returns, dev->dev_addr is definitely not NULL. So somewhere after this call but before alloc_netdev_mq() returns, that field is overwritten. By adding more of such printks from the patch (make sure to change the text that you can identify the location in the log) you could find out when this field is cleared.

[Cedric Vivier]

I rebased against 2.6.35.1, still with zcache patches applied, and the problem is gone. Feel free to close the bug.

Thanks for the help guys!

[Frank Mehnert]

Thanks for the feedback.