Ubuntu 10.04 kernel panic on high network load => Fixed in SVN

[fryckbos]

Setup:

• VirtualBox 4.0.4
• Ubuntu 10.04 host (64 bit)
• 1 Ubuntu 10.04 guest (64 bit)
• Running on a quad core AMD Opteron
• Bridged network, using e1000.

Workload:

• 4 concurrent http wgets to a local server (at gigabit)

Problem:

• No problems when using 1 core
• Kernel panic in guest when using 4 cores

Kernel panic is always in e1000_clean.

[fryckbos]

Forgot to mention that the kernel panic does not occur instantaneously but after a while, it can take up to half an hour.

We are having the same problem when running a web server at high load.

[fryckbos]

Tested this on VirtualBox 4.0.6 and the problem is still there.

[smariel]

Same problem here. Also under "heavy" load. The virtual machine is running a web server with PHP.

FYI the virtual machine got two network adapter attached (one public, one private subnet, the latest turns to be our private VLAN on the host).

I'm attaching the kernel_panic dump, and the VBox.log.

[smariel]

Stack trace of the kernel panic

[smariel]

VBox log file of the virtual machine.

[smariel]

Like fryckbos said earlier, latest VirtualBox 4.0.6 does not solve the problem.

[frank]

Just to be sure: Does the guest kernel panics always within the E1000 driver? At least it looks like because we have two similar kernel logs from two different reporters...

[frank]

An interesting question would be if similar guest kernel crashes also happen with the PCNet device and with the virtio device (Ubuntu should support both devices).

[fryckbos]

The kernel always panics in e1000_clean. We've encountered this at least 10 times.

I will repeat the test with the other 2 network devices.

[fryckbos]

The same test with Virtio is even worse. After 2 minutes the networks becomes flaky

• I can open new ssh sessions, type in the password, but I can't issue any commands.
• The VM doesn't hang fully: I can still login on the console and run commands.
• If I do a ping from within the guest, the first ping returns but the others fail.

The VBox.log file does not show any output, and the guest does not show a kernel panic.

The PCNet test is now running for one hour and is doing fine at the moment.

[frank]

Oh, one more issue: Could both of you attach the binary e1000 module to this ticket? Thanks!

[smariel]

module for Ubuntu 10.04 x86_64 : uname -a : Linux demo 2.6.32-24-server #39-Ubuntu SMP Wed Jul 28 06:21:40 UTC 2010 x86_64 GNU/Linux

[smariel]

Hi Franck,

As of now, there seems to be 2 workarounds :

• use a 1 CPU only virtual machine (as fryckbos said),
• use Am79C973 as NIC type.

Currently our test platform is running with the later option without crashing (let's wait and see, but it should have already crashed according our previous tests).

[fryckbos]

Is it possible that the PCNet NIC goes faster than 100 Mbit ?

[frank]

There is no speed limitation. The e1000 device is usually a bit faster than pcnet but even with pcnet you should be easily able to exceed 100MBit.

[frank]

fryckbos, the kernel module you attached does not fit the kernel log. Your guest kernel is apparently 2.6.35-24, please attach the correct e1000.ko module from this kernel.

[fryckbos]

e1000 driver for the 2.6.35 kernel

[frank]

Ok, the crash happens there at exact the same position.

[fryckbos]

The PCNet device also gave up, but it took about 5 hours. VirtualBox is still running, the guest is still running but network connections fail. I can still ssh, give my password, get a console but I can't type commands.

It seems to be a problem with the clocks: if I do a sleep of 10 seconds at the console, it never returns. The following message is in dmesg:

[ 936.534361] hrtimer: interrupt took 59930437 ns [18529.704149] Clocksource tsc unstable (delta = 4686851418 ns)

[frank]

That might be a different problem. Are there any relevant entries in the log to the pcnet card? Could you attach the complete output of 'dmesg' from the guest and the VBox.log file?

[fryckbos]

Log when using PCNet

[fryckbos]

Console output when using PCNet

[fryckbos]

I attached the VBox and console log, but as far as I know they don't show any errors. I don't know why the hrtime message does not show up in the console log, it was on the dmesg output, but I did not save the output of dmesg.

[frank]

During test it is always advisable to add the kernel parameters console=ttyS0 ignore_loglevel to the kernel command line and to enable a serial port in the VM settings and attach this port to a file. This will log any guest kernel messages to the external file.

[fryckbos]

Thanks, I was doing the logging to file with console=ttyS0 but did not have the ignore_loglevel.

[fryckbos]

Should I do another test with the appropriate logging ?

[frank]

Not sure if that will help. We are currently working on finding the e1000 issue.

[frank]

We found a potential problem in the E1000 implementation. We are currently fixing this issue and will provide you a test build when the fix is done.

[frank]

Or a patch as you seem to compile VBox yourself.

[fryckbos]

A patch would do fine.

[aleksey]

Patch for panic in SMP Linux kernels

[aleksey]

I've attached a tiny patch that should fix the issue. In short words it prevents context descriptors from being written back with DD bit set which caused trouble in TX cleanup function of e1000 driver because context descriptors do not have skb associated with them. This happened in SMP guests since they can add TX descriptors and do cleanup simultaneously. Could you check if the patch indeed solves the problem for you?

[rgcosma]

I will add a "mee too" because I see what seems to be the same problem on Scientific Linux 6 which has the same kernel - 2.6.32 64-bit, host is Win XP 32 bit.

[rgcosma]

SL6 crash with e1000 adapter

[fryckbos]

The tiny patch posted above fixes the problem ! Thanks a lot !

[frank]

rgcosma, could you check if  this build fixes the problem for you as well?

[rgcosma]

installed, running, transferring - didn't crash yet so I guess yes.

[frank]

Great, thanks for the feedback! The fix will be also available in the next maintenance release.

[frank]

Fixed in VBox 4.0.8.