"Not syncing: An NMI occurred" kernel panic

[Fiorangelo Peluso]

Hi all, we're experiencing several kernel panics with vboxdrv on a brand new HP ProLiant DL560 G8. We currently have Virtualbox 4.3.20 installed with about 15-20 vm running with both Windows and Linux OS. Attached you'll find screenshots of panic errors got from ILO Remote Console. We also tried to reinstall different x86_64 Linux distributions (CentOS 6.6, Oracle Linux 6.6, Red Hat 6.5) and different Virtualbox packages (4.3.10, 4.3.18, 4.3.20) but panics still occurred. The panic event is random and we're unable to reproduce it. It occurs after few days of activity and we would be very afraid to give up on this. Please let us know if any further info is needed. Thanks in advance.

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Kernel Panic Screenshot

[Fiorangelo Peluso]

Guest Kernel (currently 2.6.32-431.el6.x86_64 - Red Hat 6.5) boots with "intel_idle.max_cstate=0 idle=poll" kernel parameters in grub.conf file.

[Fiorangelo Peluso]

Googling about that, we found that a possible solution was to disable nmi_watchdog ("echo 0 > /proc/sys/kernel/nmi_watchdog" command used) since it could be a possible NMI storm issue (this article is quite old but also interesting: ​http://x86vmm.blogspot.it/2005/10/linux-nmis-on-intel-64-bit-hardware.html ).

Next steps:

• use kernel boot parameter "divider=10" since it's suggested in VirtualBox troubleshooting for Red Hat / Fedora guests;
• Disable Hyper Threading

Any help is truly appreciated anyway.

[Klaus Espenlaub]

Normally VBox should disable the NMI watchdog automatically as it interferes with VM execution (guest execution can take quite a while, and during that time the watchdog isn't re-set). No immediate idea why this doesn't work...

[Klaus Espenlaub]

Could it be that you enabled some watchdog in the BIOS? Scratching my head why we don't get many reports like yours. Must be a very special config.

[Fiorangelo Peluso]

I started with default BIOS configuration.. I really wonder if there is any watchdog enabled in BIOS. Anyway now server is working with nmi_watchdog manually disabled. So far so good.. really hope this is the key..

[Frank Mehnert]

Please attach a VBox.log file of such a VM session. What I'm after is the exact CPU specification of your host. Thank you!

[Fiorangelo Peluso]

Which VBox.log should I attach? Panic occurs on guest and I ignore the cause (doesn't seem a single vm to be the cause). Attached you'll find /proc/cpuinfo

Thanks for your feedback

[Fiorangelo Peluso]

/proc/cpuinfo file

[Frank Mehnert]

Please add a VBox.log file from such a VM anyway. It doesn't matter which one.

[Fiorangelo Peluso]

VBox.log files of one vm

[Frank Mehnert]

Thanks you for the log. I had an idea what it could be but looking at the log file my idea was proven wrong. Disabling the NMI watchdog on the host might help like in #13506. Of course that's only a workaround.

[Fiorangelo Peluso]

Thanks for your feedback. Is it safe for the host disabling NMI watchdog?

[Frank Mehnert]

Yes. The watchdog is only there for detecting unusual behavior.

[nuess0r]

I'm having similar issiues with my HP DL360 G5 Server. I got it second hand and installed it in dezember. Before that I was running memtest sucessfully.

Host running Debian 7 Stable, Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1 x86_64 GNU/Linux Guest also running Debian 7 Stable, Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1 x86_64 GNU/Linux

Virtualbox: 4.3.20

In my case I have three VMs which I have migrated from the old system. These VMs are still 32bit and with only one CPU. These three VMs are running absolutely fine, as they did on the old server.

One VM is a newly created one to have more computing power, it has SMP with six CPUs and 24 GByte of Ram configured. This VM crashes often. Sometimes it takes half a day, sometimes it runs for three days.

The symptom is that the guest is not responding to anything. A new VRDP connection keeps black and empty. On the host, six CPUs are running with 100%, the rest of the host is fine and works normal. Also on the host there is no information about the crash of the guest available (VBox.log or syslog).

After the restart of the guest, the entries in the /var/log/messages most of the time look the same, the name of the affected process differs, sometimes the Pid belongs to mysql, sometimes to apache etc.

I attached the messages log from the host and guest (each including boot process), the config of the crashing VM and the Log of the crashing VM.

One thing to add, I don't know if has an effect: Booting Linux on this HP DL360 G5 gives some warning message during boot, that the BIOS is somekind of buggy and I should complain at the vendor.

[nuess0r]

Log file of the crashing guest

[nuess0r]

Log file of the host (DL360 G5)

[nuess0r]

Yesterday I upgraded our server to VirtualBox 4.3.22r98236

Today the problematic VM crashed again, so this bug is still valid.

[nj]

I raised ticket #14034 which seems to bear some similarity to this ticket

[Frank Mehnert]

Actually I'm not sure if this is a VBox bug at all. See ​this Ubuntu ticket. Could you try to blacklist the hpwdt module like suggested there and check if this resolves your problem as well?