Opened 15 years ago
Closed 14 years ago
#4529 closed defect (fixed)
vboxdrv fails to load on 2.6.31-rc3 (says NMI watchdog is active) => Fixed in SVN
| Reported by: | Robert Hancock | Owned by: | |
|---|---|---|---|
| Component: | host support | Version: | VirtualBox 3.0.2 |
| Keywords: | Cc: | ||
| Guest type: | Windows | Host type: | Linux |
Description
When booting with kernel 2.6.31-rc3 x86_64 on Fedora 11, vboxdrv fails to load claiming the NMI watchdog is active. This occurs even if nmi_watchdog=0 is specified on the kernel command line.
vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line.
Strangely enough, simply attempting to load the driver again often succeeds.
Attachments (6)
Change History (34)
comment:1 by , 15 years ago
comment:3 by , 15 years ago
Confirm. Same problem.
https://bugs.launchpad.net/ubuntu/+source/virtualbox-ose/+bug/399732
Ubuntu 9.10 karmic, Linux 2.6.31-4-generic-pae i686 GNU/Linux
VB: 3.0.2
comment:4 by , 15 years ago
Hmmm, loading by hands - fix this issue.
$ sudo modprobe vboxdrv
[26969.453369] vboxdrv: Successfully done. [26969.453371] vboxdrv: Found 2 processor cores. [26969.453785] vboxdrv: fAsync=0 offMin=0x46a offMax=0xead [26969.453790] Platform driver 'vboxdrv' needs updating - please use dev_pm_ops [26969.453831] vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'. [26969.453833] vboxdrv: Successfully loaded version 3.0.2 (interface 0x000e0000).
After VB running good.
comment:5 by , 15 years ago
Not necessarily. I've seen a similar report where the reporter found out that
for i in `seq 1 20`; do sudo rmmod vboxdrv sudo modprobe vboxdrv done
fails sometimes.
comment:6 by , 15 years ago
srabot@khety:~/dev/autodeploy-trunk$ uname -a Linux khety 2.6.31-3-generic #19-Ubuntu SMP Tue Jul 14 16:04:41 UTC 2009 i686 GNU/Linux
srabot@khety:~/dev/autodeploy-trunk$ cat /boot/grub/menu.lst ... title Ubuntu karmic (development branch), kernel 2.6.31-3-generic root (hd0,0) kernel /boot/vmlinuz-2.6.31-3-generic root=UUID=81162d5d-7691-4918-8730-db6c4b12c9c4 ro quiet splash nmi_watchdog=0 initrd /boot/initrd.img-2.6.31-3-generic quiet ...
srabot@khety:~/dev/autodeploy-trunk$ sudo apt-get install virtualbox-ose Lecture des listes de paquets... Fait Construction de l'arbre des dépendances Lecture des informations d'état... Fait Les paquets supplémentaires suivants seront installés : virtualbox-ose-qt Paquets suggérés : virtualbox-guest-additions Les NOUVEAUX paquets suivants seront installés : virtualbox-ose virtualbox-ose-qt 0 mis à jour, 2 nouvellement installés, 0 à enlever et 40 non mis à jour. Il est nécessaire de prendre 0o/10,0Mo dans les archives. Après cette opération, 36,2Mo d'espace disque supplémentaires seront utilisés. Souhaitez-vous continuer [O/n] ? O Sélection du paquet virtualbox-ose précédemment désélectionné. (Lecture de la base de données... 233132 fichiers et répertoires déjà installés.) Dépaquetage de virtualbox-ose (à partir de .../virtualbox-ose_3.0.2-dfsg-1ubuntu1_i386.deb) ... Sélection du paquet virtualbox-ose-qt précédemment désélectionné. Dépaquetage de virtualbox-ose-qt (à partir de .../virtualbox-ose-qt_3.0.2-dfsg-1ubuntu1_i386.deb) ... Traitement des actions différées (« triggers ») pour « menu »... Traitement des actions différées (« triggers ») pour « desktop-file-utils »... Paramétrage de virtualbox-ose (3.0.2-dfsg-1ubuntu1) ... * Starting VirtualBox kernel modules * modprobe vboxdrv failed. Please use 'dmesg' to find out why Paramétrage de virtualbox-ose-qt (3.0.2-dfsg-1ubuntu1) ... Traitement des actions différées (« triggers ») pour « menu »...
srabot@khety:~/dev/autodeploy-trunk$ for i in `seq 1 20`; do sudo rmmod vboxdrv; sudo modprobe vboxdrv; done ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument ERROR: Module vboxdrv does not exist in /proc/modules FATAL: Error inserting vboxdrv (/lib/modules/2.6.31-3-generic/updates/dkms/vboxdrv.ko): Invalid argument
srabot@khety:~/dev/autodeploy-trunk$ dmesg | tail [67837.867860] vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line. [67837.922625] vboxdrv: Trying to deactivate the NMI watchdog permanently... [67837.922631] vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI [67837.922633] vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line. [67837.964244] vboxdrv: Trying to deactivate the NMI watchdog permanently... [67837.964250] vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI [67837.964252] vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line. [67838.013403] vboxdrv: Trying to deactivate the NMI watchdog permanently... [67838.013409] vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI [67838.013411] vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line.
comment:7 by , 15 years ago
All who experience this issue: Please could you attach the output of dmesg to this defect? So far we were not able to reproduce the problem.
Also, does adding the kernel parameter nmi_watchdog=-1 to the kernel boot command line (in contrast to nmi_watchdog=0 allows the module to be loaded?
comment:8 by , 15 years ago
The problem is most probably related to the new hardware performance counter framework.
by , 15 years ago
comment:9 by , 15 years ago
First attempt to load the module (at boot time) fails but running "/etc/init.d/vboxdrv start" from the command line works fine ...
comment:10 by , 15 years ago
Sorry was out of computer with this version. Probably it is already late (new version was released) but anyway here are 2 files..
by , 15 years ago
| Attachment: | dmesg-3.txt added |
|---|
by , 15 years ago
| Attachment: | dmesg2.txt added |
|---|
comment:12 by , 15 years ago
And back in VB 3.1.0 on kernel 2.6.32 (sidux/Debian Sid). Even manually reloading the module does not work:
# /etc/init.d/vboxdrv start Starting VirtualBox kernel module: modprobe vboxdrv failed. Please use 'dmesg' to find out why ... failed! # dmesg|grep vbox vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line. # ls /lib/modules/2.6.32-0.slh.2-sidux-amd64/misc vboxdrv.ko vboxnetadp.ko vboxnetflt.ko # modprobe vboxdrv FATAL: Error inserting vboxdrv (/lib/modules/2.6.32-0.slh.2-sidux-amd64/misc/vboxdrv.ko): Invalid argument
Booting with or without "nmi_watchdog=0" in Grub kernel command line does not make any difference.
comment:13 by , 15 years ago
I get the same with Fedora 11, VirtualBox-3.1-3.1.0_55467_fedora11-1.i586.rpm, custom kernel 2.6.32
# /etc/init.d/vboxdrv start Starting VirtualBox kernel module [FAILED] (modprobe vboxdrv failed. Please use 'dmesg' to find out why) # dmesg|grep vbox vboxadd: VirtualBox Guest PCI device not found. vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line. vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line. vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line. # ls /lib/modules/2.6.32-Tsa/misc/ vboxdrv.ko vboxnetadp.ko vboxnetflt.ko # modprobe vboxdrv FATAL: Error inserting vboxdrv (/lib/modules/2.6.32-Tsa/misc/vboxdrv.ko): Invalid argument
Booting with or without "nmi_watchdog=0" in Grub kernel command line does not make any difference too.
by , 15 years ago
| Attachment: | kernel-config-2.6.32.i686.Tsa.tgz added |
|---|
comment:17 by , 15 years ago
Hi, I have the same problem on slackware 13.0 and kernel 2.6.32.
1.First step.
./VirtualBox-3.1.0-55467-Linux_x86.run Verifying archive integrity... All good. Uncompressing VirtualBox for Linux installation......... VirtualBox Version 3.1.0 r55467 (2009-11-30T11:25:21Z) installer Removing previous installation of VirtualBox 3.0.12 r54655 from /opt/VirtualBox Installing VirtualBox to /opt/VirtualBox Python found: python, installing bindings... Building the VirtualBox vboxdrv kernel module Building the VirtualBox netflt kernel module Building the VirtualBox netadp kernel module VirtualBox has been installed successfully.
- ./VirtualBox
WARNING: The vboxdrv kernel module is not loaded. Either there is no module available for the current kernel (2.6.32) or it failed to load. Please recompile the kernel module
- /etc/rc.d/rc.vboxdrv setup
Stopping VirtualBox kernel module ...done. Removing old VirtualBox netadp kernel module ...done. Removing old VirtualBox netflt kernel module ...done. Removing old VirtualBox kernel module ...done. Recompiling VirtualBox kernel module ...done. Starting VirtualBox kernel module ...failed! (modprobe vboxdrv failed. Please use 'dmesg' to find out why)
- dmesg
vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line. vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line.
- ls -la /lib/modules/2.6.32/misc/vbox*
-rw-rw-r-- 1 root root 190576 2009-12-10 13:32 /lib/modules/2.6.32/misc/vboxdrv.ko -rw-rw-r-- 1 root root 9153 2009-12-10 13:32 /lib/modules/2.6.32/misc/vboxnetadp.ko -rw-rw-r-- 1 root root 18283 2009-12-10 13:32 /lib/modules/2.6.32/misc/vboxnetflt.ko
6.modprobe vboxdrv
FATAL: Error inserting vboxdrv (/lib/modules/2.6.32/misc/vboxdrv.ko): Invalid argument
tested on virtualbox 3.1 and older 3.0.12
by , 15 years ago
| Attachment: | 2.6.32-0.slh.6-sidux-amd64.config.tar.gz added |
|---|
comment:18 by , 15 years ago
Done - sorry. Will also add a feature request at Trac for auto-scrollable code boxes. Greetings, Chris
comment:19 by , 15 years ago
Just to complete my report - same behaviour with ose and non-ose version of Virtualbox. Greetings, Chris
comment:20 by , 15 years ago
The reason for the re-appearance is that they renamed CONFIG_PERF_COUNTERS to CONFIG_PERF_EVENTS in 2.6.32. The detection heuristics in the vboxdrv module has to be adapted. See r25365 plus r25366. Applying these two changesets to /usr/src/vboxdrv/linux/SUPDrv-linux.c should fix the problem for Linux 2.6.32. The next maintenance release will contain this fix as well.
comment:21 by , 15 years ago
Thanks. You are great. ;) After the changes works.
/etc/rc.d/rc.vboxdrv setup Stopping VirtualBox kernel module ...done. Removing old VirtualBox netadp kernel module ...done. Removing old VirtualBox netflt kernel module ...done. Removing old VirtualBox kernel module ...done. Recompiling VirtualBox kernel module ...done. Starting VirtualBox kernel module ...done.
/etc/rc.d/rc.vboxdrv start Starting VirtualBox kernel module ...done.
dmesg vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: Warning: 2.6.31+ kernel detected. Most likely the hardware performance vboxdrv: counter framework which can generate NMIs is active. You have to prevent vboxdrv: the usage of hardware performance counters by vboxdrv: echo 2 > /proc/sys/kernel/perf_counter_paranoid vboxdrv: Found 1 processor cores. vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'. vboxdrv: Successfully loaded version 3.1.0 (interface 0x00100001).
lsmod Module Size Used by vboxnetadp 5826 0 vboxnetflt 12170 0 vboxdrv 140301 2 vboxnetadp,vboxnetflt
ps aux |grep Vir root 23928 0.0 2.0 76756 43424 pts/10 Sl+ 17:42 0:00 /opt/VirtualBox/VirtualBox root 23947 0.0 0.2 7944 4560 pts/10 S+ 17:42 0:00 /opt/VirtualBox/VBoxXPCOMIPCD root 23956 0.0 0.4 15308 8632 ? Sl 17:42 0:00 /opt/VirtualBox/VBoxSVC --pipe 9 --auto-shutdown
comment:22 by , 15 years ago
Thanks for the patch, I can confirm that it fixes the issue. Greetings, Chris
comment:23 by , 15 years ago
Thanks for the patch, it's ok for me
vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: Warning: 2.6.31+ kernel detected. Most likely the hardware performance vboxdrv: counter framework which can generate NMIs is active. You have to prevent vboxdrv: the usage of hardware performance counters by vboxdrv: echo 2 > /proc/sys/kernel/perf_counter_paranoid vboxdrv: Found 1 processor cores. vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'. vboxdrv: Successfully loaded version 3.1.0 (interface 0x00100001).
comment:24 by , 14 years ago
[ 30.317074] vboxdrv: Trying to deactivate the NMI watchdog permanently... [ 30.317079] vboxdrv: Warning: 2.6.31+ kernel detected. Most likely the hardware performance [ 30.317081] vboxdrv: counter framework which can generate NMIs is active. You have to prevent [ 30.317083] vboxdrv: the usage of hardware performance counters by [ 30.317084] vboxdrv: echo 2 > /proc/sys/kernel/perf_counter_paranoid [ 30.317086] vboxdrv: Found 1 processor cores. [ 30.318989] vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'. [ 30.318993] vboxdrv: Successfully loaded version 3.2.10 (interface 0x00140001).
Still seeing the above on a current Lucid (see below) with VB 3.2.10. What is the benefit of putting 2 in /proc/sys/kernel/perf_event_paranoid (Lucid normally puts in a '1').
Linux version 2.6.32-25-generic (buildd@rothera) (gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #44-Ubuntu SMP Fri Sep 17 20:26:08 UTC 2010 (Ubuntu 2.6.32-25.44-generic 2.6.32.21+drm33.7)
comment:25 by , 14 years ago
| Summary: | vboxdrv fails to load on 2.6.31-rc3 (says NMI watchdog is active) → vboxdrv fails to load on 2.6.31-rc3 (says NMI watchdog is active) => Fixed in SVN |
|---|
This will be finally fixed in the next maintenance release.
comment:26 by , 14 years ago
What is the benefit of putting 2 in /proc/sys/kernel/perf_event_paranoid (Lucid normally puts in a '1')?
What is the "2.6.31+ kernel detected" warning supposed to indicate... some kind of problem with these more current kernels? I've found that Ubuntu Lucid tends to experience hard lock-ups more frequently when vbox is running and it sounds like this vbox bug might somehow be related, or at least it might be a work-around for some known 2.6.31+ kernel issue.
comment:27 by , 14 years ago
Linux kernels older than 2.6.31 allowed to disable the NMI watchdog by a defined interface. This interface didn't work anymore with 2.6.31 and later, therefore some workarounds/hacks were invented to disable the NMI anyway. An NMI is a non-maskable interrupt. If this interrupt happens during the world switch (which is done when switching from the host context to the guest context and back), then the host might freeze or reboot. The NMI is only relevant for NON-VT-x/AMD-V mode (we call it raw mode. Newer Linux kernels use the performance counter + NMI for a lot of things and it is not possible anymore to disable all these features when loading the vboxdrv kernel module. Therefore the next VBox release will contain code which disables+re-enables the NMI during each world switch. This causes more overhead but is the only fix which works reliable.
comment:28 by , 14 years ago
I've got a question related to VBox vs. NMI problem. I've got a server running RHEL4 with IPMI-based watchdog. As one may already know, IPMI-based watchdog generates NMI (nmi_wdt) on the regular intervals and kernel is expected to kill the tasks hanged that had hanged any of CPU cores for more than 5 seconds. Depending on the type of BCM IPMI watchdog either may or may not be programmed to power cycle the system in case there were no watchdog pets for a configured period of time. I'm out of luck as the BCM in my server don't support power-cycling on a watchdog expiration event. To overcome this problem and still use IPMI watchdog to control system hangs I used to disable nmi_watchdog and configure kernel to panic in case of unknown NMI event. Coupled with panic=60 it gives me almost perfect hardware-based watchdog that would reboot system 60 seconds after the expiry of the IPMI watchdog timer.
Now I want to install VBox on this server to virtualize several services and it seems that installing VirtualBOX would mean disabling watchdog and that is not what I would like to do in production environment. So, what is the current status of VBox vs. NMI vs. IPMI watchdog? Are there any hopes that it would be possible to run VirtualBOX on a system with NMI interrupts ticking? If so, what is the VirtualBOX version number this support is expected to come with?
comment:29 by , 14 years ago
| Resolution: | → fixed |
|---|---|
| Status: | new → closed |
VBox version 3.2.12 (just released) disables the NMI only before it does a world switch in non-VT-x/AMD-V mode and re-enables the NMI after coming back from the world switch. Again, this is only relevant for non-VT-x/AMD-V mode. With VT-x/AMD-V we don't need to care about the NMI at all.
Hi, I have same problem.
dmesg | grep watchdog [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-2.6.31-3-generic root=UUID=b086dc0d-0fe7-4487-855b-494761765ca7 ro quiet acpi_osi=Linux nosplash nmi_watchdog=0 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-2.6.31-3-generic root=UUID=b086dc0d-0fe7-4487-855b-494761765ca7 ro quiet acpi_osi=Linux nosplash nmi_watchdog=0 [ 14.459621] vboxdrv: Trying to deactivate the NMI watchdog permanently... [ 14.459625] vboxdrv: NMI watchdog either active or at least initialized. Please disable the NMI [ 14.459626] vboxdrv: watchdog by specifying 'nmi_watchdog=0' at kernel command line.
Look like this bug can be fixed by virtualbox. checking NMI could be done (in linux) by command:
in my case results:
All 0 mean that NMI was initialized (because it present) but it is inactive. This could be fixed by 2 ways: First) Ask developers linux kernel to remove creation NMI records if NMI is inactive. Second) Virtualbox developers could check value and if it is 0 assume than NMI is inactive.
I think Virtualbox now just check presence/absence of record for NMI. Can team to fix it?
related bugs in other sites: https://bugs.launchpad.net/ubuntu/+source/virtualbox-ose/+bug/399732 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/400227