VirtualBox

Ticket #11219 (reopened defect)

Opened 17 months ago

Last modified 3 months ago

OS X Host panic

Reported by: digititus Owned by:
Priority: major Component: other
Version: VirtualBox 4.2.4 Keywords: aio_queue_async_request
Cc: Guest type: Linux
Host type: Mac OS X

Description (last modified by frank) (diff)

I have Virtualbox 4.2.4 installed on OS X server 10.7.5 host. I have one VM running (Ubuntu 12.0.4 LTS), headless mode. I have had the system running for approximately 1 month. I have had 3 panics in this time where I have had to power cycle the host. Console log has many of the following errors:

Nov 19 00:51:25 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 00:57:00 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 01:20:33 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 02:39:51 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 05:47:14 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 06:08:30 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 06:09:06 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 07:07:31 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 07:36:31 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 07:39:01 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Nov 19 07:44:07 mail kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.

I have never seen these errors on this server before and the only new software is VBox. The server has been very stable up until this point. Any help finding the cause of this appreciated.

Change History

comment:1 Changed 17 months ago by frank

  • Description modified (diff)

comment:2 Changed 15 months ago by vanilla2

Bump. FWIW this is flagged as critical error by kernel.crit

I have same issue on 10.7.5 with vbx 4.2.6 r82870 (as of 2013-01-16, mac has all updates, and this is newest vbox). VM is running win7-64 with 2442 MB ram allotted.

12/17/12 1:16:04.000 AM kernel: aio_queue_async_request(): too many in flight for proc: 16.
12/17/12 2:04:47.000 AM kernel: aio_queue_async_request(): too many in flight for proc: 16.
12/12/12 2:39:05.000 AM kernel: aio_queue_async_request(): too many in flight for proc: 16.
...

(about one line every 10 minutes over past 34 or so days, total of 3266 crit notices and counting!)

Jan 16 16:27:39  kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Jan 16 17:07:21  kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
Jan 16 17:27:38  kernel[0]: aio_queue_async_request(): too many in flight for proc: 16
$ sudo grep -i flight /var/log/kernel.log | wc
3266   42458  339664

Tried several reboots, and also updated vbox since 12/12/2012 but kernel.crit messages still filling /var/log/kernel.log


P.S. Sounds like same issue raised here in 2010 for 10.6.2 without any resolutions:  https://forums.virtualbox.org/viewtopic.php?f=8&t=33611

Last edited 15 months ago by frank (previous) (diff)

comment:3 Changed 15 months ago by frank

If you can reproduce this problem reliably, would the tweak as posted  here help?

comment:4 Changed 13 months ago by frank

  • Status changed from new to closed
  • Resolution set to worksforme

No response, closing.

comment:5 Changed 8 months ago by CoreyAtOoyala

I have two instances of 3 Windows VMs on a Mac Mini running 10.7.5 and VirtualBox 4.2.16r86992. Both instances have run into this error. Typically after about several days of operation. I don't have data on the exact timings.

Tried the sysctl tweaks recommended above:

kern.aiomax=512
kern.aioprocmax=64
kern.aiothreads=16

This did not resolve the issue. I am currently trying the following values:

kern.aiomax=512
kern.aioprocmax=128
kern.aiothreads=32

comment:6 Changed 8 months ago by CoreyAtOoyala

The following configurations all eventually result in unresponsive VMs:

  • 10.7.4 with a single windows 7 VM
  • 10.7.5 with two windows 7 VMs
  • 10.7.5 with one windows 7 VM and one Windows XP VM

The failures are all as follows:

  • "aio_queue_async_request(): too many in flight for proc: 16"
  • after several hundred of these warnings/errors disk IO errors are reported. Later filesystem diagnostics do not always find filesystem errors.
  • The windows VMs are mostly unresponsive. They will respond to pings and initial RDP connection negotiation. However, anything involving IO freezes.
  • VBoxManage controlvm commands never complete.

All of the systems have had the HDs pulled and were determined to have working HDs. Regardless, they were replaced. Once rebuilt to the same configuration the failures continued to occur.

The follow sysctl.conf configuration eliminated the aio warnings/errors but the VMs became unresponsive much quicker. On the order of several hours vs several days.

kern.aiomax=2048
kern.aioprocmax=512
kern.aiothreads=128

I replaced Mac 10.7.5 with Ubuntu 12.0.4 on one of the systems. 3 windows 7 VMs were placed on this system and subjected to the same testing load. No errors have occurred.

The following configurations all encountered aio errors but did not have unresponsive VMs:

  • 10.6.5 with single Windows 7 VM.
  • 10.6.5 with three windows XP VMs.

These have been operating fine with

kern.aiomax=2048
kern.aioprocmax=512
kern.aiothreads=128

(previous post corrected for updated data.)

Version 1, edited 7 months ago by CoreyAtOoyala (previous) (next) (diff)

comment:7 Changed 8 months ago by CoreyAtOoyala

  • Status changed from closed to reopened
  • Resolution worksforme deleted

While I cannot provide an exact procedure for reproducing this failure, the failure still exists. Re-opening.

comment:8 Changed 6 months ago by jamiekp

I am having trouble with WIFI failing under OS X 10.9 when running VirtualBox. I started grepping my system logs for any suspicious messages and was lead here by the recurrence of the message mentioned in the report above.

It feels likely to me that this is the same issue but it is having a different effect on my system (s/kernel panic/wifi stops working/). I will try raising some of the limits mentioned to see if I have any luck.

comment:9 Changed 4 months ago by petarb

Adjusting kern.aio* sysctls helped with the panics and the minute-long lags, where kernel_task blocked everything for so long that even some fuse-mounts were "marked dead" and, subsequently, unmounted.

kernel_task is still not happy, though. And this affects everything, sound stutters, Bluetooth devices fail and the most critical issue: immediately after the lag, VMs sometimes receive the most recent keyboard input about 10 times.. which makes using the terminal somewhat risky.

The following Linux kernel messages seem to have their origin in this issue:

[  161.850031] hrtimer: interrupt took 56522379 ns
(...)
[  448.838969] [sched_delayed] sched: RT throttling activated
(...)
[ 4254.411876] atkbd serio0: Spurious NAK on isa0060/serio0. Some program might be trying to access hardware directly.

It's probably worth mentioning that the lags increase the longer a VM is running and that high kern.aio* values lead to weird VM-issues where some processes hang on io-calls for longer than 120 seconds.

P.S. The following post suggests that i7-OSX users don't have this problem:

https://www.virtualbox.org/pipermail/vbox-dev/2011-July/009942.html

P.P.S. Core 2 Duo MacBook with 8GB Ram, Snow Leopard (10.6.8, x86_64) and VirtualBox 4.3.4 (VTx enabled, one i686 and one amd64 Guest)

Last edited 4 months ago by petarb (previous) (diff)

comment:10 Changed 3 months ago by petarb

Enabling Host I/O caching for controllers that have local and file-based disks attached seems to have fixed 99% of the issues described above. Since more than 60 Minutes, one of my Host I/O cache enabled guests has been doing several tasks simultaneously (compiling a kernel, rehashing a large torrent, upgrading the bitcoin blockchain) and during that time I have observed only one system-wide lag (where everything froze for a split second followed by the guest spamming itself with my most recent keystroke). kernel_task is keeping quiet so is the most sensible host process, BluetoothAudioAgent. Without Host I/O caching, the latter would be completely unusable (i.e. the audio output over bluetooth would interrupt every few seconds).

The following line enables the Host I/O cache:

VBoxManage storagectl <vm> --name <controllername> --hostiocache on

In my case it was:

VBoxManage storagectl pintail --name SATA --hostiocache on

This probably also makes all the kern.aio* settings irrelevant.

Further implications of enabling the Host I/O cache are described here:

 http://www.virtualbox.org/manual/ch05.html#iocaching

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use