VirtualBox 5.2.0: Interrupt storm on emulated HDA controller (FreeBSD) -> fixed in 5.2.10

[wonko1953]

Scenario:

• VirtualBox 5.2.0 compiled from ports on FreeBSD  https://svnweb.freebsd.org/ports/head/emulators/virtualbox-ose/
• patch svn diff -r69153:69154 http://www.virtualbox.org/svn/vbox (iSCSI fix) applied manually
• Both host and client system are FreeBSD 11.1

Result:

• The client boots o.k.
• The client kernel reports an excessive interrupt rate for the HDA audio controller (1k..50k/sec depending on client load) even with no audio playing
• Console output:

  interrupt storm detected on "irq21:"; throttling interrupt source
  interrupt storm detected on "irq21:"; throttling interrupt source
  interrupt storm detected on "irq21:"; throttling interrupt source
  

• Running systat -vm 1 :

        users    Load  0.24  0.20  0.09                  Oct 27 16:12
     Mem usage:   7%Phy  4%Kmem
  Mem: KB    REAL            VIRTUAL                      VN PAGER   SWAP PAGER
          Tot   Share      Tot    Share    Free           in   out     in   out
  Act  162036   14172  1293824    16504 2855288  count
  All  162644   14664  1303256    25644          pages
  Proc:                                                            Interrupts
    r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        ioflt  3318 total
               96       404   13  544 2973   30             cow         atkbd0 1
                                                            zfod        ata0 14
   0.2%Sys   2.6%Intr  0.0%User  0.0%Nice 97.2%Idle         ozfod     2 ata1 15
  |    |    |    |    |    |    |    |    |    |           %ozfod    17 le0 19
  +                                                         daefr       vboxguest0
                                           1 dtbuf          prcfr  2953 hdac0 21
  Namei     Name-cache   Dir-cache     42858 desvn          totfr   228 cpu0:timer
     Calls    hits   %    hits   %      3572 numvn          react   114 cpu3:timer
         3       3 100                  1740 frevn          pdwak     1 cpu2:timer
                                                         40 pdpgs     3 cpu1:timer
  Disks   md0  ada0   cd0 pass0 pass1                       intrn
  KB/t   0.00  0.00  0.00  0.00  0.01                 64464 wire
  tps       0     0     0     0     1                 96496 act
  MB/s   0.00  0.00  0.00  0.00  0.00                 64636 inact
  %busy     0     0     0     0     0                       laund
                                                    2855288 free
  

Expected result:

• No HDA interrupts as long as no audio is playing

[wonko1953]

VBox.Log when using VirtualBox 5.2.0

[wonko1953]

VBox.Log when running the same machine using VirtualBox 5.1.30

[pentagonik]

I've just uploaded a new 5.2 test build which contains more audio fixes. Could you please give these builds a try and report back if these fix the issue for you? You can get the latest 5.2 test builds here: https://www.virtualbox.org/wiki/Testbuilds

[wonko1953]

I just re-tried this using the latest FreeBSD port, virtualbox-ose-5.2.6_2. The behavior is still the same, there are thousands of interrupts per second on hdac0.

Sorry, I might have missed your previous response. Does this test with virtualbox-ose-5.2.6_2 already include the changes in the test build you wanted me to try 8 weeks ago?

[wonko1953]

Diffing a VBox.log from 5.1.30 and from that 5.2.6, there are three additional lines involving HDA:

 HDA: Asynchronous I/O enabled

 HDA: Codec reset

 HDA: Codec reset

So maybe the issue is that on FreeBSD, async I/O with HDA does not work?

[Blofeld]

I have the same problem, and it still occurs with 5.2.8. 5.1.34 is working fine.

[wonko1953]

I have now run openSUSE tumbleweed as client under VBox 5.2.8 on a FreeBSD host in order to get some debug audio using skype in the client.

The audio is very choppy.

I have enabled audio debug acc. to https://www.virtualbox.org/wiki/AudioDebug

The resulting .wav files plus the VBox.log file are attached.

There are many messages like "OSS: Warning: Too big output size (13120 > 256), limiting to 256" in the log file.

[wonko1953]

VBox.log running openSUSE tumbleweed on 5.2.8, with audio debug enabled

[wonko1953]

I just noticed that I cannot upload the corresponding .wav files because of a file size limit of 512k. There are the following files:

% ll *.wav
-rw-r--r--  1 test  wheel  2892280 Mar 18 11:02 CaptureNonInterleaved-0.wav
-rw-r--r--  1 test  wheel  3261088 Mar 18 11:02 DebugAudioOut-0.wav
-rw-r--r--  1 test  wheel  4541392 Mar 18 11:02 hdaDMAReadSD4-0.wav
-rw-r--r--  1 test  wheel  2684576 Mar 18 11:02 hdaDMAWriteSD0-0.wav
-rw-r--r--  1 test  wheel  4541392 Mar 18 11:02 hdaStreamReadSD4-0.wav
-rw-r--r--  1 test  wheel  2754852 Mar 18 11:02 hdaStreamWriteSD0-0.wav
-rw-r--r--  1 test  wheel  4544472 Mar 18 11:02 PlayNonInterleaved-0.wav
-rw-r--r--  1 test  wheel  4544472 Mar 18 11:02 PlayNonInterleaved-2.wav
-rw-r--r--  1 test  wheel  2755028 Mar 18 11:02 StreamRead-0.wav
-rw-r--r--  1 test  wheel  4544472 Mar 18 11:02 StreamWrite-0.wav
-rw-r--r--  1 test  wheel  4544472 Mar 18 11:02 StreamWrite-1.wav
-rw-r--r--  1 test  wheel  4544472 Mar 18 11:02 StreamWrite-2.wav
% 

They all contain the same very choppy sound heard in the emulated OS, so it seems it is a case of

Distorted 	Distorted 	Device emulation problem

according to https://www.virtualbox.org/wiki/AudioDebug (some of them with a very low volume, they seem to be the microphone input fed back from the speakers).

If there is a possibility to upload these files tell me, I'll keep them for a while.

[michaln]

How are comments 6 and 7 relevant to this ticket, which is about an interrupt storm in FreeBSD guests?

The FreeBSD problem should be fixed in the latest 5.2 test build, it was a bug in the HDA emulation (incorrectly reporting an interrupt that the guest OS didn't know how to clear). Probably has zero impact on other guest OSes.

For the sake of completeness: The FreeBSD HDA interrupt handler writes the INTSTS register, which is specified as read-only. It's not clear to us what that is meant to accomplish, although the only harm it causes is slowing things down.

[jkim]

Replying to michaln:

The FreeBSD problem should be fixed in the latest 5.2 test build, it was a bug in the HDA emulation (incorrectly reporting an interrupt that the guest OS didn't know how to clear). Probably has zero impact on other guest OSes.

For the sake of completeness: The FreeBSD HDA interrupt handler writes the INTSTS register, which is specified as read-only. It's not clear to us what that is meant to accomplish, although the only harm it causes is slowing things down.

The original HDA 1.0 Spec. said INTSTS was RW1C instead of RO, it seems. Actually, Google found this:

 http://crashrecovery.org/hdaudio_pdf/hda001-a.pdf

[wonko1953]

Replying to michaln:

How are comments 6 and 7 relevant to this ticket, which is about an interrupt storm in FreeBSD guests?

The FreeBSD problem should be fixed in the latest 5.2 test build, it was a bug in the HDA emulation (incorrectly reporting an interrupt that the guest OS didn't know how to clear). Probably has zero impact on other guest OSes.

For the sake of completeness: The FreeBSD HDA interrupt handler writes the INTSTS register, which is specified as read-only. It's not clear to us what that is meant to accomplish, although the only harm it causes is slowing things down.

Actually, it was me who reported the ticket and who added both these two comments. I simply thought the problems might be related.

But thank you for providing a fix to the original issue, I'm just compiling now and will report how well the latest version works. Also thanks to jkim for quickly updating the FreeBSD port.

[michaln]

Replying to jkim:

The original HDA 1.0 Spec. said INTSTS was RW1C instead of RO, it seems. Actually, Google found this:

 http://crashrecovery.org/hdaudio_pdf/hda001-a.pdf

Ah, nice -- you found an official document showing the INTSTS write done by FreeBSD is redundant and explaining why the code was written that way :) I unfortunately don't know if there might be any hardware implementing the originally specified behavior. Actual Intel chips definitely should not need it, they always documented INTST as R/O in datasheets.

[wonko1953]

The interrupt issue is indeed resolved, thanks for the fix.

And the skype audio quality (as described in comments 6 and 7) is still bad, so indeed this is another issue. This is with a Linux client. However, amarok audio output is o.k., so maybe this is a skype issue.

Interestingly, with a Windows 10 preview client the audio is good.

Interestingly, also with a FreeBSD client the audio is good (only output tested).