VirtualBox

Ticket #10611 (new defect)

Opened 23 months ago

Last modified 3 months ago

High CPU usage from interrupts (20% on 8 idle processors)

Reported by: dezlov Owned by:
Priority: critical Component: other
Version: VirtualBox 4.1.16 Keywords: interrupts, CPU, vCPU, processor, high usage
Cc: Guest type: Windows
Host type: Linux

Description

The idle guest OS has a high CPU usage from interrupts, consuming around 20-25% of the 8 available virtual processors. When load increases so do the interrupts. If I drop the number of processors then the usage from interrupts drops substantially.

Number of processors with corresponding CPU usage from interrupts:

  • 2 processors = 5%
  • 4 processors = 4%
  • 6 processors = 5%
  • 8 processors = 20-25%

Host: Debian 6.0.5
Guest: Windows 7 SP1
VirtualBox: 4.1.16

Specs of the physical host:
2 x Xeon E5620 @ 2.40GHz, 8 cores in total, 24GB RAM

Attachments

win7-high-interrupts.png Download (159.8 KB) - added by dezlov 23 months ago.
20% cpu usage from interrupts after updates
win7-low-interrupts.png Download (167.6 KB) - added by dezlov 23 months ago.
1% cpu usage from interrupts before updates
VBox.log Download (107.3 KB) - added by dezlov 21 months ago.
VBox log file from test machine with high cpu usage from interrupts
VBox-4.1.22.log Download (108.2 KB) - added by dezlov 19 months ago.
VBox log file from test machine with high cpu usage from interrupts. Upgraded VirtualBox, Extension Pack and Guest Additions to 4.1.22. All latest Windows updates applied.
VBox-4.3.6-old-high.log Download (118.7 KB) - added by dezlov 3 months ago.
VM created in the past which had high CPU usage from interrupts and still has after upgrading to VirtualBox 4.3.6
VBox-4.3.6-old-low.log Download (116.8 KB) - added by dezlov 3 months ago.
VM created in the past which had low CPU usage from interrupts and still has after upgrading to VirtualBox 4.3.6
VBox-4.3.6-new-low.log Download (115.6 KB) - added by dezlov 3 months ago.
A newly created VM with VirtualBox 4.3.6 which has low CPU usage from interrupts

Change History

comment:1 Changed 23 months ago by dezlov

I have just installed a guest Ubuntu 12.04 Desktop x64 with same VM settings. The CPU load is around 1.5% on idle which is completely reasonable. So the issue above doesn't affect this OS.

comment:2 Changed 23 months ago by Andre.Ziegler

you can try to use xperf to trace DPC/interrupt issues. Do you see which driver causes the isues?

comment:3 Changed 23 months ago by dezlov

An interesting discovery. A fresh Win 7 SP1 installation does not experience this issue. However, the issue starts to occur once all of the security updates are installed.

I will try xperf to trace the issue and will post back.

Changed 23 months ago by dezlov

20% cpu usage from interrupts after updates

Changed 23 months ago by dezlov

1% cpu usage from interrupts before updates

comment:4 Changed 23 months ago by dezlov

I've installed performance toolkit and captured DPC/interrupts using Xperf tool (attached). Two identical virtual machines were setup, same specs, one is a fresh install and the other one is after the updates.

The number of DPC/interrupts increases after the updates. "hal.dll" seems to be largest contributor.

Any suggestions?

comment:5 Changed 23 months ago by dezlov

To clarify, I mapped the symbols and "hal.dll" points to address of "HalpRtcClockInterrupt". Whatever that means...

comment:6 Changed 23 months ago by Andre.Ziegler

run this

xperf -on latency -stackwalk profile

The stackwalking adds extra data about the callstack (if you got a warning about adding a registry key, do this and reboot). After capturing the data, open it with the viewer again, go into the "CPU sampling per Process" graph, uncheck all processes and activate ISR/DPC under <unknown>. Now load symbols, select an interval with high DPC/ISR usage make a rightclick and select "summary table". Now move the DPC/ISR column to the left to sort the data. Next activate the stack column (click on the rectangle on the left side of the Window). Now look which process causes the high ISR or DPC usage and expand the stack. Which DLL/sys causes the cpu usage? is this a VirtualBox file?

If you're unsure, upload it to your Dropox/Skydrive and post a link here (zip the file first to reduce the size)

comment:7 Changed 23 months ago by dezlov

I tried to follow your instructions but there was no ISR/DPC under <unknown>.

Here is the compressed xperf report:

 http://dl.dropbox.com/u/46074636/perf.7z

comment:8 Changed 23 months ago by Andre.Ziegler

hmm, you have a lot of bad drivers:


DPC Info

Total = 61787 Elapsed Time, > 1024 usecs AND <= 2048 usecs, 574, or 0.93% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 153, or 0.25% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 49, or 0.08% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 18, or 0.03% Elapsed Time, > 16384 usecs AND <= 32768 usecs, 1, or 0.00%

Total = 39 for module afd.sys Elapsed Time, > 4096 usecs AND <= 8192 usecs, 1, or 2.56%

Total = 3893 for module ataport.SYS Elapsed Time, > 1024 usecs AND <= 2048 usecs, 88, or 2.26% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 33, or 0.85% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 18, or 0.46% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 5, or 0.13%

Total = 80 for module i8042prt.sys Elapsed Time, > 1024 usecs AND <= 2048 usecs, 3, or 3.75% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 0, or 0.00% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 0, or 0.00% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 1, or 1.25%

So the ataport (IDE driver) and the PS/2 drivers cause some DPC spikes. But the main issue are the interrupts:

Interrupt Info


Elapsed Time, > 1024 usecs AND <= 2048 usecs, 1485, or 0.39% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 221, or 0.06% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 60, or 0.02% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 21, or 0.01% Elapsed Time, > 16384 usecs AND <= 32768 usecs, 4, or 0.00% Total, 378781

4 interrupts take 32ms to finish. this is horrible.

Total = 728 for module USBPORT.SYS Elapsed Time, > 1024 usecs AND <= 2048 usecs, 11, or 1.51% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 3, or 0.41%

Total = 391 for module VBoxGuest.sys Elapsed Time, > 1024 usecs AND <= 2048 usecs, 4, or 1.02% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 5, or 1.28% Total,

Total = 12204 for module ataport.SYS Elapsed Time, > 1024 usecs AND <= 2048 usecs, 255, or 2.09% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 59, or 0.48% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 4, or 0.03% Total, 12204

Total = 365153 for module hal.dll Elapsed Time, > 2048 usecs AND <= 4096 usecs, 154, or 0.04% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 56, or 0.02% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 21, or 0.01% Elapsed Time, > 16384 usecs AND <= 32768 usecs, 4, or 0.00%

So the usport, vboxguest and ATAdriver have large spikes.

I looked the callstack and the hal.dll calls HalpHpetProgramRolloverTimer. But I don't know what the function does. It seems to set some ACPI register. That's why I have an idea. Maybe it is caused by powersaving issues. Change the powerplan in Win7 ( http://www.sevenforums.com/tutorials/774-power-plan-select.html) to high performance. Maybe this solves the other driver issues, too. The Linux guest and Win7 powersaving maybe conflict. Try it out.

comment:9 follow-up: ↓ 10 Changed 23 months ago by dezlov

No luck so far. I've changed power plan to the "performance" and rebooted, but still high interrupts. I've also tried disabling devices in the VM setings: usb, netword card, sound card. This also had no effect.

The high usage from interrupts definatly starts after the security updates are applied, and doesn't happen with a fresh install. This is reproducable.

I am willing to try whatever it takes to get this fixed. Any suggestions?

comment:10 in reply to: ↑ 9 Changed 23 months ago by Andre.Ziegler

Replying to dezlov:

The high usage from interrupts definatly starts after the security updates are applied, and doesn't happen with a fresh install. This is reproducable.

I am willing to try whatever it takes to get this fixed. Any suggestions?

ok, install only the half of the updates. If the issue doesn't happen, install the agaon 50% of the offered updates, until you figured out which update causes it. Also upload your Vbox.log, may the Developers can see something useful.

comment:11 Changed 23 months ago by dezlov

I have tried installing Win 7 on fresh VM about 10 times, with various order of installation of updates and virualbox additions. So much time wasted.

I was certain that installing Additions first and then Updates did cause problems, but not the other way around. In some cases machine hang on shutdown, and was never able to boot getting stuck at the Windows logo. In other cases, after few reboots and all updates installed, the interrupts became high at >20% cpu usage.

After today's discovery I cannot be certain of anything anymore. One instance of VM where Additions first and then Updates were installed booted up fine and no high cpu usage from interrupts. That is a 1 out of 10.

What can developers say about all of this?

The sad truth is that I can no longer see VirtualBox as reliable virtualization solution.

comment:12 follow-up: ↓ 13 Changed 22 months ago by klaus

The developers can't say anything, as so far no one has provided the bare minimum information - a VBox.log file. There are lots of not so important details already, so it'd be appreciated if someone could provide the basics.

In general, expecting great performance when running a VM with as many cores as the host isn't really clever. The virtualization overhead is the root cause of te inefficiency in this case, and that's impossible to fix in general.

Changed 21 months ago by dezlov

VBox log file from test machine with high cpu usage from interrupts

comment:13 in reply to: ↑ 12 Changed 21 months ago by dezlov

Replying to klaus:

The developers can't say anything, as so far no one has provided the bare minimum information - a VBox.log file. There are lots of not so important details already, so it'd be appreciated if someone could provide the basics.

In general, expecting great performance when running a VM with as many cores as the host isn't really clever. The virtualization overhead is the root cause of te inefficiency in this case, and that's impossible to fix in general.

I've attached the VBox.log. Perhaps I should've done this earlier, but nobody has requested it specifically.

There is nothing wrong with using all available cores, the same setup works just fine before applying all updates and virtualbox additions. Also, similar setup with Debian as host OS works fine as well.

This isn't about what's clever or not. It is about a problem with hardware/driver compatability as far as I can see. So, klaus, your attitude isn't helping anyone.

P.S. I switched this setup to VMware platform and didn't have a single problem so far.

comment:14 Changed 20 months ago by betaloid

I have the same problem.
I'm running a Ubuntu 12.04 Desktop 64bit as host, with Virtualbox 4.1.18 (78361).
I have 10 headless VMs of Windows 7 SP1 32bit running.

Before applying all security updates I have low idle CPU.
After updates, idle CPU is around 10-20% for each VM.

Let me know if I can provide any info for solving this issue.

Last edited 20 months ago by betaloid (previous) (diff)

comment:15 Changed 20 months ago by frank

Any change with VBox 4.1.20?

comment:16 Changed 20 months ago by dezlov

No change. Installed 4.1.20 on the host, updated additions, applied latest windows updates - but high CPU usage from interrupts is still there.

comment:17 Changed 20 months ago by frank

Thanks. I was asking because I saw that your VM config has the HPET enabled and there were several fixes in that regards. Could you attach a VBox.log file for running your guest with VBox 4.1.20?

comment:18 Changed 19 months ago by betaloid

I can add that we are running on a Xeon E5620 like Frank is.

Changed 19 months ago by dezlov

VBox log file from test machine with high cpu usage from interrupts. Upgraded VirtualBox, Extension Pack and Guest Additions to 4.1.22. All latest Windows updates applied.

comment:19 Changed 19 months ago by dezlov

Issue still exists in VirtualBox 4.1.22. Also an obsevrastion: cpu load seems to be spread equally between all available cores, according to the top command on the host.

comment:20 Changed 18 months ago by Darth_kurt

Issue still exists in VirtualBox 4.2.0.
Host: Ubuntu 12.04 Desktop kernel 3.2.32 x86 with pae.
Guest: Windows7 x86.
CPU: Core2 Quad 6600.
All windows updates, Guest Additions(4.2.0) are installed.
CPU usage with 3 Virtual processors ~25%.
CPU usage with 2 Virtual processors ~15%.
Also this effect is cumulative. After some high-CPU-load operations (start&stop services with DB, SVN updates etc.) the CPU usage don't low back and hold about 25% on 2 VP.

comment:21 Changed 14 months ago by sp

it has nothing to do with the host drivers, the issue exists on windows 7 x64 as well

comment:22 Changed 5 months ago by frank

Any change with VBox 4.3.2?

comment:23 Changed 3 months ago by dezlov

I've tried VirtualBox 4.3.6 and there is a mix of observations.

The VM that previously had high CPU load from interrupts still has it after upgrading host to vbox 4.3.6, installing latest additions and windows updates on the guest.

However, I've tried setting up a fresh copy of Windows 7 SP1 64-bit with same 8 virtual processors configuration. I installed the additions first and then applied all windows updates. The resulting VM does not have high CPU load from interrupts.

I don't know if this is always reproducible since I've tried it only once with the latest VirtualBox, because with older versions I too got a system (occasinally) without high CPU load from interrupts. In all cases no extra software was installed.

Changed 3 months ago by dezlov

VM created in the past which had high CPU usage from interrupts and still has after upgrading to VirtualBox 4.3.6

Changed 3 months ago by dezlov

VM created in the past which had low CPU usage from interrupts and still has after upgrading to VirtualBox 4.3.6

Changed 3 months ago by dezlov

A newly created VM with VirtualBox 4.3.6 which has low CPU usage from interrupts

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use