VirtualBox

Opened 12 years ago

Last modified 7 years ago

#10611 reopened defect

High CPU usage from interrupts (20% on 8 idle processors)

Reported by: Denis Kozlov Owned by:
Component: other Version: VirtualBox 4.1.16
Keywords: interrupts, CPU, vCPU, processor, high usage Cc:
Guest type: Windows Host type: Linux

Description

The idle guest OS has a high CPU usage from interrupts, consuming around 20-25% of the 8 available virtual processors. When load increases so do the interrupts. If I drop the number of processors then the usage from interrupts drops substantially.

Number of processors with corresponding CPU usage from interrupts:

  • 2 processors = 5%
  • 4 processors = 4%
  • 6 processors = 5%
  • 8 processors = 20-25%

Host: Debian 6.0.5
Guest: Windows 7 SP1
VirtualBox: 4.1.16

Specs of the physical host:
2 x Xeon E5620 @ 2.40GHz, 8 cores in total, 24GB RAM

Attachments (12)

win7-high-interrupts.png (159.8 KB ) - added by Denis Kozlov 12 years ago.
20% cpu usage from interrupts after updates
win7-low-interrupts.png (167.6 KB ) - added by Denis Kozlov 12 years ago.
1% cpu usage from interrupts before updates
VBox.log (107.3 KB ) - added by Denis Kozlov 12 years ago.
VBox log file from test machine with high cpu usage from interrupts
VBox-4.1.22.log (108.2 KB ) - added by Denis Kozlov 12 years ago.
VBox log file from test machine with high cpu usage from interrupts. Upgraded VirtualBox, Extension Pack and Guest Additions to 4.1.22. All latest Windows updates applied.
VBox-4.3.6-old-high.log (118.7 KB ) - added by Denis Kozlov 10 years ago.
VM created in the past which had high CPU usage from interrupts and still has after upgrading to VirtualBox 4.3.6
VBox-4.3.6-old-low.log (116.8 KB ) - added by Denis Kozlov 10 years ago.
VM created in the past which had low CPU usage from interrupts and still has after upgrading to VirtualBox 4.3.6
VBox-4.3.6-new-low.log (115.6 KB ) - added by Denis Kozlov 10 years ago.
A newly created VM with VirtualBox 4.3.6 which has low CPU usage from interrupts
VBox-20150206-2vCPU.log (99.6 KB ) - added by Denis Kozlov 9 years ago.
2 vCPU configuration = 5-6% cpu usage from interrupts
VBox-20150206-6vCPU.log (111.2 KB ) - added by Denis Kozlov 9 years ago.
6 vCPU configuration = 15-20% cpu usage from interrupts
VBox-20150206-8vCPU.log (117.2 KB ) - added by Denis Kozlov 9 years ago.
8 vCPU configuration = 30-60% cpu usage from interrupts
20150206-8vCPU-usage.png (27.0 KB ) - added by Denis Kozlov 9 years ago.
8 vCPU configuration = CPU usage graph, 55% from interrupts
dlahodad2.tar.gz (96.9 KB ) - added by dzmitry.lahoda 7 years ago.

Download all attachments as: .zip

Change History (48)

comment:1 by Denis Kozlov, 12 years ago

I have just installed a guest Ubuntu 12.04 Desktop x64 with same VM settings. The CPU load is around 1.5% on idle which is completely reasonable. So the issue above doesn't affect this OS.

comment:2 by Andre.Ziegler, 12 years ago

you can try to use xperf to trace DPC/interrupt issues. Do you see which driver causes the isues?

comment:3 by Denis Kozlov, 12 years ago

An interesting discovery. A fresh Win 7 SP1 installation does not experience this issue. However, the issue starts to occur once all of the security updates are installed.

I will try xperf to trace the issue and will post back.

by Denis Kozlov, 12 years ago

Attachment: win7-high-interrupts.png added

20% cpu usage from interrupts after updates

by Denis Kozlov, 12 years ago

Attachment: win7-low-interrupts.png added

1% cpu usage from interrupts before updates

comment:4 by Denis Kozlov, 12 years ago

I've installed performance toolkit and captured DPC/interrupts using Xperf tool (attached). Two identical virtual machines were setup, same specs, one is a fresh install and the other one is after the updates.

The number of DPC/interrupts increases after the updates. "hal.dll" seems to be largest contributor.

Any suggestions?

comment:5 by Denis Kozlov, 12 years ago

To clarify, I mapped the symbols and "hal.dll" points to address of "HalpRtcClockInterrupt". Whatever that means...

comment:6 by Andre.Ziegler, 12 years ago

run this

xperf -on latency -stackwalk profile

The stackwalking adds extra data about the callstack (if you got a warning about adding a registry key, do this and reboot). After capturing the data, open it with the viewer again, go into the "CPU sampling per Process" graph, uncheck all processes and activate ISR/DPC under <unknown>. Now load symbols, select an interval with high DPC/ISR usage make a rightclick and select "summary table". Now move the DPC/ISR column to the left to sort the data. Next activate the stack column (click on the rectangle on the left side of the Window). Now look which process causes the high ISR or DPC usage and expand the stack. Which DLL/sys causes the cpu usage? is this a VirtualBox file?

If you're unsure, upload it to your Dropox/Skydrive and post a link here (zip the file first to reduce the size)

comment:7 by Denis Kozlov, 12 years ago

I tried to follow your instructions but there was no ISR/DPC under <unknown>.

Here is the compressed xperf report:

http://dl.dropbox.com/u/46074636/perf.7z

comment:8 by Andre.Ziegler, 12 years ago

hmm, you have a lot of bad drivers:


DPC Info

Total = 61787 Elapsed Time, > 1024 usecs AND <= 2048 usecs, 574, or 0.93% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 153, or 0.25% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 49, or 0.08% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 18, or 0.03% Elapsed Time, > 16384 usecs AND <= 32768 usecs, 1, or 0.00%

Total = 39 for module afd.sys Elapsed Time, > 4096 usecs AND <= 8192 usecs, 1, or 2.56%

Total = 3893 for module ataport.SYS Elapsed Time, > 1024 usecs AND <= 2048 usecs, 88, or 2.26% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 33, or 0.85% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 18, or 0.46% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 5, or 0.13%

Total = 80 for module i8042prt.sys Elapsed Time, > 1024 usecs AND <= 2048 usecs, 3, or 3.75% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 0, or 0.00% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 0, or 0.00% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 1, or 1.25%

So the ataport (IDE driver) and the PS/2 drivers cause some DPC spikes. But the main issue are the interrupts:

Interrupt Info


Elapsed Time, > 1024 usecs AND <= 2048 usecs, 1485, or 0.39% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 221, or 0.06% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 60, or 0.02% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 21, or 0.01% Elapsed Time, > 16384 usecs AND <= 32768 usecs, 4, or 0.00% Total, 378781

4 interrupts take 32ms to finish. this is horrible.

Total = 728 for module USBPORT.SYS Elapsed Time, > 1024 usecs AND <= 2048 usecs, 11, or 1.51% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 3, or 0.41%

Total = 391 for module VBoxGuest.sys Elapsed Time, > 1024 usecs AND <= 2048 usecs, 4, or 1.02% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 5, or 1.28% Total,

Total = 12204 for module ataport.SYS Elapsed Time, > 1024 usecs AND <= 2048 usecs, 255, or 2.09% Elapsed Time, > 2048 usecs AND <= 4096 usecs, 59, or 0.48% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 4, or 0.03% Total, 12204

Total = 365153 for module hal.dll Elapsed Time, > 2048 usecs AND <= 4096 usecs, 154, or 0.04% Elapsed Time, > 4096 usecs AND <= 8192 usecs, 56, or 0.02% Elapsed Time, > 8192 usecs AND <= 16384 usecs, 21, or 0.01% Elapsed Time, > 16384 usecs AND <= 32768 usecs, 4, or 0.00%

So the usport, vboxguest and ATAdriver have large spikes.

I looked the callstack and the hal.dll calls HalpHpetProgramRolloverTimer. But I don't know what the function does. It seems to set some ACPI register. That's why I have an idea. Maybe it is caused by powersaving issues. Change the powerplan in Win7 (http://www.sevenforums.com/tutorials/774-power-plan-select.html) to high performance. Maybe this solves the other driver issues, too. The Linux guest and Win7 powersaving maybe conflict. Try it out.

comment:9 by Denis Kozlov, 12 years ago

No luck so far. I've changed power plan to the "performance" and rebooted, but still high interrupts. I've also tried disabling devices in the VM setings: usb, netword card, sound card. This also had no effect.

The high usage from interrupts definatly starts after the security updates are applied, and doesn't happen with a fresh install. This is reproducable.

I am willing to try whatever it takes to get this fixed. Any suggestions?

in reply to:  9 comment:10 by Andre.Ziegler, 12 years ago

Replying to dezlov:

The high usage from interrupts definatly starts after the security updates are applied, and doesn't happen with a fresh install. This is reproducable.

I am willing to try whatever it takes to get this fixed. Any suggestions?

ok, install only the half of the updates. If the issue doesn't happen, install the agaon 50% of the offered updates, until you figured out which update causes it. Also upload your Vbox.log, may the Developers can see something useful.

comment:11 by Denis Kozlov, 12 years ago

I have tried installing Win 7 on fresh VM about 10 times, with various order of installation of updates and virualbox additions. So much time wasted.

I was certain that installing Additions first and then Updates did cause problems, but not the other way around. In some cases machine hang on shutdown, and was never able to boot getting stuck at the Windows logo. In other cases, after few reboots and all updates installed, the interrupts became high at >20% cpu usage.

After today's discovery I cannot be certain of anything anymore. One instance of VM where Additions first and then Updates were installed booted up fine and no high cpu usage from interrupts. That is a 1 out of 10.

What can developers say about all of this?

The sad truth is that I can no longer see VirtualBox as reliable virtualization solution.

comment:12 by Klaus Espenlaub, 12 years ago

The developers can't say anything, as so far no one has provided the bare minimum information - a VBox.log file. There are lots of not so important details already, so it'd be appreciated if someone could provide the basics.

In general, expecting great performance when running a VM with as many cores as the host isn't really clever. The virtualization overhead is the root cause of te inefficiency in this case, and that's impossible to fix in general.

by Denis Kozlov, 12 years ago

Attachment: VBox.log added

VBox log file from test machine with high cpu usage from interrupts

in reply to:  12 comment:13 by Denis Kozlov, 12 years ago

Replying to klaus:

The developers can't say anything, as so far no one has provided the bare minimum information - a VBox.log file. There are lots of not so important details already, so it'd be appreciated if someone could provide the basics.

In general, expecting great performance when running a VM with as many cores as the host isn't really clever. The virtualization overhead is the root cause of te inefficiency in this case, and that's impossible to fix in general.

I've attached the VBox.log. Perhaps I should've done this earlier, but nobody has requested it specifically.

There is nothing wrong with using all available cores, the same setup works just fine before applying all updates and virtualbox additions. Also, similar setup with Debian as host OS works fine as well.

This isn't about what's clever or not. It is about a problem with hardware/driver compatability as far as I can see. So, klaus, your attitude isn't helping anyone.

P.S. I switched this setup to VMware platform and didn't have a single problem so far.

comment:14 by betaloid, 12 years ago

I have the same problem.
I'm running a Ubuntu 12.04 Desktop 64bit as host, with Virtualbox 4.1.18 (78361).
I have 10 headless VMs of Windows 7 SP1 32bit running.

Before applying all security updates I have low idle CPU.
After updates, idle CPU is around 10-20% for each VM.

Let me know if I can provide any info for solving this issue.

Last edited 12 years ago by betaloid (previous) (diff)

comment:15 by Frank Mehnert, 12 years ago

Any change with VBox 4.1.20?

comment:16 by Denis Kozlov, 12 years ago

No change. Installed 4.1.20 on the host, updated additions, applied latest windows updates - but high CPU usage from interrupts is still there.

comment:17 by Frank Mehnert, 12 years ago

Thanks. I was asking because I saw that your VM config has the HPET enabled and there were several fixes in that regards. Could you attach a VBox.log file for running your guest with VBox 4.1.20?

comment:18 by betaloid, 12 years ago

I can add that we are running on a Xeon E5620 like Frank is.

by Denis Kozlov, 12 years ago

Attachment: VBox-4.1.22.log added

VBox log file from test machine with high cpu usage from interrupts. Upgraded VirtualBox, Extension Pack and Guest Additions to 4.1.22. All latest Windows updates applied.

comment:19 by Denis Kozlov, 12 years ago

Issue still exists in VirtualBox 4.1.22. Also an obsevrastion: cpu load seems to be spread equally between all available cores, according to the top command on the host.

comment:20 by Darth_kurt, 11 years ago

Issue still exists in VirtualBox 4.2.0.
Host: Ubuntu 12.04 Desktop kernel 3.2.32 x86 with pae.
Guest: Windows7 x86.
CPU: Core2 Quad 6600.
All windows updates, Guest Additions(4.2.0) are installed.
CPU usage with 3 Virtual processors ~25%.
CPU usage with 2 Virtual processors ~15%.
Also this effect is cumulative. After some high-CPU-load operations (start&stop services with DB, SVN updates etc.) the CPU usage don't low back and hold about 25% on 2 VP.

comment:21 by sp, 11 years ago

it has nothing to do with the host drivers, the issue exists on windows 7 x64 as well

comment:22 by Frank Mehnert, 10 years ago

Any change with VBox 4.3.2?

comment:23 by Denis Kozlov, 10 years ago

I've tried VirtualBox 4.3.6 and there is a mix of observations.

The VM that previously had high CPU load from interrupts still has it after upgrading host to vbox 4.3.6, installing latest additions and windows updates on the guest.

However, I've tried setting up a fresh copy of Windows 7 SP1 64-bit with same 8 virtual processors configuration. I installed the additions first and then applied all windows updates. The resulting VM does not have high CPU load from interrupts.

I don't know if this is always reproducible since I've tried it only once with the latest VirtualBox, because with older versions I too got a system (occasinally) without high CPU load from interrupts. In all cases no extra software was installed.

by Denis Kozlov, 10 years ago

Attachment: VBox-4.3.6-old-high.log added

VM created in the past which had high CPU usage from interrupts and still has after upgrading to VirtualBox 4.3.6

by Denis Kozlov, 10 years ago

Attachment: VBox-4.3.6-old-low.log added

VM created in the past which had low CPU usage from interrupts and still has after upgrading to VirtualBox 4.3.6

by Denis Kozlov, 10 years ago

Attachment: VBox-4.3.6-new-low.log added

A newly created VM with VirtualBox 4.3.6 which has low CPU usage from interrupts

comment:24 by Denis Kozlov, 9 years ago

This issue still persists and seems to be getting worst! Currently running Windows 7 SP1 (x64) guest on Debian 6.0.10 (x64) headless, using VirtualBox 4.3.14. Getting 20-60% CPU usage from Interrupts while machine is idle or semi-idle (all process use max 10% of CPU).

It no longer looks like it was caused by Windows updates. It seems to be getting steadily worst as the number of background processes increases (even though they are idle).

Example setup: A fresh Windows 7 SP1, with all updates applied, MySQL 5.5 Server, MS Office and few other applications installed. VM is not used, freshly booted, all processes combined use under 10% of CPU, yet, usage from interrupts stays between 20%-30%. Everything is just really sluggish. Start using any applications actively and interrupts usage jumps up to 60%.

This particular VM is configured with 6 processors and 8GB of RAM. Specs of the host haven't changed.

Any progress on this issue at all?

Developers, are you seeing this?

Version 0, edited 9 years ago by Denis Kozlov (next)

comment:25 by Frank Mehnert, 9 years ago

Resolution: invalid
Status: newclosed

Actually you run a VM with 8 guest CPUs on a host which has only 4 physical cores (according to your VBox.log). So what do you expect?

comment:26 by Denis Kozlov, 9 years ago

What are you talking about?

This is 2 x Xeon E5620 @ 2.40GHz, 8 cores in total. HT disabled.

There are 8 physical cores on the server!

comment:27 by Denis Kozlov, 9 years ago

Resolution: invalid
Status: closedreopened

comment:28 by Denis Kozlov, 9 years ago

Just clarify once again. There are 2 physical processors Xeon E5620, each has 4 cores. Together they have 8 physical processing cores.

Last edited 9 years ago by Denis Kozlov (previous) (diff)

comment:29 by Frank Mehnert, 9 years ago

Sorry. Please attach a recent VBox.log file for such a VM session.

by Denis Kozlov, 9 years ago

Attachment: VBox-20150206-2vCPU.log added

2 vCPU configuration = 5-6% cpu usage from interrupts

by Denis Kozlov, 9 years ago

Attachment: VBox-20150206-6vCPU.log added

6 vCPU configuration = 15-20% cpu usage from interrupts

by Denis Kozlov, 9 years ago

Attachment: VBox-20150206-8vCPU.log added

8 vCPU configuration = 30-60% cpu usage from interrupts

by Denis Kozlov, 9 years ago

Attachment: 20150206-8vCPU-usage.png added

8 vCPU configuration = CPU usage graph, 55% from interrupts

comment:30 by Denis Kozlov, 9 years ago

I have attached VBox logs for the same VM running with 2, 6 and 8 vCPU configurations.

Also attached a graph of CPU usage for 8 vCPU configuration, where CPU usage from interrupts varies between 30-60% while VM is sitting idle. This configuration was extremely slow/sluggish. Opening Notepad or Paint takes up to 10 seconds, and CPU usage from interrupts jumps as high as 70%.

This is the only VM running on the host at this moment.

Last edited 9 years ago by Denis Kozlov (previous) (diff)

comment:31 by Denis Kozlov, 9 years ago

I have prepared another similar setup on different host hardware. I was hoping to find clues by the process of elimination.

  • Hardware: 2 x Intel Xeon E5 2643 V2 @ 3.50GHz (12 physical cores, HT disabled), 128GB DDR3 @ 933MHz, Supermicro X9DRW.
  • Host OS: Debian 6.0.10 (64 bit)
  • Guest OS: Windows 7 SP1 (64 bit)
  • VirtualBox: 4.3.20
  • VM configuration: tried 2,4,6,8,10 Processors, 8GB Memory

Similar degradation in performance seems to occur as the number of virtual processors increases.

comment:32 by Denis Kozlov, 9 years ago

Tried another variation of the test configuration above with these differences:

Similar degradation in performance occurs as the number of virtual processors increases.

Running VM with 10-12 virtual CPUs on a host with 12 cores @ 3.50GHz is soo sluggish that it feels like it is running on a host with 1 core @ 1Ghz Pentium style processor.

comment:33 by Denis Kozlov, 9 years ago

Another observation...

The same VM disk image runs fine on other virtualization platforms (XenServer 6.5, VMware ESXi 5.0), with CPU usage from interrupts being virtually zero.

I do understand that VirtualBox might never achieve the same level of performance as "type 1" hypervisors (bare-metal), but it should still be able to use multiple virtual processors without severe degradation in performance.

comment:34 by Denis Kozlov, 9 years ago

Possibly relevant ticket #14075 - Windows VM crashes Debian host, NMI for unknown reason points to "vboxdrv".

by dzmitry.lahoda, 7 years ago

Attachment: dlahodad2.tar.gz added

comment:35 by dzmitry.lahoda, 7 years ago

Have same stuff. Ubuntu 16.04 host Windows 10 guest with VBox 5.1.10 with client additions. See attached dlahodad2 archive.

Hardware (I7-28XX Dell Inspiron laptop): https://github.com/asd-and-Rizzo/asd-and-Rizzo.github.io/blob/master/MyWorkstation.md.

comment:36 by mintylamb, 7 years ago

Just my 2 cents as I've been having this problem for ages (from VBox 4 to current 5.1.12) with Windows Guests on MAC OS X host. In my case I was finally able to make the guest responsive again by adjusting power settings in the Windows Guest to stop it trying to throttle the CPU. I used this article to expose the power settings and set minimum and maximum CPU to 100% http://isboxer.com/wiki/HOWTO:Disable_CPU_Throttling_in_Windows

I have yet to try this VM with and without these settings on other Hyper-visors to confirm this finding as others here have stated guest works on other virtualisation platforms without these changes. I'll post back my findings.

Note: See TracTickets for help on using tickets.

© 2023 Oracle
ContactPrivacy policyTerms of Use