High CPU usage with idling guest induced by RTC interrupts

[Nerijus]

Bug possibly related to: #1884

Occurs on both Vbox 2.1.4 and 2.2.0 Beta2

With Windows 7 guest, and possibly Windows Vista, Windows XP SP3, Windows Server 2008, Windows Server 2008 R2 guests.

When running:

• MS SQL server 2008 (express or not),
• SQL server 2005 SP3
• TwinCat
• Google Chrome
• Or possibly another application which uses 70h interrupts for timing

virtual box uses ~60-70% of single CPU, while guest OS is almost idle.

What was expected to happen: Host should be almost idle too.

Upon further investigation, it appears that these programs use "Real-Time Clock Interrupt", which fires 1024 times per second.

Source: ​http://blogs.msdn.com/psssql/archive/2008/12/16/how-it-works-sql-server-no-longer-uses-rdtsc-for-timings-in-sql-2008-and-sql-2005-service-pack-3-sp3.aspx

Problem with SQL server 2008: Forum thread: ​http://forums.virtualbox.org/viewtopic.php?f=2&t=15668

Same thing happens with Google Chrome. Bug ticket: ​http://www.virtualbox.org/ticket/1884

TwinCat Forum thread: ​http://forums.virtualbox.org/viewtopic.php?f=2&t=12941

How problem can be observed with Windows 7 guest (and possibly other guests):

1. Open "perfmon", and add "Processor->Interupts/sec" counter to monitor.
2. Start Google Chrome (or another mentioned application), and wait for it to finish initialization.
3. Watch interrupt count to increase by 1024 per second. This is expected. Guest remains idle.
4. However, on host, CPU usage increase significantly.

[vbox4me2]

Suggestion: assuming some interrupts are emulated, add some form of flood control such done as: VBoxManage setextradata VMNAME "VBoxInternal/Devices/piix3ide/0/Config/IRQDelay" 1

But then something like: VBoxManage setextradata VMNAME "VBoxInternal/Devices/IRQ" 70h 90 Which would mean: return 90% of IRQ 70h calls with last value(cached) and release VM time instead.

[Nerijus]

I could set this

VBoxManage setextradata VMNAME "VBoxInternal/Devices/piix3ide/0/Config/IRQDelay" 1

But not this: (too many parameters)

VBoxManage setextradata VMNAME "VBoxInternal/Devices/IRQ" 70h 90

I tried to include first parameter in path:

VBoxManage setextradata VMNAME "VBoxInternal/Devices/IRQ/70h" 90

But my vm failed to start completelly, so i removed all these parameters.

Now I am simply not using any programs that couse this problem (except chrome sometimes).

[Nerijus]

Another program: Google Talk. Increased host CPU usage when user is logged in.

[vbox4me2]

Nerijus, try READING next time, I clearly stated 'suggestion', which means a suggested way to solve the problem(and others alike), meaning2: it does NOT exist yet...

[Nerijus]

Problem does not occur when running several virtual machines at once. As soon as one vm is stopped, VirtualBox starts using more CPU again (provided that one of mentioned applications is running in vm).

[vbox4me2]

Sounds familiar, see also: ​http://www.google.nl/search?hl=nl&q=high+cpu+second+vm+site%3Aforums.virtualbox.org&meta= Its been discussed a few times, I suspect the loadbalancing system not being enabled when one VM is running.

[Nerijus]

Yes, I am running secondary very small VM without boot media and minimum RAM, and it solves problem temporary.

[Alexander Kojevnikov]

I can confirm this with VB 2.1.4 running on Ubuntu 8.10 x86_64 host and XP x86 guest.

When the guest is idle, the host uses 30% of the CPU (Intel Core 2 Duo @ 3.2 GHz) As soon as the instance of SQL Server 2008 is stopped, the host's CPU usage drops to less than 5%

[hipitihop]

I am experiencing 100% load on one of two cpu cores. I have checked and I am not running Chrome or SQL Server. Host is Ubuntu 8.10 64bit with Windows XP SP3 x86 guest migrated from actual machine as VMDK image. I have tried to use the HALu utility (mentioned in another bug or forum thread) to switch the migrated XP to single processor hal, but the switch seems to fail. Allows me to select, asks if I want to reboot, but does not reboot. When VM is restarted, hardware manager shows multy hal again. No idea why, this may be related to guest editions being loaded ?

If there is anything I can run on the host to give you logs or details then let me know

[Vladimír Bláha]

As proposed on some forums, running concurrently another virtual machine turns the behavior of machine with an idling guest to what one would expect as normal.

E.g. I have created for this purpose a fake machine, with minimal dedicated RAM and boot from the VitualBox addition ISO image. When this machine starts, it hangs of course and I can let be or pause it. This proven mechanism might hopefully lead to the final solution.

[Alexander Kojevnikov]

Replying to VladimirCZ:

As proposed on some forums, running concurrently another virtual machine turns the behavior of machine with an idling guest to what one would expect as normal.

E.g. I have created for this purpose a fake machine, with minimal dedicated RAM and boot from the VitualBox addition ISO image. When this machine starts, it hangs of course and I can let be or pause it. This proven mechanism might hopefully lead to the final solution.

Confirming. I created a new empty VM which doesn't even have a bootable OS. Starting it drastically reduces the CPU usage of the original fully-loaded VM.

[whatevermon]

Will find logs and append when I get a chance, as this is happening to me as well with win7 and a monitoring app, Netcool running constantly.

[Andreas Stempfhuber]

I confirm, too. Starting a second, empty VM which does not even contain an OS fixes the issue.
Old CPU Load when guest is idle: 90%
New CPU load when guest is idle: 7% (plus 2% for the second VM)

[Dustin Lambert]

I can also confirm that starting an empty second VM works... I've just added this to my login file: VBoxManage controlvm "DummyHost" poweroff sleep 1 VBoxHeadless -v off -s "DummyHost" & sleep 1 VBoxManage controlvm "DummyHost" pause

It's odd, DummyHost has 4mb memory, no devices enabled, 1mb display memory, no audio, storage, networking, usb, etc... Even VT-x/AMD-V is disabled... And it still prevents this "bug" from happening.

[Frank Mehnert]

This is a known behavior. The reason is that the the idle detection works a little bit different if at least two parallel VMs are started. We will fix this problem in a future release.

[dlee]

This bug is still reproducible as of version 4.1.8. Any updates or new workarounds?

[cowmix]

4.1.11, still there.

[Martin Wildam]

Ubuntu 10.04 32-bit Host with VirtualBox 4.1.12 r77245 and Win2008r2 (64-bit) Guest. Looks like Tomcat7 within the guest is creating that (or similar?) issue for me (Don't ask why running a Tomcat on Windows - in that case the program needs it that way).

[Martin Wildam]

In my case found the following to drastically reduce interrupt activities:

1. Video memory was set to low. Default for new VirtualBox machine for Windows 2008r2 for example is less than 32 MB although that is the minimum you should use. I am currently using 64 MB
2. Network Adapters (in the Windows guest) make different problems (done for each adapter):
   1. Disabled IPv6 for all adapters
   2. added local main IP to %WINDIR%\System32\drivers\etc\hosts
   3. Disabled Link-Layer topologies (in my case not needed, but depending on server role might be needed)
   4. Clicking "configure" button in Power management tab disable "Allow the computer to turn off this device to save power".

[Martin Wildam]

Oh, I also disabled absolute pointing device (under System->Motherboard) in the VB settings - don't know if that made a difference also

[Martin Wildam]

Replying to Martin Wildam: Further thing changed which seem to help: In MS SQL Server client configuration I forced named pipes protocoll and added aliases for the local computername.

In my case it seems that the main problem is with the network card. Probably everything that I do to reduce the CPU dealing with it helps.

[kanaifu]

Hi,

I have 11 virtual machines - all of which run fine except those that run MS SQL. Those VM's have 56% interrupts-based CPU activity which severely degrade its performance. Machines with 16GB RAM and 16 CPUs work perfectly as an Exchange cluster, but the same machines with MS SQL on them turn to 4x worse performance.

I will try the previous poster's suggestions, but what i really need is a solution to this interrupt problem.

Have we found out what is the real solution here? Perhaps some one-liner with vboxmanage cli?

[Martin Wildam]

Replying to kanaifu:

I have 11 virtual machines - all of which run fine except those that run MS SQL.

Oh, the tomcat service on my machine is also using the installed SQL Server, so maybe it is more the SQL Server producing the issue than is the tomcat.

In the meantime I am on v4.1.16r78094

Interrupts currently do consume 5-11 % CPU usage, however, performance of the machine as a whole is "acceptable" (for a windows 2008r2 with 2GB RAM).

I tried shutting down the SQL server services if they are the culprit for that interrupt activity but no. Shut down really every service that is possible to shut down - remaining these that can't be shut down:

   DCOM Server Process Launcher
   Group Policy Client
   Plug and Play
   Power
   Remote Procedure Call (RPC)
   RPC Endpoint Mapper
   Security Accounts Manager
   Task Scheduler
   Windows Event Log

Interrupts did not drop further. So if that interrupt level could still be decreased for Windows 2008r2 improving performance, it must be a matter of VirtualBox core. BTW: It did not seem to have any particular relevance for interrupts whether ClientAdditions service was running or not.

In a second step I also compared interrupts with a Windows 2003 machine, host is the same (so same version of virtualbox on Ubuntu 10.04 32-bit Host, Kernel 2.6.32-41-generic-pae): Interrupts signifficant less - from non-existant to 2 % (occassionally).

Third step: Connected via RDP to a Windows 2008r2 production server (running several services including MS SQL Server and virus scanner) - interrupts continously between 0.70 and 1.20 %. That server ASFAIK is a virtual machine running on VMWARE.

So I see pretty much difference with Windows Server 2008r2 on VirtualBox regarding interrupts.

As a last test I gave 128 MB display memory to the VirtualBox machine for a 1-monitor 1440x783 resolution in the VM - did not make any difference.

Note: All my interrupts usage is shown by sysinternals process explorer while in idle mode (me doing nothing but watching). Of course when launching applications interrupts increase above those values.

[kanaifu]

Hello!

Just for the sake of experiment I have converted VDI disk to VMDK, created new (similar) virtual machine in VMware Workstation, attached the disk to that machine, and voila - no more abnormal (>50%) interrupts based CPU activity.

Of course MS SQL is the culprit here, but the real solution has to be found inside VirtualBox core - in it must be possible to handle the huge amount of interrupts arising from MS SQL.

So, basically, this is addressed to the Oracle VirtualBox developers - is this an issue they are working on? Can we rely on VirtualBox for MS SQL based applications in the future versions of VirtualBox?

I would really like to have an immediate solution to this problem, either in form CLI vboxmanage command (preferable), or as a new version of VirtualBox.

Thanks!

Replying to Martin Wildam:

Replying to kanaifu:

I have 11 virtual machines - all of which run fine except those that run MS SQL.

Oh, the tomcat service on my machine is also using the installed SQL Server, so maybe it is more the SQL Server producing the issue than is the tomcat.

In the meantime I am on v4.1.16r78094

Interrupts currently do consume 5-11 % CPU usage, however, performance of the machine as a whole is "acceptable" (for a windows 2008r2 with 2GB RAM).

[lbgaus]

Can I offer a little bit of encouragement to the VirtualBox developers to take another look at this issue? It seems very well described/documented in this bugtracker, and has been in here for 4 years. If this problem with interrupts can be fixed, I know it would make a lot of grateful users like me very happy.

Problem still manifests with 4.2.14.

Thanks!

Replying to kanaifu:

Hello!

Just for the sake of experiment I have converted VDI disk to VMDK, created new (similar) virtual machine in VMware Workstation, attached the disk to that machine, and voila - no more abnormal (>50%) interrupts based CPU activity.

Of course MS SQL is the culprit here, but the real solution has to be found inside VirtualBox core - in it must be possible to handle the huge amount of interrupts arising from MS SQL.

So, basically, this is addressed to the Oracle VirtualBox developers - is this an issue they are working on? Can we rely on VirtualBox for MS SQL based applications in the future versions of VirtualBox?

I would really like to have an immediate solution to this problem, either in form CLI vboxmanage command (preferable), or as a new version of VirtualBox.

Thanks!

Replying to Martin Wildam:

Replying to kanaifu:

I have 11 virtual machines - all of which run fine except those that run MS SQL.

Oh, the tomcat service on my machine is also using the installed SQL Server, so maybe it is more the SQL Server producing the issue than is the tomcat.

In the meantime I am on v4.1.16r78094

Interrupts currently do consume 5-11 % CPU usage, however, performance of the machine as a whole is "acceptable" (for a windows 2008r2 with 2GB RAM).

[Frank Mehnert]

I would like to see a VBox.log file for VBox 4.2.14 or even better, for 4.2.16. Thank you!

[lbgaus]

VBox.log file

[lbgaus]

I have attached a VBox.log file. This VM runs 4 virtual processors with 12GB of RAM for Microsoft SQL Server 2008 R2, as soon as the SQL service is started, interrupt CPU time jumps and system performance is pretty degraded - consistent with the above mentioned interrupts CPU% and gets worse the more RAM or vCPUs the VM is given.

I would expect this problem to be easily replicated

Running the exact same VM, disk converted to a vhd file, with the same RAM and vCPU count within Microsoft Hyper-V on the same server works perfectly.

[alexd]

I would like to point the developers once more to this existing issue - well documented and out there for 5 years now, and still reproducible with the current version 4.3.6. I have Windows 2008 R2 guest + SQL 2008 Express on a Mac OSX Mavericks host and the described problems occur as soon as SQL service is running.

[Kim]

It seems this bug can be fixed by adding "-T 8038" to the start-up parameters of SQL server (I am not sure this works for everyone, but it solved my issues).

• Change the setting using the SQL Server Configuration Manager.
  
• Right-click your "SQL Server (<your name>)" instance under "SQL Server Services" and select properties.
  
• Switch to the Advanced tab and find the "Startup Parameters" setting. Add "-T8038;" to the beginning (no spaces before or after the text as this will lead to problems with ignored settings - mine starts out "-T8038;-dC:\Progra...").
  
• Restart the service and you should be good to go.

More info at: ​http://www.zeninteractions.com/blog/2011/11/08/parallels-virtualbox-using-lots-of-cpu-during-idle/

Tested with VirtualBox 4.3.10 and MSSQL 2008 R2 with the latest updates installed.

[Attila]

After applying the "-T 8038" start-up parameter, the host OS CPU usage went lower, but SQL Server 2012 Express instances were consuming still 30-40% CPU time on the guest machines. Increasing the memory dedicated for the guest machine to 5GB solved the problem, but then I had too low remaining memory for other guests. Finally, the solution was to uninstall VirtualBox Guest Additions on all the guest machines, where SQL Server 2012 Express needed to be running. After this, CPU (and memory) use on the guest machines went nice low. I used VirtualBox 4.3.12 on Windows 7 machine. Guest machines were running Windows Server 2012 R2 with 1-2 instances of SQL Server 2012 Express.

[aeichner]

Please reopen if still relevant with a recent VirtualBox release.