VirtualBox

Ticket #3613 (new defect)

Opened 5 years ago

Last modified 3 months ago

High CPU usage with idling guest induced by RTC interrupts

Reported by: Nerijus Owned by:
Priority: major Component: VMM/RAW
Version: VirtualBox 2.1.4 Keywords:
Cc: Guest type: Windows
Host type: other

Description (last modified by frank) (diff)

Bug possibly related to: #1884

Occurs on both Vbox 2.1.4 and 2.2.0 Beta2

With Windows 7 guest, and possibly Windows Vista, Windows XP SP3, Windows Server 2008, Windows Server 2008 R2 guests.

When running:

  • MS SQL server 2008 (express or not),
  • SQL server 2005 SP3
  • TwinCat
  • Google Chrome
  • Or possibly another application which uses 70h interrupts for timing

virtual box uses ~60-70% of single CPU, while guest OS is almost idle.

What was expected to happen: Host should be almost idle too.

Upon further investigation, it appears that these programs use "Real-Time Clock Interrupt", which fires 1024 times per second.

Source:  http://blogs.msdn.com/psssql/archive/2008/12/16/how-it-works-sql-server-no-longer-uses-rdtsc-for-timings-in-sql-2008-and-sql-2005-service-pack-3-sp3.aspx

Problem with SQL server 2008: Forum thread:  http://forums.virtualbox.org/viewtopic.php?f=2&t=15668

Same thing happens with Google Chrome. Bug ticket:  http://www.virtualbox.org/ticket/1884

TwinCat Forum thread:  http://forums.virtualbox.org/viewtopic.php?f=2&t=12941

How problem can be observed with Windows 7 guest (and possibly other guests):

  1. Open "perfmon", and add "Processor->Interupts/sec" counter to monitor.
  2. Start Google Chrome (or another mentioned application), and wait for it to finish initialization.
  3. Watch interrupt count to increase by 1024 per second. This is expected. Guest remains idle.
  4. However, on host, CPU usage increase significantly.

Attachments

VboxLog.log Download (38.1 KB) - added by Nerijus 5 years ago.
VBox.20130711.log Download (104.2 KB) - added by lbgaus 10 months ago.
VBox.log file

Change History

Changed 5 years ago by Nerijus

comment:1 Changed 5 years ago by vbox4me2

Suggestion: assuming some interrupts are emulated, add some form of flood control such done as: VBoxManage setextradata VMNAME "VBoxInternal/Devices/piix3ide/0/Config/IRQDelay" 1

But then something like: VBoxManage setextradata VMNAME "VBoxInternal/Devices/IRQ" 70h 90 Which would mean: return 90% of IRQ 70h calls with last value(cached) and release VM time instead.

comment:2 Changed 5 years ago by Nerijus

I could set this

VBoxManage setextradata VMNAME "VBoxInternal/Devices/piix3ide/0/Config/IRQDelay" 1

But not this: (too many parameters)

VBoxManage setextradata VMNAME "VBoxInternal/Devices/IRQ" 70h 90

I tried to include first parameter in path:

VBoxManage setextradata VMNAME "VBoxInternal/Devices/IRQ/70h" 90

But my vm failed to start completelly, so i removed all these parameters.

Now I am simply not using any programs that couse this problem (except chrome sometimes).

comment:3 Changed 5 years ago by Nerijus

Another program: Google Talk. Increased host CPU usage when user is logged in.

comment:4 Changed 5 years ago by vbox4me2

Nerijus, try READING next time, I clearly stated 'suggestion', which means a suggested way to solve the problem(and others alike), meaning2: it does NOT exist yet...

comment:5 Changed 5 years ago by Nerijus

Problem does not occur when running several virtual machines at once. As soon as one vm is stopped, VirtualBox starts using more CPU again (provided that one of mentioned applications is running in vm).

comment:6 Changed 5 years ago by vbox4me2

Sounds familiar, see also:  http://www.google.nl/search?hl=nl&q=high+cpu+second+vm+site%3Aforums.virtualbox.org&meta= Its been discussed a few times, I suspect the loadbalancing system not being enabled when one VM is running.

comment:7 Changed 5 years ago by Nerijus

Yes, I am running secondary very small VM without boot media and minimum RAM, and it solves problem temporary.

comment:8 Changed 5 years ago by alexk

I can confirm this with VB 2.1.4 running on Ubuntu 8.10 x86_64 host and XP x86 guest.

When the guest is idle, the host uses 30% of the CPU (Intel Core 2 Duo @ 3.2 GHz) As soon as the instance of SQL Server 2008 is stopped, the host's CPU usage drops to less than 5%

comment:9 Changed 5 years ago by hipitihop

I am experiencing 100% load on one of two cpu cores. I have checked and I am not running Chrome or SQL Server. Host is Ubuntu 8.10 64bit with Windows XP SP3 x86 guest migrated from actual machine as VMDK image. I have tried to use the HALu utility (mentioned in another bug or forum thread) to switch the migrated XP to single processor hal, but the switch seems to fail. Allows me to select, asks if I want to reboot, but does not reboot. When VM is restarted, hardware manager shows multy hal again. No idea why, this may be related to guest editions being loaded ?

If there is anything I can run on the host to give you logs or details then let me know

comment:10 Changed 5 years ago by frank

  • Summary changed from High CPU usage with idling guest when 70h interrupts are in use to High CPU usage with idling guest induced by RTC interrupts

comment:11 follow-up: ↓ 12 Changed 5 years ago by VladimirCZ

As proposed on some forums, running concurrently another virtual machine turns the behavior of machine with an idling guest to what one would expect as normal.

E.g. I have created for this purpose a fake machine, with minimal dedicated RAM and boot from the VitualBox addition ISO image. When this machine starts, it hangs of course and I can let be or pause it. This proven mechanism might hopefully lead to the final solution.

comment:12 in reply to: ↑ 11 Changed 5 years ago by alexk

Replying to VladimirCZ:

As proposed on some forums, running concurrently another virtual machine turns the behavior of machine with an idling guest to what one would expect as normal.

E.g. I have created for this purpose a fake machine, with minimal dedicated RAM and boot from the VitualBox addition ISO image. When this machine starts, it hangs of course and I can let be or pause it. This proven mechanism might hopefully lead to the final solution.

Confirming. I created a new empty VM which doesn't even have a bootable OS. Starting it drastically reduces the CPU usage of the original fully-loaded VM.

comment:13 Changed 5 years ago by whatevermon

Will find logs and append when I get a chance, as this is happening to me as well with win7 and a monitoring app, Netcool running constantly.

comment:14 follow-up: ↓ 15 Changed 4 years ago by 1ras

I confirm, too. Starting a second, empty VM which does not even contain an OS fixes the issue.
Old CPU Load when guest is idle: 90%
New CPU load when guest is idle: 7% (plus 2% for the second VM)

comment:15 in reply to: ↑ 14 Changed 4 years ago by vivitron

I can also confirm that starting an empty second VM works... I've just added this to my login file: VBoxManage controlvm "DummyHost" poweroff sleep 1 VBoxHeadless -v off -s "DummyHost" & sleep 1 VBoxManage controlvm "DummyHost" pause

It's odd, DummyHost has 4mb memory, no devices enabled, 1mb display memory, no audio, storage, networking, usb, etc... Even VT-x/AMD-V is disabled... And it still prevents this "bug" from happening.

comment:16 Changed 4 years ago by frank

This is a known behavior. The reason is that the the idle detection works a little bit different if at least two parallel VMs are started. We will fix this problem in a future release.

comment:17 Changed 2 years ago by dlee

This bug is still reproducible as of version 4.1.8. Any updates or new workarounds?

comment:18 Changed 2 years ago by cowmix

4.1.11, still there. Mac OSX Lion is the host OS.

Last edited 2 years ago by cowmix (previous) (diff)

comment:19 Changed 2 years ago by Martin Wildam

Ubuntu 10.04 32-bit Host with VirtualBox 4.1.12 r77245 and Win2008r2 (64-bit) Guest. Looks like Tomcat7 within the guest is creating that (or similar?) issue for me (Don't ask why running a Tomcat on Windows - in that case the program needs it that way).

comment:20 follow-up: ↓ 22 Changed 2 years ago by Martin Wildam

In my case found the following to drastically reduce interrupt activities:

  1. Video memory was set to low. Default for new VirtualBox machine for Windows 2008r2 for example is less than 32 MB although that is the minimum you should use. I am currently using 64 MB
  2. Network Adapters (in the Windows guest) make different problems (done for each adapter):
    1. Disabled IPv6 for all adapters
    2. added local main IP to %WINDIR%\System32\drivers\etc\hosts
    3. Disabled Link-Layer topologies (in my case not needed, but depending on server role might be needed)
    4. Clicking "configure" button in Power management tab disable "Allow the computer to turn off this device to save power".
Last edited 2 years ago by Martin Wildam (previous) (diff)

comment:21 Changed 2 years ago by Martin Wildam

Oh, I also disabled absolute pointing device (under System->Motherboard) in the VB settings - don't know if that made a difference also

comment:22 in reply to: ↑ 20 Changed 2 years ago by Martin Wildam

Replying to Martin Wildam: Further thing changed which seem to help: In MS SQL Server client configuration I forced named pipes protocoll and added aliases for the local computername.

In my case it seems that the main problem is with the network card. Probably everything that I do to reduce the CPU dealing with it helps.

comment:23 follow-up: ↓ 24 Changed 2 years ago by kanaifu

Hi,

I have 11 virtual machines - all of which run fine except those that run MS SQL. Those VM's have 56% interrupts-based CPU activity which severely degrade its performance. Machines with 16GB RAM and 16 CPUs work perfectly as an Exchange cluster, but the same machines with MS SQL on them turn to 4x worse performance.

I will try the previous poster's suggestions, but what i really need is a solution to this interrupt problem.

Have we found out what is the real solution here? Perhaps some one-liner with vboxmanage cli?

comment:24 in reply to: ↑ 23 ; follow-up: ↓ 25 Changed 2 years ago by Martin Wildam

Replying to kanaifu:

I have 11 virtual machines - all of which run fine except those that run MS SQL.

Oh, the tomcat service on my machine is also using the installed SQL Server, so maybe it is more the SQL Server producing the issue than is the tomcat.

In the meantime I am on v4.1.16r78094

Interrupts currently do consume 5-11 % CPU usage, however, performance of the machine as a whole is "acceptable" (for a windows 2008r2 with 2GB RAM).

I tried shutting down the SQL server services if they are the culprit for that interrupt activity but no. Shut down really every service that is possible to shut down - remaining these that can't be shut down:

   DCOM Server Process Launcher
   Group Policy Client
   Plug and Play
   Power
   Remote Procedure Call (RPC)
   RPC Endpoint Mapper
   Security Accounts Manager
   Task Scheduler
   Windows Event Log

Interrupts did not drop further. So if that interrupt level could still be decreased for Windows 2008r2 improving performance, it must be a matter of VirtualBox core. BTW: It did not seem to have any particular relevance for interrupts whether ClientAdditions service was running or not.

In a second step I also compared interrupts with a Windows 2003 machine, host is the same (so same version of virtualbox on Ubuntu 10.04 32-bit Host, Kernel 2.6.32-41-generic-pae): Interrupts signifficant less - from non-existant to 2 % (occassionally).

Third step: Connected via RDP to a Windows 2008r2 production server (running several services including MS SQL Server and virus scanner) - interrupts continously between 0.70 and 1.20 %. That server ASFAIK is a virtual machine running on VMWARE.

So I see pretty much difference with Windows Server 2008r2 on VirtualBox regarding interrupts.

As a last test I gave 128 MB display memory to the VirtualBox machine for a 1-monitor 1440x783 resolution in the VM - did not make any difference.

Note: All my interrupts usage is shown by sysinternals process explorer while in idle mode (me doing nothing but watching). Of course when launching applications interrupts increase above those values.

comment:25 in reply to: ↑ 24 ; follow-up: ↓ 26 Changed 2 years ago by kanaifu

Hello!

Just for the sake of experiment I have converted VDI disk to VMDK, created new (similar) virtual machine in VMware Workstation, attached the disk to that machine, and voila - no more abnormal (>50%) interrupts based CPU activity.

Of course MS SQL is the culprit here, but the real solution has to be found inside VirtualBox core - in it must be possible to handle the huge amount of interrupts arising from MS SQL.

So, basically, this is addressed to the Oracle VirtualBox developers - is this an issue they are working on? Can we rely on VirtualBox for MS SQL based applications in the future versions of VirtualBox?

I would really like to have an immediate solution to this problem, either in form CLI vboxmanage command (preferable), or as a new version of VirtualBox.

Thanks!

Replying to Martin Wildam:

Replying to kanaifu:

I have 11 virtual machines - all of which run fine except those that run MS SQL.

Oh, the tomcat service on my machine is also using the installed SQL Server, so maybe it is more the SQL Server producing the issue than is the tomcat.

In the meantime I am on v4.1.16r78094

Interrupts currently do consume 5-11 % CPU usage, however, performance of the machine as a whole is "acceptable" (for a windows 2008r2 with 2GB RAM).

Last edited 2 years ago by kanaifu (previous) (diff)

comment:26 in reply to: ↑ 25 Changed 10 months ago by lbgaus

Can I offer a little bit of encouragement to the VirtualBox developers to take another look at this issue? It seems very well described/documented in this bugtracker, and has been in here for 4 years. If this problem with interrupts can be fixed, I know it would make a lot of grateful users like me very happy.

Problem still manifests with 4.2.14.

Thanks!

Replying to kanaifu:

Hello!

Just for the sake of experiment I have converted VDI disk to VMDK, created new (similar) virtual machine in VMware Workstation, attached the disk to that machine, and voila - no more abnormal (>50%) interrupts based CPU activity.

Of course MS SQL is the culprit here, but the real solution has to be found inside VirtualBox core - in it must be possible to handle the huge amount of interrupts arising from MS SQL.

So, basically, this is addressed to the Oracle VirtualBox developers - is this an issue they are working on? Can we rely on VirtualBox for MS SQL based applications in the future versions of VirtualBox?

I would really like to have an immediate solution to this problem, either in form CLI vboxmanage command (preferable), or as a new version of VirtualBox.

Thanks!

Replying to Martin Wildam:

Replying to kanaifu:

I have 11 virtual machines - all of which run fine except those that run MS SQL.

Oh, the tomcat service on my machine is also using the installed SQL Server, so maybe it is more the SQL Server producing the issue than is the tomcat.

In the meantime I am on v4.1.16r78094

Interrupts currently do consume 5-11 % CPU usage, however, performance of the machine as a whole is "acceptable" (for a windows 2008r2 with 2GB RAM).

comment:27 Changed 10 months ago by frank

  • Description modified (diff)

I would like to see a VBox.log file for VBox 4.2.14 or even better, for 4.2.16. Thank you!

Changed 10 months ago by lbgaus

VBox.log file

comment:28 Changed 10 months ago by lbgaus

I have attached a VBox.log file. This VM runs 4 virtual processors with 12GB of RAM for Microsoft SQL Server 2008 R2, as soon as the SQL service is started, interrupt CPU time jumps and system performance is pretty degraded - consistent with the above mentioned interrupts CPU% and gets worse the more RAM or vCPUs the VM is given.

I would expect this problem to be easily replicated

Running the exact same VM, disk converted to a vhd file, with the same RAM and vCPU count within Microsoft Hyper-V on the same server works perfectly.

comment:29 Changed 3 months ago by alexd

I would like to point the developers once more to this existing issue - well documented and out there for 5 years now, and still reproducible with the current version 4.3.6. I have Windows 2008 R2 guest + SQL 2008 Express on a Mac OSX Mavericks host and the described problems occur as soon as SQL service is running.

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use