VirtualBox

Opened 15 years ago

Closed 10 years ago

#4392 closed defect (fixed)

Huge IO-APIC/guest SMP overhead with 32 bits guests

Reported by: rafael Owned by:
Component: VMM/HWACCM Version: VirtualBox 3.0.6
Keywords: Cc:
Guest type: other Host type: other

Description (last modified by Frank Mehnert)

Symptoms are high CPU usage while running e.g. 32 bits Windows XP/2000 guests with IO-APIC enabled.

Current status:

  • fixed for AMD-V in VirtualBox 3.0.6 (requires 3.0.6 guest additions to be installed)
  • fixed for VT-x, but only if the Intel CPU supports 64 bits (leaves out Core Solo/Duo); fixed in 3.0.12 (requires 3.0.12 guest additions to be installed)
  • Solaris/OpenSolaris guests have similar issues, but a kernel fix is underway.

Does not apply to:

  • 64 bits guests
  • 32 bits Windows Vista or Windows 7

Attachments (8)

VBox.log (41.6 KB ) - added by George Greer 15 years ago.
Windows 2000 SMP under Ubuntu 9.0.4 64-bit VT-x
VBox-WinXP-JeRa.log (57.5 KB ) - added by J.E. Raaijmakers 14 years ago.
VB log of SMP WinXP (32-bit) running on Kubuntu 9.10 (64-bit) experiencing tremendous lagging and performance degradation
56 MST 2009 (46.9 KB ) - added by Brian J. Cohen 14 years ago.
VBox.2.log (53.4 KB ) - added by Mark Cranness 14 years ago.
SMP Windows 2000 Server
VBox.3.log (64.0 KB ) - added by Marc Steffens 14 years ago.
VBox.4.log (47.9 KB ) - added by Michael Kromer 14 years ago.
VBox.log with bad 2003 r2 32-bit performance
VBox.5.log (53.9 KB ) - added by hansb 14 years ago.
VBox.bernat.log (61.7 KB ) - added by Vincent Bernat 14 years ago.
VBOX.log with 100% CPU usage, VT-X enabled, on a QuadCore

Download all attachments as: .zip

Change History (58)

comment:1 by rafael, 15 years ago

noticed the following, if I disable the IOAPIC and set VCPU to 1, install goes fine and fast ...

comment:2 by Sander van Leeuwen, 15 years ago

Summary: windows xp VM extremely slow when AMD-v is enable with 2 cpuhuge IO-APIC/guest SMP overhead with 32 bits guests (AMD-V only)

Known problem and mentioned in the manual. 32 bits guests are extremely slow with the IO-APIC enabled on AMD cpus. Will be fixed in a future release.

comment:3 by mike.mcse, 15 years ago

I can confirm this behavior on 3.0.2 on AMD Phenom, Mandriva 2009.1. I can also confirm that changing the HAL on the guest VM to Advanced Configuration and Power Interface (ACPI) PC, changing the VM properties to disable IO-APIC, single vCPU. The guest then runs quickly and without problems - but of course, single vCPU. Is there any roadmap for a fix?

The exact same VM runs fine with multiple vCPU and IOAPIC on Intel VT-x, what's the problem with AMD-V ?

comment:4 by Sander van Leeuwen, 15 years ago

It's being worked on. Intel provides an optimization feature for APIC TPR access. AMD does not, so I have fix it myself.

comment:5 by mike.mcse, 15 years ago

Similar statements also apply to VMs running Standard PC HAL compared with MPS Multiprocessor HAL. Switching to Standard speeds it up very much with the IOAPIC turned off in the VM settings.

Do we have a possible timeline for a fix, either a whole new version of VB3 incorporating AMD-friendly technology, or a patch to 3.0.2 ?

comment:6 by Sander van Leeuwen, 15 years ago

Summary: huge IO-APIC/guest SMP overhead with 32 bits guests (AMD-V only)huge IO-APIC/guest SMP overhead with 32 bits guests (AMD-V only) -> fixed in SVN/3.0.6

Fixed in 3.0.6, but only after you've installed the 3.0.6 guest additions.

comment:7 by Frank Mehnert, 15 years ago

Resolution: fixed
Status: newclosed

comment:8 by mike.mcse, 15 years ago

Resolution: fixed
Status: closedreopened

Thanks for the good work on this bug. I will be testing it as soon as I can get my hands on 3.0.6.

Can I please ask you to regression-test this fix against Windows XP 64-bit Edition on AMD-V [3.0.4 crashes when Nested Paging is on, machine occasionally aborts without warning or logging an error when Nested Paging is off]. Host is 32-bit Mandriva 2009 and 2008.

by George Greer, 15 years ago

Attachment: VBox.log added

Windows 2000 SMP under Ubuntu 9.0.4 64-bit VT-x

comment:9 by George Greer, 15 years ago

It's not AMD-V only; my VT-x does it too, even with 3.0.6 Guest Additions installed. (See attachment.)

Windows 2000 SMP with Ubuntu 9.0.4 64-bit VT-x.

comment:10 by Sander van Leeuwen, 15 years ago

Description: modified (diff)
Summary: huge IO-APIC/guest SMP overhead with 32 bits guests (AMD-V only) -> fixed in SVN/3.0.6Huge IO-APIC/guest SMP overhead with 32 bits guests
Version: VirtualBox 3.0.0VirtualBox 3.0.6

mike: please don't mention unrelated problems in this defect. If you still see this xp64 abort with 3.0.6, then open another defect with a proper description and VBox.log.

greerga: your Intel CPU doesn't support the APIC access optimization. I intend to enable the same fix once we get more feedback about AMD-V.

comment:11 by Sander van Leeuwen, 15 years ago

Description: modified (diff)

comment:12 by Frank Mehnert, 14 years ago

Component: otherVMM/HWACCM

in reply to:  11 comment:13 by mike.mcse, 14 years ago

Replying to sandervl73:Point taken about 64-bit, I will keep testing that and raise a new issue for it.

However... my testing of 3.0.6 with XP 32-bit on AMD-V has shown that it's better - and it's worse. It's better - when you choose a guest ACPI Multprocessor HAL or ACPI Uniprocessor HAL, it runs without taking half an hour to boot up and flatlining one or two CPU cores at 100%. Most of the time. However today it went back to behaving like the un-patched 3.0.4 problem. I fixed this by running sudo /etc/init.d/vboxdrv restart . Clearly something in the VirtualBox drivers can revert to the old behavior given some [unknown] circumstances.

If you build your guest with the Advanced Power Configuration Interface (ACPI) HAL [i.e. no IOAPIC, as was required on 3.0.4 to work], it runs, but not as quickly as it used to on 3.0.4. So the fix has crippled non-IOAPIC but has not completely raised the performance level of IOAPIC-based configurations.

The typical performance issue is that the SysInternals Process Explorer is showing that System and Services.exe are together taking 6% or more all the time, even with dual 2.3GHz vCPUs. This is way short of physical machine performance. So the system has a performance draining wait loop somewhere inside it, meaning that activities like Windows Installer take far longer to complete than expected unless you intervene with Process Explorer and raise the process priority to "High".

It's a shame - this so nearly worked. The fact is that 3.0.4 with IOAPIC disabled and one vCPU massively outperforms 3.0.6 with dual vCPUs, which seems counter-intuitive.

Also I have had numerous aborts when Guest Additions 3D Acceleration is enabled.

comment:14 by ToddAndMargo, 14 years ago

Just getting myself on the Cc: list (couldn't find another way). Just ignore.

comment:15 by Sander van Leeuwen, 14 years ago

Description: modified (diff)

comment:16 by Frank Mehnert, 14 years ago

Description: modified (diff)

comment:17 by Sander van Leeuwen, 14 years ago

Description: modified (diff)

comment:18 by Ralf Jung, 14 years ago

I have exactly the described issue with VirtualBox 3.0.10, 32bit Kubuntu 9.04 host and 32bit Windows XP guest. My processor is a Core 2 Duo E6420 which does support 64bit. However, reading the "current status", I'm not sure what exactly is in SVN only - the fix for Intel processor supporting 64bit or the fix for those which don't.

comment:19 by Frank Mehnert, 14 years ago

VBox 3.1.0 will contain a fix for Windows guests which works if the guest additions are installed and if the host CPU supports VT-x and 64-bit. I known, it sounds a bit strange that 64-bit support is required to workaround some 32-bit issue :)

comment:20 by Frank Mehnert, 14 years ago

Well, the current status in the ticket description should be clear.

comment:21 by Ralf Jung, 14 years ago

Ok, thanks for clarification - I hope for version 3.1 then :)

comment:22 by fuUser, 14 years ago

Is there any plan/solution you are working on for 32 Bit Hosts? I use a (beloved) Pentium M Notebook with 2 GB Ram which can handled non-APIC VM's with acceptable speed (for office etc) but nearly grinds to hold and suffers huge delays when using an almost exactly configured VM, which was installed (and has to run) with APIC enabled. This causes a rather huge administrative burden for me, because I have to keep two Versions of practically the same VM updated and running. One for my 64 Bit Host(s) and one for my 32 Bit Host.

comment:23 by Ralf Jung, 14 years ago

The 3.0.12 changelog says: VMM: reduced IO-APIC overhead for 32 bits Windows NT/2000/XP/2003 guests; requires 64 bits support (VT-x only; bug #4392) However, I still experience this issue after upgrading to the latest version.

comment:24 by Frank Mehnert, 14 years ago

ralf: Please attach a VBox.log file of such a session.

comment:25 by Sander van Leeuwen, 14 years ago

ralf: you need to install the 3.0.12 additions and the OS type in the VM settings needs to be correct. The workaround is only enabled for 32 bits NT/2K/XP/2K3 OSes.

comment:26 by Frank Mehnert, 14 years ago

Yes, that was the reason I was asking for the VBox.log file :)

comment:27 by Ralf Jung, 14 years ago

The issue occurs long before I can even install the guest additions, during OS installation. I also have an old VM installed when there was no SMP, where I installed the 3.0.12 additions and after doing so copied the hal.dll and ntoskrnl.exe from a multiprocessor system. I will get you a logfile of that one ASAP. Do I have to re-install the additions after changing the HAL?

comment:28 by Frank Mehnert, 14 years ago

The performance will not improve until you installed the guest additions and until they are active. That is, the fix/workaround will not work during boot.

I think it is not necessary to re-install the guest additions after you changed the HAL but it cannot hurt. Make sure to use 3.0.12 additions in any case.

comment:29 by Ralf Jung, 14 years ago

Ok, I installed the new HAL and re-installed the latest guest additions. Then I started a new VirtualBox session (so the log file I'll attach soon starts here), started Windows, and indeed the "full load while doing nothing" vanished some time after the additions were loaded, so something improved. However, then I started Warcraft III (to test performance), and it is definitely slower than without IO-APIC (the sound stutters, and the animations are not fluent). I also have the impression that the task-manager process list still flickers a tiny bit, but it's nothing compared to how it behaves without the work-around. At least it's now able to play the Windows shut-down sound without stuttering (the start-up sound is of course still broken).

Any chance of a work-around which is independent of the guest doing some cooperation? In the current state, IO-APIC and SMP will stay disabled here. However, I obviously don't understand anything of the internals of this issue. Since I don't use the VM that often it's not a big deal. It'd just be nice to be able to use both cores. Anyway, keep up the good work, and thanks a lot for making such a great open-source VM :)

comment:30 by J.E. Raaijmakers, 14 years ago

Same problem here, trying to run guest Windows XP (32-bit) on host Kubuntu 9.10 (64-bit, Intel Q9450) using more than one processor and IO-APIC is incredibly slow whilst reinstalling without IO-APIC makes it run fast again. Installing guest additions did not help in my case.

in reply to:  30 comment:31 by ToddAndMargo, 14 years ago

Replying to JeRa:

Same problem here, trying to run guest Windows XP (32-bit) on host Kubuntu 9.10 (64-bit, Intel Q9450) using more than one processor and IO-APIC is incredibly slow whilst reinstalling without IO-APIC makes it run fast again. Installing guest additions did not help in my case.

How much RAM is your guest using? On two other bugs, boosting it to 512M solved the problem.

comment:32 by J.E. Raaijmakers, 14 years ago

Replying to ToddAndMargo:

How much RAM is your guest using? On two other bugs, boosting it to 512M solved the problem.

I allocated 1024 MB of RAM for the guest. The host has 4096 MB of RAM available. I could try to lower the amount of RAM to 512 MB just for testing purposes (you never know), but eventually I want to run the guest using the entire gigabyte.

comment:33 by Sander van Leeuwen, 14 years ago

JeRa: without the VBox.log from the VM session there's nothing we can do.

by J.E. Raaijmakers, 14 years ago

Attachment: VBox-WinXP-JeRa.log added

VB log of SMP WinXP (32-bit) running on Kubuntu 9.10 (64-bit) experiencing tremendous lagging and performance degradation

in reply to:  33 comment:34 by J.E. Raaijmakers, 14 years ago

Replying to sandervl73:

JeRa: without the VBox.log from the VM session there's nothing we can do.

Well then, here you are. I'd also like to note that I had additions installed (didn't help) and switched settings on 2d and 3d acceleration (dito).

comment:35 by Sander van Leeuwen, 14 years ago

JeRa: your problem is completely different. See #5501

comment:36 by Brian J. Cohen, 14 years ago

I can confirm this is an issue. Running Ubuntu 9.04 host, VirtualBox 3.0.12 r54644, and Guest extensions 3.0.12. With my Windows XP guest idle., the host CPU sits at around 50% usage. Things that had no effect: disabling USB controller on the VM, setting Null Audio Driver on the VM, enabling/disabling IO-APIC on the VM. I will attach my VBox.log.

by Brian J. Cohen, 14 years ago

Attachment: 56 MST 2009 added

comment:37 by Sander van Leeuwen, 14 years ago

brianjcohen: you need to enable AMD-V in your BIOS.

comment:38 by Sander van Leeuwen, 14 years ago

Description: modified (diff)

comment:39 by Brian J. Cohen, 14 years ago

sandervI73: My CPU does not support AMD-V.

comment:40 by Mark Cranness, 14 years ago

The fix in 3.0.12 does not work for me.

Host: Windows XP Pro (32bit), Intel Core 2 Duo E8400 (64bit capable), VirtualBox 3.0.12. Guest: Windows 2000 Server, GA 3.0.12 r54655 installed and running.

IO APIC on, 'ACPI Multiprocessor PC' HAL: When VM has 1 CPU: host runs at ~50% CPU usage, roughly split between each CPU. When VM has 2 CPUs: host runs at ~100% CPU usage.

hal.dll: 5.00.2195.7006 kernel32.dll: 5.00.2195.7135 ntdll.dll: 5.00.2195.7006 ntkrnlpa.exe: 5.00.2195.7319 ntoskrnl.exe: 5.00.2195.7319 win32k.sys: 5.00.2195.7322 winsrv.dll: 5.00.2195.7135

IO APIC on, 'ACPI Uniprocessor PC' HAL or IO APIC xx, 'Advanced Configuration and Power Interface (ACPI) PC' HAL, host CPU is as expected.

by Mark Cranness, 14 years ago

Attachment: VBox.2.log added

SMP Windows 2000 Server

comment:41 by Marc Steffens, 14 years ago

Can anyone let me know if there are there plans for a fix for Core Duo processors? I have an Intel T2500 Core Duo and have the problem when trying to use both cores in a VM.

comment:42 by Sander van Leeuwen, 14 years ago

MarkCranness: IO-APIC overhead has been solved for your CPU since 3.0 (iirc) as it supports the hardware workaround. There must be some other reason why the VM is using so much cpu time.

by Marc Steffens, 14 years ago

Attachment: VBox.3.log added

comment:43 by Marc Steffens, 14 years ago

Ok I must've misunderstood the notes in the description (leaves out Core Solo/Duo)? It'd be greatly appreciated if you have time to look at the log I just attached to see if there's anything obvious in there. When I run resource intensive apps in the VM on a single core it's quite slow so it'd be great to get both cores going. thanks

comment:44 by Michael Kromer, 14 years ago

I have the same Problem with an Windows 2003 R2 Guest on Various HP ML 385 Machines ... One host with 8 CPUs, and 5 CPU's dedicated for the Guest. Only one other VBox is running, however with only 1 CPU, and it doesn't take any load at all. The System (5 CPUs) is a terminal server, as soon as >2 people are doing things on the machine (like opening such applications as office) performance is lousy. Ticking Kernel Time in Task Manager shows up Extremely high Kernel Times ... Also top/htop reveal extremely high "System" load, consumed by VBox. The same machine without virtualization and same load not even takes 10% of what I needs under VirtualBox 3.1.2 running on 2.6.31.7 x86_64 kernel. Log follows in a second

Any suggestions?

by Michael Kromer, 14 years ago

Attachment: VBox.4.log added

VBox.log with bad 2003 r2 32-bit performance

comment:45 by hansb, 14 years ago

Just checked with Virtualbox 3.1.4. r57640, Ubuntu Karmic kernel 2.6.32-02063205-generic.

W2k guest with SMP enabled is unusable, because it uses 100% of the host's cpu as soon as IO APIC is enabled. The guest itself seems to run fine, but he host is borked. Current guest additions installed. All non-essential peripherals disabled. Log file follows.

by hansb, 14 years ago

Attachment: VBox.5.log added

comment:46 by Sander van Leeuwen, 14 years ago

hansb: karmic problems should be discussed in ticket #5501

comment:47 by Frank Mehnert, 14 years ago

Marc Steffens, please have a look at #6250, this problem applies to you as well.

comment:48 by Richard, 14 years ago

I can confirm this issue exists in VirtualBox 3.2.0. I am running Windows XP Professional SP3 as a guest OS with Gentoo Linux as a host. I am using app-emulation/virtualbox-bin-3.2.0 from Gentoo Linux's portage tree.

Here are some details regarding my host system:

# uname -a Linux Laptop 2.6.34 #1 SMP PREEMPT Tue May 18 09:57:44 EST 2010 i686 Genuine Intel(R) CPU T2400 @ 1.83GHz GenuineIntel GNU/Linux

comment:49 by Vincent Bernat, 14 years ago

I am using VirtualBox 3.2.4 from Debian and still have the issue. VT-X is enabled in the BIOS. The processor is Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz. I have set perf_counter to 2. The kernel is stock 2.6.32 from Debian Sid.

I have installed 3.2.4 additions without any change.

by Vincent Bernat, 14 years ago

Attachment: VBox.bernat.log added

VBOX.log with 100% CPU usage, VT-X enabled, on a QuadCore

comment:50 by Frank Mehnert, 10 years ago

Description: modified (diff)
Resolution: fixed
Status: reopenedclosed

Please reopen if still relevant with VBox 4.3.2.

Note: See TracTickets for help on using tickets.

© 2023 Oracle
ContactPrivacy policyTerms of Use