VirtualBox

Ticket #4392 (closed defect: fixed)

Opened 5 years ago

Last modified 6 months ago

Huge IO-APIC/guest SMP overhead with 32 bits guests

Reported by: rafael_chang Owned by:
Priority: major Component: VMM/HWACCM
Version: VirtualBox 3.0.6 Keywords:
Cc: Guest type: other
Host type: other

Description (last modified by frank) (diff)

Symptoms are high CPU usage while running e.g. 32 bits Windows XP/2000 guests with IO-APIC enabled.

Current status:

  • fixed for AMD-V in VirtualBox 3.0.6 (requires 3.0.6 guest additions to be installed)
  • fixed for VT-x, but only if the Intel CPU supports 64 bits (leaves out Core Solo/Duo); fixed in 3.0.12 (requires 3.0.12 guest additions to be installed)
  • Solaris/OpenSolaris guests have similar issues, but a kernel fix is underway.

Does not apply to:

  • 64 bits guests
  • 32 bits Windows Vista or Windows 7

Attachments

VBox.log Download (41.6 KB) - added by greerga 5 years ago.
Windows 2000 SMP under Ubuntu 9.0.4 64-bit VT-x
VBox-WinXP-JeRa.log Download (57.5 KB) - added by JeRa 4 years ago.
VB log of SMP WinXP (32-bit) running on Kubuntu 9.10 (64-bit) experiencing tremendous lagging and performance degradation
56 MST 2009 Download (46.9 KB) - added by brianjcohen 4 years ago.
VBox.2.log Download (53.4 KB) - added by MarkCranness 4 years ago.
SMP Windows 2000 Server
VBox.3.log Download (64.0 KB) - added by msteffens@… 4 years ago.
VBox.4.log Download (47.9 KB) - added by mkromer 4 years ago.
VBox.log with bad 2003 r2 32-bit performance
VBox.5.log Download (53.9 KB) - added by hansb 4 years ago.
VBox.bernat.log Download (61.7 KB) - added by Vincent Bernat 4 years ago.
VBOX.log with 100% CPU usage, VT-X enabled, on a QuadCore

Change History

comment:1 Changed 5 years ago by rafael_chang

noticed the following, if I disable the IOAPIC and set VCPU to 1, install goes fine and fast ...

comment:2 Changed 5 years ago by sandervl73

  • Summary changed from windows xp VM extremely slow when AMD-v is enable with 2 cpu to huge IO-APIC/guest SMP overhead with 32 bits guests (AMD-V only)

Known problem and mentioned in the manual. 32 bits guests are extremely slow with the IO-APIC enabled on AMD cpus. Will be fixed in a future release.

comment:3 Changed 5 years ago by mike.merrett@…

I can confirm this behavior on 3.0.2 on AMD Phenom, Mandriva 2009.1. I can also confirm that changing the HAL on the guest VM to Advanced Configuration and Power Interface (ACPI) PC, changing the VM properties to disable IO-APIC, single vCPU. The guest then runs quickly and without problems - but of course, single vCPU. Is there any roadmap for a fix?

The exact same VM runs fine with multiple vCPU and IOAPIC on Intel VT-x, what's the problem with AMD-V ?

comment:4 Changed 5 years ago by sandervl73

It's being worked on. Intel provides an optimization feature for APIC TPR access. AMD does not, so I have fix it myself.

comment:5 Changed 5 years ago by mike.merrett@…

Similar statements also apply to VMs running Standard PC HAL compared with MPS Multiprocessor HAL. Switching to Standard speeds it up very much with the IOAPIC turned off in the VM settings.

Do we have a possible timeline for a fix, either a whole new version of VB3 incorporating AMD-friendly technology, or a patch to 3.0.2 ?

comment:6 Changed 5 years ago by sandervl73

  • Summary changed from huge IO-APIC/guest SMP overhead with 32 bits guests (AMD-V only) to huge IO-APIC/guest SMP overhead with 32 bits guests (AMD-V only) -> fixed in SVN/3.0.6

Fixed in 3.0.6, but only after you've installed the 3.0.6 guest additions.

comment:7 Changed 5 years ago by frank

  • Status changed from new to closed
  • Resolution set to fixed

comment:8 Changed 5 years ago by mike.merrett@…

  • Status changed from closed to reopened
  • Resolution fixed deleted

Thanks for the good work on this bug. I will be testing it as soon as I can get my hands on 3.0.6.

Can I please ask you to regression-test this fix against Windows XP 64-bit Edition on AMD-V [3.0.4 crashes when Nested Paging is on, machine occasionally aborts without warning or logging an error when Nested Paging is off]. Host is 32-bit Mandriva 2009 and 2008.

Changed 5 years ago by greerga

Windows 2000 SMP under Ubuntu 9.0.4 64-bit VT-x

comment:9 Changed 5 years ago by greerga

It's not AMD-V only; my VT-x does it too, even with 3.0.6 Guest Additions installed. (See attachment.)

Windows 2000 SMP with Ubuntu 9.0.4 64-bit VT-x.

comment:10 Changed 5 years ago by sandervl73

  • Version changed from VirtualBox 3.0.0 to VirtualBox 3.0.6
  • Description modified (diff)
  • Summary changed from huge IO-APIC/guest SMP overhead with 32 bits guests (AMD-V only) -> fixed in SVN/3.0.6 to Huge IO-APIC/guest SMP overhead with 32 bits guests

mike: please don't mention unrelated problems in this defect. If you still see this xp64 abort with 3.0.6, then open another defect with a proper description and VBox.log.

greerga: your Intel CPU doesn't support the APIC access optimization. I intend to enable the same fix once we get more feedback about AMD-V.

comment:11 follow-up: ↓ 13 Changed 5 years ago by sandervl73

  • Description modified (diff)

comment:12 Changed 5 years ago by frank

  • Component changed from other to VMM/HWACCM

comment:13 in reply to: ↑ 11 Changed 5 years ago by mike.merrett@…

Replying to sandervl73:Point taken about 64-bit, I will keep testing that and raise a new issue for it.

However... my testing of 3.0.6 with XP 32-bit on AMD-V has shown that it's better - and it's worse. It's better - when you choose a guest ACPI Multprocessor HAL or ACPI Uniprocessor HAL, it runs without taking half an hour to boot up and flatlining one or two CPU cores at 100%. Most of the time. However today it went back to behaving like the un-patched 3.0.4 problem. I fixed this by running sudo /etc/init.d/vboxdrv restart . Clearly something in the VirtualBox drivers can revert to the old behavior given some [unknown] circumstances.

If you build your guest with the Advanced Power Configuration Interface (ACPI) HAL [i.e. no IOAPIC, as was required on 3.0.4 to work], it runs, but not as quickly as it used to on 3.0.4. So the fix has crippled non-IOAPIC but has not completely raised the performance level of IOAPIC-based configurations.

The typical performance issue is that the SysInternals Process Explorer is showing that System and Services.exe are together taking 6% or more all the time, even with dual 2.3GHz vCPUs. This is way short of physical machine performance. So the system has a performance draining wait loop somewhere inside it, meaning that activities like Windows Installer take far longer to complete than expected unless you intervene with Process Explorer and raise the process priority to "High".

It's a shame - this so nearly worked. The fact is that 3.0.4 with IOAPIC disabled and one vCPU massively outperforms 3.0.6 with dual vCPUs, which seems counter-intuitive.

Also I have had numerous aborts when Guest Additions 3D Acceleration is enabled.

comment:14 Changed 4 years ago by ToddAndMargo

Just getting myself on the Cc: list (couldn't find another way). Just ignore.

comment:15 Changed 4 years ago by sandervl73

  • Description modified (diff)

comment:16 Changed 4 years ago by frank

  • Description modified (diff)

comment:17 Changed 4 years ago by sandervl73

  • Description modified (diff)

comment:18 Changed 4 years ago by ralf

I have exactly the described issue with VirtualBox 3.0.10, 32bit Kubuntu 9.04 host and 32bit Windows XP guest. My processor is a Core 2 Duo E6420 which does support 64bit. However, reading the "current status", I'm not sure what exactly is in SVN only - the fix for Intel processor supporting 64bit or the fix for those which don't.

comment:19 Changed 4 years ago by frank

VBox 3.1.0 will contain a fix for Windows guests which works if the guest additions are installed and if the host CPU supports VT-x and 64-bit. I known, it sounds a bit strange that 64-bit support is required to workaround some 32-bit issue :)

comment:20 Changed 4 years ago by frank

Well, the current status in the ticket description should be clear.

comment:21 Changed 4 years ago by ralf

Ok, thanks for clarification - I hope for version 3.1 then :)

comment:22 Changed 4 years ago by fuUser

Is there any plan/solution you are working on for 32 Bit Hosts? I use a (beloved) Pentium M Notebook with 2 GB Ram which can handled non-APIC VM's with acceptable speed (for office etc) but nearly grinds to hold and suffers huge delays when using an almost exactly configured VM, which was installed (and has to run) with APIC enabled. This causes a rather huge administrative burden for me, because I have to keep two Versions of practically the same VM updated and running. One for my 64 Bit Host(s) and one for my 32 Bit Host.

comment:23 Changed 4 years ago by ralf

The 3.0.12 changelog says: VMM: reduced IO-APIC overhead for 32 bits Windows NT/2000/XP/2003 guests; requires 64 bits support (VT-x only; bug #4392) However, I still experience this issue after upgrading to the latest version.

comment:24 Changed 4 years ago by frank

ralf: Please attach a VBox.log file of such a session.

comment:25 Changed 4 years ago by sandervl73

ralf: you need to install the 3.0.12 additions and the OS type in the VM settings needs to be correct. The workaround is only enabled for 32 bits NT/2K/XP/2K3 OSes.

comment:26 Changed 4 years ago by frank

Yes, that was the reason I was asking for the VBox.log file :)

comment:27 Changed 4 years ago by ralf

The issue occurs long before I can even install the guest additions, during OS installation. I also have an old VM installed when there was no SMP, where I installed the 3.0.12 additions and after doing so copied the hal.dll and ntoskrnl.exe from a multiprocessor system. I will get you a logfile of that one ASAP. Do I have to re-install the additions after changing the HAL?

comment:28 Changed 4 years ago by frank

The performance will not improve until you installed the guest additions and until they are active. That is, the fix/workaround will not work during boot.

I think it is not necessary to re-install the guest additions after you changed the HAL but it cannot hurt. Make sure to use 3.0.12 additions in any case.

comment:29 Changed 4 years ago by ralf

Ok, I installed the new HAL and re-installed the latest guest additions. Then I started a new VirtualBox session (so the log file I'll attach soon starts here), started Windows, and indeed the "full load while doing nothing" vanished some time after the additions were loaded, so something improved. However, then I started Warcraft III (to test performance), and it is definitely slower than without IO-APIC (the sound stutters, and the animations are not fluent). I also have the impression that the task-manager process list still flickers a tiny bit, but it's nothing compared to how it behaves without the work-around. At least it's now able to play the Windows shut-down sound without stuttering (the start-up sound is of course still broken).

Any chance of a work-around which is independent of the guest doing some cooperation? In the current state, IO-APIC and SMP will stay disabled here. However, I obviously don't understand anything of the internals of this issue. Since I don't use the VM that often it's not a big deal. It'd just be nice to be able to use both cores. Anyway, keep up the good work, and thanks a lot for making such a great open-source VM :)

comment:30 follow-up: ↓ 31 Changed 4 years ago by JeRa

Same problem here, trying to run guest Windows XP (32-bit) on host Kubuntu 9.10 (64-bit, Intel Q9450) using more than one processor and IO-APIC is incredibly slow whilst reinstalling without IO-APIC makes it run fast again. Installing guest additions did not help in my case.

comment:31 in reply to: ↑ 30 Changed 4 years ago by ToddAndMargo

Replying to JeRa:

Same problem here, trying to run guest Windows XP (32-bit) on host Kubuntu 9.10 (64-bit, Intel Q9450) using more than one processor and IO-APIC is incredibly slow whilst reinstalling without IO-APIC makes it run fast again. Installing guest additions did not help in my case.

How much RAM is your guest using? On two other bugs, boosting it to 512M solved the problem.

comment:32 Changed 4 years ago by JeRa

Replying to ToddAndMargo:

How much RAM is your guest using? On two other bugs, boosting it to 512M solved the problem.

I allocated 1024 MB of RAM for the guest. The host has 4096 MB of RAM available. I could try to lower the amount of RAM to 512 MB just for testing purposes (you never know), but eventually I want to run the guest using the entire gigabyte.

comment:33 follow-up: ↓ 34 Changed 4 years ago by sandervl73

JeRa: without the VBox.log from the VM session there's nothing we can do.

Changed 4 years ago by JeRa

VB log of SMP WinXP (32-bit) running on Kubuntu 9.10 (64-bit) experiencing tremendous lagging and performance degradation

comment:34 in reply to: ↑ 33 Changed 4 years ago by JeRa

Replying to sandervl73:

JeRa: without the VBox.log from the VM session there's nothing we can do.

Well then, here you are. I'd also like to note that I had additions installed (didn't help) and switched settings on 2d and 3d acceleration (dito).

comment:35 Changed 4 years ago by sandervl73

JeRa: your problem is completely different. See #5501

comment:36 Changed 4 years ago by brianjcohen

I can confirm this is an issue. Running Ubuntu 9.04 host, VirtualBox 3.0.12 r54644, and Guest extensions 3.0.12. With my Windows XP guest idle., the host CPU sits at around 50% usage. Things that had no effect: disabling USB controller on the VM, setting Null Audio Driver on the VM, enabling/disabling IO-APIC on the VM. I will attach my VBox.log.

Changed 4 years ago by brianjcohen

comment:37 Changed 4 years ago by sandervl73

brianjcohen: you need to enable AMD-V in your BIOS.

comment:38 Changed 4 years ago by sandervl73

  • Description modified (diff)

comment:39 Changed 4 years ago by brianjcohen

sandervI73: My CPU does not support AMD-V.

comment:40 Changed 4 years ago by MarkCranness

The fix in 3.0.12 does not work for me.

Host: Windows XP Pro (32bit), Intel Core 2 Duo E8400 (64bit capable), VirtualBox 3.0.12. Guest: Windows 2000 Server, GA 3.0.12 r54655 installed and running.

IO APIC on, 'ACPI Multiprocessor PC' HAL: When VM has 1 CPU: host runs at ~50% CPU usage, roughly split between each CPU. When VM has 2 CPUs: host runs at ~100% CPU usage.

hal.dll: 5.00.2195.7006 kernel32.dll: 5.00.2195.7135 ntdll.dll: 5.00.2195.7006 ntkrnlpa.exe: 5.00.2195.7319 ntoskrnl.exe: 5.00.2195.7319 win32k.sys: 5.00.2195.7322 winsrv.dll: 5.00.2195.7135

IO APIC on, 'ACPI Uniprocessor PC' HAL or IO APIC xx, 'Advanced Configuration and Power Interface (ACPI) PC' HAL, host CPU is as expected.

Changed 4 years ago by MarkCranness

SMP Windows 2000 Server

comment:41 Changed 4 years ago by msteffens@…

Can anyone let me know if there are there plans for a fix for Core Duo processors? I have an Intel T2500 Core Duo and have the problem when trying to use both cores in a VM.

comment:42 Changed 4 years ago by sandervl73

MarkCranness: IO-APIC overhead has been solved for your CPU since 3.0 (iirc) as it supports the hardware workaround. There must be some other reason why the VM is using so much cpu time.

Changed 4 years ago by msteffens@…

comment:43 Changed 4 years ago by msteffens@…

Ok I must've misunderstood the notes in the description (leaves out Core Solo/Duo)? It'd be greatly appreciated if you have time to look at the log I just attached to see if there's anything obvious in there. When I run resource intensive apps in the VM on a single core it's quite slow so it'd be great to get both cores going. thanks

comment:44 Changed 4 years ago by mkromer

I have the same Problem with an Windows 2003 R2 Guest on Various HP ML 385 Machines ... One host with 8 CPUs, and 5 CPU's dedicated for the Guest. Only one other VBox is running, however with only 1 CPU, and it doesn't take any load at all. The System (5 CPUs) is a terminal server, as soon as >2 people are doing things on the machine (like opening such applications as office) performance is lousy. Ticking Kernel Time in Task Manager shows up Extremely high Kernel Times ... Also top/htop reveal extremely high "System" load, consumed by VBox. The same machine without virtualization and same load not even takes 10% of what I needs under VirtualBox 3.1.2 running on 2.6.31.7 x86_64 kernel. Log follows in a second

Any suggestions?

Changed 4 years ago by mkromer

VBox.log with bad 2003 r2 32-bit performance

comment:45 Changed 4 years ago by hansb

Just checked with Virtualbox 3.1.4. r57640, Ubuntu Karmic kernel 2.6.32-02063205-generic.

W2k guest with SMP enabled is unusable, because it uses 100% of the host's cpu as soon as IO APIC is enabled. The guest itself seems to run fine, but he host is borked. Current guest additions installed. All non-essential peripherals disabled. Log file follows.

Changed 4 years ago by hansb

comment:46 Changed 4 years ago by sandervl73

hansb: karmic problems should be discussed in ticket #5501

comment:47 Changed 4 years ago by frank

msteffens@…, please have a look at #6250, this problem applies to you as well.

comment:48 Changed 4 years ago by Shining Arcanine

I can confirm this issue exists in VirtualBox 3.2.0. I am running Windows XP Professional SP3 as a guest OS with Gentoo Linux as a host. I am using app-emulation/virtualbox-bin-3.2.0 from Gentoo Linux's portage tree.

Here are some details regarding my host system:

# uname -a Linux Laptop 2.6.34 #1 SMP PREEMPT Tue May 18 09:57:44 EST 2010 i686 Genuine Intel(R) CPU T2400 @ 1.83GHz GenuineIntel GNU/Linux

comment:49 Changed 4 years ago by Vincent Bernat

I am using VirtualBox 3.2.4 from Debian and still have the issue. VT-X is enabled in the BIOS. The processor is Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz. I have set perf_counter to 2. The kernel is stock 2.6.32 from Debian Sid.

I have installed 3.2.4 additions without any change.

Changed 4 years ago by Vincent Bernat

VBOX.log with 100% CPU usage, VT-X enabled, on a QuadCore

comment:50 Changed 6 months ago by frank

  • Status changed from reopened to closed
  • Resolution set to fixed
  • Description modified (diff)

Please reopen if still relevant with VBox 4.3.2.

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use