Very bad latency numbers with more than 1 vCPU

[JasonM]

VirtualBox 6.1.12 verified on Win10 host (or openSUSE TW host), with Linux guests Ubuntu 16/18 or openSUSE TW.

Basically, I have distilled this down to a small perl script, that sleeps for 1ms for 1000 times. Ideally, this would be 1000ms total -- but on real hardware, the script takes just a shade under 1100ms per iteration (perl overhead, etc), and when VirtualBox is behaving, about 1350-1425ms per iteration.

On real hardware, 100% of 1ms sleeps are <2ms. When VirtualBox is behaving, 99% of 1ms sleeps are <2ms, and the upper bound is around 10-15ms for a few outliers.

The problem is when vCPU is 2+ ... the more added, the worse latency gets. 1ms sleeps start taking 100ms, 300ms, 500ms, occasionally even 900ms to return.

[JasonM]

perl script that measures latency of 1ms sleep calls

[JasonM]

Log file per twiki

[JasonM]

For clarification: What you'll want to do is set vCPU to 1 and boot Linux and run the script. Note the good latency numbers. Now power off, set vCPU to 4, and run the script again. The reported latency numbers are terrible. If you have a cpu with 8 threads, you can try 8 vCPUs ... the script barely runs, nearly all the 1ms sleep calls are horribly long (as an extreme example). Even 2 vCPUs is not nearly as good as 1 vCPU.

Also the same problem whether Win10 host or openSUSE TW host. Same problem with openSUSE TW guest or Ubuntu 16/18 guest.

[JasonM]

I made some progress on hunting the root cause. Turning off the Linux guest nohz scheduler makes the problem go away. Something about vbox <-> tickless Linux guest isn't working very well with multiple vcpus. I don't know who is at fault here though, either vbox is doing something that upsets the guest timing, or the guest nohz scheduler isn't robustly handling being in virtualization. This could explain why turning --hpet on improves but doesn't entirely cure the guest issue.

[fth0]

Associated forum thread: ​https://forums.virtualbox.org/viewtopic.php?f=3&t=99329