Severe time drift with KVM paravirt and Linux guests

[lpancescu]

I see very strong time drift with the kvm paravirt interface, on an OS X host - reproducible with both "debian/jessie64" and "centos/7". I let it run overnight for a more precise measurement. The guest time drifts very fast:

• 60 host seconds -> 8 guest seconds
• 8h37m host -> 0h53m38s guest

Raw measurements (host time as comment - OS X is set to the Europe/Berlin timezone):

[vagrant@localhost ~]$ date # 0:59:00
Tue May 10 18:56:18 EDT 2016
[vagrant@localhost ~]$ date # 1:00:00
Tue May 10 18:56:26 EDT 2016
[vagrant@localhost ~]$ date # 9:37:00
Tue May 10 19:50:04 EDT 2016

I found two workarounds to get a stable time:

1. set the paravirt interface to legacy, instead of kvm
2. use kvm as the paravirt interface, but set the guest kernel clocksource to acpi_pm (hpet also works, if i enable it - tsc too, i think).

Both the Debian Jessie and CentOS kernels default to kvm-clock as its clocksource, if KVM is available. I can reproduce this problem with VirtualBox 5.0.20 and 5.0.18 on OS X 10.11.4, both with and without the Guest Additions (I use a NTP daemon in the guest, when the Guest Additions are not installed, but the drift is too big to compensate). The attached log is with the CentOS guest.

The official Vagrant images for CentOS don't come with the Guest Additions preinstalled. If you have Vagrant and Ansible installed on the host, you can use ​https://github.com/lpancescu/cloud-instance-starter-kit for automatically updating the system and installing the guest additions (you'll need to issue vagrant reload after it's done).

[lpancescu]

VirtualBox log from last night's run

[lpancescu]

VirtualBox 5.0.2 log, no guest additions

[lpancescu]

I uninstalled VirtualBox 5.0.20 and installed 5.0.2, hoping to be able to bisect in which minor version of the 5.0 series this problem appeared. Unfortunately, 5.0.2 also shows this behavior. Interestingly, the time drift doesn't stop when I switch the clock source to tsc, but it stops completely when I switch to acpi_pm:

[vagrant@localhost ~]$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock
[vagrant@localhost ~]$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock tsc acpi_pm
[vagrant@localhost ~]$ date # 13:22:00
Wed May 11 07:21:38 EDT 2016
[vagrant@localhost ~]$ date # 13:23:00
Wed May 11 07:22:21 EDT 2016
[vagrant@localhost ~]$ date # 13:24:00
Wed May 11 07:22:28 EDT 2016
[vagrant@localhost ~]$ date # 13:25:00
Wed May 11 07:22:37 EDT 2016
[vagrant@localhost ~]$ date # 13:26:00
Wed May 11 07:22:45 EDT 2016
[vagrant@localhost ~]$ echo tsc | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc
[vagrant@localhost ~]$ date # 13:28:00
Wed May 11 07:23:03 EDT 2016
[vagrant@localhost ~]$ date # 13:29:00
Wed May 11 07:23:11 EDT 2016
[vagrant@localhost ~]$ echo acpi_pm | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource
acpi_pm
[vagrant@localhost ~]$ date # 13:30:00
Wed May 11 07:23:43 EDT 2016
[vagrant@localhost ~]$ date # 13:31:00
Wed May 11 07:24:43 EDT 2016
[vagrant@localhost ~]$ date # 13:32:00
Wed May 11 07:25:43 EDT 2016

This test was performed with a fresh centos/7 VM without the guest additions installed (to save some time). I attached the corresponding log.

[lpancescu]

Having read that kvm-clock is a virtualized version of tsc, I wrote a C program to help me plot the TSC values against the monotonic system clock, running inside the CentOS guest. I generated two plots, using the kvm and legacy paravirtualization interfaces - please see the attached files kvm.eps and legacy.eps (test.c is the C program used for generating the data). I set the Linux clocksource to acpi_pm in each paravirtualization mode, before running the measurements, since it's the only way to make sure that the monotonous clock in the VM corresponds to the real clock from the host.

When using the kvm paravirtualization interface, the TSC values increase much slower as soon as I call usleep, whose argument also seems to affect the slope of TSC (the values are always read two times per host-second); this doesn't happen when using the legacy paravirtualization interface - the slope always remains the same.

[lpancescu]

Measure TSC vs monotonous clock - to be run inside the guest

[lpancescu]

Plot TSC vs monotonous clock, using kvm paravirt

[lpancescu]

Plot TSC vs monotonous clock, using legacy paravirt

[Frank Mehnert]

Please reopen if still relevant with VBox 5.1.22.

[Tim S]

I have the same issue.

Guest clock runs slow by between 50% and a factor of 6 times slower. Sometimes there are only 10 guest seconds in a real world / host minute.

Currently using VirtualBox 5.2.12. The same behavior was also seen on previous 5.1 versions. I did not run versions older than that on my current system. Host: Mac OS 10.13.4. Guest: Debian Stretch.

I don't know if this is relevant, but my MacBook is old (late 2009 - 2.26 GHz Intel Core 2 Duo). Same behavior with 4 GB on the host and 8 GB on the host, 2 GB guest RAM of the 4 GB host RAM and 5 GB guest RAM of the 8 GB host RAM. Same behavior with 1 or 2 guest CPU.

I found a workaround:
VBoxManage setextradata VM_NAME VBoxInternal/TM/TSCModeSwitchAllowed 0
before running VirtualBox on the host, but this disables using the real TSC for the guest clock and so I wonder if it has performance issues.

[Tim S]

Extreme drift VBox.log for tim-coding

[Tim S]

Extreme drift workaround VBox.log for tim-coding

[Sebastian H]

I can still observe this problem. *I think* Quite massive clock drift in a debian guest, making MQTT connections with apache Paho fail.

Host:
Mac OS X 10.11.6, BuildVersion: 15G22010, Model Identifier: Macmini3,1 (interestingly also a core2duo chip)

Guest:
Linux host 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux (Debian 9)

VBoxService -v shows:

VBoxService 5.2.22 r126460 (verbosity: 1) linux.amd64 (Nov  8 2018 20:49:41) release log
00:00:00.001087 main     Log opened 2019-01-05T10:20:56.988351000Z
00:00:00.001806 main     OS Product: Linux
00:00:00.001894 main     OS Release: 4.9.0-8-amd64
00:00:00.001917 main     OS Version: #1 SMP Debian 4.9.130-2 (2018-10-27)
00:00:00.001920 main     Executable: /opt/VBoxGuestAdditions-5.2.22/sbin/VBoxService
00:00:00.001923 main     Process ID: 9340
00:00:00.001926 main     Package type: LINUX_64BITS_GENERIC
00:00:00.012121 main     5.2.22 r126460 started. Verbose level = 1
00:00:00.012121 main     Daemonizing...

The VM was generated with the wizard via GUI on my host machine, more or less all defaults. Debian minimal net install, apt update & upgrade, get kernel headers, then installed the VBox guest addons from the ISO file.

I had no luck keeping time in sync with commands like "VBoxService --timesync-set-threshold 10000" and similar: Massive slow time in guest os, when the system time was then being set the MQTT connection would obscurely break..

The clock drift seems to be stopped by applying the fix in the guest mentioned in comment by lpancescu:

echo acpi_pm | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource