Opened 8 years ago
Last modified 6 years ago
#15405 reopened defect
Severe time drift with KVM paravirt and Linux guests
Reported by: | lpancescu | Owned by: | |
---|---|---|---|
Component: | VMM | Version: | VirtualBox 5.0.20 |
Keywords: | Cc: | ||
Guest type: | Linux | Host type: | Mac OS X |
Description
I see very strong time drift with the kvm paravirt interface, on an OS X host - reproducible with both "debian/jessie64" and "centos/7". I let it run overnight for a more precise measurement. The guest time drifts very fast:
- 60 host seconds -> 8 guest seconds
- 8h37m host -> 0h53m38s guest
Raw measurements (host time as comment - OS X is set to the Europe/Berlin timezone):
[vagrant@localhost ~]$ date # 0:59:00 Tue May 10 18:56:18 EDT 2016 [vagrant@localhost ~]$ date # 1:00:00 Tue May 10 18:56:26 EDT 2016 [vagrant@localhost ~]$ date # 9:37:00 Tue May 10 19:50:04 EDT 2016
I found two workarounds to get a stable time:
- set the paravirt interface to legacy, instead of kvm
- use
kvm
as the paravirt interface, but set the guest kernelclocksource
toacpi_pm
(hpet
also works, if i enable it -tsc
too, i think).
Both the Debian Jessie and CentOS kernels default to kvm-clock
as its clocksource
, if KVM is available. I can reproduce this problem with VirtualBox 5.0.20 and 5.0.18 on OS X 10.11.4, both with and without the Guest Additions (I use a NTP daemon in the guest, when the Guest Additions are not installed, but the drift is too big to compensate). The attached log is with the CentOS guest.
The official Vagrant images for CentOS don't come with the Guest Additions preinstalled. If you have Vagrant and Ansible installed on the host, you can use https://github.com/lpancescu/cloud-instance-starter-kit for automatically updating the system and installing the guest additions (you'll need to issue vagrant reload
after it's done).
Attachments (7)
Change History (12)
by , 8 years ago
comment:1 by , 8 years ago
I uninstalled VirtualBox 5.0.20 and installed 5.0.2, hoping to be able to bisect in which minor version of the 5.0 series this problem appeared. Unfortunately, 5.0.2 also shows this behavior. Interestingly, the time drift doesn't stop when I switch the clock source to tsc
, but it stops completely when I switch to acpi_pm
:
[vagrant@localhost ~]$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource kvm-clock [vagrant@localhost ~]$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource kvm-clock tsc acpi_pm [vagrant@localhost ~]$ date # 13:22:00 Wed May 11 07:21:38 EDT 2016 [vagrant@localhost ~]$ date # 13:23:00 Wed May 11 07:22:21 EDT 2016 [vagrant@localhost ~]$ date # 13:24:00 Wed May 11 07:22:28 EDT 2016 [vagrant@localhost ~]$ date # 13:25:00 Wed May 11 07:22:37 EDT 2016 [vagrant@localhost ~]$ date # 13:26:00 Wed May 11 07:22:45 EDT 2016 [vagrant@localhost ~]$ echo tsc | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource tsc [vagrant@localhost ~]$ date # 13:28:00 Wed May 11 07:23:03 EDT 2016 [vagrant@localhost ~]$ date # 13:29:00 Wed May 11 07:23:11 EDT 2016 [vagrant@localhost ~]$ echo acpi_pm | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource acpi_pm [vagrant@localhost ~]$ date # 13:30:00 Wed May 11 07:23:43 EDT 2016 [vagrant@localhost ~]$ date # 13:31:00 Wed May 11 07:24:43 EDT 2016 [vagrant@localhost ~]$ date # 13:32:00 Wed May 11 07:25:43 EDT 2016
This test was performed with a fresh centos/7
VM without the guest additions installed (to save some time). I attached the corresponding log.
comment:2 by , 8 years ago
Having read that kvm-clock
is a virtualized version of tsc
, I wrote a C program to help me plot the TSC values against the monotonic system clock, running inside the CentOS guest. I generated two plots, using the kvm
and legacy
paravirtualization interfaces - please see the attached files kvm.eps
and legacy.eps
(test.c
is the C program used for generating the data). I set the Linux clocksource
to acpi_pm
in each paravirtualization mode, before running the measurements, since it's the only way to make sure that the monotonous clock in the VM corresponds to the real clock from the host.
When using the kvm
paravirtualization interface, the TSC
values increase much slower as soon as I call usleep
, whose argument also seems to affect the slope of TSC
(the values are always read two times per host-second); this doesn't happen when using the legacy
paravirtualization interface - the slope always remains the same.
by , 8 years ago
Measure TSC vs monotonous clock - to be run inside the guest
comment:3 by , 7 years ago
Resolution: | → obsolete |
---|---|
Status: | new → closed |
Please reopen if still relevant with VBox 5.1.22.
comment:4 by , 6 years ago
Resolution: | obsolete |
---|---|
Status: | closed → reopened |
I have the same issue.
Guest clock runs slow by between 50% and a factor of 6 times slower. Sometimes there are only 10 guest seconds in a real world / host minute.
Currently using VirtualBox 5.2.12. The same behavior was also seen on previous 5.1 versions. I did not run versions older than that on my current system. Host: Mac OS 10.13.4. Guest: Debian Stretch.
I don't know if this is relevant, but my MacBook is old (late 2009 - 2.26 GHz Intel Core 2 Duo). Same behavior with 4 GB on the host and 8 GB on the host, 2 GB guest RAM of the 4 GB host RAM and 5 GB guest RAM of the 8 GB host RAM. Same behavior with 1 or 2 guest CPU.
I found a workaround:
VBoxManage setextradata VM_NAME VBoxInternal/TM/TSCModeSwitchAllowed 0
before running VirtualBox on the host, but this disables using the real TSC for the guest clock and so I wonder if it has performance issues.
by , 6 years ago
Attachment: | VBox-tim-nodrift-disable-ModeSwitchAllowed.log added |
---|
Extreme drift workaround VBox.log for tim-coding
comment:5 by , 6 years ago
I can still observe this problem. *I think* Quite massive clock drift in a debian guest, making MQTT connections with apache Paho fail.
Host:
Mac OS X 10.11.6, BuildVersion: 15G22010, Model Identifier: Macmini3,1 (interestingly also a core2duo chip)
Guest:
Linux host 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux (Debian 9)
VBoxService -v shows:
VBoxService 5.2.22 r126460 (verbosity: 1) linux.amd64 (Nov 8 2018 20:49:41) release log 00:00:00.001087 main Log opened 2019-01-05T10:20:56.988351000Z 00:00:00.001806 main OS Product: Linux 00:00:00.001894 main OS Release: 4.9.0-8-amd64 00:00:00.001917 main OS Version: #1 SMP Debian 4.9.130-2 (2018-10-27) 00:00:00.001920 main Executable: /opt/VBoxGuestAdditions-5.2.22/sbin/VBoxService 00:00:00.001923 main Process ID: 9340 00:00:00.001926 main Package type: LINUX_64BITS_GENERIC 00:00:00.012121 main 5.2.22 r126460 started. Verbose level = 1 00:00:00.012121 main Daemonizing...
The VM was generated with the wizard via GUI on my host machine, more or less all defaults. Debian minimal net install, apt update & upgrade, get kernel headers, then installed the VBox guest addons from the ISO file.
I had no luck keeping time in sync with commands like "VBoxService --timesync-set-threshold 10000" and similar: Massive slow time in guest os, when the system time was then being set the MQTT connection would obscurely break..
The clock drift seems to be stopped by applying the fix in the guest mentioned in comment by lpancescu:
echo acpi_pm | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource
VirtualBox log from last night's run