Double caching degrades performance

[David Chisnall]

VirtualBox does not appear to be setting the F_NOCACHE fcntl() on virtual disk images. This means that every read by the guest results in memory used in the disk cache in the host. The guest OS is also caching the read data in its disk cache, so counting the copy given to the userspace program, you are using 12KB of RAM for each 4KB disk read. This makes resource allocation hard (a guest given 512MB of RAM on the host may also use another 512MB of kernel RAM for disk caching), gives no performance gain for the guest (because the guest is caching the disk access anyway, and there are very few occasions when the VM's cache is missed but the host OS's cache is hit), and seriously degrades performance of other apps by causing them to be swapped out in preference to pages mapped from the guest's disk image, which are not being accessed.

[Sander van Leeuwen]

Quite a bold claim as we've measured the performance with and without host cache and there is a considerable difference.

An internal disk cache is being worked on and will be enabled in a future release.

[David Chisnall]

Have you measured the performance of other applications under OS X? After VirtualBox has been running for a while on OS X hosts, other applications become unresponsive because they are pushed out of RAM and switching to them causes heavy swapping. Meanwhile, the VM is fine and its kernel reports that most disk I/O is being handled from the guest's disk cache.

Cache hits from the guest's disk cache will always be faster than cache hits from the host's disk cache, and with no ability to disable use of the host's disk cache it is impossible to accurately restrict the guest from overconsumption of kernel resources on the host.

An internal disk cache is pointless. If you want better caching, you can just allocate more RAM to the guest and it will use it for disk caching (more accurately than the VM can, because it gets usage hints from running applications and can, for example, not read-ahead cache files that are no longer open or have been marked for random access by guest apps).

The only time a disk cache in the VM will improve performance is if the guest either doesn't have enough RAM (in which case, the user has explicitly restricted the guest's resources and a disk cache in the VM is violating the policy set by the user) or if the guest's disk cache is making poor decisions and yet the VM's cache is making better decisions with less information.

[Sander van Leeuwen]

I'm not going to start an argument with you. The numbers speak for themselves and there's a good explanation for them regardless of your opinion. (hint: there's more to disk access than just reading)

The whole point of the internal disk cache is to limit memory consumption yet not let disk performance go down the drain.

There's already a way to disable the host's disk cache:

VBoxManage setextradata <VM name> "VBoxInternal/PDM/AsyncCompletion/File/HostCacheEnabled" 0

I haven't checked if this override still works.

[David Chisnall]

If you're talking about buffering writes, then that's completely irrelevant to what I am saying. Obviously you don't want to prevent the host OS from doing that or you will lose write reordering and coalescing. Setting the F_NOCACHE fcntl(), as I suggested, does not prevent write caching, it prevents the relevant pages hanging around in the disk cache after they have been sync'd with the disk or the application (whichever is the consumer). I'll try the configuration option you suggest and see whether it makes a difference over a few days of use.

Your comments make me wonder if VirtualBox is actually doing the right thing on OS X at all when it comes to disk I/O. I hope you're not relying on fsync() to commit data to the disk on this platform...

[Sander van Leeuwen]

Well, if you think you're cleverer than we are, then I suggest you look at the source code.

[David Chisnall]

Irrelevant to my original issue, but it appears that you flush writes from the guest to the disk with this function on OS X:

RTR3DECL(int) RTFileFlush(RTFILE File) {

if (fsync((int)File))

return RTErrConvertFromErrno(errno);

return VINF_SUCCESS;

}

On Darwin, this is categorically not guaranteed to flush the data to disk. It may still be in non-volatile storage somewhere between the OS and the disk. This function, on Darwin, should read:

RTR3DECL(int) RTFileFlush(RTFILE File) {

if (fcntl((int)File, F_FULLFSYNC))

return RTErrConvertFromErrno(errno);

return VINF_SUCCESS;

}

This will guarantee that the data is actually committed to nonvolatile storage.

Looking at the code, it appears that F_NOCACHE is set if RTFILE_O_NO_CACHE is requested for Darwin, however this option doesn't seem to be used.

It does appear that the existing code expects that this will disable write buffering and immediately flush data to disk on a write when this is set, which is slightly worrying because that is not at all what this flag does (it marks the pages associated with the vnode that the file is attached to as inactive so that they are top of the list to evict when the system is short of RAM).

[Frank Mehnert]

Regarding the host cache: The guest cannot optimize the disk access in the same degree as the host because the guest has just no information about the layout of the physical hard disk. Successive disk sectors of the guest hard disk which are mapped to successive sectors within the virtual disk file on the host are not necessarily mapped to successive disk sectors on the host hard disk. So doing the elevator algorithm isn't possible. This is one reason why you need a cache on the host.

[Sander van Leeuwen]

I couldn't quickly find a confirmation for the F_NOCACHE behaviour you mention. If true, then it's very different from other host platforms.

On Windows disabling the cache affects both read and write operations. The same goes for Linux with synchronous direct writes. Hence the need for an intermediate cache.

[Christian Pötzsch]

About the F_FULLFSYNC: see ​fcntl and ​fsync. There is nothing stated from "guarantee" but tighter guarantees. Also is there clearly mentioned that there are drives which ignore that request. After all the same count for other OSs and shouldn't make any real difference.