Using SaveState is corrupting disk image when VirtualBox has been opened more than once (by multiple users)

[jphiggs]

We have had a clients production VM disk image get corrupted twice so have spent a fair amount of time investigating the issue and trying to reproduce it. Can now reproduce fairly easily (see below).

Corruption original occurred on the client machine configured as followed:
Host = Win Server 2007 Web edition
Guest = Centos 6.5
Virtual Box 4.3.12-93773
Other Info = On the client machine we are running VBoxVMService to run the VM as a service on the host. Which can result in two VBoxSVC processes running if the service is running and you open the VBox control panel GUI.

I have also been able to reproduce corrupting the VM disk on my dev machine just by using the Virtual box control panel as two separate users:
Host = Windows 8.1 Pro 64
Guest = Centos 6.5
VBox Version = 4.3.12-93773 and also uninstalled and reinstalled latest version (VBox 4.3.26) and could still reproduce.

The easiest way to reproduce the problem is:

1. Open Vbox as userA and start the virtual machine.
2. Open Vbox as userB (this creates a second VBoxSVC.exe process)
3. Stop the VM as userA using SaveState (which saves the state of the VM to a .SAV file in the snapshots folder). See attached screenshot 'userA.png' for what the control panel looks like at this point.
4. Swap to userB and you will see in the control panel it doesn't show the saved state of the VM. (see screenshot userB.png).
5. Start the VM using the userB control panel. Which starts the VM from scratch, and does NOT use the saved state of the VM from userA.
6. Make any changes to the VM – e.g. change something in the DB, generate some files etc
7. Stop the VM in control panel (userB) using either full shutdown or save state
8. Swap back to userA and start VM again - this will load the VM’s savestate from Step 3, and will apply it to the disk file which has been changed by step 4-7, which results in corrupting the disk image.
9. On restarting the guest OS, it does a disk scan and shows the file system is corrupted (see screenshot DiskCorruption.png). How bad the corruption is seems to depend on the amount of changes made to the disk in steps 4-7 above.

Basically anytime there are two VBoxSVC.exe processes running at the same time, if one of them saves the state of the VM, the other process doesn't know about this saved state, so the 2nd proces then start the VM and make changes to the disk, which can then corrupt the disk when the 1st process starts the VM again.

The biggest issue can see with this is where VirtualBox is running on a server which multiple users log into the server, and some users leave the control panel open. When a user stops the VM using savestate other users who had the control panel open could modify the disk without loading the saved state.

From hunting around online it looks like others people have had similar issues where two separate processes modify the disk image where one process stopped the VM using save state. Here is example of similar issue where savestate corrupts the VM, but in a very different scenario. ​https://forums.virtualbox.org/viewtopic.php?f=1&t=56109

Have also attached the VBox.log from the last time the VM was started (where it shows the disk corruption).

[jphiggs]

Log file for when starting the VM when it shows the disk corruption

[jphiggs]

Showing state of VM in control panel for user a

[jphiggs]

Showing state of VM in control panel for user b

[jphiggs]

Showing disk corruption in VM.

[jphiggs]

Also as an extra note. On some client servers have noticed that we can't open the VBoxVmService and the VBox Control panel at the same time. I.e. On some machines it looks like we can't have two VBoxSVC.exe processes running at the same time. On these machines we need to fully stop the one process before starting the other.

The advantage that when doing this, we aren't able to corrupt the disks because starting the second the process after the first one is closed means the 2nd process can see the savestate of the first process. So starting the 2nd process loads the save state from process 1.

The downside is that we have to fully stop the service before we can open the control panel.

Also the main issue is that on other machines it is possible to have two VBoxSVC processes running at the same time. Have spent several hours trying to restrict / limit this so we can only have one VBoxSVC process running at any point in time (to at least stop risk of corrupting the disk again), but I haven't been able to find how to prevent multiple VBoxSVC processes from starting.

[Frank Mehnert]

Running two or more VBoxSVC processes on the same host is no problem as long as these processes use different home directories. I don't know your setup (we don't support VBoxVMService) but the VBoxSVC.log files for all VBoxSVC processes should show which settings they use.

To state it clearly, one VBox home directory (usually c:\Users\USERNAME\.VirtualBox) must never be accessed by more than one VBoxSVC instance. And from your description it sounds like you are doing exactly this and this leads to the data corruption you report here.