Multiple Instances of VBoxXPCOMIPCD, VBoxSVC, VBoxHeadless

[mh367]

Hello!

After upgrading from debian 8/jessie running virtualbox 5.1.24 to debian 9/stretch and virtualbox 5.1.26, multiple processes (at least two) for a single virtual machine may be started (with two seperate VBoxSVC-processes as their parents).

The commands "vboxmanage showvminfo" and also "vboxmanage list runningvms" sometimes report a wrong - "powered off" or "aborted" - state for running and fully functional machines.

Several Win10 and Win8 guests are not able to start anymore due to severe filesystem damages.

The relevant part of "ps aufx" looks something like this:

...
vbox      1788  Sl   /usr/lib/virtualbox/vboxwebsrv --background -F /var/log/vboxweb.log -R 10 -S 1048576 -I 86400
vbox      1800  S    /usr/lib/virtualbox/VBoxXPCOMIPCD
vbox      1809  Sl   /usr/lib/virtualbox/VBoxSVC --auto-shutdown
vbox      3834  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM221 --startvm 5e5c64e7-...
vbox      4093  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM320 --startvm cfe3de02-...
vbox      4186  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM050 --startvm 43d2653e-...
vbox      4187  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM222 --startvm 6206dddc-...
vbox      4441  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM223 --startvm 056443b9-...
vbox      4936  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM057 --startvm 0bd9f77d-...
vbox     18145  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM322 --startvm 0429d803-...
vbox      2083  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM321 --startvm 5b36d602-...
...
vbox      2169  S    /usr/lib/virtualbox/VBoxXPCOMIPCD
vbox      2174  Sl   /usr/lib/virtualbox/VBoxSVC --auto-shutdown
vbox      2218  Sl   \_ /usr/lib/virtualbox/VBoxHeadless --comment VM322 --startvm 0429d803-...
...

in this excerpt "VM322" is running as PID 18145 and 2218.

The host system is a DELL PowerEdge T620 server powered by two Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz running Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux. Guests are about 36 machines running MS Windows (Win2000 - Win10).

A downgrade to the virtualbox 5.1.24 deb-package installed in debian "stretch" did not solve the problem.

Thanks a lot for any advice!

[Frank Mehnert]

Are you using the distribution-provided VirtualBox package? Perhaps there is a bug in the package. Please try to use the VirtualBox package for Debian 9 from the official Oracle download server.

Please do also check the permissions of your /tmp directory, it should be set to 'drwxrwxrwt'. Furthermore, the /tmp directory should contain a directory .vbox-vbox-ipc (the 2nd 'vbox' is the user how is running a VM) and that directory should contain 'ipcd' and 'lock'. That path is also shown in /home/vbox/.VirtualBox/VBoxSVC.log.

On which file system is your /tmp directory located?

[mh367]

Hi! Thanks a lot for your reply!

I'm using the packages from ​http://download.virtualbox.org/virtualbox/debian (stretch, contrib/non-free).

The permissions of the "/tmp" directory already match your suggestion. The "/tmp" directory is a regular directory (not a mount-point) within a ext4 filesystem mounted as root ("/").

A directory "/tmp/.vbox-vbox-ipc" exists and also contains a socket "ipcd" and a regular file "lock". The lockfile contains a pid that matches one of the (currently two running) VBoxXPCOMIPCD processes. Both running VBoxXPCOMIPCD processes have been started as user "vbox" (according to "ps aufx") as shown in my previous post.

But there is also a "/tmp/.vbox-root-ipc" directory containing socket and lockfile (with non-existing pid) - both with quite new timestamps. I didn't start vbox as root, but I do have some shell-scripts called via cron parsing the output of

sudo -u vbox vboxmanage showvminfo ... --machinereadable 

for monitoring purposes (and restarting aborted machines). These scripts have been running for years, but maybe calls like "sudo -u vbox vboxmanage" could cause this issue in debian stretch?

[Frank Mehnert]

The /tmp/.vbox-root-ipc directories are definitely the problem. You need to make sure that the USER environment variable is always set to 'vbox' when you start any VBox service. I assume that this doesn't work properly (wrong USER variable) if you start any service from a script. Keep in mind that you can also display the active environment variables for a certain process. I assume that USER=root for one VBoxSVC prorcess while USER=vbox for the other VBoxSVC process and that means trouble.

[mh367]

Hi! Thanks again for your help.

You're right: The USER environment variable is the problem. One of the VBoxXPCOMIPCD processes has been started with an empty USER variable.

I haven't yet figured out why this happens - i.e. the PATH and HOME variables are both set properly. But it's definitely not a virtualbox bug. So this ticket may be closed.

However, being able to start a guest multiple times simultaneously is really critical - it will take me a long time to repair and restore all the damaged file-systems. Perhaps is would be good to have a slightly more "robust" locking mechanism - a locking flag in the vhds, in the vms' definition or a lock-file in the vms' directory maybe.

[Frank Mehnert]

Locking is the real problem. You cannot reliably lock on Linux file systems if you need to consider file systems like NFS etc. And putting the .vbox files on NFS is possible and is actually done by certain users...

[Klaus Espenlaub]

Strange. I thought that for a long time VirtualBox wasn't depending on the content of $USER any more for figuring out the IPC stuff. It was converted to getting the numeric uid and and resolving the name for it with getpwuid() or something similar. Only if this fails (and it really shouldn't on a properly installed OS) then the environment variable $LOGNAME or $USER are used as emergency fallbacks.

Sure that you're not messing with other environment variables which define which IPC socket is used (i.e. $VBOX_IPC_SOCKETID)? If you search for that environment variable you'll quickly find references how it can be used to run multiple VirtualBox configurations from one user account.

[Frank Mehnert]

For reference, here​ is the corresponding code.