VirtualBox

Ticket #14218 (closed defect: worksforme)

Opened 7 years ago

Last modified 7 years ago

VBoxSVC-Restart triggered from a shutdown of a Headless-VM: syslog:NS_ERROR_ABORT-COMGETTER VBoxSVC.log:VBOX_E_OBJECT_IN_USE

Reported by: pm_bernhard Owned by:
Component: VM control Version: VirtualBox 4.3.28
Keywords: VBoxSVC VBOX_E_OBJECT_IN_USE NS_ERROR_ABORT COMGETTER Cc:
Guest type: Linux Host type: Linux

Description

We have a reproducable Crash (Powerdown) of ALL running VM's when one VM shuts down triggered from our backup scripts.
The process VBoxSVC terminates and restarts itself, but the other VM's stay offline.
Because it occours on different virtual machines (backuped allways in the same order) we guess it could be a race condition in the process VBoxSVC.
It does not matter if we shutdown the machine internal or with 'vboxmanage controlvm vm_pm01 poweroff'

Environment:
Host: Debian 8 (Jessie) 64Bit
Guests: Ubuntu 12.04, Ubuntu 14.04, Debian 8, all without guest additions
VirtualBox: from Oracle-Repository 4.3.28 + ExtensionPack
(Same behavior occoured with the virtualbox version delivered from debian)

All Processes are startet as root

Maybe important: All VM's use a vmdk with direct physical hard disk access (to a LogicalVolume).
I.e. The file pm01_disk_description.vmdk was created with 'vboxmanage internalcommands createrawvmdk -filename pm01_disk_description.vmdk -rawdisk /dev/vg_md0/lv_pm01'

Attachments

syslog Download (1.8 KB) - added by pm_bernhard 7 years ago.
/var/log/syslog
VBoxSVC.log.1 Download (3.6 KB) - added by pm_bernhard 7 years ago.
/root/.config/VirtualBox/VBoxSVC.log.1
2015_10_30_VBoxSVC.log Download (4.0 KB) - added by pm_bernhard 7 years ago.
2015_10_30_syslog Download (2.8 KB) - added by pm_bernhard 7 years ago.
2015_11_02_gdb.log Download (4.2 KB) - added by pm_bernhard 7 years ago.
2015_11_02_syslog Download (5.6 KB) - added by pm_bernhard 7 years ago.
2015_11_02_VBoxSVC.log Download (3.7 KB) - added by pm_bernhard 7 years ago.

Change History

Changed 7 years ago by pm_bernhard

/var/log/syslog

Changed 7 years ago by pm_bernhard

/root/.config/VirtualBox/VBoxSVC.log.1

comment:1 Changed 7 years ago by pm_bernhard

Bug still exists in package 5.0.4-102546~Debian~jessie

comment:2 Changed 7 years ago by frank

Please describe the problem in more detail. You talk about a crash and a powerdown in the same sentence. If VBoxSVC crashes, then please provide a core dump.

comment:3 Changed 7 years ago by pm_bernhard

Sorry for the late reply frank, i configured the mailnotification incorrect.

I upgraded to version 5.0.8 and tried to get a coredump of VBoxSVC. When started from the commandline without the usual "--auto-shutdown" parameter VBoxSVC does not crash but it still stops working.

Please correct my interpretation of the problem:

  • our scripts starts via vboxmanage all headless frontents vm_svn01, vm_web01, ...
  • the first process of the virtual machine also starts the process VBoxSVC which is needed for internal communication
  • our script updates all machines via ssh-commands, reboot the machines, power down them to to create lvm-snapshots of the raw disk-data,...
  • at some point of the procedure (not everytime on the same position) the vboxmanage command fails.
  • At that time there is a VBOX_E_OBJECT_IN_USE in the logfile VBoxSVC.log
  • Now the Process VBoxSVC exits somehow (no crash, as i have seen).
  • The processes of ALL other currently running machines also stops now somehow (found no core dumps).

I attached the file /var/log/syslog and VBoxSVC.log from our problem occurred today while executing the backup/update-scripts.

Changed 7 years ago by pm_bernhard

Changed 7 years ago by pm_bernhard

comment:4 Changed 7 years ago by frank

Ok, useful information but probably not yet complete.

First, if you start VBoxSVC in a separate terminal without any parameter, that process should never terminate. If it does, this is probably the first thing we should debug. You say it does not crash and the syslog indeed doesn't mention any crash. Also the VBoxSVC.log mentions the HostPowerServiceLinux destructor which is only called on a clean shutdown.

Are you 100% sure that you start VBoxSVC as separate process without any parameters and that this process terminates eventually?

Here are the 5.0.8 debug symbols for the Jessie package. Could you install this package, then start VBoxSVC as separate process with gdb and set a break point at HostPowerServiceLinux::~HostPowerServiceLinux()? Of course make sure that VBoxSVC and the VBoxManage call are started as the same user on the same machine!

If the breakpoint is hit I would be interested to see the backtrace.

comment:5 Changed 7 years ago by pm_bernhard

Sorry, my explanation was not exact. The logfiles i attached last time ( 2015_10_30*) were created from a regular startup, not from the manual startup of VBoxSVC.

I attached now logfiles of the following procedure:

  • all processes are started as user root on the same machine (i will improve that in future)
  • VBoxSVC is started via gdb from the commandline without any other parameters.
  • VBoxSVC does not end until i quit the debugger
  • The breakpoint was reached an i created a callstack: 2015_11_02_gdb.log
  • I also attached the logfiles of VBoxSVC and syslog, but there is nothing new in it.
  • Please keep in mind that the last lines in VBoxSVC.log are created after the first error.

Changed 7 years ago by pm_bernhard

Changed 7 years ago by pm_bernhard

Changed 7 years ago by pm_bernhard

comment:6 Changed 7 years ago by frank

Thanks! After more thinking and checking your backtrace it turned out that I was partly wrong.

It is true that VBoxSVC should not terminate but some services will indeed stop a short time after the last client vanished. The VBoxSVC process even shows log file for these cases (make sure to start it in a separate terminal as you did before): Informational: VirtualBox object created (rc=NS_OK). means that the service is ready for service client requests. The VirtualBox object will remain active as long as at least one client stays connected to the service. For instance, if you start the VM selector it will keep this object reference open.

When the last client terminated, VBoxSVC will print a message Informational: VirtualBox object deleted.. At this time the breakpoint HostPowerServiceLinux::~HostPowerServiceLinux() is hit. That's completely normal.

Back to your original problem: Messages like ERROR [COM]: aRC=VBOX_E_OBJECT_IN_USE (0x80bb000c)... are also completely normal although I admit that observing such messages is confusing. Just do the experiment: Start VBoxSVC manually in a terminal, then open the VM selector. Do

tail -f /root/.config/VirtualBox/VBoxSVC.log

Now terminate the VM selector. After a few seconds (~5) you will see the Informational: VirtualBox object deleted. message appear on the VBoxSVC terminal and you will also see a couple of aRC=VBOX_E_OBJECT_IN_USE messages in the VBoxSVC log. Now, the next client (VM selector or just VBoxManage) will re-open the VirtualBox client object before it closes again when no client is longer available.

I fear we would need a scenario which allows us to reproduce your problem. In ideal case you could provide scripts which execute VBoxManage commands which fail at some time (without ssh involved if that's possible).

comment:7 Changed 7 years ago by pm_bernhard

ok, thank you for your explanations. we try to create such a script.

comment:8 Changed 7 years ago by pm_bernhard

Hello frank, we tried to isolate a simple case but we were not successful. We switched now to KVM via virsh on our server.

Thank you very much for your support! From our side the ticket can be closed.

comment:9 Changed 7 years ago by frank

  • Status changed from new to closed
  • Resolution set to worksforme

So closing. If someone else experiences the same problem: We are still looking for a reproduction scenario.

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use