Taking snapshot deadlock

[a.urakov]

There is possible deadlock when taking snapshot of one machine. Deadlock occurs due to wrong XPCOM IPC service queue processing.

When calling IConsole::takeSnapshot() then IProgress object is returned and client calls IProgress::waitForCompletion(). But there are many calls of Progress::SetNextOperation() and when one of these calls comes a bit later than IProgress::waitForCompletion() then described situation may occur.

Let's there is one worker thread of XPCOM IPC service. So when IProgress::waitForCompletion() request comes to ipcDConnectService::OnMessageAvailable() it appends it to request queue and yields the CPU. If worker's thread has not received a time slice then it will not get a request and next check will say that there is already one waiting worker and no need for one more. Second request of Progress::SetNextOperation() comes shortly after that and situation repeats. So we have two requests and one worker thread. If worker gets IProgress::waitForCompletion() request first then it indefinitely waits because this progress object can't complete (Progress::SetNextOperation() can't be processed). So deadlock occurs.

In attached log you can find that request 17537 (IProgress::waitForCompletion()) is executed but 17539 (Progress::SetNextOperation()) is never executed.

Thank you in advance!

[a.urakov]

VBoxHeadless log

[frank]

Fix is part of VBox 4.3.22.