[vbox-dev] SOLVED: VirtualBox Teleporting and SOAP API "waitForCompletion"

Sun Sep 19 01:18:24 GMT 2010

Hi All,

I had the chance to follow-up and do some very extensive testing and with 
Klaus's insight via our discussion on IRC, have found the error of my ways ... 
Thanks Klaus.  Your feedback was instrumental in helping me find the problem!!!

For anyone else that might be interested, I'll describe the mistake I made and 
outline the steps that through testing I have confirmed works every time.

First:  What am I trying to do and why:
On occasion, though not often, I need to take a live host off-line (to upgrade 
memory, replace/change internal hardware, etc).  These live hosts can have 
anywhere from 9 - 15 VMs running at a time.  To take this host offline without 
powering-down the VMs, the VirtualBox teleport feature is ideal.  My goal then 
was to teleport each of the VMs from the existing host to another host (must be 
on the same network and have access to the same networked storage subsystem 
where the virtual drives are).  VBoxManage does provide a command line option to 
initiate the teleport request but does require coordinating the execution on 
both the source and target hosts.  As I have control of all the VirtualBox hosts 
via the SOAP Webservice APIs, I decided to automate the entire process.

What I was doing wrong:
The steps I outline in the original message (see below) are actually correct 
except for Step #7.  The call to power-up the VM with a new Session via the 
IConsole object returnes a progressRef object and I assumed it was necessary to 
wait for completion before moving forward.  THIS IS WRONG!  In reality, the 
correct way to determine the VM is ready for the teleport-in step is to watch 
the state of the target VM and wait for the state to change to 'TELEPORTINGIN'.  
Once in this state, the teleport step on the source VM can be done.

The correct steps to follow therefore are:

Target Host:
----------------
    1) Get a new ISession for the machine to be teleported
    2) Get the refID for the machine from the ISession object
    3) set-up for teleport (set teleporterEnabled, teleporterAddress, 
teleporterPort, teleporterPassword values)
    4) Save the machine settings
    5) Close the ISession
    6) Power-Up the VM (get new ISession, etc) (though progressRef is returned, 
ignore it!!!)

    7) For a configured duration (I am using 10 seconds), check the state of the 
VM every second.  When the VM state changes to 'TELEPORTINGIN, the target VM is 
ready for the teleport operation.

Source Host:
----------------
    8) Get Console for source VM
    9) Call Console::teleport
For Step #7, if a timeout occurs for any reason, or if VirtualBox determines the 
teleport cannot be done, or if Step #9 fails, your error handling must include 
resetting the IMachine::teleporterEnabled value to false for the target VM (or 
use VirtualBox modifyvm --teleporter off).  If this step is not done, any 
subsequent attempt to start the target VM will immediately cause it to go into 
the TELEPORTINGIN state and the VM will not start!

After extensive testing, I have confirmed the automated teleport of all running 
VMs from a source host to a target host works every time.

A few interesting observations:
a) Teleporting VMs sequentially (i.e: waiting for a teleport to complete before 
initiating the next teleport) works consistently faster than trying to teleport 
multiple VMs in parallel.  From what Klaus has explained (or at least what I 
believe I understand from him), a fair amount of network traffic is used during 
the teleport process.  Additionally, CPU resources are heavily relied-on also.  
Add to that the network traffic for the virtual disk I/O for the machine being 
teleported and all other VMs that share the same networked storage subsystem, it 
seems even with a 1GB pipe that the network could possibly be part of the 
bottleneck.  I'll try to do some testing in the future to see if I can uncover 
more details here.

b) The target VM typically takes ~3 seconds to be in the TELEPORTINGIN state 
though I have seen it sometimes as high as 6 seconds.  My hosts are quite busy 
and think that might be part of the reason.

Hope this helps someone.  Thanks again Klaus!

Joe

________________________________
From: Joseph Smith <joseph1962smith at yahoo.com>
To: vbox-dev at virtualbox.org
Sent: Mon, September 6, 2010 9:24:25 AM
Subject: [vbox-dev] VirtualBox Teleporting and SOAP API "waitForCompletion"

Hi,

I searched bugtracker and did not see this reported .. so I believe this may be 
a new problem.

Version: 3.2.8 PUEL
Host-1: openSuSE 11.1
Host-2: openSuSE 11.2
Host-3: Windows XP
Host-4: Mac OSX 10.6.4
VMs: WinXP, openSuSE, RedHat, FreeBSD, Vista, OpenSolaris, etc

I have not had to teleport any VM's for a while until just recently and have 
encountered a problem that I do not recall happening before.  It seems calling 
progressRef::waitForCompletion with a -1 (means wait forever) during a power-up 
for a teleport operation never returns.  This is using the PHP Wrappers and my 
".ini" settings for my script are set to never  time-out.

The steps I follow are:

Target Host:
----------------
1) Get a new ISession for the machine to be teleported
2) Get the refID for the machine from the ISession object
3) set-up for teleport (set teleporterEnabled, teleporterAddress, 
teleporterPort, teleporterPassword values)
4) Save the machine settings
5) Close the ISession
6) Power-Up the VM (get new ISession, etc) (progressRef is returned)
7) Call progressRef::waitForCompletion (-1)

Source Host:
----------------
8) Get Console for source VM
9) Call Console::teleport

Step #7 will wait forever (ok .. so I only waited for up-to 10 minutes) and it 
does not come back.  This problem happens regardless of the OS running in the 
guest that is being teleported and regardless of the  host on which I originate 
the request.

At the same time, on the target host, the console shows "Waiting for incoming VM 
… (2/3)" in a modal dialog .. and then eventually aborts from a timeout.

I do not recall this happening in v3.1.8 (might be wrong here) and am not really 
in a position to downgrade to confirm.

In playing around, I have found that by setting my timeout long enough (I'm 
using 10,000ms now), though progressWait returns after the timeout (not because 
of completion but because of reaching the end of the timeout), the VM teleport 
is actually ready to continue and I can go ahead and complete the teleport 
process by calling Console::teleport on the source.  The teleport will then 
complete successfully.  If I set my timeout much less than 10sec (say 3sec - 
5sec) .. attempting to continue will result in an exception and the teleport 
will fail.  Depending  on what's happening on the target host, the time the 
target VM can be ready to teleport that I have observed can vary from 3sec to as 
much as 8sec.

Though setting the timeout long-enough works .. depending on the speed of the 
target system, the time to prep for the teleportation does vary.

Has anyone seen this problem?  Is there something else I need to be checking for 
or doing?  I'll wait to see what responses I get to determine if this should be 
opened in bugtracker.

Thanks,
Joe

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20100918/d7f0ac81/attachment.html>