VRDP connection error (VERR_VRDP_PROTOCOL_ERROR) in XP guest/XP host -> Fixed in 1.5.4.

[Pierre]

When attempting to connect to a running virtual machine with Remote Desktop Client from WinXP SP2 (version 5.1.2600.2180), the client hangs for several minutes and finally aborts. The following is extracted from the VM log file:

00:05:27.519 VRDP: New connection: 00:05:38.022 VRDP: Channel: [rdpdr] [1004]. Not supported. 00:05:38.022 VRDP: Channel: [cliprdr] [1005]. Accepted. 00:05:38.022 VRDP: Channel: [rdpsnd] [1006]. Accepted. 00:05:38.051 VRDP: Failed to process incoming RDP packet: VERR_VRDP_PROTOCOL_ERROR!!! 00:05:38.080 VRDP: The RDP packet content: 00:05:38.080 00:05:38.080 0ef74634 0000: 03 00 01 4f 02 f0 80 64-00 01 03 eb 70 81 40 48 ...O...d....p.@H 00:05:38.080 0ef74644 0010: 00 00 00 0c 04 0c 04 b3-43 00 00 9c ec f4 cc 5c ........C......\ 00:05:38.080 0ef74654 0020: ac bc 8c 1c 2c 34 4c 9b-2c 53 18 af 48 54 08 9c ....,4L.,S..HT.. 00:05:38.080 0ef74664 0030: ec 44 44 9c ac 04 5c ce-ec e0 cc 35 dc d1 ec 2e .DD...\....5.... 00:05:38.080 0ef74674 0040: 6c c6 20 8d dc 4a cc d8-ec b2 50 88 ec 52 d8 2d l. ..J....P..R.- 00:05:38.080 0ef74684 0050: c0 8a f4 2b 8c dc 8c 14-9c 8b 18 e6 a0 2c 7c 6b ...+.........,|k 00:05:38.080 0ef74694 0060: ac b1 d8 36 18 8c f4 d7-50 d7 d0 ff 10 14 48 76 ...6....P.....Hv 00:05:38.080 0ef746a4 0070: 54 fd 84 87 10 e4 8c 31-c8 d1 00 bf 64 ee fc f0 T......1....d... 00:05:38.080 0ef746b4 0080: dc 75 20 af e0 9c 54 f3-ac 23 18 49 8c a8 b8 a2 .u ...T..#.I.... 00:05:38.080 0ef746c4 0090: 10 bc 94 78 58 7c 10 34-78 f4 13 33 73 7c 80 39 ...xX|.4x..3s|.9 00:05:38.080 0ef746d4 00a0: 48 36 10 85 f0 37 14 d8-06 f8 b8 f9 84 cd f4 04 H6...7.......... 00:05:38.080 0ef746e4 00b0: dc f6 90 2d 18 5c 80 20-44 06 14 20 84 50 24 d4 ...-.\. D.. .P$. 00:05:38.080 0ef746f4 00c0: 8c 18 58 24 38 ec d8 b8-18 06 2c 70 80 a0 24 24 ..X$8.....,p..$$ 00:05:38.080 0ef74704 00d0: 64 b8 c8 90 8c e8 64 b0-7c dc 38 ac 30 78 44 d2 d.....d.|.8.0xD. 00:05:38.080 0ef74714 00e0: 68 d8 34 bd 08 4b a8 08-e8 74 10 08 d8 a8 d8 58 h.4..K...t.....X 00:05:38.080 0ef74724 00f0: 08 58 05 d9 a8 aa 1c b5-dc f7 e0 24 bc f4 ac 55 .X.........$...U 00:05:38.080 0ef74734 0100: 05 a9 a4 ac 08 92 b0 bd-ec 64 80 38 dc 8c 9c e0 .........d.8.... 00:05:38.080 0ef74744 0110: 70 08 d0 48 f0 b8 28 a8-06 d8 24 e0 e0 28 28 c8 p..H..(...$..((. 00:05:38.080 0ef74754 0120: 58 3c dc 06 18 48 f0 d8-28 08 ec 08 c8 90 68 18 X<...H..(.....h. 00:05:38.080 0ef74764 0130: dc e0 05 df 68 34 b8 55-5c da ec 05 ec 58 18 24 ....h4.U\....X.$ 00:05:38.080 0ef74774 0140: ec c1 87 1f 5b c0 f0 2c-38 1b c0 e0 24 05 ec ....[..,8...$.. 00:05:38.080 00:05:38.081 VRDP: Connection closed: 00:05:38.081 VRDP: Logoff: PEDRO (<NULL>) build 2600. User: [<NULL>] Domain: [<NULL>] Reason 0xFFFFFFFF.

[sunlover]

Does this happen always? Or there are also successful connects?

[Pierre]

Replying to sunlover:

Does this happen always? Or there are also successful connects?

No successful connection so far, all fail following the same pattern.

[sunlover]

I've tried the same RDP client and can't reproduce the problem.

Which RDP options do you use for the connection?

Also please attach the complete VBox.log.

[Pierre]

Full log file

[Pierre]

Which RDP options do you use for the connection?

Tried with all without success, log file is generated with the Modem (28.8 Kbits/s) one. Tried also to quit my firewall but it does not change anything.

[sunlover]

I've attached a test version of the VRDP server. Replace existing VBoxVRDP.dll with the new one. Then please reproduce the problem and attach the new VBox.log.

[Pierre]

Full log file using supplied VBoxVRDP.dll

[Pierre]

I replaced the file VBoxVRDP.dll in the install directory (C:\Program Files\innotek VirtualBox) by the one supplied. Restarted VirtualBox and generated the log file attached. Hope this helps.

[sunlover]

I've updated the dll again. Please retry. Attach the VBox.log if it still fails.

[Pierre]

With second dll patch.

[Pierre]

Still fails but no more silently. I get an immediate Remote Desktop error pop-up saying : "Client could not connect to distant computer due to a security error. Check you are connected to the network and try again" (translated from french) I am trying to connect locally, from the machine running Virtualbox (localhost:9696) and from another machine and get the same behaviour in both cases. Authentification method for VRDP is set to "null" in the preference dialog.

[sunlover]

Thanks. Please make the log with the updated dll.

[Pierre]

New log file with more detailed VRDP trace

[Pierre]

Same symptoms, log file updated. Thanks !

[sunlover]

Thanks for testing. Here is another dll to try.

[Pierre]

Interesting new symptoms:

• no more error message
  
• once vm has booted, host cpu is loaded at 100% by virtualbox
  
• once RDP connection is initiated, cpu is loaded 70% by mstsc.exe, 30% vitualbox
  
• the connection is not established but cannot be canceled either
  

[sunlover]

Good, thanks for the information. Please try the new dll.

[Pierre]

Congratulations, the connection has opened a window ! However, both when launched with gui or using vboxvrdp command I observe the same phenomenon:
as soon as the VM is started, host cpu in stuck at 100% (95% used by virtualbox). This continues until the rdp connection is initiated, then virtualbox cpu usage drops to 5-10% range. I attached a new log if it helps. Many thanks for your support.

[sunlover]

Very good, thanks for testing.

I've updated the dll once again. Removed excessive logging and cleaned the code. Please try it to make sure I did not break anything.

[Pierre]

Behavior is the same. VRDP connection is functional (great !) but CPU usage is not normal. I have investigated a little more :

Once the VM is fully booted both host and guest CPUs are 100% loaded. On the guest, the vboxservice process is the culprit. I left the PC alone for 10 minutes without a change, the situation seems to be stable.

Doing anything either on the guest or on the host machine is painfull (firefox takes 1.5 minutes to launch on the guest). As soon as a VRDP session is initiated, everything goes to normal and stays normal even after the session has been closed.

Last point, if I disable the VRDP server before starting the machine, the CPU consumption is at 100% and I did not find any way to have it back to normal. In this state, both guest and host machines are barely usable.

Any hint to overcome this ?

[sunlover]

Well, there seems to be a problem in the guest additions (see ticket #824). But I can't reproduce it here (XP host/XP guest as well).

Try to resize the guest window when the CPU is 100%. Does it have any effect?

[Klaus Espenlaub]

Do you have shared clipboard enabled? If that malfunctions it can also block things.

[Pierre]

Try to resize the guest window when the CPU is 100%. Does it have any effect?

No effect on my machine.

[sunlover]

Here is a debug version of the vboxservice. It will write log to vboxservice.dbg at the root of the system drive. Copy it over the one in the guest system32. Start the VM again to reproduce the 100% CPU. Please send me the log then.

[Pierre]

Do you have shared clipboard enabled? If that malfunctions it can also block things.

Yes, it was enabled, bidirectional.

Disabling it completely or enabling only one direction (either way) seems to cure the problem. Klaus, thanks for the trick. Sunlover, if you need a beta tester, I have some time at the moment ;-)

[Pierre]

Replying to sunlover:

Here is a debug version of the vboxservice. It will write log to vboxservice.dbg at the root of the system drive. Copy it over the one in the guest system32. Start the VM again to reproduce the 100% CPU. Please send me the log then.

I re-enabled bidirectional clipboard, boot the VM.

As long as I do not initiate the VRDP connection, the log file grows and the CPU is stuck at 100%.

As soon as I hit "connect" on remote desktop client, CPU cools down and log stops to grow. Log attached, enjoy !

[sunlover]

Thanks! The problem is identified. Please try the new attached vboxservice.

[Pierre]

Replying to sunlover:

Thanks! The problem is identified. Please try the new attached vboxservice.

Thanks to you ! Everything looks OK now. It was a pleasure to get your support. I am amazed by your reactivity. I guess those fixes will find their way in a 1.5.3 release ? Cheers.

[sunlover]

Please try the attached VBoxSharedClipboard.dll (host) and VBoxService.exe (guest). They include the right fix for the 100% CPU problem. Let me know whether it works.

[Frank Mehnert]

Note softgosh: The only valid answer here is 'Yes, it works fine' :)

[Pierre]

I downloaded and tested your last fix. RDP connection works both from localhost and over the network. Conclusion is "Yes, it works fine" ;-)

However, now I moved to my second step, pluging a logitech webcam to test some video security application on the virtual machine. It seems that USB video streaming freezes the VM. I guess I should open another ticket for that... Thanks for fixing RDP.

[sunlover]

Good. Both RDP and VBoxServioce fixed will be included in 1.5.4.

About the USB problem. I believe there are already similar tickets, take a look at #242 for example. Please post your findings there.