VirtualBox

Ticket #813 (closed defect: fixed)

Opened 6 years ago

Last modified 6 years ago

VRDP connection error (VERR_VRDP_PROTOCOL_ERROR) in XP guest/XP host -> Fixed in 1.5.4.

Reported by: softgosh Owned by:
Priority: major Component: other
Version: VirtualBox 1.5.2 Keywords: VRDP protocol error XP
Cc: Guest type: other
Host type: other

Description

When attempting to connect to a running virtual machine with Remote Desktop Client from WinXP SP2 (version 5.1.2600.2180), the client hangs for several minutes and finally aborts. The following is extracted from the VM log file:

00:05:27.519 VRDP: New connection: 00:05:38.022 VRDP: Channel: [rdpdr] [1004]. Not supported. 00:05:38.022 VRDP: Channel: [cliprdr] [1005]. Accepted. 00:05:38.022 VRDP: Channel: [rdpsnd] [1006]. Accepted. 00:05:38.051 VRDP: Failed to process incoming RDP packet: VERR_VRDP_PROTOCOL_ERROR!!! 00:05:38.080 VRDP: The RDP packet content: 00:05:38.080 00:05:38.080 0ef74634 0000: 03 00 01 4f 02 f0 80 64-00 01 03 eb 70 81 40 48 ...O...d....p.@H 00:05:38.080 0ef74644 0010: 00 00 00 0c 04 0c 04 b3-43 00 00 9c ec f4 cc 5c ........C......\ 00:05:38.080 0ef74654 0020: ac bc 8c 1c 2c 34 4c 9b-2c 53 18 af 48 54 08 9c ....,4L.,S..HT.. 00:05:38.080 0ef74664 0030: ec 44 44 9c ac 04 5c ce-ec e0 cc 35 dc d1 ec 2e .DD...\....5.... 00:05:38.080 0ef74674 0040: 6c c6 20 8d dc 4a cc d8-ec b2 50 88 ec 52 d8 2d l. ..J....P..R.- 00:05:38.080 0ef74684 0050: c0 8a f4 2b 8c dc 8c 14-9c 8b 18 e6 a0 2c 7c 6b ...+.........,|k 00:05:38.080 0ef74694 0060: ac b1 d8 36 18 8c f4 d7-50 d7 d0 ff 10 14 48 76 ...6....P.....Hv 00:05:38.080 0ef746a4 0070: 54 fd 84 87 10 e4 8c 31-c8 d1 00 bf 64 ee fc f0 T......1....d... 00:05:38.080 0ef746b4 0080: dc 75 20 af e0 9c 54 f3-ac 23 18 49 8c a8 b8 a2 .u ...T..#.I.... 00:05:38.080 0ef746c4 0090: 10 bc 94 78 58 7c 10 34-78 f4 13 33 73 7c 80 39 ...xX|.4x..3s|.9 00:05:38.080 0ef746d4 00a0: 48 36 10 85 f0 37 14 d8-06 f8 b8 f9 84 cd f4 04 H6...7.......... 00:05:38.080 0ef746e4 00b0: dc f6 90 2d 18 5c 80 20-44 06 14 20 84 50 24 d4 ...-.\. D.. .P$. 00:05:38.080 0ef746f4 00c0: 8c 18 58 24 38 ec d8 b8-18 06 2c 70 80 a0 24 24 ..X$8.....,p..$$ 00:05:38.080 0ef74704 00d0: 64 b8 c8 90 8c e8 64 b0-7c dc 38 ac 30 78 44 d2 d.....d.|.8.0xD. 00:05:38.080 0ef74714 00e0: 68 d8 34 bd 08 4b a8 08-e8 74 10 08 d8 a8 d8 58 h.4..K...t.....X 00:05:38.080 0ef74724 00f0: 08 58 05 d9 a8 aa 1c b5-dc f7 e0 24 bc f4 ac 55 .X.........$...U 00:05:38.080 0ef74734 0100: 05 a9 a4 ac 08 92 b0 bd-ec 64 80 38 dc 8c 9c e0 .........d.8.... 00:05:38.080 0ef74744 0110: 70 08 d0 48 f0 b8 28 a8-06 d8 24 e0 e0 28 28 c8 p..H..(...$..((. 00:05:38.080 0ef74754 0120: 58 3c dc 06 18 48 f0 d8-28 08 ec 08 c8 90 68 18 X<...H..(.....h. 00:05:38.080 0ef74764 0130: dc e0 05 df 68 34 b8 55-5c da ec 05 ec 58 18 24 ....h4.U\....X.$ 00:05:38.080 0ef74774 0140: ec c1 87 1f 5b c0 f0 2c-38 1b c0 e0 24 05 ec ....[..,8...$.. 00:05:38.080 00:05:38.081 VRDP: Connection closed: 00:05:38.081 VRDP: Logoff: PEDRO (<NULL>) build 2600. User: [<NULL>] Domain: [<NULL>] Reason 0xFFFFFFFF.

Attachments

Web Dev-2007-10-24-19-23-23.log Download (31.0 KB) - added by softgosh 6 years ago.
Full log file
Web Dev-2007-10-25-16-59-59.log Download (31.1 KB) - added by softgosh 6 years ago.
Full log file using supplied VBoxVRDP.dll
Web Dev-2007-10-25-18-44-04.log Download (29.2 KB) - added by softgosh 6 years ago.
With second dll patch.
Web Dev-2007-10-25-20-25-58.log Download (34.9 KB) - added by softgosh 6 years ago.
New log file with more detailed VRDP trace
Web Dev-2007-10-25-21-38-54.log Download (38.4 KB) - added by softgosh 6 years ago.
Web Dev-2007-10-26-12-14-11.log Download (51.4 KB) - added by softgosh 6 years ago.
VBoxVRDP.dll Download (364.0 KB) - added by sunlover 6 years ago.
vboxservice.7z Download (2.5 KB) - added by softgosh 6 years ago.
vboxclipb.zip Download (34.8 KB) - added by sunlover 6 years ago.

Change History

comment:1 in reply to: ↑ description ; follow-up: ↓ 2 Changed 6 years ago by sunlover

Does this happen always? Or there are also successful connects?

comment:2 in reply to: ↑ 1 Changed 6 years ago by softgosh

Replying to sunlover:

Does this happen always? Or there are also successful connects?

No successful connection so far, all fail following the same pattern.

comment:3 follow-up: ↓ 4 Changed 6 years ago by sunlover

I've tried the same RDP client and can't reproduce the problem.

Which RDP options do you use for the connection?

Also please attach the complete VBox.log.

Changed 6 years ago by softgosh

Full log file

comment:4 in reply to: ↑ 3 Changed 6 years ago by softgosh

Which RDP options do you use for the connection?

Tried with all without success, log file is generated with the Modem (28.8 Kbits/s) one. Tried also to quit my firewall but it does not change anything.

comment:5 follow-up: ↓ 6 Changed 6 years ago by sunlover

I've attached a test version of the VRDP server. Replace existing VBoxVRDP.dll with the new one. Then please reproduce the problem and attach the new VBox.log.

Changed 6 years ago by softgosh

Full log file using supplied VBoxVRDP.dll

comment:6 in reply to: ↑ 5 Changed 6 years ago by softgosh

I replaced the file VBoxVRDP.dll in the install directory (C:\Program Files\innotek VirtualBox) by the one supplied. Restarted VirtualBox and generated the log file attached. Hope this helps.

comment:7 follow-up: ↓ 8 Changed 6 years ago by sunlover

I've updated the dll again. Please retry. Attach the VBox.log if it still fails.

Changed 6 years ago by softgosh

With second dll patch.

comment:8 in reply to: ↑ 7 Changed 6 years ago by softgosh

Still fails but no more silently. I get an immediate Remote Desktop error pop-up saying : "Client could not connect to distant computer due to a security error. Check you are connected to the network and try again" (translated from french) I am trying to connect locally, from the machine running Virtualbox (localhost:9696) and from another machine and get the same behaviour in both cases. Authentification method for VRDP is set to "null" in the preference dialog.

comment:9 follow-up: ↓ 10 Changed 6 years ago by sunlover

Thanks. Please make the log with the updated dll.

Changed 6 years ago by softgosh

New log file with more detailed VRDP trace

comment:10 in reply to: ↑ 9 Changed 6 years ago by softgosh

Same symptoms, log file updated. Thanks !

comment:11 follow-up: ↓ 12 Changed 6 years ago by sunlover

Thanks for testing. Here is another dll to try.

Changed 6 years ago by softgosh

comment:12 in reply to: ↑ 11 Changed 6 years ago by softgosh

Interesting new symptoms:

  • no more error message
  • once vm has booted, host cpu is loaded at 100% by virtualbox
  • once RDP connection is initiated, cpu is loaded 70% by mstsc.exe, 30% vitualbox
  • the connection is not established but cannot be canceled either

comment:13 Changed 6 years ago by sunlover

Good, thanks for the information. Please try the new dll.

Changed 6 years ago by softgosh

comment:14 Changed 6 years ago by softgosh

Congratulations, the connection has opened a window ! However, both when launched with gui or using vboxvrdp command I observe the same phenomenon:
as soon as the VM is started, host cpu in stuck at 100% (95% used by virtualbox). This continues until the rdp connection is initiated, then virtualbox cpu usage drops to 5-10% range. I attached a new log if it helps. Many thanks for your support.

Changed 6 years ago by sunlover

comment:15 follow-up: ↓ 16 Changed 6 years ago by sunlover

Very good, thanks for testing.

I've updated the dll once again. Removed excessive logging and cleaned the code. Please try it to make sure I did not break anything.

comment:16 in reply to: ↑ 15 Changed 6 years ago by softgosh

Behavior is the same. VRDP connection is functional (great !) but CPU usage is not normal. I have investigated a little more :

Once the VM is fully booted both host and guest CPUs are 100% loaded. On the guest, the vboxservice process is the culprit. I left the PC alone for 10 minutes without a change, the situation seems to be stable.

Doing anything either on the guest or on the host machine is painfull (firefox takes 1.5 minutes to launch on the guest). As soon as a VRDP session is initiated, everything goes to normal and stays normal even after the session has been closed.

Last point, if I disable the VRDP server before starting the machine, the CPU consumption is at 100% and I did not find any way to have it back to normal. In this state, both guest and host machines are barely usable.

Any hint to overcome this ?

comment:17 follow-up: ↓ 19 Changed 6 years ago by sunlover

Well, there seems to be a problem in the guest additions (see ticket #824). But I can't reproduce it here (XP host/XP guest as well).

Try to resize the guest window when the CPU is 100%. Does it have any effect?

comment:18 follow-up: ↓ 21 Changed 6 years ago by klaus

Do you have shared clipboard enabled? If that malfunctions it can also block things.

comment:19 in reply to: ↑ 17 Changed 6 years ago by softgosh

Try to resize the guest window when the CPU is 100%. Does it have any effect?

No effect on my machine.

comment:20 follow-up: ↓ 22 Changed 6 years ago by sunlover

Here is a debug version of the vboxservice. It will write log to vboxservice.dbg at the root of the system drive. Copy it over the one in the guest system32. Start the VM again to reproduce the 100% CPU. Please send me the log then.

comment:21 in reply to: ↑ 18 Changed 6 years ago by softgosh

Do you have shared clipboard enabled? If that malfunctions it can also block things.

Yes, it was enabled, bidirectional.

Disabling it completely or enabling only one direction (either way) seems to cure the problem. Klaus, thanks for the trick. Sunlover, if you need a beta tester, I have some time at the moment ;-)

Changed 6 years ago by softgosh

comment:22 in reply to: ↑ 20 Changed 6 years ago by softgosh

Replying to sunlover:

Here is a debug version of the vboxservice. It will write log to vboxservice.dbg at the root of the system drive. Copy it over the one in the guest system32. Start the VM again to reproduce the 100% CPU. Please send me the log then.

I re-enabled bidirectional clipboard, boot the VM.

As long as I do not initiate the VRDP connection, the log file grows and the CPU is stuck at 100%.

As soon as I hit "connect" on remote desktop client, CPU cools down and log stops to grow. Log attached, enjoy !

comment:23 follow-up: ↓ 24 Changed 6 years ago by sunlover

Thanks! The problem is identified. Please try the new attached vboxservice.

comment:24 in reply to: ↑ 23 Changed 6 years ago by softgosh

Replying to sunlover:

Thanks! The problem is identified. Please try the new attached vboxservice.

Thanks to you ! Everything looks OK now. It was a pleasure to get your support. I am amazed by your reactivity. I guess those fixes will find their way in a 1.5.3 release ? Cheers.

Changed 6 years ago by sunlover

comment:25 follow-up: ↓ 27 Changed 6 years ago by sunlover

Please try the attached VBoxSharedClipboard.dll (host) and VBoxService.exe (guest). They include the right fix for the 100% CPU problem. Let me know whether it works.

comment:26 Changed 6 years ago by frank

Note softgosh: The only valid answer here is 'Yes, it works fine' :)

comment:27 in reply to: ↑ 25 Changed 6 years ago by softgosh

I downloaded and tested your last fix. RDP connection works both from localhost and over the network. Conclusion is "Yes, it works fine" ;-)

However, now I moved to my second step, pluging a logitech webcam to test some video security application on the virtual machine. It seems that USB video streaming freezes the VM. I guess I should open another ticket for that... Thanks for fixing RDP.

comment:28 Changed 6 years ago by sunlover

  • Status changed from new to closed
  • Resolution set to fixed
  • Summary changed from VRDP connection error (VERR_VRDP_PROTOCOL_ERROR) in XP guest/XP host to VRDP connection error (VERR_VRDP_PROTOCOL_ERROR) in XP guest/XP host -> Fixed in 1.5.4.

Good. Both RDP and VBoxServioce fixed will be included in 1.5.4.

About the USB problem. I believe there are already similar tickets, take a look at #242 for example. Please post your findings there.

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use