rdesktop connecting to VRDP crashes guest VM

[jbrown10]

After upgrading to 4.2.4 from 4.2.2 I have begun to experience periodic events where a guest vm crashes as soon as I connect to it via rdesktop.

syslog contains the following record:

kernel: [61012.245046] VRDP-OUT[9395]: segfault at 1400000004 ip 00007f8f622a4f92 sp 00007f8f2a847b10 error 6 in libc-2.15.so[7f8f62222000+1b5000]

Host system: Ubuntu 12.10 64-bit
Linux explorer 3.5.0-18-generic #29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Guest OS: Windows 7 64-bit

rdesktop: Version 1.7.1.

Log snippet from the vm beginning where the connection comes in:

03:59:16.128063 VRDP: New connection: 
03:59:16.172195 VRDP: Methods 0x00000003
03:59:16.172234 VRDP: Channel: [cliprdr] [1004]. Accepted.
03:59:16.172256 VRDP: Channel: [rdpdr] [1005]. Accepted.
03:59:16.323294 VRDP: Client seems to be rdesktop.
03:59:16.323337 VRDP: Logon: explorer (127.0.0.1) build 2600. User: [jpb] Domain: [] Screen: 0
03:59:16.324743 AUTH: User: [jpb]. Domain: []. Authentication type: [External]
03:59:16.325248 AUTH: ConsoleVRDPServer::Authenticate: loading external authentication library 'VBoxAuth'
03:59:16.325908 AUTH: Using entry point 'AuthEntry'.
03:59:16.563754 AUTH: external authentication module returned 'access granted'
03:59:16.563783 AUTH: Access granted.
03:59:16.568895 VBVA: VRDP acceleration has been requested.
03:59:17.620661 VRDP: SunFlsh disabled.

The log terminated with the last line above and the entry in syslog appeared.

To recreate:

1. Activate RDP on a Windows 7 64-bit VM.
   1. port 33881
   2. authentication method External
   3. authentication timeout 5000
   4. allow multiple connections checked
2. Start the VM
3. Open a linux command line on the host
4. Attempt to connect to the guest:

   /var/log$ rdesktop -u <userid> -p <password> localhost:33881
   

Output in the command window:

Autoselected keyboard map en-us
ERROR: recv: Connection reset by peer
ERROR: send: Broken pipe

Syslog:

Oct 31 19:45:08 explorer VirtualBox: pam_ecryptfs: seteuid error
Oct 31 19:45:08 explorer kernel: [85957.901303] VRDP-OUT[22947]: segfault at 1400000004 ip 00007fe1f82baf92 sp 00007fe1c09319a0 error 6 in libc-2.15.so[7fe1f8238000+1b5000]

I was able to repeat this with a Ubuntu 12.10 64-bit guest vm as well.

It appears to be a problem with the reported pam error and then an unhandled exception coming from the VRDPAUTH.

Changing the authentication method from External to Null for RDP and all works fine since it doesn't access pam via VRDPAUTH.

[jbrown10]

Additional information: 4.2.0 and 4.2.2 worked under Ubuntu 12.10. I ran 4.2.0 for several days under 12.10 and then upgraded to 4.2.2. Since 4.2.4 RDP has crashed the Vbox guest when using external authentication.

[Frank Mehnert]

Can you provide a core dump?

[jbrown10]

Replying to frank:

Can you provide a core dump?

Dump sent to Frank M per the core dump instructions.

[Frank Mehnert]

That looks like memory corruption so unfortunately the information in the core dump is not sufficient. We have to reproduce this problem. Can you give any hints how to faster trigger this bug with an Ubuntu guest? So far we had no success reproducing this on a 32-bit 12.10 host (I know, your host is 64-bit). Or does every attempt to connect to the VM via RDP trigger this crash?

[Frank Mehnert]

We also cannot reproduce the crash with a 64-bit Linux host. To me it seems that the memory corruption is caused by some defined guest screen update and the crash with external authentication is only a crash due to the memory corruption.

[jbrown10]

Replying to frank:

We also cannot reproduce the crash with a 64-bit Linux host. To me it seems that the memory corruption is caused by some defined guest screen update and the crash with external authentication is only a crash due to the memory corruption.

Can you give any hints how to faster trigger this bug with an Ubuntu guest? So far we had no success reproducing this on a 32-bit 12.10 host (I know, your host is 64-bit). Or does every attempt to connect to the VM via RDP trigger this crash?

Great question on frequency. It has been very consistent for me. I did just attempt it and interestingly, the first 3 attempts were successful. I began the attempts when the VBox Bios screen started and then closed the RDP window, re-established it as it got past the main boot but while it was still coming up and closed it, and finally re-established it again when the desktop was showing in the guest (Ubuntu 12.10 64-bit). It was working so I thought it might have been that I missed setting the External setting and shut down the OS. After validating that I did have the External authentication set, I started the VM and tried the sequence again. This time, it killed immediately. And for the next 7 tries, it killed immediately. I tried at different points of the boot from the bios through desktop to see if there was a point that I had launched rdesktop that didn't cause a crash -- all failed.

Next I proceeded to another VM (still linux 64-bit) on a different port, launched it, attempted an RDP connection and it instantly crashed the VM.

I also performed a "/etc/init.d/vboxdrv force-reload" and then using the ubuntu VM that had 3 sequential successful RDP connections followed by crashing with each of the next 7 RDP connections. This time the first attempt to RDP to it crashed immediately.

Next, thinking back to what I did just prior to the first successful RDP connection, I recalled I had just gone in and activated the remote display. So, I unchecked the "enable server" on the remote display tab, did a vboxdrv force-reload and launched the ubuntu VM again. This time instead of failing, I again had success. First time just after the bios displayed and each time after that, it worked fine (still the VM's initial launch.

I then preceded to shut down the VM from within the Guest (using the normal shutdown in the guest) and launched it again. This time the first attempt at RDP connection crashed the vm. Each subsequent attempt caused it to crash.

Next, I again did a vboxdrv force-reload and attempted to do an RDP connection. This again crashed immediately. To further validate the sequence of activities, I then went into the VBox GUI and unchecked the enable server setting, clicked OK, and then rechecked it, clicked OK. Restarted the VM and RDP immediately crashed the vm.

Going back to the original sequence that seemed to recover it - uncheck enable server, click OK, force-reload vboxdrv, check enable server, click OK... This time first attempt crashed. Attempted this sequence again and it crashed again.

Though I had something there, but to make sure I didn't miss anything -- I had a number of other VM's running prior to my initial vboxdrv force-reload that I had shut down, I ran through one more scenario:
1) Start a different VM with "Enable Server" checked.
2) Uncheck "Enable Server" in the Ubuntu VM that is the test vm and click OK.
3) Shutdown the VM from step 1
4) vboxdrv force-reload
5) check "Enable Server" in the Ubuntu VM and click OK.
6) Launch Ubuntu VM.
7) Attempt RDP access -- result: successful connection
8) shutdown Ubuntu VM via normal desktop shutdown from within the guest.
9) Launch Ubuntu VM again.
10) Attempt RDP access -- result: crash

Using the above sequence which involves 2 different VM's, I was able to consistently reproduce this crash event.

[barjohn]

I can confirm that this bug exists and is reproducable.

It has been happening to me all morning with a freshly upgraded host.

What I have noticed is that it is happening ( in my data center ) on hosts with AMD processors and it is not happening on Intel processors. (Occurs on Dell r415 and not on dell poweredge 2950's or intel based supermicros)

Host: Debian Squeeze with backported linux 3.2.0 kernel

Provision a windows server 2008 guest and connect with VRDE during initial startup. Disconnect then reconnect and the VM will crash. I am in the process of rolling back to 4.1 until this is resolved for this particular host.

[jbrown10]

Replying to barjohn:

What I have noticed is that it is happening ( in my data center ) on hosts with AMD processors and it is not happening on Intel processors. (Occurs on Dell r415 and not on dell poweredge 2950's or intel based supermicros)

Thanks for mentioning this. I didn't include this in my original post, but my host is also an AMD chip (Phenom II X3 720 processor). I'm running with 16GB ram on the host.

[sunlover]

jbrown10, here is an extpack build with detailed VRDP logging:

​http://www.virtualbox.org/download/testcase/Oracle_VM_VirtualBox_Extension_Pack-4.2.5r82022-log.vbox-extpack

Could you please install it, reproduce the crash and then send VBox.log to Vitali dot Pelenjow at oracle dot com

Thanks!

[jbrown10]

Replying to sunlover:

jbrown10, here is an extpack build with detailed VRDP logging:

​http://www.virtualbox.org/download/testcase/Oracle_VM_VirtualBox_Extension_Pack-4.2.5r82022-log.vbox-extpack

Could you please install it, reproduce the crash and then send VBox.log to Vitali dot Pelenjow at oracle dot com

Thanks!

I installed this extension pack (4.2.5) on top of the 4.2.4 VB install and so far have not been able to reproduce the crash. I reverted back to the 4.2.4 extension pack and instantly crashed the VM. Reinstalling the 4.2.5 extension pack, I have not been able to get it to crash. I'll continue to try.

[jbrown10]

Replying to jbrown10:

I installed this extension pack (4.2.5) on top of the 4.2.4 VB install and so far have not been able to reproduce the crash. I reverted back to the 4.2.4 extension pack and instantly crashed the VM. Reinstalling the 4.2.5 extension pack, I have not been able to get it to crash. I'll continue to try.

I did manage to get a crash. Log sent to Vitali.

[Florentino]

I'm having the same issue here, however it only affects headless, not frontend started VMs. My host is a AMD Opteron running CentOS 6. VirtualBox 4.2.2 Extension Pack 4.2.2

[jbrown10]

Since 11/17/2012 I've been running with this version of the Extension Pack: ​http://www.virtualbox.org/download/testcase/Oracle_VM_VirtualBox_Extension_Pack-4.2.5r82068.vbox-extpack

To date, I have had 0 crashes. Vitali tells me it handles stack space differently than before when using external authentication.

[Frank Mehnert]

Thanks for the feedback. The fix is included in VBox 4.2.6.