VirtualBox

Ticket #13901 (closed defect: fixed)

Opened 3 years ago

Last modified 3 years ago

rdesktop-vrdp segmentation fault when attaching remote USB devices

Reported by: carlcarl Owned by:
Priority: major Component: RDP
Version: VirtualBox 4.3.22 Keywords: rdesktop-vrdp USB redirect
Cc: Guest type: Windows
Host type: Linux

Description

I tried to access the Win7 guest on CentOS 5 host from another machine running CentOS 5 using "rdesktop-vrdp -r usb". VirtualBox 4.3.22 and the extension Pack was installed on host and the VRDP client.

USB filter was defined for a USB HID TouchPad that I want to use when I access the Win7 from the VRDP client. I have tested the TouchPad on the VM host and the Win7 guest can use it without any problem.

Then I access the Win7 guest via VRDP. Everything still fine. Then I plugged the TouchPad to the VRDP client machine, the rdesktop-vrdp crash with "segmentation fault" returned.

On the host side, I got a pop-up dialog showing "NS_ERROR_FAILURE ... Failed to create a proxy device for the USB device..."

Following is the log from VBox.log on the host:

VRDP: New connection:
VRDP: Connection opened {IPv6}: 16
VRDP: Negotiating security method with the client.
VRDP: Methods 0x00000003
VRDP: Channel: [cliprdr] [1004]. Accepted.
VRDP: Channel: [rdpsnd] [1005]. Accepted.
VRDP: Channel: [snddbg] [1006]. Not supported.
VRDP: Channel: [vrdpusb] [1007]. Accepted.
VRDP: Channel: [rdpdr] [1008]. Accepted.
VRDP: Client seems to be rdesktop.
VRDP: Logon: vmclient01 {192.168.1.10} build 2600. User: [root] Domain: [] Screen: 0
AUTH: User: [root]. Domain: []. Authentication type: [Null]
AUTH: Access granted.
VBVA: VRDP acceleration has been requested.
Remote USB: Received negotiate response. Flags 0x00.
VRDP: remote USB protocol version 1.
Remote USB: ++++ Vendor 04CA. Product 0061. Name = [USB Optical Mouse].
Remote USB: ++++ Vendor 413C. Product 2107. Name = [Dell USB Entry Keyboard].
Remote USB: ++++ Vendor 1267. Product 0701. Name = [TouchPad].
ERROR [COM]: aRC=NS_ERROR_FAILURE {0x80004005} aIID={8ab7c520-2442-4b66-8d74-4ff1e195d2b6} aComponent={Console} aText={Failed to create a proxy device for the USB device. {Error: VERR_READ_ERROR}}, preserve=false

I then fallback to use version 4.2.18, and finally can use the USB redirect function after searched for many articles/blogs/form posts. But required some USB device operations manually. As it was another issue, maybe I will discuss about it in forum or create another ticket.

Really hope the VBox experts to help, so that I can use the powerful VRDP USB redirect function.

Attachments

core-rdesktop-vrdp-11-0-0-3168-1425295936.gz.split.01 Download (512.0 KB) - added by carlcarl 3 years ago.
Core dump of rdesktop-vrdp (Part 1 of 2)
core-rdesktop-vrdp-11-0-0-3168-1425295936.gz.split.02 Download (501.9 KB) - added by carlcarl 3 years ago.
Core dump of rdesktop-vrdp (Part 2 of 2)

Change History

comment:1 Changed 3 years ago by frank

To debug this we need a core dump of the crashing rdesktop process.

Changed 3 years ago by carlcarl

Core dump of rdesktop-vrdp (Part 1 of 2)

Changed 3 years ago by carlcarl

Core dump of rdesktop-vrdp (Part 2 of 2)

comment:2 Changed 3 years ago by carlcarl

The core dump is attached. But since it is too large to be attached in a single file, I gzip it and the split it into two files using "split" command. Please let me know if you cannot open it, and tell me how to send to you in another way.

comment:3 Changed 3 years ago by frank

I had a look at your core dump but it seems that my system is too difference from yours. Could you actually do the following?

  • compile rdesktop-vrdp.tar.gz from /usr/share/virtualbox
  • start rdesktop-vrdp in gdb and on crash enter 'bt' to get a backtrace

and post the resulting backtrace here. Thank you!

comment:4 Changed 3 years ago by carlcarl

I have tried to compile the rdesktop-vrdp.tar.gz from /usr/share/virtualbox, but failed to make. I got the following error messages:

Runtime/r3/posix/path2-posix.o: In function `RTPathSetTimesEx':
/usr/share/virtualbox/rdesktop-1.7.0-vrdp/Runtime/r3/posix/path2-posix.cpp:196: warning: warning: lutimes is not implemented and will always fail
Runtime/r3/dir.o: In function `rtDirOpenCommon':
/usr/share/virtualbox/rdesktop-1.7.0-vrdp/Runtime/r3/dir.cpp:529: undefined reference to `RTPathAbs'
/usr/share/virtualbox/rdesktop-1.7.0-vrdp/Runtime/r3/dir.cpp:543: undefined reference to `RTPathAbs'
Runtime/r3/dir.o: In funtion `RTDirCreateFullPath':
/usr/share/virtualbox/rdesktop-1.7.0-vrdp/Runtime/r3/dir.cpp:64: undefined reference to `RTPathAbs'
collect2: ld returned 1 exit status
make: *** [rdesktop] Error 1

Did I miss anything?

comment:5 Changed 3 years ago by carlcarl

I finally found the RTPathAbs-generic.cpp from the virtualbox source tarball. I included it and compiled successfully.

I start rdesktop-vrdp in gdb and got the following:

(gdb) run -g 1024x768 -a 24 -r usb 192.168.1.1:3391
Starting program: /usr/share/virtualbox/rdesktop-1.7.0-vrdp/rdesktop -g 1024x768 -a 24 -r usb 192.168.1.1:3391
[Thread debugging using libthread_db enabled]
Autoselected keyboard map en-us

Program received signal SIGSEGV, Segmentation fault.
usbProxyLinuxOpen (pProxyDev=0x844f840, pszAddress=0x844e638 "sysfs:/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2//device:/dev/vboxusb/001/009",
    pvBackend=0x0) at vrdp/linux/USBProxyDevice-linux.cpp:652
652             RTListInit(&pDevLnx->ListFree);
(gdb) bt
#0 usbProxyLinuxOpen (pProxyDev=0x844f840, pszAddress=0x844e638 "sysfs:/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2//device:/dev/vboxusb/001/009",
    pvBackend=0x0) at vrdp/linux/USBProxyDevice-linux.cpp:652
#1 0x0808646b in op_usbproxy_back_open (s=0x8117880) at vrdp/rdpusb.c:102
#2 rdpusb_process (s=0x8117880) at vrdp/rdpusb.c:489
#3 0x08062f55 in sec_recv (rdpver=0xbfff921f "\003p\347\377\277t\347\377\277X\342\377\277XR\006\b\200x\021\bp\222\377\277") at secure.c:837
#4 0x08064b12 in rdp_recv (type=0xbfffe24b "") at rdp.c:122
#5 0x08065278 in rdp_loop (deactivated=0xbfffe774, ext_disc_reason=0xbfffe770) at rdp.c:1638
#6 0x0806606c in rdp_main_loop (deactivated=0xbfffe774, ext_disc_reason=0xbfffe770) at rdp.c:1619
#7 0x0806e66a in main (argc=8, argv=0xbfffe824) at rdesktop.c:1088
(gdb) print pDevLnx
$1 = (USBPROXYDEVLNX *) 0x0
(gdb)

comment:6 Changed 3 years ago by frank

Thank you for this analysis! We have fixed at least the rdesktop-vrdp.tar.gz archive to compile properly. Sorry for not noticing this; we compile the application for the packages using a different mechanism.

As for the backtrace: The crash makes much sense (pDevLnx=NULL) but it needs some deeper analysis why this happens here.

comment:7 Changed 3 years ago by carlcarl

I found the code segment that cause the pDevLnx=NULL in function rdpusb_process(STREAM s) of vrdp/rdpusb.c (line 466):

    proxy = (PUSBPROXYDEV) xmalloc (sizeof USBPROXYDEV);
    if (!proxy)
    {
     ....
    }

    proxy->pvInstanceDataR3 = xmalloc(g_USBPorxyDeviceHost.cbBackend);
    if (!proxy->pvInstanceDataR3)
    {
     ....
    }

    memset (proxy, 0, sizeof(USBPROXYDEV));   // <-- this line put pDevLnx=NULL !!!!

    ...

I think that code is used to initialize the new object proxy, but unfortunately it was done after another object is allocated into proxy.

Then I fixed it and tried again to see if USB Redirect work or not. Then I got another bug...double free!! But this time, gdb only listed out some Linux library call when backtrace.

So I did another deep analysis again, and found that it was caused by the function call xfree(pUrb) in function rdpusb_reap_urbs(void) in vrdp/rdpusb.c (line 419).

rdpusb_reap_urbs(void)
{
...
    PVUSBRB pUrb = NULL;
...
    while (...)
    {
        pUrb = op_usbproxy_back_reap_urb(proxy, 0);
        
        if (pUrb)
        {
            ...
            xfree(pUrb);   // <-- program aborted with double free error here !!!
        }
    }
}

As I am still trying to understand the USB Proxy mechanism, I have no idea how to fix it yet.

comment:8 Changed 3 years ago by frank

Thanks for debugging! Actually after having a closer look I think I found the problem. Apparently this code wasn't tested for a while...

comment:9 Changed 3 years ago by frank

  • Status changed from new to closed
  • Resolution set to fixed

Fix is part of VBox 4.3.26. Please reopen if necessary!

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use