VirtualBox

Ticket #7981 (closed defect: fixed)

Opened 3 years ago

Last modified 3 years ago

rdesktop-vrdp crashes with memory corruption - regression from 3.2.12-68302 -> fixed as of 14 Jan 2011

Reported by: kronenpj Owned by:
Priority: major Component: RDP
Version: VirtualBox 4.0.0 Keywords: regression
Cc: Guest type: Linux
Host type: Linux

Description

rdesktop-vrdp -r usb {host}

Autoselected keyboard map en-us
WARNING: Remote desktop changed from 800x600 to 1020x550.
*** glibc detected *** rdesktop-vrdp: free(): invalid next size (fast): 0x000000000236a930 ***
*** glibc detected *** rdesktop-vrdp: malloc(): memory corruption: 0x000000000236a950 ***

From here the window doesn't update and sometimes causes the rdesktop client (Fedora 14) to freeze / crash.

Problematic rdesktop-vrdp is from VirtualBox-4.0-4.0.0_69151_fedora14-1.x86_64.rpm. Previous rdesktop-vrdp from VirtualBox-3.2-3.2.12_68302_fedora14-1.x86_64.rpm works perfectly.

Attachments

backtrace.txt Download (14.3 KB) - added by kronenpj 3 years ago.
Backtrace from failed session.
VBox.log Download (46.7 KB) - added by kronenpj 3 years ago.
VBox log file from last comment.
valgrind.log Download (37.0 KB) - added by kronenpj 3 years ago.
Valgrind output with debug-enabled binary - updated 7-Jan-2011

Change History

comment:1 Changed 3 years ago by frank

Any easy way to reproduce this crash?

comment:2 Changed 3 years ago by kronenpj

The glibc memory corruption occurs almost immediately, but only when redirecting USB devices. I have only seen the system freeze symptom a few times and not recently.

comment:3 Changed 3 years ago by kronenpj

Ok, there may be another factor involved. Either it's due to my USB scanner (from NeatWorks) or the fact that I'm using IPv6 at home and not from work... Let me investigate further and report back.

Changed 3 years ago by kronenpj

Backtrace from failed session.

comment:4 Changed 3 years ago by kronenpj

I've tested a few scenarios and it's not IPv6. I've attached a backtrace though I don't get one every time. The command that generated it was: $ rdesktop-vrdp -r usb -z -P -a 16 host:3407
Though the options don't seem to matter much but the USB device connection / disconnection is important.
Let me stop rambling...

Here's a sequence of steps with outcomes:

Start the VM (WinXP SP3) and attach rdesktop-vrdp -r usb host:3407 while the VBox splash screen was still visible. XP saw the USB devices.
Unplug the scanner and rdesktop crashes.
Re-connected rdesktop, crashed almost immediately.
Re-plugged in the scanner and reconnected rdesktop, it was fine but no USB devices appeared.
Rebooted Win XP, still no USB devices.
Completely shut down VM, started VM and reconnected with scanner attached, devices are seen.
Detached rdesktop, unplugged scanner, re-connected rdesktop. Crash again.
Detached rdesktop, re-plug scanner, connect with rdesktop-vrdp from 3.2_68302.
Do all of the above stuff, never crashes, all works as expected.
Detached rdesktop, re-connect with 4.0's rdesktop-vrdp and with scanner connected, seems happy.

Hopefully this is something you can use. I can test with a debug version (64-bit) or try other combinations.

Changed 3 years ago by kronenpj

VBox log file from last comment.

comment:5 Changed 3 years ago by michael

If you can reproduce this using

 http://www.virtualbox.org/download/testcase/rdesktop-vrdp

which contains debug symbols, could you try running it in valgrind to see what output is produced? Thanks.

comment:6 Changed 3 years ago by michael

Did the run under Valgrind also abort with the error you reported initially? I am slightly confused, as I can't see any trace of it in the output.

comment:7 Changed 3 years ago by michael

And could you also provide the output of "VBoxManage list usbhost" with and without the scanner plugged in, in a situation in which you can reproduce the crash? Thanks again.

comment:8 Changed 3 years ago by kronenpj

I honestly don't remember... Sorry. I did get it to crash this time, so I'll replace the log file. Looks much more interesting. I'll try to get the list usbhost, but it doesn't give me a lot of time before it bombs...

Changed 3 years ago by kronenpj

Valgrind output with debug-enabled binary - updated 7-Jan-2011

comment:9 Changed 3 years ago by kronenpj

Unless I'm mistaken, list usbhost only shows the USB devices actually attached to the host. I have two of them, but they're attached to a different VM and they're not the scanner involved. That's why I'm using RDP with USB extensions. :)

comment:10 Changed 3 years ago by michael

You are right of course - is it possible to install VirtualBox on the system you are attaching the scanner to, in order to do that test?

comment:11 Changed 3 years ago by michael

Although hopefully the Valgrind output will be enough to fix this - this time it does look like what I need.

comment:12 Changed 3 years ago by kronenpj

Um yes, though it'll be *much* slower. I'll see what I can do and get back to you.

comment:13 Changed 3 years ago by michael

Don't worry, I think I have fixed this locally. More to follow.

comment:14 Changed 3 years ago by michael

I have uploaded a version of VirtualBox containing a hopefully fixed copy of rdesktop-vrdp. This is a test build from the stable branch of VirtualBox (disclaimer here). It is a distribution-agnostic build with a shell script installer. To unpack it without installing, do

$ ./VirtualBox-*.run --keep --noexec

which will create a subdirectory called "install" in the current directory with the installer files, including a tar.bz2 containing the actual executables. The URL for the test build is

 http://www.virtualbox.org/download/testcase/VirtualBox-2011-01-13-16-01-33-lin64-rel-4.0.1-r69417.run

comment:15 Changed 3 years ago by kronenpj

My testing shows that this looks like it fixed it. I've pulled and re-inserted the scanner a few times, no crashes. Thanks!

comment:16 Changed 3 years ago by michael

  • Summary changed from rdesktop-vrdp crashes with memory corruption - regression from 3.2.12-68302 to rdesktop-vrdp crashes with memory corruption - regression from 3.2.12-68302 -> fixed as of 14 Jan 2011

Thanks for testing.

comment:17 Changed 3 years ago by frank

  • Status changed from new to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use