rdesktop-vrdp segfaults in XPutImage on Debian Squeeze/i386

[Maximilian Gass]

I created a Debian package for rdesktop-vrdp from VirtualBox 3.1.6. It works fine on a Debian Squeeze amd64 workstation, but segfaults after login on a Debian Squeeze i386 live system I am creating.

I have localized the problem in ui_desktop_restore in xwin.c. The call to XCreateImage returns NULL sometimes and then XPutImage triggers a segmentation fault.

(gdb) bt
#0  0xb75f7d92 in XPutImage (dpy=0x9cad588, d=4194309, gc=0x9cc4688, image=0x0, req_xoffset=0, req_yoffset=0, x=184, 
    y=89, req_width=432, req_height=234) at ../../src/PutImage.c:967
#1  0x080523b8 in ui_desktop_restore (offset=0, x=184, y=89, cx=432, cy=234) at xwin.c:3595
#2  0x0806726f in process_desksave (s=0x80c0600, num_orders=43) at orders.c:384
#3  process_orders (s=0x80c0600, num_orders=43) at orders.c:1256
#4  0x0806c77e in rdp5_process (s=0x80c0600) at rdp5.c:94
#5  0x08063528 in rdp_recv (type=0xbfbb84ef "") at rdp.c:110
#6  0x08063cb4 in rdp_loop (deactivated=0xbfbb888c, ext_disc_reason=0xbfbb8888) at rdp.c:1420
#7  0x0806489c in rdp_main_loop (deactivated=0xbfbb888c, ext_disc_reason=0xbfbb8888) at rdp.c:1405
#8  0x0804df46 in main (argc=12, argv=0xbfbb8934) at rdesktop.c:994

If you need more details, please let me know.

[Maximilian Gass]

Some more infos: The problem also happens with a recent trunk checkout of rdesktop and if the live system is amd64.

[Frank Mehnert]

We can't reproduce this problem, tried this with Debian/unstable. The function in question (ui_desktop_restore()) is not changed ​upstream. Your backtrace shows an image parameter of 0 which is probably the cause of the segfault so adding a simple sanity check might solve this problem. As your backtrace is probably not from rdesktop-vrdp I assume your backtrace was taken with Debians rdesktop.

[Maximilian Gass]

No, I did use rdesktop-vrdp. I built my own package based on the tarball included with VirtualBox. But I *also* tried with upstream's trunk, same problem.

XCreateImage returns NULL if the arguments are invalid. Adding a simple sanity check will create artifacts on the screen because parts are not redrawn properly.

I have been trying to get more details with gdb, but so far it's been pretty fruitless.

[Frank Mehnert]

Before trying out a fix I would like to reproduce the problem. Any idea how to reproduce the problem? The regular Lenny rdesktop should have the same problem...

[Maximilian Gass]

Unfortunately, as of yet, I have no idea why the problem happens on the live system, but not on an installed Debian system. live-helper uses vanilla Debian packages, so there are no patches applied to X libraries or anything.

[Frank Mehnert]

Perhaps a different video mode / video driver.

[Maximilian Gass]

I now tried to use the live system including rdesktop-vrdp on real hardware (before that I used QEMU/KVM). The problem just does not exist there.

[Maximilian Gass]

Debian bug describing the same problem in vanilla rdesktop: ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=437664