[vbox-dev] Guests abort when resizing

[Klaus Espenlaub]

On 25.07.2012 16:21, Perry Halbert wrote:
> gdb shows:
>
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007f717ea0f4e2 in VNCServerImpl::VRDEUpdate
> (hServer=0x7f7178002bc0, uScreenId=<optimized out>,
> pvUpdate=0x7f7153fe0338, cbUpdate=<optimized out>) at
> /trunk/src/VBox/ExtPacks/VNC/VBoxVNC.cpp:457

The crash is in the code which converts between RGB and BGR color order 
when updating a rectangle. So either the buffer pointers are busted or 
it incorrectly copies more data than it should.

> libvncserver0 = 0.9.8.2

Hope that you either built the package on the same system, or on a 
system which has exactly the same version. The data structures for the 
VNC library change occasionally, and they're not particularly good with 
symbol versioning which means that the application will link fine 
against the .so file, but later crash because the calling code assumes a 
different data layout than the library.

> Dump is over 200MB. Needless to say this is going to take sometime for
> me to figure out. I can ftp it to appsdev if you like.

The problem with such cores for custom built packages is that we'd need 
the matching package, debug info and information about the OS version, 
bitness, special packages and so on. Otherwise gdb would most likely say 
that it can't make sense out of the core.

Klaus

>
>
>
>
> On 07/25/2012 07:00 AM, Klaus Espenlaub wrote:
>> On 24.07.2012 03:12, Perry Halbert wrote:
>>> Update. I turned off the remote display and this solved the issue.
>>> I use VNC and build the extension pack which has been working really well.
>>> So is there a way to get this back to operational status or do I need to
>>> switch back to VRDP?
>> My suspicion is that the VM process crashes... can you enable core dumps
>> to see if this hypothesis is right, or run the VM process directly in
>> gdb? You build everything yourself, so it shouldn't be a problem to have
>> the debug info available. Could be that your libvncserver is newer and
>> has a slightly incompatible API somewhere.
>>
>> I've tried a similar setup (VBox trunk, matching GA, VNC enabled, VT-x
>> disabled), but I couldn't get any crashes. Used a different guest OS and
>> a different linux distro on the host, but this shouldn't really have a
>> significant influence. Tried with and without a VNC viewer connected.
>>
>> I guess that you run the VM in the GUI, because I wasn't able to find a
>> VNC viewer which would trigger resizes when its window is resized.
>>
>> Any obvious patterns for the resize resolution (we removed a "multiple
>> of 4" restriction for the horizontal resolution recently) which causes
>> crashes? Does it ever survive a resize?
>>
>> Klaus
>>
>>>
>>> On 07/23/2012 06:27 PM, Perry Halbert wrote:
>>>> Strange issue this time.
>>>>
>>>> Latest batch of builds, not sure exactly when, but within the last
>>>> week. When resizing the guest (mouse click drag) it aborts. Nothing in
>>>> the guests log file (no shut down, resize information, or reason it
>>>> just stops recording) or syslog of the guest. Just like you pulled the
>>>> plug. Only indication is the main manager shows aborted.
>>>>
>>>> Host Ubuntu 12.04 x86_64 (no VT-x)
>>>> Guest Ubuntu 12.04 X86_32 with PAE
>>>>
>>>> Everything else works good, including all new features, well except it
>>>> is really slow and has been for some time, but usable to test with.
>>>>
>>>> I increased the vRAM to 64MB from 12MB but that does not seem to be
>>>> the issue.
>>>> I removed Version matching GAs and installed 4.1.18, but no change so
>>>> I don't think it is the GAs.
>>>> Turned off 3D so that is not the issue.
>>>>
>>>> Note: I do not see this on a matching Ubuntu host with VT-x.
>>>>
>>>> Didn't I say strange?