Solaris 11 virtual console switch causes Xorg to dump core in Guest Additions

[Jon Aimone]

I'm running a Solaris 11 build 151a as a VirtualBox guest on a Windows Vista64 host - VBox 3.2.10.

I followed the instructions carefully to enable virtual consoles, including logging out and back in.

​http://hub.opensolaris.org/bin/view/Project+vconsole/

When I pressed Alt+Ctl+F2 to switch consoles, the guest window changed size indicating that something about the display was different, but the window did not show a text login prompt as expected. The window showed artifacts of the original X desktop.

I had an ssh session into the guest so I could poke around during this experiment.

I typed my user name and password to the console window, still no change in display. However, from the ssh session I could see that I was now logged in.

Wreckdangle:/var/log
[root] (512)> who
aimone     vt/7         Dec  3 08:27    (:0)
aimone     pts/1        Dec  3 08:29    (:0.0)
aimone     pts/2        Dec  3 08:29    (10.0.2.2)

Wreckdangle:/var/log
[root] (513)> who
aimone     vt/7         Dec  3 08:27    (:0)
aimone     pts/1        Dec  3 08:29    (:0.0)
aimone     vt/2         Dec  3 08:31
aimone     pts/2        Dec  3 08:29    (10.0.2.2)

Apparently the keyboard is working, just the display is hosed.

When I attempted to switch back to the X desktop using the hotkey, Alt+Ctl+F7, Xorg dumped core.

I've attached the Xorg.0.log files from before I pressed the first hotkey and after the coredump, and the core file for the Xorg process.

Wreckdangle:/var/cores
[aimone] (4537)> mdb Xorg.1291394087.core
Loading modules: [ Xorg libc.so.1 libproc.so.1 ld.so.1 ]
> $G
C++ symbol demangling enabled
> ::stack
libint10.so`x_inb+0x90()
libint10.so`x86emuOp_in_byte_AL_DX+0x23()
libint10.so`X86EMU_exec+0x8f()
libint10.so`xf86ExecX86int10+0x26()
libvbe.so`VBESaveRestore+0x15f()
vboxvideo_drv.so`VBOXSaveRestore+0x85()
vboxvideo_drv.so`VBOXLeaveVT+0x47()
AbortDDX+0x95()
AbortServer+0x1d()
FatalError+0xf5()
OsSigHandler+0x92()
libc.so.1`__sighndlr+6()
libc.so.1`call_user_handler+0x2a4()
libc.so.1`sigacthandler+0xdb(b, fffffd7fffdf73c0, fffffd7fffdf7060)
vboxvideo_drv.so`vbox_crtc_mode_set+0xde()
xf86CrtcSetModeTransform+0x5c7()
xf86SetDesiredModes+0x3bb()
vboxvideo_drv.so`VBOXEnterVT+0x3e()
xf86RandR12EnterVT+0x4a()
libshadowfb.so`ShadowEnterVT+0x4b()
xf86VTSwitch+0x507()
WakeupHandler+0xdf()
WaitForSomething+0x521()
Dispatch+0x12d()
main+0x681()
0x46ed1c()
> 

[Jon Aimone]

Xorg log file before pressing hot-key.

[Jon Aimone]

Xorg log file following core dump.

[Jon Aimone]

The core file for Xorg is too large to attach. I'll save it for a while in case anyone wants it later.

[aeichner]

Please reopen if still relevant with a recent VirtualBox release.