Solaris 11 virtual console switch causes Xorg to dump core in Guest Additions

[Jon Aimone]

I'm running a Solaris 11 build 151a as a VirtualBox guest on a Windows Vista64 host - VBox 3.2.10.

I followed the instructions carefully to enable virtual consoles, including logging out and back in.

​http://hub.opensolaris.org/bin/view/Project+vconsole/

When I pressed Alt+Ctl+F2 to switch consoles, the guest window changed size indicating that something about the display was different, but the window did not show a text login prompt as expected. The window showed artifacts of the original X desktop.

I had an ssh session into the guest so I could poke around during this experiment.

I typed my user name and password to the console window, still no change in display. However, from the ssh session I could see that I was now logged in.

Wreckdangle:/var/log [root] (512)> who aimone vt/7 Dec 3 08:27 (:0) aimone pts/1 Dec 3 08:29 (:0.0) aimone pts/2 Dec 3 08:29 (10.0.2.2)

Wreckdangle:/var/log [root] (513)> who aimone vt/7 Dec 3 08:27 (:0) aimone pts/1 Dec 3 08:29 (:0.0) aimone vt/2 Dec 3 08:31 aimone pts/2 Dec 3 08:29 (10.0.2.2)

Apparently the keyboard is working, just the display is hosed.

When I attempted to switch back to the X desktop using the hotkey, Alt+Ctl+F7, Xorg dumped core.

I've attached the Xorg.0.log files from before I pressed the first hotkey and after the coredump, and the core file for the Xorg process.

Wreckdangle:/var/cores [aimone] (4537)> mdb Xorg.1291394087.core Loading modules: [ Xorg libc.so.1 libproc.so.1 ld.so.1 ]

$G

C++ symbol demangling enabled

::stack

libint10.so`x_inb+0x90() libint10.so`x86emuOp_in_byte_AL_DX+0x23() libint10.so`X86EMU_exec+0x8f() libint10.so`xf86ExecX86int10+0x26() libvbe.so`VBESaveRestore+0x15f() vboxvideo_drv.so`VBOXSaveRestore+0x85() vboxvideo_drv.so`VBOXLeaveVT+0x47() AbortDDX+0x95() AbortServer+0x1d() FatalError+0xf5() OsSigHandler+0x92() libc.so.1`sighndlr+6() libc.so.1`call_user_handler+0x2a4() libc.so.1`sigacthandler+0xdb(b, fffffd7fffdf73c0, fffffd7fffdf7060) vboxvideo_drv.so`vbox_crtc_mode_set+0xde() xf86CrtcSetModeTransform+0x5c7() xf86SetDesiredModes+0x3bb() vboxvideo_drv.so`VBOXEnterVT+0x3e() xf86RandR12EnterVT+0x4a() libshadowfb.so`ShadowEnterVT+0x4b() xf86VTSwitch+0x507() WakeupHandler+0xdf() WaitForSomething+0x521() Dispatch+0x12d() main+0x681() 0x46ed1c()

[Jon Aimone]

Xorg log file before pressing hot-key.

[Jon Aimone]

Xorg log file following core dump.