Solaris 10u5 guest hangs and consumes 100% CPU

[Shane Hjorth]

Running Virtualbox 2.0.2 on a Sun Ultra 24 running Ubuntu 8.04 (64bit) and I am finding stability to be a real problem when Intel VT-x is enabled which in turn allows me to run Solaris 10u5 x86 in 64-bit mode.

Running with VT-x disabled (same guest no running in 32-bit mode) is however stable.

After a period of time (30minutes+ depends on activity on the guest itself) when VT-x is enabled the Solaris 10u5 (64-bit) VirtualBox process will become non-responsive (hang) and CPU usage will shoot up to 100%.

The spinning thread is looping in the following call:

(gdb) where
#0  0x00007f727cbab3c7 in ioctl () from /lib/libc.so.6
#1  0x00007f727c414adf in ?? () from /usr/lib/virtualbox/VBoxRT.so
#2  0x00007f72744409ee in VMMR3HwAccRunGC ()
   from /usr/lib/virtualbox/VBoxVMM.so
#3  0x00007f727446ee6e in EMR3ExecuteVM () from /usr/lib/virtualbox/VBoxVMM.so
#4  0x00007f72744445d8 in ?? () from /usr/lib/virtualbox/VBoxVMM.so
#5  0x00007f727c3eee2c in ?? () from /usr/lib/virtualbox/VBoxRT.so
#6  0x00007f727c410c62 in ?? () from /usr/lib/virtualbox/VBoxRT.so
#7  0x00007f727d3523f7 in start_thread () from /lib/libpthread.so.0
#8  0x00007f727cbb2b2d in clone () from /lib/libc.so.6
#9  0x0000000000000000 in ?? ()

strace output

ioctl(33, 0x56c1, 0)                    = 0
ioctl(33, 0x56c1, 0)                    = 0
ioctl(33, 0x56c1, 0)                    = 0
ioctl(33, 0x56c1, 0)                    = 0
ioctl(33, 0x56c1, 0)                    = 0
ioctl(33, 0x56c1, 0)                    = 0

lsof output

COMMAND     PID  USER   FD   TYPE             DEVICE       SIZE     NODE NAME
VirtualBo 13331 shane   33u   CHR              10,63               24235 /dev/vboxdrv

I have tried a number of troubleshooting steps to solve this issue:

• patched the Ultra 24 bios to the latest available.
• Upgraded virtualbox from 2.0.0 -> 2.0.2
• Ran the Solaris 10u5 system with and without VBox tools installed on the guest.
• Installed a fresh copy of Solaris 10u5 (64bit).

Nothing has helped so far.

This issue appears to be triggered by load within the guest itself. For example I had to re-attempt the installation of Sun Communication Suite 5 three times before it would complete without the VM hanging half-way through the process. Installing the same product when the same OS is running in 32bit mode completes without a problem.

I currently have 3 Solaris 10u5 guests running.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13413 shane     20   0 1329m 1.1g  16m S    5 18.6  10:07.66 VirtualBox
13331 shane     20   0 1313m 1.1g  15m S   94 18.3  44:35.35 VirtualBox
13362 shane     20   0  775m 540m  15m S    4  9.1   8:24.34 VirtualBox

13331 => problem Solaris 10u5 64-bit instance with a small amount of application load 13413 => no-problem running Solaris 10u5 64-bit with no application load 13362 => no-problem running Solaris 10u5 32-bit with same level of application load as 13331

13331 Session Information (Runtime Attributes):

Screen Resolution 720x400
VT-x/AMD-V        Enabled
Nested Paging     Disabled
Guest Additions   Version 1.4

Guest OS Type     Solaris

Hard Disk Statistics
IDE Primary Master
DMA Transfers      79,145

PIO Transfers      2,873
Data Read          576,124,416 B
Data Written       1,436,581,888 B

CD/DVD-ROM Statistics
DMA Transfers
PIO Transfers      4,094
Data Read          7,942,144 B
Data Written       0 B

Network Adapter Statistics
Adapter 0
Data Transmitted  476,950 B
Data Received     66,507,622 B

[Shane Hjorth]

VBox Log file after hung process forcefully stopped.