| 1 | From: Red Hat Support [support@redhat.com]
|
|---|
| 2 | Sent: Thursday, February 06, 2014 6:30 PM
|
|---|
| 3 | To: Damir Cizmic
|
|---|
| 4 | Subject: (Waiting on Customer) Case #01035356 (RHEL 6.4 hosts crash when
|
|---|
| 5 | VirtualBox 3.1.8 is used) [
|
|---|
| 6 | ref:00DA0000000HxWHMA0.500A000000HnJbyIAF:ref ]
|
|---|
| 7 |
|
|---|
| 8 | ---------------------------------------
|
|---|
| 9 | | Case Information |
|
|---|
| 10 | ---------------------------------------
|
|---|
| 11 | https://access.redhat.com/support/cases/01035356
|
|---|
| 12 | Case Title : RHEL 6.4 hosts crash when VirtualBox 3.1.8 is used
|
|---|
| 13 | Case Number : 01035356
|
|---|
| 14 | Case Open Date : 2014-02-05 13:29:21
|
|---|
| 15 |
|
|---|
| 16 | Most recent comment: On 2014-02-06 13:59:37, Singh, Buland commented:
|
|---|
| 17 | "Hi Damir,
|
|---|
| 18 |
|
|---|
| 19 | The crash analysis of vmcore indicates that kernel panicked in g_abExecMemory() function of VirtualBox Linux kernel driver (vboxdrv).
|
|---|
| 20 |
|
|---|
| 21 | Here's an excerpt from the crash analysis.
|
|---|
| 22 |
|
|---|
| 23 | System Information:
|
|---|
| 24 |
|
|---|
| 25 | crash> sys
|
|---|
| 26 | KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-358.18.1.el6.x86_64/vmlinux
|
|---|
| 27 | DUMPFILE: vmcore [PARTIAL DUMP]
|
|---|
| 28 | CPUS: 24
|
|---|
| 29 | DATE: Fri Jan 31 05:01:13 2014
|
|---|
| 30 | UPTIME: 6 days, 04:55:22
|
|---|
| 31 | LOAD AVERAGE: 2.63, 2.94, 2.90
|
|---|
| 32 | TASKS: 1579
|
|---|
| 33 | NODENAME: epgsim166.seln.ete.ericsson.se
|
|---|
| 34 | RELEASE: 2.6.32-358.18.1.el6.x86_64
|
|---|
| 35 | VERSION: #1 SMP Fri Aug 2 17:04:38 EDT 2013
|
|---|
| 36 | MACHINE: x86_64 (3324 Mhz)
|
|---|
| 37 | MEMORY: 96 GB
|
|---|
| 38 | PANIC: "Oops: 0002 [#1] SMP " (check log for details)
|
|---|
| 39 |
|
|---|
| 40 | Kernel Ring Buffer:
|
|---|
| 41 |
|
|---|
| 42 | crash> log
|
|---|
| 43 | [..]
|
|---|
| 44 | BUG: unable to handle kernel paging request at 00007ff90cde5880
|
|---|
| 45 | IP: [<ffffffffa0466db1>] g_abExecMemory+0x24791/0x180000 [vboxdrv] PGD 70d928067 PUD 182bbfb067 PMD 14a17e5067 PTE 8000000cb9d96007
|
|---|
| 46 | Oops: 0002 [#1] SMP
|
|---|
| 47 | last sysfs file: /sys/module/ipv6/initstate CPU 13 Modules linked in: bridge tun vboxnetadp(U) vboxnetflt(U) vboxdrv(U) nfs fscache mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf dell_rbu nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc target_core_iblock target_core_file target_core_pscsi target_core_mod configfs 8021q garp stp llc ipv6 ext3 jbd uinput power_meter ses enclosure sg bnx2 dcdbas microcode serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
|
|---|
| 48 |
|
|---|
| 49 | Pid: 5157, comm: VBoxHeadless Not tainted 2.6.32-358.18.1.el6.x86_64 #1 Dell Inc. PowerEdge R610/0F0XJ6
|
|---|
| 50 | RIP: 0010:[<ffffffffa0466db1>] [<ffffffffa0466db1>] g_abExecMemory+0x24791/0x180000 [vboxdrv]
|
|---|
| 51 | RSP: 0018:ffff881829d99a28 EFLAGS: 00010206
|
|---|
| 52 | RAX: 0000000000000063 RBX: 00007ff90cde5880 RCX: 00007ff95a383478
|
|---|
| 53 | RDX: 00000000000000e3 RSI: 0000000000091948 RDI: ffffc9002c3880a8
|
|---|
| 54 | RBP: ffff881829d99aa8 R08: 00007ff95abc9008 R09: 0000000000000000
|
|---|
| 55 | R10: 0000000000005752 R11: 0000000000000001 R12: 0000000000000003
|
|---|
| 56 | R13: ffffc9002c3a2000 R14: ffff880051ed5000 R15: 0000000000000000
|
|---|
| 57 | FS: 00007ff98ba0e700(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
|
|---|
| 58 | CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
|
|---|
| 59 | CR2: 00007ff90cde5880 CR3: 000000010225d000 CR4: 00000000000027e0
|
|---|
| 60 | DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
|
|---|
| 61 | DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process VBoxHeadless (pid: 5157, threadinfo ffff881829d98000, task ffff88182a6fc080)
|
|---|
| 62 | Stack:
|
|---|
| 63 | ffff881829d99aa8 ffffffffa046ba63 ffffc9002c387000 00007ff98b28a000 <d> ffffc9002c387000 010101002c387000 00007ff95a383478 00007ff98b26a478 <d> ffff881829d99b20 00007ff90cde5880 00007ff95abc9000 00007ff90cde5880 Call Trace:
|
|---|
| 64 | [<ffffffffa046ba63>] ? g_abExecMemory+0x29443/0x180000 [vboxdrv] [<ffffffffa046de04>] g_abExecMemory+0x2b7e4/0x180000 [vboxdrv] [<ffffffffa046c0d6>] g_abExecMemory+0x29ab6/0x180000 [vboxdrv] [<ffffffffa0450d05>] g_abExecMemory+0xe6e5/0x180000 [vboxdrv] [<ffffffff8100bbee>] ? invalidate_interrupt1+0xe/0x20 [<ffffffffa04804a2>] ? g_abExecMemory+0x3de82/0x180000 [vboxdrv] [<ffffffffa047fd8d>] ? g_abExecMemory+0x3d76d/0x180000 [vboxdrv] [<ffffffffa044ef3b>] ? g_abExecMemory+0xc91b/0x180000 [vboxdrv] [<ffffffffa0449ff2>] g_abExecMemory+0x79d2/0x180000 [vboxdrv] [<ffffffffa04811e7>] g_abExecMemory+0x3ebc7/0x180000 [vboxdrv] [<ffffffffa045371e>] g_abExecMemory+0x110fe/0x180000 [vboxdrv] [<ffffffffa042c7ea>] supdrvIOCtlFast+0x6a/0x70 [vboxdrv] [<ffffffffa042c1bc>] VBoxDrvLinuxIOCtl+0x4c/0x1c0 [vboxdrv] [<ffffffffa045371e>] ? g_abExecMemory+0x110fe/0x180000 [vboxdrv] [<ffffffff8100bbee>] ? invalidate_interrupt1+0xe/0x20 [<ffffffff811952f2>] vfs_ioctl+0x22/0xa0 [<ffffffff8100bbee>] ? invalidate_interrupt1+0xe/0x20 [<ffffffff811c1480>] ? bio_integrity_tag+0x10/0xb0 [<ffffffff81195494>] do_vfs_ioctl+0x84/0x580 [<ffffffff810d94ed>] ? unroll_tree_refs+0xed/0x120 [<ffffffff81182c3e>] ? fput+0xe/0x30 [<ffffffff81195a11>] sys_ioctl+0x81/0xa0 [<ffffffff810dc685>] ? __audit_syscall_exit+0x265/0x290 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
|
|---|
| 65 | Code: f6 c2 02 0f 85 9c 01 00 00 45 84 ff 0f 85 a7 fe ff ff 80 7d af 00 66 66 66 90 0f 85 99 fe ff ff 84 d2 0f 89 a8 01 00 00 83 c8 20 <88> 03 41 80 08 20 48 8b 55 b0 0f b6 02 89 c2 83 ca 20 80 7d ae RIP [<ffffffffa0466db1>] g_abExecMemory+0x24791/0x180000 [vboxdrv] RSP <ffff881829d99a28>
|
|---|
| 66 | CR2: 00007ff90cde5880
|
|---|
| 67 | [..]
|
|---|
| 68 |
|
|---|
| 69 | Backtraces:
|
|---|
| 70 |
|
|---|
| 71 | crash> bt
|
|---|
| 72 | PID: 5157 TASK: ffff88182a6fc080 CPU: 13 COMMAND: "VBoxHeadless"
|
|---|
| 73 | #0 [ffff881829d99610] machine_kexec at ffffffff81035d6b
|
|---|
| 74 | #1 [ffff881829d99670] crash_kexec at ffffffff810c0e22
|
|---|
| 75 | #2 [ffff881829d99740] oops_end at ffffffff81511c20
|
|---|
| 76 | #3 [ffff881829d99770] no_context at ffffffff81046c1b
|
|---|
| 77 | #4 [ffff881829d997c0] __bad_area_nosemaphore at ffffffff81046ea5
|
|---|
| 78 | #5 [ffff881829d99810] bad_area_nosemaphore at ffffffff81046f73
|
|---|
| 79 | #6 [ffff881829d99820] __do_page_fault at ffffffff810476d1
|
|---|
| 80 | #7 [ffff881829d99940] do_page_fault at ffffffff81513b6e
|
|---|
| 81 | #8 [ffff881829d99970] page_fault at ffffffff81510f25
|
|---|
| 82 | [exception RIP: g_abExecMemory+149393] <<<-----[ Exception occurred at g_abExecMemory+149393 ]
|
|---|
| 83 | RIP: ffffffffa0466db1 RSP: ffff881829d99a28 RFLAGS: 00010206
|
|---|
| 84 | RAX: 0000000000000063 RBX: 00007ff90cde5880 RCX: 00007ff95a383478
|
|---|
| 85 | RDX: 00000000000000e3 RSI: 0000000000091948 RDI: ffffc9002c3880a8
|
|---|
| 86 | RBP: ffff881829d99aa8 R8: 00007ff95abc9008 R9: 0000000000000000
|
|---|
| 87 | R10: 0000000000005752 R11: 0000000000000001 R12: 0000000000000003
|
|---|
| 88 | R13: ffffc9002c3a2000 R14: ffff880051ed5000 R15: 0000000000000000
|
|---|
| 89 | ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
|
|---|
| 90 | RIP: 0000003ba94e0a47 RSP: 00007ff98ba0dca0 RFLAGS: 00000213
|
|---|
| 91 | RAX: 0000000000000010 RBX: ffffffff8100b072 RCX: 0000003ba94e0a47
|
|---|
| 92 | RDX: 0000000000000000 RSI: 00000000000056c1 RDI: 000000000000000f
|
|---|
| 93 | RBP: 00007ff98ba0dca0 R8: 000655cbf5d35e73 R9: 0000000001ff3739
|
|---|
| 94 | R10: 0000000000989680 R11: 0000000000000246 R12: 00007ff98b86e780
|
|---|
| 95 | R13: 00007ff98b86e000 R14: 00007ff98b853000 R15: 00007ff98b86e000
|
|---|
| 96 | ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
|
|---|
| 97 |
|
|---|
| 98 | The BUG() function was trigger while copying the value of %rbx register into lower byte (%al) of the %ax register.
|
|---|
| 99 |
|
|---|
| 100 | crash> dis -l g_abExecMemory+149393
|
|---|
| 101 | 0xffffffffa0466db1 <g_abExecMemory+149393>: mov %al,(%rbx) <<<
|
|---|
| 102 |
|
|---|
| 103 | %rbx: 00007ff90cde5880
|
|---|
| 104 | %rax: 0000|0000|0000|0063
|
|---|
| 105 |
|
|---|
| 106 | The following is list of all unsigned(U) modules linked into the kernel.
|
|---|
| 107 |
|
|---|
| 108 | crash> mod -t
|
|---|
| 109 | NAME TAINTS
|
|---|
| 110 | vboxnetadp (U)
|
|---|
| 111 | vboxnetflt (U)
|
|---|
| 112 | vboxdrv (U)
|
|---|
| 113 |
|
|---|
| 114 | Details of VirtualBox unsigned(U) kernel modules (vboxdrv).
|
|---|
| 115 |
|
|---|
| 116 | crash> mod |grep -i vboxdrv
|
|---|
| 117 | ffffffffa05cc9e0 vboxdrv 1783385 (not loaded) [CONFIG_KALLSYMS]
|
|---|
| 118 |
|
|---|
| 119 | crash> module ffffffffa05cc9e0|grep -e version
|
|---|
| 120 | version = 0xffff881824de85a0 "3.1.8 (0x00100001)",
|
|---|
| 121 | srcversion = 0xffff88182a623ba0 "61ABD52580FB8A46A7F5C97",
|
|---|
| 122 |
|
|---|
| 123 | Please note that vboxdrv module is not shipped by Red Hat, hence I don't have a debug-info or source package of this module for further analysis.
|
|---|
| 124 |
|
|---|
| 125 | I would suggest you to contact vendor of vboxdrv module and provide this vmcore to them for analysis. If they found any issues from Operating System side then please share their finding with us for further analysis.
|
|---|
| 126 |
|
|---|
| 127 | Let me know if you have any questions.
|
|---|
| 128 |
|
|---|
| 129 | Regards,
|
|---|
| 130 | Buland Kumar Singh
|
|---|
| 131 | Red Hat, Inc."
|
|---|
| 132 |
|
|---|
| 133 | ---------------------------------------
|
|---|
| 134 |
|
|---|
| 135 | Thank you for your latest interaction with Red Hat Global Support Services. We are currently working to resolve your case.
|
|---|
| 136 |
|
|---|
| 137 | Your case has transitioned to "Waiting On Customer" status. This means that the Red Hat associate working on your case needs information or action from you to proceed. To help us resolve your case as quickly as possible, you will need to update your case online at https://access.redhat.com
|
|---|
| 138 |
|
|---|
| 139 | Once you update the case, we can continue working to resolve your issue.
|
|---|
| 140 |
|
|---|
| 141 | If you wish to contact Red Hat, visit https://access.redhat.com to find phone and web contact information relevant to your region and support contract.
|
|---|
| 142 |
|
|---|
| 143 |
|
|---|
| 144 | Thank you,
|
|---|
| 145 |
|
|---|
| 146 | Red Hat Global Support Services
|
|---|
| 147 | ---------------------------------------
|
|---|
| 148 |
|
|---|
| 149 | Have you joined the discussion in Groups yet? Visit http://red.ht/RHgroups to participate in our rapidly growing user community.
|
|---|
| 150 |
|
|---|
| 151 | Supporting success. Exceeding expectations.
|
|---|
| 152 |
|
|---|
| 153 | Red Hat Customer Portal http://red.ht/accessRH Red Hat Support Twitter http://bit.ly/rhsupport Red Hat Support Facebook http://bit.ly/RHsupportFB Red Hat Support Google+ http://bit.ly/RHsupportG
|
|---|
| 154 |
|
|---|
| 155 | If you need immediate assistance, please reference https://access.redhat.com/support/contact/technicalSupport.html
|
|---|
| 156 | ---------------------------------------
|
|---|
| 157 | [ ref:00DA0000000HxWHMA0.500A000000HnJbyIAF:ref ]
|
|---|
