ZFS replication level decrease possibly resulted in deferred kernel panic

[baitisj]

Kernel crash occurred after host storage ZFS volume going offline. System hang occurred while SATA device was retried. Host system continued to function in degraded mode, but virtual machine crashed.

Host is FreeBSD 10.0-RELEASE-p4 running VB 4.3.8-OSE.

dmesg from guest:

Jun 17 18:10:20 manbox kernel: [1453901.719191] PGD 5a33067 PUD 5a32067 PMD 0 
Jun 17 18:10:20 manbox kernel: [1453901.719200] Oops: 0002 [#1] SMP 
Jun 17 18:10:20 manbox kernel: [1453901.719208] Modules linked in: sg st fuse bnep bluetooth rfkill snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_page_alloc snd_timer snd soundcore intel_agp intel_gtt ac ppdev processor joydev parport_pc parport i2c_piix4 button microcode psmouse serio_raw evdev nfs lockd sunrpc fscache vboxvideo(O) drm i2c_core vboxsf(O) vboxguest(O) ext4 crc16 mbcache jbd2 sr_mod cdrom ata_generic sd_mod pata_acpi virtio_net ohci_hcd ahci libahci usbcore usb_common virtio_pci virtio_ring virtio ata_piix libata scsi_mod
Jun 17 18:10:20 manbox kernel: [1453901.719283] CPU: 1 PID: 16815 Comm: mozStorage #6 Tainted: G           O 3.10.40-1-MANJARO #1
Jun 17 18:10:20 manbox kernel: [1453901.719290] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jun 17 18:10:20 manbox kernel: [1453901.719296] task: ffff88001cf02140 ti: ffff88000eac8000 task.ti: ffff88000eac8000
Jun 17 18:10:20 manbox kernel: [1453901.719301] RIP: 0010:[<ffffffffa01f31b6>]  [<ffffffffa01f31b6>] vbglPhysHeapExcludeBlock+0x16/0x60 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.719317] RSP: 0018:ffff88000eac9890  EFLAGS: 00010202
Jun 17 18:10:20 manbox kernel: [1453901.719323] RAX: 0000000000000001 RBX: ffff88001de3014c RCX: 0000000000000074
Jun 17 18:10:20 manbox kernel: [1453901.719328] RDX: ffff88001de30124 RSI: ffff88001de30124 RDI: ffff88001de30198
Jun 17 18:10:20 manbox kernel: [1453901.719333] RBP: ffff88000eac9890 R08: 0000000000000000 R09: 0000000000000002
Jun 17 18:10:20 manbox kernel: [1453901.719338] R10: fff0000000000fff R11: ffff800000000000 R12: ffff88001de30124
Jun 17 18:10:20 manbox kernel: [1453901.719343] R13: 0000000000000000 R14: 0000000000000002 R15: ffff88001de30198
Jun 17 18:10:20 manbox kernel: [1453901.720339] FS:  00007f8bbb0ff700(0000) GS:ffff88001fd00000(0000) knlGS:00000000f6400840
Jun 17 18:10:20 manbox kernel: [1453901.720350] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 17 18:10:20 manbox kernel: [1453901.720355] CR2: 0000000000000019 CR3: 000000001c88d000 CR4: 00000000000006e0
Jun 17 18:10:20 manbox kernel: [1453901.720364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 17 18:10:20 manbox kernel: [1453901.720370] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 17 18:10:20 manbox kernel: [1453901.720375] Stack:
Jun 17 18:10:20 manbox kernel: [1453901.720380]  ffff88000eac98b0 ffffffffa01f36d9 ffff88001de3014c 0000000000000002
Jun 17 18:10:20 manbox kernel: [1453901.720390]  ffff88000eac98c8 ffffffffa01f1aee ffff88000eac9be0 ffff88000eac9a78
Jun 17 18:10:20 manbox kernel: [1453901.720396]  ffffffffa01f23a4 ffff88000eac99b0 0000000000000246 00000001a0215280
Jun 17 18:10:20 manbox kernel: [1453901.720404] Call Trace:
Jun 17 18:10:20 manbox kernel: [1453901.720438]  [<ffffffffa01f36d9>] VbglPhysHeapFree+0xc9/0xe0 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720452]  [<ffffffffa01f1aee>] VbglGRFree+0x1e/0x30 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720466]  [<ffffffffa01f23a4>] VbglR0HGCMInternalCall+0x6c4/0xf70 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720479]  [<ffffffffa01ee530>] ? VBoxGuestHGCMAsyncWaitCallbackWorker+0x240/0x240 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720514]  [<ffffffffa01ee887>] ? VBoxGuestCommonIOCtl_HGCMCall+0x257/0x320 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720564]  [<ffffffff81129c2e>] ? __alloc_pages_nodemask+0x15e/0xb30
Jun 17 18:10:20 manbox kernel: [1453901.720584]  [<ffffffffa01efc19>] ? VBoxGuestCommonIOCtl+0x239/0x1aa0 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720597]  [<ffffffffa01ee887>] VBoxGuestCommonIOCtl_HGCMCall+0x257/0x320 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720610]  [<ffffffffa01ed251>] ? VBoxGuestIDCCall+0x61/0x70 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720623]  [<ffffffffa01efc19>] VBoxGuestCommonIOCtl+0x239/0x1aa0 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720636]  [<ffffffffa01ed251>] VBoxGuestIDCCall+0x61/0x70 [vboxguest]
Jun 17 18:10:20 manbox kernel: [1453901.720649]  [<ffffffffa0024756>] vbglDriverIOCtl+0x16/0x20 [vboxsf]
Jun 17 18:10:20 manbox kernel: [1453901.720774]  [<ffffffffa0025423>] VbglHGCMCall+0x43/0xa0 [vboxsf]
Jun 17 18:10:20 manbox kernel: [1453901.720791]  [<ffffffffa0025a9a>] vboxCallClose+0x5a/0x70 [vboxsf]
Jun 17 18:10:20 manbox kernel: [1453901.720969]  [<ffffffffa0021a37>] sf_dir_open+0x1b7/0x1d0 [vboxsf]
Jun 17 18:10:20 manbox kernel: [1453901.720982]  [<ffffffff81188f68>] do_dentry_open+0x198/0x2b0
Jun 17 18:10:20 manbox kernel: [1453901.720994]  [<ffffffffa0021880>] ? sf_dir_release+0x30/0x30 [vboxsf]
Jun 17 18:10:20 manbox kernel: [1453901.721005]  [<ffffffff81189351>] finish_open+0x31/0x50
Jun 17 18:10:20 manbox kernel: [1453901.721016]  [<ffffffff81198eca>] do_last+0x4da/0xf80
Jun 17 18:10:20 manbox kernel: [1453901.721151]  [<ffffffff81199a20>] path_openat+0xb0/0x540
Jun 17 18:10:20 manbox kernel: [1453901.721161]  [<ffffffff8117240c>] ? kfree+0xac/0x150
Jun 17 18:10:20 manbox kernel: [1453901.721171]  [<ffffffff8119ae98>] do_filp_open+0x38/0xa0
Jun 17 18:10:20 manbox kernel: [1453901.721183]  [<ffffffff811a6d38>] ? __alloc_fd+0x78/0x110
Jun 17 18:10:20 manbox kernel: [1453901.721190]  [<ffffffff8118a198>] do_sys_open+0xe8/0x1d0
Jun 17 18:10:20 manbox kernel: [1453901.721248]  [<ffffffff8118a29e>] SyS_open+0x1e/0x20
Jun 17 18:10:20 manbox kernel: [1453901.721262]  [<ffffffff814d79dd>] system_call_fastpath+0x1a/0x1f
Jun 17 18:10:20 manbox kernel: [1453901.721443] Code: 05 00 00 c7 05 a8 22 02 00 00 00 00 00 5d c3 66 0f 1f 44 00 00 66 66 66 66 90 48 8b 47 10 55 48 89 e5 48 85 c0 74 0c 48 8b 57 18 <48> 89 50 18 48 8b 47 10 48 8b 57 18 48 85 d2 74 19 48 89 42 10 
Jun 17 18:10:20 manbox kernel: [1453901.721443]  RSP <ffff88000eac9890>
Jun 17 18:10:20 manbox kernel: [1453901.721443] CR2: 0000000000000019
Jun 17 18:10:20 manbox kernel: [1453901.722753] ---[ end trace 0436ca8276f4d407 ]---

Corresponding dmesg of host:

Jun 17 16:16:58 bsdraid kernel: ahcich3: is 00000000 cs f0000000 ss f0000000 rs f0000000 tfd 00 serr 00000000 cmd 0020fc17
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 0b d4 e5 40 06 00 00 01 00 00
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): CAM status: Command timeout
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): Error 5, Periph was invalidated
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 63 de e5 40 06 00 00 01 00 00
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): CAM status: Unconditionally Re-queue Request
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): Error 5, Periph was invalidated
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 63 df e5 40 06 00 00 01 00 00
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): CAM status: Unconditionally Re-queue Request
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): Error 5, Periph was invalidated
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 63 e0 e5 40 06 00 00 01 00 00
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): CAM status: Unconditionally Re-queue Request
Jun 17 16:16:58 bsdraid kernel: (ada2:ahcich3:0:0:0): Error 5, Periph was invalidated
Jun 17 16:16:59 bsdraid kernel: (ada2:ahcich3:0:0:0): Periph destroyed
Jun 17 16:17:47 bsdraid kernel: ahcich3: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jun 17 16:17:47 bsdraid kernel: ahcich3: Poll timeout on slot 31 port 0
Jun 17 16:17:47 bsdraid kernel: ahcich3: is 00000000 cs 80000000 ss 00000000 rs 80000000 tfd 00 serr 00000000 cmd 0020ff17
Jun 17 16:17:47 bsdraid kernel: (aprobe0:ahcich3:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Jun 17 16:17:47 bsdraid kernel: (aprobe0:ahcich3:0:0:0): CAM status: Command timeout
Jun 17 16:17:47 bsdraid kernel: (aprobe0:ahcich3:0:0:0): Error 5, Retries exhausted
Jun 17 16:31:31 bsdraid kernel: pid 42632 (VirtualBox), uid 1001: exited on signal 11

[aeichner]

Please reopen if still relevant with a recent VirtualBox release.