Ticket #9975 (new defect)

Opened 2 years ago

Last modified 9 days ago

Virtual HDD becomes unavailable for guest after a Canceled write is logged

Reported by: karmapolis Owned by:
Priority: blocker Component: virtual disk
Version: VirtualBox 4.1.6 Keywords:
Cc: Guest type: Windows
Host type: Linux

Description (last modified by frank) (diff)

The guest is a Windows SBS Server 2008 with 2 virtual CPUs, 4 GB of virtual RAM and two virtual HDDs (C: with 40 GB and D: with 20 GB, both with lots of free space). This SBS server acts as an SBS primary server (i.e., AD controller); it has had its MS Exchange and MS SharePoint removed and an SQL Server 2008 has been installed (besides the SQL Server 2005 Express that comes with SBS and which cannot be removed). This is a supported configuration for SBS, even if it's a bit unusual. It has not been customized significantly yet because we were just testing feasibility. The guest is running the latest Guest Additions (matching the host's VirtualBox version).

The problem is the guest works fine... until it doesn't. When the problem appears, guest applications start hanging pretty fast. A few tests have shown that the problem is that the (virtual) C: drive has stopped responding, and appications die only when they try to access it. The failing drive can also be the D: drive instead. The guest system cannot be brought down in this situation: the only option is powering it off or waiting for the Windows kernel to bluescreen and reset.

When the guest system is hanged like that, one or several messages like this one can be seen in the Vbox.log file:

53:11:46.112 AHCI#0: Canceled write at offset 29189742592 (4096 bytes left) returned rc=VINF_SUCCESS

The return code, if that's what it is, is always VINF_SUCCESS. The offset and the amount of bytes left may vary.

Thinking it could be a problem with flushing I have tried setting:

VBoxManage setextradata "VM002" "VBoxInternal/Devices/ahci/0/LUN#0/Config/IgnoreFlush" 0

VBoxManage setextradata "VM002" "VBoxInternal/Devices/ahci/0/LUN#1/Config/IgnoreFlush" 0

VBoxManage setextradata "VM002" "VBoxInternal/Devices/ahci/0/LUN#0/Config/FlushInterval" 1000000

VBoxManage setextradata "VM002" "VBoxInternal/Devices/ahci/0/LUN#1/Config/FlushInterval" 1000000


VBoxManage setextradata "VM002" "VBoxInternal/Devices/ahci/0/LUN#0/Config/IgnoreFlush" 0

VBoxManage setextradata "VM002" "VBoxInternal/Devices/ahci/0/LUN#1/Config/IgnoreFlush" 0

VBoxManage setextradata "VM002" "VBoxInternal/Devices/ahci/0/LUN#0/Config/FlushInterval" 1

VBoxManage setextradata "VM002" "VBoxInternal/Devices/ahci/0/LUN#1/Config/FlushInterval" 1

But both ended up failing with the same error.

The problem is not easily reproduceable. It sometimes fails several times in a row, then sometimes it works for a couple of days before failing again. It might be a VirtualBox bug somehow related to host load (but this server is nowhere near full utilization).

This problem has been found on an HP ML370 G6 dual Quad Xeon host with lots of RAM and 4 SAS 15000 RPM HDDs working in a RAID10 array. The host runs CentOS 5.7 and this host is, as you might guess, unusually fast. This host also runs another VirtualBox guest: it's a Windows Server 2008 R2 system that works fine and hasn't suffered from this problem (fingers crossed). This other guest runs under a different host user.


VBox.log Download (127.5 KB) - added by karmapolis 2 years ago.

Change History

Changed 2 years ago by karmapolis

comment:1 Changed 2 years ago by frank

Did you see any messages in the host kernel log (dmesg) when this happens?

comment:2 Changed 2 years ago by karmapolis

No, no messages in dmesg at that time, and nothing suspicious before that.

comment:3 Changed 2 years ago by karmapolis

In case you're interested, I have a core dump of VBoxHeadless while the guest is locked up.

comment:4 Changed 2 years ago by karmapolis

More info obtained through further testing:

The "Canceled write" messages sometimes appear a while before any symptom appears in the guest. Other times it happens simultaneously. On one occasion I left it hanged and the Windows guest showed a blue screen with error 0x1000009f. On a different occasion I had seen a different error code which I don't have at hand now, but I researched it at the time and it was clearly related to a "missing" C: drive.

comment:5 Changed 2 years ago by karmapolis

This problem has now been confirmed with the same virtual machine on a different host (an HP ML110 G7 with SATA disks).

comment:6 Changed 2 years ago by karmapolis

On only one occasion I got the following error message *after* all the "Canceled write" ones:

00:01:40.232 VRDP: shadowBufferOrder: pointer 00002aab91f36981/606183713 is out of range [00002aaab5f3b344; 00002aaab6150aa2) after op 7, 00002aaab5f3b364, 2185022

Also, a couple of times I saw the same guest symptoms while there had been no "Canceled write" messages in the log. I now think the canceled write messages may be just a symptom, but not near to the cause of the problem; they are just correlated to it.

I have a couple of snapshots where the problem arises soon almost every time upon doing some operation. Let me know if any of this is useful to you.

comment:7 Changed 2 years ago by karmapolis

This bug is still present in VirtualBox 4.1.8 . Please let me know how I can help this get resolved. Thanks.

comment:8 Changed 2 years ago by Zanthamos

Try setting your HDD as an IDE drive rather than a SATA, I was having the same issue until I did so.

comment:9 Changed 2 years ago by drando

I'm hit by the same bug. Under Ubuntu 8.04 Linux 64bits, on 4.1.6 and 4.1.8 Virtualbox, suddently, AHCI was causing this error (was running ok for a long time). I had to switch to ide. The guest S.O. is Ubuntu 8.04 server.

On another server, where i have 8 VMs (all Windows XP Guest), i got hit by this bug randomnly on each of those VMs.

comment:10 Changed 21 months ago by danf84

I am having issues similar to this, switching to IDE *seems* to resolve the issue.

comment:11 Changed 21 months ago by frank

  • Description modified (diff)

Then I assume that enabling the host I/O cache for the SATA controller would have the same effect (VM settings).

comment:12 Changed 9 days ago by camojiet

Have same problem with 4.3.8-92456
313:49:08.022482 AHCI#0P2: Cancelled task 5
313:49:08.069531 AHCI#0: Port 2 reset
313:49:08.425542 AHCI#0P2: Canceled write at offset 13813657088 (4096 bytes left) returned rc=VINF_SUCCESS
313:49:09.082520 AHCI#0: Port 2 reset

This is easily reproduceable, you need the low speed disk system (in my case it is RAID6 array) on the host, ACHI on the guest and heavy write data flow to disk(not long).
For me it's a really problem. I was faced with this problem around half a year ago, and long time sought any information, tried a lot of variants of configuration, and have no result.
Now i move critical important vdi files to SSD, but sometimes like today i faced with this problem on not SSD drive and need to reboot guest. It's very very very very very poorly.
What i should to do to resolve or circumvent this problem? iSCSI?

Last edited 9 days ago by camojiet (previous) (diff)
Note: See TracTickets for help on using tickets.
ContactPrivacy policyTerms of Use