Disk errors ("No space left on device" with nearly empty disks)

[deever]

In a FreeBSD guest, doing disk I/O intensive things like copying large/deep directory trees, I'm getting errors after a while:

/: create/symlink failed, no inodes free
./configure: cannot create config.log: No space left on device

although df reports 97% free inodes and 95% free space on disk. These aren't reproducible for exactly the same filenames and somethimes, after restoring the current snapshot and redoing the same operation again, it succeeds.

I had something similar also in the previous x.0 versions (2.0 and 3.0) of VirtualBox, where there were no specific error messages, but general disk failure, so no way to run df anymore.

Maybe it is a matter that I use Linux LVM for storing the VM's data?

[deever]

Definitely not LVM-specific. Had the same issue in a Linux guest not on a LV.

[Frank Mehnert]

You get this error in the guest, right? Please attach the VBox.log file of such a VM session.

[deever]

Back in 3.2.12, I also have a similar (or the same) issue. The logs for it are:

[...]
00:15:04.355 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0xca (10009485 usec ago) CmdIf1=0x00 (-1 usec ago)
00:15:06.368 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:08.372 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:10.376 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:12.379 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:14.380 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:16.382 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:18.385 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:20.389 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:22.395 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:15:23.259 PIIX3 ATA: execution time for ATA command 0xca was 28 seconds
00:15:23.260 PIIX3 ATA: Ctl#0: finished processing RESET
[...]

Changing the IDE type to PIIX3 (from PIIX4), I'm having a different error in the host:

[...]
Device ad0s1a went missing before all of the data could be written to it; expect data loss
[...]

and the following log (although I have changed the IDE type):

[...]
00:30:44.690 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0xca (10006962 usec ago) CmdIf1=0x00 (-1 usec ago)
00:30:46.702 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:30:48.703 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:30:50.705 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:30:52.710 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:30:54.713 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:30:56.720 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:30:58.722 PIIX3 ATA: Async I/O thread probably stuck in operation, interrupting
00:31:00.440 PIIX3 ATA: execution time for ATA command 0xca was 25 seconds
00:31:00.440 PIIX3 ATA: Ctl#0: finished processing RESET
[...]

Changing the IDE type to ICH6, host error:

[...]
Device ad0s1a went missing before all of the data could be written to it; expect data loss
objcopy: unable to rename 'kernel.debug' reason: Device not configured
[...]

and the following log:

[...]
00:15:29.024 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0xca (10019103 usec ago) CmdIf1=0x00 (-1 usec ago)
00:15:30.067 PIIX3 ATA: execution time for ATA command 0xca was 11 seconds
00:15:30.067 PIIX3 ATA: Ctl#0: finished processing RESET
[...]

After removing the VM and creating a new one with 4.0.4 again and with ICH6 from the beginning, the same "Device not configured error" and the log:

[...]
00:39:46.520 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0xca (9999814 usec ago) CmdIf1=0x00 (-1 usec ago)
00:39:46.858 PIIX3 ATA: execution time for ATA command 0xca was 10 seconds
00:39:46.858 PIIX3 ATA: Ctl#0: finished processing RESET
[...]

Why actually always the use of PIIX3 even if I have changed to something other?

I hope you can find out something...I'd be keen on (keep) using VirtualBox (instead of all the other stuff)...! :)

Greetings

[Frank Mehnert]

As long as you don't provide the information I've asked for I can't help you.

[deever]

VBox.log

[deever]

OK, if that helps? ;)

Thank you!

[Frank Mehnert]

So far nothing new from the log file. The log file shows that one ATA operation issued by the guest took very long, most probably because the host was busy during this time. You already posted that.

Don't worry about the PIIX3 string, this is only the internal device emulation which is used (PIIX3 and PIXX4 don't differ that much).

Is this problem easy to reproduce? Do you really have some disk-intensive operations run in parallel to the VM? You could also try to use the SATA emulation. And make sure to disable the host I/O cache (default for SATA) as this should minimize such effects (VBox will use its own disk cache in that case).