Suggestions: Resize/Compact VDI size

Please change type of ticket from "defect" to "enhancement". Thanks and sorry. Maddes

Compacting is available in 1.3.4, but resizing a hard disk is still a feature-request.

How then do you compact?

VBoxManage modifyvdi VDINAME compact

Make sure that all unused sectors are zeroed, otherwise you won't gain much. Note that this currently cannot effectively operate on disks which have snapshots, it can only compact the base image but not the diff image(s).

Add a comment to just show I'm interested by this feature too.

As "VBoxManage modifyvdi VDINAME compact" does only release zeroed sectors, we need a tool to fill unused sectors of a file system with zeros, e.g. for an NTFS partition. Does anyone know such a tool?

Check out the "-z" parameters of SysInternals? SDelete:

http://www.microsoft.com/technet/sysinternals/fileanddiskutilities.mspx

Thanks for the tip about SDelete. That should solve the problem for NTFS partitions. Does anyone know a similar tool for Linux partitions?

There is a simple way to do this on a Linux partition. Simply use the dd tool. i.e.

dd if=/dev/zero of=/fillerup.zero

then remove the /fillerup.zero file and it's done.

I have a vm with the following config

host: windows xp guest: ubuntu 7.04 hdd : expanding on usage

I took 2 snapshots, and the file size is around 4 G. I removed the snapshots and ran dd if=/dev/zero of=/fillerup.zero, until it was about 3 G, and then ran

VBoxManage modifyvdi VDINAME compact

There is no change in the vdi file size.

Can you help me with this?

How do I add myself to the cc list for this bug?

Replying to Laibsch:

How do I add myself to the cc list for this bug?

You have just done it.

Although it is a somewhat roundabout way, it works quite well: Create a new VDI-drive of the desired size and then add it as second disk to the system to be resized. Tie in a CD-imgage like Knoppix or Gparted and launch the system. Then simply use dd or ddrescue to transfer the system to the new drive. Of course you will also need to resize the partitions etc.

For filling the unused blocks of ext2/ext3 filesystems with zeros there is also the tool zerofree. It claims do be significantly faster than dd.

http://intgat.tigress.co.uk/rmy/uml/index.html

I would just like to note that when using sdelete to zero out free space in Windows the correct switch is "-c", not "-z".

Example:

sdelete -c 

I also vote for the VDI size extension feature.
We would need at least a converter or resizer utility.

My VDI became full, and I can't use it until I do not resize the virtual hard disk.

Sae request here. Started out with too small a disk size. Funny, because it seems like a verrrrrrry simple thing to implement given that the disk image file is already a dynamically expanding image?

That implies to me, that all that is required is change the upper-bound for the disk size (only an administrative change) to a value greater than before. Afterwards, one could choose from a wide range of tooling (gparted, qtparted, parted, partition magic, ntfstools, AIX smitty etc) to resize and rearrange partitions and filesystems...

Please, santa? Can we have it?

Please add grahamperrin to the cc list - thanks

Working with VBox 1.6.2:

1. I can confirm that the 'dd if=/dev/zero of=/zerofile; rm /zerofile' does NOT allow to recover (all) unused space

2. Using the 'sfill -z /' utility from the 'secure-delete' (Debian/Ubuntu) package does NOT help either

A WORKING (quick and dirty) solution is:

3.a Boot using a GParted LiveCD

3.b Downsize existing partition(s) to the strict minimum

3.b Create temporary unformatted partition(s) in the unused disk space

3.d Fill the temporary unformatted partition(s) with zeroes, using 'dd if=/dev/zero of=/dev/...'

(and optionally do the same for the swap partition)

3.e Remove the temporary unformatted partition(s)

3.f Grow the partition(s) previously downsized in 3.b back to their original size

3.g Shutdown GParted

3.h Compact VDI image with 'VBoxManage modifyvdi ...'

Hope this helps ;-)

Replying to cdufour: Instead of using "rm /zerofile", one could use "shred --interations=1 --zero /zerofile". I have tried this with different distributions and it is working quite well (although shred is taking it's time).

Replying to hufi: There is a catch in using shred: It does not work effectively with certain filesystems, especially those not writing the data directly "on-site" such as ext3, reiser, xsf, etc. I think that all those systems have an option to switch to the correct behaviour. Using ext3, you can simply mount it with the data=ordered or data=writeback option. I think the former is already the default mode.

Replying to cdufour:

Wow, finally something that worked. You're the man. If I were to guess... the reason this works is because the new partition isn't formatted and perhaps ext3 writes some info to the disk right after being zeroed as part of its inherent nature.

Glad I didn't have to use zerofree, not that it wouldn't have worked, just that I read his comments and that he claimed he hadn't fully tested it with ext3 and that kinda shied me away from it since my data is important.

Thanks

Working with VBox 1.6.2: 1. I can confirm that the 'dd if=/dev/zero of=/zerofile; rm /zerofile' does NOT allow to recover (all) unused space 2. Using the 'sfill -z /' utility from the 'secure-delete' (Debian/Ubuntu) package does NOT help either A WORKING (quick and dirty) solution is: 3.a Boot using a GParted LiveCD 3.b Downsize existing partition(s) to the strict minimum 3.b Create temporary unformatted partition(s) in the unused disk space 3.d Fill the temporary unformatted partition(s) with zeroes, using 'dd if=/dev/zero of=/dev/...' (and optionally do the same for the swap partition) 3.e Remove the temporary unformatted partition(s) 3.f Grow the partition(s) previously downsized in 3.b back to their original size 3.g Shutdown GParted 3.h Compact VDI image with 'VBoxManage modifyvdi ...' Hope this helps ;-)

This is an important issue (compacting). I'm just adding this comment to 'subscribe' to this ticket.

Also subscribing to this ticket. I tried the "dd" trick and it doubled the size of my .vdi instead of shrinking it.

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

Replying to frank:

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

frank, I did exactly that. I'm seeing the same problem reported by others above. I'm trying the GParted trick (above) now.

Replying to franl:

Replying to frank:

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

frank, I did exactly that. I'm seeing the same problem reported by others above. I'm trying the GParted trick (above) now.

I can confirm that using GParted (as described above) works for compacting VDI disk images.

Decreasing the size of the guest partition and filling the unused space with zero before doing VBoxManage modifyvdi compact should help in any case. Doing the dd if=/dev/zero of=/bigfile ; rm /bigfile should help as well (of course do not use dd if=/dev/null of=/bigfile). The gain with dd might be less than filling an unused partition with zeros as not all space of a partition is occupied by raw data (because of the file system maintenance structures).

I'd also like add my vote for greater control over VDI files after they are created. In my case, I overestimated the amount of hard drive space I had to play with on the host system, and having got the guest OS (WindowsXP) just right didn't want to start again from scratch. It's basically along the lines of what other people have mentioned but I thought I would detail the process I went through in case it proved useful to someone else:

- After defragmenting the Windows partition in the guest OS, I downloaded the demo version of BootIt?, ran it as a bootable ISO, and reduced the size of the primary partition to my target size of 4Gb from an initial 10Gb. - In the remaining space I created a secondary partition. - After booting the guest into WIndows XP, I used sdelete -c (as suggested above) to zero the data on the secondary parition; this actually hung, I presume when it reached the physical (as opposed to virtual) limit of the data, but seemed to do the trick anyway. - I then rebooted through the BootIt? ISO and deleted the zeroed secondary partition - Finally, in the host I ran:

VBoxManage modifyvdi my.vdi compact

So while the VDI image still has the same theoretical upper limit, the guest OS won't be able to write beyond the 4Gb partition, and the VDI file won't grow beyond this limit either.

I also vote for official VDI manipulation support. It's a very useful but dangerous for users to do the tricks by themselves.

In the site http://www.techthrob.com/tech/securedelete.php I found the following statement:

"Filesystems Starting with ext3, Linux filesystems overwrite files with zeros when you delete them, rather than just marking the file as "free space." Previous Linux filesystems, as well as the Windows filesystems (NTFS, FAT) use the latter method, which is why it is so easy to recover deleted files with software tools freely downloadable on the Internet. This is something to consider when deciding how much time you should spend wiping your drive."

If this is true, for Linux ext3 partitions zero free-space fill tools are not needed since the OS is writing zeroes after any file is deleted.

The question: How true is this statement?

Thanks

@cmozuelos (concerning ext3 secure-delete feature): absolute crap! I can't believe writers of articles about secure deletion tools can make such statement (that's security for you...)!

EXT3, as currently implemented, does NOT securely delete files. This feature has been provisioned (via the 's' extended attribute)... but has NOT been implemented yet; see the manual page for 'chattr', section 'BUGS AND LIMITATIONS' for further information.

You can convince yourself empirically of that by creating a VERY large file on your disk, and then deleting it. While creating the file may take several minutes, deletion never takes more than a few seconds. That just tells you that NO time is spent going through the physical media, "overwriting" data with zeroes/random/whatever:

time dd if=/dev/zero of=/scratch/big-file bs=1k count=10000000

10000000+0 records in 10000000+0 records out 10240000000 bytes (10 GB) copied, 194.35 s, 52.7 MB/s

real 3m15.530s user 0m1.864s sys 0m42.123s

chattr -V +s /scratch/big-file

chattr 1.40.8 (13-Mar-2008) Flags of /scratch/big-file set as s-----------------

time rm /scratch/big-file

real 0m15.235s user 0m0.000s sys 0m0.804s

;-)

(Sorry for the slightly OT answer...)

Thanks cdufour, you are right I tried:

Nothing: did not worked as you said DD = worked but poorly, time consuming and did not shrink the VDI to its right size. zerofree = quick and effective

I started trying sfill, but I was too tired and happy with the zerofree solution.

Thanks again.

I have a (technical) suggestion to make compating much faster:

introduce a system call with which a guest program can tell the host VM that a particular block of the disk is no longer used. Programs like sdelete, zerofree oder gparted could be modified to simply notify the host instead of writing zeroed blocks. The host would maintain a list of unused blocks and remove them again if something is written to them. "VBoxManage" could then simply use that list instead of looking for zero blocks. I hope someone can implement that soon.

I've found a nice solution to this problem. This method has the advantage that is doesn't require any more disk space than your virtual disk is alreadying using, because the virtual disk is compacted in-place. This method uses the "zerofree" tool, which finds unallocated filesystem blocks and fills them with zero bytes.

0. Backup your virtual machine! You have been warned!

1. Install the libext2fs library if you don't already have it. On Ubuntu/Debian distros, just run "sudo apt-get install ext2fs-dev".

2. Download zerofree from:

http://intgat.tigress.co.uk/rmy/uml/zerofree-1.0.1.tgz

3. Build zerofree by unpacking the above tarball and running "make" in the source directory.

4. Copy the zerofree executable onto the virtual disk to be compacted.

5. While your VM is still running, note the device names of all the partitions (e.g., /dev/sda1, /dev/sda2, etc.), including the swap partition (run "/sbin/swapon -s" to see the device name for the swap partition).

6. Shutdown the virtual machine!

7. Boot off a live CD (e.g., the Ubuntu install CD).

8. For each of the partitions on the VM's disk, do the following:

8-A. Mount the filesystem read-only:

sudo mount -r /dev/XXX /mnt

8-B. Zero the filesystem's free blocks:

sudo /mnt/.../zerofree -v /dev/XXX

8-C. Unmount the filesystem:

sudo umount /mnt

8-D. Check the filesystem:

sudo fsck.ext2 -f /dev/XXX

9. Zero the swap partition:

sudo dd if=/dev/zero of=/dev/XXX bs=512

10. Shutdown the system, and power off the VM.

11. Compact the virtual disk:

VBoxManage modifyhd FILENAME --compact

WARNING: This will take a long time.

add to ticket watch / CC list please

I have version 3.1.2 and it seems to be still not implemented. Voting for it too.