VirtualBox

Ticket #28 (closed enhancement: fixed)

Opened 7 years ago

Last modified 2 years ago

Suggestions: Resize/Compact VDI size => fixed in SVN

Reported by: maddes.b Owned by:
Priority: minor Component: virtual disk
Version: VirtualBox 1.3.2 Keywords: resize vdi
Cc: Guest type: other
Host type: other

Description

It would be a great advantage if you could change or at least extend the size of a VDI, e.g. you start with 4GB of virtual harddisk space, and when necessary resize it to 8GB to install more programs to it.

Another suggestion related to VDI is an option to compact an existing flexible VDI by removing all unused/free sectors from the VDI file. This way you can reduce disk usage on the host system.

Regards Maddes

Change History

comment:1 Changed 7 years ago by maddes.b

Please change type of ticket from "defect" to "enhancement". Thanks and sorry. Maddes

comment:2 Changed 7 years ago by umoeller

  • Type changed from defect to enhancement

comment:3 Changed 7 years ago by maddes.b

Compacting is available in 1.3.4, but resizing a hard disk is still a feature-request.

comment:4 Changed 7 years ago by ja_guy

How then do you compact?

comment:5 Changed 7 years ago by klaus

VBoxManage modifyvdi VDINAME compact

Make sure that all unused sectors are zeroed, otherwise you won't gain much. Note that this currently cannot effectively operate on disks which have snapshots, it can only compact the base image but not the diff image(s).

comment:6 Changed 7 years ago by JusTiCe8

Add a comment to just show I'm interested by this feature too.

comment:7 Changed 7 years ago by chdh

As "VBoxManage modifyvdi VDINAME compact" does only release zeroed sectors, we need a tool to fill unused sectors of a file system with zeros, e.g. for an NTFS partition. Does anyone know such a tool?

comment:8 Changed 7 years ago by maddes.b

Check out the "-z" parameters of SysInternals SDelete:

 http://www.microsoft.com/technet/sysinternals/fileanddiskutilities.mspx

comment:9 Changed 7 years ago by chdh

Thanks for the tip about SDelete. That should solve the problem for NTFS partitions. Does anyone know a similar tool for Linux partitions?

comment:10 Changed 7 years ago by dschulz

There is a simple way to do this on a Linux partition. Simply use the dd tool. i.e.

dd if=/dev/zero of=/fillerup.zero

then remove the /fillerup.zero file and it's done.

comment:11 Changed 7 years ago by shvelmur

I have a vm with the following config

host: windows xp guest: ubuntu 7.04 hdd : expanding on usage

I took 2 snapshots, and the file size is around 4 G. I removed the snapshots and ran dd if=/dev/zero of=/fillerup.zero, until it was about 3 G, and then ran

VBoxManage modifyvdi VDINAME compact

There is no change in the vdi file size.

Can you help me with this?

comment:12 follow-up: ↓ 13 Changed 6 years ago by Laibsch

How do I add myself to the cc list for this bug?

comment:13 in reply to: ↑ 12 Changed 6 years ago by michael

Replying to Laibsch:

How do I add myself to the cc list for this bug?

You have just done it.

comment:14 Changed 6 years ago by hufi

Although it is a somewhat roundabout way, it works quite well: Create a new VDI-drive of the desired size and then add it as second disk to the system to be resized. Tie in a CD-imgage like Knoppix or Gparted and launch the system. Then simply use dd or ddrescue to transfer the system to the new drive. Of course you will also need to resize the partitions etc.

comment:15 Changed 6 years ago by nmandery

For filling the unused blocks of ext2/ext3 filesystems with zeros there is also the tool zerofree. It claims do be significantly faster than dd.

 http://intgat.tigress.co.uk/rmy/uml/index.html

comment:16 Changed 6 years ago by jesse_kahtava

I would just like to note that when using  sdelete to zero out free space in Windows the correct switch is "-c", not "-z".

Example:

sdelete -c 

comment:17 in reply to: ↑ description Changed 6 years ago by klor

I also vote for the VDI size extension feature.
We would need at least a converter or resizer utility.

My VDI became full, and I can't use it until I do not resize the virtual hard disk.

comment:18 Changed 6 years ago by sehe

Sae request here. Started out with too small a disk size. Funny, because it seems like a verrrrrrry simple thing to implement given that the disk image file is already a dynamically expanding image?

That implies to me, that all that is required is change the upper-bound for the disk size (only an administrative change) to a value greater than before. Afterwards, one could choose from a wide range of tooling (gparted, qtparted, parted, partition magic, ntfstools, AIX smitty etc) to resize and rearrange partitions and filesystems...

Please, santa? Can we have it?

comment:19 Changed 6 years ago by grahamperrin

Please add grahamperrin to the cc list - thanks

comment:20 follow-ups: ↓ 21 ↓ 24 Changed 6 years ago by cdufour

Working with VBox 1.6.2:

  1. I can confirm that the 'dd if=/dev/zero of=/zerofile; rm /zerofile' does NOT allow to recover (all) unused space
  1. Using the 'sfill -z /' utility from the 'secure-delete' (Debian/Ubuntu) package does NOT help either

A WORKING (quick and dirty) solution is:

3.a Boot using a GParted LiveCD

3.b Downsize existing partition(s) to the strict minimum

3.b Create temporary unformatted partition(s) in the unused disk space

3.d Fill the temporary unformatted partition(s) with zeroes, using 'dd if=/dev/zero of=/dev/...'

(and optionally do the same for the swap partition)

3.e Remove the temporary unformatted partition(s)

3.f Grow the partition(s) previously downsized in 3.b back to their original size

3.g Shutdown GParted

3.h Compact VDI image with 'VBoxManage modifyvdi ...'

Hope this helps ;-)

comment:21 in reply to: ↑ 20 ; follow-up: ↓ 22 Changed 6 years ago by hufi

Replying to cdufour: Instead of using "rm /zerofile", one could use "shred --interations=1 --zero /zerofile". I have tried this with different distributions and it is working quite well (although shred is taking it's time).

comment:22 in reply to: ↑ 21 Changed 6 years ago by hufi

Replying to hufi: There is a catch in using shred: It does not work effectively with certain filesystems, especially those not writing the data directly "on-site" such as ext3, reiser, xsf, etc. I think that all those systems have an option to switch to the correct behaviour. Using ext3, you can simply mount it with the data=ordered or data=writeback option. I think the former is already the default mode.

comment:23 Changed 6 years ago by frank

  • Component changed from other to virtual disk

comment:24 in reply to: ↑ 20 Changed 6 years ago by DougNoTronik

Replying to cdufour:

Wow, finally something that worked. You're the man. If I were to guess... the reason this works is because the new partition isn't formatted and perhaps ext3 writes some info to the disk right after being zeroed as part of its inherent nature.

Glad I didn't have to use zerofree, not that it wouldn't have worked, just that I read his comments and that he claimed he hadn't fully tested it with ext3 and that kinda shied me away from it since my data is important.

Thanks

Working with VBox 1.6.2:

  1. I can confirm that the 'dd if=/dev/zero of=/zerofile; rm /zerofile' does NOT allow to recover (all) unused space
  1. Using the 'sfill -z /' utility from the 'secure-delete' (Debian/Ubuntu) package does NOT help either

A WORKING (quick and dirty) solution is:

3.a Boot using a GParted LiveCD

3.b Downsize existing partition(s) to the strict minimum

3.b Create temporary unformatted partition(s) in the unused disk space

3.d Fill the temporary unformatted partition(s) with zeroes, using 'dd if=/dev/zero of=/dev/...'

(and optionally do the same for the swap partition)

3.e Remove the temporary unformatted partition(s)

3.f Grow the partition(s) previously downsized in 3.b back to their original size

3.g Shutdown GParted

3.h Compact VDI image with 'VBoxManage modifyvdi ...'

Hope this helps ;-)

comment:25 Changed 6 years ago by qwavel

This is an important issue (compacting). I'm just adding this comment to 'subscribe' to this ticket.

comment:26 in reply to: ↑ description Changed 5 years ago by franl

Also subscribing to this ticket. I tried the "dd" trick and it doubled the size of my .vdi instead of shrinking it.

comment:27 follow-up: ↓ 28 Changed 5 years ago by frank

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

comment:28 in reply to: ↑ 27 ; follow-up: ↓ 29 Changed 5 years ago by franl

Replying to frank:

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

frank, I did exactly that. I'm seeing the same problem reported by others above. I'm trying the GParted trick (above) now.

comment:29 in reply to: ↑ 28 Changed 5 years ago by franl

Replying to franl:

Replying to frank:

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

frank, I did exactly that. I'm seeing the same problem reported by others above. I'm trying the GParted trick (above) now.

I can confirm that using GParted (as described above) works for compacting VDI disk images.

comment:30 Changed 5 years ago by frank

Decreasing the size of the guest partition and filling the unused space with zero before doing VBoxManage modifyvdi compact should help in any case. Doing the dd if=/dev/zero of=/bigfile ; rm /bigfile should help as well (of course do not use dd if=/dev/null of=/bigfile). The gain with dd might be less than filling an unused partition with zeros as not all space of a partition is occupied by raw data (because of the file system maintenance structures).

comment:31 Changed 5 years ago by neurojam

I'd also like add my vote for greater control over VDI files after they are created. In my case, I overestimated the amount of hard drive space I had to play with on the host system, and having got the guest OS (WindowsXP) just right didn't want to start again from scratch. It's basically along the lines of what other people have mentioned but I thought I would detail the process I went through in case it proved useful to someone else:

  • After defragmenting the Windows partition in the guest OS, I downloaded the demo version of BootIt, ran it as a bootable ISO, and reduced the size of the primary partition to my target size of 4Gb from an initial 10Gb.
  • In the remaining space I created a secondary partition.
  • After booting the guest into WIndows XP, I used sdelete -c (as suggested above) to zero the data on the secondary parition; this actually hung, I presume when it reached the physical (as opposed to virtual) limit of the data, but seemed to do the trick anyway.
  • I then rebooted through the BootIt ISO and deleted the zeroed secondary partition
  • Finally, in the host I ran:

VBoxManage modifyvdi my.vdi compact

So while the VDI image still has the same theoretical upper limit, the guest OS won't be able to write beyond the 4Gb partition, and the VDI file won't grow beyond this limit either.

comment:32 Changed 5 years ago by zhijun

I also vote for official VDI manipulation support. It's a very useful but dangerous for users to do the tricks by themselves.

comment:33 Changed 5 years ago by cmozuelos

In the site  http://www.techthrob.com/tech/securedelete.php I found the following statement:

"Filesystems Starting with ext3, Linux filesystems overwrite files with zeros when you delete them, rather than just marking the file as "free space." Previous Linux filesystems, as well as the Windows filesystems (NTFS, FAT) use the latter method, which is why it is so easy to recover deleted files with software tools freely downloadable on the Internet. This is something to consider when deciding how much time you should spend wiping your drive."

If this is true, for Linux ext3 partitions zero free-space fill tools are not needed since the OS is writing zeroes after any file is deleted.

The question: How true is this statement?

Thanks

comment:34 Changed 5 years ago by cdufour

@cmozuelos (concerning ext3 secure-delete feature): absolute crap! I can't believe writers of articles about secure deletion tools can make such statement (that's security for you...)!

EXT3, as currently implemented, does NOT securely delete files. This feature has been provisioned (via the 's' extended attribute)... but has NOT been implemented yet; see the manual page for 'chattr', section 'BUGS AND LIMITATIONS' for further information.

You can convince yourself empirically of that by creating a VERY large file on your disk, and then deleting it. While creating the file may take several minutes, deletion never takes more than a few seconds. That just tells you that NO time is spent going through the physical media, "overwriting" data with zeroes/random/whatever:

time dd if=/dev/zero of=/scratch/big-file bs=1k count=10000000

10000000+0 records in 10000000+0 records out 10240000000 bytes (10 GB) copied, 194.35 s, 52.7 MB/s

real 3m15.530s user 0m1.864s sys 0m42.123s

chattr -V +s /scratch/big-file

chattr 1.40.8 (13-Mar-2008) Flags of /scratch/big-file set as s-----------------

time rm /scratch/big-file

real 0m15.235s user 0m0.000s sys 0m0.804s

;-)

(Sorry for the slightly OT answer...)

comment:35 Changed 5 years ago by cmozuelos

Thanks cdufour, you are right I tried:

Nothing: did not worked as you said DD = worked but poorly, time consuming and did not shrink the VDI to its right size. zerofree = quick and effective

I started trying sfill, but I was too tired and happy with the zerofree solution.

Thanks again.

comment:36 follow-up: ↓ 42 Changed 5 years ago by titan

I have a (technical) suggestion to make compating much faster:

introduce a system call with which a guest program can tell the host VM that a particular block of the disk is no longer used. Programs like sdelete, zerofree oder gparted could be modified to simply notify the host instead of writing zeroed blocks. The host would maintain a list of unused blocks and remove them again if something is written to them. "VBoxManage" could then simply use that list instead of looking for zero blocks. I hope someone can implement that soon.

comment:37 in reply to: ↑ description Changed 4 years ago by franl

I've found a nice solution to this problem. This method has the advantage that is doesn't require any more disk space than your virtual disk is alreadying using, because the virtual disk is compacted in-place. This method uses the "zerofree" tool, which finds unallocated filesystem blocks and fills them with zero bytes.

  1. Backup your virtual machine! You have been warned!
  1. Install the libext2fs library if you don't already have it. On Ubuntu/Debian distros, just run "sudo apt-get install ext2fs-dev".
  1. Download zerofree from:

 http://intgat.tigress.co.uk/rmy/uml/zerofree-1.0.1.tgz

  1. Build zerofree by unpacking the above tarball and running "make" in the source directory.
  1. Copy the zerofree executable onto the virtual disk to be compacted.
  1. While your VM is still running, note the device names of all the partitions (e.g., /dev/sda1, /dev/sda2, etc.), including the swap partition (run "/sbin/swapon -s" to see the device name for the swap partition).
  1. Shutdown the virtual machine!
  1. Boot off a live CD (e.g., the Ubuntu install CD).
  1. For each of the partitions on the VM's disk, do the following:

8-A. Mount the filesystem read-only:

sudo mount -r /dev/XXX /mnt

8-B. Zero the filesystem's free blocks:

sudo /mnt/.../zerofree -v /dev/XXX

8-C. Unmount the filesystem:

sudo umount /mnt

8-D. Check the filesystem:

sudo fsck.ext2 -f /dev/XXX

  1. Zero the swap partition:

sudo dd if=/dev/zero of=/dev/XXX bs=512

  1. Shutdown the system, and power off the VM.
  1. Compact the virtual disk:

VBoxManage modifyhd FILENAME --compact

WARNING: This will take a long time.

comment:38 Changed 4 years ago by hanasaki

add to ticket watch / CC list please

comment:39 Changed 4 years ago by klinok

I have version 3.1.2 and it seems to be still not implemented. Voting for it too.

comment:40 Changed 4 years ago by paul.lambert

I vote for this feature as well!

comment:41 Changed 4 years ago by cdufour

I don't think it is VirtualBox's job to do (for the sake of compatibility with current and future guest OSes). The feature IS here: it is called "VBoxManage ... compact". The fact that is does not always work as expected is out of VirtualBox's realm: it comes from the way the guest filesystem(s) is (are) handled. And they are many filesystems out there, each with its own particularities, changing as versions go by. Just for Linux, there must be more than 10 filesystems available (with the most common being: ext2, ext3, ext4, reiserfs, xfs, etc.). It would not be a good idea to have VirtualBox try to keep up with all those (it would certainly lead to more problems, like data corruption). In the case of ext3/4, the votes should better be redirected to their Linux maintainers, so they implement the "secure delete" (zero delete) option. Windows is no problem, since "sdelete" does the job. My 1-penny opinion ;-)

comment:42 in reply to: ↑ 36 Changed 4 years ago by titan

Linux and Windows7 do now support the ATA  http://en.wikipedia.org/wiki/TRIM command for SSD. Virtualbox should add support for that and collect the positions of the erased blocks for compacting. The utilities sdelete, zerofree etc could be modified to send the command for other OSs.

Replying to titan:

introduce a system call with which a guest program can tell the host VM that a particular block of the disk is no longer used. Programs like sdelete, zerofree oder gparted could be modified to simply notify the host instead of writing zeroed blocks.

comment:43 Changed 4 years ago by macias

Adding to CC.

Btw. maybe this report should be splitted, because except common target "hard disk", compacting the image (file) has very little to do with resizing the virtual disk. The first one does not change the disk size on guest side, the second does.

comment:44 Changed 4 years ago by klaus

I don't see how splitting brings us any closer to a solution.

comment:45 Changed 4 years ago by macias

I don't see it too, it is just matter of avoiding duplicates, keeping 1 issue = 1 report principle, in short -- a bit of order, that's all.

comment:46 Changed 4 years ago by haridsv

Just adding to CC. Interested in seeing this fixed.

comment:47 Changed 4 years ago by GlennCoombs

Yes, well, there was already a separate enhancement request that I filed a year ago:

 http://www.virtualbox.org/ticket/3718

on being able to resize a VDI file. It just got closed and marked as a duplicate of this one :-( For such a trivial change I can't see how it hasn't already been fixed.

comment:48 Changed 4 years ago by sehe

just trying to unsubscribe from this never-ending issue... perhaps I'll have that option when I submit a reply? [sorry if that doesn't work, I cannot find the link]

comment:49 Changed 4 years ago by aeichner

@GlennCoombs: You can contribute a patch of course if you think the change is trivial...

comment:50 Changed 4 years ago by sschapiro

Well, I use VMDK files with VirtualBox and then the vmware-vdiskmanager comes in handy to just enlarge the disk.

I guess as long as VirtualBox does not implement such simple things one stays stuck to use both VMware and VirtualBox.

And, of course one does not need a VMware Workstation license for this trick...

comment:51 Changed 4 years ago by aeichner

  • Summary changed from Suggestions: Resize/Compact VDI size to Suggestions: Resize/Compact VDI size => fixed in SVN

Increasing the size of VDI images was added in trunk and will be available in the next major release. If anyone wants to try it out use a revision after r31920 (the usual disclaimer for trunk applies of course). This was the last part of the original request so I'll mark that ticket as "fixed in SVN"

comment:52 follow-up: ↓ 53 Changed 3 years ago by frank

  • Status changed from new to closed
  • Resolution set to fixed

VBox 4.0 supports resizing of VDIs. The shrink function will be implemented later.

comment:53 in reply to: ↑ 52 Changed 3 years ago by adnoh

Replying to frank:

VBox 4.0 supports resizing of VDIs. The shrink function will be implemented later.

... but it doesn't work with Snapshots !? So I've a Base.vdi and some Snapshots, I can expand the Base.vdi but I don't see the new size of the Disk in my Snapshots (or current stage)... It's not an option to me to merge all my Snapshots or to clone to a new HDD just to extend a VDI :-(

comment:54 Changed 2 years ago by bluezeak

sdelete: There's been some confusion about -c vs -z. The current version of sdelete requires -z, not -c (I know from having just shrank my drive with after using -c unsuccessfully, then used -z and the shrink worked). Also, the current documentation for sdelete says to use -z, and it's correct.

The documentation for Virtual Box says to use sdelete -c, and it is wrong.

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use