VirtualBox eats my RAM

[anthony]

I have a powerful server with 24GB of ram and an Intel Xenon quad core processor that I'm using to run a bunch of virtual machines for an office on using VirtualBox.

Lately there has been a problem though. If I run free this is what I get:

total used free shared buffers cached Mem: 24733688 24531132 202556 0 79212 23492972 -/+ buffers/cache: 958948 23774740 Swap: 23437304 0 23437304

I have sysstat install so I ran sar -r on the command line and I found:

02:05:01 PM 19725876 5007812 20.25 309100 511408 4694764 9.75 02:15:01 PM 19725004 5008684 20.25 309352 511412 4697920 9.75 02:25:01 PM 19719736 5013952 20.27 309508 515608 4694840 9.75 02:35:01 PM 19708688 5025000 20.32 309696 520112 4701960 9.76 02:45:01 PM 215180 24518508 99.13 198292 19825416 4695892 9.75 02:55:01 PM 207744 24525944 99.16 198592 19833400 4695276 9.75 03:05:01 PM 200148 24533540 99.19 198904 19837592 4697116 9.75 03:15:01 PM 199720 24533968 99.19 199240 19837604 4700032 9.76

a little before 2:45 PM I had my RAM eaten. Going back through my brain I remember that at that time I ran the command VBoxManage clonehd XXXXX.vdi to clone one of my virtual hard drives.

I tried this again to make sure that the VBoxManage clonehd command was really the thing that was eating my RAM and it was, or at least running the command caused it.

I'm running VirtualBox 3.2.10 on Ubuntu Server 10.04

[anthony]

VBox.log file of the vm I was trying to clone

[Klaus Espenlaub]

You forgot to make a statement what the result of your repetition of "VBoxManage clonehd" was. When I clone a VDI file I see no memory leak. VBoxSVC temporarily needs a few MB more memory to efficiently perform the cloning, but that's not permanent.

Can you tell which process eats the memory? For such huge leaks it should be easy to find.

[Bayard Bell]

Following on previous comment: sar provides system aggregate resource utilisation. Something like top that provides per-process statistics would be more useful (e.g. "top -o rsize -i 10 -l 0"). sar is rarely the best instrument (I seem to recall that it's tied to SysV standards definitions that don't much correspond to modern problems in memory management), but it looks to be saying that 80% of the utilisation in the ten-minute sample from 02:25:01 is from filesystem cache consumption, up from 10% in the previous sample, where both the . This would be simply the OS opportunistically caching the filesystem data touched by the clone operation, not the process allocating the memory to itself and holding it thereafter as swap, heap, or shared. Such pages should be released from the cache in case of memory pressure from processes. This also appears consistent with the output from free, which shows the vast majority of memory consumption under "cached". This would also be consistent with the size of the VDI image, which appears to be 20GB (41943040 sectors at 512 bytes/sector, although I'm not sure whether the length in sector reflects dynamic consumption or the size presented to the guest, assuming that it's a dynamic VDI) but would be double that in case of clone operation (one file has to be read, the other written). That would essentially be a system- rather than application-level behaviour, triggered by use of buffered filesystem I/O in the absence of other demand.

Is there any evidence that this memory is actually unavailable for applications rather than kept around by the OS for lack of other demand? Running something like top to produce data every ten seconds or so would be preferable to system-aggregate data every ten seconds would be more beneficial in establishing this. It would be easy to establish what behaviours are in effect by performing large I/O operatings like cloning and then trying to reclaim the memory for applications by firing up enough VMs to shift back most of the memory from the cache to the VMs.

[anthony]

Hi. Thanks for your responses. I realize now that the RAM isn't being eaten but simply cached. I was running into issues a little while ago where windows VMs running on my server would suddenly reboot without explanation and I was looking for causes and found that the RAM being cached might have been an issue. I didn't know about the existence of the log files in the machines sub-directory.

I do have a couple of questions though:

1) With the pagefusion option on I seem to get VERR_PGM_PHYS_INVALID_PAGE_ID (you can see this error starting on line 905 of the above log file). I would like to take advantage of the memory de-duplication feature but it seems to cause issue with the windows VMs on this server.

2) How can I tell if my processor can use the nestedpaging and largepages features of VirtualBox?

~Thanks

[Frank Mehnert]

exquest, sorry for the late response. For asking such questions better use the forums. Anyway:

1. Not sure if this is already fixed but if not, definitely not related to this ticket.
2. Nested paging seems to be disabled in your VM configuration. Once you enabled it, either look for HWACCM: Enabled nested paging in the VBox.log file or hover the mouse over the 'V' icon of your VM.