Huge new memory leak => Fixed in SVN

[luxifer]

It seems like since VirtualBox 4.0 there is some huge memory leak. I have a VM in which my firewall runs. It was created with a 3.2 Version of VirtualBox and transfered to a new server with VirtualBox 4.0. I even tried creating a new VM on the 4.0 host and just reusing the old disk image (I need this :-))

So the VM mainly does network i/o... nat, routing, vpn, firewall stuff...

The VM i configured like so:

• 192MB RAM, 5MB VRAM, OS: Linux 2.6
• Chipset: PIIX3, IO-APIC not active, no absolute pointing device
• 1 processor, pea/nx active
• VT and nested paging active
• no 3D accel
• no remote control
• ide controller ich6; sec master is dvd; uses host i/o cache
• sata controler ahci; port 0 is disk image; uses host i/o cache
• no audio
• 2x nic bridged to the same physical nic, using virtio
• no serial interfaces
• no usb
• no shared folders

oh and: no guest addition installation possible

It didn't even run for 24 hours and I have performance counters that makes the question why my server seems to slow down a no-brainer. As of the moment I just shut it down:

• Private Bytes: 8.442.504 KB (With Peak Private Bytes just a few KB above)
• Virtual Size: 8.575.196 KB
• Working Set 28.624 KB (Whaat?)
• Peak Working Set: 3.023.380 KB (Ah, got you... how many times did I tell you to clean up after you leave a mess???)

Thing is: that thing even started to route slower. Webinterface was slow. GUI was completely unresponsive - as were any other manager gui windows. Eventually I had to kill this thing.

Total CPU time of almost 24h of running was about 40 minutes. The VM process ran under "above normal" priority.

Host is Windows Server 2008 R2; VirtualBox 3.2.x worked fine!

So dear devs: could you fix it, please? :-) it's really nasty

if you need anything else to get to the root of this numbers, I'm happy to provide it, if i can.

[luxifer]

HAH, I got you twice, bad bug! I am able to reproduce the bug reliably!

Do the following:

• Boot the Linux VM (still boot menu, no kernel running :-))
• close the VirtualBox Manager Window (the one with the VMs listed, not the one with the VM of couse)
• Start VirtualBox again (reopen the VirtualBox Manager window via a Shortcut to VirtualBox)
• BAM: The Private Bytes of the VM just exploded from some Megabytes to whopping 4 Gigabytes in just the blink of an eye (an I'm not joking)

Do it as often as you will...

Workaround so far: Don't close that freakin Manager window... (but that kind-of sucks)

Cheers

[Frank Mehnert]

Please attach the VBox.log file of the VM session. The memory leak you observed could depend on the VM configuration.

[Alil]

I have same too and it is exactly reproducable
I think it can be related to other crashes too

Host: Win7 x64
Guest: WinXP SP3 x86

[luxifer]

There's no log entry when the bug takes place... I noticed though, that this bug seem to crash the VM that was started last. I have had a second VM running with Ubuntu Server 10.10 64 Bit. As soon as I triggered the bug *this* machine was hit by it... There's no log entry at this time though. So I give the log for this one...

[sunlover]

The memory leak is not reproducible here so far.

diagiman, please attach VBox.log for the VM, where you have this problem.

luxifer, is the bug reproducible when the VM network uses NAT instead of bridged mode? What happens if you disable VM Preview in the VirtualBox Manager window (double click on the Preview title).

Thanks.

[Alil]

Host Win7 x64
So, here are reproduce tests on newly created VM:

1. Create new VM, Guest WinXP default, 256 MB RAM(config attached)
2. Run it from manager (so there is two VirtualBox.exe processes: one manager, other vm itself). VBox warns about boot cancel. VBox will stop with boot error.
   RAM: Manager process ~20MB, VM ~56MB, VBoxSVC ~10MB, Driver locked ~47MB
   
3. Leave VM to stay in boot fail. Close Manger application. Then start it again
   RAM: Manager process ~14MB, VM ~4GB!!!, VBoxSVC ~10MB, Driver locked ~47MB
4. After that VM hangs when trying to close it (stops at 14% and only increases remain time). So I killed VM process. Ant it gets aborted state
   

VBox log is attached but it doesn't have any memory leak related entry

Yes it is preview related!! There is no leak if preview disabled
And it looks like that afected only "current" VM in VirtualBox, not the other ones.

[sunlover]

Thanks diagiman! I still can't reproduce the leak, but the information that it is related to the Preview is very helpful.

[luxifer]

I just reproduced it on my machine at work, so for me it's already 3 boxes, that show this bug (though all are w7 or s2k8r2 64 bit)...

I tried turning off the VM Preview first... the bug doesn't show if VM Preview is disabled but it comes right back the moment you enable VM Preview again... that should narrow it down a bit ;)

Interesting thing though: When the manager GUI starts it does something different with the preview thingy than when you turn it on again... If you restart the Manager with preview disabled, there's no memory leak... if you re-enable it it works as expected and there's also no memory leak... only if you restart the manager with preview set to on

[sunlover]

Please try

​http://www.virtualbox.org/download/testcase/VirtualBox-win-rel-4.0.1-r69271-MultiArch.exe

I've found one place were the memory could be leaked. Let me know how the build behaves. Thanks.

[Alil]

It is still reproducible by the same manner with 4.0.1 r69271
The only difference it saw was Manger restart slowness, but the memory still leaks
It looks strange that the leak stops at 4GB limit (I have 8GB RAM)

[Frank Mehnert]

Thanks for testing. Could you attach a VBox.log file of a VM session from this test build as well?

[Alil]

Strange thing happened - I can't reproduce it any more :)
The only thing I did on the host was host adapter reconfig (as it resets after upgrade)
After that I removed all VB software form host, rebooted and reinstalled 4.0.0 final. But I still can't reproduce it in any case.
I don't know what happened to VBox or host itself, hope bug is gone in test build.

Anyway thank you, for your great work

[hex]

Same problem here, 4.0.0

Installed it fresh on w2k8R2, created one VM, installed ubuntu 10.10 and left it running for about 24h. Then when I started manager it leaked massively so a whole server stopped for about a minute.

I turned of preview and i think now is ok.

[sunlover]

VirtualBox 4.0.2 has been released and includes the memory leak fix. Please try the new version and let us know if it fixes the problem.

[hex]

Nice, Thanks. You forgot to mention this bug in changelog so i thought it wasn't fixed...

I have preview turned off now and I don't want to play with it because I don't need it and it is production server...

[Alil]

The bug came again after VBox 4.0.2 install
It is reproducable by now if VM preview is enabled

I don't know what extra info is needed to fix it.

After leak hapens live snapshot (also sometimes suspend), usb and may be result to crash.

Currently I don't need/use preview of VM, but I can test if needed.

[Frank Mehnert]

diagiman, any chance to attach a VBox.log file with VBox 4.0.2? You don't need to wait until all memory is consumed but please wait for some time with preview enabled before you stop the VM. Unfortunately we still cannot reproduce the problem.

[sunlover]

diagiman, please reproduce the problem with 4.0.2 and send me VBox.log and the VM XML config file. The memory is still leaked in the VM process, right?

Also when the leak happens, is the preview working correctly, that is it contains the actual image of the VM and the image is updated?

[Alil]

It created new VM guest-type WinXP with default options
Ran it, vbox showed dialog for new VMs - canceled it and it stoped at boot error
Then I executed desktop shortcut "Oracle VM Virtualbox" which launched with a litle delay. After that I see that TestVM process gets 4G RAM. I leaved it open for a couple minutes, changed video mode, nat cable connect. VBox crashed only when I launched another VM with 3G guest ram. May be it was host initaited beacause memory lacked.
If there is few other running VMs leak only Vbox manager current VM and only if preview enabled. Preview it self is drawn right.
So memory leaks only at VBox manager start when preview is enabled and only for current running VM in the managers list.

[sunlover]

Thanks diagiman. BTW you said "I executed desktop shortcut "Oracle VM Virtualbox" which launched with a litle delay. After that I see that TestVM process gets 4G RAM." Does this mean that the TestVM process memory increased to 4GB immediately, or it increased gradually and reached 4GB after some time?

[Alil]

I think it was gradually but very quick as of i saw 2G value in task manager when is was changing. And the delay was related with host os. I tried to switch off preview update immediately after another Oracle VM Vbox launch, but was unable to get it so quick.

[sunlover]

Thanks. This is important information. So it took just a couple of seconds to reach 4GB, right?

[Alil]

Yes

[sunlover]

diagiman, please try ​this testbuild?

Thanks.

[Alil]

It is still reproducible with testbuild

[Frank Mehnert]

From your log it seems that you just started a dummy VM (no boot medium), is that correct? And you still saw 4GB memory consumption with that VM which did not happen if you disable the preview window? Unfortunately still unable to reproduce here using a Windows 7 host.

[Alil]

Yes that's right. I created dummy VM (TestVM), it's vbox file is already attached. But it leaks for other long-lived VMs too. May be it is x64 host related? It looks strange that it consumes only 4Gb.

[pentagonik]

This issue should be fixed in the latest sources already and a bugfix will be available with the next upcoming release of VBox. Please re-open the ticket if this issue is not solved for you using the upcoming version. Thanks for the report!

[sunlover]

diagiman, to make sure that the problem has been fixed, please try ​this new build.

Thanks.

[Frank Mehnert]

This ticket will be closed when the next maintenance release was published.

[BARTHES]

Hello eveybody,

I also think there is a big memory problem. In my server (ubuntu server 10.04LTS AMD64 - 1 XEONL5520 -MB SUPERMICRO X8ST3-F - 18GB of RAM) i use virtualbox 4.02.

However, with only 1 VM under windowsXPPro 64 with 1024Mo of RAM, virtualbox uses ALL the available memory (that is to say, 17Gb......not fair.)
And the other process have not more memory....) I attached the log of virtualbox.

Thanks. Christophe

[sunlover]

cbarthes35, do you also use the Preview in the VirtualBox console? If yes, then try to disable it.

[BARTHES]

I tried with preview disabled, but unsuccessfully. Memory still fills in inexorably.

Moreover, I tried to compile svn version but I'm stuck with this error message and don't know what to do since the paths are correct and the files are not missing...

Config.kmk:2028: ....out/linux.amd64/release/GCCConfig.kmk:
    Aucun fichier ou dossier de ce type
Config.kmk:4685: ....out/linux.amd64/release/revision.kmk:
    Aucun fichier ou dossier de ce type
kBuild: Generating ....out/linux.amd64/release/revision.kmk

Thanks. Chris.

[sunlover]

cbarthes35, could you try the build:

​http://www.virtualbox.org/download/testcase/VirtualBox-lin64-4.0-4.0.3-r69765.run

Thanks.

[Frank Mehnert]

Regarding the compile messages: These messages are normal and you can ignore them.

[BARTHES]

It seems that it has fixed the memory problem.

Thanks very much.

Chris.

[BARTHES]

i'll keep you informed if there is a change.

[Alil]

It seams too be fixed in r69765. There is no leak now.
But in log at previously "leaked place" appears strange entry:
00:00:06.287 ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={09eed313-cd56-4d06-bd56-fac0f716b5dd} aComponent={Display} aText={Argument height is invalid (must be height <= 32767)}, preserve=false

Thank you for great work!

[sunlover]

Thanks for testing. The E_INVALIDARG message is expected with this build. This is exactly the situation which caused the leak. The message has been already fixed in SVN and the fix will be included in the next VBox release.

[glombus]

I opened the duplicate ticket #8215.

Didn't notice this one when searching for similar problems.

​This build did not resolve my issue. Memory usage still jumps at very fast rate after GUI is closed and re-opened after having launched a VM.

[Alil]

Hi. Try ​this build.

[Frank Mehnert]

glombus, please test the build diagiman advised. The 69722 build did not contain the final fix. Thank you!

[flyerlevrai]

ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={09eed313-cd56-4d06-bd56-fac0f716b5dd} aComponent={Display} a Text={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false

[flyerlevrai]

ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={09eed313-cd56-4d06-bd56-fac0f716b5dd} aComponent={Display} aText= {Argument address is NULL}, preserve=false

[glombus]

Thank you, the build from diagiman worked perfectly. As far as I can tell the issue is now resolved for me.

[Frank Mehnert]

Should be finally fixed in 4.0.4.

[Marc]

aaahooogah vbox.log

[Marc]

I'm running VirtualBox 4.0.4r70112 on Windows 7 x64.

Single VM running FreeNAS, configured to use 1024MB RAM.

Windows Task Manager reports VirtualBox.exe using 3.7GB RAM. How much RAM should I expect VirtualBox.exe to use?

[Marc]

Update:

• If I stop the FreeNAS VM, the memory is returned to the operating system.
• If I restart the FreeNAS VM, Task Manager reports that virualBox is using about 100 MB of RAM.

The issue seems to pop up when I'm copying large amounts of data into the FreeNAS VM.

I have the VM configured to use 10 virtual drives (created with VBoxManage, each 900 GB in size). I have FreeNAS configured to use all 10 drives in a RAID 5 stripe.

Hope this helps track down the issue.

[Edmar (Macrovita)]

I'm using VirtualBox 4.0.4r70112, WinXP SP3 32-bit host, Linux Fedora 14 guest.

I'm getting memory leak in the host VirtualBox.exe process. To verify, install bonnie++ Linux package on the guest and run the following loop (as non-root user, to run as root add [-u root] option to bonnie++):

while (true) do date; bonnie++ -b

break; done

Host VirtualBox.exe process memory rises continually over time. Eventually (few hours later) the VM fails on the host, with Windows EventLog: Source: Application Error, EventID: 1000, Message: Application Failure virtualbox.exe 4.0.4.0 in vboxrt.dll 0.0.0.0 at offset 00023458.

The problem occurs with Fedora 14 64-bit guest (tested chipset PIIX3 or chipset ICH9, SATA AHCI) and Fedora 14 32-bit guest (tested chipset PIIX3, no IOAPIC on the host, SATA AHCI or SCSI LSI Logic). I have not tested other guest OSes.

For self reference (please ignore): BTM3505.

[Edmar (Macrovita)]

Correction:

while (true) do date; bonnie++ -b || break; done

[Edmar (Macrovita)]

Using VirtualBox-3.2.12-68302-Win.exe (instead of VirtualBox-4.0.4-70112-Win.exe), the memory leak identified through bonnie++ does not occur. WinXP 32-bit host.

[Frank Mehnert]

macrovita-erw, are you 100% sure that the memory leak happens as a result of the bonnie++ test? Could you check what happens if you leave that VM idle?

[Frank Mehnert]

Oh, and please attach a VBox.log file of one of these VM sessions when you performed the bonnie++ benchmark.

[Frank Mehnert]

Confirmed, attaching the log is not necessary anymore. Thanks for the testcase! Although this leak is completely different from the one of the original reporter.

[Frank Mehnert]

And this leak will be fixed in the next maintenance release of course. For reference: This is r36568.

[Edmar (Macrovita)]

Hi frank, thanks a lot for the updates! I was able to confirm the memory leak again, but apparently due to some problem at the website/bugtracker, I wasn't able to login/update the bugtracker for days ( ​http://forums.virtualbox.org/viewtopic.php?f=1&t=40539 ).

Anyway, very happy the bug was identified and fixed for the next release. I'm migrating from VMware Server 1.0.9, and the active good work on VirtualBox is very reasssuring. Thanks!!