VirtualBox

Ticket #7353 (closed defect: duplicate)

Opened 4 years ago

Last modified 4 years ago

The Virtual Machine has a serious memory leak when using NAT in OpenSolaris

Reported by: gu99roax Owned by:
Priority: major Component: network/NAT
Version: VirtualBox 3.2.8 Keywords:
Cc: Guest type: Windows
Host type: Solaris

Description

When I set up a WinXP x64 system in Virtualbox to seed some files using uTorrent while monitoring the memory usage on the Opensolaris host (snv_b134) the memory usage kept growing until the system choked. The memory leakage stopped when I closed the bittorrent client. When I shut down the virtual machine, the memory usage returned to normal levels. The memory usage doesn't grow this way when I use bridged networking.

Other measures taken when troubleshooting while using NAT:

limiting maximum ZFS ARC : no effect disabling virtual IDE controller cache: no effect

My System: AMD based 4 core using Phenom II x4 with 4GB RAM with various ZFS based filesystems.

Attachments

VBox.log Download (82.5 KB) - added by gu99roax 4 years ago.
VBox.log.1 Download (73.9 KB) - added by gu99roax 4 years ago.
VBox.log.2 Download (44.7 KB) - added by gu99roax 4 years ago.
VBox.log.2.2 Download (44.7 KB) - added by gu99roax 4 years ago.
VBox.log.3 Download (72.8 KB) - added by gu99roax 4 years ago.
VBox(Pure.Test.Run).log Download (75.0 KB) - added by gu99roax 4 years ago.
This log was recorded specifically for the duration of using NAT
VBox.2.log Download (197.5 KB) - added by gu99roax 4 years ago.
Logs the crash that occurs when running the Machine with the modified VBoxDD.so (amd64)
VBox.3.log Download (57.5 KB) - added by gu99roax 4 years ago.
Test run with 2010.10.13 testcase package
messages(reduced) Download (390.6 KB) - added by gu99roax 4 years ago.
Submitted after the system crashes when running virtual machine using Bridged Adapter (testcase 2010.10.13)
messages(reduced2) Download (175.5 KB) - added by gu99roax 4 years ago.
The system also crashes when closing the QT GUI (should be somewhere between 15:40 and 15:54). This log comes immediately before the messages(reduced) log
VBox.4.log Download (83.1 KB) - added by gu99roax 4 years ago.
These are the logs from the 48 hour NAT test run of build 66678

Change History

comment:1 Changed 4 years ago by ramshankar

Could you please upload VBox.log for the VM? Also does this happen only with using Bit-Torrent on the guest or with other network transfers as well?

Changed 4 years ago by gu99roax

Changed 4 years ago by gu99roax

Changed 4 years ago by gu99roax

Changed 4 years ago by gu99roax

Changed 4 years ago by gu99roax

comment:2 Changed 4 years ago by gu99roax

I'm quite sure that this is not related to bittorrent but it is difficult to see in other circumstances such as down/uploading a single file using a web browser. If you know any other means of simulating a sustained flow of data and inbound/outbound packet requests I could give it a try and let it stay on for an hour to see if that will give rise to a memory leak.

These are the logs I've gathered during the past week experimenting with two different virtual machines; Magellan - Win 7 x64 (not interesting here) and McKenzie - Win XP x64. If you want I could do another test run of the NAT mode and submit a fresh log.

I also see that there are occasional failures to access shared directories (vboxsvr) for reading and writing but I guess that should be put in a separate bug report.

Changed 4 years ago by gu99roax

This log was recorded specifically for the duration of using NAT

comment:3 Changed 4 years ago by gu99roax

I have now done a pure test run using NAT and uTorrent seeding for a duration long enough to see the memory leak in the System Monitor. Then I shut down the torrent client and let the machine run at idle for a few minutes until it was clear that the memory is no longer leaking.

comment:4 Changed 4 years ago by ramshankar

We had a memory leak fix in VBoxHeadless recently, but your very last low (Pure) shows that it runs with the VirtualBox QT GUI and you still observe this leak. Am I right?

comment:5 Changed 4 years ago by gu99roax

This is a post I submitted in the mailing list 2010-09-06:

The tests I ran was while using the graphical manager (where all virtual machines are listed and shown graphically) right on the GNOME desktop on the host computer. I have run virtual machines remotely using screen and VBoxHeadless as well but not on this occasion since I used the GUI to temporarily change the virtual network adapter from Bridged Mode to NAT for my test runs.

comment:6 Changed 4 years ago by Hachiman

When you're using uTorrent what target (guest disk or shared folder) for storing are you using?

comment:7 Changed 4 years ago by gu99roax

I use a shared folder for storage located in a tank that is separate from the system files and the disk image files of the virtual machines.

comment:8 follow-up: ↓ 9 Changed 4 years ago by gu99roax

If you suspect that this leakage issue is due to vboxsrv; remember that it doesn't happen when I use Bridged Networking.

comment:9 in reply to: ↑ 8 Changed 4 years ago by Hachiman

Replying to gu99roax: Could you please verify  VBoxDD.so if incorparated changes fix issue for you? (to involve bits, please replace VBoxDD.so from your VBox installation with one pointed with URL)

comment:10 Changed 4 years ago by gu99roax

I have now tried the to replace the VBoxDD.so located in /opt/VirtualBox/amd64/ with the one supplied here. I also made sure that it had the same name, owner:group and permissions as the old file.

When starting virtual machines after that replacement they won't even start anymore. They crash with a "Guru Meditation" window. This happens regardless of whether I use NAT or Bridged Networking and affects all Virtual Machines. When flipping back to the old VBoxDD.so file the problems disappeared.

Changed 4 years ago by gu99roax

Logs the crash that occurs when running the Machine with the modified VBoxDD.so (amd64)

comment:11 Changed 4 years ago by gu99roax

The submitted log shows a failed attempt to start the Windows 7 x64 machine. It would be great if I received an indication of whether the file replacement was correct and if I should wait with the upgrade to 3.2.10 until this bug is sorted out.

comment:12 Changed 4 years ago by Hachiman

Could you please try the whole package  http://www.virtualbox.org/download/testcase/VirtualBox-3.2.8-SunOS-amd64-r64453.tar.gz? The offered fix was posponed till 3.2.12, so It wasn't included in 3.2.10.

comment:13 follow-up: ↓ 14 Changed 4 years ago by gu99roax

Could you perhaps guide me on how I preserve the settings of the previous install and leave the virtual machines unchanged while updating to that package?

comment:14 in reply to: ↑ 13 Changed 4 years ago by Hachiman

Replying to gu99roax:

Could you perhaps guide me on how I preserve the settings of the previous install and leave the virtual machines unchanged while updating to that package?

It's exactly the revision you've used before just with local changings. So settings of your VMs and VMs itself won't be affected with installing this package.

comment:15 Changed 4 years ago by gu99roax

pkgadd allows only one instance of SUNWvbox. I'm afraid that when removing the old instance of VirtualBox using pkgrm, all files will be removed including the old configuration files.

comment:16 Changed 4 years ago by ramshankar

Removing the VirtualBox package (pkgrm) only removes the VirtualBox program files. The VMs and VM config files (i.e. everything under ~/.VirtualBox) will not be touched.

What will be removed is program files (stuff under /opt/VirtualBox), drivers, services etc. So you can safely go ahead and remove/add the package.

comment:17 Changed 4 years ago by gu99roax

I have now updated the amd64 testcase version that is submitted here and I can now run the virtual machines.

The bad news is that the situation remains the same. It still gobbles memory when having a sustained data transfer with uTorrent using NAT. When the machine was booted up, 15.6% memory was allocated. It took about 32 minutes for the memory allocation to grow to 50% when using the Torrent client. The memory allocation stopped growing when closing the client, and after shutting down the virtual machine the memory usage went back down to 13.6%.

comment:18 Changed 4 years ago by ramshankar

Just to clear up somethings about memory...

Solaris does not reclaim the memory from the heap during free from mallocs. So once process memory grows you will not notice shrinking.

You're not using RAM preallocation, not all of the guest memory is allocated up front. As and when the guest touches it's memory pages the size of memory on the host will grow.

Are you sure what you're seeing is not a combination of these two conditions? Also you're talking about percentage of available host memory. 15.6%, growing, then back again is normal. You'd have to provide more precise details. Perhaps output of "vmstat 5" and VBox.log?

The more important bit is "grow to 50% when using the Torrent client". Does it stop growing at 50% ?

Changed 4 years ago by gu99roax

Test run with 2010.10.13 testcase package

comment:19 Changed 4 years ago by gu99roax

The file transfer is rather choppy; The stream is continuous for a few minutes and then it hangs for about half a minute and resumes again for another duration of a few minutes until it hangs again ... I don't know if this is due to the network connection (I also have this behaviour when using Bridged Networking) or performance issues with the storage pool. There are no hardware errors detected by the system and I've been advised to update to at least build 145. I will update when there is a stable release of OpenIndiana and it is safe to update to it.

When examining the log I see that the time-stamps don't match the time when I ran the system. In the log it says that the log is opened at 12:32 while I actually started the machine at 14:32.

After shutting down VirtualBox entirely (including the QT GUI) the system crashed/rebooted. It usually don't behave this way.

comment:20 Changed 4 years ago by gu99roax

When I did my test run I only let it grow to 50% just to establish that it is a memory leak and not a normal memory allocation. It doesn't stop at 50%, I stop it at that point by shutting down the bittorrent client. If I let it stay on for a few hours it will continue to leak memory until the system chokes. I don't have this behaviour when using a Bridged Adapter for networking.

I monitored the memory usage by looking at the "System Monitor". The Task Manager in the guest OS doesn't report anything out of the ordinary. The memory usage in it is constant and doesn't change.

But this build of VirtualBoxkeeps crashing my system! When I switched back to Bridged Adapter the computer crashes each time I start the virtual machine, both from the QT GUI and when using VBoxHeadless.

comment:21 Changed 4 years ago by ramshankar

Regarding your host crashing, that's a separate issue. Upload /var/adm/messages and we can see if it has some crash info.

Changed 4 years ago by gu99roax

Submitted after the system crashes when running virtual machine using Bridged Adapter (testcase 2010.10.13)

comment:22 Changed 4 years ago by ramshankar

I've fixed the system crash in the bridging code, thank you for the report. Too bad this issue with bridged cropped up now on this ticket with NAT.

comment:23 Changed 4 years ago by gu99roax

Also note that this crash also occurs each time I quit from the QT GUI (that is started automatically upon Gnome login). I have reverted back to the old 3.2.8 until there is a build available that doesn't crash this way.

Changed 4 years ago by gu99roax

The system also crashes when closing the QT GUI (should be somewhere between 15:40 and 15:54). This log comes immediately before the messages(reduced) log

comment:24 Changed 4 years ago by Hachiman

comment:25 Changed 4 years ago by gu99roax

Congratulations! It seems that you have made a breakthrough. I've been running the system for about half an hour and the memory allocation looks normal now (it swings back and forth between 16 and 17% of 4GB). I'll leave the system on for a day or so and return to you with the results.

I can see that there is an occasional drop in the data transfer, I have also experienced this drop when using Bridged Adapter in VBox3.2.8. So, I feel that I need to investigate this further...

comment:26 Changed 4 years ago by Hachiman

Thanks, for feedback.

comment:27 follow-up: ↓ 28 Changed 4 years ago by gu99roax

I have now run the Virtual Machine for about 48 hours while maintaining a constant stream of data transfer at ~1MB/s and the memory allocation seems to be at normal levels. The memory allocation has grown from 16-17% of 4GB to 18.9% which I assume could be due to any other factors.

The NAT implementation isn't flawless however, as I experience significant performance issues with the network connection. After keeping the bittorrent client on for a while, the connection inside the virtual machine becomes sluggish and irresponsive. For example the google.com page takes "forever" to load on a web browser while the bittorrent client is seeding. This is not due to limited bandwidth (I'm using 100Mbit fiberLAN) as other computers on the network as well as the host OS don't have these issues while the torrent client is active. Also, even after the bittorrent client is shut down the connection continues to be sluggish. I have also experienced this issue while using the "bridged adapter" option. Unless you object I will issue another bug report for this when I have tried out a few more options...

Changed 4 years ago by gu99roax

These are the logs from the 48 hour NAT test run of build 66678

comment:28 in reply to: ↑ 27 Changed 4 years ago by Hachiman

Replying to gu99roax:

Unless you object I will issue another bug report for this when I have tried out a few more options...

Ok, let's put performance issue in separate defect (and it would be better to split NAT and bridged cases).

comment:29 Changed 4 years ago by Hachiman

  • Status changed from new to closed
  • Resolution set to duplicate

duplicate of #6918.

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use