Virtualbox casuses Xorg to leak memory on host -> X.Org issue

[Nick B.]

I've noticed that after running a VM in full screen for an hour or so and switching back and forth from the VM to the host (possible cause?) many times, Xorg on the host is using over 1GB of memory. Guest OS does not seem to matter. Host is Kubuntu 16.04 using the Nvidia graphics driver.

Steps to reproduce

1. Start up a VM in full screen mode (guest addons were installed in my tests)
2. Minimize the VM several times (~25-50 times?)
3. After a while Xorg on the host will be using a lot of memory (>1GB on my system which has 8 GB of RAM)

Update: since the leaks do not go away when VirtualBox is terminated we will treat this as an issue in X.Org (or drivers) and not investigate it ourselves. We might take this up with someone familiar with the relevant parts of the X.Org source code though.

[Frank Mehnert]

We tested this with Ubuntu 16.04 (no Kubuntu so our test used a different window manager). We were not able to reproduce the problem.

[Nick B.]

Will there be anything useful in the VBox log? I didn't see anything that stood out from looking through it. I can attach that if so. I also tested this in windowed mode and had the same problem.

[Nick B.]

Actually this doesn't seem to be related to being full screen or windowed. It seems to be related to the mouse. I noticed that when the mouse was moving the memory continued to increase on the host but once I stopped moving the mouse around the memory stopped increasing. ​Here's a video of it. (The jittery mouse is probably a different bug)

[Michael Thayer]

Does the resource usage of X.Org go back to normal again after all VirtualBox processes are terminated? Otherwise I think it is more likely to be a bug in the X server. If so, please try reporting it as such, and if the X server maintainers are doubtful please ask them to comment on this ticket.

[Nick B.]

No, it does not get released after quiting VirtualBox. I have to restart X.

[Frank Mehnert]

That sounds more like a bug in the X.org server. If an X.org client is responsible for allocating memory then the X.org server should free that memory after the client terminated.

[dcendents]

I've got the same problem and I was looking at possible bugs in X.org and X.org server and right now I can only conclude the bug is from VirtualBox. This is new since VirtualBox 5.1 and I've tried nearly all releases of 5.1 and 5.2 since and they all cause that problem. I still use 5.0 because of that.

I don't know much about the internals of Xorg, but I read elsewhere that it is normal that memory will increase as its cache grows, but running free will show that used memory does not really increase and available memory does not really decrease. This is not the case for me, available slowly decrease over time, I run nothing on the host (Ubuntu 16.04.3, gdm 3, linux integrated graphics, latest pacthes and drivers installed) except VirtualBox running Windows 7 guest.

[dcendents]

Some more information, in case you find that useful.

xrestop is very low:

res-base Wins  GCs Fnts Pxms Misc   Pxm mem  Other   Total   PID Identifier    
1000000     2    1    0    8 1557    54140K     36K  54177K   ?   <unknown>
1200000    22    3    0   11  178    11515K      4K  11520K   ?   mutter guard window
1c00000    26    3    1    6 113727     2775K   2667K   5442K  3162 Win7 [Running] - Oracle V
1600000     4    3    0    5  317     2747K      7K   2754K  2286 System Monitor
2000000     9    2    1    6  364      768K      9K    777K  3225 Terminal
1a00000   135    2    2    0  152        0B      8K      8K  3105 Oracle VM VirtualBox Manager
...

and ps_mem shows most memory used is private and not shared (guest has 16 GB base memory + 256 MB video memory, 2 displays)

sudo ps_mem -p 1475,3162
 Private  +   Shared  =  RAM used	Program 

855.6 MiB +   8.9 MiB = 864.5 MiB	Xorg
 16.5 GiB +  22.4 MiB =  16.5 GiB	VirtualBox
---------------------------------
                         17.4 GiB
=================================

[Michael Thayer]

Once again, if VirtualBox causes the Xorg process to leak memory (specifically if the memory is not recovered after VirtualBox exits) then that is not a bug in VirtualBox. It might be a bug in Xorg or behaviour which is expected for whatever reason. However it it Xorg which is responsible for keeping track of resources which it allocated on behalf of VirtualBox and for freeing them when appropriate.

[dcendents]

It might be a bug in Xorg but something has changed in VirtualBox from version 5.1 onwards that expose said bug. Knowing that, it might be useful to verify what has changed and see if those modifications are necessary and can be reverted or not.

Furthermore, it seems shared memory created by the client needs to be released by the client, so it might be dangerous to assume Xorg will free allocated resources on its own. (Based on what I can read here: ​https://www.x.org/releases/X11R7.7/doc/xextproto/shm.html)

I'd love to update VirtualBox and be able to use the latest version but that is impossible and I'm merely trying to figure out what is the problem and I'm caught in that yo-yo game where everyone directs me to see the other one. I know nothing about developing Xorg apps and it is confusing enough as it is that I though this should at least be reviewed on your side by someone before blindly assuming it is Xorg.

A few years ago there was a bug reported in Hibernate and after a while it was proven that it was in fact a bug in the linux jdk, regardless of that the hibernate team "fixed" their code to make sure it worked on linux and did not wait for the java team to fix their bug.

I appreciate your hard work and I hope someone on the team can spend some time to review what has changed and evaluate if the leak can be avoided.

[Michael Thayer]

If it started with version 5.1 it might be due to the switch from Qt4 to Qt5. I wonder if there are any other similar reports involving Qt5.

[Michael Thayer]

And following up to comment:10, I doubt that our team will be able to find time to follow this up. Since you do seem to be interested the question seems to me to be if there is anything that I can do which will help you investigate without requiring any serious time investment on my side. Unfortunately I currently do not even have time to work on things I consider to be much higher priority than this, so while I am happy if I can help you I do not want to put other things aside for it.

It would be good to know what your technical skills are regarding building and debugging software.

[vicsanca]

Same problem here. xubuntu 16.04 updated and virtualbox 5.2.6. AMD R7 260X and X6 1055T with 8GB RAM. Open source drivers. Big memory leak in xorg using 3D+2D video acceleration with Windows 10. As a workaround I have disabled both options.

[odbo]

I'm experiencing the same.

Host: Linux Mint 18.3

Guests: Linux Mint, Ubuntu (16.04 and 18.04)

Graphics: Nvidia gt 710 w/ nvidia drivers - dual monitor

The leak usually starts within 30 minutes of starting a guest. Sometimes it is immediate. Once started, the Xorg memory usage jumps by a few hundred kB each time these events occur:

1) The mouse is moved over a link in firefox

2) In firefox, the mouse is moved over text where the cursor changes from pointer arrow to the 'text' cursor.

I have not seen the memory jump on mouse motions in other applications beside firefox, but I've only tested file managers, terminals, and window manager stuff.

Like the others have posted, once the memory leaks it is not returned after the guest is shutdown. My Xorg memory usage has gone upwards of 4GB before I've had to log out and restart X on the host.

Recently, I noticed the leak stops when I physically disconnect one of my monitors and then plug it back in. The same result can be achieved from the command line using xrandr to disable and then enable a monitor.

Although this stops the leak, it does not return the memory that has already leaked. It also does not prevent the leak from starting again. But it does stop it for the moment, until whatever seemingly random process starts it again.

[Michael Thayer]

I will close this ticket, since unless I am very much mistaken this is an issue with the X server (or a driver), since it does not go away when VirtualBox is shut down. We might take this up with someone familiar with the relevant parts of the X.Org source code though.