vbi_internal_alloc() should not call contig_alloc() for large allocations => Fixed in SVN

[blakej]

A user of VirtualBox 4.1.16, with Solaris s11u1_16 as the host, commented that he was seeing "virtualbox's guru-meditation-screen-of-death due to not being able to allocate and lock memory [...] even though plenty of memory is available." The relevant comment from his logfile was:

PGMR3PhysAllocateLargePage: allocating large pages takes too long

(last attempt 8462 ms; nr of timeouts 1); DISABLE

I asked him to look into what VirtualBox was doing when it generated that error message. He found that it was in the vbox driver, making a call from vbi_internal_alloc() -> contig_alloc() -> page_create_io() -> delay().

Further investigation showed that vbi_internal_alloc() was being called with a size of 2MB, an alignment of 2MB, a high-address limitation of 2⁶⁴ - 1 (i.e. no limitation), and a request for contiguity. In other words, it just wanted a single large page.

Looking at the VBox 4.1.16 source, vbi_internal_alloc() calls contig_alloc() to get this page. As it turns out, contig_alloc() is just about the slowest imaginable way to allocate a single large page -- it assumes that it needs to respect the rest of the ddi_dma_attr_t, and it doesn't assume that it's nicely page-aligned, so it may end up trawling through huge amounts of the system's memory trying to stitch together a valid region for the caller.

If you would like information on a better approach for doing this, please contact me directly -- my email is blake.a.jones at oracle.com. (I am the technical lead for the VM2 project in Solaris.) I expect that a different approach might go 100x faster.

[jmcp]

Does this path still go through vbi_pages_alloc(), calling in to page_resv() with KM_NOSLEEP?

[blakej]

draft implementation of large page allocation/free support for Solaris

[blakej]

I've attached a sample implementation of how to allocate and free large pages using proper Solaris VM interfaces. I haven't tried compiling it, much less running it, but you're welcome to give it a try. Please feel free to contact me if you have problems or questions.

[blakej]

For those in the Solaris group looking at this, I believe my fix should work both before and after the VM2 putback.

[ramshankar]

I will have a look at this, thanks for the patch.

[ramshankar]

Calling contig_alloc() for the large page (2M contiguous allocation via PGM/GMM) is a bug (see #if 0 in memobj-r0drv-solaris). We should call into vbi_large_page_alloc() and not vbi_phys_alloc()->contig_alloc().

We shouldn't have any code that requires > 2M contiguous physical memory.

[ramshankar]

Fixed in SVN and backported to 4.1. Fix should be available in 4.1.20.

[frank]

4.1.20 was just released.