VirtualBox x64 fails to start a VM with top-down memory allocation and DLL loading is enabled

[Alex Vakulenko]

I was testing some x64 application for 64-bit specific issues and for this I enabled the Top-Down memory allocation and DLL loading in Windows to ensure that the pointer and values are located in the upper portion of memory. I noticed that with these settings VirtualBox fails to start a virtual machine with the following error:

Failed to open a session for the virtual machine WinXP.

The loader resolved an external symbol to an address to big for the image format. (VERR_SYMBOL_VALUE_TOO_BIG).

Result Code: E_FAIL (0x80004005) Component: Console Interface: IConsole {1968b7d3-e3bf-4ceb-99e0-cb7c913317bb}

It looks like there is a problem in VB that mistreats large pointers (either by truncating the pointer value, or using it as a signed value somewhere).

In order to enable these two settings in Windows, go to registry editor and change or add the "AllocationPreference" value of "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" key to DWORD:0x00100000 as well as add bit 0x2000000 to "GlobalFlag" value of "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager" (this will normally be set to 0x20000400 in this case).

Then after reboot try starting a virtual machine.

You can read about these flags here:

​http://msdn.microsoft.com/en-us/library/windows/desktop/bb613473(v=vs.85).aspx (at the end of the page) ​http://technet.microsoft.com/en-us/library/cc779664(WS.10).aspx (search for FLG_LDR_TOP_DOWN)

[Frank Mehnert]

Thanks for the report. I could only test the Top-down memory flag as I don't have Windows server installed but even this flag makes the VM crash sooner or later. We will investigate.

[Frank Mehnert]

Actually we just found and fixed a bug which shows when the top-down allocation strategy is enabled. So yes, this is indeed a valuable test! The fix will be part of the next maintenance release.

[Frank Mehnert]

Could you check if you still see crashes with ​this test build?

[Frank Mehnert]

Fix is part of 4.2.2.