PAE/NX enabled, but NX (No-Execute Page Protection) still disabled -> fixed in SVN/3.1.4

[Jan Ritzerfeld]

I created a Vista-VirtualBox (OSE) with the PAE/NX setting enabled. A few days ago, I noticed that hardware DEP is not available. In the VBox.log I see that PAE is enabled, but NX (No-Execute Page Protection) is disabled:

[...]
************************* CFGM dump *************************
  EnablePAE          <integer> = 0x0000000000000001 (1)
[...]
********************* End of CFGM dump **********************
CPUMSetGuestCpuIdFeature: Enabled PAE
Logical host processors: 2, processor active mask: 0000000000000003
************************* CPUID dump ************************
         RAW Standard CPUIDs
[...]
PAE - Physical Address Extension       = 1 (1)
[...]
         RAW Extended CPUIDs
[...]
PAE - Physical Address Extension       = 0 (0)
[...]
NX - No-Execute Page Protection        = 0 (1)

I did not find any helpful information about NX and VirtualBox, neither in the forums (NX is too short to search, DEP finds nothing) nor via google. So I do not even know whether DEP/NX is really supposed to work with VirtualBox (OSE) and Vista.

[Jan Ritzerfeld]

Still no luck with VirtualBox 3.1.2.

[Frank Mehnert]

The reason is that your guest doesn't use the PAE mode. Is it possible that you installed your Vista guest with PAE disabled and enabled the PAE setting later? If you installed your guest with PAE enabled then your guest doesn't use PAE for another reason.

[Sander van Leeuwen]

Add a new log for 3.1.2 please. Vista will use PAE if available and not explicitly disabled in the boot config.

[Jan Ritzerfeld]

I am pretty sure that I enabled PAE/NX right from the beginning. However, I did a new installation of Vista with SP1 within VirtualBox 3.1.2 but again: still no luck.

[Jan Ritzerfeld]

Log of Vista SP1 installation with PAE/NX enabled within VirtualBox 3.1.2

[Sander van Leeuwen]

Right, NX is not enabled on 32 bits hosts. A workaround is to select a 64 bits guest type. I'll fix it.

[Sander van Leeuwen]

Note that I've fixed it only for VT-x and AMD-V enabled VMs. Software virtualization does not support NX.

[Jan Ritzerfeld]

Thank you! (VT-x was enabled anyway, I have no problem with that.)

[Sander van Leeuwen]

Please check if 3.1.4 beta 1 solves the problem: ​http://forums.virtualbox.org/viewtopic.php?f=15&t=27300

[Jan Ritzerfeld]

Using VirtualBox 3.1.4_OSE r57050 (from openSUSE OBS), Vista tells me that my processor supports hardware-enforced DEP: | C:\Users\jan>wmic OS Get DataExecutionPrevention_Available | DataExecutionPrevention_Available | TRUE Thank you!

[Sander van Leeuwen]

Thanks for the feedback.