DEP doesn't prevent execution access to non executable memory

[andrewboy]

I tried to write an exploit to demonstrate how DEP prevent simple buffer overflow attacks and what other ways are existing to bypass it, but VirtualBox surprised me. All of my standard buffer overflow exploits worked well and they didn't hit DEP.

I checked everything, and it looks like it is a VirtualBox bug. With or without "PAE/NX enabled" config (at VM settings), and with DEP always on settings under Win7 (32 bit, Ultimate N)there is NO working DEP, just Win tells you that the hw is DEP capable and DEP is on, but there is no restriction to access non executable memory and run the payload directly there!

Then I changed my guest to XP SP3, DEP is ok there!

I had to make free space on my HDD so I removed the win7 guest -> no VBox.log jet :(

$ cat /etc/issue
Ubuntu 10.04.1 LTS \n \l

$ uname -a
Linux dragon 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:58:24 UTC 2010 x86_64 GNU/Linux

$ dpkg -l | grep virtualbox
ii  virtualbox-3.2                       3.2.8-64453~Ubuntu~lucid                        Oracle VM VirtualBox

$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     P9700  @ 2.80GHz
stepping        : 10
cpu MHz         : 800.000
cache size      : 6144 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts
rep_good aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave
lahf_lm ida tpr_shadow vnmi flexpriority
bogomips        : 5585.80
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     P9700  @ 2.80GHz
stepping        : 10
cpu MHz         : 800.000
cache size      : 6144 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts
rep_good aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave
lahf_lm ida tpr_shadow vnmi flexpriority
bogomips        : 5585.95
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

[Sander van Leeuwen]

All very interesting, but not very useful without the VBox.log.

[andrewboy]

Replying to sandervl73:

All very interesting, but not very useful without the VBox.log.

I'll reinstall the guest (end of the next week) and send the log also.

Until that maybe others can confirm that.

[Sander van Leeuwen]

Ok, thanks.

[Technologov]

How is the progress on reinstalling and sending the vbox log ?

-Technologov

[Technologov]

VBox.log has not been uploaded. Please close this bug as INVALID.

-Technologov

[Frank Mehnert]

Note that NX is only supported for VMs running in VT-x/AMD-V mode.

[aeichner]

Please reopen if still relevant with a recent VirtualBox release.