MultiThreaded C# solution crashes Visual Studio 2010 debugger with corrupted memory

[Mike]

I use Virtual Box 4.1.2 with a Windows 7 x64 Guest. On my guest is Visual Studio 2010 Pro.

I'm developing a C# project which makes heavy use of threads. When stepping through the code on my guest the Visual Studio debugger throws the exception:

"Attempted to read or write protected memory. This is often an indication that other memory is corrupt"

Using Visual Studio 2010 on the host, the debugger never throws this exception. I tested multiple hardware and rebuilt guest images with the same result. I believe that Virtual Box is corrupting the memory of the Visual Studio debugger.

Attached is the C# project demonstrating the issue.

[Mike]

MultiThreaded C# project showing Visual Studio Debugger memory corruption

[Mike]

Log

[Byron Sommardahl]

I'm trying to fix this problem on my dev box as well. In absence of responses to my exact problem, I've been looking for possibly related issues. I found this question on SO that might be helpful. ​http://stackoverflow.com/questions/5187367/caching-application-block-crashing-iis

[Mike]

I made a clean install image of Win 7 32bit, tossed on VS 2010 and still ran into the problem. It's not a corrupted framework. Until the issue gets resolved I've switched to using VMWare Player which doesn't exhibit the problem.

[Jakob Østergaard Hegelund]

Replying to mbursill:

I use Virtual Box 4.1.2 with a Windows 7 x64 Guest. On my guest is Visual Studio 2010 Pro.

We see the same on 4.0.12 with WinXP x86 guests. We see this with Visual Studio 2005 as well.

The guests are dual processor - we will try to see if switching to single processor will do anything.

The problem happens several times a day for the developers.

[Mike]

I attempted many different VBox configurations and found that running under a single processor caused the exception to throw less frequently. Not much less, but noticeable.

[Jakob Østergaard Hegelund]

Replying to mbursill:

I attempted many different VBox configurations and found that running under a single processor caused the exception to throw less frequently. Not much less, but noticeable.

We still see this on 4.1.8r75467

[TimBailey]

Ubuntu 64bit host 4 cpu 8gb RAM, Win2008Svr guest 2 cpu 4gb RAM, VS2010SP1 Pro Virtualbox 4.1.20, with guest additions installed

"System.AccessViolationException was unhandled Message: Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

When debugging by attaching to w3wp.exe (IIS) I "often" get a similar message as above and the debugging session terminates.

I've disabled nested paging as suggested in this post ​http://altamodatech.com/blogs/?p=62 but still having the issue.

I can pretty easily repro, not sure what would be usefull as far as dumps. I can dump vs or w3p or something else. I've saved a "Minidump with Heap" from within VS, but its 120mb compressed. I can send if that would be useful.

Would like to help get this fixed as it is really hurting my debugging....

Thanks Tim Bailey

[Willie]

I've seen this issue too. But also when I run VS on the host. It looks like it's a VS bug but for me it manifests more often in a virtual machine.

[Petr Vones]

Same issue with Visual Studio 2012 Update 2, VirtualBox 4.2.12, Core i5 2540M, VT-x enabled. Windows 7 x64 host, Windows Server 2008 R2 guest.

Disabling Nested Paging in configuration did not help.
Using a single processor in configuration did not help.
Setting x86 platform in Visual Studio debug configuration did not help.

The issue is very annoying and probably forces me to use VMWare which I don't like because it is way too much complex and consumes more system resources.

I can always reproduce it in 10 seconds with the attached sample project. I also tried that in VMware and it works there. Please fix it, I don't believe it is a VS bug.

[Petr Vones]

The occurence of the AccessViolationException in the example code (x86 platform) decreases when "Native code debugging" is enabled but still happens. If so, it is always in the same place:

_ZwWaitForSingleObject@12:
7738F89C  mov         eax,1  
7738F8A1  mov         ecx,0Dh  
7738F8A6  lea         edx,[esp+4]  
7738F8AA  call        dword ptr fs:[0C0h]  
7738F8B1  add         esp,4  <---------------- here 
7738F8B4  ret         0Ch  
7738F8B7  nop

It looks like a corrupted stack.

[Petr Vones]

Microsoft Visual Studio Debugger QA staff resolution ​https://connect.microsoft.com/VisualStudio/feedback/details/785380/please-reopen-bug-662764 is "It sounds like this is an issue with Virtual Box. We are not aware of how to fix this on our side so we would recommend reaching out to Oracle with the feedback that their product breaks Visual Studio in a particular way."

There is 100% reproducible test case attached. Detailed steps to reproduce it:

1. download attached MemoryLeakTest.rar
2. extract (better to delete 'bin' and 'obj' directories) and open the solution in Visual Studio 2012 or 2010
3. open NumberCounter.cs in code editor
4. goto line 79, put breakpoint there
5. press F5 (Run)
6. press 'Start' button on the form
7. when the debugger stops at the breakpoint, press and hold F10 key

The debugged application will crash with AccessViolationException after approx 10 seconds. If you enable "Native code debugging" in the example project it will take more time to crash and the AV address is still the same, see the ticket comment 9 for details. Note that it never happens in non-virtualized environment or using VMware.

System information: Visual Studio 2012 Update 2 or 2010 SP1, VirtualBox 4.2.12, Core i5 2540M, VT-x enabled, Nested Paging disabled, 2 virtual CPU configured, Windows 7 x64 host, Windows Server 2008 R2 x64 guest.

Could you please indicate whether you will try to fix it or not ?

[Petr Vones]

Similar issue reported here ​https://forums.virtualbox.org/viewtopic.php?f=6&t=30841 for Visual Studio 2008 and version 3.1.6. The issue went away by downgrading to version 3.1.2. Any clue ?

[f456]

100% reprod. indeed.

It appears this issue occurs when 32 bit application is debugged on a x64 machine. Looks to me that somehow the calling into or returning from the "heavesn gate", ie the 'connection' between x64 and x86 mode, corrupts a stack.

This does not appear to be a VS bug, and should really be investigated, as the root cause may be much deeper.

[Petr Vones]

Even more complicated, I can reproduce it by all combinations. 32 bit application on x64 virtual machine, 64 bit application on x64 virtual machine and 32 bit application on x86 virtual machine. All these environments produces the same error under Visual Studio debugger when stopping on a breakpoint. Note that VMware nor Virtual PC does not have the issue on the same hardware.

[Mihai Hanor]

Can you reproduce the crash if you comment out the accessors for the Count property?
I can reproduce an access violation using a simpler C# application. But not if I use a field, instead of a property.

[Petr Vones]

Actually the issue does not have anything with deugging multithreaded code. I have much more simpe test case that is always 100% reproducible as well:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;

namespace ConsoleApplication1
{
  class Program
  {
    static void Main(string[] args)
    {
      for (int i = 0; i < 100000; i++)
      {
        Thread.Sleep(0); // <---- put breakpoint here, press and hold F10 to step over and wait few seconds to crash
      }
    }

  }
}

That is really odd. I have tried additional virtualization settings available via VBoxManage.exe only, namely '--hwvirtexeexcl on', '--vtxvpid off' and '--largepages off' but nothing helped.

[Petr Vones]

mhanor: Yes, can reproduce it as well with the property but not as a field.

[Mihai Hanor]

In your example, the function, inside the loop, can be any function. It will crash with anything that exercises the call stack.

[Petr Vones]

Yes, but this one throws SEHException instead of AccessViolationException

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication1
{
  class Program
  {
    static void Main(string[] args)
    {
      for (int i = 0; i < 1000; i++)
      {
        M1(i); // <- put breakpoint here, press and hold F5 key
      }
    }

    private static void M1(int i)
    {
    }
  }
}

Still don't know whether we can get attention to start investigate the bug. I have already asked it in vbox-dev forum https://www.virtualbox.org/pipermail/vbox-dev/2013-April/011417.html without any response yet.

Another note, setting devenv.exe process CPU affinity to single CPU core only via Task Manager affects the occurence of the bug.

[f456]

affects the occurence of the bug.

how? less frequently, after a longer period of time? This is one wired bug.

In native c++ i cannot reproduce the bug on x86-x86. Yet. I ll install c# later on to check your examples.

[Mark Cranness]

I also get occasional "Attempted to read or write protected memory. This is often an indication that other memory is corrupt" messages when stepping thru C# code in VS 2010.

Another debugging problem I have, which I'm hoping is related is: Sometimes F10 or F11 Step does not step, but instead acts as F5 Run.

I often have to set breakpoints to make sure that F10 or F11 does not run away too far ahead of where I need to debug.

[Petr Vones]

how? less frequently, after a longer period of time?

Yes, after a bit longer period of time, more step over the breakpoint line iterations. But it is still there in 100% cases anyway. I also can not reproduce it with a native C++ project but that's completely different platform and debugger.

I have also tried to insert System.Diagnostics.Debugger.Break() before the M1() method call instead of setting a breakpoint in Visual Studio and it worked fine. It seems to be as a thread synchronization (CPU cache coherence ?) issue between the debugger process (Visual Studio) and debugged one. Still, it is weird that only debugged process is affected and there are no similar issues with multithreaded code in general. Maybe there is something in VirtualBox that handles processes with 'debugger attached flag' in a different way.

Does all of you run VirtualBox on Windows host ? Any chance to check the issue on different one ?

I think there isn't anything else to try. We have to wait if the issue gets attention and will be investigated. If not, the only solution is probably to move to different vitualization software because such serious bug makes software development almost impossible.

[Mihai Hanor]

I can reproduce it on an Ubuntu 13.04 x86 host, running Windows 7 x86 as guest.

[jeff.deseret.tech]

Getting the same results here with Linux 3.8.x on ArchLinux and OS X Mountain Lion 10.8.2 and 10.8.3. Occurs much more frequently on OS X. Tolerably rare on ArchLinux.

On the OS X host, guest also occasionally throws an "illegal operation" in IIS Express while debugging.

Just attempted to disable nested paging and will see if it improves the frequency on Mountain Lion.

Guest:

• Windows 7 Professional x64 SP1
• Visual Studio 2012 Update 2
• Debugging ASP.NET MVC and WPF projects

Hosts:

• ArchLinux 3.8.x; Core i7-3770K; 32GB RAM
• Mountain Lion 10.8.3; Retina MBP mid-2012; 8GB RAM

This has happened across several recent versions of Virtual 4.2.

VT-X is enabled, with nested paging and some other stuff. Guest has 4 CPUs. As above, just disabled nested paging and will see if that improves things. Will also attempt to set CPU affinity on the debugged process.

I wonder if something has made this bug more severe in recent editions, since it went mostly unnoticed for nearly a year and a half, and now has a lot of activity.

[Uwe Geiges]

I also experience this problem for a very long time now.

Host Win 7 x64 - Core i7 2600K, Guests Win 7/2008R2 x64

I have some additional info: I can prevent this bug from happening by assigning only one cpu to the guest AND restricting the VirtualBox.exe process to only one cpu on the host (TaskManager -> Set Affinity). So the bug occurs when the VirtualBox Process is switching cpu on the host.

And, as mbursill already stated, VMware does not have this problem: The same VM that exhibits the problem in VirtualBox, when converted to VMDK runs without a problem in VMware Player.

@jeff.deseret.tech: One cause for this bug staying unnoticed for so long might be that many people (me included) never thought this to be a bug in VirtualBox but blamed Visual Studio.

[Petr Vones]

@AltGr: VMware (Workstation and ESX) nor Virtual PC have the issue. I've been using Visual Studio 2005, 2008, 2010 without a single occurence of it for many years. As you have experienced, the same virtual machine have the issue in VirtualBox only so it is hard to blame Visual Studio debugger.

I'll try your workaround to set affinity of the host process (I'd have never thought it could help). Unfortunately I do a lot of multithread code and I'd like to test it in more real environment than a single-CPU one. After all, supporting multiple cores was one of reasons to move from Virtual PC to VirtualBox. Sad.

Good news is that I've just got a reply in vbox-dev mailing list that confirms the issue https://www.virtualbox.org/pipermail/vbox-dev/2013-May/011456.html

Does anyone have a chance to check the issue on AMD CPU based system ?

[Petr Vones]

Replying to AltGr:

I have some additional info: I can prevent this bug from happening by assigning only one cpu to the guest AND restricting the VirtualBox.exe process to only one cpu on the host (TaskManager -> Set Affinity). So the bug occurs when the VirtualBox Process is switching cpu on the host.

Yes, I can confirm it helps a bit but the issue is still present anyway.

[pgates]

Over the last few versions of VirtualBox, this issue occurs more frequently on my VB VMs, too. It has become somewhat of a hindrance. Seems to show up consistently when debugging C# apps.

Host: OSX 10.8.3, i7-2820QM

Guest: Win 7 Enterprise x64, VS2012 Update 2, VS2012 SP1

[Mihai Hanor]

It can also be reproduced by enabling the option "when breakpoint is hit", by selecting "continue execution", even when delaying each execution by one second, using a script.

My 2-CPU guest crashes if I set the CPU affinity to one CPU (STOP 0x101), but doing that for a single-CPU guest, I haven't observed any improvements.

[Ramshankar Venkataraman]

So I finally got some time today to setup a Windows 7 VM, Visual Studio 2012 and debug the C# console application testcase provided in comment 18. I can confirm that it's indeed broken with VirtualBox 4.2.x and even with single VCPU VMs and is a VirtualBox bug.

The new VT-x code in trunk doesn't have this problem. I didn't have time to investigate even deeper. Unfortunately, identifying and backporting the exact fix to the 4.2.x branch will not be trivial and will require more investigation effort and time. All I can say now is, it should be fixed in the next major release, not in 4.2.x.

Maybe later we could provide you with a test build here for you to try that the fix works. Thank you for the report.