Shared folders in Windows guest randomly getting ERROR_NOT_SUPPORTED error from CreateFile call

[Kevin Hoffman]

Inside of a guest VM running Windows 7 amd64 or Server 2008 R2 amd64, we are accessing shared folders via the \vboxsvr custom kernel mini redirector.

We have found that in rare cases an attempt to create a file (CreateFile Win32 API call) fails (returns an invalid handle) and GetLastError returns error 0x32 (decimal 50, ERROR_NOT_SUPPORTED). If you attempt to call CreateFile with exactly the same arguments, the next API call will succeed.

We have written a short test program that reproduces the problem. You first install Windows into a guest VM, install the guest tools, and then run the test program. The test program will enter an infinite loop where it attempts to create/delete the same file 2000 times and then pause for 5 seconds. The test program source/binary is attached. You run the test program like:

C:\TestVBoxFS.exe "\VBOXSVR\data\temp2.chk" > c:\test.log.txt

Also, we have found that if nothing accesses the \vboxsvr filesystem for about 10-15 minutes, and then we run our test application, the very first CreateFile call fails, but the remaining call fails. We have found this is the best way to reproduce this problem. However, if the filesystem is being accessed 24/7 we still see the CreateFile failure in about 1 out of every million CreateFile calls.

This could be potentially the cause of other tickets that mention random problems with shared folders (e.g., #7160 or #4494) [or it might not be related].

We've tried this on both Windows hosts and Solaris hosts and the error seen by the guest is the same. The ticket mentions 4.1.2 I believe we have reproduced this on 4.1.4 as well.

In general we've found the shared folder mechanism to be both reliable and fast for basic file I/O (e.g., 500 MB/sec), but the random CreateFile failures is impacting the reliability of applications using the shared folder filesystem.

Thanks a lot for your time and consideration.

[Kevin Hoffman]

Source code of test program

[Kevin Hoffman]

Binary of test program (requires MSVC 2005 amd64 CRT)

[Alex Bobryshev]

I found way to fix this bug.

Problem was found not in driver VBoxSF.sys, but in system driver rdbss.sys (filesystem provider helper).

Sometimes it access to shared folder driver as named pipe in Win 7 x64. It happens very rarely.

I try to apply some fix strategies, but only one works fine. When VBoxSF.sys reveive a request typed NET_ROOT_PIPE, it manually changing to NET_ROOT_WILD in function VBoxMRxCreateVNetRoot.

I was made driver builds for x86 and x64 and try to test it on Windows XP x32 and Windows 7 x64. It works perfectly, but it need full-scale testing.

There are no bug shows on fixed drivers.

I attach an archive with fixed source file netroot.c (src\VBox\Additions\WINNT\SharedFolders\redirector\sys\netroot.c) and driver builds for x32 and x64 (sys and pdb) for testing.

Do not forget that digital signature is need for testing drivers on x64 systems. Driver signing with test certificate should take place at testing machine. Test certificate generates with microsoft tools MakeCert: ​http://msdn.microsoft.com/en-us/library/windows/hardware/ff540213%28v=vs.85%29.aspx[[BR]]

Then you must install it to trusted root certifiacte storage. And sign a driver file (VBoxSF.sys): ​http://msdn.microsoft.com/en-us/library/windows/hardware/ff553467%28v=vs.85%29.aspx[[BR]]

There is more easy way: select "Turn off driver signing enforcement" in boot menu of testing OS and then use patched driver.

[Alex Bobryshev]

fixed netroot.c

[Technologov]

Host: Win 7 x64 + Core i7 2600K + WD Black Caviar 7200 RPM + VBox 4.1.6

Guest: Win 7 32-bit + GA 4.1.6

I used mounted Shared Folder M:\:

TestVBoxFS.exe M:\123.txt > test.log.txt

UNREPRODUCIBLE. (tested until I got 30 MB log and 1 million lines there, about 30 min test.)

-Technologov

[Technologov]

Same tests were run with Win7 x64 guest. Same result: UNREPRODUCIBLE.

-Technologov

[Kevin Hoffman]

We were able to reproduce this 100% of the time on several different host systems (both Windows and Solaris), the guest in every case was Windows Server 2008 R2 SP1 x64. You had to let the guest system idle for about 10 to 20 minutes without touching the host share (e.g., \vboxsvr\myshare) and then at the command prompt run:

C: C:\TestVBoxFS.exe \vboxsvr\myshare\myfile.txt > test.log.txt

We also had another company's tech team reproduce this issue in-house -- we originally thought it was a problem with their software, but it turns out the problem was reproducible with the test program we created.

Additionally, 'helloworld' (another party) was able to reproduce the issue reliably enough to debug the issue and provide a fix. We have tested the fix on all of the systems where we saw the issue before, and the patch does fully resolve the problem.

Please consider including the patched changes into mainline trunk. Thanks.

[Alex Bobryshev]

Updated issue solution. There are some changes to file 'netroot.c' (it fix some artifacts from issue 7160 now). It placing at src\VBox\Additions\WINNT\SharedFolders\redirector\sys\netroot.c in VBox sources.

[Frank Mehnert]

Replying to helloworld:

I found way to fix this bug.

Problem was found not in driver VBoxSF.sys, but in system driver rdbss.sys (filesystem provider helper).

Sometimes it access to shared folder driver as named pipe in Win 7 x64. It happens very rarely.

I try to apply some fix strategies, but only one works fine. When VBoxSF.sys reveive a request typed NET_ROOT_PIPE, it manually changing to NET_ROOT_WILD in function VBoxMRxCreateVNetRoot.

And this is the interesting part. It would be nice if this change fixes the behavior but we need to understand why this translation is necessary.