Shared folders in Windows guest randomly getting ERROR_NOT_SUPPORTED error from CreateFile call

[Kevin Hoffman]

Inside of a guest VM running Windows 7 amd64 or Server 2008 R2 amd64, we are accessing shared folders via the \vboxsvr custom kernel mini redirector.

We have found that in rare cases an attempt to create a file (CreateFile Win32 API call) fails (returns an invalid handle) and GetLastError returns error 0x32 (decimal 50, ERROR_NOT_SUPPORTED). If you attempt to call CreateFile with exactly the same arguments, the next API call will succeed.

We have written a short test program that reproduces the problem. You first install Windows into a guest VM, install the guest tools, and then run the test program. The test program will enter an infinite loop where it attempts to create/delete the same file 2000 times and then pause for 5 seconds. The test program source/binary is attached. You run the test program like:

C:\TestVBoxFS.exe "\VBOXSVR\data\temp2.chk" > c:\test.log.txt

Also, we have found that if nothing accesses the \vboxsvr filesystem for about 10-15 minutes, and then we run our test application, the very first CreateFile call fails, but the remaining call fails. We have found this is the best way to reproduce this problem. However, if the filesystem is being accessed 24/7 we still see the CreateFile failure in about 1 out of every million CreateFile calls.

This could be potentially the cause of other tickets that mention random problems with shared folders (e.g., #7160 or #4494) [or it might not be related].

We've tried this on both Windows hosts and Solaris hosts and the error seen by the guest is the same. The ticket mentions 4.1.2 I believe we have reproduced this on 4.1.4 as well.

In general we've found the shared folder mechanism to be both reliable and fast for basic file I/O (e.g., 500 MB/sec), but the random CreateFile failures is impacting the reliability of applications using the shared folder filesystem.

Thanks a lot for your time and consideration.

[Kevin Hoffman]

Source code of test program

[Kevin Hoffman]

Binary of test program (requires MSVC 2005 amd64 CRT)

[Alex Bobryshev]

I found way to fix this bug.

Problem was found not in driver VBoxSF.sys, but in system driver rdbss.sys (filesystem provider helper).

Sometimes it access to shared folder driver as named pipe in Win 7 x64. It happens very rarely.

I try to apply some fix strategies, but only one works fine. When VBoxSF.sys reveive a request typed NET_ROOT_PIPE, it manually changing to NET_ROOT_WILD in function VBoxMRxCreateVNetRoot.

I was made driver builds for x86 and x64 and try to test it on Windows XP x32 and Windows 7 x64. It works perfectly, but it need full-scale testing.

There are no bug shows on fixed drivers.

I attach an archive with fixed source file netroot.c (src\VBox\Additions\WINNT\SharedFolders\redirector\sys\netroot.c) and driver builds for x32 and x64 (sys and pdb) for testing.

Do not forget that digital signature is need for testing drivers on x64 systems. Driver signing with test certificate should take place at testing machine. Test certificate generates with microsoft tools MakeCert: ​http://msdn.microsoft.com/en-us/library/windows/hardware/ff540213%28v=vs.85%29.aspx[[BR]]

Then you must install it to trusted root certifiacte storage. And sign a driver file (VBoxSF.sys): ​http://msdn.microsoft.com/en-us/library/windows/hardware/ff553467%28v=vs.85%29.aspx[[BR]]

There is more easy way: select "Turn off driver signing enforcement" in boot menu of testing OS and then use patched driver.

[Alex Bobryshev]

fixed netroot.c

[Technologov]

Host: Win 7 x64 + Core i7 2600K + WD Black Caviar 7200 RPM + VBox 4.1.6

Guest: Win 7 32-bit + GA 4.1.6

I used mounted Shared Folder M:\:

TestVBoxFS.exe M:\123.txt > test.log.txt

UNREPRODUCIBLE. (tested until I got 30 MB log and 1 million lines there, about 30 min test.)

-Technologov

[Technologov]

Same tests were run with Win7 x64 guest. Same result: UNREPRODUCIBLE.

-Technologov

[Kevin Hoffman]

We were able to reproduce this 100% of the time on several different host systems (both Windows and Solaris), the guest in every case was Windows Server 2008 R2 SP1 x64. You had to let the guest system idle for about 10 to 20 minutes without touching the host share (e.g., \vboxsvr\myshare) and then at the command prompt run:

C: C:\TestVBoxFS.exe \vboxsvr\myshare\myfile.txt > test.log.txt

We also had another company's tech team reproduce this issue in-house -- we originally thought it was a problem with their software, but it turns out the problem was reproducible with the test program we created.

Additionally, 'helloworld' (another party) was able to reproduce the issue reliably enough to debug the issue and provide a fix. We have tested the fix on all of the systems where we saw the issue before, and the patch does fully resolve the problem.

Please consider including the patched changes into mainline trunk. Thanks.

[Alex Bobryshev]

Updated issue solution. There are some changes to file 'netroot.c' (it fix some artifacts from issue 7160 now). It placing at src\VBox\Additions\WINNT\SharedFolders\redirector\sys\netroot.c in VBox sources.

[Frank Mehnert]

Replying to helloworld:

I found way to fix this bug.

Problem was found not in driver VBoxSF.sys, but in system driver rdbss.sys (filesystem provider helper).

Sometimes it access to shared folder driver as named pipe in Win 7 x64. It happens very rarely.

I try to apply some fix strategies, but only one works fine. When VBoxSF.sys reveive a request typed NET_ROOT_PIPE, it manually changing to NET_ROOT_WILD in function VBoxMRxCreateVNetRoot.

And this is the interesting part. It would be nice if this change fixes the behavior but we need to understand why this translation is necessary.

[Alex Bobryshev]

Debugging showed next behavior: after long standby system 'forget' file share and when we calling CreateFile it try to reconnect. First connection received as pipe, then - as file share. On first connection VBoxSF.sys returned 'ERROR_NOT_SUPPORTED' and this code sending to calling context. Next connection system make as network FS - and all return successing. When we manually change request type to NET_ROOT_WILD (network FS) for rdbss.sys when receiving NET_ROOT_PIPE - calling context understands that response.

[Frank Mehnert]

I'm attaching the relevant diff here for discussion:

--- netroot.c-old
+++ netroot.c-new
@@ -175,7 +175,7 @@
     PMRX_SRV_CALL pSrvCall = pNetRoot->pSrvCall;
 
     BOOLEAN fTreeConnectOpen = TRUE; // RxContext->Create.ThisIsATreeConnectOpen;
-    BOOLEAN fInitializeNetRoot;
+    BOOLEAN fInitializeNetRoot = FALSE; //// !! Fix
 
     Log(("VBOXSF: VBoxMRxCreateVNetRoot: pNetRoot = %p, fTreeConnectOpen = %d\n", pNetRoot, pRxContext->Create.ThisIsATreeConnectOpen));
 
@@ -185,16 +185,25 @@
      * returning and then return STATUS_PENDING. Otherwise Win64 will hang.
      */
 
-    if (pNetRoot->Type == NET_ROOT_MAILSLOT || pNetRoot->Type == NET_ROOT_PIPE)
+    if (pNetRoot->Type == NET_ROOT_PIPE)
     {
+        pNetRoot->Type = NET_ROOT_WILD;
+    }
+
+
+    if (pNetRoot->Type == NET_ROOT_MAILSLOT)
+    {
+
         /* DDK sample returns this status code. And our driver does get such NetRoots. */
         Log(("VBOXSF: VBoxMRxCreateVNetRoot: Mailslot or Pipe open (%d) not supported!\n", pNetRoot->Type));
+
         pVNetRoot->Context = NULL;
 
         Status = STATUS_NOT_SUPPORTED;
         goto l_Exit;
     }
 
+
     // The V_NET_ROOT is associated with a NET_ROOT. The two cases of interest are as
     // follows
     // 1) the V_NET_ROOT and the associated NET_ROOT are being newly created.

[Alex Bobryshev]

Also we must return STATUS_NOT_SUPPORTED when system try to access VBoxSF device as mailslot. If we do not make this there are some problems with access to shared folders when it connected as network drive

[sunlover]

helloworld, thanks for your findings. When the net root type is set to NET_ROOT_WILD then the VBoxMRxCreateVNetRoot function returns STATUS_BAD_NETWORK_NAME, which seems to be a "good" status code. Simply returning STATUS_BAD_NETWORK_NAME for a pipe open also fixed the problem.

Actually it turned out that the following code is enough:

    if (pNetRoot->Type == NET_ROOT_PIPE)
    {
        pVNetRoot->Context = NULL;
        Status = STATUS_MORE_PROCESSING_REQUIRED;
        goto l_Exit;
    }
    if (pNetRoot->Type == NET_ROOT_MAILSLOT)
    ...

This seems to work because the pipe open request is related to DSF and ​MSDN says:

The Windows client returns STATUS_MORE_PROCESSING_REQUIRED to the calling application
to indicate that the path does not correspond to a DFS Namespace or a SYSVOL/NETLOGON
share, and that the I/O operation is not complete.

Could you please also test that this fix works? Thanks.

[Frank Mehnert]

We've built new 4.1 Additions containing the above fix. helloworld and other users, could you test if ​this build fixes your problems?

[Frank Mehnert]

These changes are part of VirtualBox 4.1.10.