VirtualBox cannot catch USB devices if RemoteFX USB Redirection enabled

[spuch]

Host: Windows 7 SP1 Pro 64 bit. VirtualBox Version 5.2.10 r122406 Guest: Mageia Linux 6 with kernel 4.14.30 The guest additions have been installed

I had to enable RemoteFX USB Redirection on my Windows 7 Laptop using Group Policy Editor (Enabled for Administrator and User). This feature is necessary in order to redirect "other" devices like Dongles etc. to a remote computer using Microsoft RDP protocol version 8.0. But if this feature is enabled VirtualBox is not able to catch any USB device, neither a simple mass storage device like an USB stick nor a camera... The first try to connect doesn't show anything, the second try results in an error "USB device is busy with a previous request"

I tested this on Windows 7 SP1 Pro 64 bit host with a Mageia Linux 6 Guest and with a Knoppix Live ISO (Guest). When disabling RemoteFX USB Redirection on the host again, everything works as expected.

How to reproduce:

1. Open Group Policy Editor (gpedit.msc) from the command line and
2. navigate to: Computer Configuration \ Administrative Templates \ Windows Components \ Remote Desktop Services \ Remote Desktop Connection Client \ RemoteFX USB Device Redirection \ Allow RDP redirection of other supported RemoteFX USB devices from this computer
3. Set to Enabled and the rights to Administrator and User
4. Run gpupdate /force from an elevated command prompt or better do a reboot of the host as advised within the description
5. Connect a simple USB Stick to Windows host
6. Start a virtual machine within VirtualBox and try to catch this USB stick within the Guest
7. The USB device will not be connected. A second try will result in the error: "USB device is busy with a previous request"

Related tickets: There may be some parallels to Ticket #13513 but the setting is little bit different. Host is a native computer (not a guest somewhere) and Windows 7 itself is not connected via RDP.

Expected behaviour: The setting of "RemoteFX USB Redirection" should not have any impact on the USB driver used within VirtualBox

--> maybe it would be helpful to ad this setting to "USB basics and troubleshooting" within the VirtualBox forum as it took me hours to narrow this bug down

[Socratis]

Hey spuch, socratis here, the author/editor of the ​USB basics and troubleshooting article in the forums that you mentioned. Just to clarify a couple of things, I'm not a VirtualBox developer I'm a simple user, and the FAQ is compiled after years of troubleshooting and distilling different issues in a compact as possible article. So, you're lucky I was browsing the bug tracker that I saw this, otherwise your recommendation would be "lost in translation"... :D

Your case is included in the FAQ, maybe not by name, I'll add it. I'll quote the related passage, before I do the edits:

Error: Busy with a previous request (VERR_RESOURCE_BUSY)

• This error can also mean that another filter is installed on your host and is actively monitoring/filtering your USB connections. Such filters can be installed with either antivirus programs or Wireshark and its USBPcap component.

Those two cases (antivirus, Wireshark) were seen in the forums, that's why they made it by name. RemoteFX USB Redirection is going to be added to the known adversaries list. I wasn't even aware of the MS offering.

Now, take a step back and think how these "USB filters" work. As soon as a USB device is plugged in, if a USBfilter exists, it grabs the USB and potentially evaluates a set of rules, whether to redirect it or not.

Wireshark, as far as I know, keeps a copy of the USB packets for analysis. The antivirus grabs it to scan for viruses. VirtualBox checks its filters and potentially redirects the USB device to a VM. RemoteFX redirects the device to a remote computer (via RDP).

The order of the USBfilters determines which one gets the first shot at a USB device. And I'm asking you:

• If you insert a USB stick, which USBfilter would you prefer?
• At which point? Based on what criteria?
• You may want to redirect it to a VM. Or a remote computer. Or scan it. Or debug it. Which one?

Now, since these are kernel lever drivers, they all load at boot time. Methinks that all the USBfilters should include a "filtering" mechanism, i.e. don't grab everyhing in the known universe, but do something like VirtualBox does: I'm interested in *this* USB device with such and such VendorID, and such and such ProductID. And being able to change it "live". Then you'd know which device should be grabbed by which filter...

EDIT: ​USB basics and troubleshooting has been updated. Hopefully the search engines will pick up on the "RemoteFX USB Redirection".

Error: Busy with a previous request (VERR_RESOURCE_BUSY)

• This error can also mean that another filter is installed on your host and is actively monitoring/filtering your USB connections. Such filters can be installed with either antivirus programs, Microsoft's RemoteFX USB Redirection, or Wireshark and its USBPcap component. Or any other application with a USB filtering mechanism.

[spuch]

Hey socratis, thank you for the detailed explanations. I thought that Microsoft's RemoteFX USB Redirection provides some kind of "filtering" mechanism, because "USB devices can now be selected for redirection even in the middle of a remote session, and can be swapped between remote sessions or to the local PC."

Source: ​https://cloudblogs.microsoft.com/enterprisemobility/2012/09/11/remotefx-usb-redirection-in-windows-server-2012-and-windows-8/

Obviously this mechanism interferes with the one of VirtualBox so that it prevents to pass USB devices to a VM when enabled. I don't know if there is a workaround possible to use both filters together, but at least it is now documented (thanks again) and hopefully saves time for other affected users.

Maybe this problem is located only to Windows 7 hosts and Windows 8, 10, etc. have a better implementation (someone needs to test).

[Socratis]

Replying to spuch:

I don't know if there is a workaround possible to use both filters together

I described the workaround:

all the USBfilters should include a "filtering" mechanism, i.e. don't grab everyhing in the known universe, but do something like VirtualBox does: I'm interested in *this* USB device with such and such VendorID, and such and such ProductID. And being able to change it "live". Then you'd know which device should be grabbed by which filter...

But that would require a rewrite from Microsoft's part for that feature to work. A rewrite so that Microsoft can "play" well with others.

Do you see the oxymoron in my last sentence? Yes, you do... ;)

EDIT: I just thought of something... Could you enable/disable that RemoteFX service/driver from the Services in the Control Panel? I don't know if it's going to work, but give it a shot...

[spuch]

Replying to socratis:

EDIT: I just thought of something... Could you enable/disable that RemoteFX service/driver from the Services in the Control Panel? I don't know if it's going to work, but give it a shot...

I'm not 100% sure if I understood what you suggested, so here is what I did:

1. I enabled RemoteFX USB Redirection using Group Policy Editor and rebooted my system
2. I checked successfully, that now no USB device could be passed to a VM within VirtualBox
3. I stopped the "Remote Desktop Services UserMode Port Redirector" using 'net stop UmRdpService' from an elevated command prompt. -> This doesn't help to get get USB devices connecting again to VMs within VirtualBox
4. I additionally stopped the "Remote Desktop Protocol Device Redirector" service (Rdpdr.sys) using 'net stop rdpdr' from an elevated command prompt. As well no change to get USB devices connecting to a VM (a did a full shutdown of the VM in between).
5. Finally I stopped the whole terminal service 'net stop termservice' without any difference. The first two services are enabled again when restarting termservice again.

Disabling RemoteFX USB Redirection using Group Policy Editor and another reboot of my system reproducible resolves the USB issues from VirtualBox.

While I don't belief, that Microsoft will "play" a better game (at least not for Windows 7) I don't see any chance to dynamically switch USB devices either through a RDP session or to a VM within VirtualBox without rebooting the host :(

[Socratis]

Yes. It's definitely less than optimal, but that's what it is I guess...