Starting from VirtualBox 2.1, packets are duplicated when written to a TAP interface

[Florian]

I think the title says it all. After using VBox 2.0 for a long time, I switched to 2.2, and noticed all packets sent to a Tap interface (created using "tunctl") from the virtualized OS were duplicated. Ping would report "DUP!" packets, and this is confirmed by Wireshark for all protocols. I went back to VBox 2.1, and the problem is there too. VBox 2.0 works normaly.

To reproduce, it's quite simple: using latest VBox, configure a VM to boot using a tap interface (create it using "sudo tunctl") as network interface by selecting the Tap in "Network Configuration" -> "Access By Bridge" -> and select "tap0". Then start tcpdump or Wireshark on tap0, and then boot the VM. As soon as the VM is going to emits packets (for DHCP requests, for ARP, broadcast etc...) you will notice that all packets are duplicated. Try again with VBox 2.0, no such problem.

As always, if more info is needed, I would be happy to provide it. I would really like this bug fixed, it's quite a blocker for some software I'm working on...

[Florian]

After going to #vbox on freenode, I was suggested to use hostonly networks to see if it helped. It doesn't, either bridged mode or hostonly duplicate packets.

I've also tried to create a bridge comprising only of a tap, give that tap interface to the vbox bridge mode, and sniff the bridge and the tap using two distinct Wireshark. Well ho and behold, I get two packets on the tap (as usual), and only one on the bridge. So apparently a bridge has some inside logic not to write a packet twice.

I can confirm that observation by giving the bridge directly to the bridge mode of vbox, and sniffing both bridge and tap, and in that case, I get, as one would expect, two packets on the bridge interface (wireshark gets it before the bridge does, so it cannot filter yet) and only one on the tap (so the bridge has filtered out the duplicated packet).

Here are the vbox log from a session with simply a tap given to the bridge mode of vbox. As always, packets were duplicated. Vbox is running as root in this particular session, but I've done most of my testing using my regular user account, results are the same.

~/.VirtualBox/Machines/name_of_vm/name_of_vm.xml -> ​http://pastebin.com/m50293cde ~/.VirtualBox/Machines/name_of_vm/Logs/VBox.log -> ​http://pastebin.com/m414e874

Tests were done using latest VBox 2.2.2

[Aleksey Ilyushin]

Could you turn on debug logging and reproduce the problem? To turn on kernel module logging add ' -DDEBUG -DDEBUG_aleksey' to line 221 in <vbox_install_dir>/bin/src/vboxnetflt/Makefile. The line you need to change looks like this:

KFLAGS  += -DVBOX_WITH_64_BITS_GUESTS

Just replace it with:

KFLAGS  += -DVBOX_WITH_64_BITS_GUESTS -DDEBUG -DDEBUG_aleksey

Then, please, rebuild the modules with:

sudo /etc/init.d/vboxdrv setup

Start VM, do a couple of 'pings' and attach the kernel log (/var/log/kern.log) here.

[Florian]

Kernl logs with debug activated in vboxnetflt

[Florian]

PCAP file corresponding to the kernel logs activity

[Florian]

Ok, did that. My setup was the following: Vbox 2.2.2 from the official .deb, with an interface connfigured as 'Access by bridge" connected to a TAP (tap0 in the logs). As usual packets are duplicated, I've also attached the associated pcap capture. As you can see, all packets are repeated identically within ~60µs. I've browsed the kernel logs rapidly, and it definitely looks like it's VBox which is writing the packets twice. Thanks for taking the time.

[Aleksey Ilyushin]

Note that all packets captured are multicast packets. Since vboxnetflt "connects" to host's network stack in the "middle" it is completely normal that multicast packets are sent twice by VirtualBox: the first to the wire (down host's stack) and the second to the host itself (up host's stack). Both copies are captured and logged by the sniffer. Can you do a couple of pings so we see some unicast packets in the capture?

[Florian]

Ok, did that too, took me longer this time because at first I could not reproduce the bug. Apparently, the problem does not appear if I ping between a virtual box on a TAP interface and the host (meaning I give an IP to the TAP, and ping it from the Vbox), or if I ping between two virtualbox on the same TAP. But, it does appear again when I do the following setup: two virtualBox, each one on its own TAP, and a bridge linking the two. In that case, unicast packets are duplicated again. But, as I mentioned previously, bridges have a filter that removes duplicate packets apparently, as I get only one of the two packets in the other TAP after crossing the bridge. As before, I've attached the pcap capture from one of the two TAPs, and the corresponding kernel logs.

[Florian]

BTW, I still think having packets duplicated in a sniffer input should still be considered a bug, even if they are multicast or broadcast packets.

In the software I'm working on (a network simulator), I use a raw socket to read packets in promiscuous mode on a TAP interface, which is standard procedure, as it is the way it's done in libpcap if I'm not mistaken. And reading duplicate packets, even for multicast, is a pb, as it distorts the simulation by introducing copies of packets that were never emitted by the guest. It's not catastrophic, but still, it makes the simulation less realistic. Without wanting to start a flame war, KVM does not exhibit this behaviour :)

[Aleksey Ilyushin]

I could not find any logic for detecting duplicate packets in Linux bridging code. You can see for yourself: ​http://lxr.linux.no/linux+v2.6.29/net/bridge/br_device.c, take a look at br_dev_xmit(). The reason you see duplicate packets on one of the TAPs is that the bridge puts all interfaces attached to it into promiscuous mode, in which case VirtualBox sends all packets both up and down the stack so both actual recipient and the sniffer (in this case, the bridge) get them. Only the ones going up will reach the other TAP connected to the bridge (that is why you see no duplicates on the other TAP), the ones going down are sent to the wire (in case of TAP that would be a user-space program reading from TAP's control device).

Do you need to analyse the traffic in real-time or post-mortem analysis will do? If no real-time is required you may consider using VirtualBox' embedded sniffer (check -nictrace option of VBoxManage modifyvm).

Btw, why would you want to use bridge to connect two VMs? It can be easily done using the same host-only adapter in both VMs. In such case both VMs will share the same internal 'virtual' network which will be 'connected' to outside world through host-only interface (vboxnetN).

As for KVM I cannot tell for sure, most likely it does not use this kind of attachment/injection and uses TAP control device to relay packets to/from host's network stack, just like VirtualBox did in older versions.

[Florian]

Indeed, you are right, there's apparently no code to handle duplicate packets in linux bridges. And after your explanation, I understand better what's happening.

Concerning the bridge between two VMs, that's simply because I was looking for a simple use case that would reproduce the pb. It's the closest I could find without needing you to download my project to test, as the behaviour of the bridge is almost identical to what I'm doing. Well the difference being that the bridge code lives in kernel land, and thus only receives packets coming up the stack, where my sniffer receives both, according to your reasoning.

I do have one gripe with your reasoning though, and it's that I don't understand why my sniffer would receive packets going *up* the stack ? Shouldn't it be too "late" ? Also, couldn't VBox simply inject its packets at the bottom of the stack and let them trickle up ? Why do you inject them half-way, both up and down ? And if I understand correctly, the only reason you send the down also, is to allow sniffers to see those packets. Am I right ?

Also, using -nictrace won't do, as my software needs the packets as soon as they're sent by the guest.

May I ask why you changed the way VBox does its networking stuff ? Previous versions worked fine for me :)

All in all, possibly not a bug per se, but an undesirable behaviour from my point of view :)

[Florian]

Me again. I ran some more tests today:

• I have exactly the same behaviour if I use a "normal" interface such as eth0.
• I tried using a more recent version of Ubuntu (9.04), but still the same.
• Also, same behaviour if I use the "host only" network configuration.

Just to make sure, you do see that behaviour on your end too, right ?

[Florian]

Sorry, I figured the '*' would be converted into a list, and did not bother previewing. here is the same message as above, properly displayed:

---

Me again. I ran some more tests today:

• I have exactly the same behaviour if I use a "normal" interface such as eth0.
  
• I tried using a more recent version of Ubuntu (9.04), but still the same.
  
• Also, same behaviour if I use the "host only" network configuration.

Just to make sure, you do see that behaviour on your end too, right ?

[Aleksey Ilyushin]

Replying to fvichot:

Concerning the bridge between two VMs, that's simply because I was looking for a simple use case that would reproduce the pb. It's the closest I could find without needing you to download my project to test, as the behaviour of the bridge is almost identical to what I'm doing. Well the difference being that the bridge code lives in kernel land, and thus only receives packets coming up the stack, where my sniffer receives both, according to your reasoning.

The bridge is very different from a sniffer since it is a device and it registers callbacks through which it gets the packets while the sniffer is a packet handler which registers a callback that is called both from dev_queue_xmit (dev_queue_xmit_nit to be precise) and netif_rx (via netif_receive_skb). VirtualBox netfilter code acts just like any other sniffer. I suggest you get familiar with net/core/dev.c if you want to understand how VirtualBox "connects" to kernel stack.

I do have one gripe with your reasoning though, and it's that I don't understand why my sniffer would receive packets going *up* the stack ? Shouldn't it be too "late" ? Also, couldn't VBox simply inject its packets at the bottom of the stack and let them trickle up ? Why do you inject them half-way, both up and down ? And if I understand correctly, the only reason you send the down also, is to allow sniffers to see those packets. Am I right ?

Just like I said before your sniffer (as well as any other sniffer, including vboxnetflt.ko) registers itself as a packet handler and is called both in receive and transmit paths. As for injecting packets VirtualBox does it at the kernel/driver boundary which can be considered as the bottom of hardware-independent stack. Still packets must "go out" to external network via hardware driver (dev_queue_xmit) as well as "go in" to host's stack (netif_rx). For unicast packets it is done selectively depending on where the packet must be delivered, unless the interface is in promiscuous mode which forces unicast packets to be delivered to the host as well even if they are destined to external network.

May I ask why you changed the way VBox does its networking stuff ? Previous versions worked fine for me :)

The majority of users is not concerned with duplicate multicast packets in sniffers while appreciating the ease of "setting up the bridge" between real and virtual networks.

All in all, possibly not a bug per se, but an undesirable behaviour from my point of view :)

I'll try to find a way to "hide" redundant packets from sniffers while not affecting other users, but I cannot promise it.

[Florian]

Ok, I get it now. Quick recap if anybody stumbles one this page for the same problem: My sniffer sees two packets because vboxnetflt sends it twice, once to the host's stack, and once to external hosts. And raw sockets are not the way to go for you because they only send packets written to them to external hosts, not to the host's stack.

So not a bug, but an undesirable feature :)

If you find some way to hide the duplicate packets from the sniffer, that'd be awesome. Possibly, if you have some way of detecting tap interface, maybe you can can skip sending the packets up the stack, AFAIK a tap doesn't have a stack (notably, can't ping a tap interface).