Vbox bridge loses outer QinQ tag

[Arc]

Virtualox bridge loses outer QinQ tag if bridged with Linux TAP interface

Steps to reproduce:

1. Install OS Linux on PC. In my case: Ubuntu 14.04.5 kernel 4.2.0-42-generic X86_64

2. Install VirtualBox (in my case ver. 5.1.10 r112026 (Qt5.6.1)) from ​here

3. Download and install guest Linux OS. In my case Ubuntu 16.04.01 from ​here

4. On host system create tap interface:

   ip tuntap add dev tap10 mode tap
   ip link set dev tap10 up
   

5. In guest VM netwrok settings set network adapter to "bridge" mode to interface tap10

   vboxmanage modifyvm TESTVM --nic2 bridged
   vboxmanage modifyvm TESTVM --bridgeadapter2 tap10
   

All other network settings may be set to default. I tried to change the "adapter type" and "promiscuous mode". Their change does not change the result.

6. Create QINQ subinterface on host interface tap10 with inner vlan 2000 and outer vlan 100 and set ip address (type of protocol 802.1q or 802.1ad not important. This change does not change the result):

   ip link add link tap10 tap10.100 type vlan proto 802.1q id 100
   ip link set up tap10.100
   ip link add link tap10.100 tap10.100.2000 type vlan proto 802.1q id 2000
   ip link set up tap10.100.2000
   ip addr add 10.11.12.1/24 dev tap10.100.2000
   

7. Start up guest VM and determine which interface bridged with tap10

In my case this is a enp0s8

8. Create QINQ subinterface on VM interface which bridged with tap10 with inner vlan 2000 and outer vlan 100 and set ip address (type of protocol 802.1q or 802.1ad not important. This change does not change the result):

   ip link set up enp0s8
   ip link add link enp0s8 enp0s8.100 type vlan proto 802.1q id 100
   ip link set up enp0s8.100
   ip link add link enp0s8.100 enp0s8.100.2000 type vlan proto 802.1q id 2000
   ip link set up enp0s8.100.2000
   ip addr add 10.11.12.2/24 dev enp0s8.100.2000
   

9. start watch for traffic on host PC on tap10 interface:

   tcpdump -i tap10 -vvv -e
   

   for creating pcap file

   tcpdump -i tap10 -vvv -e -w qinqPC.pcap
   

10. start watch for traffic in guest VM on "physical" interface:

    tcpdump -i enp0s8 -vvv -e
    

    for creating pcap file

    tcpdump -i enp0s8 -vvv -e -w qinqVM.pcap
    

11. Try to ping 10.11.12.2 from 10.11.12.1 (Host PC => Guest VM)

    ping 10.11.12.2
    

12. You should get the following results:
    on physical PC (tap10):

    21:22:06.345997 f6:4e:f0:31:76:45 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 50: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.2 tell 10.11.12.1, length 28
    21:22:07.346093 f6:4e:f0:31:76:45 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 50: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.2 tell 10.11.12.1, length 28
    21:22:08.346032 f6:4e:f0:31:76:45 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 50: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.2 tell 10.11.12.1, length 28
    

    on VM (enp0s8):

    21:30:21.737754 f6:4e:f0:31:76:45 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.2 tell 10.11.12.1, length 46
    21:30:22.738035 f6:4e:f0:31:76:45 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.2 tell 10.11.12.1, length 46
    21:30:23.737981 f6:4e:f0:31:76:45 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.2 tell 10.11.12.1, length 46
    

As seen, on VM side is missing outer vlan (vlan 100)

If a try to ping from VM to PC the outer vlan in right place only in ARP request. In ARP reply (PC -> VM) the outer vlan is missing. on physical PC:

21:35:12.342498 08:00:27:5d:26:01 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.1 tell 10.11.12.2, length 42
21:35:12.342501 08:00:27:5d:26:01 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.1 tell 10.11.12.2, length 42
21:35:12.342521 f6:4e:f0:31:76:45 (oui Unknown) > 08:00:27:5d:26:01 (oui Unknown), ethertype 802.1Q (0x8100), length 50: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.11.12.1 is-at f6:4e:f0:31:76:45 (oui Unknown), length 28
21:35:13.343489 08:00:27:5d:26:01 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.1 tell 10.11.12.2, length 42
21:35:13.343495 08:00:27:5d:26:01 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.1 tell 10.11.12.2, length 42
21:35:13.343516 f6:4e:f0:31:76:45 (oui Unknown) > 08:00:27:5d:26:01 (oui Unknown), ethertype 802.1Q (0x8100), length 50: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.11.12.1 is-at f6:4e:f0:31:76:45 (oui Unknown), length 28

on VM:

21:35:10.614413 08:00:27:5d:26:01 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 50: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.1 tell 10.11.12.2, length 28
21:35:10.614556 f6:4e:f0:31:76:45 (oui Unknown) > 08:00:27:5d:26:01 (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.11.12.1 is-at f6:4e:f0:31:76:45 (oui Unknown), length 46
21:35:11.615368 08:00:27:5d:26:01 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 50: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.1 tell 10.11.12.2, length 28
21:35:11.615549 f6:4e:f0:31:76:45 (oui Unknown) > 08:00:27:5d:26:01 (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.11.12.1 is-at f6:4e:f0:31:76:45 (oui Unknown), length 46
21:35:12.615063 08:00:27:5d:26:01 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 50: vlan 100, p 0, ethertype 802.1Q, vlan 2000, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.1 tell 10.11.12.2, length 28

It does not look as a TAP bug. If a try to bridge two tap interface via linux bridge, and create QINQ subinterfaces for each of them, this problem does not appear.

PCAP-s from VM and PC (qinqVM.pcap and qinqPC.pcap) in attachement.

Thanks in advance!

[Arc]

PCAP-s from VM

[Arc]

PCAP from PC

[Ovidiu Poncea]

It looks like it drops the first VLAN tag from host to quest even if it's only one.

I did a similar test your test Arc. Virtualbox 5.1.20 on Ubuntu 16.04 with an Ubuntu 16.04 guest. Both with virtio-net and Intel PRO/1000 MT Desktop.

Host config:

############
# Settings #
############

# Set variables
VLAN=10
BR="vbtest"
IP="10.1.1.1/24"

if [ $OPERATION == "create" ]; then
    echo "Creating bridges and interfaces"
    set -x
    # Create bridges
    sudo brctl addbr $BR

    # Create peers
    sudo ip link add ${BR}-v type veth peer name ${BR}-vp
    sudo ip link add link ${BR}-v name ${BR}-v.${VLAN} type vlan id ${VLAN}

    sudo ip link set up dev ${BR}
    sudo ip link set up dev ${BR}-v
    sudo ip link set up dev ${BR}-vp
    sudo ip link set up dev ${BR}-v.${VLAN}

    sudo ip link set ${BR}  promisc on
    sudo ip link set ${BR}-v promisc on
    sudo ip link set ${BR}-vp promisc on

    sudo brctl addif ${BR} ${BR}-vp
    sleep 1
    sudo ip a a ${IP} dev ${BR}-v.${VLAN}
    set +x
fi

The host 'vboxtest' bridge is configured as a bridged interface in virtualbox for my ubuntu guest.

Guest config (it has 3 interface, a NAT, a host only and the third one for VLAN testing):

from guest history:
   50  ip link add link enp0s9 name enp0s9.10 type vlan id 10
   51  ip a a 10.1.1.2/24 dev enp0s9.10
   52  ip l set up dev enp0s9
   53  ip l set up dev enp0s9.10
   56  ip l set promisc on dev enp0s9
   57  ip l set promisc on dev enp0s9

When I ping from guest to host (ping 10.1.1.1 from 10.1.1.2). On guest enp0s9 I get:

root@ubuntu-vbox:~# tcpdump -ne -i enp0s9
19:08:05.985552 08:00:27:a0:1b:ad > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Request who-has 10.1.1.1 tell 10.1.1.2, length 28
19:08:05.985697 5a:d9:37:48:4a:fc > 08:00:27:a0:1b:ad, ethertype ARP (0x0806), length 60: Reply 10.1.1.1 is-at 5a:d9:37:48:4a:fc, length 46

As you can see the VLAN tag is missing in the guest. On the host bridge on the other heand it is present:

ovidiu@laptop-pc:~$ sudo tcpdump -ne -i vbtest
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vbtest, link-type EN10MB (Ethernet), capture size 262144 bytes
19:09:24.826942 08:00:27:a0:1b:ad > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 10, p 0, ethertype ARP, Request who-has 10.1.1.1 tell 10.1.1.2, length 46
19:09:24.826977 5a:d9:37:48:4a:fc > 08:00:27:a0:1b:ad, ethertype 802.1Q (0x8100), length 46: vlan 10, p 0, ethertype ARP, Reply 10.1.1.1 is-at 5a:d9:37:48:4a:fc, length 28