Ticket #7059 (closed defect: fixed)
Severe networking issues with Intel 82575/82576 NICs and igb driver with VBox Bridging
|Reported by:||crash0veride||Owned by:|
Description (last modified by frank) (diff)
The Issue manifests itself as the following:
- Slow networking performance in the guest with I/O intensive network operations.
- Slow network performance in the guest with certain network protocols and operations.
- Excessively large amounts of TCP re-transmissions
- Constant out of order TCP packet deliveries
- Excessive TCP segmentation
- Excessive TCP recv window overflows
- Adapter resets
The cause seems to be some improper interaction with VirtualBox bridging and the Generic Receive Offloading (GRO) feature which is now defaulted “on” in the igb driver. This only occurs on systems with the 82575 or 825756 ethernet controller and does not occur on other ethernet controllers or on Intel ethernet controllers using the e1000e or e1000 driver.
- The issue can be reproduced with Linux or OpenSolaris as the host OS with the common thread being the igb driver.
- I cannot reproduce this issue with Windows 7 x64, Windows Server 2003 R2 x64, or Windows Server 2008 R2 x64 as a host OS.
- The version of the Intel igb driver is out of date on some distros (EG: opensuse 11.2 = 1.3.16-k2 and centos was a 2.x version of the driver)
- I have compiled and installed the latest version of the "igb" driver from intel on tested out of date distros, (2.2.9)
- With the new driver however the issue is still there
- Changing the networking from bridged to NAT on the guest makes this problem go away compeletely.
- Guest network adapter type did not matter (EG: intel, virtio)]
- Occurs under all guest OS (EG: Windows, Linux, OpenSolaris)
The easiest way I have found thus far to reproduce this issue: Load a system that has and 82575 or 82576 ethernet controller with Linux (any linux works choose your distro) or OpenSolaris b134. Load VBox and install a guest (Guest OS does not matter) Bridge the guest to an interface EG: eth0, igb0 Within the guest do a "svn co http://<ip addy of svn server>/path/to/project" from a SVN server local to the same network as the guest is bridged.
Wireshark can be utilized to capture this in action via the VBox nictrace (VBoxManage modifyvm <vmname> --nictrace1 on --nictracefile1 /path/to/tracefile). Additionally a simple "netstat -s ethX" where "ethX" is the interface to which the guest is bridged to will show some of the symptoms as TCP statistics as well.
The workaround for this at the moment is to disable GRO on the host. This can be done via ethtool by issuing the following: "ethtool –K gro off"
I opened a bug with the Intel igb driver on this (see here) however it appears that this may be a bug in the VirtualBox bridging implementation and the handling of LRO/GRO. Below are some links to changes made to the upstream bridging code in linux for LRO/GRO. Essentially the bridge disables LRO/GRO on any device that hands it a coalesced packet.
Forum thread: http://forums.virtualbox.org/viewtopic.php?f=7&t=32171