network layer hangs (TX stops) when it's slammed w/ a stress test

[Ken Yee]

I'm running on an XP host with the max socket# set for 32K instead of the default 5000. I configured a VirtualBox VM running Win2003 and make sure to turn off SynAttackProtect so Windows 2003's network layer isn't shutting off.

The MSSQL server runs on XP. The Win2003 VM runs IIS and my web app but it uses the MSSQL server on the XP VirtualBox host. The VM is set to use the Host Interface for networking so it has an actual IP address on the network.

The OpenSTA stress tester runs on the XP machine. I set up 1000 virtual users to run the captured stream back to the web app and it works ok up to about 500 users. After that, the response time first spikes hard up to 10sec, then the network layer falls over (can't ping from XP machine to Win2003 VM or vice versa)...restarting the VM fixes this network layer hang.

If I bring up the Session Information for the VirtualBox VM, I can see the Receive Byte count go up, but Transmit doesn't change.

FWIW, VMWare scales up to 800ish users before OpenSTA craps out and shuts down w/o saving any results (VMWare's networking stays up). VirtualBox scales up faster, before the networking layer hangs :-)

I've checked for tickets with "network hang" w/o luck, so hopefully this isn't a duplicate. This is the first issue I've found w/ the 2.1 release.

[anonymous]

id say its a duplicate/add to #2474.

looks like this is happening on linux, windows and solaris hosts with different guests :(

[misha]

Replying to kenyee:

Try updating to Virtualbox 2.1.4 and let me know if the problem is still there. Thanks.

[Ken Yee]

I read bug 2474 and don't believe it's the same bug. To fix it, I have to restart the *guest*, not the *host* as described in 2474. I also have a static IP address set up for host and guest.

I tried 2.1.4 and the problem still happens. Same symptom. Slam it hard w/ a stress test and in the Session Information Dialog for the guest window, the Data Transmitted for the network is stuck but it keeps receiving data, even after I kill off my OpenSTA stress generator.

[anonymous]

you are correct, i read over that important detail. :( all my comments in #2474 should have been posted here instead.

short summary of what i have noticed so far: ubuntu 8.10 64b host, win2k guest. the host interface has an ip alias where a samba deamon is listening. the guest has one of that samba shares mounted. i can always break the connectivity of the guest in a matter of seconds by using flashfxp (a win32 ftp program). when i save the downloaded file directly onto the samba share, it breaks almost always after less than 100MB (even, if the source is inside the vm (a local ftpd)). the receive counter is increasing as kenyee noted. packets from the host reach the guest. the guest applications seem to reply, but the packets never go through. i observed this with wireshark inside the guest.

[Ken Yee]

FWIW, I had a shared folder on the Win2003 guest, so I tried copying about 2GB of files to it at the same time. No problems and I could still ping it afterwards, so the network interface was fine. I was hoping this would make reproducing the bug easier than slamming it with a web stress tool.

[steffan]

Not sure whether it's related, but with VirtualBox 2.1.4, trying to install Red Hat from an http server seems to result in a network hang. Basically, the install gets about half-way through, then stops while downloading an RPM (it hangs at a different RPM every time).

Happens with both NAT and host interface networking. VMWare Server running on the same machine doesn't have this issue (nor does a physical server on the same network).

Host is Windows XP.

[Ken Yee]

steffan: look at the Session Information Dialog for the guest window as mentioned in the bug and if the window is receiving network data but not transmitting, it's probably the same bug. If you see network tx/rx both increasing, you have a different bug...

[steffan]

Thanks kenyee.

Looks like I've got a different issue - nothing changes in session information once it hangs.

Steffan

[Jason Smith]

I run Apache in an Ubuntu Server 8.04.1 guest on a Mac OS X host. Browsing my website typically results in about 30% broken images or missing stylesheets due to connection problems between host and guest. I use NAT to connect host and guest.

The network stack never hangs completely, but continuously keeps rejecting connections randomly.

[Ken Yee]

FYI, this still happens w/ virtualbox 2.2.2. Same symptom. The Session Info dialog for the virtualbox window says it transmitted 74MB of data on the network before it's stuck...but receive still continues from 10MB and keeps increasing.

[misha]

Replying to kenyee: Could you try a different type of Virtual NIC (that could be changed via VBox Network settings GUI) and see if it solves the problem in any way?

[Frank Mehnert]

kenyee, any news?

[Ken Yee]

on vacation this week...will try to reproduce this by switching to a different virtual nic next week.

[Ken Yee]

Sorry for the late reply. I tried (this is what it says in the session info dialog after I select it in the settings GUI): "PCnet-FAST III (bridged adapter, NVidia nForce 10/100/100 Mbps Ethernet #2)". This locked up (stopped transmit but kept receiving) at 10.8MB network data transferred. Same thing happened when choosing the Intel MT server network adapter. Just noticed that Virtualbox 3.0 was released. I'll download that and give it another try...

[Ken Yee]

FYI, the PCnet-FAST III adapter locks up in VirtualBox 3.0 as well.

However, the Intel Pro/1000 MT adapter (last one on the adapter list) does *not*, so it appears to be a bug w/ the PCnet adapter code in VirtualBox. If any admins can do it, please change the title of this bug to reflect that the adapter is what is buggy and not the VirtualBox network layer...

[Frank Mehnert]

We just fixed a potential hang in rare cases for the PCNet device emulation. Please recheck the next maintenance release (3.0.8) and reopen if the problem persists.

[Ken Yee]

Hate to tell you, but it's still broken. I didn't check until now, but installed the latest 3.0.12 version of VirtualBox and the PCnet adapter still hangs (the transmitted byte count gets stuck and the receive bytes keeps increasing).

Info from the session dialog box:

Network Adapter 1:

PCnet-FAST III (Bridged adapter, NVIDIA nForce 10/100/1000 Mbps Ethernet #2)

Adapter 1

Data Transmitted 45,086,341 B

Data Received 9,962,961 B

[misha]

Replying to kenyee:

The "latest" is now VBox 3.1, could you check if it's still an issue there?

[Ken Yee]

Wow, you guys are quick. Can't wait to play w/ the teleportation feature :-)

And yes, I'm afraid it's still broken w/ 3.1. It does seem to go a little further before locking up the TX:

Network Statistics Adapter 1

Data Transmitted

56,842,291 B

Data Received

8,765,269 B

but symptoms are exactly the same (can't ping it, RX increases, TX is stuck, and w/ the PC-Net III adapter)...

[Frank Mehnert]

Please reopen if still relevant with VBox 4.0.6.