NAT: crash with "WSAEnumNetworkEvents tcp error 10038" -> fixed in svn

[Pinepara]

I have a Arch Linux installed in VirtualBox, and the VirutalBox crashes every time when I tried to install some new packages or perform a system upgrade in it. I believe that it must have something to do with the heavy disk I/O.

I never came to this situation until about two weeks ago, so I believe that this is a latest introduced bug of VirtualBox.

My host system is: Windows 7 SP1 x64; My guest system is: Arch Linux x64.

Upgrading to the latest version of Virtualbox does not solve my problem.

[Pinepara]

Sorry for the arbitrary judgement.

I found the following line in the log file:

00:06:21.850 WSAEnumNetworkEvents tcp error 10038

Seems like a network error, not a disk one.

[Hachiman]

Thanks for reporting the issue. Have you noticed which networking activity provokes this crush for you?

[Pinepara]

Everytime I run 'yaourt -Syu' or 'yaourt -S blabla' in gnome-terminal, virtualbox crashes in less than five minutes.

Since downloading packages can be continued after reboot, I can perform a full system upgrade within a few times of reboot. But it is very annoying.

Furthermore, I tried to use bridged network instead of NAT after I reported this issue, by which I managed to perform a 30-minutes' system upgrade without crash.

As soon as I switched to NAT again and tried to run 'yaourt -S texlive-most', it crashed in three minutes.

So I think maybe NAT causes this. No idea how.

Replying to Hachiman:

Thanks for reporting the issue. Have you noticed which networking activity provokes this crush for you?

[Hachiman]

Replying to Pinepara: Thanks for update.

[Hachiman]

Could you please try the build  VirtualBox-4.1.15-77799-Win.exe whether it fix the issue for you?

[Pinepara]

Oh, it crashed again during a 'yaourt -Syu'.

I will attach the log, but it looks like nothing different from the last one to me.

Replying to Hachiman:

Could you please try the build  VirtualBox-4.1.15-77799-Win.exe whether it fix the issue for you?

[Pinepara]

Virtualbox version 4.1.15 r77799

[ArchEnthusiast]

VirtualBox log with the crash

[ArchEnthusiast]

Greetings! I'm also experiencing a similar issue with my Arch Linux disto. The log reports "WSAEnumNetworkEvents tcp error 10038" when I run the command:

wget  ftp://ftp.adobe.com/pub/adobe/reader/unix/9.x/9.5.1/enu/AdbeRdr9.5.1-1_i486linux_enu.tar.bz2

under ~/AURBuilds as a non-root user.

It's a 4-5 day old installation of Arch on a dynamic 200gb .vmdk, which I just created by cloning a static 8gb .vmdk as by  this tutorial. I ran gparted after the cloning to resize by /dev/sda3 and /dev/sda4 to be about 30gb and 10gb respectively. I was mucking around with the .xml config files for both .VirtualBox and my specific Arch VM previously, so I assumed this crash was caused by something I completely screwed up in the process, but it only seems to happen during intensive network operations (like wget).

This bug is kind of a show-stopper.

Feel free to contact me if you need more information.

[Pinepara]

I also used gparted to resize partions just a few days ago. Besides that I did nothing you mentioned here.

Is there something to do with gparted? That sounds crazy...

Replying to ArchEnthusiast:

Greetings! I'm also experiencing a similar issue with my Arch Linux disto. The log reports "WSAEnumNetworkEvents tcp error 10038" when I run the command:

wget  ftp://ftp.adobe.com/pub/adobe/reader/unix/9.x/9.5.1/enu/AdbeRdr9.5.1-1_i486linux_enu.tar.bz2

under ~/AURBuilds as a non-root user.

It's a 4-5 day old installation of Arch on a dynamic 200gb .vmdk, which I just created by cloning a static 8gb .vmdk as by  this tutorial. I ran gparted after the cloning to resize by /dev/sda3 and /dev/sda4 to be about 30gb and 10gb respectively. I was mucking around with the .xml config files for both .VirtualBox and my specific Arch VM previously, so I assumed this crash was caused by something I completely screwed up in the process, but it only seems to happen during intensive network operations (like wget).

This bug is kind of a show-stopper.

Feel free to contact me if you need more information.

[Pinepara]

Are you using NAT? Have you tried bridged network?

My issues won't happen while network set to bridged.

Replying to ArchEnthusiast:

Greetings! I'm also experiencing a similar issue with my Arch Linux disto. The log reports "WSAEnumNetworkEvents tcp error 10038" when I run the command:

wget  ftp://ftp.adobe.com/pub/adobe/reader/unix/9.x/9.5.1/enu/AdbeRdr9.5.1-1_i486linux_enu.tar.bz2

under ~/AURBuilds as a non-root user.

It's a 4-5 day old installation of Arch on a dynamic 200gb .vmdk, which I just created by cloning a static 8gb .vmdk as by  this tutorial. I ran gparted after the cloning to resize by /dev/sda3 and /dev/sda4 to be about 30gb and 10gb respectively. I was mucking around with the .xml config files for both .VirtualBox and my specific Arch VM previously, so I assumed this crash was caused by something I completely screwed up in the process, but it only seems to happen during intensive network operations (like wget).

This bug is kind of a show-stopper.

Feel free to contact me if you need more information.

[Hansel]

I'm getting this issue, but with an install of Ubuntu. I'm trying to compile OpenELEC XBMC, but the VM session crashes with "VirtualBox Stopped Responding". It always does it whilst trying to download a package.

I also have "WSAEnumNetworkEvents tcp error 10038" in my log (which is how I found my way here).

This was on the latest release, and also the release prior. I'm now on 4.0.16 to see if that helps :(

[ArchEnthusiast]

Replying to Pinepara:

Are you using NAT? Have you tried bridged network?

My issues won't happen while network set to bridged.

I've been able to complete my wget after switching to bridged. It was in NAT mode before.

[Hansel]

Just to confirm, it's working fine with the 4.0.16 version, so I guess it's something that was introduced in later versions.

[Hachiman]

Could you please download the build of  VBoxDD.dll with enabled logging? After downloading please, replace original VBoxDD.dll from VirtualBox 4.1.15 r77799 with downloaded DLL (please, safe original DLL in safe place). Then please launch your VM in the following way

# set VBOX_LOG_DEST=file=nat.log
# set VBOX_LOG=drv_nat.e.f.l2
# VirtualBox -startvm <your-vm>

Result log please attach to the defect. Note: VBoxDD.dll compiled against revision pointed (4.1.15 r77799) earlier to the defect.

[Jacek_]

Seems that I have the same problem with "WSAEnumNetworkEvents tcp error 10038"

My setup is:

Host: Win7
VBox: 4.1.12 r77245
Guest: xubuntu
Network: NAT

I'm pretty sure that in my case the VM Abort appears after sometime of using FileZilla to access some FTP server. Unfortunately, I can't use the bridged mode and need to access the FTP server, which make this issue a real pain.

[Hachiman]

Replying to Jacek_: Would you mind to try instructions I've provided in the comment 15 and attach result log to the defect?

[Pinepara]

I did as you said. It crashed again, as it always does.

Since the log files exceed the file size limit here, I uploaded them to:

 http://file.pinepara.info/nat.log  http://file.pinepara.info/VBox.log

Replying to Hachiman:

Could you please download the build of  VBoxDD.dll with enabled logging? After downloading please, replace original VBoxDD.dll from VirtualBox 4.1.15 r77799 with downloaded DLL (please, safe original DLL in safe place). Then please launch your VM in the following way

# set VBOX_LOG_DEST=file=nat.log
# set VBOX_LOG=drv_nat.e.f.l2
# VirtualBox -startvm <your-vm>

Result log please attach to the defect. Note: VBoxDD.dll compiled against revision pointed (4.1.15 r77799) earlier to the defect.

[Hachiman]

Replying to Pinepara: Thank you. I will take a look.

[Hachiman]

Still log doesn't clarify what wrong is going on so could you please to apply instructions from comment 15 and use DDL with improved logging  VBoxDD.dll. Additionally could you please, dump network traffic using following instructions.

[Pinepara]

Replying to Hachiman:

I have done so many times of system update recently that I have nothing to upgrade now...

So I try to do some downloading using wget but haven't triggered any bug yet.

Very weird, I will try again later.

[gendouhydeist]

Replying to Hachiman:

Ok I've tried what you instructed and replace VBoxDD.dll and got an error Unable to load R3 module VBoxDD.dll GetLastError=193 (VER_BAD_EXE_FORMAT)

Result Code: E_FAIL (0x80004005) Component: Console Interface: IConsole {1968b7d3-e3bf-4ceb-99e0-cb7c913317bb}

Still log doesn't clarify what wrong is going on so could you please to apply instructions from comment 15 and use DDL with improved logging  VBoxDD.dll. Additionally could you please, dump network traffic using following instructions.

[Hachiman]

Replying to gendouhydeist:

Replying to Hachiman:

Ok I've tried what you instructed and replace VBoxDD.dll and got an error Unable to load R3 module VBoxDD.dll GetLastError=193 (VER_BAD_EXE_FORMAT)

I'll build/upload 32-bit VBoxDD, today, for you. Btw, have you installed 4.1.15 r77799, mentioned in comment 18?

[gendouhydeist]

Replying to Hachiman:

yes I did.

Replying to gendouhydeist:

Replying to Hachiman:

Ok I've tried what you instructed and replace VBoxDD.dll and got an error Unable to load R3 module VBoxDD.dll GetLastError=193 (VER_BAD_EXE_FORMAT)

I'll build/upload 32-bit VBoxDD, today, for you. Btw, have you installed 4.1.15 r77799, mentioned in comment 18?

[Hachiman]

Replying to gendouhydeist:

Replying to Hachiman:

Could you please try  VBoxDD.dll, it's 32-bit library.

[gendouhydeist]

Replying to Hachiman:

It's bigger than I though it would be had to compress it.

 http://www.mediafire.com/?ft3niamk1inx35s

Replying to gendouhydeist:

Replying to Hachiman:

Could you please try  VBoxDD.dll, it's 32-bit library.

[Hachiman]

Replying to gendouhydeist: Thank you for cooperation, I'll take a look.

[Pinepara]

Replying to Hachiman:

I'm back with the crash report here:

 http://file.pinepara.info/crashlog-pinepara-0515.zip

[Hachiman]

Replying to Pinepara: Thanks for report.

[Hachiman]

Could you please verify following VBoxDD.dll builds: 64-bit  VBoxDD.dll, 32-bit  VBoxDD.dll. Both binaries built again 4.1.15 r77799 so please follow instructions in comment 15 for installation.

[Pinepara]

Replying to Hachiman:

Comment 18 is mine.

Should I reproduce the crash again using this dll?

[Hachiman]

Replying to Pinepara:

Replying to Hachiman:

Thanks, for correction. indeed comment 15, and yes please try to reproduce the issue.

[gendouhydeist]

Replying to Hachiman:

 http://www.mediafire.com/?iwtybby8uadxlsp

Do you also need the file.pcap? If so then I'll reproduce this later tommorow.

Could you please verify following VBoxDD.dll builds: 64-bit  VBoxDD.dll, 32-bit  VBoxDD.dll. Both binaries built again 4.1.15 r77799 so please follow instructions in comment 15 for installation.

[Hachiman]

Replying to gendouhydeist:

Do you also need the file.pcap? If so then I'll reproduce this later tommorow.

Thanks for logs, no pcaps files needed.

[Pinepara]

Replying to Hachiman:

I packed 3 files: file.pcap, nat.log, VBox.log in:

 http://file.pinepara.info/crashlog-pinepara-0517.zip

[Jacek_]

Replying to Hachiman:

Replying to Jacek_: Would you mind to try instructions I've provided in the comment 15 and attach result log to the defect?

Hi,

Sorry for not answering for a while. After the upgrade to 4.1.15 r77799 and changing VBoxDD.dll it all works fine for me. I've not experienced any other crashes since then.

[Hachiman]

Replying to Jacek_:

Replying to Hachiman:

Replying to Jacek_:

Thanks for feedback. But now I'm confused here, I don't see any crushes here either. But looks like Pinepera and gendouhydeist still experiences some crushes.

[Pinepara]

Replying to Hachiman:

Yes, I'm still experiencing crushes even after I upgraded to 4.1.6

[Hachiman]

Replying to Pinepara:

Replying to Hachiman:

Yes, I'm still experiencing crushes even after I upgraded to 4.1.6

That intended, fixes above wasn't back ported to 4.1, all binaries above were a custom build.

[gendouhydeist]

Replying to Hachiman:

Yep, still the same here.

Replying to Jacek_:

Replying to Hachiman:

Replying to Jacek_:

Thanks for feedback. But now I'm confused here, I don't see any crushes here either. But looks like Pinepera and gendouhydeist still experiences some crushes.

[Hachiman]

Replying to Hachiman:

Replying to Pinepara:

Could you please try this build  http://www.virtualbox.org/download/testcase/VirtualBox-4.1.17-78168-Win.exe?

no additional actions required, so far.

[Pinepara]

Replying to Hachiman:

Crashed again, during a 'yaourt -Syu'

[Hachiman]

Replying to Pinepara: Ok, perhaps we need to try from other side. Could you please generate the mini dump for this crash please look at Mini Dump for more details?

[gendouhydeist]

Replying to Hachiman:

Here's what I did on task manager>VirtualBox.exe>right click menu>create dump file.

 http://www.mediafire.com/?9w7s032711040jd

this actually a full dump.

Replying to Pinepara: Ok, perhaps we need to try from other side. Could you please generate the mini dump for this crash please look at Mini Dump for more details?

[Hachiman]

Replying to gendouhydeist:

Replying to Hachiman:

Here's what I did on task manager>VirtualBox.exe>right click menu>create dump file.

thank you, will take a look.

[gendouhydeist]

Replying to Hachiman:

This is the mini dump created through process explorer.

 http://www.mediafire.com/download.php?1925b49rx2hhrt3

Replying to gendouhydeist:

Replying to Hachiman:

Here's what I did on task manager>VirtualBox.exe>right click menu>create dump file.

thank you, will take a look.

[whitemp]

I'm seeing something similar but with a different OS Host is Windows 7 x86_64 - my guest OS is PC-BSD 9.0

If I use NAT, the VM will crash if I am using AppCafe (or pbi_installer, which is about the same thing.) The same error, WSAEnumNetworkEvents tcp error 10038, is in the log.

I switched to bridge, and it's been up for about 24hrs - I can get it crash the VM in about 8 minutes otherwise.

[Hachiman]

Replying to whitemp:

I'm seeing something similar but with a different OS Host is Windows 7 x86_64 - my guest OS is PC-BSD 9.0

Thanks for report I'll try to reproduce this scenario in local environment.

[Hachiman]

Could you please upgrade VBox up to 4.1.18 and replaces VBoxDD.dll in the same manner we've done before with binaries below?

 VBoxDD.dll - 32-bit VBoxDD.dll
 VBoxDD.dll - 64-bit VBoxDD.dll

[gendouhydeist]

Replying to Hachiman:

I would definitely say that it has been fixed, but it didn't have nat.log. Using your VBoxDD.dll posted below. Though I haven't tried the original VBoxDD.dll on the 4.1.18 installer.

Edit: Still crashing on 4.1.18 VBoxDD.dll.

Could you please upgrade VBox up to 4.1.18 and replaces VBoxDD.dll in the same manner we've done before with binaries below?

 VBoxDD.dll - 32-bit VBoxDD.dll
 VBoxDD.dll - 64-bit VBoxDD.dll

[Hachiman]

Replying to gendouhydeist: Thank you for testing.

[Pinepara]

Replying to Hachiman:

Wow! It seems that this fixed my problem too. Good Job!

[Hachiman]

Replying to Pinepara:

Wow! It seems that this fixed my problem too. Good Job!

Thank you for feed back.

[mkucharski]

virtualbox: 4.1.18 r78361
host: Windows 7 SP1 64-bit
guest: OpenBSD 5.2-beta amd64

VBoxManage.exe modifyvm openbsd-amd64 --natdnshostresolver2 on
VBoxManage.exe modifyvm openbsd-amd64 --natsettings2 1500,128,128,128,128
VBoxManage.exe startvm openbsd-amd64 --type headless

Issue is still present, "WSAEnumNetworkEvents tcp error 10038" is last line in the log when VirtualBox is crashing. It always happens during high network load generated on guest with wget/curl/ftp.

[Hachiman]

Replying to mkucharski:

Issue is still present, "WSAEnumNetworkEvents tcp error 10038" is last line in the log when VirtualBox is crashing. It always happens during high network load generated on guest with wget/curl/ftp.

Could you please attach the log? Could you please describe how do you generate huge load on the guest?

[mkucharski]

Replying to Hachiman:

Could you please attach the log? Could you please describe how do you generate huge load on the guest?

Sure, it doesn't always crash. When it crashes again I'll provide the log. FYI: I don't have any debugging DLLs insalled. I'm downloading on that machine OpenBSD snapshots quite often from various http or ftp mirrors. Nothing fancy really.

[mkucharski]

VB 4.1.18 crash, Win7 64-bit host, OpenBSD 5.2-beta amd64 guest, NAT

[mkucharski]

Crashed during download of  ftp://ftp5.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/install52.iso

[Hachiman]

thanks for update.

[frank]

VirtualBox 4.1.20 contains another fix. Could you please test if the bug was finally fixed?

[mkucharski]

Crash still happens with VirtualBox 4.1.20 r80170

[mkucharski]

Attachment didn't bump last modified time, so bumping by this comment - Yeah, it fails again.

[Hachiman]

Replying to mkucharski:

Attachment didn't bump last modified time, so bumping by this comment - Yeah, it fails again.

Could you please collect network dump ? (for more details please look Network_tips)

[Eric]

Crash on 4.1.20 on eavy bandwidth load

[Eric]

If you need me for testing, I get a systematic way to make it crash by downloading a FTP file with by 100Mb connection on my dedicated server. Trying to get now to collect "network dump"...

[Hachiman]

Replying to Eric:

If you need me for testing, I get a systematic way to make it crash by downloading a FTP file with by 100Mb connection on my dedicated server. Trying to get now to collect "network dump"...

Thank you Eric for your help. Could you please send it to me (vasily _dot_ levchenko _at_ oracle _dot_ com) by mail?

[Eric]

Replying to Hachiman:

Replying to Eric:

If you need me for testing, I get a systematic way to make it crash by downloading a FTP file with by 100Mb connection on my dedicated server. Trying to get now to collect "network dump"...

Thank you Eric for your help. Could you please send it to me (vasily _dot_ levchenko _at_ oracle _dot_ com) by mail?

My VB crashes typically after 400MB downloaded, which makes the capture very large. Could you provide me a debug version with logs to try? I'm trying to setup a build environnement to help solving this issue with an instrumented slirp/socket classes to handle the 10038 (not a socket error WSAENOTSOCK). It will be faster if you do it by yourself... Eric

[Hachiman]

Replying to Eric:

Replying to Hachiman:

Replying to Eric:

I'm trying to setup a build environnement to help solving this issue with an instrumented slirp/socket classes to handle the 10038 (not a socket error WSAENOTSOCK). It will be faster if you do it by yourself...

Ok, I will build log enabled bits and share it with you. Btw, Does it change anything for you in case of single CPU guest?

[Hachiman]

Log enabled build VBoxDD.dll is available at  http://www.virtualbox.org/download/testcase/VBoxDD.dll.4.1.20_10513 (please look at comment 15 for more detail how to install and use it). It'd be better to have both logs and pcap file. I suppose that result files will be huge, so you can upload ones to  ftp://ftp.oracle.com/appsdev/incoming, please note this folder is WO so please let me know the file name(s).

[mkucharski]

after replacing VBoxDD.dll with VBoxDD.dll.4.1.20_10513 virtualbox fails to start

[mkucharski]

After replacing VBoxDD.dll with VBoxDD.dll.4.1.20_10513 virtualbox fails to start, see previously attached file for details. Not sure is that related to the fact, that Oracle_VM_VirtualBox_Extension_Pack-4.1.20.vbox-extpack fails to install on my machine with error

Failed to install the Extension Pack C:/Users/mkucharski/.VirtualBox/Oracle_VM_VirtualBox_Extension_Pack-4.1.20.vbox-extpack.

The installer failed with exit code 1: VBoxExtPackHelperApp.exe: error: Failed to rename the extension pack directory: VERR_ALREADY_EXISTS

rcExit=1.

Result Code: E_FAIL (0x80004005)
Component: ExtPackManager
Interface: IExtPackManager {3295e6ce-b051-47b2-9514-2c588bfe7554}

[mkucharski]

Reverting VBoxDD.dll to previous (original) version allows to start VM again, even though I have vbox-extpack 4.1.18 installed.

[mkucharski]

Issue still happening with 4.1.22

[mkucharski]

Crashed during capturing (in different way), 4.1.22

[mkucharski]

Finally had a chance to look into this again. All below tests are done with 4.1.22 (the latest reported by Help->Check for Updates at the time of writing this). Enabled packet capturing on the guest, which look more or less as following command:

VBoxManage.exe modifyvm openbsd-amd64 --nictrace2 on --nictracefile2 openbsd-amd64.pcap

First started download of:

 http://ftp5.eu.openbsd.org/ftp/pub/OpenBSD/5.1/amd64/install51.iso

however file finished successfully. So I looked again at my previous downloads and noticed that on that particular mirror I was using FTP to trigger the crash. So I decided to download again the same file via:

 ftp://ftp5.eu.openbsd.org/pub/OpenBSD/5.1/amd64/install51.iso

Left VirtualBox unattended for a while. Came back and wget transfer was stalled. Disappointed decided to CTRL+C the transfer (which worked fine) and then typed 'halt -p' to power off the guest OS. In a moment my CPU fan kicked off like crazy. Windows' TaskMgr.exe showed that VBoxHeadless.exe is using more CPU than usual. Few seconds later I see crash report from Windows about VBoxHeadless.

So, this crash is different than previously reported. Not sure why. See file attached to this ticket very recently (log.gz) for the log from that session. The pcap file is around 330MB uncompressed, 289MB gzipped. I'm going to upload it to my web server and send you a link via email given in previous comments in this ticket. In Oracle VM VirtualBox Manager guest is reported as Aborted.

Please let me know, if you need anything else from me. Thanks.

[Hachiman]

Replying to mkucharski:

Left VirtualBox unattended for a while. Came back and wget transfer was stalled. Disappointed decided to CTRL+C the transfer (which worked fine)

Do I understand you right, that CTRL+C stopped the wget ?

and then typed 'halt -p' to power off the guest OS. In a moment my CPU fan kicked off like crazy. Windows' TaskMgr.exe showed that VBoxHeadless.exe is using more CPU than usual. Few seconds later I see crash report from Windows about VBoxHeadless.

And guest is really crashed on 'halt -p'?

I'm going to upload it to my web server and send you a link via email given in previous comments in this ticket. In Oracle VM VirtualBox Manager guest is reported as Aborted.

Thanks for a link. It looks like according to log that crashed VBoxSVC rather than guest.

[mkucharski]

Replying to Hachiman:

Do I understand you right, that CTRL+C stopped the wget ?

Yes, CTRL+C stopped wget.

And guest is really crashed on 'halt -p'?

I do not understand the question. I'm not sure did guest crashed. I think VirtualBox running the guest crashed. Not 100% sure. It crashed in usual way for me, however triggered in different way that always. This time not by constant network traffic.

Thanks for a link. It looks like according to log that crashed VBoxSVC rather than guest.

Not sure what VBoxSVC is. Is there anything more which I can test? I think I'll repeat the tests few times again.

[Hachiman]

Could you please generate minidump of your crash?

[mkucharski]

Can you point me do the document which explains what is minidump and how to generate it?

[Hachiman]

Replying to mkucharski:

Can you point me do the document which explains what is minidump and how to generate it?

Please look at our Wiki resource in the part related to Windows system.

[frank]

No response. Please reopen if still relevant with VBox 4.2.18.