NAT-related crash of ubuntu guest on OSX host.

[c_t]

When trying to install node.js npm dependencies inside an ubuntu 12.04 guest system, which causes a lot of simultaneous HTTP connections, virtualbox crashes reproducably with the following crash report:

Process:         VirtualBoxVM [52317]
Path:            /Applications/VirtualBox.app/Contents/MacOS/VirtualBoxVM
Identifier:      VirtualBoxVM
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  VBoxSVC [29251]

Date/Time:       2013-03-27 11:32:37.325 +0100
OS Version:      Mac OS X 10.7.5 (11G63)
Report Version:  9
Sleep/Wake UUID: 6554FC2E-A188-4914-A585-61EF6051F79C

Crashed Thread:  20  NAT

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000010


Thread 20 Crashed:: NAT
0   VBoxDD.dylib                  	0x000000010db6ed59 VBoxDevicesRegister + 838601
1   VBoxDD.dylib                  	0x000000010db65b85 VBoxDevicesRegister + 801269
2   VBoxDD.dylib                  	0x000000010db5d5bb VBoxDevicesRegister + 767019
3   VBoxVMM.dylib                 	0x000000010280f96a PDMR3ThreadCreate + 970
4   VBoxRT.dylib                  	0x00000001002409df RTThreadCreateF + 271
5   VBoxRT.dylib                  	0x00000001002930fc RTThreadPoke + 540
6   libsystem_c.dylib             	0x00007fff8e6cb8bf _pthread_start + 335
7   libsystem_c.dylib             	0x00007fff8e6ceb75 thread_start + 13

Full crash log attached.

I experience this on Mac OS X 10.7.5 and Mac OS X 10.8.3 with VirtualBox 4.1.23 and 4.2.10.

We are currently seeing this issue on 3 different host machines and it "just appeared" without any updates to virtual box or the guest OS.

Maybe it was introduced with a Mac OS X update?

[vasily Levchenko]

If it easy to reproduce in you environment could you please collect coredump and upload it on ​ftp://ftp.oracle.com/appsdev/incoming, together with attaching log file from crushing session.

[c_t]

I've uploaded a tarball called vbox-crash-ticket-11649.tar.gz into ​ftp://ftp.oracle.com/appsdev/incoming including the core dump, the crash report and some corresponding lines from /var/log/system.log

Thank you for your help.

[vasily Levchenko]

Thanks for uploading the core file, but could you please attache log file from your VBox session (please look at ​http://www.virtualbox.org/manual/ch12.html#idp16864224 for more details)

[c_t]

Oh, sorry for the misunderstanding. I've uploaded another tarball which now includes the corresponding VBox.log to make sure all files are consistent.

Thanks again!

[vasily Levchenko]

Could you please describe what your guest is doing? I've been able to get stack trace from your core file, but It doesn't point to obvious problematic place. It looks like crush related to dnsproxy code which is enabled in your scenario, and I've tried several DNS related benchmarks to reproduce the issue you're experiencing, but wasn't succeed.

[c_t]

On the guest system (Ubuntu 12.04) im performing an

npm install

which installs about 132 npm packages for node.js (​http://nodejs.org/). These packages are hosted on HTTP locations, so the process is resolving a lot of DNS names and downloading many files in parallel. One work-around we have found, is manually installing the 132 packages in series and not in parallel, which does not cause crashes (so only few concurrent DNS lookups and HTTP requests). However doing so in parallel, results reproduceably in the described crash.

Interestingly, I now did the same at home (so logged into a different network with the host machine) and the problem does not occur! Nothing else has changed with my machine except for the network connection of the host machine!

Maybe that's helping? I'll be asking my colleagues, who are experiencing the same issues at the office, whether they can try to reproduce the problems at home.

Do you have any suspicion on what in the office network could cause these weird crashes? Could it be something IPV6 related? I see the message

00:00:16.757430 NAT: IPv6 not supported

is one of the last messages in the log... I will try to check with our ISP at the office whether they changed anything concerning IPv6 or DNS lately.

At least something like this would explain why the crashes suddenly started to occur on many different computers at the same time without any changes in the software configuration of these machines.

[vasily Levchenko]

Replying to c_t:

Could you please try the ​build? I've changed clean up on processing timeouts in dnsproxy code, which could potentially cause the issue.

On the guest system (Ubuntu 12.04) im performing an

npm install

Interestingly, I now did the same at home (so logged into a different network with the host machine) and the problem does not occur! Nothing else has changed with my machine except for the network connection of the host machine!

Hmm, interesting observation (what is in delta of your environments? (number of DNS servers?))

Maybe that's helping? I'll be asking my colleagues, who are experiencing the same issues at the office, whether they can try to reproduce the problems at home.

Do you have any suspicion on what in the office network could cause these weird crashes?

there should be some difference in your DNS settings or DNS servers behaviour that provoke the bug in NAT/dnsproxy.

Could it be something IPV6 related? I see the message

No it isn't relates to IPv6.

00:00:16.757430 NAT: IPv6 not supported

is one of the last messages in the log... I will try to check with our ISP at the office whether they changed anything concerning IPv6 or DNS lately.

At least something like this would explain why the crashes suddenly started to occur on many different computers at the same time without any changes in the software configuration of these machines.

[c_t]

Thanks for your investigations. I'll try the build out on tuesday when I'll be back in the office and I'll also have a deeper look at the differences in DNS config.

[c_t]

Just tried out the new build, but the problem persists.

I've uploaded a new core dump as vbox-crash-ticket-11649-3.tar.gz

I will check whether I can tune DNS settings or whether configuring an external DNS server on my computer will help (thus circumventing the router's DNS cache). Is there anything else I can do?

Thanks again!

[vasily Levchenko]

Replying to c_t:

Just tried out the new build, but the problem persists.

I've uploaded a new core dump as vbox-crash-ticket-11649-3.tar.gz

thank you for feed back I will look at the core file and will back to you after analisys.

[c_t]

Replying to Hachiman:

Replying to c_t:

Just tried out the new build, but the problem persists.

I've uploaded a new core dump as vbox-crash-ticket-11649-3.tar.gz

thank you for feed back I will look at the core file and will back to you after analisys.

Did you already manage to have a look at the core dump? Is there any other information I could provide or do you have any other idea, what I could check?

[vasily Levchenko]

Yes I've looked at the coredump, I've done mass changes but still not sure whether I've missed anything, and still I will appreciate to see the some kind of diff your home and office resolv.conf settings, are there some dead in the list, difference in timeout settings?

[c_t]

OK, we now observed the following:

1. In the case of no special network configuration on the host machine (Mac OS X) the DNS-Server is set via DHCP and the /etc/resolv.conf looks like this:

#
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
domain fritz.box
nameserver 192.168.20.1

The nameserver IP is pointing to the local router which is also the default gateway (192.168.20.1). It is this device: ​https://www.avm.de/en/Produkte/FRITZBox/FRITZ_6360_Cable/index.php

2. We now went to the Mac OS X network configuration on the host system and changed the DNS server's IP to 8.8.8.8 (i.e. a google nameserver). After a reboot of the host machine and a restart of the virtual box the problem is gone. /etc/resolv.conf now looks like this:

#
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
domain fritz.box
nameserver 8.8.8.8

Setting the nameserver IP to the google nameserver solved the problem on both, you 4.2.11 build as well as on virtual box 4.2.6.

So as you already pointed out it's definitely the nameserver! Unfortunately I have no idea how to find out how the nameserver on the cable-router is configured since you can only access it through a web frontend that doesn't really reveal any nameserver settings...

[vasily Levchenko]

Can you check how differs query time (e.g. ;; Query time: 66 msec in dig report) for google dns server and in your router settings e.g. with dig

# dig www.de @8.8.8.8

btw: I believe that your router has got just own DNSPROXY forwarding requests to your internet provider's nameservers configured while setup. So perhaps some statistics of resolving times against them will give me a hint for emulation environment closer to your settings.

[c_t]

Here are some results (I stripped all the irrelevant stuff from the dig output):

$ dig www.de @8.8.8.8

;; Query time: 27 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)

$ dig www.de @192.168.20.1

;; Query time: 148 msec
;; SERVER: 192.168.20.1#53(192.168.20.1)

$ dig www.de @192.168.20.1

;; Query time: 9 msec
;; SERVER: 192.168.20.1#53(192.168.20.1)

$ dig @83.169.184.33 www.de

;; Query time: 152 msec
;; SERVER: 83.169.184.33#53(83.169.184.33)

$ dig @8.8.8.8 www.de

;; Query time: 25 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)

So all in all the google nameserver is very fast ~25ms, the ISP's nameserver, which is used by the cable modem/router (83.169.184.33), seems to be a lot slower, ~152 ms as you can see. From the router's dns proxy (192.168.20.1) we see the expected: first request is roughly the same response time as the ISP's nameserver, once the result is cached responses are very fast (9ms).

Does that information help?

[vasily Levchenko]

Thank you for update I will emulate corresponding timeouts here.

[vasily Levchenko]

I've checked against your timeouts here and it works fine, let's check in your environment.

Could you please verify the build ​build? Note: It's a trunk build, so it'd better backup yours vm settings, before try.

[c_t]

Unfortunately no luck. With the router's NS it still crashes, with the google NS it works...

[vasily Levchenko]

Replying to c_t:

Unfortunately no luck. With the router's NS it still crashes, with the google NS it works...

Could you please upload the core file for this session?

[betterdevs]

Even though this seems to be dormant: I seem to have the same problem: NAT thread crashes on MacOS, FritzBox as router (different model, though), after I read this I put the Google DNS which seems to work. Interestingly enough, it works with a different Mac and FritzBox at home (different provider, though).

VirtualBox version is 4.2.16, Host OS is Mac OS 10.8, Guest OS is CentOS 6.4. The problem happens when trying to install the first package from the repositories (which ususally tries to contact all mirrors to find the fastest one).

I can try to post crash logs if that is still interesting.

[machete143]

I've got the same issue on a Windows 7 host with the most recent Virtual Box (4.3.6) and vagrant!

When the networking type is 'NAT', 'npm install bower' crashes the guest vm. When I switch to 'Network Bridge', the crash doesn't happen.

Please also see: https://www.virtualbox.org/ticket/12588

[machete143]

Interesting though, because the same VM works fine on Mac OS...

[vasily Levchenko]

Replying to machete143:

Interesting though, because the same VM works fine on Mac OS...

Could you please collect pcap file from your guest?

[machete143]

Windows 7 Host PCAP File

[machete143]

I've attached the pcap file you requested. The commands I used (after login) were:

sudo -s
npm install bower grunt-cli

The VM crashed very soon after (like .5 - 1 seconds later)

[vasily Levchenko]

Replying to machete143:

I've attached the pcap file you requested. The commands I used (after login) were:

thank you.

[vasily Levchenko]

Could you please also share your VBox.log file? Rather interesting to look at your NAT configuration.

[vasily Levchenko]

Could you please check, whether v11649-test.sh crashes your VM? It's sends DNS requests similar, which are the last at your file. Note, it require xxd (usually exists with vim) and socat are installed in the guest.

[machete143]

Windows 7 Host / Precise 32 Guest Vbox Log

[machete143]

I've attached the VBox log. Unfortunately I can not run the sh file, because it tells me:

line 23: Syntax error: Bad for loop variable.

However, that was solvable by replacing #!/bin/sh with #!/bin/bash

But unfortunately the script didn't crash the vm.

Please see my updated posts below, the vm crashes when executing the script via nodejs

I think this issue could be related to nodejs' non blocking I/O architecture. What happens (at least I think this is what happens) is that npm asks very fast for multiple DNS entries and queueing the answer in a callback. So we get multiple simoultaneously DNS requests although the answer hasn't arrived yet. What your script does is (if I read that correctly) opening sequently 100 dns queries and *waiting* for the reply before opening another. Could this be right?

I'm from another field of development, but we have there something called "Race condition" where one thread is faster than the other and causes a fatal error. This could be related to this issue (again, I'm wild guessing here) where a second DNS query arrives *before* the first DNS query and maybe that causes the VM to crash? That could also be the reason why this error happens randomly (sometimes I can install 200 packages, sometimes the VM crashes right at the 2nd package).

I have no background with nodejs whatsoever, but I'll ask a friend of mine to write a nodejs script that opens many simoultaneously connections to one server and maybe that script will crash the VM as well.

Please also note that I have a very fast internet link (100mbit) with a very low ping (usually < 10ms) and a direct connection (lan) to the router, so that might favor this issue. My MacBook is connected through WLAN, maybe that's one of the reasons why this doesn't happen there.

[machete143]

VM not crashing

[machete143]

I've now also added a logfile with a VM I've set up manually (not through vagrant). This VM *does not crash* when executing 'npm install'.

However, when creating a vagrant box out of it and running 'vagrant up', the crash occurs again! So my wild guess is that vagrant is setting some weird option, which causes the system to crash!

Maybe a diff on those two can help you identifying the issue.

I've added two logfiles:

• VM not crashing (above this post, Athene2-vm)
• Vagrant VM based on the non crashing VM (below this post, vagrant_default)

The first logfile is the one, I've setup manually - this one doesn't crash with 'npm install'

The second logfile is a VM setup by vagrant. It uses the image of the 'VM not crashing' VM. This one crashes when executing 'npm install'

[machete143]

Vagrant VM based on the non crashing VM

[machete143]

I've now attached two nodejs scripts. One ('request') executes multiple requests. The other, executes your script. BOTH crash the VM!

However, they only crash 'vagrant_default', not 'Athene2-vm'!

So this is just as I expected, the non blocking I/O architecture of nodejs is causing the trouble

[machete143]

I've made multiple edits to my previous posts so if you're only monitoring the mailing list, please check those!

tracing bugs really is fun :D

[vasily Levchenko]

Thank you for investigation, right DNS calls are done extremely intensive in short period. Could you please attach pcap files for Athene2-vm and vagran_default? there is other thing that I've noticed in file.pcap, the order of responses, that could potentially be a reason. And one thing actually Athene2-vm isn't exactly clone of vagrant_default : the first one uses list of DNS servers exported by VirtualBox to guest, while valgrant_default uses dnsproxy. I believe that once you change it Athene2-vm will also crash.

[machete143]

nodejs scripts

[machete143]

I will provide you with a pcap once I'm back home. Can you provide me with a command to switch the dns behaviour (for both VMs)? If that's related to the dnsproxy setting, is there a possibility to fix this? Or should I contact the vagrant developers and tell them that the other setting should be used instead?

[machete143]

Using

config.vm.provider :virtualbox do |vb|
  vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']
end

the VM doesn't crash any more! Do you still need the file.pcap of Athene2-vm?

[vasily Levchenko]

'--natdnshostresolver1' is alternative '--natdnsproxy1', difference is that NAT ask host to resolve name via C API, rather forward UDP packet to registered servers, to switch them off possible via VBoxManage modifyvm --natdns{hostresolver,proxy}1 off, actually if they're both on --dnshostresolver1 will win.

[vasily Levchenko]

Replying to machete143:

the VM doesn't crash any more! Do you still need the file.pcap of Athene2-vm?

yes please, but perhaps you just can switch dnsproxy on and off (but please dont forget to switch hostresolver off first) because if all requests are identical, perhaps because of crash real problematic request/response is missed in serialized pcap.

[vasily Levchenko]

Replying to machete143:

I will provide you with a pcap once I'm back home. Can you provide me with a command to switch the dns behaviour (for both VMs)? If that's related to the dnsproxy setting, is there a possibility to fix this? Or should I contact the vagrant developers and tell them that the other setting should be used instead?

--natdnsproxy1 is more preferable, it was in shadow of hostresolver too long, so it better to find root cause and fix it correctly :).

[machete143]

Unfortunately the pcap file was too big for me to upload, so I put it on dropbox: ​https://www.dropbox.com/s/qozdlzfu6lzvvnz/athene.pcap

Hope this helps :)

[machete143]

Replying to Hachiman:

'--natdnshostresolver1' is alternative '--natdnsproxy1', difference is that NAT ask host to resolve name via C API, rather forward UDP packet to registered servers, to switch them off possible via VBoxManage modifyvm --natdns{hostresolver,proxy}1 off, actually if they're both on --dnshostresolver1 will win.

So if I understood correctly, I should do:

vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']
vb.customize ['modifyvm', :id, '--natdnsproxy1', 'off']

?

[vasily Levchenko]

Replying to machete143:

Replying to Hachiman:

'--natdnshostresolver1' is alternative '--natdnsproxy1', difference is that NAT ask host to resolve name via C API, rather forward UDP packet to registered servers, to switch them off possible via VBoxManage modifyvm --natdns{hostresolver,proxy}1 off, actually if they're both on --dnshostresolver1 will win.

So if I understood correctly, I should do:

vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']
vb.customize ['modifyvm', :id, '--natdnsproxy1', 'off']

?

No both should be off.

[machete143]

With those settings, vagrant doesn't work :) If any one else has this problem, this setting worked for me:

vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']

[Takis]

Any progress on this? I'm most likely having the same issue: Using bower crashes my CentOS 6.5 virtual machine (setup using Vagrant).

[Takis]

I can confirm that the workaround provided above works for me:

config.vm.provider :virtualbox do |vb|
  vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']
end

[Valery Ushakov]

This might be the same underlying use-after-free problem as the one described in #13994. It was fixed after 4.3.26 and will be available in the next 4.3 release. Meanwhile you may give a recent testbuild a try.

[Valery Ushakov]

The fix is part of 4.3.28. Please, give it a try.

[fotoflo]

forgive me if this seems like a silly question, but where do we put this workaround?

[Valery Ushakov]

Replying to fotoflo:

forgive me if this seems like a silly question, but where do we put this workaround?

Why do you need the workaround? Do you still experience this problem with 4.3.28 or later?

[Frank Mehnert]

Please reopen if necessary. The above comments clearly say that the fix is part of VBox 4.3.28 or later (VBox 5.0.0 includes the fix as well).