VirtualBox

Ticket #4801 (closed defect: fixed)

Opened 7 months ago

Last modified 5 months ago

IPSEC via NAT-Interface does not work properly in Linux Guest -> fixed in svn/3.0.8

Reported by: vondralbra Assigned to:
Priority: major Component: network/NAT
Version: VirtualBox 3.0.4 Keywords: ipsec nat
Cc: Guest type: Linux
Host type: Linux

Description

IPSEC from the guest does not work when the Guest is behind a NAT Interface of VBox 3.0.4.

A tcpdump of the interface of the guest shows that 11 packets are transmitted to the VPN Gateway having the 217.XXX.XXX.XXX address when the connection is initialized. 

listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
09:32:30.119518 arp who-has 10.0.2.2 tell 10.0.2.15
09:32:30.120196 arp reply 10.0.2.2 is-at 52:54:00:12:35:02
09:32:31.714162 IP 10.0.2.15.500 > 217.XXX.XXX.XXX.500: isakmp: phase 1 I ident
09:32:31.767945 IP 217.XXX.XXX.XXX.500 > 10.0.2.15.500: isakmp: phase 1 R ident
09:32:31.799922 IP 10.0.2.15.500 > 217.XXX.XXX.XXX.500: isakmp: phase 1 I ident
09:32:31.905178 IP 217.XXX.XXX.XXX.500 > 10.0.2.15.500: isakmp: phase 1 R ident
09:32:31.935712 IP 10.0.2.15.4500 > 217.XXX.XXX.XXX.4500: NONESP-encap: isakmp: phase 1 I ident[E]
09:32:31.935821 IP 10.0.2.15 > 217.XXX.XXX.XXX: udp
09:32:41.907620 IP 217.XXX.XXX.XXX.500 > 10.0.2.15.500: isakmp: phase 1 R ident
09:32:41.941562 IP 10.0.2.15.4500 > 217.XXX.XXX.XXX.4500: NONESP-encap: isakmp: phase 1 I ident[E]
09:32:41.941620 IP 10.0.2.15 > 217.XXX.XXX.XXX: udp
^C
11 packets captured
11 packets received by filter
0 packets dropped by kernel

On the interface of the host there are only 6 packets to the VPN gateway seen in the same time. 

tcpdump -n -i eth0 host 217.XXX.XXX.XXX
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
09:32:24.257707 IP 217.XXX.XXX.XXX.500 > 192.168.0.22.40258: isakmp: phase 1 ? ident
09:32:30.842839 IP 192.168.0.22.40258 > 217.XXX.XXX.XXX.500: isakmp: phase 1 I ident
09:32:30.895337 IP 217.XXX.XXX.XXX.500 > 192.168.0.22.40258: isakmp: phase 1 R ident
09:32:30.928742 IP 192.168.0.22.40258 > 217.XXX.XXX.XXX.500: isakmp: phase 1 I ident
09:32:31.031873 IP 217.XXX.XXX.XXX.500 > 192.168.0.22.40258: isakmp: phase 1 R ident
09:32:41.029828 IP 217.XXX.XXX.XXX.500 > 192.168.0.22.40258: isakmp: phase 1 R ident
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

As it could be seen that the request to the IPSEC-GW on 4500/udp is not visible on the host interface.

The IPSEC-Connection works properly when the guest interface is running in bridged mode.

The only message regarding NAT is that ipv6 is not supported.

Attachments

FW-Debian-2009-08-22-05-09-21.log (50.1 kB) - added by vondralbra on 2009-08-22 05:13:37.
A saved logfile of the pcap debugging session mentioned below

Change History

2009-08-18 19:04:59 changed by vondralbra

I just saw that I forgot to mention: Host OS is either Debian Lenny or Ubuntu 9.10 (both tested positive regarding this issue) and the message regarding IPv6 is seen in the vBox-log.

(follow-up: ↓ 3 ) 2009-08-21 06:28:49 changed by Hachiman

Could you please attach the log file anyway?

(in reply to: ↑ 2 ) 2009-08-21 07:27:36 changed by Hachiman

Replying to Hachiman:

Could you please attach the log file anyway?

Could you please send me [vasily _dot_ levchenko _at_ Sun _dot_ COM] pcap file?

2009-08-22 05:13:37 changed by vondralbra

  • attachment FW-Debian-2009-08-22-05-09-21.log added.

A saved logfile of the pcap debugging session mentioned below

2009-09-07 11:22:39 changed by Hachiman

please try 3.0.6 b1?

2009-09-08 06:34:20 changed by vondralbra

Thank you very much for the link to the upcoming 3.0.6 release. I've mailed you a logfile and three .pcaps of an ipsec session.
Seems like the VBox-NAT engine does currently not forward the NONESP-encap packets on port 4500/udp.

2009-09-08 07:12:17 changed by Hachiman

Could you please send me pcap files you've got with 3.0.6 b1?

2009-09-08 07:17:03 changed by vondralbra

In addition to that I tried every hardware type which Virtualbox provides to rule out driver dependencies on the guest operation system but to no avail.
After analyzing the pcaps made this morning, I saw that the missing UDP-packets are fragmented. 

tcpdump -vvnr trace-guest-nat.pcap | less

leads to:

06:02:31.836466 IP (tos 0x0, ttl 64, id 47184, offset 0, flags [+], proto UDP (17), length 1500) 10.0.2.15.4500 > 217.XXX.XXX.XXX.4500: NONESP-encap: [.. omitted ..])
06:02:31.836515 IP (tos 0x0, ttl 64, id 47184, offset 1480, flags [none], proto UDP (17), length 436) 10.0.2.15 > 217.XXX.XXX.XXX: udp

The NONESP-Packet is split into two fragments according to the MTU of the 'nat'ed interface of the guest.
None of the fragments show up on the host interface. Don't know whether this finding is directly related to this issue though.

(follow-up: ↓ 9 ) 2009-09-08 07:34:42 changed by vondralbra

Forgot to mention: This finding is made from the "trace-guest-nat.pcap" sent to you this morning.

(in reply to: ↑ 8 ) 2009-09-08 07:45:10 changed by Hachiman

Replying to vondralbra:

Forgot to mention: This finding is made from the "trace-guest-nat.pcap" sent to you this morning.

Could you please re-send it because I've not got it yet?

2009-09-08 08:04:43 changed by Hachiman

Good, finely received, thanks. BTW does skip packets with if change adapter on pcnet or e1k ?

2009-09-08 20:11:25 changed by vondralbra

Like i said at 2009-09-08 07:17:03: I switched through every available hardware to check whether a driver problem is in question.
I made a pcap of all sessions involved in this driver switch.
All pcaps had the same pattern: The fragmented NONESP on 4500/udp did not get through the NAT layer.
The sourcecode on the changeset query is recent and all the same over the different flavours of VirtualBox?
I saw some interesting debugging hooks lurking around there and I wonder whether they are compile time only.

2009-09-12 10:05:43 changed by vondralbra

Issue exists also in the 3.0.6 release as of 9/9/9.
To reproduce the packet drop of fragmented packets in this 3.0.6 release install hping2 on the guest and run: 

hping --udp -p 4500 -d 1500 217.XXX.XXX.XXX

on the guest. This sends a 1500 byte UDP-Packet on Port 4500 to the host 217.XXX.XXX.XXX.

When monitoring the traffic to 217.XXX.XXX.XXX on the host interface no traffic is seen by the command 

tcpdump -n -i eth0 host 217.XXX.XXX.XXX

on the host.

If the "-d " parameter to hping2 is set to 1300 then the traffic passes flawlessly. 

tcpdump -n -i eth0 host 217.XXX.XXX.XXX
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
09:55:30.075483 IP 192.168.186.199.34351 > 217.XXX.XXX.XXX.4500: UDP-encap: ESP(...), length 1300
09:55:31.073671 IP 192.168.186.199.60190 > 217.XXX.XXX.XXX.4500: UDP-encap: ESP(...), length 1300
09:55:32.069253 IP 192.168.186.199.45788 > 217.XXX.XXX.XXX.4500: UDP-encap: ESP(...), length 1300
^C
3 packets captured
4 packets received by filter
0 packets dropped by kernel

The "-d" parameter should be modified accordingly to your setup and network's MTU size.
I saw a ICMP-Frag-needed packet was able to switch the NAT-Engine to pass the fragments through. 

root@fw:~# hping --udp -p 4500 -d 1470 217.XXX.XXX.XXX
HPING 217.XXX.XXX.XXX (eth1 217.XXX.XXX.XXX): udp mode set, 28 headers + 1470 data bytes
ICMP Fragmentation Needed/DF set from ip=192.168.186.254

like seen here: 

09:51:00.002204 IP 192.168.186.199.37574 > 217.XXX.XXX.XXX.4500: UDP-encap: ESP(...), length 1464
09:51:00.002212 IP 192.168.186.199 > 217.XXX.XXX.XXX: udp
09:51:00.998175 IP 192.168.186.199.60287 > 217.XXX.XXX.XXX.4500: UDP-encap: ESP(...), length 1464
09:51:00.998181 IP 192.168.186.199 > 217.XXX.XXX.XXX: udp
09:51:01.997823 IP 192.168.186.199.38287 > 217.XXX.XXX.XXX.4500: UDP-encap: ESP(...), length 1464
09:51:01.997831 IP 192.168.186.199 > 217.XXX.XXX.XXX: udp
09:51:02.990996 IP 192.168.186.199.44635 > 217.XXX.XXX.XXX.4500: UDP-encap: ESP(...), length 1464
09:51:02.991000 IP 192.168.186.199 > 217.XXX.XXX.XXX: udp

2009-09-28 20:43:33 changed by Hachiman

vondralbra, Could you please check if http://virtualbox.org/download/testcase/virtualbox-3.0_3.0.7-52933_Debian_lenny_amd64.deb fixes problem for you?

(follow-up: ↓ 15 ) 2009-09-29 06:22:08 changed by vondralbra

Thank you very much.
Installation went flawlessly atop 3.0.6 production.
But IPSEC is still not working in NAT but in bridged mode.

md5sum of the DeviceDriver? is: c55788251663d30656ba8d598b22e2cd VBoxDD.so
pcaps taken on the guest and on the host simultaneously show,
that the sending side (guest) is ok now.

The receiving side on the guest receives malformed packets during IKE_AUTH.

(in reply to: ↑ 14 ) 2009-09-29 06:39:37 changed by Hachiman

Replying to vondralbra:

Thank you very much.
Installation went flawlessly atop 3.0.6 production.
But IPSEC is still not working in NAT but in bridged mode.
md5sum of the DeviceDriver? is: c55788251663d30656ba8d598b22e2cd VBoxDD.so
pcaps taken on the guest and on the host simultaneously show,
that the sending side (guest) is ok now.

The receiving side on the guest receives malformed packets during IKE_AUTH.

Could you please send pcaps files to me ?

(follow-up: ↓ 17 ) 2009-09-29 08:19:47 changed by Hachiman

Thanks for pcap files. From them I can conclude that udp socket fetches received server response not fully, to verify this I'd offer you debug library to get more detailed log.

(in reply to: ↑ 16 ; follow-up: ↓ 18 ) 2009-09-29 09:44:13 changed by Hachiman

Replying to Hachiman:

Thanks for pcap files. From them I can conclude that udp socket fetches received server response not fully, to verify this I'd offer you debug library to get more detailed log.

Could you please download http://virtualbox.org/download/testcase/VBoxDD.so.4801? It's debug build compatible you've downloaded before. Could you please replace corresponded bits and do the following: 

# export VBOX_LOG=drv_nat.e.l2.l3.l4
# export VBOX_LOG_DEST=file=nat.log
# VirtualBox -startvm <your-vm>

I'd very appreciate if you'll able to send to me nat.log and release log from ~/.VirtualBox plus pcap files :).

(in reply to: ↑ 17 ) 2009-09-29 11:04:40 changed by Hachiman

Replying to Hachiman:

Replying to Hachiman:

Other bits http://virtualbox.org/download/testcase/VBoxDD.so.4801.exp 

# export VBOX_LOG=drv_nat.e.l2.l3.l4
# export VBOX_LOG_DEST=file=nat.log
# VirtualBox -startvm <your-vm>

I'd very appreciate if you'll able to send to me nat.log. Here I've add code checking the rest bits on the socket available for reading after first read.

2009-09-30 06:41:12 changed by Hachiman

In received pcap files I observed that IKE_AUTH, indeed not fully forwarded to guest. I've updated VBoxDD.so.4801.exp bits to log socket operation and latter fragmentation, to detect at which steps the corruption is happens. I'd appreciate if you collect pcap files with the logs like we did it before.
Thank you for cooperation.

2009-09-30 08:54:51 changed by Hachiman

I've updated VBoxDD.so.4801 bits, that fixes socket reading bug. Please try if it fixes problem for you. If no, please send me logs and pcaps will dig deeper.

(follow-up: ↓ 22 ) 2009-10-01 06:29:01 changed by vondralbra

Just tested the binary.
IPSEC in a NAT'ed guest works like a charm and out of the box now using this VBoxDD.so.4801. md5sum (for the record) of it is :
60ea1d7d8ce6e03c016b48b8e25055eb VBoxDD.so
Thank you very very much for your hard and eager work on this issue.

(in reply to: ↑ 21 ) 2009-10-01 06:45:02 changed by Hachiman

  • summary changed from IPSEC via NAT-Interface does not work properly in Linux Guest to IPSEC via NAT-Interface does not work properly in Linux Guest -> fixed in svn..

Replying to vondralbra:

Just tested the binary.
IPSEC in a NAT'ed guest works like a charm and out of the box now using this VBoxDD.so.4801. md5sum (for the record) of it is :
60ea1d7d8ce6e03c016b48b8e25055eb VBoxDD.so
Thank you very very much for your hard and eager work on this issue.

Great, will send you release bits.

2009-10-01 07:13:47 changed by Hachiman

here are release bits

2009-10-01 20:07:13 changed by vondralbra

Release-binary installs without any glitches atop 3.0.6 production.
IPSEC via NAT works right out of the box.
You guys - and especially hachiman - did a pretty good job!
Thanks a lot and all the best wishes!

2009-10-06 10:29:41 changed by sandervl73

  • status changed from new to closed.
  • resolution set to fixed.
  • summary changed from IPSEC via NAT-Interface does not work properly in Linux Guest -> fixed in svn. to IPSEC via NAT-Interface does not work properly in Linux Guest -> fixed in svn/3.0.8.


Contact – Privacy policy