VirtualBox

Ticket #14833 (closed defect: invalid)

Opened 6 years ago

Last modified 5 years ago

NAT doesn't work behind Microsoft Forefront TMG server

Reported by: Giangi Owned by:
Component: network/NAT Version: VirtualBox 5.0.10
Keywords: Cc:
Guest type: other Host type: other

Description (last modified by vushakov) (diff)

I have upgraded my VB from 4.3.12 directly to 5.0.10 and now none of my guests configured as NAT are able to navigate.

This ticket is related to this forum post:  https://forums.virtualbox.org/viewtopic.php?f=1&t=74498#p344920 I have found this bug report #13292 but is for an older VB release (v4)

I'm primarily using VB on a network which has Microsoft Forefront TMG as proxy/firewall. I do have full admin access on TMG and enabling the logging I do not see any errors but I do not see any "real traffic" too, just the start/close session

On my pc I have the Forefront TMG Client installed and enabled, could it be that its DLLs are being blocked?

In the logs there are many references to these DLLs, like the following.

1388.1bb0: supHardenedWinVerifyImageByHandle: -> 0 (\Device\HarddiskVolume2\Program Files\Forefront TMG Client\FwcWsp.dll) WinVerifyTrust
1388.1bb0: supR3HardenedWinVerifyCacheInsert: \Device\HarddiskVolume2\Program Files\Forefront TMG Client\FwcWsp.dll
1388.1bb0: supR3HardenedMonitor_LdrLoadDll: pName=C:\Program Files\Forefront TMG Client\FwcWsp.dll (rcNtResolve=0xc0150008) *pfFlags=0x0 pwszSearchPath=007c7b5c:C:\Program Files\Oracle\VirtualBox;C:\Windows\system32 [calling]
1388.1bb0: supR3HardenedScreenImage/NtCreateSection: cache hit (VINF_SUCCESS) on \Device\HarddiskVolume2\Program Files\Forefront TMG Client\FwcWsp.dll
1388.1bb0: supR3HardenedDllNotificationCallback: load 74bb0000 LB 0x001fc000 C:\Program Files\Forefront TMG Client\FwcWsp.dll [fFlags=0x0]
1388.1bb0: supR3HardenedScreenImage/LdrLoadDll: cache hit (VINF_SUCCESS) on \Device\HarddiskVolume2\Program Files\Forefront TMG Client\FwcWsp.dll

The guest NIC is configured as:

Configurazione IP di Windows
        Nome host . . . . . . . . . . . . . . : TESTXP1NEW
        Suffisso DNS primario  . . . . . . .  :
        Tipo nodo . . . . . . . . . . . . . .  : Ibrido
        Routing IP abilitato. . . . . . . . . : No
        Proxy WINS abilitato . . . . . . . .  : No
        Elenco di ricerca suffissi DNS. . . . : master.local
Scheda Ethernet Lan:
        Suffisso DNS specifico per connessione: master.local
        Descrizione . . . . . . . . . . . . . : AMD PCNET Family PCI Ethernet Adapter
        Indirizzo fisico. . . . . . . . . . . : 08-00-27-BB-9E-71
        DHCP abilitato. . . . . . . . . . . . : Sì
        Configurazione automatica abilitata   : Sì
        Indirizzo IP. . . . . . . . . . . . . : 10.0.2.15
        Subnet mask . . . . . . . . . . . . . : 255.255.255.0
        Gateway predefinito . . . . . . . . . : 10.0.2.2
        Server DHCP . . . . . . . . . . . . . : 10.0.2.2
        Server DNS . . . . . . . . . . . . .  : 10.0.2.3
        Lease ottenuto. . . . . . . . . . . . : lunedì 16 novembre 2015 13.39.27
        Scadenza lease . . . . . . . . . . .  : martedì 17 novembre 2015 13.39.27

DNS resolution is working...

C:\Documents and Settings\Utente>nslookup
*** Impossibile trovare nome server per l'indirizzo 10.0.2.3: Non-existent domain
*** I server predefiniti non sono disponibili
Server predefinito:  UnKnown
Address:  10.0.2.3
> set q=any
> google.com
Server:  UnKnown
Address:  10.0.2.3
Risposta da un server non di fiducia:
google.com      internet address = 173.194.112.137
google.com      internet address = 173.194.112.133
google.com      internet address = 173.194.112.130
google.com      internet address = 173.194.112.131
google.com      internet address = 173.194.112.136
google.com      internet address = 173.194.112.142
google.com      internet address = 173.194.112.134
google.com      internet address = 173.194.112.135
google.com      internet address = 173.194.112.128
google.com      internet address = 173.194.112.132
google.com      internet address = 173.194.112.129
google.com      nameserver = ns1.google.com
google.com      nameserver = ns3.google.com
google.com      nameserver = ns4.google.com
google.com      nameserver = ns2.google.com
google.com
        primary name server = ns1.google.com
        responsible mail addr = dns-admin.google.com
        serial  = 107925622
        refresh = 900 (15 mins)
        retry   = 900 (15 mins)
        expire  = 1800 (30 mins)

On my home network all the guests are connecting to internet without problems.

Attachments

VBox.zip Download (23.1 KB) - added by Giangi 6 years ago.
VBoxHardening.zip Download (29.8 KB) - added by Giangi 6 years ago.
BtMmHook.zip Download (109.4 KB) - added by Giangi 6 years ago.
VBupdate.zip Download (6.8 KB) - added by Giangi 6 years ago.
TMG log for VB update check
VBiso_dl1.zip Download (6.7 KB) - added by Giangi 6 years ago.
TMG log for first guest additions download
VBoxSVC.zip Download (2.1 KB) - added by Giangi 6 years ago.
VB host log for first guest additions download
guest.pcapng.gz Download (233.9 KB) - added by Giangi 6 years ago.
guest2.pcap.gz Download (224.0 KB) - added by Giangi 6 years ago.
ipc.txt Download (3.8 KB) - added by Giangi 6 years ago.

Change History

Changed 6 years ago by Giangi

Changed 6 years ago by Giangi

comment:1 in reply to: ↑ description Changed 6 years ago by Giangi

Replying to Giangi:

I have upgraded my VB from 4.3.12 directly to 5.0.10 and now none of my guests configured as NAT are able to navigate.

I forgot to mention that I have installed the current test build 5.0.11 but nothing changed

comment:2 follow-up: ↓ 4 Changed 6 years ago by vushakov

You are probably running into hardening problems. Hardening on Windows was introduced in 4.3.14, so #13292 is relevant.

comment:3 Changed 6 years ago by vushakov

  • Description modified (diff)

comment:4 in reply to: ↑ 2 Changed 6 years ago by Giangi

Replying to vushakov:

so #13292 is relevant.

I have posted there too, I guess this one could be closed?

comment:5 Changed 6 years ago by vushakov

In your case it looks like the certificate used to sign your bluetooth software was revoked.

comment:6 follow-up: ↓ 10 Changed 6 years ago by Giangi

Thanks, that is good to know but I do not think I will ever need to use the BT stack from within VB either in the host or in a guest! :-)

BTW, installing a more recent Broadcom BT suite (if available) could it solve?

Addendum, I have checked certificate for one of the files: BtMmHook.dll The certificate has been revoked before the signing!! How could it be possible??? https://i.imgur.com/oqevLNU.png

comment:7 Changed 6 years ago by Giangi

With sigcheck I do no see the "revoke statement"....

C:\Program Files\WIDCOMM\Bluetooth Software\BtMmHook.dll:
	Verified:	Signed
	Catalog:	C:\Program Files\WIDCOMM\Bluetooth Software\BtMmHook.dll
	Signers:
	   Broadcom Corporation
		Status:		Un certificato richiesto non rientra nel suo periodo di validità se verificato rispetto all'ora corrente del sistema o al timestamp sul file firmato.
		Valid Usage:	Code Signing
		Serial Number:	3A 8E 49 11 EA 41 4D E5 37 BC
				EE 2A AA B7 4F C7
		Thumbprint:	D1E1DF6516A9912556F3E471B431916D03944D0D
		Algorithm:	SHA1
		Valid from:	01:00 27/02/2009
		Valid to:	00:59 21/04/2012
	   VeriSign Class 3 Code Signing 2004 CA
		Status:		Un certificato richiesto non rientra nel suo periodo di validità se verificato rispetto all'ora corrente del sistema o al timestamp sul file firmato.
		Valid Usage:	Client Auth,
				Code Signing
		Serial Number:	41 91 A1 5A 39 78 DF CF 49 65
				66 38 1D 4C 75 C2
		Thumbprint:	197A4AEBDB25F0170079BB8C73CB2D655E0018A4
		Algorithm:	SHA1
		Valid from:	01:00 16/07/2004
		Valid to:	00:59 16/07/2014
	   VeriSign Class 3 Public Primary CA
		Status:		Valid
		Valid Usage:	Email Protection,
				Client Auth,
				Code Signing,
				Server Auth
		Serial Number:	70 BA E4 1D 10 D9 29 34 B6 38
				CA 7B 03 CC BA BF
		Thumbprint:	742C3192E607E424EB4549542BE1BBC53E6174E2
		Algorithm:	MD2
		Valid from:	01:00 29/01/1996
		Valid to:	00:59 02/08/2028
	Signing date:	00:25 26/03/2011
	Counter Signers:
	   VeriSign Time Stamping Services Signer - G2
		Status:		Un certificato richiesto non rientra nel suo periodo di validità se verificato rispetto all'ora corrente del sistema o al timestamp sul file firmato.
		Valid Usage:	Timestamp Signing
		Serial Number:	38 25 D7 FA F8 61 AF 9E F4 90
				E7 26 B5 D6 5A D5
		Thumbprint:	ADA8AAA643FF7DC38DD40FA4C97AD559FF4846DE
		Algorithm:	SHA1
		Valid from:	01:00 15/06/2007
		Valid to:	00:59 15/06/2012
	   VeriSign Time Stamping Services CA
		Status:		Un certificato richiesto non rientra nel suo periodo di validità se verificato rispetto all'ora corrente del sistema o al timestamp sul file firmato.
		Valid Usage:	Timestamp Signing
		Serial Number:	47 BF 19 95 DF 8D 52 46 43 F7
				DB 6D 48 0D 31 A4
		Thumbprint:	F46AC0C6EFBB8C6A14F55F09E2D37DF4C0DE012D
		Algorithm:	SHA1
		Valid from:	01:00 04/12/2003
		Valid to:	00:59 04/12/2013
	   Thawte Timestamping CA
		Status:		Valid
		Valid Usage:	Timestamp Signing
		Serial Number:	00
		Thumbprint:	BE36A4562FB2EE05DBB3D32323ADF445084ED656
		Algorithm:	MD5
		Valid from:	01:00 01/01/1997
		Valid to:	00:59 01/01/2021
	Publisher:	Broadcom Corporation
	Company:	Broadcom Corporation.
	Description:	Multimedia Keys Hook DLL
	Product:	Bluetooth Software
	Prod version:	6.3.0.8200
	File version:	6.3.0.8200
	MachineType:	32-bit

comment:8 Changed 6 years ago by vushakov

Please, can you upload the offending dll?

Changed 6 years ago by Giangi

comment:9 Changed 6 years ago by Giangi

Added!!

BTW: my problem is with NAT and Forefront TMG server, the Bluetooth dll being blocked is a marginal problem... :-)

comment:10 in reply to: ↑ 6 Changed 6 years ago by vushakov

Replying to Giangi:

Addendum, I have checked certificate for one of the files: BtMmHook.dll The certificate has been revoked before the signing!! How could it be possible???

Revocation requests are often created well in advance, often along with the original certificate, so that you can distribute the revocation when necessary.

comment:11 follow-up: ↓ 12 Changed 6 years ago by vushakov

On my home network all the guests are connecting to internet without problems.

Do you mean the same guests on the same laptop moved to a different network?

but I do not see any "real traffic" too, just the start/close session

What do you mean by "start/close session"? Please, provide logs and, ideally, packet captures.

comment:12 in reply to: ↑ 11 Changed 6 years ago by Giangi

Replying to vushakov:

On my home network all the guests are connecting to internet without problems.

Do you mean the same guests on the same laptop moved to a different network?

Almost: same guest (on an USB disk) but different host

but I do not see any "real traffic" too, just the start/close session

What do you mean by "start/close session"? Please, provide logs and, ideally, packet captures.

Tomorrow I can prepare the logs from the TMG server, the "start/close session" is referred to the TMG server: each "sessions" have at least two entries into the log, a start and close. Normally "in between" there is the traffic. With this build there is nothing, like nothing is leaving my pc directed to the TMG server! If you need a Wireshark logs please provide a step-by-step procedure because I know only how to start it! :-) If there are some "debug" parameters on VirtualBox just let me know!

BTW: the TMG client DLLs are referenced into the hardening log, do they are there for a warning or were blocked?

comment:13 Changed 6 years ago by bird

The Forefront Threat Management Gateway Client DLL (FwcWsp.dll) was loaded successfully from a quick glance at the logs.

I've taken a peek at the client code, just to see if there was something typical with respect to VBox hardening that stood out (e.g. OpenProcess or OpenThread calls from services), but I couldn't immediately spot anything... Seeing that you're upgrading from a non-hardened (4.3.12) to a hardened VBox (5.0.x), it's natural to suspect the problem is hardening related. That said, quite a few other things may have changed since 4.3 unrelated to hardening, of course.

I wonder if you could verify that the update check works (VirtualBox VM selector, file menu, check for updates). The VM selector process is not hardened, but it helps verify that we've can talk https over the Forefront TMG.

Another potentially useful test would be to see if we can download the additions iso:

  1. go to c:\program files\oracle\virtualbox and rename VBoxGuestAdditions.iso to something else.
  2. start a vm (empty vm without anything installed is fine)
  3. from the "devices" menu select "insert guest additions cd image" (bottom)
  4. It should now ask whether to download it, tell it to download it.
  5. If downloading the iso from virtualbox.org works, then basic http works from a hardened VirtualBox VM process, which means it's probably a NAT problem. (Otherwise, it's either hardening or buggy download code in the GUI. Trunk seems to have the latter atm.)
  6. Undo 1.

comment:14 Changed 6 years ago by Giangi

The update check works, the TMG's log follows as VBupdate.xls and the allowed entry is the one in bold/red. In human-readable format is

Allowed Connection SRVTMG02 18/11/2015 10:58:38 
Log type: Web Proxy (Forward) 
Status: 0 The operation completed successfully.  
Rule: Tutto per Giangi's pc 
Source: Internal (10.192.138.101:59637) 
Destination: External (update.virtualbox.org 137.254.60.34:443) 
Request: update.virtualbox.org:443 
Filter information: Req ID: 13c00194; Compression: client=No, server=No, compress rate=0% decompress rate=0% 
Protocol: SSL-tunnel 
User: anonymous 
 Additional information 
Object source: Internet (Source is the Internet. Object was added to the cache.)
Cache info: 0x0
Processing time: 0 MIME type:

Changed 6 years ago by Giangi

TMG log for VB update check

comment:15 Changed 6 years ago by Giangi

The first guest addition download test failed! Error is "During network request Connection refused." VB's proxy settings is Direct connection to the internet

TMG's log is VBiso_dl1.xls, here there is any connection! ...only the "Initiated/Closed" messages...

I'm attaching the VBoxSVC.log file too, but I've seen there only the warning for the missing guest additions iso...

Changed 6 years ago by Giangi

TMG log for first guest additions download

Changed 6 years ago by Giangi

VB host log for first guest additions download

comment:16 Changed 6 years ago by Giangi

The second guest addition download test failed as well! VB's proxy settings was changed to Auto-detect

comment:17 Changed 6 years ago by Giangi

The third guest addition download test failed too! VB's proxy settings was changed to Manual.

This time the error is different and I've seen the connection on TMG! VB message is "Host not found".

On TMG the url path was to  http://download.virtualbox.org/virtualbox/5.0.11/VBoxGuestAdditions_5.0.11.iso

But I have pasted that link into my IE and I've got an error 404. I guess I have first to downgrade to 5.0.10; I think the iso for .11 as not been released yet...

The allowed log on TMG is:

Allowed Connection SRVTMG02 18/11/2015 11:32:43 
Log type: Web Proxy (Forward) 
Status: 404 Not Found 
Rule: Tutto per Giangi's pc 
Source: Internal (10.192.138.101:60569) 
Destination: External (a213-254-17-111.deploy.akamaitechnologies.com 213.254.17.111:80) 
Request: HEAD http://download.virtualbox.org/virtualbox/5.0.11/VBoxGuestAdditions_5.0.11.iso 
Filter information: Req ID: 13c7b5e2; Compression: client=No, server=No, compress rate=0% decompress rate=0% 
Protocol: http 
User: anonymous 
 Additional information 
Client agent: Mozilla/5.0 (AgnosticOS; Blend) IPRT/64.42
Object source: Internet (Source is the Internet. Object was added to the cache.)
Cache info: 0x40800000 (Response includes the LAST-MODIFIED header. Response should not be cached.)
Processing time: 562 MIME type: text/html
Last edited 6 years ago by Giangi (previous) (diff)

comment:18 Changed 6 years ago by Giangi

Ok, I have downgraded to the official 5.0.10 r104061

The update checks works perfectly will all the three available proxy setting options

comment:19 Changed 6 years ago by Giangi

I have re-done the tests with the three options. With either auto-detect or direct there is no traffic logged on TMG... With the proxy manually configured VB has downloaded the iso file near to the end but has reported "Network operation failed with Unknows reason."

On TMG the error was "connection forcibly closed by the remote host"

Failed Connection Attempt SRVTMG02 18/11/2015 11:53:48 
Log type: Web Proxy (Forward) 
Status: 10054 An existing connection was forcibly closed by the remote host.  
Rule: Tutto per Giangi's pc 
Source: Internal (10.192.138.101:61062) 
Destination: External (213.254.17.111:80) 
Request: GET http://download.virtualbox.org/virtualbox/5.0.10/VBoxGuestAdditions_5.0.10.iso 
Filter information: Req ID: 13cc23e8; Compression: client=No, server=Yes, compress rate=0% decompress rate=0% 
Protocol: http 
User: anonymous 
 Additional information 
Client agent: Mozilla/5.0 (AgnosticOS; Blend) IPRT/64.42
Object source: Internet (Source is the Internet. Object was added to the cache.)
Cache info: 0x40800000 (Response includes the LAST-MODIFIED header. Response should not be cached.)
Processing time: 4945 MIME type: application/octet-stream

comment:20 Changed 6 years ago by Giangi

Uhm... it doesn't work from my IE too!! Looks like there is currently a problem on the remote host!

Technical Information (for support personnel)
##Error Code 10060: Connection timeout
##Background: The gateway could not receive a timely response from the website you are trying to access. This might indicate that the network is congested, or that the website is experiencing technical difficulties.
##Date: 11/18/2015 11:01:32 AM [GMT]

comment:21 Changed 6 years ago by Giangi

No way, I have tried with both the TMG client enabled and disabled but the iso download goes, slowly, up to about 55% and then it stops with "Network operation failed with Unknown reason."

From my pc it get downloaded from IE; using FreeDownloadManager it's like from within the guest: starts slowly, then once it stopped (and on the TMG there was the error "connection forcibly closed by the remote host") then restarted and complete successfully...

comment:22 Changed 6 years ago by Giangi

I'm pretty sure there is something wrong on the remote server... I have disconnected the internal LAN cable and enabled a wi-fi connection that is outside the internal LAN and it doesn't go through the TM server (TM client is disable).

Same error: progress goes up to 55% and then it stops.

comment:23 follow-up: ↓ 25 Changed 6 years ago by Giangi

Last report for today... :-) Using the "external Wi-Fi" connection the guest I've used for creating this ticket is connecting to internet via NAT without any problem!

comment:24 follow-up: ↓ 27 Changed 6 years ago by vushakov

Well, whatever it is with the server, the important thing is that you can connect to the server from the VirtualBox process.

Connections from within the guest still don't work, I take it?

Please, start a wireshark packet capture on the host. Then from the guest try to make an http connection to, say, www.virtualbox.org. Upload the resulting pcap file here.

comment:25 in reply to: ↑ 23 ; follow-up: ↓ 26 Changed 6 years ago by vushakov

Replying to Giangi:

Using the "external Wi-Fi" connection the guest I've used for creating this ticket is connecting to internet via NAT without any problem!

What do you mean by "using the external wi-fi connection"?

comment:26 in reply to: ↑ 25 ; follow-up: ↓ 28 Changed 6 years ago by Giangi

Replying to vushakov:

What do you mean by "using the external wi-fi connection"?

I mean that I have selected a Wi-Fi whose IP is not into the "internal LAN" range, call it a "public Wi-Fi" access point...

comment:27 in reply to: ↑ 24 Changed 6 years ago by Giangi

Replying to vushakov:

Please, start a wireshark packet capture on the host. Then from the guest try to make an http connection to, say, www.virtualbox.org. Upload the resulting pcap file here.

Done, WS has many file format... I have saved the capture with the first two format (gzipped). I was able to reopen them from WS... :-)

The TMG server (actually they are two with NLB) is called proxy-re.master.local with ip 192.168.16.23 (yes, the TMG server is not my default gateway...)

Attached you'll find my ipconfig too (it's in Italian...)

Changed 6 years ago by Giangi

Changed 6 years ago by Giangi

Changed 6 years ago by Giangi

comment:28 in reply to: ↑ 26 Changed 6 years ago by vushakov

Replying to Giangi:

Replying to vushakov:

What do you mean by "using the external wi-fi connection"?

I mean that I have selected a Wi-Fi whose IP is not into the "internal LAN" range, call it a "public Wi-Fi" access point...

In which case TMG is not used, I assume?

comment:29 Changed 6 years ago by vushakov

  • Status changed from new to closed
  • Resolution set to invalid

Your captures show that the host talks remote-winsock protocol (tcp/1745) to your TMG server. The protocol is proprietary and wireshark doesn't even have a dissector for it, though they seem to have a dissector for an earlier UDP based MS Proxy (udp/1745) protocol.

I guess TMG loads a winsock component into the client program that intercepts winsock API calls and translates them into remote-winsock instead. As the same sequence of socket calls does work without TMG, I'd say the problem you see are either a bug in TMG or a problem with TMG configuration. Since I can't look inside the TMG traffic without extra effort, I can't even venture a guess, unfortunately.

So I don't think this is VirtualBox problem and that there's anything we can do to help you. Your best bet at this point, I guess, is to turn on very verbose logging in TMG and try to work out what's going on on its side.

comment:30 Changed 6 years ago by Giangi

I can understand your point of view, but before closing this ticket please explain me this: why VB 4.3.12 is working with the same TMG server\TMG client ??

To me this is neither a bug nor a problem on TMG

Last edited 6 years ago by Giangi (previous) (diff)

comment:31 Changed 6 years ago by vushakov

I can't explain something I can't analyse. A different sequence of winsock calls may be? It works with plain winsock. TMG is supposed to be more-or-less transparent to the application. If it's not (plain works, TMG doesn't), then it's TMG part that needs to be analysed and since it's proprietary and closed, there's not much I can do.

At the very minimum a very verbose/debug TMG logs with protocol level details is necessary. And even then it's TMG and not VirtualBox that we are investigating.

comment:32 Changed 6 years ago by Giangi

Exactly, TMG is supposed to be as much as transparent. Something has changed internally in VB from v4 to v5 and it's that change that should be analysed... ...and I bet that is (again) that nice hardening feature added... :-( Anyway, thank a lot for your help and patience. Fortunately the TMG server will be replaced early 2016... until then I can:

  1. disconnect my LAN cable and use the "public wifi"
  2. downgrade, again, to the best VB ever: 4.3.12!! :-)

comment:33 Changed 6 years ago by Giangi

Just in case someone else maybe interested, I have solved with a trick: installed cNTLM on the host and configured the guest with NAT to use it as a proxy. It works.

comment:34 Changed 5 years ago by Martinko

I've got the same problem with Forefront TMG. If my guest does a request to host on the local address table (see FwcTool printconfig), then the process VBoxHeadless.exe running in user context opens an connection to the host and everything is fine. If the guest tries to connect to an host secured by TMG, then the [System Process] with PID 0 opens a connection to the TMG server port 1745 instead. As the system process ain't running in user context the permission isn't granted. Running the same process natively, all connections are opened by the same java process in user context. I used Sysinternals TCPView to check on connection and processes. Using Virtualbox 5.0.0. Using Vagrant-Options to pimp nat-settings:

v.customize ["modifyvm", :id, "--nataliasmode1", "sameports"] v.customize ["modifyvm", :id, "--natbindip1", "<local-ip>"]

Was the "hardening" moving some networking parts from userspace to kernelspace? See "Hardening on Windows was introduced in 4.3.14, so #13292 is relevant." above.

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use