Infinite Loop in Guest Addition for Windows XP (1 Core always 100% on VBoxTray.exe)

[Francesco Bucciantini]

Hi guys, on July 8th 2024 I raised an issue with the development snapshot uploaded on 2024-07-04T18:53:02, namely version 7.0.97 r163779 on Fedora 40, kernel 6.9.6 as it was cycling through something. In my 4c/8th configuration the process VirtualBoxVM constantly used at least 15% CPU time which means that it's always using 1 core at 100% despite the guest being completely idling. This seems to be occurring regardless of the guest, although I made the tests in particular on Windows XP. Anyway everything pointed towards an issue within the hypervisor. This didn't happen with the last but one development snapshot version and it also doesn't happen with the last but one stable build, namely 7.0.18 r162988. I was hoping this was gonna be debugged and caught early before the stable release, but unfortunately I can now confirm that the issue has been carried over from the development snapshot 7.0.97 r163779 to the last stable release 7.0.20 r163906.

Logs: ​https://forums.virtualbox.org/download/file.php?id=52945

Forum reference for 7.0.97 r163779 Development Snapshot ​https://forums.virtualbox.org/viewtopic.php?p=548925#p548925

Forum reference for 7.0.20 r163906 Stable: ​https://forums.virtualbox.org/viewtopic.php?t=111893

[Francesco Bucciantini]

Screenshot highlighting the problem with the guest addition (25% CPU in 4c/4th means 1 core is always at 100%)

[Francesco Bucciantini]

Logs of the VM experiencing the issue

[Francesco Bucciantini]

I can also confirm that reverting to the guest additions version 7.0.19 r163782 works and the infinite loop is gone.

So now it's just a matter of comparing the changes between 7.0.19 r163782 and 7.0.20 r163906 and spot which commit caused the issue.

[Francesco Bucciantini]

Virtualbox 7.1.0 r164395

[Francesco Bucciantini]

The issue is also occurring on 7.1.0 r164369 and 7.1.0 r164395 from August 15th, 2024. Given that 7.1.0 is Beta, it has also been reported in the dedicated Beta section of the forum: ​https://forums.virtualbox.org/viewtopic.php?t=111997

Once again, downgrading the guest additions to 7.0.19 r163782 (or, for those who prefer stable, 7.0.18 ​https://download.virtualbox.org/virtualbox/7.0.18/VBoxGuestAdditions_7.0.18.iso) solves the issue. In other words, I can confirm that the last guest additions to not show the issue are 7.0.19 r163782 and using those solves the issue on both 7.0.20 and 7.1.0.

[pentagonik]

Thanks for the report; I'll have a look into this.

[pentagonik]

It could be that this particular snapshot version(s) had the same bug we were seeing in some other reports, which caused a high host CPU load.

Did the host CPU load drop as soon as you terminated VBoxTray on the guest?

[Francesco Bucciantini]

Yes, the host CPU load dropped as soon as I killed the VBoxTray.exe process. That is, if you're using Bridged Network and not NAT, if you're using NAT, then you still had the high CPU load in the host despite the guest using basically nothing. This was until yesterday's release Virtualbox 7.1.0 r164443 which addressed the NAT issue.

In other words:

Test 1:

Virtualbox: 7.1.0 r164369 / r164395 Guest Additions: 7.1.0 r164395 Network: NAT

Guest CPU: 25% for a 4c configuration (infinite loop in VBoxTray.exe in one core) Host CPU: 66% for a 4c/8th configuration (high CPU usage beyond 1 core in infinite loop)

Test 2:

Virtualbox: 7.1.0 r164369 / r164395 Guest Additions: 7.1.0 r164395 Network: Bridged Network

Guest CPU: 25% for a 4c configuration (infinite loop in VBoxTray.exe in one core) Host CPU: 15% for a 4c/8th configuration (1 core stuck in infinite loop changing every few seconds, typical in Linux + a little vm overhead, which is normal)

Test 3:

Virtualbox: 7.1.0 r164369 / r164395 Guest Additions: 7.0.19 r163782 Network: NAT

Guest CPU: 5% for a 4c configuration (normal usage) Host CPU: 55% for a 4c/8th configuration (high CPU usage)

Test 4:

Virtualbox: 7.1.0 r164369 / r164395 Guest Additions: 7.0.19 r163782 Network: Bridged Network

Guest CPU: 5% for a 4c configuration (normal usage) Host CPU: 9% for a 4c/8th configuration (normal usage)

In other words, using the old guest additions makes the infinite loop in the guest go away, however it does nothing to prevent the infinite loop in the host if you're using NAT. That isn't the case if you're using bridged network, though. That is until yesterday's release, Virtualbox 7.1.0 r164443, as I was saying. With yesterday's release, I can confirm that the NAT infinite loop on the host is solved:

Virtualbox: 7.1.0 r164443 Guest Additions: 7.0.19 r163782 Network: NAT

Guest CPU: 5% for a 4c configuration (normal usage) Host CPU: 9% for a 4c/8th configuration (normal usage)

Virtualbox: 7.1.0 r164443 Guest Additions: 7.0.19 r163782 Network: Bridged Network

Guest CPU: 5% for a 4c configuration (normal usage) Host CPU: 9% for a 4c/8th configuration (normal usage)

So... one problem down, one to go. :D Thank you for fixing the first problem by the way (the NAT one). Now all is left is to also fix the issue in the guest addition for the guest. :)

[pentagonik]

Does this still apply to Beta 2?

[Francesco Bucciantini]

Yes, it also applies to Virtualbox 7.1.0 r164448 (aka BETA 2). New logs depicting the same issue occurring with Beta 2 in attachment (WindowsEmbedded-2024-08-25-18-51-31.zip).

[Francesco Bucciantini]

Issue still occurring with Virtualbox 7.1.0 r164448 (aka BETA 2)

[Francesco Bucciantini]

Logs of the issue occurring on Virtualbox 7.1.0 r164448 (aka BETA 2)

[Francesco Bucciantini]

I've just installed the new Guest Addition version 7.1.0 r164481 and the problem persists. I tried to set the environment variables as it says here: https://www.virtualbox.org/wiki/AdditionsLogging however it didn't work as it didn't create any file in the specified folder. In attachment the normal logs for 7.1.0 r164481 (WindowsEmbedded-2024-08-28-10-17-42.zip).

[Francesco Bucciantini]

Virtualbox 7.1.0 r164481

[Francesco Bucciantini]

Virtualbox 7.1.0 r164481

[Francesco Bucciantini]

New logs, also from within the guest. Essentially, I went into

and I created the DWORD entry LoggingEnabled with Decimal 255 (Hex ff)

Then I added the two Environment Variables:

and I created the "Temp" folder in C: to allow logs to be written there:

At this point, I rebooted and logged in with my normal Active Directory user and sure enough the logs were there. :)

Unfortunately, however, when I opened them, I was a bit surprised by the lack of details. It looks like there isn't much being logged... :(

00:00:00.011718 main     VBoxTray 7.1.0_BETA2 r164482 win.x86 (Aug 21 2024 16:45:56) release log
00:00:00.011718 main     Log opened 2024-08-30T06:41:27.714716700Z
00:00:00.011718 main     OS Product: Windows XP Professional
00:00:00.012695 main     OS Release: 5.1.2600
00:00:00.012695 main     Executable: C:\WINDOWS\system32\VBoxTray.exe
00:00:00.012695 main     Process ID: 3836
00:00:00.012695 main     Package type: WINDOWS_32BITS_GENERIC
00:00:00.012695 main     Verbosity level: 0

When I shut down the guest, I've got the message that VBoxTray couldn't be stopped correctly (probably 'cause it was still using resources being stuck in an infinite loop with the same 25% CPU usage in a 4c/4th configuration):

Then I killed it and I shut down the host:

Anyway, in attachment you can find the logs, both from the Host and the Guest.

[Francesco Bucciantini]

VBoxTray logs 7.1.0 r164481

[Francesco Bucciantini]

Virtualbox logs 7.1.0 r164481

[Francesco Bucciantini]

The Guest Additions were updated today at 13:14:26 BST in the development snapshot page, so I installed the new version, namely r164641 and I rebooted the guest. Unfortunately, things got... uhm... worse, but you know what they say "things have to get worse before they get better". Jokes aside, I have no idea what the new VBoxTray.exe is doing, but whatever it is, it's no longer single threaded with 1 core out of 4 being at 100% resulting in 25% CPU usage, but rather multithreaded with the load being spread across 2 cores and the CPU usage being 50% now in a 4c configuration. After waiting for a while (around 20 minutes), however, the CPU usage dropped down to 25% once again, thus going back to the old behavior. Things look interesting for the guest logs this time round as there are more things going on.

00:00:00.007812 main     VBoxTray 7.1.0_BETA2 r164641 win.x86 (Sep  2 2024 13:06:42) release log
00:00:00.007812 main     Log opened 2024-09-02T16:15:36.763906500Z
00:00:00.008789 main     OS Product: Windows XP Professional
00:00:00.009765 main     OS Release: 5.1.2600
00:00:00.009765 main     Executable: C:\WINDOWS\system32\VBoxTray.exe
00:00:00.009765 main     Process ID: 3572
00:00:00.009765 main     Package type: WINDOWS_32BITS_GENERIC
00:00:00.009765 main     Verbosity level: 0
00:00:00.014648 main     Windows version 5.1 build 2600 (uNtVersion=0x50010000000a28)
00:00:23.383789 main     Starting services ...
00:00:23.383789 main     Starting service 'display' ...
00:03:01.079101 main     Service 'display' started
00:03:01.080078 main     Starting service 'clipboard' ...
00:03:01.081054 main     Shared Clipboard: New Clipboard API not available (VERR_SYMBOL_NOT_FOUND)
00:03:01.088867 main     Shared Clipboard: Guest features: 0x7 - Host features: 0x3
00:03:01.089843 main     Shared Clipboard: Using chunk size 65536 (maximum is 134213632)
00:05:54.214843 shclwnd  Shared Clipboard: Initialized OLE in window thread
00:05:54.232421 shclwnd  Shared Clipboard: Window thread running
00:08:18.444335 clipboard Shared Clipboard: Worker loop running
00:08:18.445312 main     Service 'clipboard' started
00:08:18.446289 clipboard Shared Clipboard: Initialized OLE in worker thread
00:08:18.447265 main     Starting service 'seamless' ...
00:10:45.871093 main     Service 'seamless' started
00:10:45.871093 main     Starting service 'VRDP' ...
00:13:08.093750 main     Service 'VRDP' started
00:13:08.094726 main     Starting service 'IPC' ...
00:13:08.096679 main     VBoxIPCInit: Local IPC server now running at "\\.\pipe\VBoxTrayIPC-Bucciantinif"
00:15:32.291015 main     Service 'IPC' started
00:15:32.291015 main     Starting service 'LA' ...
00:15:32.291992 main     LA: RegQueryValueExW: failed [SOFTWARE\Oracle\VirtualBox Guest Additions/VBoxTrayLog]
00:15:32.291992 main     LA: RegQueryValueExW: failed [SOFTWARE\Oracle\VirtualBox Guest Additions/VBoxTrayLA]
00:15:32.292968 main     LA: DetachOnDisconnect=true
00:18:04.646484 main     Service 'LA' started
00:18:04.647460 main     Starting service 'draganddrop' ...
00:21:04.146484 dndwnd   DnD: Guest features: 0x0 - Host features: 0x0
00:21:34.138671 main     DnD: Failed to initialize proxy window, rc=VERR_NOT_SUPPORTED
00:22:34.139648 main     DnD: Initializing drag and drop service failed with rc=VERR_NOT_SUPPORTED
00:22:34.139648 main     Service 'draganddrop' is not supported on this system
00:22:34.139648 main     All services started
00:23:25.402343 dndwnd   DnD: Creating drop target failed with hr=0x80070057

The total run of the guest was 45 minutes after which I shut it down. In attachment you can find the screenshots along with the logs from both the host and the guest running version 7.1.0 r164448 with Guest Addition r164641.

[Francesco Bucciantini]

CPU Usage 7.1.0 r164448 with Guest Addition r164641

[Francesco Bucciantini]

Host logs Virtualbox 7.1.0 r164448 with Guest Addition r164641

[Francesco Bucciantini]

Guest logs Virtualbox 7.1.0 r164448 with Guest Addition r164641

[Francesco Bucciantini]

I've updated to Virtualbox 7.1.0 r164699 with guest addition version r164700 uploaded on September 6th and unfortunately the issue still occurs. Looks like I'm still getting the line "New Clipboard API not available (VERR_SYMBOL_NOT_FOUND)" in the logs. New logs in attachment.

[Francesco Bucciantini]

Host logs Virtualbox 7.1.0 r164699 with guest addition version r164700

[Francesco Bucciantini]

Guest logs Virtualbox 7.1.0 r164699 with guest addition version r164700

[Francesco Bucciantini]

Updated to Virtualbox 7.1.0 r164728. Unfortunately the issue is still there. New logs (both host and guest) in attachment.

[Francesco Bucciantini]

Host logs Virtualbox 7.1.0 r164728

[Francesco Bucciantini]

Guest logs Virtualbox 7.1.0 r164728

[Francesco Bucciantini]

Tested on Virtualbox 7.1.2 r164945 stable and unfortunately the problem is still there.

[Francesco Bucciantini]

Tested on Virtualbox 7.1.3 r165012 development snapshot and unfortunately the problem is still there.

[Francesco Bucciantini]

logs: ​https://forums.virtualbox.org/download/file.php?id=53572&sid=2631c1e579d63953ebe4a40033f88c7a

screenshot: ​https://forums.virtualbox.org/download/file.php?id=53571&sid=2631c1e579d63953ebe4a40033f88c7a

[Francesco Bucciantini]

Host logs Virtualbox 7.1.3 r165012

[Francesco Bucciantini]

Tested on Virtualbox 7.1.4 r165100 Stable and unfortunately the problem is still there.

[Francesco Bucciantini]

Host logs Virtualbox 7.1.4 r165100 Stable

[Francesco Bucciantini]

Virtualbox 7.1.4 r165100 Stable CPU Usage screenshot

[Francesco Bucciantini]

Fast forward to the end of November and the situation hasn't changed. If anything, it has gotten worse. Starting with Virtualbox 7.1.97 r165364 released in October, all the way through Virtualbox 7.1.97 r165570 released on October 25th the installation of guest additions simply fails on Windows XP x86.

As of Virtualbox 7.1.97 r166019 with guest additions r165950 from November 20th 2024, the latest one, Windows XP x86 still fails to install them with the same errors displayed above. This has also been reported here: ​https://forums.virtualbox.org/viewtopic.php?p=552102

[Francesco Bucciantini]

Virtualbox 7.1.97 r166019 with guest additions r165950 installation fail Windows XP x86 Screenshot 1

[Francesco Bucciantini]

Virtualbox 7.1.97 r166019 with guest additions r165950 installation fail Windows XP x86 Screenshot 2

[Francesco Bucciantini]

Virtualbox 7.1.97 r166019 with guest additions r165950 installation fail Windows XP x86 Screenshot 3

[Francesco Bucciantini]

New year, old problems. Virtualbox 7.1.97 r166619 with guest additions 7.1.97 r166611 fails the installation on XP x86.

[Francesco Bucciantini]

Virtualbox 7.1.97 r166619 screenshot 1

[Francesco Bucciantini]

Virtualbox 7.1.97 r166619 screenshot 2

[Francesco Bucciantini]

Virtualbox 7.1.6 r167084 Stable has been released and unfortunately this issue managed to creep its way into this production release. This is really a shame... :(

[Francesco Bucciantini]

Virtualbox 7.1.6 r167084 Stable (error picture)

[Francesco Bucciantini]

Virtualbox 7.1.6 r167084 Stable (Guest Addition Installation Logs - Error 1)

[Francesco Bucciantini]

Virtualbox 7.1.6 r167084 Stable (Guest Addition Installation Logs - Error 2)

[Francesco Bucciantini]

Virtualbox 7.1.7 r167129 (test build) has been released and it fixes the installation problem affecting 7.1.97 r165364 up to 7.1.6 r167084 Stable, which is definitely good news, however the original issue, namely the infinite loop issue when shared clipboard is enabled and bidirectional between Fedora (Xorg X11) and Windows XP x86 Professional remains.

[Francesco Bucciantini]

Virtualbox 7.1.7 r167129 logs (infinite loop VBoxTray.exe)

[Francesco Bucciantini]

Virtualbox 7.1.7 r167129 picture (infinite loop VBoxTray.exe)

[Francesco Bucciantini]

Virtualbox 7.1.7 Process Explorer analysis of VBoxTray.exe

[Francesco Bucciantini]

Virtualbox 7.1.7 after killing the problematic thread while the VBoxTray.exe main process survives

[Francesco Bucciantini]

Things got interesting with 7.1.7. The installation proceeds correctly, but then there's VBoxTray.exe stuck at 25% of CPU (i.e 1 core at 100% in a 4c/4th configuration).

Now, I tried to dig things out a bit using process explorer and as you can see the situation is as follows:

As you can see, it's looping at 25% CPU usage intervals, then it goes very briefly to 0% and it goes back to 25% again, stays there for an extensive period of time, then briefly goes to 0%, then 25% again etc, in other words, it loops with something going bad, but shared clipboard between guest and host works fine.

Digging this further we can see the following:

Interestingly enough out of all the spawned threads by the VBoxTray.exe process there's only one causing the loop. What's even more interesting is that killing the affected thread has no undesirable effect, in fact once I killed it the situation goes back to normal. Not only the process survives, but shared clipboard keeps working normally between the host and the guest and the CPU usage is well within control this time.

I've let it run for almost 40 minutes and there are no signs of the issue coming back, which means that out of all the VBoxTray.exe spawned threads only one is going in loop and it's not the one dedicated to handling copy/paste of text between host and guest and viceversa (i.e guest to host).

[pentagonik]

Thank you for this further investigation -- we have this bug for several versions now, but weren't able to successfully reproduce this. Those data points are very valuable for us.

Since VBox 7.1 VBoxTray has the ability to "only run" or disable sub services within VBoxTray, which hopefully should make it easier pinpointing the issue.

For example, to disable the Shared Clipboard sub service within VBoxTray, start VBoxTray with "VBoxTray --disable-clipboard". To *only* run VBoxTray with the Shared Clipboard sub service (i.e. all other sub services are disabled), do a "VBoxTray --only-clipboard". See the syntax help "VBoxTray /?" for all available sub services.

Can you please try further pinpointing the issue using the options above?

[bird]

Thanks for the "Stack for thread 3296" - it is really helpful! The VBoxTray.exe addresses in there translates as follows:

 [D:\tinderbox\add-7.1\src\VBox\Runtime\r3\win\thread-win.cpp @ 378] (00000000`00d3ffa0)   VBoxTray!rtThreadNativeMain+0x90   |  (00000000`00d40070)   VBoxTray!rtThreadNativeUninitComAndOle
0:000> ln VBoxTray.exe + 2839c
Browse module
Set bu breakpoint

 [D:\tinderbox\add-7.1\src\VBox\Runtime\common\misc\thread.cpp @ 745] (00000000`00d38360)   VBoxTray!rtThreadMain+0x3c   |  (00000000`00d38430)   VBoxTray!rtThreadReInitObtrusive
0:000> ln VBoxTray.exe + 2c02
Browse module
Set bu breakpoint

 [D:\tinderbox\add-7.1\src\VBox\Additions\WINNT\VBoxTray\VBoxTray.cpp @ 227] (00000000`00d12be0)   VBoxTray!vboxTrayServiceThread+0x22   |  (00000000`00d12c90)   VBoxTray!vboxTrayServicesStart
0:000> ln VBoxTray.exe + ec53
Browse module
Set bu breakpoint

 [D:\tinderbox\add-7.1\src\VBox\Additions\WINNT\VBoxTray\VBoxIPC.cpp @ 590] (00000000`00d1eab0)   VBoxTray!VBoxIPCWorker+0x1a3   |  (00000000`00d1ede0)   VBoxTray!vboxIPCSessionThread
0:000> ln VBoxTray.exe + 27ee5
Browse module
Set bu breakpoint

 [D:\tinderbox\add-7.1\src\VBox\Runtime\common\misc\thread.cpp @ 795] (00000000`00d37e50)   VBoxTray!RTThreadCreate+0x95   |  (00000000`00d37f80)   VBoxTray!RTThreadSelfName
0:000> ln VBoxTray.exe + 3046c
Browse module
Set bu breakpoint

 [D:\tinderbox\add-7.1\src\VBox\Runtime\r3\win\thread-win.cpp @ 425] (00000000`00d40420)   VBoxTray!rtThreadNativeCreate+0x4c   |  (00000000`00d404a0)   VBoxTray!rtThreadNativeDestroy

This means we're probably spinning in the IPC service for some weird reason. There is no pushback (Sleep(1)) built into that thread, unfortunately, so it will spin hard if something with the connection logic goes south.

P.S. Please ignore the "VBoxTray --only-clipboard" stuff above, my coworker confuses VBoxTray and VBoxService options there. There isn't currently any way to disable the functionality in VBoxTray, but I guess we will try introduce that promptly as well as try figure out what's wrong here.

[pentagonik]

I've fixed the symptom but didn't find the cause for it yet.

The Windows XP Embedded you're using as a guest, is this some special build? Does the same error also happen with a regular Windows XP (or any other Windows) guest?

I'll upload a new test build shortly.

[Francesco Bucciantini]

Thank you Penta! Thank you Bird! I look forward to testing the new build. :D As for Windows XP Embedded, it's most commonly used on systems like tills, fuel pumps, water control systems etc, but it's really not different from a normal version of Windows XP, the only difference is that customers didn't have to pay extra for the Extended Support Updates.

In other words, it's pretty much the same as Windows XP Professional x86 with the Microsoft Premier Support (i.e Extended Support updates until July 2019). Those updates were supposed to be bought separately by corporations and they were only delivered via WSUS which was configured via Group Policy according to their own OU. I have a second system with Windows XP Professional x86 registered to the Active Directory domain and the same exact issue occurs.

[pentagonik]

I've uploaded new 7.1 test builds which can be found here: https://www.virtualbox.org/wiki/Testbuilds

VirtualBox 7.1 builds >= 167262 should fix the symptoms. For that the Guest Additions need to be updated and the guest rebooted.

Please let me know if that fixes the issue for you.

[Francesco Bucciantini]

I installed Virtualbox 7.1.7 r167269 and the guest additions to 7.1.7 r167267, however that didn't quite work.

Here's a video depicting the issue: ​https://we.tl/t-uUQksvPLln (link available for 3 days)

I've added an heavily compressed version of that video to the ticket as well, but it's better to use the WeTransfer version since I had to compress the bugtracker one down to 500 KB to make it fit.

[Francesco Bucciantini]

Virtualbox 7.1.7 r167269 video depicting the issue

[pentagonik]

Thanks for the video. Did you check if VBoxTray.exe actually has been upgraded to r167269? That can be checked via the file properties of VBoxTray.exe.

I've now integrated the changes I was talking about in comment:26.

So to investigate this further, kill a running instance of VBoxTray.exe first, open a command line and manually start VBoxTray via

VBoxTray --foreground -vvvvv --only-ipc

That only will start the IPC sub service (which is what we think is the culprit here).

[pentagonik]

You also can then proof if that really is related to the IPC sub service by starting VBoxTray (this time in the background) via

VBoxTray --disable-ipc

[Francesco Bucciantini]

I've checked that the VBoxTray.exe has been updated and indeed it was:

I've disabled it from the auto-start and rebooted. At that point, I've then tried to run it using the following command:

VBoxTray.exe --foreground -vvvvv --only-ipc

which led to the following logs being generated:

C:\Documents and Settings\Bucciantinif>VBoxTray.exe --foreground -vvvvv --only-i
pc

C:\Documents and Settings\Bucciantinif>00:00:00.001953 main     VBoxTray 7.1.7 r
167269 win.x86 (Jan 31 2025 09:26:59) release log
00:00:00.001953 main     Log opened 2025-02-04T09:07:40.469201900Z
00:00:00.001953 main     OS Product: Windows XP Professional
00:00:00.001953 main     OS Release: 5.1.2600
00:00:00.001953 main     Executable: C:\WINDOWS\system32\VBoxTray.exe
00:00:00.001953 main     Process ID: 3164
00:00:00.001953 main     Package type: WINDOWS_32BITS_GENERIC
00:00:00.002929 main     Verbosity level: 5
00:00:00.005859 main     Log group settings are: all=0xf7
00:00:00.006835 main     Verbosity level: 5
00:00:00.053710 main     Showing balloon tip "pre-initializing": Reporting statu
s to host
00:00:00.063476 main     Entering main loop
00:00:00.063476 main     Showing balloon tip "initializing": Reporting status to
 host
00:00:00.064453 main     Starting services ...
00:00:00.065429 main     Skipping starting service 'display' (disabled)
00:00:00.065429 main     Skipping starting service 'clipboard' (disabled)
00:00:00.065429 main     Skipping starting service 'seamless' (disabled)
00:00:00.066406 main     Skipping starting service 'VRDP' (disabled)
00:00:00.066406 main     Starting service 'IPC' ...
00:00:00.068359 main     vbtrIPCInit: Local IPC server now running at "\\.\pipe\
VBoxTrayIPC-Bucciantinif"
00:00:53.627929 IPC      IPC: Worker thread started
00:00:53.628906 IPC      IPC: New client connected with session 0x1d55d8
00:00:53.628906 main     Service 'IPC' started
00:00:53.628906 main     Skipping starting service 'LA' (disabled)
00:00:53.628906 main     Skipping starting service 'draganddrop' (disabled)
00:00:53.628906 main     1/7 service(s) started
00:00:53.629882 main     Showing balloon tip "active/running": Reporting status
to host

and the CPU loop was indeed there:

I've then killed the process and restarted it using the following command:

VBoxTray.exe --disable-ipc

however, to my surprise, this also resulted in the CPU loop:

New video here: ​https://we.tl/t-Lc5Y7mQDXE

[pentagonik]

Very interesting, so with and without the IPC sub service there apparently is high CPU usage ...

Can you try disabling the other sub services (as shown via /help) so figure out what could make this go away on your guest? I really was convinced that this is the IPC sub service ...

[bird]

The last stack you posted with the ipc service disabled can't be spinning in VBoxTray, as it's the main thread creating the service threads. So, my guess is that it's spinning in the kernel.

Going back to the stack from comment:31, it looks like you may have some antivirus software installed (I don't recognizes aswArPot.sys & aswSnx.sys as anything shipping with windows). What exactly is it that you've installed (product & version, please)?

[Francesco Bucciantini]

The antivirus is Avast, in particular Avast Premier version 8.8.2356 build 18.8.4084.409. The security definitions are updated on a daily basis (currently 250204-6).

I already thought about the antivirus before opening this ticket months ago, which I why I did set C:\WINDOWS\system32\VBoxTray.exe as an exclusion in all shields back then and it has been there ever since.

Here are the various tests with VBoxTray.exe started from the command line in the various configurations:

VBoxTray.exe --foreground -vvvvv --only-display

VBoxTray.exe --foreground -vvvvv --only-clipboard

VBoxTray.exe --foreground -vvvvv --only-seamless

VBoxTray.exe --foreground -vvvvv --only-VRDP

VBoxTray.exe --foreground -vvvvv --only-IPC

VBoxTray.exe --foreground -vvvvv --only-LA

VBoxTray.exe --foreground -vvvvv --only-draganddrop

New video here: ​https://we.tl/t-uWUgVptxtb

[bird]

You said you excluded VBoxTray.exe way back, but the stacks in comment:31 shows two avast drivers on the stack. The last HAL64G.DLL entry hints at aswArPot.sys doing some an async proceduce call (ASP), but I'm not sure I've got the right symbols here as you clearly have a 128GB patch of some kind installed (details would be helpful).

The last comments doesn't have any kernel stacks, so it hard to say what's going on there or whether it's related to Avast. As I already pointed out, we don't have anything in our code that can loop forever in the comment:34 stacks, which strongly indicates that it's something else that busted, not VBoxTray.exe...

The threadlist for VBoxTray.exe also includes some 0patchLoader.dll thread, which would be nice to know what is...

[Francesco Bucciantini]

After a long and very painful debugging session I was able to confirm that it was indeed Avast. Unfortunately, it doesn't matter that I've put

C:\Program Files\Oracle\* C:\WINDOWS\system32\VBoxTray.exe C:\WINDOWS\system32\VBoxService.exe C:\WINDOWS\system32\VBoxControl.exe

in the exclusions, Avast will still go ahead and inject itself in the threads created by the VBoxTray.exe process and cause the dreaded lockup. I uninstalled Avast and removed internet from the VM (for safety reasons) and just like magic the problem disappeared:

That's VirtualBox 7.1.7 r167389 with the updated guest additions. I left it there running for 1h and there were no signs of the issue, everything worked perfectly. I've reported it to Avast but the support team doesn't seem much keen. I would like to apologise for the use of your time, guys, both bird and pentagonik. You've been with me until the very end and I really appreciate this, I mean it, thank you for everything you've done. From this I take out the positives, namely the fact that we now have command line options in VBoxTray.exe and also the fact that the signature issue was fixed. I still have no idea why Avast would go rogue on the poor threads created by the VBoxTray.exe initialized process nor why it's injecting aswArPot.sys and aswSnx.sys when I've put that process in the exceptions. I have created a full memory dump, let me know if you're interested and I'll send you privately on the forum to avoid any potential leak of private info, otherwise no worries I'll just delete it. Nonetheless, this is just avast acting weird and at this point I have no idea what changed from Virtualbox 7.0.18 r162988 to make it go nut on that poor process, but it's very much an Avast issue at this point, not a Virtualbox one, so I've reported it there even though they don't seem very interested in the report. I know that it's not related, but since you asked, 0Patch is basically a program that can patch other programs in RAM when an official Microsoft patch isn't available and there's a known vulnerability (which would be the case for Windows XP and its derivatives given that the extended support ended in 2019): ​https://0patch.com/ As to the HAL64G.DLL and PAE128 they're basically part of the unlocked PAE which can be selected at boot time when PAE is enabled to use more than 4GB of RAM in a 32bit system. They've never caused any issues. By default PAE is disabled on XP and its derivatives because Microsoft didn't think drivers were stable enough to work in PAE mode and they only really enabled it on Windows Server 2003 as they thought that server hardware was gonna be tested to work with PAE friendly drivers, however it was possible to enable it on other variants like Professional since SP1. The VM has 12GB of RAM from the host assigned and it recognizes them all perfectly. I know it's not related, but a little bit of history and quirks about XP. :)

Anyway, once again, thank you so much for all the effort and dedication you've put on this. I just hope that Avast is gonna fix this behavior, but my expectations are very low... :(

[Francesco Bucciantini]

I've submitted a whitelisting request to the Avast Team. I attached the latest Guest Addition ISO along with the following message:

Avast is causing issues with Virtualbox versions newer than 7.0.18 r162988 when Guest Additions are installed. Although the Guest Additions themselves, when scanned, don't show up in a report, Avast is injecting itself in the threads created by the VBoxTray.exe process and it causes them to go into an infinite loop. In a 4c/4th VM, you would see 25% usage which means that 1 core is cycling at 100%. Inserting the C:\WINDOWS\system32\VBoxTray.exe under the exception in all shields doesn't help and if a stack trace is performed we can see that aswArPot.sys and aswSnx.sys are there. To confirm that this is the case, uninstalling avast makes the VBoxTray.exe process execute normally with all its threads. There's nothing from the Virtualbox side that could cause any lockup, so this needs to be addressed on the Avast side. Please release a new definitions update in which the Virtualbox Guest Additions are whitelisted so that they won't cause an issue on Windows Guests.

Now we wait, I guess.

[galitsyn]

Hello,

We just released VirtualBox 7.1.8 which is available on the Downloads page. This issue should be fixed in this version.