VM lockup when serial port is enabled in file mode to /dev/null

[Thomas Dreibholz]

My VM has COM1 enabled, with data to file "/dev/null". It boots the RELEASE-12.1 FreeBSD installer CD (i.e.  https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.1/FreeBSD-12.1-RELEASE-amd64-disc1.iso). The VM locks up during boot, with high CPU utilisation. Turning the COM1 port off, everything is okay.

Using a file (e.g. /tmp/test1.txt) for output of COM1 works as well.

[Thomas Dreibholz]

Output of "vboxmanage showvminfo -machinereadable" for the VM

[Thomas Dreibholz]

With some more testing, I got a reproducible scenario where the VirtualBox process locks up (not just the VM), i.e. the GUI is not reacting any more:

• COM1 in VM configuration: enabled, but disconnected
• Booting of a Ubuntu 20.04 VM, with additional kernel cmdline parameters: console=tty0 console=ttyS0,115200n8 (the idea is to have a normal console, as well as an additional console over COM1)
• VM locks up during boot
• Forcing VirtualBox to reset the VM via the VM's GUI
• The VirtualBox GUI locks up as well

[Thomas Dreibholz]

Note, I am using VirtualBox Version 6.1.12 r139181 (Qt5.12.8), i.e. the latest version from the  https://www.virtualbox.org website.

[Thomas Dreibholz]

Configuration of the Ubuntu VM

[aeichner]

Not reproducible here, please try the latest 6.1 testbuild from Testbuilds, it contains some serial fixes. Please attach a VBox.log of the affected VM in case it still is reproducible.

[MWutzke]

I am seeing similar lockups when using the 'Serial Port' option 'enabled ; disconnected'.

The key to the reproduction is that the issue is only seen when using 2 or more vCPU's for the VirtualBox machine instance. Using a single vCPU, the issue is not observed.

This issue is observed with a custom Linux kernel we generate, but I have also been able to reproduce it with the Ubuntu 20.10 live-server ISO as well when using either of these 2 kernel options:

• linux /casper/vmlinuz console=ttyS0,115200 console=tty1,0 --
• linux /casper/vmlinuz console=tty1,0 console=ttyS0,115200 --

When using the first option, the lockup occurs (generally) once the initial Language selection dialog occurs on the VGA. I suspect this is the same time that the user is prompted to select the interface style on the (disconnected) serial port.

I have reproduced this issue with on both Windows10 and Ubuntu 20.04 hosts as follows:

• Windows 10 host: VirtualBox 2.1.16, 2.1.18
• Ubuntu 20.04 host: VirtualBox 2.1.10, 2.1.19 (testing 142299)

Using the same machine with VirtualBox 5.2 or 6.0.24, and multiple vCPU's, the issue is not observed.

In addition to the VirtualBoxVM lockup, when using the VirtualBox VM debugger, the debugger locks up as well, and becomes unresponsive.

[aeichner]

The lock up in disconnected mode should be fixed in the next maintenance release. You can try the latest testbuild from here.

[MWutzke]

I've just tried 6.1.19-142777-Linux_amd64, and my initial testing shows that this is resolved. Thanks for addressing this.