VirtualBox

Ticket #1548 (closed defect: fixed)

Opened 6 years ago

Last modified 12 months ago

Missing characters using serial port.

Reported by: Teuniz Owned by:
Priority: major Component: uart
Version: VirtualBox 3.1.6 Keywords: Serial port
Cc: Guest type: other
Host type: other

Description

When I use the serial port in Windows XP on a Linux host, I miss characters during receiving. Serial port params are 115K2 8N1 (no flowcontrol).

This happens when an external hardware device is sending a continues datastream to the serial port at the highest baudrate (115K2). It looks likes Virtualbox is not able to handle all the data in time (empty the receivebuffer). When I use a native Linux application to receive the same data at the same baudrate without flowcontrol, no characters are missing. When I use Windows XP without virtualizing, there are no missing characters. The host is OpenSUSE 10.3.

Attachments

serial.log Download (3.9 KB) - added by BeamRider 4 years ago.
Serial log for the problem (both on working host and broken guest)
serial.patch Download (3.9 KB) - added by TKreuzer 4 years ago.
uart_buffer.patch Download (4.8 KB) - added by frank 4 years ago.
Diff against VBox 3.1.6

Change History

comment:1 Changed 6 years ago by frank

  • Component changed from other to uart

comment:2 Changed 5 years ago by TKreuzer

See also #3636 (sorry for duplicate report, maybe someone can mark it as duplicate)

comment:3 Changed 5 years ago by attoparsec

  • Version changed from VirtualBox 1.6.0 to VirtualBox 2.1.4

comment:4 Changed 5 years ago by TKreuzer

As described in bug 3636 I get the problems on Windows XP host (AMD64 x2). This also prevents me from connecting WinDbg to windows running in the VM using a pipe. As a workaround I installed com0com, a virtual com port NULL-modem. It works great and also allows me to connect WinDbg to Windows running in VBox.

comment:5 Changed 5 years ago by fvichot

I was about to file a bug report for exactly that error. In my case, vbox also loses characters when a lot of text is send through the serial port.

My setup for testing is quite simple: using a linux host and a linux guest, I run "cat somefile > /dev/ttyS0" in the guest and "socat /tmp/vbox-pipe - > thefile", /dev/ttyS0 being the virtualized serial port in the guest, and /tmp/vbox-pipe the named socket in which vbox redirects the serial port from the guest. Well after transferring a 2Mbytes file that way (base64 encoded so no risk of control characters interfering), only 1.3Mb made it to the host socket... This works fine using KVM/Qemu.

I only describe my methodology for testing from native linux to virtualized linux, but any combination of linux and windows gives the same results: characters are lost.

I would like to stress that this is very annoying, and I've been obliged to code a small protocol over the serial port connection to do error checking and resend frames that didn't get through properly. This is a pretty costly workaround in terms of man hours... Judging from the forum thread at:  http://forums.virtualbox.org/viewtopic.php?f=1&t=15761&p=67774 , I'm hardly the only one, so this is a real problem. I insist because I've been following this bug for a while, and nothing has happened in over a year.

If you need me to do more testing, I'd be happy to.

comment:6 Changed 5 years ago by Kelledin

I'm seeing the same problem in VBox 2.2.2 (PEUL build), with a Linux guest ttyS0 connected to a local socket on a Centos 5.2 host. Essentially large swatches of serial output (several sequential lines of text, with or without control characters) seem to get randomly dropped.

This is especially annoying, as I'm trying to use VB to set up an automated unit-test for an OS installation process that uses the serial port as the default system console. In fact, this is my primary use-case for proposing VirtualBox as an alternative to a certain nameless but rather popular virtualization package. ;-)

The serial port settings in this case are 19200-8-n-1, which closely matches what our real systems deploy with. I can try it with flow control turned on from the guest side, although that would be deviating from real hardware's standard config.

comment:7 Changed 5 years ago by frank

I assume the correct fix would be to implement buffering which is supported by the 16550. We are currently provide a 16450 emulation. Moving to the 16550 is on our TODO list.

comment:8 Changed 5 years ago by Kelledin

I just tried again, with 9600 and 19200, with CTS/RTS flow control both on and off. If anything, dropping the baud rate or adding flow control seems to make things even worse. I'm experimentally cranking the baud rate up to 57600 to see if that somehow improves things, but I don't have high hopes.

comment:9 Changed 5 years ago by Kelledin

Well, higher speeds haven't worked too well either. I'd have to write off the serial port emulation as virtually useless in the state it's in now.

I'm delving into the code to see what it will take to get robust 16550 emulation going.

comment:10 Changed 5 years ago by jkeil2

I think I observe the same problem on an OpenSolaris host and VirtualBox 2.2.2, when trying to use the kernel debugger in an OpenSolaris guest on the virtual serial port console (and a "socat" binary on the host to get at the serial port data). In most of kmdb's output lots of stuff is missing and is lost.

comment:11 Changed 5 years ago by nitingupta

I'm having similar problem with raw file as serial output. Originally reported in #1023.

The serial output in raw file is garbled when guest logs out at a fast rate (VirtualBox 2.2.4 r47978) Guest: Fedora 10 (kernel: 2.6.29.4-75.fc10.x86_64) Host: Fedora 10 (kernel: 2.6.29.3-60.fc10.x86_64)

Grub command line: console=tty0 console=ttyS0 Also, tried: console=tty0 console=ttyS0,115200

comment:12 Changed 4 years ago by trk

I believe this bug/defect is responsible for my inability to fax from a Windows XP guest on a Linux host?

Faxes receive fine (at 14400) but whenever we try to SEND, the fax program (currently tried winfax, joyfax, XP built-in faxing, Snappyfax and Activefax) sends the page data from 0% to 100% instantly and then the serial port becomes unresponsive from within the guest. The linux host can then reset the port by closing the VirtualBox and using minicom or similiar to send ATZ to the modem.

I have tried setting the baud rate to every speed from 115k to 1.2k without success. I have tried multiple installations of Windows and multiple versions of VirtualBox including the current.

I am about to try using the pipe setting to socat the pipe file to and from /dev/ttyS0 (if bidirectional communication via the pipe is possible in this manner) to see if it can keep up with the output of a fax stream. Or perhaps find a way to throttle the fax programs output to a speed that VirtualBox can keep up with?

comment:13 Changed 4 years ago by sk_buff

I have the same running problem VirtualBox 3.0.12 on Windows Vista.

Up to 132 byte blocks works without problems. More than 132 bytes causes a significant loss of data on the serial line.

The output is checked with a logic analyzer, so no handshaking involved.

comment:14 Changed 4 years ago by GregH

I'm seeing the same thing.

  • Linux host (kernel 2.6.27.7)
  • VBox 3.1.2
  • Windows XP SP3 guest
  • COM1 connected to ttyS0
  • 115,200 baud rate

This works in VMware Workstation (6.5.3) on the same hardware.

comment:15 Changed 4 years ago by BeamRider

Same thing with

  • Win7 x64 host
  • WinXP guest
  • Vbox 3.1.2
  • ANY baud rate and no handshake

See serial dump (with description) attached.

Changed 4 years ago by BeamRider

Serial log for the problem (both on working host and broken guest)

comment:16 Changed 4 years ago by TKreuzer

Ok, finding out what's going wrong took me 15 minutes looking at the code. ;-)

serial is passed on like this: serial_ioport_write: s->lsr &= ~UART_LSR_THRE; Transmitter Holding Register is not empty ... rc = s->pDrvChar->pfnWrite(s->pDrvChar, &ch, 1); Send the byte to the driver ... s->lsr |= UART_LSR_THRE; Transmitter Holding Register is empty

The function does not wait for the char driver to be actually ready, it assumes the driver has magically instantly taken the byte.

But the char driver just buffers it: drvCharWrite: pThis->aSendQueue[idx] = pBuffer[i]; store one byte into the char drivers buffer idx = (idx + 1) & CHAR_MAX_SEND_QUEUE_MASK; advance in ring buffer

The buffer is not checked id it's full.

This gives you an idea why the corruption starts when more than 132 bytes are sent: #define CHAR_MAX_SEND_QUEUE 0x80 #define CHAR_MAX_SEND_QUEUE_MASK 0x7f

A quick hackfix would be increasing the buffer size to an appropruate ammount like 0x800, which should probably work in most scenarios. A real fix would be to set the UART_LSR_THRE bit only after a byte has really been sent by the drvCharSendLoop thread and refusing any action in serial_ioport_write if the bit is not set.

comment:17 Changed 4 years ago by frank

Thanks for this investigation, I will take a look.

comment:18 Changed 4 years ago by nemat

@frank

Any progress with this? Had to switch to vmware only to have a serial port working :(

If it is possible make a private patched driver (even with temporary fix)? Thanks.

comment:19 Changed 4 years ago by nemat

@TKreuzer

I've downloaded sources of VB, made the following changes: 0x80 -> 0x8000 0x7f -> 0x7fff

and recompiled. Still no go with serial port. :(

Is there anything I can do in order to get it work? Thanks.

Changed 4 years ago by TKreuzer

comment:20 Changed 4 years ago by TKreuzer

I have attached a patch. The patch adds a new notification function for lsr changes. Instead of setting LSR_THRE directly after sending a character to the char driver, it will now set it only when being notified by the char send loop after sending a character to the real output. This is a hackfix and will make the char buffer only 1 char big, which is inefficient, but should work. If you want to fix that, you need to check if the buffer is completely full when sending a character and based upon that set or clear the LSR_THRE, I'm just too lazy. I don't even know if the patch compiles, because I didn't bother to setup a build environment.

comment:21 Changed 4 years ago by nemat

@TKreuzer

Thanks, but in short, it didn't work. :(

More details: patch didn't succeed automatically (I've got 3.1.6 sources) and applied it manually. compilation failed (not finding symbols: 's', 'pDrvIns'. Corrected those, compiled successfully. My serial uploader (unfortunately have no source for this) haven't seen the COM port. With stock driver it begins uploading but fails with timeout, I think protocol error there due to buggy driver.

Changed 4 years ago by frank

Diff against VBox 3.1.6

comment:22 Changed 4 years ago by frank

I've added a similar patch against VBox 3.1.6. We notify the serial device if the buffer is full. With a simulated delay when writing the byte in the driver this works fine without any character loss. It is not sure if we can include that patch into the next version as this change might cause regressions.

comment:23 Changed 4 years ago by nemat

@frank

Thanks for the patch. When I've applied it to the stock 3.1.6 source I've got reject in DrvChar.cpp, only last diff, it seems. No idea why it didn't succeed, since I've got VirtualBox-3.1.6-OSE.tar.bz2 file with md5sum 6cb3c8161ad878c2a2732137c1621dc4. Anyway, applied the patch manually and going to test it soon, will inform outcome.

comment:24 Changed 4 years ago by nemat

@frank

It didn't work :( I've got a damn legacy serial downloader which has no source, that's the only problem for virtualbox. I've removed vmware and installed qemu, which serial driver works just fine. But I prefer virtualbox to qemu.

comment:25 Changed 4 years ago by frank

  • Version changed from VirtualBox 2.1.4 to VirtualBox 3.1.6

nemat, didn't work with which guest/host? Can you describe your testcase?

comment:26 Changed 4 years ago by nemat

@frank,

VirtualBox: 3.1.6 Host: Ubuntu 9.10 Guest: Windows XP/SP2 Program: POS serial upploader Symptoms: Serial upploader shows timeout, at different percents. (Guest it sends ~100Kb file with blocks to the POS)

The same setup works with qemu from Ubuntu 9.10 repository.

comment:27 Changed 4 years ago by jpolfer

Observing same defect.

VirtualBox: 3.1.6 Host: Windows XP, Guest: Debian Linux (2.6.24-etchnhalf.1-486). Program: cat command, in-house program.

When using the "Host Device" mode, I am seeing dropped characters (tested B9600 B19200. When using the "Host Pipe" mode, I am not seeing dropped characters after following the instructions in the help docs for Serial Port.

comment:28 Changed 4 years ago by tedm

Hi All,

I'm having a bit of problem with compiling this after applying this patch. I'm getting the following under FreeBSD 8:

/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Serial/DrvChar.cpp:79: error: ISO C++ forbids zero-size array 'RTASSERTVAR' /usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Serial/DrvChar.cpp:79: error: conflicting declaration 'int RTASSERTVAR [0u]' /usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Serial/DrvChar.cpp:79: error: 'RTASSERTVAR' has a previous declaration as 'int RTASSERTVAR [1]' kmk[2]: * usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release/obj/Drivers/Serial/DrvChar.o Error 1 The failing command: @c++ -c -O2 -g -pipe -pedantic -Wall -Wextra -Wno-missing-field-initializers -Wno-unused -Wno-trigraphs -Wno-long-long -Wno-variadic-macros -march=i586 -O2 -mtune=generic -fno-omit-frame-pointer -fno-strict-aliasing -fvisibility=hidden -DVBOX_HAVE_VISIBILITY_HIDDEN -DRT_USE_VISIBILITY_DEFAULT -fvisibility-inlines-hidden -m32 -I/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices -I/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Network/slirp -I/usr/include -I/usr/X11R6/include -I/usr/local/include -I/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/include -I/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release -DVBOX -DVBOX_OSE -DVBOX_WITH_64_BITS_GUESTS -DVBOX_WITH_HARDENING -DRTPATH_APP_PRIVATE=\"/usr/local/share/virtualbox-ose\" -DRTPATH_APP_PRIVATE_ARCH=\"/usr/local/lib/virtualbox\" -DRTPATH_SHARED_LIBS=\"/usr/local/lib/virtualbox\" -DRTPATH_APP_DOCS=\"/usr/local/share/doc/virtualbox-ose\" -DRT_OS_FREEBSD -DFREEBSD -DRT_ARCH_X86 -DX86 -DIN_RING3 -DHC_ARCH_BITS=32 -DGC_ARCH_BITS=64 -DIN_IDE_R3 -DVBOX_WITH_NETFLT -DVBOX_WITH_PDM_ASYNC_COMPLETION -DVBOX_WITH_SCSI -Wp,-MD,/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release/obj/Drivers/Serial/DrvChar.o.dep -Wp,-MT,/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release/obj/Drivers/Serial/DrvChar.o -Wp,-MP -o /usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release/obj/Drivers/Serial/DrvChar.o /usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Serial/DrvChar.cpp kmk[2]: * Waiting for unfinished jobs.... kmk[2]: Leaving directory `/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE' kmk[2]: Entering directory `/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE' kmk[2]: * Exiting with status 2 kmk[1]: * [pass_libraries_this] Error 2 kmk[1]: Leaving directory `/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE' kmk: * [pass_libraries_order] Error 2 * Error code 2

Any suggestions?

comment:29 Changed 4 years ago by tedm

Just a quick update - I got it to compile by commenting out line # 79, it seems to be some sort of check in the code designed to make the build fail if the DRVCHAR struct is modified - which happened because of the insertion of the unit32_t cEntries; modifies the DRVCHAR struct.

comment:30 Changed 4 years ago by frank

Some status: Changing the serial device like described above does help under certain conditions but there are reports that this change breaks serial support for Windows guests attached to a (Windows) host pipe. Therefore this change was disabled on trunk in r28399 and r38400.

comment:31 Changed 4 years ago by tedm

Here's another update on my setup.

This patch definitely does help. Previously I could not get any Windows programs to communicate with the serial port at all, now they can. However, autobauding doesen't appear to work, also the port speed must be set by stty before starting VirtualBox, and the only modem entry in Windows that appears to work is the "standard" modem that locks the speed. The worst problem though is when the program under Windows is closed that is using the serial port, and restarted again (I am using the included hyperterminal under winxp for this) the serial port in Windows stops responding. I have to completely shut down the VirtualBox session to get it working again (and not just tell the client OS to reboot)

Serial support never worked for a windows host pipe for me, either with or without this patch.

comment:32 Changed 4 years ago by nemat

@frank

any progress on this problem?

comment:33 Changed 4 years ago by frank

Only investigation, no solution so far. With a fix like proposed above the serial interface does not loose any character anymore. This causes a regression at least for guests which should be attached to windbg at the host (e.g. some debug Windows kernels). The communication stalls because the guest writes characters to the serial port but doesn't read enough characters from the port. Seems that the guest expects that the serial port drops characters -- as real 16450 hardware would do. But this is what we want to prevent with the patch ...

comment:34 Changed 4 years ago by nemat

@frank

Any estimate when this gets fixed? The above patch didn't work for me and it is PITS using qemu (probably I used to get to virtualbox). If you need some testing, please let me know.

comment:35 Changed 4 years ago by frank

So far no ETA for a fix as we have to do some more investigation. But we are aware of this problem and we are working on it.

comment:36 Changed 4 years ago by frank

I have uploaded two testbuilds, one for Linux  x86, another one for Linux  amd64. These builds contain new code which should improve the behavior regarding lost characters. This can still happen but the probability should be lower. Feedback welcome.

comment:37 Changed 4 years ago by nemat

@frank

I'll test amd64 build and will give you feedback soon. Thanks.

comment:38 Changed 4 years ago by nemat

Ok, here goes feedback.

Seems serial port kinda works for me now. I could upload my program to the terminal. 3 out of 4 it uploaded without timeout, once I got timeout at 93%.

I assume, you are going to include this change into coming 3.2.1 release, eh?

comment:39 Changed 4 years ago by frank

Yes, they will be part of the upcoming release (which will be 3.2.2 according to our version number policy).

comment:40 in reply to: ↑ description ; follow-up: ↓ 41 Changed 4 years ago by capfed76

I've tried dir /s > COM1 from a w2k VM in an XP host in VirtualBox 3.2.4r62467.

I don't lose any characater but output is painfully slow (typewriter like), the same thing tested in the previous VirtualBox version is really fast, this is some kind of regression it seems.

comment:41 in reply to: ↑ 40 Changed 4 years ago by capfed76

Replying to capfed76: I was using vmwaregateway. Now I tried with com0com and this problem doesn't occur, don't know who's at fault here...

comment:42 Changed 4 years ago by cjq

Any ETA on a fix? Still having the same issue with 3.2.6. We are going to have to move to vmware if not fixed in the near future.

comment:43 Changed 4 years ago by drgerafe

Seeing a similar problem here (VirtualBox 3.2.6, Ubuntu 9.10 host, Win2kSP4 guest). Guest Win2k app "sees" GPS device attached to serial port, but proper communication fails due to lost characters. (Had this working well with QEMU on a FreeBSD host not long ago.)

comment:44 Changed 3 years ago by nemat

Using latest VirtualBox, 3.2.10. The problem worsened. 1 out of 5-7 attempts to upload over COM port works. :( Hope someone will fix this eventually and in the near future.

comment:45 Changed 3 years ago by andrewsharpe79

+1

comment:46 Changed 3 years ago by pfaf

Same problem with COM ports. I have a 64 bit Debian Linux Host with a 64bit W2K8 Server Guest and although the physical COM port seems to be working with loopback adapter and the PuTTY terminal program when it is attached to the W2K8 Guest, an application which I must use to communicate with a special purpose tax device (EAFDSS - ΕΑΦΔΣΣ) did not work.

I had to switch to VMWare Server on the same machine since this was an installation for a customer, and all work flawlessly till now. Petty, we had to reinstall the W2K8Srv OS since I could not easily migrate W2K8Srv into VMWare - i got Blue Screen Of Death (bsod) - used CloneZilla for cloning but it would not boot.

I would happily change back to VirtualBox once this bug is fixed, but for now VMWare seems to do it better - at least regarding serial ports that is.

comment:47 Changed 3 years ago by pfaf

Yes, I am using latest version, 3.2.10

comment:48 Changed 3 years ago by jbdemonte

Hi,

I got same problem i think.

HOST : Win 7 64b - Guess : Ubuntu 8.04 i try to send a simple buffer on ttyS0 : echo "<CB><PP><RL><F3><HW3,2><DI><RC779,113>26/11/10 08:00<RL><F2><HW1,1><DI><RC779,209>EDITE LE 26/11/2010 - 17:18<p><CB><PP><RL><F3><HW3,2><DI><RC779,113>26/11/10 08:00<p>" > /dev/ttyS0

some times, the second charaters is lost : <B> instead of <CB>

comment:49 Changed 3 years ago by jbdemonte

i use 3.2.10 too

comment:50 Changed 3 years ago by nemat

Please fix this bug, before releasing a new version.

I don't care for some fancy stuff like 3D, audio, etc under virtualbox, if very basic thing as serial port does not work.

Or do everyone now made obsolete their stuff with serial port and uses usb? (I'm not sure if usb works fully, though)

comment:51 Changed 3 years ago by GregH

I just realized that it's been a year since I first added to this bug. Unfortunate that this hasn't been fixed, as I'm still having to keep VMware around for only this reason. In all other respects, I like virtualbox better.

I understand that most desktop users are more interested in eye-candy, 3D, etc., but there's got to be lots of people like me that work with embedded systems that are still very dependent on RS-232 or RS-485.

As a side-note, as VMware became more and more feature-heavy with each new release, I liked the product less and less. That's what got me to try Virtualbox in the first place. Please don't let VB go down the same path...

comment:52 Changed 3 years ago by frank

Should NOW be much improved (after 4.0.2). The relevant changesets are r35717, r35723, r35725 and r35769. These fixes will be part of the next maintenance release.

comment:53 Changed 3 years ago by Teuniz

Why is this bug still not "assigned" ? (see top of the page)

comment:54 Changed 3 years ago by frank

We don't assign bugs in general.

comment:55 Changed 3 years ago by Teuniz

I assume the correct fix would be to implement buffering which is supported by the 16550. We are currently provide a 16450 emulation. Moving to the 16550 is on our TODO list.

Frank, when will this be done? Many users are still using a hardware-implemented serial port, specially in engineering/industry like I do.

comment:56 Changed 3 years ago by frank

The 16550 emulation was introduced with VBox 3.2.2.

comment:57 Changed 3 years ago by Teuniz

Thank you for your quick reply Frank.

Should NOW be much improved (after 4.0.2). The relevant changesets are r35717, r35723, r35725 and r35769. These fixes will be part of the next maintenance release.

Last question, can you tell me when the next release will be?

comment:58 Changed 3 years ago by frank

Sorry, we do not make release dates public, but as usual, the next maintenance release will be released in weeks, not months.

comment:59 Changed 3 years ago by frank

  • Status changed from new to closed
  • Resolution set to fixed

The character loss with the serial interface should be fixed in VBox 4.0.4. For any other issues, please open separate defects.

comment:60 Changed 3 years ago by GregH

Thanks for your efforts Frank. For those who haven't upgraded yet, 4.0.4 solved my problem, and I require baud rates up to 115,200.

Goodbye VMware!

comment:61 Changed 3 years ago by mTi3Ad9vN

V4.0.4 has greatly improved Windbg performance when debugging VMs over a virtual serial port. Thank you, VBox team!

comment:62 Changed 12 months ago by Neofit

Smth similar on VirtualBox 4.2.6 build 82870: https://www.virtualbox.org/ticket/11768

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use