Video corruption with Unscaled HiDPI Output - grey rectangles cover updated regions & follow mouse

[adaugherity]

This is a regression with 5.1.2 that did not occur with 5.0.26. Host is VirtualBox 5.1.2 on OS X 10.11.6, iMac Retina 5K late 2015.

With "Unscaled HiDPI output" enabled, the areas of the screen being updated are overwritten with grey rectangles. On a Windows 10 guest, this follows the mouse pointer (see video), as well as creating larger grey rectangles in updated fields. On an Ubuntu 16.04 guest, I see this on the login screen, but not very much within the desktop. However on the text console (Ctrl+Alt+F1) any typing or cursor movement is replaced by grey rectangles.

For both guests, minimizing the VM window and unminimzing it will temporarily restore the screen, until something else is updated, as will clicking the close button to bring up the save state/ACPI shutdown/force off menu, which dims the guest window while the menu is up, and also temporarily "fixes" the screen. Both guests have the 5.1.2 guest additions installed.

This behavior is not affected by enabling/disabling 3D support or 2D video acceleration (it happens whenever Unscaled HiDPI is enabled, regardless of 3D or 2D accel settings, although due to #14349/#14231, I have to disable 3D in the Windows 10 guest to use HiDPI; that was reported for 5.0.x and is still a problem in 5.1.2).

[adaugherity]

VBox.log for Win10 guest

[adaugherity]

Video of grey streaking following mouse pointer:  https://youtu.be/PDrMRspWiys

[Jisakiel]

+1 on a retina macbook late 2013, 10.11.6.

[xyzzy]

Same issue on Macbook Pro retina, early '15, OS X 10.10.5; Virtualbox 5.1.4; linux guest.

[kingds]

Same issue on Macbook Retina, 12 inch, early 2016, OS X 10.11.5, VirtualBox 5.1.4, Windows 10 guest. Big black boxes appear anywhere there is movement on screen.

[terataki]

The problem is not only with Windows, but also with Linux guests. There the gray area is not permanent, but after a few seconds it gets redrawn, which makes the display very tiring to look at.

[Ben2D2]

This looks very similar to a bug I've been seeing, which is a regression as of 5.1: https://www.virtualbox.org/ticket/15646

I can no longer use unscaled hidpi output for any VMs, or at least not windows 10 or linux.

[Wolfgang Glas]

Same issue on Macbook Pro Retina, 12 inch, early 2015, OS X 10.11.6, VirtualBox 5.1.8, debian jessie Linux guest. Big grey boxes appear anywhere there is movement on screen, after a few seconds a redraw is issued, unscaled HiDPI completely unusable.

How can it happen, that this major 5.1 regression is still present in 5.1.8 with no comment from Oracle and no tentative "fix version" :-/

This regression affects various guest systems, so please consider a fix.

[Leonid Dorogin]

The same problem on MacBook Pro Retina, 15 inch, mid 2015 with macOS Sierra 10.12.1, VirtualBox 5.1.8. The guest machine is Arch Linux with the standard kernel and the guest modules 5.1.8 installed ("virtualbox-guest-modules-arch", "virtualbox-guest-utils"). When switching on the "unscaled HiDPI output" in the virtual machine settings, the screen is being filled with gray rectangles under moving mouse pointer.

[Wolfgang Glas]

...still no response from Oracle's VirtualBox Team :(

[michael]

Sorry, I'm afraid that at present no knowledgeable developer has time to look at this, and at the moment there is no free developer time on the horizon for it either. (That might change, but then again might not.) I think that for now your best bet is finding someone outside of the core team with the skills and motivation to investigate and fix it.

[Dsen]

Hello guys,
could you please try the following test-build:
https://www.virtualbox.org/download/testcase/VirtualBox-5.1.9-111785-OSX.dmg

[alexn]

Replying to Dsen:

Hello guys,
could you please try the following test-build:
https://www.virtualbox.org/download/testcase/VirtualBox-5.1.9-111785-OSX.dmg

Thank you for this build, it works. Grey rectangles gone. However still problems with mouse positioning (hot spot is not at the tip of the mouse pointer when mouse integration os on). This build is not compatible with latest Extension Pack (cannot start the MV if Ext.Pack is installed). And the overall graphic performance is very very weak :(

[Dsen]

Replying to alexn:

Replying to Dsen:

Hello guys,
could you please try the following test-build:
https://www.virtualbox.org/download/testcase/VirtualBox-5.1.9-111785-OSX.dmg

Thank you for this build, it works. Grey rectangles gone. However still problems with mouse positioning (hot spot is not at the tip of the mouse pointer when mouse integration os on).

Thanks.
Could you please check whether the mouse positioning issue is gone with a Mac test-build from the test build page: https://www.virtualbox.org/wiki/Testbuilds

[adaugherity]

Seems fixed, thanks! (In 111785, the mouse pointer was off in my Ubuntu VM, but my Windows 10 VM was fine. With 111830 the mouse is now correct in Ubuntu.)

Still have to disable 3D in the Windows VM (#14349), but this grey rectangles problem seems fixed in the test build.

[Guilala]

For me, those glitching rectangles were black, and 5.1.9-111785-OSX solved it for me to. Thanks very much for solving this :-)!!

[Wolfgang Glas]

Thanks for fixing this, unfortunately I don't have a test system with HiDPI, but other users were eager to try the test builds ;-)

[bwildenhain]

Replying to Dsen:

Hello guys,
could you please try the following test-build:
https://www.virtualbox.org/download/testcase/VirtualBox-5.1.9-111785-OSX.dmg

I am having similar problems using Debian GNU/Linux as a host system. Could you provide a patch or your modified sourcecode ?

[Dsen]

Replying to bwildenhain:

Replying to Dsen:

Hello guys,
could you please try the following test-build:
https://www.virtualbox.org/download/testcase/VirtualBox-5.1.9-111785-OSX.dmg

I am having similar problems using Debian GNU/Linux as a host system. Could you provide a patch or your modified sourcecode ?

Hello,
Unfortunately (or fortunately :D) "Unscaled HiDPI Output" is the Mac OS X (Retina) specific feature, unrelated to Linux host. Which means you are experiencing another issue, could you please create another defect for that purpose?

[frank]

Fix is part of VBox 5.1.10.