WDDM driver does not validate custom video resolutions, causing graphics freeze on boot

[machfour]

See investigation on ​this forum post

For some reason (possibly the tiling WM resize bug on Linux) the custom/transient display resolution can be set to a very small size (e.g. 400x191) and this small size is saved to the registry in Windows. Once the WDDM driver is installed, Windows will try to add this mode to the target modes in the VidPn framework. This fails and the boot sequence appears to freeze on the Windows splash screen, rendering the VM unusable.

A workaround is to install the Guest additions in safe mode, then edit the registry keys storing the custom resolution to be a larger size, e.g. 800x600. See the attached image.

Reproduced on both OSE and non-OSE versions of Virtualbox, 5.1.22 with corresponding extpack and Guest Additions ISO installed.

I have attached the boot log when the problem occurs.

[machfour]

Log from clean boot/poweroff sequence when problem occurs

[machfour]

Small Custom{X,YRes and CustomYRes values in WDDM registry storage causing graphics freezes

[machfour]

Small Custom{X,Y}Res values in WDDM registry storage cause graphics freezes

[machfour]

(Sorry, I accidentally attached the regedit.png twice while editing the description. The first one can be deleted)

[machfour]

The ticket and related discussion about the resizing issue in tiling WMs is found at #15863.

[Dmitrii Grigorev]

Recent 5.2 Guest Additions should have fix for the discussed problem. Please upgrade GA in your Windows guest and if the problem will persist - attach VBox.log here.

The download link for your convinience: ​http://download.virtualbox.org/virtualbox/5.2.0_RC1/