Upgrade 5.1.28 to 5.2 headless guest can no longer be accesses, video performace degraded

[drankinatty]

Upgrade from 5.1.28 to 5.2 (Arch x86_64 host) leaves headless Win7 guests unable to capture mouse or keyboard regardless of host-key use until virtualbox is started on host machine and guest launched via gui interface and windows guest addtions updated from 5.1.28 to 5.2.

PUEL from VirtualBox-5.2.0-118431-Linux_amd64.run with Oracle_VM_VirtualBox_Extension_Pack-5.2.0.vbox-extpack

This is unexpected new behavior. The expected result is that following update, the headless client can be started and accessed as normal remotely via rdesktop and guest additions updated from within headless guest as has been the case with all prior 5.1.X updates.

An update to 5.2 on the host will leave Win7 (and probably others) unable to access the headless guests remotely. There is no way to have the mouse or keyboard captured when first accessing a headless Win7 guest remotely after the 5.2 update on the host. The login screen is displayed and visible, but there is no way to interact with it -- with or without host-key use (currently the default rt-control)

When virtualbox is started on the host machine, prior guest additions are ignored and prior setvideomodehint resolution and color depths settings are ignored. The default capture host-key dialog is shown as if no prior guest additions are present.

If the incompatibility between 5.1.2X guest additions and 5.2 is a known issue, this should be clearly stated in the changelog as it impacts remote headless guest access that guest users cannot correct without access to the host machine.

Secondly, after 5.2 guest additions are installed from the host, and the guest and virtualbox shut down, the guest can then be restarted headless and access remotely. setvideomodehint on the host must be reset to restore 32bit display color depth for remote access. Video performance within the headless guest is degraded compared to 5.1.28 with lagging window moves within the Win7 guest and artifacts at the window corners on repaint that were not present in 5.1.28. (this is with 2D accell only, no 3D or aero active)

Please let me know what additional information you may need to help with this bug and I'll be happy to provide it.

[drankinatty]

Linux guests also lose setvideomodehint resolution and color depth on host upgrade to 5.2. I use a specific resolution for headless guests run from the server, e.g.

VBoxManage controlvm arch_1_64 setvideomodehint 1366 864 32

After update the linux guests reverted to the default framebuffer resolution of 1024x768. At default resolution, linux guest experience no degradation of video performance, but when resolution and color depth are set with setvideomodehint, windows headless guests experience a significant slowdown and increased jagged window moves, etc using 5.2 compared to 5.1.28.

[drankinatty]

Is it safe to downgrade from 5.2 to 5.1.30 in the interim, or did 5.2 make changes to any of the config files that are not backwards compatible?

[drankinatty]

Here is another note that may (or may not) help. Downgraded to 5.1.30. dkms failed to unload old and reload new drivers on 5.2 -> 5.1.30 downgrade, just as it failed on the 5.1.28 -> 5.2 upgrade. dkms behavior has been flawless for all 5.1.2x upgrades, but there is a problem, for whatever reason, with dkms on the 5.1.x to 5.2 transition. Headless video performance on 5.1.30 is slightly better than 5.2, but not enough to paint a stark contrast between the two. The big issue is the failure of 5.1.28 guest additions to be compatible enough on the 5.2 upgrade to allow headless window guests access.