Auto-resize Guest Display disabled with VMSVGA

[Sam Morris]

Now that VMSVGA is the default for Linux guests, I have changed my VM over to using it.

I have found that the 'audio-resize guest display' option is unavailable.

Some error messages can be obtained from VBoxClient with:

$ VBoxClient -v -f --vmsvga
VMWARE's ctrl extension is available. Major Opcode is 128.
Failed to register resizing support, rc=VERR_INVALID_FUNCTION
Failed to get display change request, rc=VERR_ACCESS_DENIED

Corresponding kernel messages:

Aug 03 14:15:54 kernel: Service: Display SVGA X11
Aug 03 14:15:54 kernel: VMWARE's ctrl extension is available. Major Opcode is 128.
Aug 03 14:15:54 kernel: Failed to register resizing support, rc=VERR_INVALID_FUNCTION
Aug 03 14:15:54 kernel: Denying userspace vmm call type 0x000051
Aug 03 14:15:54 kernel: Failed to get display change request, rc=VERR_ACCESS_DENIED

It's not clear whether this option is not supported under VMSVGA or not. If not then that should be documented somewhere.

I'm seeing this with VirtualBox 6.1.12 but that option isn't available in the bug tracker(!)

There are a couple of similar bugs:

[Sam Morris]

https://www.virtualbox.org/ticket/19635 - same with VBoxSVGA

https://www.virtualbox.org/ticket/19418 - does not state which graphics adapter is in use

[Frank Batschulat (Oracle)]

Which Linux distro? Which Linux kernel version? Please.

[Sam Morris]

Debian testing

Kernel: Linux version 5.7.0-2-amd64 (debian-kernel@…) (gcc version 9.3.0 (Debian 9.3.0-16), GNU ld (GNU Binutils for Debian) 2.35) #1 SMP Debian 5.7.10-1 (2020-07-26)

Guest additions version installed: 6.1.12-dfsg-8

[Sam Morris]

The 'failed to register resizing support' error is preceeded by these system calls:

ioctl(3, VBG_IOCTL_CHANGE_FILTER_MASK, 0x7fff8ca697e0) = 0
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x56, 0xd, 0x24), 0x7fff8ca697d0) = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(3, VT_RESIZE, 0x559bd26a5940)     = 0

fd 3 is /dev/vboxuser.

[Sam Morris]

BTW the vboxguest kernel module I'm using is the one from the Linux kernel. Not the one shipped by the guest additions.

[stevew]

I'm seeing the same or very similar behavior with a Linux Host. I've gotten the permission denied errors once(I think with the .iso packages) using native packages I'm getting this:

vm1:~$ VBoxClient --version
6.1.10_Ubuntur138449

m1:~$ VBoxClient -f -v --vmsvga
VMWARE's ctrl extension is available. Major Opcode is 128.
An X Window protocol error occurred: BadValue (integer parameter out of range for operation) (error code 2).  Request code: 128, minor code: 2, serial number: 14
Resizing frame buffer to 1533 765 has failed
RRScreenChangeNotify event received
An X Window protocol error occurred: BadValue (integer parameter out of range for operation) (error code 2).  Request code: 128, minor code: 2, serial number: 28
Resizing frame buffer to 1533 765 has failed

This also hangs the X server until the VBoxClient is killed. Results are the same for DKMS or Kernel modules in the guest.

Guest: 
Linux vm1 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 
GNU/Linux
ii  virtualbox-guest-dkms                      6.1.10-dfsg-1~ubuntu1.20.04.1       all          x86 virtualization solution - guest addition module source for dkms
ii  virtualbox-guest-utils                     6.1.10-dfsg-1~ubuntu1.20.04.1       amd64        x86 virtualization solution - non-X11 guest utilities
ii  virtualbox-guest-x11                       6.1.10-dfsg-1~ubuntu1.20.04.1       amd64        x86 virtualization solution - X11 guest utilities

Host:

Linux pc1 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

ii  virtualbox-6.1                         6.1.12-139181~Ubuntu~bionic                                                 amd64        Oracle VM VirtualBox

[Sam Morris]

Based on ​https://github.com/torvalds/linux/blob/00e4db51259a5f936fec1424b884f029479d3981/drivers/virt/vboxguest/vboxguest_core.c#L1334, req->request_type is 0x51 or 81 in decimal.

In ​https://github.com/torvalds/linux/blob/ecfd7940b8641da6e41ca94eba36876dc2ba827b/include/uapi/linux/vbox_vmmdev_types.h#L22, 81 corresponds to VMMDEVREQ_GET_DISPLAY_CHANGE_REQ_MULTI.

That means that this case should be chosen: ​https://github.com/torvalds/linux/blob/00e4db51259a5f936fec1424b884f029479d3981/drivers/virt/vboxguest/vboxguest_core.c#L1304

So I don't see how we end up in the default case at ​https://github.com/torvalds/linux/blob/00e4db51259a5f936fec1424b884f029479d3981/drivers/virt/vboxguest/vboxguest_core.c#L1334...

[Sam Morris]

Anyway. I switched to the vboxguest kernel module provided by VirtualBox rather than Linux and auto-resizing works again!

So I think it should be documented prominently in the manual that full video support (and perhaps other features?) requires VirtualBox's kernel modules, and that the modules shipped by the Linux kernel are incomplete.

[Frank Batschulat (Oracle)]

fwiw, Hans de Goede from RedHat did fix this in the Linux vboxguest driver with:

​https://github.com/torvalds/linux/commit/412f84cff0c9aa95fd204961f21121f38d77cc13#diff-35c9a7653ea671ab94bed9d02d686ce2
​https://github.com/torvalds/linux/commit/631beddc5466731b048263a4a9d3d67150e72f8d#diff-35c9a7653ea671ab94bed9d02d686ce2
​https://github.com/torvalds/linux/commit/316b0035402f05fe9e9e5334d1ff65dae285cb7c#diff-35c9a7653ea671ab94bed9d02d686ce2

NB: The new ioctls are now supported by the upstream kernel module but this was fixed only recently and the reporter is not using the latest upstream kernel containing the fix.

[Frank Batschulat (Oracle)]

closing per my last comment https://www.virtualbox.org/ticket/19788#comment:10 as this is a problem with the upstream Linux kernel module vboxguest.