[vbox-dev] Binary client with custom IFramebuffer with 6.1.38

[Rūdolfs Bundulis]

Hi,

I had a piece of software using VirtualBox 5.2 that was able to start
a VM and inject itself via the IFramebuffer interface and capture the
displayed bitmap. Due to hardening (even though it was so long ago
that now I am not sure if 5.2 had hardening) I was chowning the
binaries with root and adding the sticky bit and that seemed to work.

Today I tried to port this to 6.1.38 but sadly when doing the chown +
setuid by root it does not work - I seem to be unable to obtain the
VirtualBox API (w/o the chown + setuid thing I get the expected error:

"VirtualBox kernel driver not accessible, permission problem.
Re-install VirtualBox. If you are building it yourself, you  should
make sure it installed correctly and that the setuid bit is set on the
executables calling VMR3Create."

When doing the setuid + chown thing I notice that the VBoxSVC is now
started as root, the /tmp/.vbox-root-ipc seems to be there, what else
should I look at to understand what is failing? Does the root setuid +
chown combo somehow disable the client validation by the runtime or is
this expected and only signed Oracle clients will work? Since sudo
VirtualBox works fine and can start a machien. Is such a setup even
possbile with VirtualBox 6.1.38? Or is my only option here building
w/o hardening?