[vbox-dev] 3.1.2 GA vboxmouse driver for x11 calculates erroneous mouse coordinates on Linux Lenny (64 bit) with server 1.4.2

[Carsten Juttner]

Hi everyone,

I decided to play a bit with Virtualbox using a 64 bit guest on an 
x86_64 Bit host.

Host is running (K)ubuntu Karmic 64 bit and on a whim I used Debian 
Lenny 64 bit (net installer) for the guest.

Virtualbox was latest public svn (OSE) with no modifications.

After installing the official ISO guest additions (3.1.2) and starting 
gdm/kdm I noticed the mouse wasn't working correctly and seemed stuck in 
the bottom right corner.

I couldn't find any hints on what was causing this in the forums and/or 
bug tracker so I decided to hunt it down by building the 
vboxmouse-driver and stepping through the callbacks. I found out that 
the incorrect results came from "xf86ScaleAxis" which is called by 
"VBoxReadInput". This function maps a value in an input range to the 
according one in a given output range. But here it was always clamped to 
the maximum output value even though the input was valid. I then 
analysed the xorg-server sources to see how "xf86ScaleAxis" was implemented.

It turns out that the prototype of xf86ScaleAxis is "extern _X_EXPORT 
int xf86ScaleAxis(int Cx, int Sxhigh, int Sxlow, int Rxhigh, int Rxlow)" 
while Virtualbox uses "xf86ScaleAxis(cx, 0, 
screenInfo.screens[0]->width, 0, 65536)". Carefully checking the input 
parameter names shows that they are obviously used in reverse regarding 
minimum and maximum value.

In theory this should not work but obviously it did so checking the git 
commitlog of the xorg server showed that the "xf86ScaleAxis"-function 
(implemented in xorg-server/hw/xfree86/common/xf86Xinput.c) was broken 
for several years because it got the final clamp check backwards. 
Finally this was fixed in xorg-server commit 
a3a7c12fcf8e4ac1418f9ea53f76091f309a721b on 2008-06-08, preceding 1.6.0. 
Now, Lenny uses server 1.4.2 which predates this change so it shouldn't 
be affected. But the Debian folks backported this patch among others to 
2:1.4.2-9 so that's the reason why it fails to work now even on a 1.4 
server.

Going by the commit log of Virtualbox the newer driver (vboxmouse_15.c) 
was only extended to work for 1.3 and 1.4 by the end of November 2009, 
so it didn't affect people before (and testing the 3.1.0 GAs confirms 
this, they were still working fine). Also, "VBoxReadInput" uses 
xf86ScaleAxis in a conditional way (it's #ifdef'ed and the comment 
reads: "Bug in the 1.4 X server series - conversion_proc was no longer 
called, but the server didn't yet do the conversion itself") so it 
obviously won't get used on later versions of the xorg server.

There is another place in the Virtualbox driver where "xf86ScaleAxis" is 
used ("VBoxConvert") but from the commit I take it that this callback 
will only get used on 1.3 because it obviously is not called on 1.4 (see 
comment in last paragraph) and is no longer necessary for >=1.5.

So things would still be fine if not for Debian backporting the 1.6 
changes to 1.4 which leads to vboxmouse_15.c using the parameters in a 
wrong order for "xf86ScaleAxis" on systems that use this patched version.

Avoiding any further headaches in trying to detect this situation my 
solution was to simply replace "xf86ScaleAxis" by a new function 
"VBoxScale" in vboxmouse_15.c.

In case anyone is interested I have attached this as a patch. It's 
licensed under MIT, in case it's needed.

It also fixes another bug due to missing brackets (the #ifdef lines for 
1.4 were inserted without noticing that the preceding if-condition only 
covers one line due to missing brackets, this will only show up in case 
of an error and even then only on 1.4 servers since for all other cases 
there is still only one line after the if)

Note that I didn't test this on any other server than the Lenny one yet 
(1.4.2-10).


Carsten

-------------- next part --------------
A non-text attachment was scrubbed...
Name: vboxmouse_fix_for_lenny.diff
Type: text/x-patch
Size: 3281 bytes
Desc: not available
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20091228/2813d1e4/attachment.bin>