Keyboard problems on X11 hosts in version 2.0

[Michael Thayer]

This ticket is for reporting problems with keyboard handling on X11 hosts (currently Linux and Solaris) in VirtualBox 2.0.

[Robert Mitchell]

Log file for unrecognized custom keyboard

[Robert Mitchell]

I use a ​custom dvorak layout that I added to the X11 keyboard database by appending the attached file to /usr/X11R6/share/X11/xkb/symbols/us -- once appended, you can switch to the new layout using:

setxkbmap -model microsoftpro -layout us -variant dvorak-robm

I get "Failed to recognize keyboard mapping" message in the log file (also attached). I'm new to VirtualBox, but (yay open-source!) it looks like VBox/Frontends/VirtualBox/src/linux/keyboard-layouts.h has pre-parsed versions of all of the keyboard layouts and attempt to determine which layout you're using.

Is there a way to use custom layouts? Maybe I'll have to recompile the code with my keymap converted to the format in keyboard-tables.h?

[Robert Mitchell]

custom keymap for X11

[Robert Mitchell]

Sorry, my trac-fu was off. Corrected link: ​http://www.flipturn.org/Dvorak_Keyboard.html

[Michael Thayer]

Does the keyboard actually work correctly in VirtualBox? I would rather not be adding custom layouts to the software.

[Robert Mitchell]

Yes, the keyboard responds and characters appear within the guest OS, but the mapping is wrong. This is confusing to explain, but I'll try to give a few examples:

In my host OS (Linux), my dvorak mapping changes the "QWERTY" on a standard keyboard to "/,.PYF". What I mean is: when I press key labeled "Q" on a standard 101 key keyboard, the "/" character appears; if I press the key labeled "Y", the "F" appears, etc. This is what I want because it's the mapping for my modified dvorak.

In the VirtualBox guest OS, if I were to type the keys labeled "QWERTY" on my keyboard, what appears in the guest OS is "[WERTY". The "Q" has been incorrectly mapped because I'm not using a standard dvorak.

If I switch to the standard X11 dvorak mapping, VirtualBox correctly figures this out due to all the hardcoded mappings in keyboard-layouts.h. Typing "QWERTY" on the physical keyboard responds with "QWERTY" in the guest OS.

I agree that you probably don't want to be adding everyone's custom layouts to the hardcoded list in the VirtualBox source; what I'm wondering is if there's some way to specify the custom layout to VirtualBox through a configuration option?

[Michael Thayer]

I tried out your layout in X using xmodmap and the file dvorak.xmodmap on your webpage. It worked fine here. I have made a couple of modifications to the keyboard code which are not in 2.0.2, so you may have more luck with 2.0.4 when we release it (don't yet know when that will be though).

[Michael Thayer]

Actually, if 2.0.2 doesn't work for you, you will probably have to wait until 2.1, as 2.0.4 will only be getting minor fixes.

[Michael Thayer]

Correction - it works for me with the future 2.0.4 codebase - have you tried 2.0.2?

[Robert Mitchell]

I actually had been using 2.0.2. Interesting about xmodmap; I had switched from the xmodmap approach a while ago in favor of the new style setxkbmap, but maybe it's worth going back.

At any rate, I was also able to work around the problem by making a custom keymap in the guest OS. I'll try 2.0.4 when it comes out to see if only xmodmap works, or if setxkbmap will also work.

Thanks.

[Michael Thayer]

I hate to say it, but the setxkbmap version works too here. (Future 2.1 codebase, not tested with 2.0.4).

[Felix Möller]

I am using VirtualBox as shipped with current openSUSE factory: # rpm -qf which VirtualBox virtualbox-ose-2.0.2-1.18

My hardware is a MacBook on which I am running Linux. The keyboard layout is german mac.

Sadly my | and @ doe not work. They are composed with mac alt (ISO_Level3_Shift) and 7 (|) and l (@).

What do I have to do to get them working. I am using Windows XP and Fedora 9 as guests. I am pretty sure they worked with 1.6.6 ...

[Michael Thayer]

Replying to fm:

What do I have to do to get them working. I am using Windows XP and Fedora 9 as guests. I am pretty sure they worked with 1.6.6 ...

Can you test this with version 2.0.4 when it is released (should be very soon now)? That version will at least produce correct warnings in the log file, instead of the bogus warnings produced by 2.0.2. Thanks!

[Felix Möller]

Hi michael thaks for your answer I tried VirtualBox-2.0.4_38406_openSUSE11-1.i586.rpm today, but my keys still do not work. :-(

How can I debug this further? Any workaround to get the keys in my virtual machine?

[Felix Möller]

On my host the Mac alt is known as ISO_Level3_Shift, running xev on my guest I get Alt_L.

The apple key is known as Super_L to my guest whereas it is Alt_L on my host...

[Michael Thayer]

fm: can you please attach a log file from running your guest with VirtualBox 2.0.4? Thanks.

[Felix Möller]

log of booring winxp

[Felix Möller]

log of booting Fedora 9

[Felix Möller]

michael: sorry for beeing so slow. I hope the logs I attached are the correct ones.

I am using openSUSE 11.1 on a MacBook of early 2006.

fm@macbook:~> rpm -q kernel-pae VirtualBox kernel-pae-2.6.27.4-2.1 VirtualBox-2.0.4_38406_openSUSE11-1

Please just tell me if you need anything more.

[Michael Thayer]

The logs do not show any obvious issues. Are you sure that it is not just a problem of the keyboard mapping you selected in the guest? On a normal German keyboard layout, such as Linux or Windows guests would use, @ and | are obtained using AltGr (ISO_Level3_Shift) + Q and < respectively. Obviously the guest will not realise that you have a MacBook keyboard layout unless you explicitly tell it. Can you check to see whether the key combinations I gave you work?

[Felix Möller]

I am sorry but I do not understand how this is supposed to work for me.

I am using:

macbook:/home/fm # rpm -q kernel-pae virtualbox-ose
kernel-pae-2.6.27.7-4.4
virtualbox-ose-2.0.4-4.15
macbook:/home/fm # uname -a
Linux macbook 2.6.27.7-4-pae #1 SMP 2008-11-25 00:02:37 +0100 i686 i686 i386 GNU/Linux

Key	Host (openSUSE 11.1) xev	Host VirtualBox (Preferences -> Input )	Guest Fedora 10 xev
ctrl	Control_L	Left Ctrl	Control_L
alt	ISO_Level3_Shift	Alt Gr	Alt_L
apple	Alt_L	Left Alt	Super_L

I tried on my Fedora 10 Hosts "Generic 105-key (Intl) PC" with German Layout and "Apple Laptop" with "German Macintosh" layout, but that did not any of the keys ...

[Felix Möller]

I tried to get my text-mode layout correct but that did not work either.

On my host I am using mac-de-latin1-nodeadkeys.map.gz

Loading this on my guest makes Enter to an "8" and backspace an "e".

[Felix Möller]

OK i "fixed" the layout for me with the following entries:

[fm@localhost ~]$ cat .Xmodmap 
keycode 64 = Mode_switch 
! ISO_Level3_Shift
keycode 133 = Alt_L
keycode 14 = 5 percent bracketleft
keycode 15 = 6 ampersand bracketright
keycode 16 = 7 slash bar
keycode 17 = 8 parenleft braceleft
keycode 18 = 9 parenright braceright
keycode 46 = l L at 
keycode 35 = plus asterisk asciitilde

this is certainly not a "solution" but at least I can use my virtual machine now.

[Michael Thayer]

Hm, is that your left Alt key which produces the ISO_Level3_Shift keycode? I would be interested if you could complete your table of keys to X11 keycodes a bit for my personal education. In any case, the keyboard handling has been changed slightly in the VirtualBox development sources, so that e.g. ISO_Level3_Shift on the host is always mapped to right Alt (a.k.a. Alt Gr) in the guest, even if you change the host mapping while VirtualBox is running, so the next major release will most likely solve your issue.

[Michael Thayer]

This has not been backported to the 2.0 line though.

[Felix Möller]

Replying to michael:

Hm, is that your left Alt key which produces the ISO_Level3_Shift keycode?

yes it is.

I would be interested if you could complete your table of keys to X11 keycodes a bit for my personal education.

All other keys can stay as they are, at least I did not hit any other problem yet...

In any case, the keyboard handling has been changed slightly in the VirtualBox development sources, so that e.g. ISO_Level3_Shift on the host is always mapped to right Alt (a.k.a. Alt Gr) in the guest, even if you change the host mapping while VirtualBox is running, so the next major release will most likely solve your issue.

Is it changed in the public svn or do I have to wait for a release? If there are no major changes compared to 2.0.4 I would just recompile the src.rpm...

[Michael Thayer]

The changes are in the public svn. However, the public svn contains a lot of changes compared to 2.0.4. Depending on what you feel comfortable with, you could either try that out, or backport the changes in the keyboard handling (this is a cleanly separated shared library, so it should be feasible - look at src/VBox/Frontends/VirtualBox4/src/X11).

[Felix Möller]

Looking at the spec it seems like rebuilding virtualbox on opensuse is not too much fun.

# Patch0:         vbox-futex.diff  
# Patch1:         %{name}-64issue.diff  
# Patch2:         vbox-kmp-vboxadd.diff  
# Patch3:         vbox-kmp-vboxvfs.diff  
# Patch4:         %{name}-init-scripts.diff  
# Patch6:         virtualbox-gcc43-fixes.diff  
# Patch7:         virtualbox-validate-op-gcc43.diff  
# Patch8:         use-o3-to-workaround-gcc-ice.diff  
# Patch9:         virtualbox-system-yasm.diff  
# Patch10:        vbox-2.6.25  
# Patch11:        vbox-vboxfs-2.6.25  
# Patch12:        vbox-buildfix  
# Patch13:        virtualbox-use-intree-yasm.diff  
# Patch14:        vbox-kbuild_unit_paths.diff  
# Patch15:        vbox-configure_python26.diff  
# Patch16:        virtualbox-ose-disable-updates.diff

I am just doing an SVN checkout but not sure how for I get, as I do not have too much time ... Is there no place to get daily builds?

[Frank Mehnert]

There are no daily builds and there will no be any daily builds. The reason is that we already get enough bug reports for our regular releases. Please wait until the next release.

[Robert Mitchell]

I'm the original reporter of this bug, for posterity's sake I thought I'd report that I discovered the cause of my problem: it is because I use loadkeys to map my custom Dvorak keyboard in the linux console before I start the X server.

[Michael Thayer]

For my personal education (with possible benefits for VirtualBox users :) ), why was this causing problems? Does loadkeys change the way X maps key codes to physical keys?

[Robert Mitchell]

Customized Dvorak layout showing mismapping in VirtualBox

[Robert Mitchell]

It turns out that loadkeys is only a factor when using the XkbDisable option in xorg.conf, and I expect that is an uncommon option in linux distros. I removed the XkbDisable option from my xorg.conf to be more in line with typical linux installs but I was able to find another way to duplicate the problem.

If you load the custom xmodmap (I just attached it to this bug) using xmodmap custom-dvorak.xmodmap before starting VirtualBox, you can see the issue.

Inside the guest OS (for my tests I used both XP and Vista) if you load up notepad and type qwerty notepad instead displays [werty. There are a few other mis-mappings: ' instead shows q, [ produces -, and others.

[Michael Thayer]

Interesting - I will give it a try, but that should actually work. It might be of course (and your VBox log should confirm this if it is indeed the case) that your keyboard layout was not detected properly due to other reasons. In particular, up until and including version 3.0.4, any keyboards lacking or remapping certain standard keys (any of left ctrl, left shift, caps lock, tab, escape, enter, up, down, left, right and F1 to F8) would not be properly detected. Quite rare, but for instance some Mac and I think some Sun keyboards lack one of these keys. This particular bug should be fixed in the 3.0.6 beta.

[Michael Thayer]

Indeed, it works fine here.

[Robert Mitchell]

Thanks for that suggestion. Separately from the dvorak mapping, I was remapping Caps Lock into another Ctrl because I don't like Caps Lock. It looks like that was the reason the keyboard was improperly detected, not the xmodmap stuff for my custom dvorak. Removing that Caps Lock/Ctrl remapping allowed 3.0.4 to properly set up the keyboard even in my modified dvorak. The keyboard in the virtual machine shows the proper mapping.

I'll try 3.0.6 to see if it provides the correct mapping even if I use the Caps Lock/Ctrl remapping.

[Robert Mitchell]

I tried 3.0.6-beta1 and found if I remap caps lock to ctrl, the keyboard is not recognized in the VBox.log when using my custom dvorak xmodmap. I tried 4 test cases using 4 different .xinitrc files, restarting the X server each time to make sure I wasn't leaving any cruft from the last test.

Test 1: (the null test)

No .xinitrc, so no remapping. Standard qwerty X11 keyboard, Caps Lock present.

This functions correctly in the guest os. The keyboard is recognized in the VBox.log; qwerty maps to qwerty

Test 2:

Using the .xinitrc to provide a standard qwerty X11 keyboard, but remapping caps lock to left ctrl

xmodmap -e 'remove Lock = Caps_Lock'
xmodmap -e 'keysym Caps_Lock = Control_L'
xmodmap -e 'add Control = Control_L'

functions correctly in the guest os; qwerty maps to qwerty. However, the keyboard is unrecognized in the VBox.log.

Test 3:

Using this .xinitrc to provide my custom dvorak X11 keyboard but leaving caps lock alone

xmodmap custom-dvorak.xmodmap

also functions correctly in the guest os; qwerty maps to qwerty, but the keyboard is still unrecognized in the VBox.log.

Test 4:

This test fails. Using this .xinitrc to provide my custom dvorak X11 keyboard and remapping caps lock to left ctrl

xmodmap custom-dvorak.xmodmap
xmodmap -e 'remove Lock = Caps_Lock'
xmodmap -e 'keysym Caps_Lock = Control_L'
xmodmap -e 'add Control = Control_L'

results in breakage in the guest os. qwerty maps to [werty, and the keyboard is still unrecognized in the VBox.log

This is obviously a corner case, and you may or may not feel it's not worth addressing. If you'd like I can open another bug that restates this comment.

[Michael Thayer]

This is expected. We use two algorithms for deciding which scan codes to map to which X11 keysyms. One of these tries to detect a known X11 keysym mapping by checking the keysyms of certain keys (as commented above) to see if they match known values. When you remap caps lock to control, this algorithm is thrown off (we do check whether caps lock and control are reversed, to make old unix people happy, but that doesn't work in your case of course).

The other algorithm looks at the way the layout dependant keys on the keyboard are mapped, to see if it matches a known national layout. This obviously fails with your custom layout. And when you combine caps lock remapping with your custom layout, both algorithms fail, and VirtualBox makes a best effort attempt.

Some time we may use the Xkb extension as a third way of determining the keyboard setup, but as things currently work for almost everyone this is not very high priority I'm afraid.

[Robert Mitchell]

No worries; I have a workaround. With some trial and error, I created a custom keymap in the guest OS.

If I get some free time, I'll download the current svn and check if adding a clause in X11DRV_InitKeyboardByType fixes my problem:

                || (   (XKeysymToKeycode(display, XK_Caps_Lock) == main_keyboard_type_list[i].lctrl)
                    && (XKeysymToKeycode(display, XK_Control_L) == main_keyboard_type_list[i].lctrl)

although even if that works I don't know if it would be an acceptable inclusion for svn since AFAIK I'm the only one with this issue. :)