Fun and games with whacked out video modes in XP guest / Linux host

[Michael H. Warfield]

This was mentioned in the comments of the (now closed) 3096 ticket.

​http://www.virtualbox.org/ticket/3096

This is not fixed in 2.1.4 as the 3096 problem was, so this is a new ticket, and I have more information and some potential workarounds.

This was my comment from there:

The problem I have may be related. Sometime between late December and now, my Windows XP virtual guests w/ the guest additions got stuck at 800x600 max 4 bit color (16 colors) max. Symptoms are very similar to the original poster. Crappy screen, random (flashy) updates, poor resolution. It initially comes up 640x480 and won't let me easily increase the resolution. Going into desktop display properties and selecting Settings -> Advanced -> Adapter -> "List All Modes" only shows 640x480 and 800x600, both in 16 color mode. Selecting 800x600 gives me a little more resolution but that's all I can get.

I first noticed it after the 1/13/2009 Windows XP updates on one of the system. Rolling XP back to a previous restore point did not resolve the problem and, at that time, I noticed a second virtual guest, which had not been updated with the 1/13 updates, was also exhibiting the same problem. Deleting the vboxvideo driver restored full resolution video through the vesa driver in both. This was with 2.1.0. Installing 2.1.2 and reinstalling the guest additions did not resolve the problem. The vboxvideo driver still gives me 800x600 w/ 4 bit color. Deleting that driver restores it. So, yeah, it's on 2.1.0 and 2.1.2.

I updated VirtualBox from 2.0.6 to 2.1.0 on 12/22 (based on timestamps of the downloaded packages) and it may have started after the guest additions back then, I just didn't notice at the time. Not sure when I updated the guest additions in the individuals VM's.

My host hardware is a Lenovo T60p laptop, which also has an unusual screen resolution of 1400x1050 although I'm generally running it in 1280x1024 for external monitors.

Unfortunately, downgrading the VirtualBox host software to 2.0.6 doesn't seem to be viable for testing and debugging, due to configuration file changes (I get an error about unable to import settings). Too bad more effort isn't put into maintaining compatibility for diagnostic purposes. However, be that as it may, I was able to extract the 2.0.6 guest additions CD from that build and install them into the virtual guests after reinstalling the 2.1.2 host software. That was no help. I was definitely NOT experiencing this problem with 2.0.6 host engine and 2.0.6 guest additions prior to December.

Deleting the driver for the Virtual Box Graphics Display adapter under the hardware device manager then does restore full resolution but it tells me it found new hardware and wants to install the driver I just uninstalled. I have to tell it to stop installation at that point.

Now, I've gone back and created a new XP SP2 machine and applied no updates (just to make sure none of the MS updates since SP2 have any impact and I can eliminate that as a source of the problem) and just installed the VirtualBox guest addons. After rebooting, I found I had 32 bit color but still only 640x480 resolution available from the desktop settings. Under advanced -> show all modes for the adapter, it showed 6 possible modes, 5 in 640 x 480 and only one in 800 x 600 (4 bit).

I thought, maybe the graphics memory. It was set to 12 Meg. So I shut down the vm and upped the video memory to 64 Meg and rebooted. Now I've only got 640 x 480 with 4 bit color and "all modes" only shows 4 modes, 640 x 480 and 800 x 600. Changing back to 12 Meg had no effect.

On a silly lark, I hit "full screen" (host window manager decoration maximize) and was surprised by the answer. It had full 32 bit color (but still only 640 x 480). But when I went back to the desktop properties, I found I could now select all the other higher resolutions. If I drop out of maximize, I've still got 32 bit color but stuck back on 640x480 and I'm back with "All modes" showing only 5 modes available.

If I manually resize the window from the host window decoration handles, I can resize it up to a bit but it doesn't change what the guest thinks (properties still says 640x480 32bit). After a certain point, it won't let me enlarge it any larger and it snaps back like a rubber band

Using the machine mode "full screen" (Host-F) give me a 1280x1024 in the VM and all modes appear properly. If I hit Host-F again, it drops back to the manually sized window with the custom resolution. From there, I can set the resolution from within Windows to any resolution smaller than the custom resolution I had it at and the host window now resizes! Yeah! I believe we have a workaround... Maybe...

Workaround that appears to work for me.

After installing the guest addins...

• Expand the window manually using the host window manager decorations as far as the screen and the window permits.

• Enter full screen mode (Host-F) and verify video resolutions and bit plane depth.

• Exit full screen mode (Host-F).

• Try setting video resolution from within Windows once again.

Unfortunately, this workaround doesn't appear to be sticky. If I shut the VM down and restart it, I'm back at 640x480 with 4 bit color. If I immediately do a Host-F, I'm at full screen with 1280x1024 w/ 32 bit color and another Host-F takes me back to 800x600 w/ 32 bit color, which I can not immediately increase from within Windows. If I use the host window manager handles to expand the window, I find it does now expand and I find more and more modes present in the "All Modes" list box in the guest, up to a certain limit (1232 x 919 for some weird reason).

At this point, I can make it work, at least for the SP2 VM. Less than elegant is the understatement of the year.

It was slightly more complicated on my fully patched fully loaded production XP SP3 system but, after increasing the video memory from 8M to 128Meg and then playing with the bit plane depth within Windows and then doing the Host-F trick, that one worked as well. Seems like it remembers some settings from one to the other.

This seems to be a real complicated interaction between video memory and window resolution from the host to within the XP guest.

The log files are from the XP-SP2 guest system and not the SP3 system.