VMM failure after Minix soft-reboot if VT-x is enabled -> fixed in SVN

[Nicolay]

I observed following problem with:

• VirtualBox version: built from trunk, r28440 (dev version after 3.1.6)
• Host OS: Windows XP SP3
• Guest OS: Minix 3.1.6 (also with trunk version - r6776)
• Host machine: with Intel VT-x support (Core Duo T2300)

Steps to reproduce:

1. Create a new VM with HW extensions enabled.
2. Attach dvd with OS image (image can be retrieved from ​http://www.minix3.org/download/minix_R3.1.6-r6084.iso.bz2).
3. Set dvd to be the primary boot device.
4. Start VM (I used VBoxSDL frontend). Minix boot monitor would be shown providing a set of choices (actually one choice).
5. Hit '1' or wait 10 seconds to select the first boot choice. Boot process should go normally. When login prompt is shown type 'root' and hit Enter. You should now be logged in.
6. Type 'shutdown' in the command prompt. This will bring Minix down and return you to the boot monitor.
7. Type 'boot' in the boot monitor command prompt. VMM will fail with VERR_VMX_INVALID_GUEST_STATE error. After that VM enters illegal state and almost any action toward frontend triggers an assertion.

Expected behavior:

Second and any further boots succeed.

My observations

I did a bit of debugging and found out that error cause is VMX_EXIT_ERR_INVALID_GUEST_STATE, i.e. there is a VM-entry failure due to invalid guest state (one of VM-entry checks failed). The failure seems to occur just after the Minix kernel image is loaded into memory.

Logs

I have attached several logs to the ticket in a zip archive:

1. VBox.log - standard VirtualBox log.
2. 2010-04-20-02-51-26.000-VBoxSDL.exe-1416.log - corresponding log, captured using VBOX_LOG="+SELM.e.f.l;+HWACCM.e.f.l;+VMM.e.f.l;".
3. 2010-04-02-11-13-34.082-VBoxSDL.exe-1300.log.tail - a tail of another similar machine run log, but with logging options VBOX_LOG="+SELM.e.f.l;+HWACCM.e.f.l;" and with R0 logging enabled.

[Nicolay]

VBox logs (see description in the ticket)

[Sander van Leeuwen]

Reproduced.

[Sander van Leeuwen]

Fixed, but you'll have to wait for the next major release.