VirtualBox Guru Meditation crash with grub2, EFI, and grub digital signatures

[JamesJohnston]

The grub2 bootloader crashes VirtualBox with a "Guru Meditation" critical error if EFI is used and digital signatures are enabled within grub.

Complete steps to reproduce this crash from a virtual machine that is made from scratch are below:

Steps to reproduce:

1. Open VirtualBox 5.0.15 r105747 test build on 64-bit Windows 7 SP1. (I have Core i5-2520M CPU with VT-x, EPT, and VT-d on ThinkPad T420.) Problem also happens with VBox 5.0.14.
2. Click the New button to make a new virtual machine with these settings:

• Name: GrubTest
• Type: Linux
• Version: Ubuntu (64-bit)
• Memory size: 4096 MB
• Virtual disk size: 6 GB, dynamically allocated, VDI file type

3. Go to the Settings of the new virtual machine; set these settings:

• Important: System: set "Enable EFI" to enable the EFI firmware. (Maybe it also happens on non-EFI BIOS, but I haven't tested).
• Storage: set DVD drive to ubuntu-15.10-server-amd64.iso (SHA-1 478446F33E336A98D5868C0B353D0D347E8A7262)
• NOTE: You can get this Ubuntu Server 64-bit installer from: ​http://releases.ubuntu.com/15.10/ubuntu-15.10-server-amd64.iso
• Network: set as you need to get on the Internet; I left mine with default NAT settings.
• USB: USB 1.1 controller enabled (not 2.0, but problem also was reproduced on VBox 5.0.14 stable with USB2 enabled).

4. Start the VM. The Ubuntu installer starts. Use default settings in the Ubuntu installer, except where noted:

• Hostname: ubuntu
• User full name: James Johnston
• Username: james
• Password: password
• Encrypt home dir: no
• Time zone: set as appropriate; mine defaulted to America/Chicago and I left it there.
• Partition disks: To simplify things, I changed default to "Guided - use entire disk" (i.e. not use LVM). Default partitioning for me was EFI system partition, ext4, and swap.
• Automatic updates: No automatic updates
• Software selection: No items selected
• Reboot when prompted by the installer. (The CD is automatically ejected)

5. Log into your new ubuntu system: james / password - then run:

   # Generate new GPG key as root (go to root shell first)
   sudo su
   gpg --gen-key
   # Press Enter; use defaults - except pick 1024 bit RSA key size (it will be faster)
   # Make up some information for the Real Name and e-mail when prompted.
   # Press Enter twice when prompted for passphrase: this means no passphrase.
   

6. The virtual machine doesn't have much entropy for key generation. To speed up GPG key generation, press Alt+F2 to go to second terminal. Log in again, and make some entropy:

   sudo su
   while true ; do echo -n . ; echo 3 > /proc/sys/vm/drop_caches ; ls / -R > /dev/nul ; done
   

Use Alt+F1 to go back to monitoring GPG key generation. When it finishes, Alt+F2 and then Ctrl+C to abort the above operation (it makes your computer slow).

7. Now you have a GPG key. Sign some grub files with it and make grub use it:

   # Go to root shell
   sudo su
   # Extract public key and install grub that uses it:
   gpg --output grubkey --export
   grub-install --pubkey=grubkey --modules=verify --no-uefi-secure-boot
   # Sign everything in /boot
   cd /boot
   find -type f -print0 | xargs -0 -n 1 gpg --batch --detach-sign --yes
   

8. Reboot the VM:

   sudo reboot
   

9. Actual behavior: When the VM reboots and attempts to start grub, I get a VirtualBox - Guru Meditation:

A critical error has occurred while running the virtual machine and the machine execution has been stopped.

Expected behavior: The VM boots into grub as usual, not crash the VM.

[Frank Mehnert]

The Guru Meditation is about a triple fault. A triple fault is normally an error condition which is worth to be investigated. However, it is also possible that your grub setup uses a triple fault to reboot the machine. What happens if you do the following:

VBoxManage setextradata VM_NAME VBoxInternal/EM/TripleFaultReset 1

Does grub then behaves normally?

[JamesJohnston]

Thanks for the response. After doing some more debugging/logging with grub in conjunction with the video capture feature of VirtualBox, I think this is a bug in grub - I think it has a problem of infinite recursion and running out of stack space; however the current default VirtualBox behavior makes it look like a VirtualBox problem as well.

I noticed that the VM hangs for several seconds before the Guru Meditation. I then set the TripleFaultReset as advised. This got rid of the Guru Meditation, and instead the VM reset after the few seconds instead. I think the hang is infinite recursion/looping.

I also tried this on VMware Workstation 12 Player. On this platform, the VM just hangs forever: no errors, no resets. Different VM platform, similar problem: more evidence of a grub problem.

I think the best suggestion I have at this point might be that VirtualBox should show some more informative UI indicating what happened (explain that a triple fault reset happened), and most importantly emphasizing that the problem is a problem with the guest. The current UI makes the user thing the problem is a defect within VirtualBox itself. Perhaps the default behavior of an unset "TripleFaultReset" can be changed to show more helpful UI instead of a Guru Meditation? And non-default behavior where the flag is set can reset the VM like it currently does.

I've attached some screenshots demonstrating what I think is the grub infinite loop. Notice that grub immediately goes into a recursive loop of calling "Opening memdisk" --> "Detecting tarfs" --> "Opening memdisk" --> "Detecting tarfs" --> a bunch of verify calls --> "Opening memdisk" --> etc. forever. This goes on forever and ever and eventually the VM dies. Never do we ever see a "Closing memdisk" log entry, which I verified normally happens right after "Detecting tarfs" on a working grub setup.

[JamesJohnston]

Update: after tracing the code shown in the grub logs, I found the problem was that I didn't include the required RSA module into the core grub image for verifying signatures. The verification code then attempted to load the required RSA module. But in order to do that, it had to run the verification function to check the signature on the RSA module. And so on...

It was 100% a grub bug and/or an issue with how I installed grub.

However, I still stand by my previous message - I think VirtualBox handled the problem poorly, by indicating the problem was a defect in VirtualBox (thus this ticket), when it was clearly a guest problem...

[Frank Mehnert]

"Explaining the triple fault" is already done (kind of) in the Guru Medidation. Triple faults are not that common and only legacy guests (perhaps DOS) use them to reboot the machine. The Guru Meditation is a hint for the user that VBox doesn't know how to continue. The trigger for the problem might be either a guest problem or a bug in VirtualBox. During the Guru Meditation the whole VM is stopped and it's possible to evaluate the VM state with a debugger. Of course we can try to improve the error handling for mis-behaving guests but at the moment we focus to fix bugs in VirtualBox (e.g. edge cases which are wrongly emulated).