Opened 3 years ago
#20223 new enhancement
Linux/64bit host XMM register usage in kernel randomly crashes Linux guest
| Reported by: | vboxcomapijunkie | Owned by: | |
|---|---|---|---|
| Component: | VMM | Version: | VirtualBox 6.1.16 |
| Keywords: | XMM register usage kernel Linux AMD64 | Cc: | |
| Guest type: | Linux | Host type: | Linux |
Description
Synopsis: src/VBox/VMM/Makefile.kmk 'knows' flag VBOX_WITH_KERNEL_USING_XMM, which is added for Windows/64bit builds only. Problem is that Linux kernel DOES MAKE USE of XMM registers, too. Examples include aes-ni kernel module (AES-NI instructions implicitly use XMM0..15) or RAID, respectively). VMM at ring 0 only implements XMM wrappers for Windows/64bit and does not have conditionals to abort build if VBOX_WITH_KERNEL_USING_XMM is set in src/VBox/VMM/Makefile.kmk for linux.amd64. This yields a kernel panic because Microsoft ABI is completely different from System V ABI (aka gcc ABI). The 6.1.18 trunk seems to rely on a complete rewrite of src/VBox/VMM/VMMR0/HMR0A.asm but 6.1.13 / 6.1.16 (6.0 also) still have this problem.
Solution is:
a) Add VBOX_WITH_KERNEL_USING_XMM to src/VBox/VMM/Makefile.kmk for linux.amd64
b) Provide conditional implementation in src/VBox/VMM/VMMR0/HMR0A.asm for gcc/amd64 expecting function parameters in System V ABI registers rdi, rsi, rdx, rcx, r8, and r9. The two wrappers hmR0VMXStartVMWrapXMM and hmR0SVMRunWrapXMM need this.
c) calling CPUMIsGuestFPUStateActive from yasm assembler to C++ requires suffix "wrt ..plt"
I have patches for this but optimized only for "my" Intel CPU. VBox users either have to switch to 6.1.18 or apply the above mentioned modifications.
Recommendation:
VBOX maintainers should think about making VBOX_WITH_KERNEL_USING_XMM the default for Linux, too (because of possible AES-NI or RAID usage, respectively).
