VERR_CPUM_IPE_2 on Intel Xeon Silver 4410T [fixed in svn]

[BenV666]

As discussed in the ​Forum:

I've assembled a new server here using a Supermicro X13SEI-F and an Intel Xeon 4410T processor with 64GB of memory. After successfully installing Arch linux on it (now on kernel 6.3.3) with Virtualbox 7.0.8, my next objective is getting a VM up and running.

Unfortunately this seems harder than it should be. Initially I ran into a kernel panic issue due to ibt, but adding ibt=off to the kernel parameters made that go away (based on ticket #21435).

However, now I'm stuck: No matter what VM I try to configure, VMs fail to boot with the same error:

00:00:00.135712 HM: HMR3Init: VT-x w/ nested paging and unrestricted guest execution hw support
00:00:00.135829 CPUM: fXStateHostMask=0xe7; initial: 0xe7; host XCR0=0x602e7
00:00:00.135831 AssertLogRel /build/virtualbox/src/VirtualBox-7.0.8/src/VBox/VMM/VMMR3/CPUM.cpp(2192) int CPUMR3Init(PVM): pVM->cpum.s.HostFeatures.cbMaxExtendedState >= sizeof(X86FXSTATE) && pVM->cpum.s.HostFeatures.cbMaxExtendedState <= sizeof(pVM->apCpusR3[0]->cpum.s.Host.XState) && pVM->cpum.s.HostFeatures.cbMaxExtendedState <= sizeof(pVM->apCpusR3[0]->cpum.s.Guest.XState)
00:00:00.136288 VMSetError: /build/virtualbox/src/VirtualBox-7.0.8/src/VBox/VMM/VMMR3/VM.cpp(341) int VMR3Create(uint32_t, PCVMM2USERMETHODS, uint64_t, PFNVMATERROR, void*, PFNCFGMCONSTRUCTOR, void*, VM**, UVM**); rc=VERR_CPUM_IPE_2
00:00:00.136293 VMSetError: CPUM internal processing error #2.
00:00:00.137475 ERROR [COM]: aRC=NS_ERROR_FAILURE (0x80004005) aIID={6ac83d89-6ee7-4e33-8ae6-b257b2e81be8} aComponent={ConsoleWrap} aText={CPUM internal processing error #2. (VERR_CPUM_IPE_2)}, preserve=false aResultDetail=-1755
00:00:00.137638 Console: Machine state changed to 'PoweredOff'
00:00:00.141898 Power up failed (vrc=VERR_CPUM_IPE_2, hrc=NS_ERROR_FAILURE (0X80004005))
00:00:00.145749 VRDP: TCP server closed.
00:00:00.160572 VBoxHeadless: exiting

I've tried all the options that I could find regarding processor extensions (e.g. nested paging / VT-x), acceleration enabled/disabled, 32/64 bit vms, all break with the same error.

This makes me believe it might either be due to the CPU being unknown to virtualbox, the kernel not playing nice, or a combination of such. (note that there is nothing weird in dmesg with the ibt=off setting)

That said, I'm not sure how to make this work - are there more kernel options to try, or perhaps VM options to somehow disable whatever it's breaking on? Suggestions welcome, I'm adding the complete log of trying to boot a basic (empty) arch VM.

Also tested 7.0.9-157502 with the same result.

[BenV666]

arch VM attempt

[zutro]

Hello,
I just ran into the same issue while migrating my VM's to new server, which is embedded with Supermicro X13SEI-TF (similar one like Ben's) and Intel Xeon Gold 6426Y. I am also using recent Arch Linux (6.5.5), which now delivers Virtualbox 7.0.10. I fiddled with it a bit more and made some advancement, based on the error posted here (same as mine) some assertion fails in here: https://www.virtualbox.org/browser/vbox/trunk/src/VBox/VMM/VMMR3/CPUM.cpp?rev=100940#L2193. What fails there in my case is that cbMaxExtendedState (0x2b00) is greater than both host and guest XState (0x2000). This block of code seems a bit unfinished as for example aligned cbMaxXState variable is never used, so I just commented this block out and recompiled whole package using provided PKGBUILD (package contents could be found here ​https://gitlab.archlinux.org/archlinux/packaging/packages/virtualbox) and other makepkg tools provided by our linux distribution. After this simple change, my VM boots normally. I understand that this is not ideal approach as this does not explain the problem itself and might have some hidden unpredictable effects, but it may help in the meantime as there is no other solution.
Have a nice day.

[4access]

Still not fixed in VirtualBox 7.0.12 -- does not work on Xeon w5-2455X (no VM can be started).

If a person who wrote the relevant XSAVE/XRESTORE code has read Intel CPUID manual they'd have known that you can't define X86FXSTATE as a compile-time constant, because future CPUs will report different state store sizes. Sapphire Rapids reports 0x2B00 because of AMX instruction set extensions which are supported and enabled in Linux and Windows 11 hosts (Windows 10 and older do not enable AMX).

I can't believe Sapphire Rapids support was not validated at launch, let alone six months later and counting.

[bird]

The assertion is wrong. Fixed in internal 160259 (trunk) and r160260 (7.0).

[Klaus Espenlaub]

The latest test builds have this fix. For those who need the source code change: see changeset:102135.

[4access]

I have tested this build and while it can boot Windows 10 guest on Windows 11 host with AMX enabled, Windows 7 SP1 x64 guest has graphical corruption (see screenshot attached) and hangs on boot if tried in UEFI mode (original Microsoft ISO was used for testing). Non-UEFI mode appears to work fine.