<div dir="ltr"><div style="font-size:12.8px">Hi Michal,</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I was only mentionning the 32 bit architecture to highlight that we're running a 32 bit OS (not just an executable) on the 64 bit host. Plus, you are entirely right that the behavior occurs only on Sandy & Ivy bridge. My understanding of when the CS & DS registers are lost is incomplete - especially since it's not systematic. Let me try to detail what I think happens, based on the behavior the provided sample can show:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">1- From the guest, we execute an FPU instruction, which sets FPU-CS &DS<br></div><div style="font-size:12.8px">2- I assume other code, which triggers a VMExit, runs inside the guest<br></div><div style="font-size:12.8px">3- Virtualbox uses the 64-bit xsave to save the guest state (see the ref 1 at the end of this mail)</div><div style="font-size:12.8px">4- Some host code is run which (sometimes only) resets the CS & DS. I see two possibilities here: maybe FPU instructions are executed which, because they are on the 64-bit host, reset CS and DS. Or maybe the host executes something like fninit. I'm not too sure about this part.<br></div><div style="font-size:12.8px">5- Virtualbox then uses the 64-bit xrstor to restore the guest state, which ignores CS / DS (see the ref 2).<br></div><div style="font-size:12.8px"></div><div style="font-size:12.8px">6- Back to the guest , we test FPU CS and DS: the expected behaviour is that between the execution of the FPU instruction and this test, CS and DS aren't reset.<br></div><div style="font-size:12.8px">7- If step 4 reset CS & DS, the previous assumption turns out wrong and we stop the program. Otherwise, we keep looping back to step 1.<br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">So, from a guest point of view, CS & DS can be sometimes reset for no obvious reason. On later versions of the CPU, the CS & DS are never saved anyway, so that wouldn't be a problem.<br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">The submitted patch forces Virtualbox to properly save FPU CS and DS when the guest state is saved using xsave. After the patch is applied, the test program should run indefinitely on a 32-bit guest, matching the behaviour of a non-virtualised 32-bit system.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Ref 1: In subchapter 13.5.1 "x87 State" of "Intel 64 and IA-32 Architectures Developer's Manual" we read that in 64-bit mode (REX.W=1), xsave uses the space of CS and DS to save upper parts of FIP and FDP, so FPU CS and DS are ignored.</div><div style="font-size:12.8px">Ref 2: Like the ref 1, xrstor doesn't restore FPU CS and DS</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Regards</div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-05-13 15:39 GMT+02:00 Michal Necasek <span dir="ltr"><<a href="mailto:michal.necasek@oracle.com" target="_blank">michal.necasek@oracle.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:Times New Roman;font-size:12pt;color:#000000"><br> Hi Quentin,<br><br> Please add a bit more detail... we're on a 64-bit system the whole time so that's not useful in explaining what happens. When we're running a 32-bit executable, it's executing a 32-bit FNSTENV and the segment registers should be saved. Where exactly are they lost? By the way, feel free to point to the relevant sections of the Intel SDM.<br><br> Also, am I understanding correctly that this problem only affects Sandy Bridge and Ivy Bridge generation CPUs? Older CPUs shouldn't be affected because they have no AVX, and Haswell and later are broken by design and never save the CS/DS at all.<br><br> Regards,<br> Michal<div><div class="h5"><br><br>----- Original Message -----<br>From: <a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a><br>To: <a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a><br>Sent: Friday, May 13, 2016 2:55:20 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna<br>Subject: Re: [vbox-dev] Virtualbox don't restore FPU segments with 32-bit guests while using xsave/xrstor<br><br><div dir="ltr"><span style="font-size:12.8px">Hi Frank,</span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">When we run a FPU instruction on a x86 system, FPU CS and DS are set to the same value as the CS and DS registers. This is an historical reason where FPU is an individual chip with its own registers. So, the expected behaviour of this sample is to run forever on a guest x86.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>On a x86_64 system, this is different because it removes this historical behavior and the FPU segments are always set to 0. So, if you run this sample on a x86_64 system, it's the normal behaviour to have "segs unset 1".</div><div><br></div><div>The problem is that Virtualbox doesn't restore properly FPU CS and DS when it uses xsave/xrstor, but does it if it uses fxsave/fxrstor. The problem happens randomly and I think that it's cause by the switch between guest code execution and host code execution (when Virtualbox save the guest state, restore the host state and after the host code execution repeated those operations in reverse order).</div><div><br></div><div>My patch fixes that problem by using the same behaviour than for fxsave / fxrstor to save CS and DS when using xsave / xrstor in that particular case.</div><div><div><br></div><div>Regards,</div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-05-13 14:53 GMT+02:00 quentin buathier <span dir="ltr"><<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Frank,<div><br></div><div>When we run a FPU instruction on a x86 system, FPU CS and DS are set to the same value as the CS and DS registers. This is an historical reason where FPU is an individual chip with its own registers. So, the expected behaviour of this sample is to run forever on a guest x86.</div><div><br></div><div><div>On a x86_64 system, this is different because it removes this historical behavior and the FPU segments are always set to 0. So, if you run this sample on a x86_64 system, it's the normal behaviour to have "segs unset 1".</div><div><br></div><div>The problem is that Virtualbox doesn't restore properly FPU CS and DS when it uses xsave/xrstor, but does it if it uses fxsave/fxrstor. The problem happens randomly and I think that it's cause by the switch between guest code execution and host code execution (when Virtualbox save the guest state, restore the host state and after the host code execution repeated those operations in reverse order).</div><div><br></div><div>My patch fixes that problem by using the same behaviour than for fxsave / fxrstor to save CS and DS when using xsave / xrstor in that particular case.</div><div><div><br></div><div>Regards,</div></div></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">2016-05-13 9:14 GMT+02:00 Frank Mehnert <span dir="ltr"><<a href="mailto:frank.mehnert@oracle.com" target="_blank">frank.mehnert@oracle.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Quentin,<br>
<br>
what is the expected behaviour of this sample? Should it run forever?<br>
Running this sample in a 32-bit guests stops with "segs unset" after<br>
a short time. After applying your patch and running the example in the<br>
guest, it runs forever.<br>
<br>
But: If I run this sample on the host (Linux 4.5.4), it will always<br>
stop with "segs unset 1" after the first turn.<br>
<br>
Kind regards,<br>
<br>
Frank<br>
<div><div><br>
On Thursday 12 May 2016 14:47:01 quentin buathier wrote:<br>
> This is a sample in C++ which reproduce the problem randomly (1 ~ 2<br>
> seconds).<br>
> On the same host / guest / cpu that my previous mail.<br>
><br>
> 2016-05-12 12:20 GMT+02:00 quentin buathier <<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a>>:<br>
> > Hi Michal,<br>
> ><br>
> > I can't now give a way to reproduce the bug but I'll send an executable if<br>
> > I manage to reproduce the problem on something minimalist.<br>
> ><br>
> > But I can give you the context of the problem:<br>
> > Host OS: Debian jessie 64-bits<br>
> > Guest OS: Debian jessie 32-bits<br>
> > Processor: i7-2600 (and all i7 tested)<br>
> ><br>
> > PS: Sorry for the previous mail that was accidently sent<br>
> ><br>
> > Regards,<br>
> ><br>
> > 2016-05-12 12:18 GMT+02:00 quentin buathier <<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a>>:<br>
> >> Hi Michal,<br>
> >><br>
> >> I can't now give a way to reproduce the bug. I'll send an executable if I<br>
> >> manage to reproduce the problem on something minimalist.<br>
> >><br>
> >> But I can give you the context of the problem:<br>
> >> Host OS: Debian jessie 64-bits<br>
> >><br>
> >> 2016-05-12 11:52 GMT+02:00 Michal Necasek <<a href="mailto:michal.necasek@oracle.com" target="_blank">michal.necasek@oracle.com</a>>:<br>
> >>> Hi Quentin,<br>
> >>><br>
> >>> Thank you for the patch!<br>
> >>><br>
> >>> Unfortunately (?) I can't reproduce the problem that was originally<br>
> >>><br>
> >>> fixed. Could you please provide a bit more information? What's the host<br>
> >>> OS,<br>
> >>> guest OS, host CPU type? How to reproduce the problem?<br>
> >>><br>
> >>> Regards,<br>
> >>><br>
> >>> Michal<br>
> >>><br>
> >>> On 5/12/2016 11:26 AM, quentin buathier wrote:<br>
> >>>> Hi,<br>
> >>>><br>
> >>>> As I understand it, there used to be a problem with restoring the FPU<br>
> >>>> segments in case of a 64-bit hosts with a 32-bit guest. This issue has<br>
> >>>> been fixed by using the macros "SAVE_32_OR_64_FPU" and<br>
> >>>> "RESTORE_32_OR_64_FPU" in "src/VBox/VMM/VMMR0/CPUMR0A.asm" (when<br>
> >>>> Virtualbox was using fxsave and fxrstor to save and restore the FPU<br>
> >>>> context).<br>
> >>>><br>
> >>>> But along with the recent support of xsave / xrstor, the bug was<br>
> >>>> reintroduced: if the CPU supports xsave/xrstor, Virtualbox uses these<br>
> >>>> instructions and the guest's FPU segments are not restored properly.<br>
> >>>><br>
> >>>> Please find attached a possible patch to fix this issue (MIT licence).<br>
> >>>><br>
> >>>> Regards,<br>
> >>>><br>
> >>>><br>
> >>>> _______________________________________________<br>
> >>>> vbox-dev mailing list<br>
> >>>> <a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a><br>
> >>>> <a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
> >>><br>
> >>> _______________________________________________<br>
> >>> vbox-dev mailing list<br>
> >>> <a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a><br>
> >>> <a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
<br>
</div></div><span><font color="#888888">--<br>
Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox<br>
ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany<br>
<br>
ORACLE Deutschland B.V. & Co. KG<br>
Hauptverwaltung: Riesstraße 25, D-80992 München<br>
Registergericht: Amtsgericht München, HRA 95603<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.<br>
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande<br>
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697<br>
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher<br>
</font></span><div><div>_______________________________________________<br>
vbox-dev mailing list<br>
<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a><br>
<a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></div></div><br>_______________________________________________<br>
vbox-dev mailing list<br>
<a href="mailto:vbox-dev@virtualbox.org">vbox-dev@virtualbox.org</a><br>
<a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
<br></blockquote></div><br></div>