<div dir="ltr">Hi Michal,<div><br></div><div>Thank you for taking the time to reply to all my messy explanations and I hope my patch will be useful.</div><div><br></div><div>Regards</div><div class="gmail_extra"><br><div class="gmail_quote">2016-05-17 16:05 GMT+02:00 Michal Necasek <span dir="ltr"><<a href="mailto:michal.necasek@oracle.com" target="_blank">michal.necasek@oracle.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi Quentin,<br>
<br>
A couple of minor points:<br>
<br>
- VM exits are not only triggered by guest code but also by external interrupts arriving on the host; that introduces a nice element of unpredictability into the mix<br>
<br>
- Some x86 CPUs may only save the FP data pointer on FP exceptions; however, the FP instruction pointer should still be saved always<br>
<br>
- One of the situations that could cause problems is when the VM process gets rescheduled on a different host CPU and the guest FPU state needs to be correctly saved/loaded; again unpredictable when that happens<br>
<br>
I'm still not certain why your testcase always immediately fails on a 64-bit host OS. I suspect that the OS actively destroys the FP CS/DS while manipulating the 64-bit FPU state, which is probably forced by the signal processing.<br>
<br>
This whole thing was really poorly designed on the hardware level and I don't understand why there's no instruction that can save/restore the full FPU state at once (16-bit segments + 64-bit offsets). Then again Intel clearly wants to get away from tracking the CS/DS completely which also "solves" the issue. It actually causes problems for some existing software because FP exception handlers will fail/crash when CS/DS is invalid.<br>
<br>
Anyway, your patch should solve the problem on the affected CPUs. We'll take another look at it (too easy to get something wrong with assembler) but I think it's doing the right thing.<br>
<br>
Thanks,<br>
Michal<div><div><br>
<br>
On 5/16/2016 11:28 AM, quentin buathier wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
Hi Michal,<br>
<br>
I was only mentionning the 32 bit architecture to highlight that we're<br>
running a 32 bit OS (not just an executable) on the 64 bit host. Plus,<br>
you are entirely right that the behavior occurs only on Sandy & Ivy<br>
bridge. My understanding of when the CS & DS registers are lost is<br>
incomplete - especially since it's not systematic. Let me try to detail<br>
what I think happens, based on the behavior the provided sample can show:<br>
<br>
1- From the guest, we execute an FPU instruction, which sets FPU-CS &DS<br>
2- I assume other code, which triggers a VMExit, runs inside the guest<br>
3- Virtualbox uses the 64-bit xsave to save the guest state (see the ref<br>
1 at the end of this mail)<br>
4- Some host code is run which (sometimes only) resets the CS & DS. I<br>
see two possibilities here: maybe FPU instructions are executed which,<br>
because they are on the 64-bit host, reset CS and DS. Or maybe the host<br>
executes something like fninit. I'm not too sure about this part.<br>
5- Virtualbox then uses the 64-bit xrstor to restore the guest state,<br>
which ignores CS / DS (see the ref 2).<br>
6- Back to the guest , we test FPU CS and DS: the expected behaviour is<br>
that between the execution of the FPU instruction and this test, CS and<br>
DS aren't reset.<br>
7- If step 4 reset CS & DS, the previous assumption turns out wrong and<br>
we stop the program. Otherwise, we keep looping back to step 1.<br>
<br>
So, from a guest point of view, CS & DS can be sometimes reset for no<br>
obvious reason. On later versions of the CPU, the CS & DS are never<br>
saved anyway, so that wouldn't be a problem.<br>
<br>
The submitted patch forces Virtualbox to properly save FPU CS and DS<br>
when the guest state is saved using xsave. After the patch is applied,<br>
the test program should run indefinitely on a 32-bit guest, matching the<br>
behaviour of a non-virtualised 32-bit system.<br>
<br>
Ref 1: In subchapter 13.5.1 "x87 State" of "Intel 64 and IA-32<br>
Architectures Developer's Manual" we read that in 64-bit mode (REX.W=1),<br>
xsave uses the space of CS and DS to save upper parts of FIP and FDP, so<br>
FPU CS and DS are ignored.<br>
Ref 2: Like the ref 1, xrstor doesn't restore FPU CS and DS<br>
<br>
Regards<br>
<br>
2016-05-13 15:39 GMT+02:00 Michal Necasek <<a href="mailto:michal.necasek@oracle.com" target="_blank">michal.necasek@oracle.com</a><br></div></div>
<mailto:<a href="mailto:michal.necasek@oracle.com" target="_blank">michal.necasek@oracle.com</a>>>:<span><br>
<br>
<br>
Hi Quentin,<br>
<br>
Please add a bit more detail... we're on a 64-bit system the whole<br>
time so that's not useful in explaining what happens. When we're<br>
running a 32-bit executable, it's executing a 32-bit FNSTENV and the<br>
segment registers should be saved. Where exactly are they lost? By<br>
the way, feel free to point to the relevant sections of the Intel SDM.<br>
<br>
Also, am I understanding correctly that this problem only affects<br>
Sandy Bridge and Ivy Bridge generation CPUs? Older CPUs shouldn't be<br>
affected because they have no AVX, and Haswell and later are broken<br>
by design and never save the CS/DS at all.<br>
<br>
Regards,<br>
Michal<br>
<br>
<br>
----- Original Message -----<br></span><div><div>
From: <a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a> <mailto:<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a>><br>
To: <a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a> <mailto:<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a>><br>
Sent: Friday, May 13, 2016 2:55:20 PM GMT +01:00 Amsterdam / Berlin<br>
/ Bern / Rome / Stockholm / Vienna<br>
Subject: Re: [vbox-dev] Virtualbox don't restore FPU segments with<br>
32-bit guests while using xsave/xrstor<br>
<br>
Hi Frank,<br>
<br>
When we run a FPU instruction on a x86 system, FPU CS and DS are set<br>
to the same value as the CS and DS registers. This is an historical<br>
reason where FPU is an individual chip with its own registers. So,<br>
the expected behaviour of this sample is to run forever on a guest x86.<br>
<br>
On a x86_64 system, this is different because it removes this<br>
historical behavior and the FPU segments are always set to 0. So, if<br>
you run this sample on a x86_64 system, it's the normal behaviour to<br>
have "segs unset 1".<br>
<br>
The problem is that Virtualbox doesn't restore properly FPU CS and<br>
DS when it uses xsave/xrstor, but does it if it uses fxsave/fxrstor.<br>
The problem happens randomly and I think that it's cause by the<br>
switch between guest code execution and host code execution (when<br>
Virtualbox save the guest state, restore the host state and after<br>
the host code execution repeated those operations in reverse order).<br>
<br>
My patch fixes that problem by using the same behaviour than for<br>
fxsave / fxrstor to save CS and DS when using xsave / xrstor in that<br>
particular case.<br>
<br>
Regards,<br>
<br>
2016-05-13 14:53 GMT+02:00 quentin buathier <<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a><br></div></div>
<mailto:<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a>>>:<span><br>
<br>
Hi Frank,<br>
<br>
When we run a FPU instruction on a x86 system, FPU CS and DS are<br>
set to the same value as the CS and DS registers. This is an<br>
historical reason where FPU is an individual chip with its own<br>
registers. So, the expected behaviour of this sample is to run<br>
forever on a guest x86.<br>
<br>
On a x86_64 system, this is different because it removes this<br>
historical behavior and the FPU segments are always set to 0.<br>
So, if you run this sample on a x86_64 system, it's the normal<br>
behaviour to have "segs unset 1".<br>
<br>
The problem is that Virtualbox doesn't restore properly FPU CS<br>
and DS when it uses xsave/xrstor, but does it if it uses<br>
fxsave/fxrstor. The problem happens randomly and I think that<br>
it's cause by the switch between guest code execution and host<br>
code execution (when Virtualbox save the guest state, restore<br>
the host state and after the host code execution repeated those<br>
operations in reverse order).<br>
<br>
My patch fixes that problem by using the same behaviour than for<br>
fxsave / fxrstor to save CS and DS when using xsave / xrstor in<br>
that particular case.<br>
<br>
Regards,<br>
<br>
2016-05-13 9:14 GMT+02:00 Frank Mehnert<br></span>
<<a href="mailto:frank.mehnert@oracle.com" target="_blank">frank.mehnert@oracle.com</a> <mailto:<a href="mailto:frank.mehnert@oracle.com" target="_blank">frank.mehnert@oracle.com</a>>>:<span><br>
<br>
Hi Quentin,<br>
<br>
what is the expected behaviour of this sample? Should it run<br>
forever?<br>
Running this sample in a 32-bit guests stops with "segs<br>
unset" after<br>
a short time. After applying your patch and running the<br>
example in the<br>
guest, it runs forever.<br>
<br>
But: If I run this sample on the host (Linux 4.5.4), it will<br>
always<br>
stop with "segs unset 1" after the first turn.<br>
<br>
Kind regards,<br>
<br>
Frank<br>
<br>
On Thursday 12 May 2016 14:47:01 quentin buathier wrote:<br>
> This is a sample in C++ which reproduce the problem<br>
randomly (1 ~ 2<br>
> seconds).<br>
> On the same host / guest / cpu that my previous mail.<br>
><br>
> 2016-05-12 12:20 GMT+02:00 quentin buathier<br></span>
<<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a> <mailto:<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a>>>:<span><br>
> > Hi Michal,<br>
> ><br>
> > I can't now give a way to reproduce the bug but I'll<br>
send an executable if<br>
> > I manage to reproduce the problem on something minimalist.<br>
> ><br>
> > But I can give you the context of the problem:<br>
> > Host OS: Debian jessie 64-bits<br>
> > Guest OS: Debian jessie 32-bits<br>
> > Processor: i7-2600 (and all i7 tested)<br>
> ><br>
> > PS: Sorry for the previous mail that was accidently sent<br>
> ><br>
> > Regards,<br>
> ><br>
> > 2016-05-12 12:18 GMT+02:00 quentin buathier<br></span>
<<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a> <mailto:<a href="mailto:qbuathier@tetrane.com" target="_blank">qbuathier@tetrane.com</a>>>:<span><br>
> >> Hi Michal,<br>
> >><br>
> >> I can't now give a way to reproduce the bug. I'll send<br>
an executable if I<br>
> >> manage to reproduce the problem on something minimalist.<br>
> >><br>
> >> But I can give you the context of the problem:<br>
> >> Host OS: Debian jessie 64-bits<br>
> >><br>
> >> 2016-05-12 11:52 GMT+02:00 Michal Necasek<br></span>
<<a href="mailto:michal.necasek@oracle.com" target="_blank">michal.necasek@oracle.com</a> <mailto:<a href="mailto:michal.necasek@oracle.com" target="_blank">michal.necasek@oracle.com</a>>>:<div><div><br>
> >>> Hi Quentin,<br>
> >>><br>
> >>> Thank you for the patch!<br>
> >>><br>
> >>> Unfortunately (?) I can't reproduce the problem that<br>
was originally<br>
> >>><br>
> >>> fixed. Could you please provide a bit more<br>
information? What's the host<br>
> >>> OS,<br>
> >>> guest OS, host CPU type? How to reproduce the problem?<br>
> >>><br>
> >>> Regards,<br>
> >>><br>
> >>> Michal<br>
> >>><br>
> >>> On 5/12/2016 11:26 AM, quentin buathier wrote:<br>
> >>>> Hi,<br>
> >>>><br>
> >>>> As I understand it, there used to be a problem with<br>
restoring the FPU<br>
> >>>> segments in case of a 64-bit hosts with a 32-bit<br>
guest. This issue has<br>
> >>>> been fixed by using the macros "SAVE_32_OR_64_FPU" and<br>
> >>>> "RESTORE_32_OR_64_FPU" in<br>
"src/VBox/VMM/VMMR0/CPUMR0A.asm" (when<br>
> >>>> Virtualbox was using fxsave and fxrstor to save and<br>
restore the FPU<br>
> >>>> context).<br>
> >>>><br>
> >>>> But along with the recent support of xsave / xrstor,<br>
the bug was<br>
> >>>> reintroduced: if the CPU supports xsave/xrstor,<br>
Virtualbox uses these<br>
> >>>> instructions and the guest's FPU segments are not<br>
restored properly.<br>
> >>>><br>
> >>>> Please find attached a possible patch to fix this<br>
issue (MIT licence).<br>
> >>>><br>
> >>>> Regards,<br>
> >>>><br>
> >>>><br>
> >>>> _______________________________________________<br>
> >>>> vbox-dev mailing list<br></div></div>
> >>>> <a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a> <mailto:<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a>><span><br>
> >>>> <a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
> >>><br>
> >>> _______________________________________________<br>
> >>> vbox-dev mailing list<br></span>
> >>> <a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a> <mailto:<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a>><span><br>
> >>> <a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
<br>
--<br>
Dr.-Ing. Frank Mehnert | Software Development Director,<br>
VirtualBox<br>
ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384<br>
Weinstadt, Germany<br>
<br>
ORACLE Deutschland B.V. & Co. KG<br>
Hauptverwaltung: Riesstraße 25, D-80992 München<br>
Registergericht: Amtsgericht München, HRA 95603<br>
<br>
Komplementärin: ORACLE Deutschland Verwaltung B.V.<br>
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande<br>
Handelsregister der Handelskammer Midden-Niederlande, Nr.<br>
30143697<br>
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val<br>
Maher<br>
_______________________________________________<br>
vbox-dev mailing list<br></span>
<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a> <mailto:<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a>><span><br>
<a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
vbox-dev mailing list<br></span>
<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a> <mailto:<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a>><span><br>
<a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
vbox-dev mailing list<br>
<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a><br>
<a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
<br>
</span></blockquote><div><div>
<br>
_______________________________________________<br>
vbox-dev mailing list<br>
<a href="mailto:vbox-dev@virtualbox.org" target="_blank">vbox-dev@virtualbox.org</a><br>
<a href="https://www.virtualbox.org/mailman/listinfo/vbox-dev" rel="noreferrer" target="_blank">https://www.virtualbox.org/mailman/listinfo/vbox-dev</a><br>
</div></div></blockquote></div><br></div></div>