[vbox-dev] Virtualbox don't restore FPU segments with 32-bit guests while using xsave/xrstor
quentin buathier
qbuathier at tetrane.com
Tue May 17 14:54:56 GMT 2016
Hi Michal,
Thank you for taking the time to reply to all my messy explanations and I
hope my patch will be useful.
Regards
2016-05-17 16:05 GMT+02:00 Michal Necasek <michal.necasek at oracle.com>:
>
> Hi Quentin,
>
> A couple of minor points:
>
> - VM exits are not only triggered by guest code but also by external
> interrupts arriving on the host; that introduces a nice element of
> unpredictability into the mix
>
> - Some x86 CPUs may only save the FP data pointer on FP exceptions;
> however, the FP instruction pointer should still be saved always
>
> - One of the situations that could cause problems is when the VM process
> gets rescheduled on a different host CPU and the guest FPU state needs to
> be correctly saved/loaded; again unpredictable when that happens
>
> I'm still not certain why your testcase always immediately fails on a
> 64-bit host OS. I suspect that the OS actively destroys the FP CS/DS while
> manipulating the 64-bit FPU state, which is probably forced by the signal
> processing.
>
> This whole thing was really poorly designed on the hardware level and I
> don't understand why there's no instruction that can save/restore the full
> FPU state at once (16-bit segments + 64-bit offsets). Then again Intel
> clearly wants to get away from tracking the CS/DS completely which also
> "solves" the issue. It actually causes problems for some existing software
> because FP exception handlers will fail/crash when CS/DS is invalid.
>
> Anyway, your patch should solve the problem on the affected CPUs. We'll
> take another look at it (too easy to get something wrong with assembler)
> but I think it's doing the right thing.
>
> Thanks,
> Michal
>
>
> On 5/16/2016 11:28 AM, quentin buathier wrote:
>
>> Hi Michal,
>>
>> I was only mentionning the 32 bit architecture to highlight that we're
>> running a 32 bit OS (not just an executable) on the 64 bit host. Plus,
>> you are entirely right that the behavior occurs only on Sandy & Ivy
>> bridge. My understanding of when the CS & DS registers are lost is
>> incomplete - especially since it's not systematic. Let me try to detail
>> what I think happens, based on the behavior the provided sample can show:
>>
>> 1- From the guest, we execute an FPU instruction, which sets FPU-CS &DS
>> 2- I assume other code, which triggers a VMExit, runs inside the guest
>> 3- Virtualbox uses the 64-bit xsave to save the guest state (see the ref
>> 1 at the end of this mail)
>> 4- Some host code is run which (sometimes only) resets the CS & DS. I
>> see two possibilities here: maybe FPU instructions are executed which,
>> because they are on the 64-bit host, reset CS and DS. Or maybe the host
>> executes something like fninit. I'm not too sure about this part.
>> 5- Virtualbox then uses the 64-bit xrstor to restore the guest state,
>> which ignores CS / DS (see the ref 2).
>> 6- Back to the guest , we test FPU CS and DS: the expected behaviour is
>> that between the execution of the FPU instruction and this test, CS and
>> DS aren't reset.
>> 7- If step 4 reset CS & DS, the previous assumption turns out wrong and
>> we stop the program. Otherwise, we keep looping back to step 1.
>>
>> So, from a guest point of view, CS & DS can be sometimes reset for no
>> obvious reason. On later versions of the CPU, the CS & DS are never
>> saved anyway, so that wouldn't be a problem.
>>
>> The submitted patch forces Virtualbox to properly save FPU CS and DS
>> when the guest state is saved using xsave. After the patch is applied,
>> the test program should run indefinitely on a 32-bit guest, matching the
>> behaviour of a non-virtualised 32-bit system.
>>
>> Ref 1: In subchapter 13.5.1 "x87 State" of "Intel 64 and IA-32
>> Architectures Developer's Manual" we read that in 64-bit mode (REX.W=1),
>> xsave uses the space of CS and DS to save upper parts of FIP and FDP, so
>> FPU CS and DS are ignored.
>> Ref 2: Like the ref 1, xrstor doesn't restore FPU CS and DS
>>
>> Regards
>>
>> 2016-05-13 15:39 GMT+02:00 Michal Necasek <michal.necasek at oracle.com
>> <mailto:michal.necasek at oracle.com>>:
>>
>>
>> Hi Quentin,
>>
>> Please add a bit more detail... we're on a 64-bit system the whole
>> time so that's not useful in explaining what happens. When we're
>> running a 32-bit executable, it's executing a 32-bit FNSTENV and the
>> segment registers should be saved. Where exactly are they lost? By
>> the way, feel free to point to the relevant sections of the Intel SDM.
>>
>> Also, am I understanding correctly that this problem only affects
>> Sandy Bridge and Ivy Bridge generation CPUs? Older CPUs shouldn't be
>> affected because they have no AVX, and Haswell and later are broken
>> by design and never save the CS/DS at all.
>>
>> Regards,
>> Michal
>>
>>
>> ----- Original Message -----
>> From: qbuathier at tetrane.com <mailto:qbuathier at tetrane.com>
>> To: vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
>> Sent: Friday, May 13, 2016 2:55:20 PM GMT +01:00 Amsterdam / Berlin
>> / Bern / Rome / Stockholm / Vienna
>> Subject: Re: [vbox-dev] Virtualbox don't restore FPU segments with
>> 32-bit guests while using xsave/xrstor
>>
>> Hi Frank,
>>
>> When we run a FPU instruction on a x86 system, FPU CS and DS are set
>> to the same value as the CS and DS registers. This is an historical
>> reason where FPU is an individual chip with its own registers. So,
>> the expected behaviour of this sample is to run forever on a guest
>> x86.
>>
>> On a x86_64 system, this is different because it removes this
>> historical behavior and the FPU segments are always set to 0. So, if
>> you run this sample on a x86_64 system, it's the normal behaviour to
>> have "segs unset 1".
>>
>> The problem is that Virtualbox doesn't restore properly FPU CS and
>> DS when it uses xsave/xrstor, but does it if it uses fxsave/fxrstor.
>> The problem happens randomly and I think that it's cause by the
>> switch between guest code execution and host code execution (when
>> Virtualbox save the guest state, restore the host state and after
>> the host code execution repeated those operations in reverse order).
>>
>> My patch fixes that problem by using the same behaviour than for
>> fxsave / fxrstor to save CS and DS when using xsave / xrstor in that
>> particular case.
>>
>> Regards,
>>
>> 2016-05-13 14:53 GMT+02:00 quentin buathier <qbuathier at tetrane.com
>> <mailto:qbuathier at tetrane.com>>:
>>
>> Hi Frank,
>>
>> When we run a FPU instruction on a x86 system, FPU CS and DS are
>> set to the same value as the CS and DS registers. This is an
>> historical reason where FPU is an individual chip with its own
>> registers. So, the expected behaviour of this sample is to run
>> forever on a guest x86.
>>
>> On a x86_64 system, this is different because it removes this
>> historical behavior and the FPU segments are always set to 0.
>> So, if you run this sample on a x86_64 system, it's the normal
>> behaviour to have "segs unset 1".
>>
>> The problem is that Virtualbox doesn't restore properly FPU CS
>> and DS when it uses xsave/xrstor, but does it if it uses
>> fxsave/fxrstor. The problem happens randomly and I think that
>> it's cause by the switch between guest code execution and host
>> code execution (when Virtualbox save the guest state, restore
>> the host state and after the host code execution repeated those
>> operations in reverse order).
>>
>> My patch fixes that problem by using the same behaviour than for
>> fxsave / fxrstor to save CS and DS when using xsave / xrstor in
>> that particular case.
>>
>> Regards,
>>
>> 2016-05-13 9:14 GMT+02:00 Frank Mehnert
>> <frank.mehnert at oracle.com <mailto:frank.mehnert at oracle.com>>:
>>
>> Hi Quentin,
>>
>> what is the expected behaviour of this sample? Should it run
>> forever?
>> Running this sample in a 32-bit guests stops with "segs
>> unset" after
>> a short time. After applying your patch and running the
>> example in the
>> guest, it runs forever.
>>
>> But: If I run this sample on the host (Linux 4.5.4), it will
>> always
>> stop with "segs unset 1" after the first turn.
>>
>> Kind regards,
>>
>> Frank
>>
>> On Thursday 12 May 2016 14:47:01 quentin buathier wrote:
>> > This is a sample in C++ which reproduce the problem
>> randomly (1 ~ 2
>> > seconds).
>> > On the same host / guest / cpu that my previous mail.
>> >
>> > 2016-05-12 12:20 GMT+02:00 quentin buathier
>> <qbuathier at tetrane.com <mailto:qbuathier at tetrane.com>>:
>> > > Hi Michal,
>> > >
>> > > I can't now give a way to reproduce the bug but I'll
>> send an executable if
>> > > I manage to reproduce the problem on something minimalist.
>> > >
>> > > But I can give you the context of the problem:
>> > > Host OS: Debian jessie 64-bits
>> > > Guest OS: Debian jessie 32-bits
>> > > Processor: i7-2600 (and all i7 tested)
>> > >
>> > > PS: Sorry for the previous mail that was accidently sent
>> > >
>> > > Regards,
>> > >
>> > > 2016-05-12 12:18 GMT+02:00 quentin buathier
>> <qbuathier at tetrane.com <mailto:qbuathier at tetrane.com>>:
>> > >> Hi Michal,
>> > >>
>> > >> I can't now give a way to reproduce the bug. I'll send
>> an executable if I
>> > >> manage to reproduce the problem on something minimalist.
>> > >>
>> > >> But I can give you the context of the problem:
>> > >> Host OS: Debian jessie 64-bits
>> > >>
>> > >> 2016-05-12 11:52 GMT+02:00 Michal Necasek
>> <michal.necasek at oracle.com <mailto:michal.necasek at oracle.com
>> >>:
>>
>> > >>> Hi Quentin,
>> > >>>
>> > >>> Thank you for the patch!
>> > >>>
>> > >>> Unfortunately (?) I can't reproduce the problem that
>> was originally
>> > >>>
>> > >>> fixed. Could you please provide a bit more
>> information? What's the host
>> > >>> OS,
>> > >>> guest OS, host CPU type? How to reproduce the problem?
>> > >>>
>> > >>> Regards,
>> > >>>
>> > >>> Michal
>> > >>>
>> > >>> On 5/12/2016 11:26 AM, quentin buathier wrote:
>> > >>>> Hi,
>> > >>>>
>> > >>>> As I understand it, there used to be a problem with
>> restoring the FPU
>> > >>>> segments in case of a 64-bit hosts with a 32-bit
>> guest. This issue has
>> > >>>> been fixed by using the macros "SAVE_32_OR_64_FPU" and
>> > >>>> "RESTORE_32_OR_64_FPU" in
>> "src/VBox/VMM/VMMR0/CPUMR0A.asm" (when
>> > >>>> Virtualbox was using fxsave and fxrstor to save and
>> restore the FPU
>> > >>>> context).
>> > >>>>
>> > >>>> But along with the recent support of xsave / xrstor,
>> the bug was
>> > >>>> reintroduced: if the CPU supports xsave/xrstor,
>> Virtualbox uses these
>> > >>>> instructions and the guest's FPU segments are not
>> restored properly.
>> > >>>>
>> > >>>> Please find attached a possible patch to fix this
>> issue (MIT licence).
>> > >>>>
>> > >>>> Regards,
>> > >>>>
>> > >>>>
>> > >>>> _______________________________________________
>> > >>>> vbox-dev mailing list
>> > >>>> vbox-dev at virtualbox.org <mailto:
>> vbox-dev at virtualbox.org>
>> > >>>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
>> > >>>
>> > >>> _______________________________________________
>> > >>> vbox-dev mailing list
>> > >>> vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org
>> >
>> > >>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
>>
>> --
>> Dr.-Ing. Frank Mehnert | Software Development Director,
>> VirtualBox
>> ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384
>> Weinstadt, Germany
>>
>> ORACLE Deutschland B.V. & Co. KG
>> Hauptverwaltung: Riesstraße 25, D-80992 München
>> Registergericht: Amtsgericht München, HRA 95603
>>
>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>> Handelsregister der Handelskammer Midden-Niederlande, Nr.
>> 30143697
>> Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val
>> Maher
>> _______________________________________________
>> vbox-dev mailing list
>> vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
>>
>>
>>
>>
>> _______________________________________________
>> vbox-dev mailing list
>> vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
>>
>>
>>
>>
>> _______________________________________________
>> vbox-dev mailing list
>> vbox-dev at virtualbox.org
>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
>>
>>
> _______________________________________________
> vbox-dev mailing list
> vbox-dev at virtualbox.org
> https://www.virtualbox.org/mailman/listinfo/vbox-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20160517/62b41806/attachment.html>
More information about the vbox-dev
mailing list