[vbox-dev] Virtualbox don't restore FPU segments with 32-bit guests while using xsave/xrstor
Frank Mehnert
frank.mehnert at oracle.com
Fri Jun 17 10:23:14 GMT 2016
Hi Quentin,
this patch (or rather parts of it) are part of VBox 5.0.22 so the problem
should be fixed now.
Kind regards,
Frank
On Tuesday 17 May 2016 16:54:56 quentin buathier wrote:
> Hi Michal,
>
> Thank you for taking the time to reply to all my messy explanations and I
> hope my patch will be useful.
>
> Regards
>
> 2016-05-17 16:05 GMT+02:00 Michal Necasek <michal.necasek at oracle.com>:
> > Hi Quentin,
> >
> > A couple of minor points:
> >
> > - VM exits are not only triggered by guest code but also by external
> >
> > interrupts arriving on the host; that introduces a nice element of
> > unpredictability into the mix
> >
> > - Some x86 CPUs may only save the FP data pointer on FP exceptions;
> >
> > however, the FP instruction pointer should still be saved always
> >
> > - One of the situations that could cause problems is when the VM process
> >
> > gets rescheduled on a different host CPU and the guest FPU state needs to
> > be correctly saved/loaded; again unpredictable when that happens
> >
> > I'm still not certain why your testcase always immediately fails on a
> >
> > 64-bit host OS. I suspect that the OS actively destroys the FP CS/DS while
> > manipulating the 64-bit FPU state, which is probably forced by the signal
> > processing.
> >
> > This whole thing was really poorly designed on the hardware level and I
> >
> > don't understand why there's no instruction that can save/restore the full
> > FPU state at once (16-bit segments + 64-bit offsets). Then again Intel
> > clearly wants to get away from tracking the CS/DS completely which also
> > "solves" the issue. It actually causes problems for some existing software
> > because FP exception handlers will fail/crash when CS/DS is invalid.
> >
> > Anyway, your patch should solve the problem on the affected CPUs. We'll
> >
> > take another look at it (too easy to get something wrong with assembler)
> > but I think it's doing the right thing.
> >
> > Thanks,
> >
> > Michal
> >
> > On 5/16/2016 11:28 AM, quentin buathier wrote:
> >> Hi Michal,
> >>
> >> I was only mentionning the 32 bit architecture to highlight that we're
> >> running a 32 bit OS (not just an executable) on the 64 bit host. Plus,
> >> you are entirely right that the behavior occurs only on Sandy & Ivy
> >> bridge. My understanding of when the CS & DS registers are lost is
> >> incomplete - especially since it's not systematic. Let me try to detail
> >> what I think happens, based on the behavior the provided sample can show:
> >>
> >> 1- From the guest, we execute an FPU instruction, which sets FPU-CS &DS
> >> 2- I assume other code, which triggers a VMExit, runs inside the guest
> >> 3- Virtualbox uses the 64-bit xsave to save the guest state (see the ref
> >> 1 at the end of this mail)
> >> 4- Some host code is run which (sometimes only) resets the CS & DS. I
> >> see two possibilities here: maybe FPU instructions are executed which,
> >> because they are on the 64-bit host, reset CS and DS. Or maybe the host
> >> executes something like fninit. I'm not too sure about this part.
> >> 5- Virtualbox then uses the 64-bit xrstor to restore the guest state,
> >> which ignores CS / DS (see the ref 2).
> >> 6- Back to the guest , we test FPU CS and DS: the expected behaviour is
> >> that between the execution of the FPU instruction and this test, CS and
> >> DS aren't reset.
> >> 7- If step 4 reset CS & DS, the previous assumption turns out wrong and
> >> we stop the program. Otherwise, we keep looping back to step 1.
> >>
> >> So, from a guest point of view, CS & DS can be sometimes reset for no
> >> obvious reason. On later versions of the CPU, the CS & DS are never
> >> saved anyway, so that wouldn't be a problem.
> >>
> >> The submitted patch forces Virtualbox to properly save FPU CS and DS
> >> when the guest state is saved using xsave. After the patch is applied,
> >> the test program should run indefinitely on a 32-bit guest, matching the
> >> behaviour of a non-virtualised 32-bit system.
> >>
> >> Ref 1: In subchapter 13.5.1 "x87 State" of "Intel 64 and IA-32
> >> Architectures Developer's Manual" we read that in 64-bit mode (REX.W=1),
> >> xsave uses the space of CS and DS to save upper parts of FIP and FDP, so
> >> FPU CS and DS are ignored.
> >> Ref 2: Like the ref 1, xrstor doesn't restore FPU CS and DS
> >>
> >> Regards
> >>
> >> 2016-05-13 15:39 GMT+02:00 Michal Necasek <michal.necasek at oracle.com
> >>
> >> <mailto:michal.necasek at oracle.com>>:
> >> Hi Quentin,
> >>
> >> Please add a bit more detail... we're on a 64-bit system the whole
> >>
> >> time so that's not useful in explaining what happens. When we're
> >> running a 32-bit executable, it's executing a 32-bit FNSTENV and the
> >> segment registers should be saved. Where exactly are they lost? By
> >> the way, feel free to point to the relevant sections of the Intel
> >> SDM.
> >>
> >> Also, am I understanding correctly that this problem only affects
> >>
> >> Sandy Bridge and Ivy Bridge generation CPUs? Older CPUs shouldn't be
> >> affected because they have no AVX, and Haswell and later are broken
> >> by design and never save the CS/DS at all.
> >>
> >> Regards,
> >>
> >> Michal
> >>
> >> ----- Original Message -----
> >> From: qbuathier at tetrane.com <mailto:qbuathier at tetrane.com>
> >> To: vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
> >> Sent: Friday, May 13, 2016 2:55:20 PM GMT +01:00 Amsterdam / Berlin
> >> / Bern / Rome / Stockholm / Vienna
> >> Subject: Re: [vbox-dev] Virtualbox don't restore FPU segments with
> >> 32-bit guests while using xsave/xrstor
> >>
> >> Hi Frank,
> >>
> >> When we run a FPU instruction on a x86 system, FPU CS and DS are set
> >> to the same value as the CS and DS registers. This is an historical
> >> reason where FPU is an individual chip with its own registers. So,
> >> the expected behaviour of this sample is to run forever on a guest
> >>
> >> x86.
> >>
> >> On a x86_64 system, this is different because it removes this
> >> historical behavior and the FPU segments are always set to 0. So, if
> >> you run this sample on a x86_64 system, it's the normal behaviour to
> >> have "segs unset 1".
> >>
> >> The problem is that Virtualbox doesn't restore properly FPU CS and
> >> DS when it uses xsave/xrstor, but does it if it uses fxsave/fxrstor.
> >> The problem happens randomly and I think that it's cause by the
> >> switch between guest code execution and host code execution (when
> >> Virtualbox save the guest state, restore the host state and after
> >> the host code execution repeated those operations in reverse order).
> >>
> >> My patch fixes that problem by using the same behaviour than for
> >> fxsave / fxrstor to save CS and DS when using xsave / xrstor in that
> >> particular case.
> >>
> >> Regards,
> >>
> >> 2016-05-13 14:53 GMT+02:00 quentin buathier <qbuathier at tetrane.com
> >>
> >> <mailto:qbuathier at tetrane.com>>:
> >> Hi Frank,
> >>
> >> When we run a FPU instruction on a x86 system, FPU CS and DS are
> >> set to the same value as the CS and DS registers. This is an
> >> historical reason where FPU is an individual chip with its own
> >> registers. So, the expected behaviour of this sample is to run
> >> forever on a guest x86.
> >>
> >> On a x86_64 system, this is different because it removes this
> >> historical behavior and the FPU segments are always set to 0.
> >> So, if you run this sample on a x86_64 system, it's the normal
> >> behaviour to have "segs unset 1".
> >>
> >> The problem is that Virtualbox doesn't restore properly FPU CS
> >> and DS when it uses xsave/xrstor, but does it if it uses
> >> fxsave/fxrstor. The problem happens randomly and I think that
> >> it's cause by the switch between guest code execution and host
> >> code execution (when Virtualbox save the guest state, restore
> >> the host state and after the host code execution repeated those
> >> operations in reverse order).
> >>
> >> My patch fixes that problem by using the same behaviour than for
> >> fxsave / fxrstor to save CS and DS when using xsave / xrstor in
> >> that particular case.
> >>
> >> Regards,
> >>
> >> 2016-05-13 9:14 GMT+02:00 Frank Mehnert
> >>
> >> <frank.mehnert at oracle.com <mailto:frank.mehnert at oracle.com>>:
> >> Hi Quentin,
> >>
> >> what is the expected behaviour of this sample? Should it run
> >> forever?
> >> Running this sample in a 32-bit guests stops with "segs
> >> unset" after
> >> a short time. After applying your patch and running the
> >> example in the
> >> guest, it runs forever.
> >>
> >> But: If I run this sample on the host (Linux 4.5.4), it will
> >> always
> >> stop with "segs unset 1" after the first turn.
> >>
> >> Kind regards,
> >>
> >> Frank
> >>
> >> On Thursday 12 May 2016 14:47:01 quentin buathier wrote:
> >> > This is a sample in C++ which reproduce the problem
> >>
> >> randomly (1 ~ 2
> >>
> >> > seconds).
> >> > On the same host / guest / cpu that my previous mail.
> >> >
> >> > 2016-05-12 12:20 GMT+02:00 quentin buathier
> >>
> >> <qbuathier at tetrane.com <mailto:qbuathier at tetrane.com>>:
> >> > > Hi Michal,
> >> > >
> >> > > I can't now give a way to reproduce the bug but I'll
> >>
> >> send an executable if
> >>
> >> > > I manage to reproduce the problem on something
> >> > > minimalist.
> >> > >
> >> > > But I can give you the context of the problem:
> >> > > Host OS: Debian jessie 64-bits
> >> > > Guest OS: Debian jessie 32-bits
> >> > > Processor: i7-2600 (and all i7 tested)
> >> > >
> >> > > PS: Sorry for the previous mail that was accidently sent
> >> > >
> >> > > Regards,
> >> > >
> >> > > 2016-05-12 12:18 GMT+02:00 quentin buathier
> >>
> >> <qbuathier at tetrane.com <mailto:qbuathier at tetrane.com>>:
> >> > >> Hi Michal,
> >> > >>
> >> > >> I can't now give a way to reproduce the bug. I'll send
> >>
> >> an executable if I
> >>
> >> > >> manage to reproduce the problem on something minimalist.
> >> > >>
> >> > >> But I can give you the context of the problem:
> >> > >> Host OS: Debian jessie 64-bits
> >> > >>
> >> > >> 2016-05-12 11:52 GMT+02:00 Michal Necasek
> >>
> >> <michal.necasek at oracle.com <mailto:michal.necasek at oracle.com
> >>
> >> > >>> Hi Quentin,
> >> > >>>
> >> > >>> Thank you for the patch!
> >> > >>>
> >> > >>> Unfortunately (?) I can't reproduce the problem that
> >>
> >> was originally
> >>
> >> > >>> fixed. Could you please provide a bit more
> >>
> >> information? What's the host
> >>
> >> > >>> OS,
> >> > >>> guest OS, host CPU type? How to reproduce the problem?
> >> > >>>
> >> > >>> Regards,
> >> > >>>
> >> > >>> Michal
> >> > >>>
> >> > >>> On 5/12/2016 11:26 AM, quentin buathier wrote:
> >> > >>>> Hi,
> >> > >>>>
> >> > >>>> As I understand it, there used to be a problem with
> >>
> >> restoring the FPU
> >>
> >> > >>>> segments in case of a 64-bit hosts with a 32-bit
> >>
> >> guest. This issue has
> >>
> >> > >>>> been fixed by using the macros "SAVE_32_OR_64_FPU" and
> >> > >>>> "RESTORE_32_OR_64_FPU" in
> >>
> >> "src/VBox/VMM/VMMR0/CPUMR0A.asm" (when
> >>
> >> > >>>> Virtualbox was using fxsave and fxrstor to save and
> >>
> >> restore the FPU
> >>
> >> > >>>> context).
> >> > >>>>
> >> > >>>> But along with the recent support of xsave / xrstor,
> >>
> >> the bug was
> >>
> >> > >>>> reintroduced: if the CPU supports xsave/xrstor,
> >>
> >> Virtualbox uses these
> >>
> >> > >>>> instructions and the guest's FPU segments are not
> >>
> >> restored properly.
> >>
> >> > >>>> Please find attached a possible patch to fix this
> >>
> >> issue (MIT licence).
> >>
> >> > >>>> Regards,
> >> > >>>>
> >> > >>>>
> >> > >>>> _______________________________________________
> >> > >>>> vbox-dev mailing list
> >>
> >> > >>>> vbox-dev at virtualbox.org <mailto:
> >> vbox-dev at virtualbox.org>
> >>
> >> > >>>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
> >> > >>>
> >> > >>> _______________________________________________
> >> > >>> vbox-dev mailing list
> >> > >>> vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org
> >> > >>>
> >> > >>> https://www.virtualbox.org/mailman/listinfo/vbox-dev
> >>
> >> --
> >> Dr.-Ing. Frank Mehnert | Software Development Director,
> >> VirtualBox
> >> ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384
> >> Weinstadt, Germany
> >>
> >> ORACLE Deutschland B.V. & Co. KG
> >> Hauptverwaltung: Riesstraße 25, D-80992 München
> >> Registergericht: Amtsgericht München, HRA 95603
> >>
> >> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >> Handelsregister der Handelskammer Midden-Niederlande, Nr.
> >> 30143697
> >> Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val
> >> Maher
> >> _______________________________________________
> >> vbox-dev mailing list
> >> vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
> >> https://www.virtualbox.org/mailman/listinfo/vbox-dev
> >>
> >> _______________________________________________
> >> vbox-dev mailing list
> >> vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
> >> https://www.virtualbox.org/mailman/listinfo/vbox-dev
> >>
> >> _______________________________________________
> >> vbox-dev mailing list
> >> vbox-dev at virtualbox.org
> >> https://www.virtualbox.org/mailman/listinfo/vbox-dev
> >
> > _______________________________________________
> > vbox-dev mailing list
> > vbox-dev at virtualbox.org
> > https://www.virtualbox.org/mailman/listinfo/vbox-dev
--
Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox
ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany
ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstraße 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher
More information about the vbox-dev
mailing list