[vbox-dev] Question regarding - tr.flags and TSS_BUSY

[Michal Necasek]

 TR can't be not reloaded, because in general there's no guarantee it didn't change.

 And no, there is no fine-grained tracking of virtual CPU state changes. After coming back from the recompiler, everything is assumed to be changed. Not super efficient, which is why we're getting away from the recompiler for the typical VT-x/AMD-V execution case.

 I don't think checking the actual values would help either because any part of the CPU state can and does change. For example if a task switch is emulated in the recompiler, TR will be changed. We can't say "TR won't change" because under some circumstances it will.

        - Michal

----- Original Message -----
From: alexander.boettcher at genode-labs.com
To: michal.necasek at oracle.com
Cc: vbox-dev at virtualbox.org
Sent: Thursday, March 23, 2017 11:41:45 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
Subject: Re: [vbox-dev] Question regarding - tr.flags and TSS_BUSY

Hi,

On 23.03.2017 00:23, Michal Necasek wrote:
>  Your setup is certainly... exotic.

Yes.

> I'm entirely convinced there's a problem, but I don't think your
suggested solution is right.

Ok.

> 
>  Based on my reading of AMD and especially Intel VT-x documentation, the busy bit must be set in the TSS that's loaded into the TR at all times. The busy bit is set on CPU reset, the LTR instruction sets it, any task switch sets it. In other words, there is no way to unset it. VT-x actively prevents loading the guest state if the busy bit in the TR is not set.

I think you are right regarding the documentation. However I think the
actual point is that the tr should not be (re-)loaded into hardware when
coming from REM if it was actually not changed/touched at all. This
would avoid all the hassle completely.

By the way, is there a standard way in VBox (a flag, e.g.) per register
to detect whether it was changed by REM - so that the HM acceleration
part may decide what to write/reload into VMCS/VMCB and what not.

I mean, just comparing the register after a exit and the state short
before resuming HM mode is possible, but nothing what is really
desirable (for various reasons).


>  So the code in VBoxRecompiler.c which clears the busy bit is just plain wrong, and there may be some code in the recompiler (task switch code) that needs adjusting as well. We'll need to fix that.

I see, thanks for taking care.

Alex.

> 
> 
>       Regards,
>          Michal
> 
> ----- Original Message -----
> From: alexander.boettcher at genode-labs.com
> To: michal.necasek at oracle.com
> Cc: vbox-dev at virtualbox.org
> Sent: Monday, March 20, 2017 10:20:00 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
> Subject: Re: [vbox-dev] Question regarding - tr.flags and TSS_BUSY
> 
> Hello,
> 
> ok, here are the information I have and some points if you want try to
> reproduce it. (You will need 2.4.1 or the upcoming 2.9 Qemu release to
> reproduce it) :
> 
> The guest is a a trivial 32bit Genode/NOVA setup (the iso and the output
> of a run in Qemu 2.4.1 see [0,1]). It uses sysenter/sysexit, I/O APIC is
> required and the output is available via the serial device.
> 
> qemu-system-x86_64 -no-kvm -cpu core2duo -nographic -m 64 -cdrom log.iso
> 
> 
> This log.iso is then added as VM to a 64bit Genode/NOVA + VBox (4.3.40)
> port to run it as VM.
> 
> This setup is executed on a emulated 64 bit host (Qemu + AMD-V SVM
> extension enabled [2,3] by using:
> 
> qemu-system-x86_64 -nographic -no-kvm -m 512 -cpu phenom -serial
> mon:stdio -cdrom vbox4_tss_bad.iso
> 
> With the proposed tss fix the issue is gone [4,5].
> 
> According to Qemu, a interrupt [6] happens in ring-3 (cpl==3), where the
> tr/tss is used to look up the ss & esp register to switch to ring-0
> (dpl==0) [7]. Then the assertion triggers as shown in [3] by the line of
> [8] in Qemu.
> 
> I hope this helps. At the moment I am not able to trigger the issue in
> the Vbox5 port reliable, thats-why I solely added the vbox4 version.
> 
> Cheers,
> 
> Alex.
> 
> 
> [0] https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/log.iso
> [1] https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/log_iso.txt
> 
> [2]
> https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/virtualbox_tss_bad.iso
> [3]
> https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/virtualbox_tss_bad.txt
> 
> [4]
> https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/virtualbox_tss_fix.iso
> [5]
> https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/virtualbox_tss_fix.txt
> 
> 
> [6]
> https://github.com/alex-ab/qemu/blob/stable-2.4/target-i386/seg_helper.c#L577
> [7]
> https://github.com/alex-ab/qemu/blob/stable-2.4/target-i386/seg_helper.c#L676
> [8]
> https://github.com/alex-ab/qemu/blob/stable-2.4/target-i386/seg_helper.c#L151
> 
> On 17.03.2017 00:00, Michal Necasek wrote:
>>
>>    Hi Alexander,
>>
>>  Thanks for the information. I believe you that there is a problem, and the comments in the code indicate as much, but I still don't understand the failure scenario on the virtual CPU level.
>>
>>  Can you confirm that there is task switching involved in a 32-bit guest? Can you please explain exactly at what point the incorrect TSS busy bit value triggers problems? Is it during a task switch (I think it has to be)? What kind of a task switch is it, i.e. what triggered it? JMP, CALL, interrupt...?
>>
>>  The busy flag management in the recompiler task switch (switch_tss()) looks fishy and is almost certainly wrong, but I need to understand more. And I think you're right that the VirtualBox<->recompiler transitions should leave the busy flag alone. But I still don't understand exactly how it's causing problems.
>>
>>  Incidentally, what is 'Qemu+SVM'? That is not SVM as in Secure Virtual Machine aka AMD-V, is it?
>>
>>
>>       Regards,
>>          Michal
>>
>> ----- Original Message -----
>> From: alexander.boettcher at genode-labs.com
>> To: michal.necasek at oracle.com, vbox-dev at virtualbox.org
>> Sent: Monday, March 13, 2017 1:45:19 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
>> Subject: Re: [vbox-dev] Question regarding - tr.flags and TSS_BUSY
>>
>> Hello,
>>
>> On 10.03.2017 17:05, Michal Necasek wrote:
>>>  Could you please describe your use case, or rather how we can reproduce
>>> the problem?
>>
>> I feared you ask - but nevertheless - when running VirtualBox inside
>> Qemu+SVM (and no kvm enabled) with our ported Version of VBox to
>> Genode/NOVA.
>>
>>> This is not code we want to change without precisely
>>> understanding why it needs to change and testing the behavior.
>>
>> I can imagine.
>>
>>>
>>>  Also, why are you using the recompiler at all? On a typical system with
>>> hardware virtualization, it should be used very little or not at all.
>>
>> Probably this is right for the vanilla Virtualbox as you provide. In our
>> Vbox 4 (4.3.40) ported version we mainly use REM + hw accelerated
>> (interface provided by the Microkernel NOVA), there it triggers reliable
>> if running in Qemu.
>>
>> For the VBox 5 (5.1.14) port to Genode we enabled also the IEM, but here
>> it also trigger in Qemu+SVM (no-kvm).
>>
>> You may have luck, Qemu just succeeds running Vbox and the VM - iif the
>> preemption (due to interrupts to be injected by the VBox VMM) of the
>> Guest VM (Microkernel+Genode setup) is that, that you ever get a tr
>> register with set busy bit.
>>
>> I understand that our use-case is maybe of no interest to the majority
>> (even we mainly use it for early debugging in Qemu) - nevertheless we
>> wanted you just let know that there is a issue.
>>
>>
>> Cheers,
>>
>> Alex.
>>
> 

-- 
Alexander Boettcher
Genode Labs

http://www.genode-labs.com - http://www.genode.org

Genode Labs GmbH - Amtsgericht Dresden - HRB 28424 - Sitz Dresden
Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth