[vbox-dev] Question regarding - tr.flags and TSS_BUSY

[Michal Necasek]

    Hi Alexander,

 Your setup is certainly... exotic. I'm entirely convinced there's a problem, but I don't think your suggested solution is right.

 Based on my reading of AMD and especially Intel VT-x documentation, the busy bit must be set in the TSS that's loaded into the TR at all times. The busy bit is set on CPU reset, the LTR instruction sets it, any task switch sets it. In other words, there is no way to unset it. VT-x actively prevents loading the guest state if the busy bit in the TR is not set.

 So the code in VBoxRecompiler.c which clears the busy bit is just plain wrong, and there may be some code in the recompiler (task switch code) that needs adjusting as well. We'll need to fix that.


      Regards,
         Michal

----- Original Message -----
From: alexander.boettcher at genode-labs.com
To: michal.necasek at oracle.com
Cc: vbox-dev at virtualbox.org
Sent: Monday, March 20, 2017 10:20:00 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
Subject: Re: [vbox-dev] Question regarding - tr.flags and TSS_BUSY

Hello,

ok, here are the information I have and some points if you want try to
reproduce it. (You will need 2.4.1 or the upcoming 2.9 Qemu release to
reproduce it) :

The guest is a a trivial 32bit Genode/NOVA setup (the iso and the output
of a run in Qemu 2.4.1 see [0,1]). It uses sysenter/sysexit, I/O APIC is
required and the output is available via the serial device.

qemu-system-x86_64 -no-kvm -cpu core2duo -nographic -m 64 -cdrom log.iso


This log.iso is then added as VM to a 64bit Genode/NOVA + VBox (4.3.40)
port to run it as VM.

This setup is executed on a emulated 64 bit host (Qemu + AMD-V SVM
extension enabled [2,3] by using:

qemu-system-x86_64 -nographic -no-kvm -m 512 -cpu phenom -serial
mon:stdio -cdrom vbox4_tss_bad.iso

With the proposed tss fix the issue is gone [4,5].

According to Qemu, a interrupt [6] happens in ring-3 (cpl==3), where the
tr/tss is used to look up the ss & esp register to switch to ring-0
(dpl==0) [7]. Then the assertion triggers as shown in [3] by the line of
[8] in Qemu.

I hope this helps. At the moment I am not able to trigger the issue in
the Vbox5 port reliable, thats-why I solely added the vbox4 version.

Cheers,

Alex.


[0] https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/log.iso
[1] https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/log_iso.txt

[2]
https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/virtualbox_tss_bad.iso
[3]
https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/virtualbox_tss_bad.txt

[4]
https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/virtualbox_tss_fix.iso
[5]
https://github.com/alex-ab/genode/raw/vbox4_tss_qemu/virtualbox_tss_fix.txt


[6]
https://github.com/alex-ab/qemu/blob/stable-2.4/target-i386/seg_helper.c#L577
[7]
https://github.com/alex-ab/qemu/blob/stable-2.4/target-i386/seg_helper.c#L676
[8]
https://github.com/alex-ab/qemu/blob/stable-2.4/target-i386/seg_helper.c#L151

On 17.03.2017 00:00, Michal Necasek wrote:
> 
>    Hi Alexander,
> 
>  Thanks for the information. I believe you that there is a problem, and the comments in the code indicate as much, but I still don't understand the failure scenario on the virtual CPU level.
> 
>  Can you confirm that there is task switching involved in a 32-bit guest? Can you please explain exactly at what point the incorrect TSS busy bit value triggers problems? Is it during a task switch (I think it has to be)? What kind of a task switch is it, i.e. what triggered it? JMP, CALL, interrupt...?
> 
>  The busy flag management in the recompiler task switch (switch_tss()) looks fishy and is almost certainly wrong, but I need to understand more. And I think you're right that the VirtualBox<->recompiler transitions should leave the busy flag alone. But I still don't understand exactly how it's causing problems.
> 
>  Incidentally, what is 'Qemu+SVM'? That is not SVM as in Secure Virtual Machine aka AMD-V, is it?
> 
> 
>       Regards,
>          Michal
> 
> ----- Original Message -----
> From: alexander.boettcher at genode-labs.com
> To: michal.necasek at oracle.com, vbox-dev at virtualbox.org
> Sent: Monday, March 13, 2017 1:45:19 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
> Subject: Re: [vbox-dev] Question regarding - tr.flags and TSS_BUSY
> 
> Hello,
> 
> On 10.03.2017 17:05, Michal Necasek wrote:
>>  Could you please describe your use case, or rather how we can reproduce
>> the problem?
> 
> I feared you ask - but nevertheless - when running VirtualBox inside
> Qemu+SVM (and no kvm enabled) with our ported Version of VBox to
> Genode/NOVA.
> 
>> This is not code we want to change without precisely
>> understanding why it needs to change and testing the behavior.
> 
> I can imagine.
> 
>>
>>  Also, why are you using the recompiler at all? On a typical system with
>> hardware virtualization, it should be used very little or not at all.
> 
> Probably this is right for the vanilla Virtualbox as you provide. In our
> Vbox 4 (4.3.40) ported version we mainly use REM + hw accelerated
> (interface provided by the Microkernel NOVA), there it triggers reliable
> if running in Qemu.
> 
> For the VBox 5 (5.1.14) port to Genode we enabled also the IEM, but here
> it also trigger in Qemu+SVM (no-kvm).
> 
> You may have luck, Qemu just succeeds running Vbox and the VM - iif the
> preemption (due to interrupts to be injected by the VBox VMM) of the
> Guest VM (Microkernel+Genode setup) is that, that you ever get a tr
> register with set busy bit.
> 
> I understand that our use-case is maybe of no interest to the majority
> (even we mainly use it for early debugging in Qemu) - nevertheless we
> wanted you just let know that there is a issue.
> 
> 
> Cheers,
> 
> Alex.
> 

-- 
Alexander Boettcher
Genode Labs

http://www.genode-labs.com - http://www.genode.org

Genode Labs GmbH - Amtsgericht Dresden - HRB 28424 - Sitz Dresden
Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth