Ticket #9053 (closed defect: obsolete)

Opened 6 years ago

Last modified 3 years ago

FSTENV assembler instruction - wrong implementation

Reported by: fritz Owned by:
Priority: major Component: other
Version: VirtualBox 4.0.8 Keywords: assembler implementation floating point
Cc: Guest type: Windows
Host type: Windows

Description (last modified by frank) (diff)


I believe VirtualBox has wrong implementation of FSTENV assembler instruction. It is not dumping last float-pointing instruction address correctly. I've created a simple application in assembler which can be used as proof.

Try and debug this application with debugger (OllyDbg for example) and take a look at buffer after the FSTENV instruction. It should contain address of FNOP assembler instruction.

format PE GUI
entry main

include ''

section '.data' data readable writeable
	buffer db "                        ", 0

section '.text' code readable executable
	mov eax, buffer
	fstenv [eax]

	invoke ExitProcess, 0

section '.idata' import data readable
	library kernel32, 'KERNEL32.DLL'
	import kernel32, ExitProcess, 'ExitProcess'

You can compile the example code with FASM.

Change History

comment:1 Changed 6 years ago by frank

I've tested this with several Linux guests (32-bit guest with/without VT-x, 64-bit guest) and it always worked correctly. Note that the instruction pointer starts at offset 12 of the buffer. Which content has the buffer in your case?

comment:2 Changed 6 years ago by fritz

I'm comparing this case with my windows host on which works fine. This is my buffer: 7F 02 FF FF 00 00 FF FF FF FF FF FF 29 FA C4 77 1B 00 7D 07 0C FF 55 01 23 00 FF FF

Instruction pointer is (starting at offset 12) 77C4FA29. It should be 00402006, that's where FNOP is located.

My host is Windows and guest is Windows also.

comment:3 Changed 6 years ago by michaln

I tried the example on Windows host and guest, in both cases I got the expected result. FYI, you don't need any of the Windows API junk for your tiny example, a simple "ret" does the job.

To eliminate the possibility of a debugger interfering, I wrote the following tiny C program - not very portable, but that doesn't matter, it does the job:

#include <stdio.h>
#include <fenv.h>

int main( void )
    fenv_t  fenv;

    __asm fnop;
    fegetenv( &fenv );
    printf( "Last FP EIP: %08x\n", fenv.ins_ptr_offset );
    return( 0 );

The program prints something along the lines "Last FP EIP: 00401018" which is the address of the FNOP instruction. The result is identical on my Windows Server 2003 host and inside a Windows XP guest.

The address 0x77C4FA29 looks very much like an address inside a DLL. You should find out what is at that address. My guess is that the debugger you are using is broken and upsets the results. I strongly suggest that rather than using a debugger, you print the last instruction pointer value from the fstenv buffer.

comment:4 Changed 6 years ago by fritz

michaln, thanks for the feedback, I'll do the test and get back at you.

comment:5 Changed 6 years ago by frank

If you can still reproduce this issue, please attach a VBox.log file from such a VM session so we can have a look at the VM configuration.

comment:6 Changed 6 years ago by fritz

michaln, frank, sorry for false alarm. When executed w/o debugger the IP is correct, but when in debugger - it's completely wrong. This is strange because debugger (OllyDbg) works on host OS and on other machines ...

Thanks for the response.

comment:7 Changed 3 years ago by frank

  • Status changed from new to closed
  • Resolution set to obsolete
  • Description modified (diff)
Note: See TracTickets for help on using tickets.
ContactPrivacy policyTerms of Use