[vbox-dev] D3D support in custom frontend

Klaus Espenlaub klaus.espenlaub at oracle.com
Tue Feb 10 14:57:01 GMT 2015

On 09.02.2015 11:54, Rūdolfs Bundulis wrote:
> Hi Klaus,
> thanks for the response, yeah I suspected that the path was read from 
> somewhere, since running regsvr32 to unregister VBoxC.dll from 
> installation directory and then registering it from my path didn't 
> chagne anything, but I'm happy - it's not an issue to run my binaries 
> from the installation folder, the main thing is that things seem to be 
> working. Now the only thing I need is to see if I can make OpenGL 
> render to a custom FBO so I can live without the window and access the 
> OpenGL pixel data, but that is my issue, if I come up with anything 
> interesting maybe that can be use fro you too. At least as far as I 
> understand right now, due to the need of a window on the host, 
> VBoxHeadless.exe does not provide any 3D acceleration support right?
 From what I understood it does work for quite a while now (at least 
4.3)... all the FBO stuff should be already in place and working for 
VBoxHeadless, as long as the process gets started in a context offers 
working GDI/Direct3D functions.

The only question is if this does what you need or only does the 
necessary work for the VRDE interface...

> Sorry to bother you so much, but the issue I'm more troubled by is the 
> crash I get - I've already described it before but I have made little 
> progress. Basically what happens is whenever I start my VM with my 
> custom frontend with more than 1 CPU it crashes on Windows startup 
> (longjmp_internal from Microsoft C runtime fails with unaligned stack, 
> the last stack entry I saw from VBoxREM.dll was cpu_exit_loop or smth) 
> - since VBoxREM.dll is built with MinGW visual studio does not help a 
> lot, and when running under debug version under MinGW I also was 
> unable to obtain any valuable info. With 1 CPU it always works fine. 
> I've tried CRT debug heap, Application Verifier etc. to find potential 
> issues with my code (since the default Qt frontend does not crash with 
> the same configuration) I blame my part of the code, but still maybe 
> there are some peculiarities with the COM interface. Do you know any 
> bugs that have reported similar issues to what I'm seeing? Or maybe 
> you can give any pointers since currently I'm kind of stuck.
> Again, great thanks for the support, my research would have hit dead 
> end without your help, yesterday I finally finished splitting so 
> thanks to VirtualBox I am able to run a single virtual 9600x5400 
> display on VirtualBox (what is weird, even with WDDM drivers) and 
> split/encode it to 25 displays, basically this cannot be done with 
> hardware (well in terms of a single homogeneous display) so I'm very 
> happy.
Could be that we're just over cautious with WDDM (assuming the behavior 
of certain greedy apps) and that there is actually no hard requirement 
for a 3rd buffer.
> Best Regards,
> Rudolfs Bundulis
> 2015-02-09 12:21 GMT+02:00 Klaus Espenlaub <klaus.espenlaub at oracle.com 
> <mailto:klaus.espenlaub at oracle.com>>:
>     Rūdolfs,
>     On 09.02.2015 10:28, Rūdolfs Bundulis wrote:
>>     Hi,
>>     yeah it turned out that when running from the VirtualBox
>>     installation path 3D support works for my frontend, seems that it
>>     comes from the fact that the COM API is using the dll's  from the
>>     installation folder. I'm just gonna leave this information here
>>     in case anyone else stumbles on the same issue.
>     VBoxC.dll contains "in-process" COM components (which live in the
>     "API client" process), and the path to it is kept in the registry.
>     Normal COM stuff.
>>     2015-02-07 18:37 GMT+02:00 Rūdolfs Bundulis
>>     <rudolfs.bundulis at gmail.com <mailto:rudolfs.bundulis at gmail.com>>:
>>         Hi,
>>         again thanks for all the previous help, I managed to build
>>         VBox with the LogRel() but when I copied the dll's over the
>>         installation i got a ton of errors about invalid image hashes
>>         (I guess that just copying over is not the way to go).
>>         Anyhow, I came up with a different approach and made a small
>>         frontend that would basically do the same stuff as my app but
>>         also draw the output to a window. When I launch the
>>         application from the VirtualBox installation folder
>>         everything works fine - 3D support works, when I take the app
>>         and all the virtualbox binaries and put them in a random
>>         folder, it again stops working. And what is interresting that
>>         using Process Explorer I was able to see that actually two
>>         copies of VBoxC.dll are loaded (one from the installation
>>         folder and one from my folder) and one VBoxRT.dll (only from
>>         the VirtualBox installation folder). I assume the ones from
>>         the installation folder are loaded by the COM object? Is this
>>         a very very bad practice to try and deploy the VirtualBox
>>         binaries outside the installation path?
>     The invalid image hashes most likely mean that VirtualBox isn't
>     accepting the signature in your binaries. It only accepts the "one
>     signer" of everything, so if you mix Oracle and your own build
>     then it won't work very well.
>     Deploying VirtualBox binaries outside the installation path isn't
>     exactly forbidden, but we don't test it right now. This will stay
>     for a while, we have still way too much noise on the hardening
>     related tickets.
>     Klaus
>>         2015-01-26 15:25 GMT+02:00 Rūdolfs Bundulis
>>         <rudolfs.bundulis at gmail.com <mailto:rudolfs.bundulis at gmail.com>>:
>>             Hi Vitali,
>>             this was what I was looking for, much appreciated :)
>>             Best Regards,
>>             Rudolfs Bundulis
>>             2015-01-26 14:59 GMT+02:00 Vadim Galitsyn
>>             <vadim.galitsyn at oracle.com
>>             <mailto:vadim.galitsyn at oracle.com>>:
>>                 Hi Rudolfs,
>>                 you could add LogRel() messages and they should
>>                 appear in release log (VBox.log file) once VBox has
>>                 been built w/ corresponding flag. Note, that this
>>                 only works for guest R0 code (miniport). Adding
>>                 LogRel() to R3 code (display driver) will not result
>>                 in extra messages in log.
>>                 Please consider to add VBOX_WITH_R0_LOGGING=1 into
>>                 LocalConfig.kmk in order to enable LogRel()’s.
>>                 Vadim
>>>                 On 26 Jan 2015, at 15:26, Rūdolfs Bundulis
>>>                 <rudolfs.bundulis at gmail.com
>>>                 <mailto:rudolfs.bundulis at gmail.com>> wrote:
>>>                 Hi Vadim,
>>>                 ok clear, thanks for the info. We'll so far I
>>>                 haven't patched anything in the VBox code itself, as
>>>                 I said I have a custom frontend. Does that code help
>>>                 you in any way? I'll make a debug build and look for
>>>                 more messages and mail back if it gives any
>>>                 additional info, thanks for all the help so far. The
>>>                 main thing I wanted to understand was is there any
>>>                 way to see the output from the WARN macros in the
>>>                 miniport driver, and as far as I understand there is
>>>                 none.
>>>                 Best Regards,
>>>                 Rudolfs Bundulis
>>>                 2015-01-26 13:42 GMT+02:00 Vadim Galitsyn
>>>                 <vadim.galitsyn at oracle.com
>>>                 <mailto:vadim.galitsyn at oracle.com>>:
>>>                     Hi Rudolfs,
>>>                     Debug build of VBox itself would probably add a
>>>                     bit more messages to VBox.log (and to debug log
>>>                     as well), but this definitely will not prevent
>>>                     the fact that debug version of GAs cause BSOD.
>>>                     Is it possible if you share a patch to OSE tree
>>>                     with your changes, so I could take a look at it
>>>                     later this week (or another)?
>>>                     Vadim
>>>>                     On 26 Jan 2015, at 14:00, Rūdolfs Bundulis
>>>>                     <rudolfs.bundulis at gmail.com
>>>>                     <mailto:rudolfs.bundulis at gmail.com>> wrote:
>>>>                     Hi Vadim,
>>>>                     Well since I also came to the the conclusion
>>>>                     that the WinId could cause this, I tried making
>>>>                     a dummy window for each of the IFramebuffer
>>>>                     instances, but nothing changed. Though the
>>>>                     windows do not have any logic in the message
>>>>                     loop, and I do not resize them when the
>>>>                     resolutions change - maybe this causes some
>>>>                     errors. That is why I asked if any logs from
>>>>                     the miniport driver is available to see where
>>>>                     does it actually crash. If I make a debug
>>>>                     VirtualBox build then would debug GAs work?
>>>>                     Best Regards,
>>>>                     Rudolfs Bundulis
>>>>                     2015-01-26 12:50 GMT+02:00 Vadim Galitsyn
>>>>                     <vadim.galitsyn at oracle.com
>>>>                     <mailto:vadim.galitsyn at oracle.com>>:
>>>>                         Hi Rudolfs,
>>>>                         As far as I can understand, returning NULL
>>>>                         WinID might cause the behaviour you are
>>>>                         observing on Windows host (please note, I
>>>>                         have not tried that locally, take it like a
>>>>                         gesture). Seems, you also already have 3D
>>>>                         host service running. Can you try to
>>>>                         provide valid WinID to Main/src-client
>>>>                         stuff as an experiment and see if things
>>>>                         changed?
>>>>                         Regarding to GAs, I am afraid you’ll have a
>>>>                         BSOD in guest once you attempt to boot VM
>>>>                         with debug GAs installed. We currently have
>>>>                         some debug R0 assertion there.
>>>>                         Vadim
>>>>>                         On 26 Jan 2015, at 13:12, Rūdolfs Bundulis
>>>>>                         <rudolfs.bundulis at gmail.com
>>>>>                         <mailto:rudolfs.bundulis at gmail.com>> wrote:
>>>>>                         Hi Vadim,
>>>>>                         thanks for the pointers, I'll look up the
>>>>>                         code, I was already afraid that something
>>>>>                         is simply missing.
>>>>>                         To describe what I am doing:
>>>>>                         I am running a research project in
>>>>>                         University of Latvia where we are trying
>>>>>                         to use VirtualBox for large scale high
>>>>>                         resolution display wall construction -
>>>>>                         basically we run VirtualBox with a custom
>>>>>                         frontend that takes the video data from
>>>>>                         the VRAM, encodes it and streams over to a
>>>>>                         display wall, currently we've been able to
>>>>>                         run a 9600x5400 surface with Windows XPDM
>>>>>                         (we set this resolution as a hint and then
>>>>>                         internally split it to  5*5 1920x1080
>>>>>                         displays) , which is really impressive, so
>>>>>                         thanks again for the great software. In
>>>>>                         terms of resolution I guess this is as far
>>>>>                         as the existing 256MB VRAM limit in
>>>>>                         VirtualBox can go (at least according to
>>>>>                         the calculations for framebuffer needs
>>>>>                         provided by Klaus Erspenlaub and mentioned
>>>>>                         in your code). Since the resolution limit
>>>>>                         is reached we are trying to address two
>>>>>                         more issues - 3D support and a weird
>>>>>                         unaligned stack crash in VBoxREM.dll with
>>>>>                         more than 1 CPU shared to the machine (for
>>>>>                         the second issue I need to do some more
>>>>>                         research as to where it is coming from
>>>>>                         before I ask any help here). As for the 3D
>>>>>                         - we'd basically want our software to
>>>>>                         function in the same way the VirtualBox QT
>>>>>                         client does, but currently it does not.
>>>>>                         Thanks for the pointer about the
>>>>>                         HGCM/OpenGL service, I'll check out that
>>>>>                         code and see what I can adapt.
>>>>>                         Our software basically uses the VirtualBox
>>>>>                         COM interface to create the needed
>>>>>                         machine, set the number of
>>>>>                         displays/resolutions, injects the
>>>>>                         IFramebuffer instances and powers up the
>>>>>                         machine via the IConsole interface. So I
>>>>>                         examined all the IFramebuffer
>>>>>                         implementations in VirtualBox frontends,
>>>>>                         and didn't see anything very special in
>>>>>                         the 3D related callbacks there. As for the
>>>>>                         log - I tried to compare the log of a
>>>>>                         session from our software with a session
>>>>>                         from the same machine under QT fronted
>>>>>                         where the 3D acceleration works. Basically
>>>>>                         there were no errors and differences, in
>>>>>                         both cases the logs contained the lines
>>>>>                         about OpenGL support (i guess this comes
>>>>>                         from the OpenGL capability test exe), I'll
>>>>>                         send both logs as soon as I get back to
>>>>>                         that machine but I doubt they'll give much
>>>>>                         help, since they don't have any errors,
>>>>>                         and in both of them the guest additions
>>>>>                         log line states that acceleration is
>>>>>                         supported. Basically the only thing that
>>>>>                         differs is that the D3D/OpenGL (we've
>>>>>                         tried both API's, since the test
>>>>>                         application is built on Unity it is
>>>>>                         trivial to force either mode) device
>>>>>                         creation fails on our frontend. That is
>>>>>                         why I asked about the get_WinId, I
>>>>>                         examined the miniport driver code and as
>>>>>                         far as I understood it uses the window id
>>>>>                         value to create a swap chain. Out frontend
>>>>>                         does not have an actual window, so I am
>>>>>                         always returning NULL, can this cause an
>>>>>                         error? I also saw that the miniport driver
>>>>>                         has a lot of WARN macros, is  the output
>>>>>                         visible anywhere? I tried using DebugView
>>>>>                         but that didn't help? Should I try
>>>>>                         building a debug version of guest
>>>>>                         additions to get some log information?
>>>>>                         Best Regards,
>>>>>                         Rudolfs Bundulis
>>>>>                         2015-01-26 10:34 GMT+02:00 Vadim Galitsyn
>>>>>                         <vadim.galitsyn at oracle.com
>>>>>                         <mailto:vadim.galitsyn at oracle.com>>:
>>>>>                             Hi Rudolfs,
>>>>>                             In order to provide 3D acceleration,
>>>>>                             VBox HGCM/OpenGL host service should
>>>>>                             be running. It is currently a part of
>>>>>                             VirtualBoxVM process (ShCrOpenGL
>>>>>                             thread). In order to provide 3D
>>>>>                             acceleration support for your
>>>>>                             frontend, I am afraid, you need to
>>>>>                             adapt stuff from this thread for your
>>>>>                             code. And I am afraid this might be
>>>>>                             not that easy (there is a bit of magic
>>>>>                             when VBox GUI conversates w/ 3D
>>>>>                             stuff). Please refer to
>>>>>                             src/VBox/HostServices/SharedOpenGL and
>>>>>                             src/VBox/GuestHost/OpenGL sources.
>>>>>                             Btw, if you describe a little bit more
>>>>>                             why do you need this and what you
>>>>>                             already have done, maybe I could
>>>>>                             explain you more things/steps you need
>>>>>                             to do in order to reach the goal.
>>>>>                             VBox.log also might be helpful.
>>>>>                             Vadim
>>>>>                             > On 24 Jan 2015, at 14:14, Rūdolfs
>>>>>                             Bundulis <rudolfs.bundulis at gmail.com
>>>>>                             <mailto:rudolfs.bundulis at gmail.com>>
>>>>>                             wrote:
>>>>>                             >
>>>>>                             > Hi,
>>>>>                             >
>>>>>                             > I'm having a bit of a hard time
>>>>>                             understanding why my custom
>>>>>                             framebuffer cannot use the 3D
>>>>>                             acceleration feature. I have a Windows
>>>>>                             7 guest machine (3D acceleration
>>>>>                             enabled, guest additions installed)
>>>>>                             and a Unity demo application. Under
>>>>>                             the default QT GUI the application
>>>>>                             launches fine, while under my frontend
>>>>>                             Unity is not able to create the D3D
>>>>>                             device. First I though that this could
>>>>>                             be due to incorrect implementation of
>>>>>                             IFramebuffer, but when I looked at
>>>>>                             UIFrameBuffer::Notify3DEvent(ULONG
>>>>>                             uType, BYTE *pData) in
>>>>>                             UIFrameBuffer.cpp there wasn't any
>>>>>                             special magic there. Are there any
>>>>>                             hints to guide me towards
>>>>>                             understanding the cause of this? Is
>>>>>                             there any tracing facilities for the
>>>>>                             guest miniport driver? Also, I don't
>>>>>                             have a window handle for my frontend
>>>>>                             since there is no actual window. If I
>>>>>                             am returning 0 in get_WinId in my
>>>>>                             framebuffer can that mess things up?
>>>>>                             >
>>>>>                             > Best Regards,
>>>>>                             > Rudolfs Bundulis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20150210/0d87c263/attachment.html>

More information about the vbox-dev mailing list