[feature-request] Nested Virtualization: VT-in-VT

[Technologov]

Hi All,

Because more and more virtualization software requires VT (Intel VMX or AMD-V) to run, it makes sense to virtualize it somehow. (at least if host CPU supports hardware VT)

It will allow the following software to run in guests: -Xen (full virtualization mode) -KVM -Hyper-V -Windows Virtual PC -- this is basically a requirement for those who wanna run Windows 7 "XP mode". i.e. It allows for more complete user experience on Win7 guests.

Others can take advantage of it too: -VirtualBox itself :) -VMware Workstation -VMware ESX

-Technologov (yes, I will take advantage of it too :-), yay! )

[Edit by klaus 2018-05-06:]

This feature is being worked on for a while. Once the implementation is complete and tested it will be available to all users.

[Technologov]

opened on 17.5.2009.

[sandervl73]

A lot of work for questionable usefulness. Definitely very low on our priority list.

[Technologov]

This is partially related to wish #2988

-Technologov

[Technologov]

KVM team has developed (untested) patches for both nested VMX and nested SVM. Maybe some ideas can be taken from those patches.

See: (General Info)

 http://avikivity.blogspot.com/2008/09/nested-svm-virtualization-for-kvm.html

and

 http://avikivity.blogspot.com/2009/09/nested-vmx-support-coming-to-kvm.html

-Technologov

[Technologov]

update: VMware Workstation 8 claims to support this feature !

 https://www.vmware.com/support/ws80/doc/releasenotes_workstation_80.html

-Technologov

[Technologov]

It seems that Linux 3.1 KVM finally achieved nested VMX.

This feature is yet another filler feature that is a "must-have":

 http://www.mjmwired.net/kernel/Documentation/virtual/kvm/nested-vmx.txt

Research paper:

 http://www.usenix.org/events/osdi10/tech/full_papers/Ben-Yehuda.pdf

-Technologov

[frank]

What you denote as "must have" needs a lot of implementation time, is very difficult to implement and the benefit of nested virtualization is questionable.

[frank]

I know that other virtualization products support nested virtualization. Even with VirtualBox it is possible to execute VirtualBox without VT-x/AMD-V as a guest of VirtualBox with VT-x/AMD-V enabled. But apart from that, implementing full virtualization is too much effort for too little benefit.

[itarchitectlev]

It seems that this ticket was evaluated a long time ago when the usefulness may have been disputed. In any case, there are real reasons people want to have nested hardware VT.

There is a lot of working happening in the cloud community and evaluating products in this arena often requires hardware VT - which implies two things:

1) You have real hardware available 2) You have nested hardware VT capable virtualization software (e.g. VMware)

Given that we are in a world of cheap memory coupled with multiple core fast CPUs, virtualization is a great boost to doing cloud infrastructure development work. The problem is that the virtualized hardware created under VBox is only one part of the picture - as you're creating virtual hardware to run infrastructure designed to virtualize. Nested Hardware VT is essential here as software emulation in this world is painfully slow.

I have just purchased a VMware product because VirtualBox doesn't support this fundamental feature that I require which will now render my use of VirtualBox redundant.

I have been a big advocate of VirtualBox, but future work now sees this being left behind.

Regards, Kev @itarchitectkev

[frank]

I don't want to start a discussion here, a public bugtracker is not the right place for doing this. But please be aware that the VirtualBox development is primary driven by paying customer demands. Of course that does not mean that community demands are not being considered. We just have limited resources. Code contribution from the community may speed up the implementation of certain features.

[Smackey's Dad]

Will nested VM work using "Intel Haswell CPU" which has VMCS feature allowing nested virtualization. If it does not it makes sense (in my opinion) to add support for nested VM with New CPU VMCS feature rather than writing the emulation layer.

[ramshankar]

No. Haswell CPUs while it has features like VMCS shadowing that help make Nested Virtualization perform better and easier to program, we still have to implement code which makes use of it. At the moment, we still feel when we implement the nested virtualization feature, having a Haswell CPU as the minimum requirement is not reasonable as it's still too new, and will exclude quite a bit of the users even two years down the line.

This feature is most definitely on our radar for a while now, but again, it's a question of managing our development resources which are limited.

[michael]

I have removed most of the comments from this ticket, as they were more of a discussion nature and not very conductive to implementation. I added the nicknames of the commenters to the CC list so that they still get updates to the ticket. Unfortunately trac does not like nicknames with spaces in in CC, so I just removed the content of those comments.

As mentioned previously, we currently do not have the available resources to implement this feature. If people in the community are willing and able to help we would certainly be interested in code contributions, but in that case please communicate with us before you start writing code to avoid wasted effort. If anyone in the community would be willing to be hired to work on this several people have indicated that they would be interested in trying to organise funding. Alternatively perhaps someone can find a suitable freelance developer. The same thing applies here: please talk to us before starting writing.

[Pro Tipper]

cc me too please?

[michael]

Pro Tipper: unfortunately the CC field can't handle user names with spaces in them. I will just leave your comment and that will cause you to get update notifications.

Current CC list is: Tao1, vbfun, DNS, Tristan, x5560, Tsso, bluezeak, peterdk, snowch, rdx, Bertrualex, EnesKorukcu, AfUnix, dario, anrichter, Jimbob, adam84luong, Coffee_fan, everflux, purpletoad, jonseymour, ciekawy, GA, ThatOneGuySean, Youness, subdian, veganaiZe, JohnStarich, Michal, todd.vierling, emiliano.bonassi, killmenot, ccarbone, azurtree, amclain, macedemo, GnomeUser, nicorac, schmunk, gael.abadin, paha, krasimir, pixie, Parfait, ikb42, yanp.bugz, hekier, maniankara, RayN, oam, domiel, sab3awy, Canada00, Ashishkel, leonexis, andjohn2000, denka_b, theBruno

[Gerry R]

cc me please

[Parkeren Schiphol]

Would really benefit from this for all my employees  https://parkos.nl/parkeren-schiphol we are all running w7 in xp mode due to a software compatibility issue. This should be a basic feature.

[argenstijn]

Could you increase priority! Nowadays this feature will come in handy.

[einar]

cc me please

[jwatilo]

Just want to add my voice -- we NEED nested virtualization. With the latest technologies this is becoming a MUST.

[socratis]

1. You don't "need" or "must have" nested virtualization, the vast majority of the world is doing great without it. You "would like" is more like it... ;)
2. Take a look at the timeline.
3. As Yoda once said:

   Patience you must have my young Padawan...

[mironex]

Replying to socratis:

1. You don't "need" or "must have" nested virtualization, the vast majority of the world is doing great without it. You "would like" is more like it... ;)
2. Take a look at the timeline.
3. As Yoda once said:

   Patience you must have my young Padawan...

Dear Socratis, Padwan is right. It's no longer a caprice. But an obvious need. Many tools, for example, for containerization are launched in a virtualbox.

That means that we really need to run a virtualbox in a virtualbox.

Maybe is possible to change the priorities. If you, Socratis, are so influential, you can change something.

If you do not have such an influence, well we must practice a Greek stoic patience or just use maybe e.g. KVM, VMWare.

Well, 9 years is a long time

Error: On virtual box is not able to start minikube

Starting local Kubernetes v1.9.0 cluster...
Starting VM...
E0214 22:14:50.003571    7064 start.go:159] Error starting host: Error creating host: Error executing step: Running precreate checks.
: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory.

 Retrying.
E0214 22:14:50.004044    7064 start.go:165] Error starting host:  Error creating host: Error executing step: Running precreate checks.
: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory

[mironex]

cc me too please?

[socratis]

@mironex
You didn't take a look at the timeline, did you?

[n00ris]

cc me please

[n00ris]

I opened the timeline, what in special shloud we see at the timeline?

I used my browser search to find "nested" in the page and I found

Changeset in vbox [71004] by vboxsync

VMM/HMSVMR0: Nested Hw.virt: Fix intercepting VMMCALL while executing the …

from 2018-02-14

Does this mean this feature is Work In Progress, but not yet in the latest release? Is there a beta available, or is it in a state that we could test if I build virtualbox directly from svn?

[maeni70]

I don't understand the timeline either. And i agree that this feature is important! E.g. Android Emulator needs VT-X. I wanted to use VirtualBox VM for mobile development, but as I learned just right know, it is limited for this purpose.

[maeni70]

cc me too please. thanks

[socratis]

The timeline simply tells you that fixes/patches are coming in all the time. It means that they're working on this feature. Hard. No, it's not ready, no, you didn't see any announcements. And keep in mind that yes, it may becoming an important feature, but a couple of years ago you didn't even have virtualization at all, so jumping from nothing to nested, that's quite a leap.

It will become available. Just (like everything else in life), don't ask for promises or specific dates. The source code is available, if you can't contribute, at least you can monitor it to see if they're working on it. And they are...

[paleozogt]

+1 cc me also

[jotenko]

I'm also waiting desperately for this. CC me too, please.

[vbkamfung]

As docker is becoming common, this request should be changed from 'want to' to 'must have'.

[michael]

Based on the number of code contributions we have seen, or the number of people expressing interest in contributing (I think none at all, but I may have missed one or two) this feature cannot be in very high demand. Be that as it may, it is being worked on.

[Darren Mackay]

Just want to add my support for this.

This past week - I now have 3 clients (2x UK and 1 x NL) who are deploying Windows Containers as standard - 1 of these clients has 3000 users (also a large Oracle JDE house). the requirement to run windows as a VM is mandatory - due to lockdown by the customer requirements - and thus containers within the locked down windows image (also includes VPM clients / etc).

Note - Running virtualbox on Mac as a consultant.

[brexit]

I occasionally need to run KVM and in order to do that I must stop VBox VMs, disable VBox, enable KVM, reboot, and revert 1-2 days later. And of course migrate VBox VMs to KVM, if I need any of them on KVM.

Test/dev, containers, compatibility requirement (for KVM-specific ISVs), etc - there are many reasons why it is desirable.

[socratis]

@brexit

Your first requirement is not about nested virtualization, it's about concurrent virtualization. And unfortunately, KVM (just like Hyper-V) doesn't want to play nice.

The trick with VT-x is that it has to be shared, since it's "only one" available slot for its use. VMware and VirtualBox do not use VT-x at the same time but rather take turns. Simply put, programs that use VT-x should do this: VT-x lock, execute, VT-x unlock. This way other processes can use VT-x in turns.

On the other hand, KVM and Hyper-V permanently take over VT-x and don't let anyone else use it. If a program uses VT-x from the moment it is loaded and doesn't let go till the moment it dies, you have a problem. So, please complain to the KVM and Hyper-V people about that.

[Stef2]

Raising the count for VT-in-VT feature. Main use case is also docker.

[klaus]

Replying to Stef2:

Raising the count for VT-in-VT feature. Main use case is also docker.

Can anyone explain why VT-in-VT is so important for users of Docker? Docker relies on containers, which is a technology not requiring virtualization.

Of course one can run Docker inside a VirtualBox VM today, so what am I missing?

[zappacor]

+1, needing this ASAP. @Klaus: any ETA for it yet?

[socratis]

Replying to zappacor:

+1, needing this ASAP.

Can you for the life of me explain to me why this is an ASAP request? What in the seven kingdoms are you trying to do that requires this ... yesterday? What's your usage scenario that requires VT-x in a VM?

@Klaus: any ETA for it yet?

I can give you one, and maybe Klaus will prove me wrong: No ETA. There is *never* an ETA. For *anything*/*everything*.

(it's not just the VirtualBox team, it's any developer with a sense of self-respect)

[hamerins]

+1 cc me also

[JurgenO]

+1 please.

Out of respect for all of us here.

Many of us here have benefited from the "FREE" and very useful technologies for our work.

Which many have had contributed their time and effort to make this happened.

And more, we have yet to spent any effort for this great software, we only make use of this for FREE.

Appreciation and respect should come to mind even before making any request.

As much as I also need to have this feature for my Android-SDK development usage, I can only make this to be heard as a humble request and NOT a DEMAND.

Definitely no ETA, just hope.

Cheers!

[Mixim]

Dear developers of VirtualBox, this requested feature will be very usefull for all software developer, which use .NET Core and Docker with it. This ticket is 9 year old, may be you can implement it now?

[passionsplay]

Replying to klaus:

Hey Klaus, thank you for taking the time to comment on this really old ticket!

Can anyone explain why VT-in-VT is so important for users of Docker? Docker relies on containers, which is a technology not requiring virtualization.

You're right that Docker doesn't necessarily require a VM to operate. Since Docker is a tech used to run Linux containers, it doesn't "need" to be run in a VM. That being said, for development reasons, in order to use Docker on a Mac or Windows host, you are required to have a Linux VM to run your Docker containers on.

My use case as developer and support engineer is to be able to see and replicate issues for Windows users of an app that makes use of Docker containers within a virtual machine.

The installation process of this app is to download Virtualbox and use docker-machine to create a sandboxed Guest VM to run docker containers within. These are basically mini websites that can be used and deployed anywhere Docker containers can be run. You can think of the structure of this app as being something like:

• Host (Windows or Mac)
  • VM created by docker-machine using boot2docker
    • docker container 1
    • docker container 2
    • ... etc...

I currently use a Mac as my main Host OS, but for testing what our end users are experiencing, we need to be able to spin up different versions of Windows. Inevitably, a Windows update will break something, so having snapshots and being able to roll back to different versions for testing and troubleshooting is important. For me, that structure would ideally look like:

• Mac (my laptop)
  • Host Windows 10 Home
    • VM created by docker-machine
      • ... Docker containers ...
  • Host Windows 10 Pro
    • VM created by docker-machine
      • ... Docker containers ...
  • Host Windows 10 v123_pre_fubared_update
  • ... more Windows guests ...

Right now, I can't do that because the docker-machine VM requires nested virtualization -- as a result, I can't test using Virtualbox. Currently, we can use Parallels Pro to get this nested virtualization. Ideally, this would be something that was available in an open-source product, but I understand how difficult things can be to create.

Performance is less of a concern for these things, whereas the ability to replicate issues as well as having disposable environments is the main goal.

I mainly wanted to share my use case, and give a concrete rationale for why this sort of thing is important. Thanks for your time!

[tylerbyte]

Hey Guys.

Docker tutorial follower here:  https://docs.docker.com/get-started/part4/#set-up-your-swarm

I am currently running everything on my host system (Windows 10) Pro. I have Vbox 5.2 & Ubuntu Server 16.04 installed on the host, The Ubuntu Server VM runs docker and Vbox 5.2 as well.

I thought I would also need the nested virtualization as discussed here but... (Knowing the CPU VT-X feature is in-fact enabled in the bios) it seems it won't help adding this feature to vbox at all.

Due to an issue with the command:

docker-machine create --driver virtualbox myvm1

Error: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory.

I ran this command:

docker-machine create --virtualbox-no-vtx-check --driver virtualbox myvm1

Which completely by-passed the Nested Virtualization Issue. But now it is stuck at
"(myvm1)Awaiting an IP"....

Any thoughts on that issue then we may have a solution to the docker guys needing this feature as it would save you guys a lot of time.

Thanks :)

UPDATE
After installing UbuntuServer 16.04 LTS fresh on brand new desktop with the VTX feature enabled, my waiting for in ip issue disappeared, virtual box has given me zero issues.

Maybe fix a nested host adapter make it bridge to the original network, might be easier than nested cpu virtualization :)

Hope this helps somebody! Lotsa luck coding!

[brexit]

Replying to socratis: @socratis

Your first requirement is not about nested virtualization, it's about concurrent virtualization. And unfortunately, KVM (just like Hyper-V) doesn't want to play nice.

That's one way to think about it, but if I could nest KVM or ESXi inside of VirtualBox, I wouldn't need to run it alongside VirtualBox.

[kulio]

So, my suggestion would be, at least as a first step that would seem relatively easier, expose the virtualbox api somehow to the guest. That is basically what they did with docker in docker, which accomplishes the same thing as nested virtualization  https://github.com/jpetazzo/dind.

This would allow kubernetes and docker to work, and I would imagine you could use some of the same code for the different architectures.

[syadnom]

Another key use case is GNS3, which requires KVM for some systems, which in turn requires nested virtualization.

Almost all use cases I can come up with are for test beds.

How about mocking up a proxmox cluster? This can be done with nested virtualization in vmware, but i/we all hate vmware and want to do this in virtualbox.

[wolfeman2120]

Hi All, Just wanted to put in an additional request for this so it gets to the top of the priority list.

My use cases are similar to everyone else. Looking to be able to run minikube in a VM for testing builds and deployments. would be nice to use this to teach docker and kubernetes for managed deployments. Now a days home PC's running x99 and x299 chipsets can easily handle this at least hardware wise.

[Darryn Brooking]

FTR;

I use VMs *a LOT*; there are simply too many development setups that interact badly - or simple need different version of the same software - to have everything install on the host at once. I'm moving to VMWare because of the lack of this feature because Docker and Android development.

[AR0x7E7]

+1 cc

[Tom B]

I'm going to +1 here but only because you were asking for use-cases. I don't *need* this feature but it would be nice to have.

I maintain several development environments that use vagrant/virtualbox. For the most part, it's fine but I have had several issues in the past where a VM configuration works on Windows 10 but not 7, or works on a Linux host but not on Windows.

For example. I had an issue where a vagrant box worked fine on Linux where I built and tested it but when someone tried it on Windows there was an issue where VirtualBox did not have the "Cable connected" box ticked in networking so had no network connection.

The issues are generally with vagrant but differences between host OS have caught me out on more than one occasion.

Since then I've been testing my development environments on different hosts (Windows 7, 10 and Linux) to find these issues before making the vagrant boxes available. It would be a lot easier if I could test my development environments on different hosts OSs inside VirtualBox VMs. Probably not a common use-case but it would make my life easier if this feature existed.

[Friis]

+1 My usecase for this feature is: I am running unraid with windows 10 and uses it like people normally would as a "host". At work I use virtualbox as a test machine, in order to do the same home, I see I need this feature. So unraid host, windows 10 virtual machine running virtualbox with windows 7/10.

[n00ris]

Let me join and describe my use-case, too: I want to build this  https://markelov.blogspot.com/2017/05/how-to-configure-openstack-tripleo.html but the host should be a virtualbox itself. In short this is a openstack tripleo deployment to test, train and development purpose. Host is a linux in virtualbox, in it some virtual machine with virtualbmc that simulate the hardware server from production. Thus we can test the director deployment with the ironic ipmi driver as it is used in production. Inside the director we should not see any difference, since we deploy the director with ansible, we should be able to deploy the virtual director in this test setup with the same code without adaptions.

To use a virtualbox as host has now advantage: I can safe the state or I can hand out the image after basic setup, such that someone can learn the director deployment and tripleo update procedure. Now, native VT-in-VT support hopefully speed things up. ( e.g. is it even possible to start an instance in the openstack for basic tests of new feature/openstack service we activated with the last deployment, VT-in-VT-in-VT..., so it really helps if at least [VT-in-VT]-in-VT is optimized )

Thanks a lot in advance.

[eduardolucioac]

VirtualBox is currently the platform of choice for infrastructure development and simulation/test in our company. We abandoned VMWare as soon as we found a solution to this problem  https://forum.manjaro.org/t/manjaro-and-virtualbox-host-only-with-internet/28722/12 - which we deem an unnecessary limitation, sorry. We are currently having trouble testing infrastructures with Hypervisor which is an important work front in our company that has several contributions to open source solutions. Note that this ticket is already 9 years old and with so many requests I do not believe this is a "priority: minor" request. I believe that these arguments are very pertinent. Many thanks and please do not get mad!

[Chipper02]

It sure would be nice to have a multi-os hypervisor that supports nested virtualization on Linux, Windows, and MacOS. It would make development environments highly portable and host OS agnostic. Instead I have to tweak things for Xen, KVM, Hyper-V, and hyperkit depending on what project we are working on. Waiting 9 years for a feature the all other modern hypervisors have had for years makes it hard to take VirtualBox seriously anymore. I have uninstalled it from all our systems and will not look back until the DEVs start taking this issue seriously.

[maeni70]

Any news? There was silence the last 2 months. I really hope too that this issue is taken seriously.

[socratis]

Replying to maeni70:

There was silence the last 2 months.

2 months??? Have you seen the date of the ticket? (1)

And if you think this is a one- or a two-month effort, you wouldn't even be in the same ballpark. This is a multi-year effort! In a couple of years (not "Olympic" years) I think it's going to be a real feature... ;)

---

(1): Hint... 2009-05-17, 9 1/2 years ago

[ramshankar]

Replying to maeni70:

Any news? There was silence the last 2 months. I really hope too that this issue is taken seriously.

See Klaus' post here:  https://forums.virtualbox.org/viewtopic.php?p=431797#p431797

Nested AMD-V is functional (although not feature complete). Nested VT-x is still work-in-progress. Hence no major announcement was made.

Also, it's incorrect to assume this feature was 9 years in the making because this ticket is 9 years old (I know socratis didn't mean that, but just clarifying before any misinterpretations are made).

[maeni70]

Replying to ramshankar:

See Klaus' post here:  https://forums.virtualbox.org/viewtopic.php?p=431797#p431797

Nested AMD-V is functional (although not feature complete). Nested VT-x is still work-in-progress. Hence no major announcement was made.

Thanks for the information! Good to know that there are activities.

[madscientist_42]

Sorry Oracle...

A 9-10 year old ticket is STILL a hang your head in shame guys, especially when most of the virtualization solutions HAVE this and more. You used to be the best...you still have a lot of what it takes if you'd take it more seriously... This isn't it.

[socratis]

@madscientist_42

1. The age of a ticket has absolutely nothing to do with the whole thing.

2. You haven't checked the newly released VirtualBox 6.0.0, right? You should be doing your homework before engaging the rage mode...

[Tom B]

I'm not sure if this is the best place to report this but nested AMD-V is not working.

I can check the box "Enable Nested VT-x/AMD-V" but with that checked the guest OS freezes on startup.

CPU: AMD Threadripper 1950x Host: Arch Linux Guest: Windows 10 (64bit)

The guest OS freezes on the spinning loading screen.

[socratis]

@Tom B

I've opened a thread in the forums, have a look at it:  Nested virtualization: Success stories (and failures).

Do share... ;)

[NeccoNeko]

Please CC me as well (or leave this comment). Thanks!

[0xs4ni]

[ignore this comment, read through docs, didn't have --nested-hw-virt available; current dist. was outdated. Upgraded, moving to Success/Failure stories thread]

[cremor]

+1 Please add support for Intel CPUs!

[Harvey ManfredStJohnsen]

Ugh, I wish I'd read the comment about spaces in nicks before creating mine. Apologies for the comment noise, just commenting so I get cc'd.

[Parkeren Schiphol Nederland]

Would really need this for my employees at  https://comfortparking.nl we are all running MS windows 10 in xp mode due to a software compatibility issue. This would be a perfect solution.

[ramshankar]

Hardware-assisted Nested virtualization on Intel CPUs has been available starting with VirtualBox 6.1.0

Closing this ticket since the feature requested has been added.

[Grunthos]

Any chance of nesting being made available on less advanced Intel CPUs? (VMCS shadowing is a 'pro' feature).

[chrisgayle]

Cette amélioration était un must, fonctionnant comme un charme. Utilisé ici pour trouver les réponses au jeu de 8 lettres sur  https://www.4images1motsolution.info/8-lettres/