[vbox-dev] Linux Host VM Cap?
Frank Mehnert
frank.mehnert at oracle.com
Tue May 10 18:46:04 GMT 2011
Hi Zak,
we have prepared this test build for you:
http://www.virtualbox.org/download/testcase/virtualbox-4.0_4.0.7-71651~Ubuntu~maverick_amd64.deb
Kind regards,
Frank
On Sunday 08 May 2011 13:41:01 Knut St. Osmundsen wrote:
> Hi Zak,
>
> I've raised the max VM limit to 8191 in r36988). We can provide a test
> build if you are unable to build from SVN, just let us know.
>
> Kind Regards,
> Knut
>
> On May 4, 2011, at 12:41 AM, Zak Perschon wrote:
> > I had to restart the machine running all of the VMs, so I decided to
> > increase the amount of files per process like you suggested, relaunched,
> > and we are now hitting the projected 1023 virtual machine cap! Here is
> > the error that VboxManage pushes out if anyone is interested (A vbox.log
> > file was also created for the VM):
> >
> > VBoxManage: error: VM creation failed (GVMM) (VERR_GVM_TOO_MANY_VMS)
> > VBoxManage: error: Unknown error creating VM (VERR_GVM_TOO_MANY_VMS)
> > VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component
> > Console, interface IConsole, callee
> >
> > Now that we made it this far, I guess the next step would be attempting
> > to remove the 1023 machine limit that virtualbox is imposing on us?
> > Thanks again for the help, its been very uplifting to have something
> > actually show progress (bashing our heads against a slew of different
> > virtualization solutions has been exhausting).
> >
> > -Zak
> >
> > > Date: Tue, 3 May 2011 20:39:14 +0200
> > > From: klaus.espenlaub at oracle.com
> > > To: vbox-dev at virtualbox.org
> > > Subject: Re: [vbox-dev] Linux Host VM Cap?
> > >
> > > On 03.05.2011 17:39, Zak Perschon wrote:
> > > > After letting my mass-deploy script complete, we seem to have hit a
> > > > cap at 1010 running VMs (A heck of a lot better than 128!). There
> > > > doesn't seem to be a log created for the 1010th VM that failed, but
> > > > here is the error message that I get when attempting to start it:
> > > >
> > > > VBoxManage: error: The Virtual Machine 'TinyCore-1011' has terminated
> > > > unexpectidly during startup with exit code 1
> > > > VboxManage: error: Details: code NS_ERROR_FAILURE (0x80004005),
> > > > component Machine, interface IMachine, callee
> > > >
> > > > Any ideas as to where to go from here?
> > >
> > > Of course :) It looks like you're running into the maximum number of
> > > open files _per process_ now, which defaults to 1024 on most Linux
> > > distros. The critical component in this respect is the VBoxXPCOMIPCD
> > > process (automatically started). This one is the central message broker
> > > for API requests. Since it also has a couple of files open the limit of
> > > API clients is a bit lower than 1024.
> > >
> > > Not sure if it's worth the effort to get 13 VMs more, but if you want
> > > to get to the current hardcoded maximum you have to stop all VMs
> > > again. Please double check with the "ps" command that VBoxXPCOMIPCD
> > > has terminated (should happen approx. 5 seconds after the last API
> > > client is gone).
> > >
> > > Then set a higher limit either straight with ulimit -n 2048, or
> > > following the descriptions (the comments are interesting as well) in
> > > http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-
> > > files/
> > >
> > > > The current system resource usage is as follows:
> > > >
> > > > CPU Usage: 10~% average idling, 20~% when we are utilizing all of the
> > > > 1010 machines.
> > >
> > > This sounds very good. Must be a really stripped down distribution
> > > which doesn't do much user friendly (i.e. CPU wasting) background
> > > activities. Another reason for the low utilization might actually be
> > > lock
> > > contention, as all those VMs need to coordinate their activities to
> > > some degree, and that could cause delays. We'd have some ideas what to
> > > change if this really happens.
> > >
> > > > Memory: 146.8GB of 2019.9GB (which seems low, we have 1010 machines
> > > > using 150MB of ram and 8 MB of vram, but we are getting replies from
> > > > all of the machines)
> > > > Disk Space: 48.6GB of 1.3TB (The machine is supposed to have an
> > > > attached SAS drive, but its still on order)
> > >
> > > The memory usage is suspiciously low. There could be a reason though -
> > > VirtualBox allocates VM memory lazily. If a VM doesn't touch all its
> > > memory then it will be below the expected value.
> > >
> > > > The original goal of 1800 was our "conservative" estimate, we are
> > > > actually aiming to get around 4,000 if possible. So far, Virtualbox
> > > > has been the only VM solution we've tried that has gotten us this
> > > > close (most either have a cap at around 300~ VMs that can't
> > > > increase, or they don't support our host machine's hardware and
> > > > crash). If you'd like, once we are finished with this I can provide
> > > > a bit more detail on the VM solutions we've tried and the
> > > > limitations we've ran in to.
> > >
> > > For over 1023 it needs some code tweaking... we'll help you with that
> > > of course. Don't know yet when our expert can spare a few minutes for
> > > looking into this.
> > >
> > > Klaus
> > >
> > > > -Zak
> > > >
> > > > > Date: Tue, 3 May 2011 17:00:55 +0200
> > > > > From: klaus.espenlaub at oracle.com
> > > > > To: zakperschon at hotmail.com
> > > > > CC: knut.osmundsen at oracle.com
> > > > > Subject: Re: [vbox-dev] Linux Host VM Cap?
> > > > >
> > > > > Hi Zak,
> > > > >
> > > > > On 03.05.2011 00:29, Zak Perschon wrote:
> > > > > > Hello,
> > > > > >
> > > > > > Here is a quick update for you (thanks a ton for the help by the
> > > > > > way!):
> > > > > >
> > > > > > I've been able to breach the 128 cap we had before w/ the fix you
> > > > > > suggested, but have ran into some "user error issues" that have
> > > > > > caused me to have to re-deploy/restart my VMs. The highest I've
> > > > > > gotten so far is 675 running VMs, but I'm sure I'll be able to
> > > > > > hit the 1023 mark. It takes a uprising amount of time to
> > > > > > deploy/start 1000 VMS....
> > > > >
> > > > > Keeping fingers crossed... with 675 VMs, 150MB RAM each (how much
> > > > > VRAM, I guessed 16MB?), you're around 280GB of RAM used by all of
> > > > > them, using the rough rule of thumb that RAM usage of a VM is
> > > > > approx.
> > > > > 1.02*(RAM+VRAM)+250MB. Varies greatly with the VM config details of
> > > >
> > > > course.
> > > >
> > > > > With your monster gear having 1.8TB of free RAM the theoretical
> > > > > maximum would be around 4300 of the tiny VMs you outlined
> > > > > (ignoring the disk and CPU capacity).
> > > > >
> > > > > I'm working on updating the manual. The information for Solaris is
> > > > > already there (in chapter 9), and the Linux information should also
> > > > > go there.
> > > > >
> > > > > Just out of curiosity - are other virtualizers significantly faster
> > > > > with such insanely high VM counts? It's really a very exotic use
> > > > > case.
> > > > >
> > > > > BTW, we'd appreciate keeping such threads on the mailing list -
> > > > > they show that VirtualBox is a serious virtualization solution,
> > > > > and also help others if they run into such extreme situations.
> > > > >
> > > > > Klaus
> > > > >
> > > > > > -Zak
> > > > > >
> > > > > > > Date: Mon, 2 May 2011 19:01:21 +0200
> > > > > > > From: klaus.espenlaub at oracle.com
> > > > > > > To: vbox-dev at virtualbox.org
> > > > > > > Subject: Re: [vbox-dev] Linux Host VM Cap?
> > > > > > >
> > > > > > > On 01.05.2011 17:56, Knut St. Osmundsen wrote:
> > > > > > > > Hi Zak!
> > > > > > > >
> > > > > > > > The VMM imposes a limit of 1023 VMs on 64-bit hosts and 127
> > > > > > > > VMs on
> > > >
> > > > > > > > 32-bit hosts. See:
> > > > http://www.virtualbox.org/browser/trunk/src/VBox/VMM/VMMR0/GVMMR0.cpp
> > > > #L121
> > > >
> > > > > > > That means your plan to have 1800 VMs will not work without
> > > >
> > > > bumping the
> > > >
> > > > > > > hardcoded limits, and in this area it needs much more thinking
> > > >
> > > > than just
> > > >
> > > > > > > changing a #define.
> > > > > > >
> > > > > > > Anyway, let's get you to that limit first...
> > > > > > >
> > > > > > > If you run "ipcs -ls" you'll get a "max number of arrays = 128"
> > > > > > > line, which is what you're most likely bumping into.
> > > > > > >
> > > > > > > You can read the kernel defaults with "sysctl kernel.sem" as
> > > >
> > > > well, and
> > > >
> > > > > > > the last number needs to be increased to a bit more than 1023,
> > > >
> > > > just to
> > > >
> > > > > > > be on the safe side if some other app also needs a few
> > > > > > > semaphores.
> > > > > > >
> > > > > > > "sysctl -w kernel.sem=250 32000 32 1200" should eliminate this
> > > > > > > limit. You can also put this in /etc/sysctl.conf, so that it
> > > > > > > is set
> > > >
> > > > straight on
> > > >
> > > > > > > system boot, as otherwise these settings are lost on a reboot.
> > > > > > >
> > > > > > > So... how many VMs can you start now?
> > > > > > >
> > > > > > > Klaus
> > > > > > >
> > > > > > > > Seeing you have 2TB of memory and 128 cores, I would assume
> > > > > > > > you're running 64-bit Ubuntu. So, it is probably some other
> > > > > > > > limit that
> > > >
> > > > you're
> > > >
> > > > > > > > hitting. It could be SysV semaphores, it could be memory
> > > > > > > > available
> > > > > >
> > > > > > below
> > > > > >
> > > > > > > > 4GB, ... Could you please provide some more details regarding
> > > > > > > > what happens when you are unable to launch more VMs? For
> > > > > > > > instance, does
> > > > > >
> > > > > > a new
> > > > > >
> > > > > > > > VBox.log file get created for the VM? If it does, please zip
> > > > > > > > it
> > > >
> > > > up and
> > > >
> > > > > > > > send it to me (or Klaus) per private mail and we'll work it
> > > > > > > > out.
> > > > > > >
> > > > > > > I think the 64bit host OS assumption is safe... I would
> > > > > > > seriously ask people to have their brain inspected if they'd
> > > > > > > use a 32bit OS for
> > > >
> > > > this.
> > > >
> > > > > > > > On Apr 29, 2011, at 7:43 PM, Zak Perschon wrote:
> > > > > > > >> Ah, sorry forgot to include the original support thread. The
> > > >
> > > > limit I
> > > >
> > > > > > > >> hit was 128 active VMs at once. Once I hit that number, I
> > > > > > > >> was
> > > >
> > > > unable
> > > >
> > > > > > > >> to launch any more unless I stopped one that was already
> > > >
> > > > running. My
> > > >
> > > > > > > >> original thread can be found here (though there is little
> > > >
> > > > information
> > > >
> > > > > > > >> contained in
> > > > > > > >> it):http://forums.virtualbox.org/viewtopic.php?f=7&t=40955
> > > > > > > >> <http://forums.virtualbox.org/viewtopic.php?f=7&t=40955>
> > > > > > > >>
> > > > > > > >> -Zak
> > > > > > > >>
> > > > > > > >> > Date: Fri, 29 Apr 2011 16:25:44 +0200
> > > > > > > >> > From:klaus.espenlaub at oracle.com
> > > >
> > > > <mailto:klaus.espenlaub at oracle.com>
> > > >
> > > > > > > >> > To:vbox-dev at virtualbox.org
> > > > > > > >> > <mailto:vbox-dev at virtualbox.org> Subject: Re: [vbox-dev]
> > > > > > > >> > Linux Host VM Cap?
> > > > > > > >> >
> > > > > > > >> > On 25.04.2011 18:00, Zak Perschon wrote:
> > > > > > > >> > > I've come across what is seemingly a hard cap for the
> > > >
> > > > amount of
> > > >
> > > > > > > >> > > concurrent VMs I can run on a Linux based machine (after
> > > >
> > > > posting
> > > >
> > > > > > > >> on the
> > > > > > > >>
> > > > > > > >> > > support forums, I was directed here). Apparently this
> > > > > > > >> > > same
> > > >
> > > > cap was
> > > >
> > > > > > > >> > > encountered on a solaris host and a fix was deployed to
> > > > > >
> > > > > > increase the
> > > > > >
> > > > > > > >> > > amount of VMs that were allowed to run at the same time.
> > > > > > > >> > > Was
> > > > > >
> > > > > > this fix
> > > > > >
> > > > > > > >> > > supposed to be included in other OS releases? Any
> > > > > > > >> > > information
> > > > > >
> > > > > > on this
> > > > > >
> > > > > > > >> > > would be greatly appreciated.
> > > > > > > >> >
> > > > > > > >> > That's a rather vague statement, and if you could tell
> > > > > > > >> > what
> > > > > >
> > > > > > number of
> > > > > >
> > > > > > > >> > concurrently running VMs you can achieve I might be able
> > > > > > > >> > to
> > > > > >
> > > > > > infer what
> > > > > >
> > > > > > > >> > limit is causing trouble for you.
> > > > > > > >> >
> > > > > > > >> > It could be the SysV semaphore count, the number of open
> > > >
> > > > files per
> > > >
> > > > > > > >> > process, and so on. All these defaults depend on the
> > > > > > > >> > kernel configuration and/or distribution, so it's hard to
> > > > > > > >> > generalize.
> > > > > > > >> >
> > > > > > > >> > Solaris is easy compared to that, as it is very uniform
> > > >
> > > > compared to
> > > >
> > > > > > > >> Linux.
> > > > > > > >>
> > > > > > > >> > Klaus
> > > > > > > >> >
> > > > > > > >> > > Here is the information on the hardware/VM profiles I am
> > > > > > > >> > > using
> > > > > > > >> > >
> > > > > > > >> > > Host Machine:
> > > > > > > >> > > 128 Processor Cores (64 dual core Xenon processors
> > > > > >
> > > > > > running at 2.27 GHz)
> > > > > >
> > > > > > > >> > > 2.0 TB Memory (1.8~ Usable)
> > > > > > > >> > > 1.8 TB RAID 0 (6 x 300GB HDD running at 10k RPM)
> > > > > > > >> > > 16TB SAS
> > > > > > > >> > >
> > > > > > > >> > > VM Profile
> > > > > > > >> > > Compiled TinyCore Linux 2.6
> > > > > > > >> > > Required Memory: 150~MB
> > > > > > > >> > > Required HDD Space: 50MB
> > > > > > > >> > >
> > > > > > > >> > > -Zak
> > > > > > > >> >
> > > > > > > >> > _______________________________________________
> > > > > > > >> > vbox-dev mailing list
> > > > > > > >> >
> > > > > > > >> >vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
> > > > > > > >> >http://vbox.innotek.de/mailman/listinfo/vbox-dev
> > > > > > > >>
> > > > > > > >> _______________________________________________
> > > > > > > >> vbox-dev mailing list
> > > > > > > >> vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
> > > > > > > >> http://vbox.innotek.de/mailman/listinfo/vbox-dev
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Kind regards / Mit freundlichen Gruessen / Vennlig hilsen,
> > > > > > > > bird
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > ORACLE Deutschland B.V. & Co. KG Knut St. Osmundsen
> > > > > > > > Werkstrasse 24 Senior Staff Engineer, VirtualBox
> > > > > > > > 71384 Weinstadt, Germany mailto:bird at sun.com
> > > > > > > >
> > > > > > > > Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> > > > > > > > Registergericht: Amtsgericht Muenchen, HRA 95603
> > > > > > > >
> > > > > > > > Komplementaerin: ORACLE Deutschland Verwaltung B.V.
> > > > > > > > Rijnzathe 6, 3454PV De Meern, Niederlande
> > > > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr.
> > > > > > > > 30143697 Geschaeftsfuehrer: J. Kunz, M. van de Molen, A. van
> > > > > > > > der Ven
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > vbox-dev mailing list
> > > > > > > > vbox-dev at virtualbox.org
> > > > > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev
> > > > > > >
> > > > > > > --
> > > > > > > Oracle <http://www.oracle.com>
> > > > > > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop
> > > > > > > Virtualization
> > > > > > > Phone: +49 7151 60405 205 <tel:+49715160405205>
> > > > > > > Oracle VM VirtualBox
> > > > > > >
> > > > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384
> > > > > > > Weinstadt
> > > > > > >
> > > > > > > ORACLE Deutschland B.V. & Co. KG
> > > > > > > Hauptverwaltung: Riesstr. 25, D-80992 München
> > > > > > > Registergericht: Amtsgericht München, HRA 95603
> > > > > > >
> > > > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V.
> > > > > > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> > > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr.
> > > > > > > 30143697 Geschäftsführer: Jürgen Kunz, Marcel van de Molen,
> > > > > > > Alexander van
> > > >
> > > > der Ven
> > > >
> > > > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is
> > > >
> > > > committed to
> > > >
> > > > > > > developing practices and products that help protect the
> > > > > > > environment
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > vbox-dev mailing list
> > > > > > > vbox-dev at virtualbox.org
> > > > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev
> > > > >
> > > > > --
> > > > > Oracle <http://www.oracle.com>
> > > > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop
> > > > > Virtualization
> > > > > Phone: +49 7151 60405 205 <tel:+49715160405205>
> > > > > Oracle VM VirtualBox
> > > > >
> > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt
> > > > >
> > > > > ORACLE Deutschland B.V. & Co. KG
> > > > > Hauptverwaltung: Riesstr. 25, D-80992 München
> > > > > Registergericht: Amtsgericht München, HRA 95603
> > > > >
> > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V.
> > > > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> > > > > Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van
> > > > > der Ven
> > > > >
> > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is committed
> > > > > to developing practices and products that help protect the
> > > > > environment
> > > >
> > > > _______________________________________________
> > > > vbox-dev mailing list
> > > > vbox-dev at virtualbox.org
> > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev
> >
> > _______________________________________________
> > vbox-dev mailing list
> > vbox-dev at virtualbox.org
> > http://vbox.innotek.de/mailman/listinfo/vbox-dev
--
ORACLE Deutschland B.V. & Co. KG Dr.-Ing. Frank Mehnert
Werkstrasse 24 Staff Engineer, VirtualBox
71384 Weinstadt, Germany mailto:frank.mehnert at oracle.com
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20110510/25b59679/attachment.sig>
More information about the vbox-dev
mailing list