[vbox-dev] Linux Host VM Cap?
Zak Perschon
zakperschon at hotmail.com
Thu May 12 05:24:19 GMT 2011
I VPN'd into our system to check the progress, and it finally seems to have hit an error at VM 1498.
The following error occurs whenever I attempt to start a new VM (with 1497 currently running):
VBoxManage: error: The Virtual machine 'TinyCore-1498' has terminated unexpectedly during startup with exit code 1
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component Machine, interface IMachine, callee
I'll investigate this error a bit more when I get into the office tomorrow.
-Zak
From: zakperschon at hotmail.com
To: vbox-dev at virtualbox.org
Date: Wed, 11 May 2011 15:50:04 -0600
Subject: Re: [vbox-dev] Linux Host VM Cap?
Hi,
Thanks for the build, we are currently at 1150 VMs and counting! One thing we noticed though is that launching the VMs this time around seems to be significantly slower. Before, it took us 40~ minutes to launch all 1023 VMs, and with this current build, it took about 2 and a half hours to get back to 1023. We are seeing a variance of between 5 and 45 seconds to start each VM (closer to the 30-45 second mark), whereas before it was pretty stable around 2-5 seconds per VM. Some things may have changed with our SAS that I am investigating, but I'm not entirely convinced that is what is causing the slowdown.
-Zak
From: frank.mehnert at oracle.com
To: vbox-dev at virtualbox.org
Date: Tue, 10 May 2011 20:46:04 +0200
Subject: Re: [vbox-dev] Linux Host VM Cap?
Hi Zak,
we have prepared this test build for you:
http://www.virtualbox.org/download/testcase/virtualbox-4.0_4.0.7-71651~Ubuntu~maverick_amd64.deb
Kind regards,
Frank
On Sunday 08 May 2011 13:41:01 Knut St. Osmundsen wrote:
> Hi Zak,
>
> I've raised the max VM limit to 8191 in r36988). We can provide a test
> build if you are unable to build from SVN, just let us know.
>
> Kind Regards,
> Knut
>
> On May 4, 2011, at 12:41 AM, Zak Perschon wrote:
> > I had to restart the machine running all of the VMs, so I decided to
> > increase the amount of files per process like you suggested, relaunched,
> > and we are now hitting the projected 1023 virtual machine cap! Here is
> > the error that VboxManage pushes out if anyone is interested (A vbox.log
> > file was also created for the VM):
> >
> > VBoxManage: error: VM creation failed (GVMM) (VERR_GVM_TOO_MANY_VMS)
> > VBoxManage: error: Unknown error creating VM (VERR_GVM_TOO_MANY_VMS)
> > VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component
> > Console, interface IConsole, callee
> >
> > Now that we made it this far, I guess the next step would be attempting
> > to remove the 1023 machine limit that virtualbox is imposing on us?
> > Thanks again for the help, its been very uplifting to have something
> > actually show progress (bashing our heads against a slew of different
> > virtualization solutions has been exhausting).
> >
> > -Zak
> >
> > > Date: Tue, 3 May 2011 20:39:14 +0200
> > > From: klaus.espenlaub at oracle.com
> > > To: vbox-dev at virtualbox.org
> > > Subject: Re: [vbox-dev] Linux Host VM Cap?
> > >
> > > On 03.05.2011 17:39, Zak Perschon wrote:
> > > > After letting my mass-deploy script complete, we seem to have hit a
> > > > cap at 1010 running VMs (A heck of a lot better than 128!). There
> > > > doesn't seem to be a log created for the 1010th VM that failed, but
> > > > here is the error message that I get when attempting to start it:
> > > >
> > > > VBoxManage: error: The Virtual Machine 'TinyCore-1011' has terminated
> > > > unexpectidly during startup with exit code 1
> > > > VboxManage: error: Details: code NS_ERROR_FAILURE (0x80004005),
> > > > component Machine, interface IMachine, callee
> > > >
> > > > Any ideas as to where to go from here?
> > >
> > > Of course :) It looks like you're running into the maximum number of
> > > open files _per process_ now, which defaults to 1024 on most Linux
> > > distros. The critical component in this respect is the VBoxXPCOMIPCD
> > > process (automatically started). This one is the central message broker
> > > for API requests. Since it also has a couple of files open the limit of
> > > API clients is a bit lower than 1024.
> > >
> > > Not sure if it's worth the effort to get 13 VMs more, but if you want
> > > to get to the current hardcoded maximum you have to stop all VMs
> > > again. Please double check with the "ps" command that VBoxXPCOMIPCD
> > > has terminated (should happen approx. 5 seconds after the last API
> > > client is gone).
> > >
> > > Then set a higher limit either straight with ulimit -n 2048, or
> > > following the descriptions (the comments are interesting as well) in
> > > http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-
> > > files/
> > >
> > > > The current system resource usage is as follows:
> > > >
> > > > CPU Usage: 10~% average idling, 20~% when we are utilizing all of the
> > > > 1010 machines.
> > >
> > > This sounds very good. Must be a really stripped down distribution
> > > which doesn't do much user friendly (i.e. CPU wasting) background
> > > activities. Another reason for the low utilization might actually be
> > > lock
> > > contention, as all those VMs need to coordinate their activities to
> > > some degree, and that could cause delays. We'd have some ideas what to
> > > change if this really happens.
> > >
> > > > Memory: 146.8GB of 2019.9GB (which seems low, we have 1010 machines
> > > > using 150MB of ram and 8 MB of vram, but we are getting replies from
> > > > all of the machines)
> > > > Disk Space: 48.6GB of 1.3TB (The machine is supposed to have an
> > > > attached SAS drive, but its still on order)
> > >
> > > The memory usage is suspiciously low. There could be a reason though -
> > > VirtualBox allocates VM memory lazily. If a VM doesn't touch all its
> > > memory then it will be below the expected value.
> > >
> > > > The original goal of 1800 was our "conservative" estimate, we are
> > > > actually aiming to get around 4,000 if possible. So far, Virtualbox
> > > > has been the only VM solution we've tried that has gotten us this
> > > > close (most either have a cap at around 300~ VMs that can't
> > > > increase, or they don't support our host machine's hardware and
> > > > crash). If you'd like, once we are finished with this I can provide
> > > > a bit more detail on the VM solutions we've tried and the
> > > > limitations we've ran in to.
> > >
> > > For over 1023 it needs some code tweaking... we'll help you with that
> > > of course. Don't know yet when our expert can spare a few minutes for
> > > looking into this.
> > >
> > > Klaus
> > >
> > > > -Zak
> > > >
> > > > > Date: Tue, 3 May 2011 17:00:55 +0200
> > > > > From: klaus.espenlaub at oracle.com
> > > > > To: zakperschon at hotmail.com
> > > > > CC: knut.osmundsen at oracle.com
> > > > > Subject: Re: [vbox-dev] Linux Host VM Cap?
> > > > >
> > > > > Hi Zak,
> > > > >
> > > > > On 03.05.2011 00:29, Zak Perschon wrote:
> > > > > > Hello,
> > > > > >
> > > > > > Here is a quick update for you (thanks a ton for the help by the
> > > > > > way!):
> > > > > >
> > > > > > I've been able to breach the 128 cap we had before w/ the fix you
> > > > > > suggested, but have ran into some "user error issues" that have
> > > > > > caused me to have to re-deploy/restart my VMs. The highest I've
> > > > > > gotten so far is 675 running VMs, but I'm sure I'll be able to
> > > > > > hit the 1023 mark. It takes a uprising amount of time to
> > > > > > deploy/start 1000 VMS....
> > > > >
> > > > > Keeping fingers crossed... with 675 VMs, 150MB RAM each (how much
> > > > > VRAM, I guessed 16MB?), you're around 280GB of RAM used by all of
> > > > > them, using the rough rule of thumb that RAM usage of a VM is
> > > > > approx.
> > > > > 1.02*(RAM+VRAM)+250MB. Varies greatly with the VM config details of
> > > >
> > > > course.
> > > >
> > > > > With your monster gear having 1.8TB of free RAM the theoretical
> > > > > maximum would be around 4300 of the tiny VMs you outlined
> > > > > (ignoring the disk and CPU capacity).
> > > > >
> > > > > I'm working on updating the manual. The information for Solaris is
> > > > > already there (in chapter 9), and the Linux information should also
> > > > > go there.
> > > > >
> > > > > Just out of curiosity - are other virtualizers significantly faster
> > > > > with such insanely high VM counts? It's really a very exotic use
> > > > > case.
> > > > >
> > > > > BTW, we'd appreciate keeping such threads on the mailing list -
> > > > > they show that VirtualBox is a serious virtualization solution,
> > > > > and also help others if they run into such extreme situations.
> > > > >
> > > > > Klaus
> > > > >
> > > > > > -Zak
> > > > > >
> > > > > > > Date: Mon, 2 May 2011 19:01:21 +0200
> > > > > > > From: klaus.espenlaub at oracle.com
> > > > > > > To: vbox-dev at virtualbox.org
> > > > > > > Subject: Re: [vbox-dev] Linux Host VM Cap?
> > > > > > >
> > > > > > > On 01.05.2011 17:56, Knut St. Osmundsen wrote:
> > > > > > > > Hi Zak!
> > > > > > > >
> > > > > > > > The VMM imposes a limit of 1023 VMs on 64-bit hosts and 127
> > > > > > > > VMs on
> > > >
> > > > > > > > 32-bit hosts. See:
> > > > http://www.virtualbox.org/browser/trunk/src/VBox/VMM/VMMR0/GVMMR0.cpp
> > > > #L121
> > > >
> > > > > > > That means your plan to have 1800 VMs will not work without
> > > >
> > > > bumping the
> > > >
> > > > > > > hardcoded limits, and in this area it needs much more thinking
> > > >
> > > > than just
> > > >
> > > > > > > changing a #define.
> > > > > > >
> > > > > > > Anyway, let's get you to that limit first...
> > > > > > >
> > > > > > > If you run "ipcs -ls" you'll get a "max number of arrays = 128"
> > > > > > > line, which is what you're most likely bumping into.
> > > > > > >
> > > > > > > You can read the kernel defaults with "sysctl kernel.sem" as
> > > >
> > > > well, and
> > > >
> > > > > > > the last number needs to be increased to a bit more than 1023,
> > > >
> > > > just to
> > > >
> > > > > > > be on the safe side if some other app also needs a few
> > > > > > > semaphores.
> > > > > > >
> > > > > > > "sysctl -w kernel.sem=250 32000 32 1200" should eliminate this
> > > > > > > limit. You can also put this in /etc/sysctl.conf, so that it
> > > > > > > is set
> > > >
> > > > straight on
> > > >
> > > > > > > system boot, as otherwise these settings are lost on a reboot.
> > > > > > >
> > > > > > > So... how many VMs can you start now?
> > > > > > >
> > > > > > > Klaus
> > > > > > >
> > > > > > > > Seeing you have 2TB of memory and 128 cores, I would assume
> > > > > > > > you're running 64-bit Ubuntu. So, it is probably some other
> > > > > > > > limit that
> > > >
> > > > you're
> > > >
> > > > > > > > hitting. It could be SysV semaphores, it could be memory
> > > > > > > > available
> > > > > >
> > > > > > below
> > > > > >
> > > > > > > > 4GB, ... Could you please provide some more details regarding
> > > > > > > > what happens when you are unable to launch more VMs? For
> > > > > > > > instance, does
> > > > > >
> > > > > > a new
> > > > > >
> > > > > > > > VBox.log file get created for the VM? If it does, please zip
> > > > > > > > it
> > > >
> > > > up and
> > > >
> > > > > > > > send it to me (or Klaus) per private mail and we'll work it
> > > > > > > > out.
> > > > > > >
> > > > > > > I think the 64bit host OS assumption is safe... I would
> > > > > > > seriously ask people to have their brain inspected if they'd
> > > > > > > use a 32bit OS for
> > > >
> > > > this.
> > > >
> > > > > > > > On Apr 29, 2011, at 7:43 PM, Zak Perschon wrote:
> > > > > > > >> Ah, sorry forgot to include the original support thread. The
> > > >
> > > > limit I
> > > >
> > > > > > > >> hit was 128 active VMs at once. Once I hit that number, I
> > > > > > > >> was
> > > >
> > > > unable
> > > >
> > > > > > > >> to launch any more unless I stopped one that was already
> > > >
> > > > running. My
> > > >
> > > > > > > >> original thread can be found here (though there is little
> > > >
> > > > information
> > > >
> > > > > > > >> contained in
> > > > > > > >> it):http://forums.virtualbox.org/viewtopic.php?f=7&t=40955
> > > > > > > >> <http://forums.virtualbox.org/viewtopic.php?f=7&t=40955>
> > > > > > > >>
> > > > > > > >> -Zak
> > > > > > > >>
> > > > > > > >> > Date: Fri, 29 Apr 2011 16:25:44 +0200
> > > > > > > >> > From:klaus.espenlaub at oracle.com
> > > >
> > > > <mailto:klaus.espenlaub at oracle.com>
> > > >
> > > > > > > >> > To:vbox-dev at virtualbox.org
> > > > > > > >> > <mailto:vbox-dev at virtualbox.org> Subject: Re: [vbox-dev]
> > > > > > > >> > Linux Host VM Cap?
> > > > > > > >> >
> > > > > > > >> > On 25.04.2011 18:00, Zak Perschon wrote:
> > > > > > > >> > > I've come across what is seemingly a hard cap for the
> > > >
> > > > amount of
> > > >
> > > > > > > >> > > concurrent VMs I can run on a Linux based machine (after
> > > >
> > > > posting
> > > >
> > > > > > > >> on the
> > > > > > > >>
> > > > > > > >> > > support forums, I was directed here). Apparently this
> > > > > > > >> > > same
> > > >
> > > > cap was
> > > >
> > > > > > > >> > > encountered on a solaris host and a fix was deployed to
> > > > > >
> > > > > > increase the
> > > > > >
> > > > > > > >> > > amount of VMs that were allowed to run at the same time.
> > > > > > > >> > > Was
> > > > > >
> > > > > > this fix
> > > > > >
> > > > > > > >> > > supposed to be included in other OS releases? Any
> > > > > > > >> > > information
> > > > > >
> > > > > > on this
> > > > > >
> > > > > > > >> > > would be greatly appreciated.
> > > > > > > >> >
> > > > > > > >> > That's a rather vague statement, and if you could tell
> > > > > > > >> > what
> > > > > >
> > > > > > number of
> > > > > >
> > > > > > > >> > concurrently running VMs you can achieve I might be able
> > > > > > > >> > to
> > > > > >
> > > > > > infer what
> > > > > >
> > > > > > > >> > limit is causing trouble for you.
> > > > > > > >> >
> > > > > > > >> > It could be the SysV semaphore count, the number of open
> > > >
> > > > files per
> > > >
> > > > > > > >> > process, and so on. All these defaults depend on the
> > > > > > > >> > kernel configuration and/or distribution, so it's hard to
> > > > > > > >> > generalize.
> > > > > > > >> >
> > > > > > > >> > Solaris is easy compared to that, as it is very uniform
> > > >
> > > > compared to
> > > >
> > > > > > > >> Linux.
> > > > > > > >>
> > > > > > > >> > Klaus
> > > > > > > >> >
> > > > > > > >> > > Here is the information on the hardware/VM profiles I am
> > > > > > > >> > > using
> > > > > > > >> > >
> > > > > > > >> > > Host Machine:
> > > > > > > >> > > 128 Processor Cores (64 dual core Xenon processors
> > > > > >
> > > > > > running at 2.27 GHz)
> > > > > >
> > > > > > > >> > > 2.0 TB Memory (1.8~ Usable)
> > > > > > > >> > > 1.8 TB RAID 0 (6 x 300GB HDD running at 10k RPM)
> > > > > > > >> > > 16TB SAS
> > > > > > > >> > >
> > > > > > > >> > > VM Profile
> > > > > > > >> > > Compiled TinyCore Linux 2.6
> > > > > > > >> > > Required Memory: 150~MB
> > > > > > > >> > > Required HDD Space: 50MB
> > > > > > > >> > >
> > > > > > > >> > > -Zak
> > > > > > > >> >
> > > > > > > >> > _______________________________________________
> > > > > > > >> > vbox-dev mailing list
> > > > > > > >> >
> > > > > > > >> >vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
> > > > > > > >> >http://vbox.innotek.de/mailman/listinfo/vbox-dev
> > > > > > > >>
> > > > > > > >> _______________________________________________
> > > > > > > >> vbox-dev mailing list
> > > > > > > >> vbox-dev at virtualbox.org <mailto:vbox-dev at virtualbox.org>
> > > > > > > >> http://vbox.innotek.de/mailman/listinfo/vbox-dev
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Kind regards / Mit freundlichen Gruessen / Vennlig hilsen,
> > > > > > > > bird
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > ORACLE Deutschland B.V. & Co. KG Knut St. Osmundsen
> > > > > > > > Werkstrasse 24 Senior Staff Engineer, VirtualBox
> > > > > > > > 71384 Weinstadt, Germany mailto:bird at sun.com
> > > > > > > >
> > > > > > > > Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> > > > > > > > Registergericht: Amtsgericht Muenchen, HRA 95603
> > > > > > > >
> > > > > > > > Komplementaerin: ORACLE Deutschland Verwaltung B.V.
> > > > > > > > Rijnzathe 6, 3454PV De Meern, Niederlande
> > > > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr.
> > > > > > > > 30143697 Geschaeftsfuehrer: J. Kunz, M. van de Molen, A. van
> > > > > > > > der Ven
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > vbox-dev mailing list
> > > > > > > > vbox-dev at virtualbox.org
> > > > > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev
> > > > > > >
> > > > > > > --
> > > > > > > Oracle <http://www.oracle.com>
> > > > > > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop
> > > > > > > Virtualization
> > > > > > > Phone: +49 7151 60405 205 <tel:+49715160405205>
> > > > > > > Oracle VM VirtualBox
> > > > > > >
> > > > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384
> > > > > > > Weinstadt
> > > > > > >
> > > > > > > ORACLE Deutschland B.V. & Co. KG
> > > > > > > Hauptverwaltung: Riesstr. 25, D-80992 München
> > > > > > > Registergericht: Amtsgericht München, HRA 95603
> > > > > > >
> > > > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V.
> > > > > > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> > > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr.
> > > > > > > 30143697 Geschäftsführer: Jürgen Kunz, Marcel van de Molen,
> > > > > > > Alexander van
> > > >
> > > > der Ven
> > > >
> > > > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is
> > > >
> > > > committed to
> > > >
> > > > > > > developing practices and products that help protect the
> > > > > > > environment
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > vbox-dev mailing list
> > > > > > > vbox-dev at virtualbox.org
> > > > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev
> > > > >
> > > > > --
> > > > > Oracle <http://www.oracle.com>
> > > > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop
> > > > > Virtualization
> > > > > Phone: +49 7151 60405 205 <tel:+49715160405205>
> > > > > Oracle VM VirtualBox
> > > > >
> > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt
> > > > >
> > > > > ORACLE Deutschland B.V. & Co. KG
> > > > > Hauptverwaltung: Riesstr. 25, D-80992 München
> > > > > Registergericht: Amtsgericht München, HRA 95603
> > > > >
> > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V.
> > > > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> > > > > Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van
> > > > > der Ven
> > > > >
> > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is committed
> > > > > to developing practices and products that help protect the
> > > > > environment
> > > >
> > > > _______________________________________________
> > > > vbox-dev mailing list
> > > > vbox-dev at virtualbox.org
> > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev
> >
> > _______________________________________________
> > vbox-dev mailing list
> > vbox-dev at virtualbox.org
> > http://vbox.innotek.de/mailman/listinfo/vbox-dev
--
ORACLE Deutschland B.V. & Co. KG Dr.-Ing. Frank Mehnert
Werkstrasse 24 Staff Engineer, VirtualBox
71384 Weinstadt, Germany mailto:frank.mehnert at oracle.com
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven
_______________________________________________ vbox-dev mailing list vbox-dev at virtualbox.org http://vbox.innotek.de/mailman/listinfo/vbox-dev
_______________________________________________ vbox-dev mailing list vbox-dev at virtualbox.org http://vbox.innotek.de/mailman/listinfo/vbox-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.virtualbox.org/pipermail/vbox-dev/attachments/20110511/4974b2a9/attachment.html>
More information about the vbox-dev
mailing list