VBoxDrv: Too many CPUs (288) for the GIP (max 256)

[okkhoy]

The initialization here:

include/iprt/types.h:# define RTCPUSET_MAX_CPUS      256

and the check :

    /*
     * Check the CPU count.
     */
    cCpus = RTMpGetArraySize();
    if (   cCpus > RTCPUSET_MAX_CPUS
#if RTCPUSET_MAX_CPUS != 256
        || cCpus > 256 /* ApicId is used for the mappings */
#endif
        )
    {
        SUPR0Printf("VBoxDrv: Too many CPUs (%u) for the GIP (max %u)\n", cCpus, RT_MIN(RTCPUSET_MAX_CPUS, 256));
        return VERR_TOO_MANY_CPUS;
    }

in file src/VBox/HostDrivers/Support/SUPDrvGip.cpp prevents installation of virtualbox on (Linux) server with > 256 cores.

[okkhoy]

The installation fails with the following error:

***
[6276452.882765] vboxdrv: Found 288 processor cores
[6276452.885176] VBoxDrv: Too many CPUs (288) for the GIP (max 256)
***

[Frank Mehnert]

That's indeed a bug. I think we have a similar ticket elsewhere but currently I'm unable to find it.

Could you attach the output of

lshw -c cpu

? It might be necessary to install the 'lshw' package on your Linux distribution.

[okkhoy]

Thanks for the quick response! Here is the output of lshw. However, I am running this as non-root.

lshw -c cpu
WARNING: you should run this program as super-user.
  *-cpu:0                 
       product: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
       vendor: Intel Corp.
       physical id: 1
       bus info: cpu@0
       size: 1199MHz
       capacity: 3300MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts cpufreq
  *-cpu:1
       product: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
       vendor: Intel Corp.
       physical id: 4
       bus info: cpu@1
       size: 1200MHz
       capacity: 3300MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts cpufreq
  *-cpu:2
       product: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
       vendor: Intel Corp.
       physical id: 6
       bus info: cpu@2
       size: 1200MHz
       capacity: 3300MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts cpufreq
  *-cpu:3
       product: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
       vendor: Intel Corp.
       physical id: 7
       bus info: cpu@3
       size: 1200MHz
       capacity: 3300MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts cpufreq
  *-cpu:4
       product: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
       vendor: Intel Corp.
       physical id: 8
       bus info: cpu@4
       size: 1199MHz
       capacity: 3300MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts cpufreq
  *-cpu:5
       product: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
       vendor: Intel Corp.
       physical id: 9
       bus info: cpu@5
       size: 1200MHz
       capacity: 3300MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts cpufreq
  *-cpu:6
       product: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
       vendor: Intel Corp.
       physical id: a
       bus info: cpu@6
       size: 1223MHz
       capacity: 3300MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts cpufreq
  *-cpu:7
       product: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
       vendor: Intel Corp.
       physical id: b
       bus info: cpu@7
       size: 1200MHz
       capacity: 3300MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts cpufreq
  *-processor
       description: Co-processor
       product: Xeon Phi coprocessor 5100 series
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:02:00.0
       version: 11
       width: 64 bits
       clock: 33MHz
       capabilities: bus_master cap_list
       configuration: driver=mic latency=0
       resources: iomemory:210c0-210bf irq:26 memory:210c00000000-210dffffffff memory:abd00000-abd1ffff
  *-processor
       description: Co-processor
       product: Xeon Phi coprocessor 5100 series
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0001:02:00.0
       version: 11
       width: 64 bits
       clock: 33MHz
       capabilities: bus_master cap_list
       configuration: driver=mic latency=0
       resources: iomemory:21220-2121f irq:46 memory:212200000000-2123ffffffff memory:dbe00000-dbe1ffff
WARNING: output may be incomplete or inaccurate, you should run this program as super-user.

[Klaus Espenlaub]

Wouldn't exactly call it a bug, as it's strictly speaking a (deliberate) hardcoded limitation, to limit the code complexity and overhead of CPU management in VirtualBox.

Which has all nice until Intel started selling CPUs with more than 16 cores (i.e. with the Haswell generation), as in an 8 socket server this brings the count beyond 256 (including hyperthreading).

All in all I'm of course nitpicking, because such systems will become more and more widespread. It's not really the typical kind of machine for running VirtualBox on, but who are we to tell people they do the wrong thing with running VirtualBox on very expensive hardware (around $56k list price for the CPUs alone)... can't tell yet by when we'll be able to implement this.

If you're somewhat desperate to get VirtualBox working: all such big servers I've seen have settings in the BIOS/UEFI setup for limiting the core count. Going with 16 cores per socket should do the trick (reducing the overall compute power of the system by 12.5%).

[okkhoy]

Thank you. I guess for now, we may go ahead and try setting the core count limit in the BIOS.

[Socratis]

Related discussion in the forums: ​https://forums.virtualbox.org/viewtopic.php?f=7&t=84245