[vbox-dev] Problem with kernel 5.8

[Larry Finger]

On 6/8/20 4:12 AM, Frank Batschulat wrote:
> Larry, thanks - this work will be tracked with:
> 
> Ticket #19644 (accepted defect)
> Linux kernel version: 5.8 - we need changes
> https://www.virtualbox.org/ticket/19644

With each update to the mainline kernel, another API change for Kernel 5.8 
emerges. Thus far, there have been three different changes:

1. In struct mm_struct, member mmap_sem is renamed to mmap_lock. The comment in 
the commit says "Rename the mmap_sem field to mmap_lock.  Any new uses of this 
lock should now go through the new mmap locking api.  The mmap_lock is still
implemented as a rwsem, though this could change in the future." Until that 
unknown future time, only the member name needs to change, and that is what I 
propose.

2. The shadow set of kernel quantities stored in cpu_tlbstate are no longer 
available. The only quantity used from that structure is CR4. My fix is to use 
__read_cr4() rather than this_cpu_read(cpu_tlbstate.cr4) to read CR4, and 
ASMSetCR4(uNew) rather than this_cpu_write(cpu_tlbstate.cr4, uNew) to write a 
new value into CR4. This change has tested OK.

3. Kernel routine __get_vm_area() was first changed to __get_vm_area_caller(), 
and then the latter routine was no longer exported. In addition, routine 
map_vm_area() has been removed. I do not have a fix for this. At first it seemed 
that reverting to the method used before kernel 2.6.13 would work by not 
defining RTMEMALLOC_EXEC_VM_AREA and using RTMEMALLOC_EXEC_HEAP instead. 
Unfortunately, this code fails to allocate the memory correctly, and VMs fail to 
start. I have been unable to fix that problem.

Until there is a fix for the code using RTMEMALLOC_EXEC_HEAP, I will be 
recommending that the openSUSE versions of 5.8 include a patch that exports 
__get_vm_area_caller() and map_kernel_range(). The latter routine is used to 
replace map_vm_area. With these two changes, the VB modules build and execute. I 
will be posting the patches for vboxdrv and the proposed kernel patch at 
https://www.virtualbox.org/ticket/19644.

As an editorial aside, it is clear that VirtualBox needs to get at least part of 
vboxdrv into the kernel as a module in the way that VMWare has done so that this 
incompatibility with memory management never occurs again.

Larry