VirtualBox

Ticket #13335 (closed defect: fixed)

Opened 3 years ago

Last modified 3 years ago

Leftover signal handler in VBoxOGLcrutil.so crashes if no 3D acceleration is enabled -> fixed in 4.2 and later releases after 2 September 2014

Reported by: koe Owned by:
Priority: major Component: guest additions
Version: VirtualBox 4.3.10 Keywords:
Cc: Guest type: Linux
Host type: Linux

Description

gnome-session on RHEL7 crashes when it receives a signal. This seems to be caused by a bug in the virtual box guest video driver.

Here are the relevant parts of a strace run:

The VBox OpenGL library is loaded and mapped into memory

open("/lib64/VBoxOGLcrutil.so", O_RDONLY|O_CLOEXEC) = 7
[...]
mmap(NULL, 1885440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 7, 0) = 0x7f6197e9e000
mprotect(0x7f6197f4f000, 1044480, PROT_NONE) = 0
mmap(0x7f619804e000, 61440, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0xb0000) = 0x7f619804e000
mmap(0x7f619805d000, 54528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f619805d000
close(7)                                = 0
[...]

It registers several signal handlers (among them sigint at 0x7f61980b56e0)

rt_sigaction(SIGTERM, {0x7f61980b56e0, [TERM], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, {0x7f61b2377610, [], SA_RESTORER, 0x7f61b12fc130}, 8) = 0
rt_sigaction(SIGINT, {0x7f61980b56e0, [INT], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, 8) = 0
[...]
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f61a9152a00}, {0x7f61980b56e0, [INT], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f61a9152a00}, {SIG_DFL, [], 0}, 8) = 0
[...]
rt_sigaction(SIGINT, {0x7f61980b56e0, [INT], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f61a9152a00}, NULL, 8) = 0
[...]

It fails to initialize since 3D acceleration is not enabled for the VM. It gets unloaded again but does *NOT* unregister the signal handlers first so they stay intact for invalid addresses.

write(2, "OpenGL Warning: Failed to connec"..., 93OpenGL Warning: Failed to connect to host. Make sure 3D acceleration is enabled for this VM.) = 93
write(2, "libGL error: ", 13libGL error: )           = 13
write(2, "failed to load driver: vboxvideo"..., 33failed to load driver: vboxvideo) = 33
write(2, "libGL error: Try again with LIBG"..., 66libGL error: Try again with LIBGL_DEBUG=verbose for more details.) = 66
munmap(0x7f619806b000, 2284776)         = 0
munmap(0x7f6197e9e000, 1885440)         = 0
[...]

The software rendering driver gets loaded. By chance it is in the same address range the VBox OpenGL driver was before. So the signal handlers now point somewhere into this library.

open("/usr/lib64/dri/swrast_dri.so", O_RDONLY|O_CLOEXEC) = 7
[...]
mmap(NULL, 9876608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 7, 0) = 0x7f619792d000
mprotect(0x7f6197e8e000, 2093056, PROT_NONE) = 0
mmap(0x7f619808d000, 155648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0x560000) = 0x7f619808d000
mmap(0x7f61980b3000, 1987712, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61980b3000
close(7)                                = 0
[...]

A SIGINT signal gets triggered (in this case by hand). The signal handler is called, it points into the bss section of the software rendering library which is not executable and a segfault occurs.

--- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=7475, si_uid=0} ---
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7f61980b56e0} ---
+++ killed by SIGSEGV (core dumped) +++

Change History

comment:1 Changed 3 years ago by michael

Could you give this (4.3.14-compatible) Additions build a try? Thanks.

https://www.virtualbox.org/download/testcase/VBoxGuestAdditions_4.3.15-95813.iso

comment:2 Changed 3 years ago by koe

In this version VBoxOGLcrutil.so stays in the process and it does not crash when the signal handler is called. Thanks for the quick fix.

comment:3 Changed 3 years ago by michael

  • Summary changed from Leftover signal handler in VBoxOGLcrutil.so crashes if no 3D acceleration is enabled to Leftover signal handler in VBoxOGLcrutil.so crashes if no 3D acceleration is enabled -> fixed in 4.2 and later releases after 2 September 2014

Thanks for the feedback.

comment:4 Changed 3 years ago by frank

  • Status changed from new to closed
  • Resolution set to fixed

Fix is part of VBox 4.3.16.

Note: See TracTickets for help on using tickets.

www.oracle.com
ContactPrivacy policyTerms of Use