This issue is described in the following thread on the
openindiana-discuss@… mailing list:
VirtualBox Client Extensions Broken
https://www.mail-archive.com/openindiana-discuss@openindiana.org/msg26504.html
OpenIndiana / Illumos changed the names of some kernel thread-related
routines which VirtualBox references as described here:
https://www.illumos.org/issues/13917
The resolution is also described in the above issue as being located at:
https://github.com/omniosorg/omnios-extra/pull/921
which made it into the OpenIndiana VirtualBox Guest Additions package:
system/virtualbox/virtualbox-additions
So the fix for your issue is to install the VirtualBox Guest Additions
from the OpenIndiana pkg repo rather than the downloaded version from
virtualbox.org.