[vbox-dev] [PATCH] Runtime fix for LLVM 8.0+

[Kyle Evans]

Hi,

On FreeBSD, we encountered a runtime regression when our base system
LLVM updated from 7 to 8+. Careful triage revealed that LLVM commit
18bfb3a5ec41 caused clang to start applying tail-call-optimization to
XPTC_InvokeByIndex, resulting in the tail call being moved to *after*
the stack prepared for it was already clobbered.

Examination of the area in VirtualBox trunk/ revealed that a
similar-looking issue had been uncovered in changeset 75061 with GCC
sanitizers. The attached patch -- also available at
https://people.freebsd.org/~kevans/vbox/xptc-clang.diff -- adds
!defined(__clang__) to these #ifndefs so that clang will use the GCC
with sanitizer method for passing the stack through.

It's worth noting that our LLVM folks mentioned that MacOS may not
goofed the codegen on this due to other differences, e.g. not using
LLVM libunwind, if it turns out MacOS is using a version of LLVM in
the above version range.

Again I suspect this patch is considered de minimus and this one in
particular likely won't end up in its current form committed anyways,
but in case it's necessary: this may be licensed under the MIT
license.

Thanks,

Kyle Evans

[0] https://reviews.llvm.org/rL34516