Can't enter encryption password OSX 10.12.5

[Neekoh]

Hello,

Every machine that using a saved state + virtual encryption, can't be unlocked. Either no password prompt shows up or the password prompt that appears does not allow any typing in it (can't be clicked).

Only solution for now is to discard the saved state and start up normally.

Another remark: When pausing a VM, no password has to be entered to unlock/decrypt, ideally behaviour should be the same then we starting/starting from saved state.

[Socratis]

Please take a look at #16560. It sounds really similar (or closely related). Take a look at the procedure to capture the VBox.log (and ZIP IT then). Which you should have included in the first place...

Also, is this a new behavior, did that start happening with 5.1.26, or was it happening before that?

[Neekoh]

VBox logs

[Neekoh]

Hello,

You're right, always mind the logs :-) It's not new behaviour, it has always been there.

Scenario:

When my laptop battery is low, default behaviour of VB is to save state and then power off. When starting the laptop, it opens VB as my OS X session is restored. VB is running, but the VM is marked as 'Saved'.

I do a normal start, the VM starts to resume, I can see my last session and whatever application I had open but it's greyed out. Whatever session, application, file I had opened is displayed as a (greyed out/erase ever other row screenshot). It shouldn't disclose anything without authentication.

As soon as it is full resumed, a pop up window appears, asking for a password. I then try to click to focus/activate the password input field but I can't get any sort of control over it. Same story for the VM window, no focus/activate. Via the VB application icon in my dock, I can do a poweroff. But that poweroff is also not capable of controlling the VM window. So it stays visible. I can't close it, not via the red dot/cross, not via ctrl-left click. The VB app in my dock does the exact same. Whatever I do to close it: No go.

I have to Force quit / kill -6 or even -9 the application via terminal or activity monitor.

So attached to this mail, the requested logs: Vbox0.log: Vbox1.log:

[Socratis]

I can only say that in 10.11.6 (and previously 10.9.5), I've never seen that behavior, so I cannot confirm (or deny) that behavior.

Any particularities about the VM? I.e. are you running it full screen? Anything out of the ordinary running on your OSX? Please don't say no, no computer is ever as it would be if it was factory installed. Mainly interested for things that are/might be affecting the system...

[Neekoh]

Nothing really special:

The VM is a test environment that runs Kali, a Debian based distro. OS X: I have brew installed / Xcode / Xquartz

I made a screenshot

I can't click/activate the password window, clicking OK, Cancel or the input field does not work, if I do OS X produces a sound alert (error). Sometimes this window does not even full appear, it's displayed for a second or so and disappears. Resizing the window makes it appear again.

I tried changing the resolution of the OS X host because changing resolution means that all windows have to be redrawn: No luck.

All menu options are greyed out:

Only way out is to click the VB program icon in my dock, click the VM and do save state or power off:

Update

After 10 to 20 seconds the VB daemon that hosted the VM crashed, I attached the crashlog to this ticket.

Behaviour is like:

​https://bugreports.qt.io/browse/QTBUG-5058

[Socratis]

I see a lot of weird things in the screenshots, like the fonts. Someone has been messing with the system. Big time!

Plus, I don't like the "brew" part. Are your default paths modified in any way, shape or form? Do you have the Qt toolkit installed as part of the brew ports?

[Neekoh]

Hello!

Your are mistaking. The fonts = UTF-8 encoded text in a pseudo alphabet. I was bored one day ;-).

No really the OS X installed is fine. I have no problem whatsoever with anything other then not being able to enter the decryption password.

Brew is I properly installed, no paths have been modified, everything in /usr/local, duplicates of existing commands or libraries are not included in $PATH as they reside in /usr/local/opt/.

The crashreport gives an overview of all files in use by VirtualBox:

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [0]

VM Regions Near 0:
--> 
    __TEXT                 0000000109903000-0000000109925000 [  136K] r-x/rwx SM=COW  /Applications/VirtualBox.app/Contents/MacOS/VirtualBoxVM

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   '''org.qt-project.QtCoreVBox'''     	0x000000010a9503aa QMetaObject::cast(QObject*) const + 26
1   '''VirtualBox.dylib'''              	0x0000000109f801c9 0x109e4b000 + 1266121
2   '''VirtualBox.dylib'''              	0x0000000109f802fe 0x109e4b000 + 1266430
3   '''org.qt-project.QtCoreVBox'''     	0x000000010a976a63 QObjectPrivate::deleteChildren() + 243
4  ''' org.qt-project.QtCoreVBox'''     	0x000000010a976820 QObject::~QObject() + 2016
5   VirtualBox.dylib              	0x0000000109f51777 0x109e4b000 + 1075063
6   VirtualBox.dylib              	0x000000010a1e758d 0x109e4b000 + 3786125
7   org.qt-'''project.QtCoreVBox'''     	0x000000010a977438 QObject::event(QEvent*) + 744
8   org.qt-'''project.QtWidgetsVBox'''  	0x000000010b319cf6 QApplicationPrivate::notify_helper(QObject*, QEvent*) + 294
9   org.qt-project.QtWidgetsVBox  	0x000000010b31d26e QApplication::notify(QObject*, QEvent*) + 8558
10  org.qt-'''project.QtCoreVBox'''     	0x000000010a94a7e7 QCoreApplication::notifyInternal2(QObject*, QEvent*) + 167
11  org.qt-project.QtCoreVBox     	0x000000010a94b44b QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) + 587
12  libqcocoa.dylib               	0x000000010eb2862e QCocoaEventDispatcherPrivate::processPostedEvents() + 190
13  libqcocoa.dylib               	0x000000010eb28f31 QCocoaEventDispatcherPrivate::postedEventsSourceCallback(void*) + 33
14  com.apple.CoreFoundation      	0x00007fffb7e82321 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
15  com.apple.CoreFoundation      	0x00007fffb7e63197 __CFRunLoopDoSources0 + 423
16  com.apple.CoreFoundation      	0x00007fffb7e62716 __CFRunLoopRun + 934
17  com.apple.CoreFoundation      	0x00007fffb7e62114 CFRunLoopRunSpecific + 420
18  com.apple.HIToolbox           	0x00007fffb73c3ebc RunCurrentEventLoopInMode + 240
19  com.apple.HIToolbox           	0x00007fffb73c3cf1 ReceiveNextEventCommon + 432
(repeated until crash)

Maybe it's not clear enough in the screenshots but the pop-up window that appears for password entry never reaches the 'upper layer', it's sort of glued stuck in the VM screen window and that only happens when a VM is in saved state. If I remove the saved state it works normally and I have a popup window of which is context/Input activated by clicking on it.

The crash looks like VirtualBox requesting access while the process is already dead (so after power off of the VM). It's a consequence of the original problem but I don't think it relates to the cause.

I should know what I'm doing having about 20 years of experience in stuff as this.

Thanks!

[Socratis]

Replying to Neekoh:

Replying to socratis:

Someone has been messing with the system. Big time!

Your are mistaking. The fonts = UTF-8 encoded text in a pseudo alphabet. I was bored one day ;-).

I'm afraid that I'm not mistaken. These are the SYSTEM fonts. If you're messing with the system every time that you're bored, who knows what else you've been up to... ;-)

No really the OS X installed is fine. I have no problem whatsoever with anything other then not being able to enter the decryption password.

Do you want to make a bet that if you do a fresh install you won't have that problem? So, something in your setup is interfering with the proper operation. Any Qt libraries installed? Any other Qt related apps?

The crash looks like VirtualBox requesting access while the process is already dead (so after power off of the VM). It's a consequence of the original problem but I don't think it relates to the cause.

As I mentioned, I tried your steps to the "t" (except wait for the battery to die). No problem here, although I have to mention that I'm on 10.11.6.

I should know what I'm doing having about 20 years of experience in stuff as this.

Same here ;-)

[hjartstrom]

Same thing on Kubuntu 16.04.3 LTS using VirtualBox 5.1.26 r117224 (Qt 5.5.1). Froze the UI completely, except for mouse movement, so I had to exit to CLI (Ctrl+Alt+F1) and kill the VirtualBox process.

I've also seen this several times on Windows 10 using different releases. Simply close a running (encrypted) VirtualBox machine and save the state, then try to start it again.