Windows VBox dev. build 6.1.19 142917 host won't start VMs but 6.1.19 142777 does

[MikeDiack]

Bug report:

Am running Windows 8.1 Pro x64 fully patched (build 9600.19941). Symantec Endpoint Protection: Version: 14.0.3929.1200 (14.0.3929.1200.105 to be exact)

With developer build Virtual Box 6.1.19 142917, I receive the following (or similar) when trying to start any VM (XP 32 bit, but also same for a 64 bit Windows 10 guest and 32 bit Windows 7 guests) The previous developer build: 6.1.19 142777 was fine. The only change is the build of Virtual Box, and indeed reverting back from 142917 to 142777 fixes the problem.

Error messages:

Failed to open a session for the virtual machine XP.

The virtual machine 'XP' has terminated unexpectedly during startup with exit code 1 (0x1). More details may be available in 'C:\VBoxMachines\XP\Logs\VBoxHardening.log'.

Result Code: E_FAIL (0x80004005) Component: MachineWrap Interface: IMachine {85632c68-b5bb-4316-a900-5eb28d3413df}

Note: The following bug report against 142917 also looks relevant: ​https://forums.virtualbox.org/viewtopic.php?f=38&t=101896

[MikeDiack]

Hardening log file during crash of trying to start a windows guest with VBox 6.1.19 bld 142917

[MikeDiack]

Note: I've now updated to Virtual Box dev build 6.1.19 142946 - the problem remains.

[frg]

Hardening errror

[frg]

Same here. Test build 6.1.19-142894 for Windows was the last good one.

[Klaus Espenlaub]

Mysterious. If you know for sure that 142894 was the last good build then there were just 2 changes (142912 and 142917) until things went bang.

142912 is a small fix in the guest additions (i.e. the host package was just rebuilt with exactly the same code and signing), and 142917 added a forgotten ':' for one line of the "VBoxManage showvminfo" output. Neither go anywhere near areas which can impact code signing.

You can check that yourself by just looking at the installer (but also feel free to look at VBoxDrv.sys in \Windows\System32\Drivers, it should be signed with the exact same certs). It's dual signed, and we didn't change anything in the last weeks.

Yet, because the SHA2 certificate (which is the only one which Windows 10 should care about - but the SHA1 signature should also be perfectly good) will expire in about 3 weeks, so we will need to make changes soon.

[frg]

I added the hardening log from 142894 from the same vm. Forget to add a comment to it. Sorry. Maybe you can spot a difference. All I can say is it works for me. Looking at the sys and the dll files the individual certificates seems to be ok too even in the later builds. 142912 test build was never available on the website or I missed it so I can't say if it was good too.

[Jacob Klein]

How can I test 142894?

I can only confirm that:
v6.1.19 Test Build 142777 = Works
v6.1.19 Test Build 142917 = Broken
v6.1.19 Test Build 142946 = Broken

I have installers for these, too.

[fth0]

I've compared the two VBoxHardening.log files from frg, and noticed the following differences that may be important:

== r142894 (ok) ==
824.27c: VirtualBoxVM.exe: timestamp 0x6033704c (rc=VINF_SUCCESS)
824.27c: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe: Signature #1/2: info status: 24202
824.27c: '\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe' has no imports
824.27c: '\Device\HarddiskVolume5\Windows\System32\ntdll.dll' has no imports
824.27c: supR3HardenedWinInit: SUPHARDNTVPKIND_SELF_PURIFICATION_LIMITED -> VINF_SUCCESS, cFixes=0
824.27c: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe: Signature #1/2: info status: 24202
824.27c: '\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe' has no imports
824.27c: supHardenedWinVerifyImageByHandle: -> '''24202''' (\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe)
[...]
660.b74: supR3HardenedScreenImage/NtCreateSection: cache hit (Unknown Status 24202 (0x5e8a)) on \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxSupLib.dll [lacks WinVerifyTrust]
[...]
660.b74: supR3HardenedWinVerifyCacheProcessWvtTodos: '''0 (was 24202)''' fWinVerifyTrust=1 for '\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe'

== r142946 (not ok) ==
9f8.814: VirtualBoxVM.exe: timestamp 0x60379419 (rc=VINF_SUCCESS)
9f8.814: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe: Signature #1/2: info status: 24202
9f8.814: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe: Signature #2/2: VERR_CR_X509_CPV_NOT_VALID_AT_TIME for 0x60379419; retrying against current time: 0x6038df9c.
9f8.814: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe: Signature #2/2: VERR_CR_X509_CPV_NOT_VALID_AT_TIME (-23033) w/ timestamp=0x6038df9c/now.
9f8.814: '\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe' has no imports
9f8.814: '\Device\HarddiskVolume5\Windows\System32\ntdll.dll' has no imports
9f8.814: supR3HardenedWinInit: SUPHARDNTVPKIND_SELF_PURIFICATION_LIMITED -> VINF_SUCCESS, cFixes=0
9f8.814: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe: Signature #1/2: info status: 24202
9f8.814: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe: Signature #2/2: VERR_CR_X509_CPV_NOT_VALID_AT_TIME for 0x60379419; retrying against current time: 0x6038df9d.
9f8.814: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe: Signature #2/2: VERR_CR_X509_CPV_NOT_VALID_AT_TIME (-23033) w/ timestamp=0x6038df9d/now.
9f8.814: '\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe' has no imports
9f8.814: supHardenedWinVerifyImageByHandle: -> '''0''' (\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe)
[...]
a14.8c: supR3HardenedScreenImage/NtCreateSection: cache hit (VINF_SUCCESS) on \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxSupLib.dll [lacks WinVerifyTrust]
[...]
a14.8c: supR3HardenedWinVerifyCacheProcessWvtTodos: '''0 (was 0)''' fWinVerifyTrust=1 for '\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe'

This sequence of events is the first major difference, and it occurs multiple times for multiple files (VirtualBoxVM.exe, VBoxSupLib.dll). In the "ok" case, the check of the first signature leads to the info status 24202 (VINF_CR_DIGEST_DEPRECATED) (because of SHA-1) and this status prevails. In the "not ok" case, the check of the second signature leads to an additional error status -23033 (VERR_CR_X509_CPV_NOT_VALID_AT_TIME) and this status does not prevail.

== r142894 (ok) ==
660.b74: SUPR3HardenedMain: Load Runtime...
660.b74: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll: Signature #1/2: info status: 24202
[...]
660.b74: supHardenedWinVerifyImageByHandle: -> 0 (\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll) WinVerifyTrust
660.b74: supR3HardenedWinVerifyCacheInsert: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll

== r142946 (not ok) ==
a14.8c: SUPR3HardenedMain: Load Runtime...
a14.8c: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll: Signature #1/2: info status: 24202
a14.8c: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll: Signature #2/2: Unknown Status -5659 (0xffffe9e5) w/ timestamp=0x60379419/link.
a14.8c: supR3HardenedMonitor_LdrLoadDll: returns rcNt=0x0 hMod=00007ff9e7560000 'C:\Windows\system32\rsaenh.dll'
a14.8c: supHardenedWinVerifyImageByHandle: -> -5659 (\Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll) WinVerifyTrust
a14.8c: Error (rc=0):
a14.8c: supR3HardenedScreenImage/LdrLoadDll: rc=Unknown Status -5659 (0xffffe9e5) fImage=1 fProtect=0x0 fAccess=0x0 \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll: Signature #2/2: Not valid kernel code signature.: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll
a14.8c: supR3HardenedWinVerifyCacheInsert: \Device\HarddiskVolume5\Program Files\Oracle\VirtualBox\VBoxRT.dll
a14.8c: Error (rc=0):
a14.8c: supR3HardenedMonitor_LdrLoadDll: rejecting 'C:\Program Files\Oracle\VirtualBox\VBoxRT.dll' (C:\Program Files\Oracle\VirtualBox\VBoxRT.dll): rcNt=0xc0000190

This sequence of events is the second major difference. In the "ok" case, the check of the first signature leads to the info status 24202 (VINF_CR_DIGEST_DEPRECATED) (because of SHA-1) and this status does not prevail. In the "not ok" case, the check of the second signature leads to an additional error status -5659 (VERR_SUP_VP_NOT_VALID_KERNEL_CODE_SIGNATURE), this status prevails and leads to rejecting VBoxRT.dll.

This second major difference leads to the final result, since VirtualBox cannot run without the VBoxRT.dll.

Note that in the first major difference above, the "prevailing" seems to be illogical in both the "ok" and "not ok" case.

[fth0]

Replying to Jacob Klein:

I can only confirm that:
v6.1.19 Test Build 142777 = Works
v6.1.19 Test Build 142917 = Broken
v6.1.19 Test Build 142946 = Broken

Jacob Klein provided me with the VBoxRT.dll files from those revisions. I compared the signatures and certificates by eye, using the Windows Explorer file property dialogs and the Sysinternals sigcheck tool (​https://live.sysinternals.com/sigcheck.exe) and could not find any important differences.

[boxer01]

This probably has the same root cause, so I wouldn't open a new ticket and write here. As of 6.1.19 revision 14294, if I try to install the Extension Pack, I get the following message:

Signature #2/2: Not valid kernel code signature.: \Device\HarddiskVolume4\PortableApps\Portable-VirtualBox\app64\ExtensionPacks\Oracle_VM_VirtualBox_Extension_Pack\win.amd64\VBoxPuelMain.dll.


Error code:
E_FAIL (0x80004005)
Component:
ExtPackManagerWrap
Interface:
IExtPackManager {70401eef-c8e9-466b-9660-45cb3e9979e4}

If I install the previous version (142299), the installation goes without problems, but at the start of the VM I get the messages, described by the others above.

I put both VBoxHardening files (from pack installation and VM start itself) in archive. I hope this would help to find the cause of this mess.

Another problem here: there is no archive of the test builds. If you didn't download some of the older versions, you have no chance to do it later. Only the latest version i available. So I can't test now with 142777 or 142894 for example.

[boxer01]

VBoxHardening files (from pack installation and VM start itself)

[fth0]

I had to dig a little deeper, and I've finally discovered the root cause (pun intended ;)):

TL;DR: One of the two Microsoft cross-certificates expired on 2021-02-22, the other one will expire on 2021-04-15.

Some background information: The Windows Explorer file property dialogs and the Sysinternals sigcheck tool both show 12 (actually 11) certificates being involved and valid, but there are two additional certificates in the VBoxRT.dll files: The DigiCert root certificate used for the SHA-1 code signing is cross-certified by a Microsoft certificate that will expire on 2021-04-15. The Verisign root certificate used for the SHA-256 code signing is cross-certified by a Microsoft certificate that expired on 2021-02-22.

@klaus: This could be a major problem, see ​https://docs.microsoft.com/en-us/windows-hardware/drivers/install/deprecation-of-software-publisher-certificates-and-commercial-release-certificates for details. Greetings to your lead developer. :)

[Klaus Espenlaub]

I'm aware of the phasing out of the cross-signing solution. It means that using test builds will become a lot more painful, since the effort for the automated attestation signing (which is now possible, but thanks, Microsoft, for not doing this years ago!) is at the moment too big. It might actually stay this way, because attestation signing is a very time consuming process. Unless Microsoft changed something fundamental it takes on average 45 minutes to get our 4MB driver package through. And that we have to additionally do it for the guest additions drivers is adding to this pain significantly, because that will slow down the build process tremendously.

[fth0]

What about release builds? Can you still build and sign a VirtualBox release (e.g. 6.1.20) that is accepted by the VirtualBox hardening code itself?

[MikeDiack]

Good question fth0 - this was something I was wondering about release builds? Great diagnosis by the way, I'd just begun to wonder myself if it was something like that - but great work on finding the root cause.

[Klaus Espenlaub]

Of course, release builds will be as before (except guest additions, that will need some thinking). They are already going through attestation signing today. This will add Microsoft's stamp of approval directly which is the new signing process they're describing at the URL mentioned above.

[frg]

Personally I would stop testing if I would need to disable driver signing. I can understand not signing the next release bleeding edge alphas but for the regular next minor release maybe do a weekly build or so. Could always back this out if a prolem comes up and wait for the next.

[Jacob Klein]

I would like to know what the plan is to resolve this issue for TestBuilds, so that I may continue to test them without hitting the error. I also would not be able to test if I was forced to disable driver signing. I understand you are just working through the problem now, but please advise for us testers, whenever you come up with a plan. Thank you.

[fth0]

Replying to klaus:

Of course, release builds will be as before (except guest additions, that will need some thinking). They are already going through attestation signing today. This will add Microsoft's stamp of approval directly which is the new signing process they're describing at the URL mentioned above.

The "as before" strikes me. I must be missing some detail. To elaborate:

The VBoxRT.dll files from VirtualBox 6.1.18r142142, 6.1.19r142777 and 6.1.19.r142917 all use the same expired Microsoft cross-certificate. The major difference is the point in time when the code signature was created. Therefore, I would expect that a rebuild of 6.1.18 today or a new build of 6.1.20 would also be rejected by the VirtualBox hardening code. What am I missing?

Was attestation signing already used when building the 6.1.18 release, or will it be only used starting with the next official release? If it was the former, why would the VirtualBox hardening code on a new release build today not reach the check of the expired cross-certificate and reject VBoxRT.dll?

[Klaus Espenlaub]

The "as before" means that release builds were attestation signed for years now, which means that for those nothing changes. For test builds I don't know at this point how inconvenient they will be in the future. Microsoft defines the rules, and they don't look test build friendly.

[Klaus Espenlaub]

BTW, 142894 (no longer available for download, but of course we'd have it in our build archive if someone's life depends on it) was the last 6.1.19 build during the validity of the mentioned cross-signing cert.

I'll make an experiment now with using the new SHA2 code signing cert a little earlier than absolutely required (first with trunk, which we currently don't provide as test builds due to problems with the macOS package). If the working theory is right then this will produce working test builds until 15 April 2021. After that the new driver signing regime hits us with full vengeance.

[Jacob Klein]

Thank you.

[Klaus Espenlaub]

New test builds are available. They should work out of the box, but as mentioned, that's short lived fun since the now relevant cross-signing cert expires in about 6 weeks and Microsoft has decided to kill this way of signing drivers. There are likely very few options other than "test mode" of Windows to get the future test builds working. Painful for users.

[Jacob Klein]

Thank you. Oracle VirtualBox v6.1.19 Test Build 142995 does indeed work for me.

The previously mentioned link, contains useful info for Testbuild users. You might consider posting that link on the Testbuilds page: ​https://docs.microsoft.com/en-us/windows-hardware/drivers/install/deprecation-of-software-publisher-certificates-and-commercial-release-certificates

Also, on that page, is a link that goes into detail about the TESTSIGNING boot option, which you might also want to link to, to help out Testbuild users: ​https://docs.microsoft.com/en-us/windows-hardware/drivers/install/the-testsigning-boot-configuration-option

Thanks for working with us, to continue to support you. I will consider turning on TESTSIGNING to continue to test VirtualBox Testbuilds.

[Klaus Espenlaub]

Yes, we'll update the information on the Testbuilds page as soon as it is required. We have some weeks of "business as usual" before everyone needs to worry.

[MikeDiack]

I can also add to Jacob Klein's update (as you probably expect)

• Test build 6.1.19 build 142995 is working fine for me on my Win 8.1 x64 Pro fully patched system (UBR: 9600.19968) this morning.

[MikeDiack]

Just as an incidental note, I'm pleased to say that test build 6.1.19 build 143292 is working well for me today on Win 8.1 x64 host with Win 10 guests. :)

[MikeDiack]

Given that the date of 15 April 2021, when the changes to the signing of certificates will kick in (quoting below) is now only 2 days away.....

What's the news on signing/test builds/future plans for Virtual Box. Like I say 15 April is now < 48 hours away?

Quoting:

Some background information: The Windows Explorer file property dialogs and the Sysinternals sigcheck tool both show 12 (actually 11) certificates being involved and valid, but there are two additional certificates in the VBoxRT.dll files: The DigiCert root certificate used for the SHA-1 code signing is cross-certified by a Microsoft certificate that will expire on 2021-04-15. The Verisign root certificate used for the SHA-256 code signing is cross-certified by a Microsoft certificate that expired on 2021-02-22.

[MikeDiack]

So we've passed the 15 April 2021 deadline for the change in certificate behaviour (I see there was a build 143777 dated 14 April 2021) - is there an update - for what the plans are (if any) going forward, for test builds?

Klaus - can you comment at all?

[Jacob Klein]

Replying to klaus:

If the working theory is right then this will produce working test builds until 15 April 2021. After that the new driver signing regime hits us with full vengeance.

I just found and tested Oracle VirtualBox v6.1.21 Test Build 143957. Signed 4/22/2021. And It appeared to work.

• Was it expected to work?
• Why?
• Did you change your building/signing processes to accommodate allowing Test Builds to work?
• Does this mean the issue is fully resolved?

[Klaus Espenlaub]

Replying to Jacob Klein:

Replying to klaus:

If the working theory is right then this will produce working test builds until 15 April 2021. After that the new driver signing regime hits us with full vengeance.

I just found and tested Oracle VirtualBox v6.1.21 Test Build 143957. Signed 4/22/2021. And It appeared to work.

• Was it expected to work?
• Why?
• Did you change your building/signing processes to accommodate allowing Test Builds to work?
• Does this mean the issue is fully resolved?

We had to adjust to Microsoft's new driver signing regime already with the 6.1.20 release, because that ended up past the cross-signg cert expiry (not intentionally, but we had regressions requiring a respin). Which caused quite a bit of fun for us, since it broke some sanity checks in hardening which we needed to work around.

Good question why 6.1.21 works actually, because it definitely didn't go through attestation signing. Seems that Microsoft isn't checking the cross-cert expiry so far as well as our hardening code did... we'll have to see what happens with future Windows updates.

[MikeDiack]

As an aside, I've just tried the v6.1.21 Test Build 143957 on my Windows 8.1 x64 fully patched (build 9600.19995) box with success. Starts up fine and started a Win 10 x64 host just fine.

Replying to klaus:

Replying to Jacob Klein:

Replying to klaus:

If the working theory is right then this will produce working test builds until 15 April 2021. After that the new driver signing regime hits us with full vengeance.

I just found and tested Oracle VirtualBox v6.1.21 Test Build 143957. Signed 4/22/2021. And It appeared to work.

• Was it expected to work?
• Why?
• Did you change your building/signing processes to accommodate allowing Test Builds to work?
• Does this mean the issue is fully resolved?

We had to adjust to Microsoft's new driver signing regime already with the 6.1.20 release, because that ended up past the cross-signg cert expiry (not intentionally, but we had regressions requiring a respin). Which caused quite a bit of fun for us, since it broke some sanity checks in hardening which we needed to work around.

Good question why 6.1.21 works actually, because it definitely didn't go through attestation signing. Seems that Microsoft isn't checking the cross-cert expiry so far as well as our hardening code did... we'll have to see what happens with future Windows updates.