Invalid check of authenticated attributes from certificates when loading custom device plugin

[yanfu]

I use a custom virtual PCI device plugin. I've signed it with a code-signing key from GoDaddy, but VirtualBox 5.0.0 refuses to load my DLL on Windows 7.

Error

It gives the follow error:

Failed to open a session for the virtual machine vbox-signed.

Unable to load R3 module C:\Users\vbox-win7\VirtualBox VMs\vbox-signed\img_ipc_device\img_ipc_device.dll (img_ipc_device): : \Device\HarddiskVolume2\Users\vbox-win7\VirtualBox VMs\vbox-signed\img_ipc_device\img_ipc_device.dll (VERR_INTERNAL_ERROR_3).

Result Code: E_FAIL (0x80004005)
Component: ConsoleWrap
Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}

Debugging

In order to track the error, I have tried to build a debug version of VirtualBox 5.0.0 OSE with test signing. When loading my custom dll (cross signed), VERR_INTERNAL_ERROR_3 was hit in ..\src\VBox\Runtime\common\crypto\pkcs7-sanity.cpp as follows:

static int rtCrPkcs7SignedData_CheckSanityExtra(PCRTCRPKCS7SIGNEDDATA pSignedData, uint32_t fFlags, PRTERRINFO pErrInfo, const char *pszErrorTag)
{
...

  if (RTCrPkcs7Attributes_IsPresent(&pSignerInfo->AuthenticatedAttributes))
  {
     ...
     AssertReturn(pAttrib->enmType == RTCRPKCS7ATTRIBUTETYPE_UNKNOWN, VERR_INTERNAL_ERROR_3);
  }

}

Bug Analysis

From debugging into the failure, it showed loading my custom dlls failed when VBox checked the attribute of Sigining Time (1.2.840.113549.1.9.5) in our certificate.

In the VBox source code, it expected the type enum of this attribute (pAttrib->enmType) to be RTCRPKCS7ATTRIBUTETYPE_UNKNOWN. However, it turned out to be RTCRPKCS7ATTRIBUTETYPE_SIGNING_TIME when I debugged to check.

I think it is a bug here, because it is surely not an unknown attribute and shouldn't be considered as an error. The attribute of Signing Time should be checked to match it's own pAttrib->enmType as SIGNING_TIME, instead of UNKNOWN.

I have attached relevant log files for reference.

[yanfu]

VBoxStartup log

[yanfu]

VBox log

[yanfu]

Log file of VBox debug version

[Frank Mehnert]

Please reopen if still relevant with VBox 5.0.14.