Automatic update of VBox and Failed to open a session for the virtual machine Ubuntu => fixed in SVNsvn

[AZidai]

Few weeks ago, when I had updated VirtualBox, I had faced an error with opening a session for the virtual machine (guest OS is Ubuntu). Found a suggestion that VBox version 6.1.32 is good and I had downgraded the VBox and everything works fine, I can start a session for the VM. I had turned off the automatic update of VirtualBox, but few times since then, VirtualBox was updated automatically to the newest version.

For some, unknown to me reason, I can't use the newest VBox version, but my VBox keep upgrading even when automatic update is turned off. What is the solution? Should I downgrade my VBox every day?

[galitsyn]

Hi AZidai,

The error on screenshot suggests that host does not have enough free memory. If it is not a case, please attach VBox.log.

[AZidai]

Which kind of memory is missing? Dedicated RAM or Hard disk space? With VB version 6.1.32 everything works. Any newer versions produce the same issue. This is the log:

Failed to open a session for the virtual machine Ubuntu.

Failed to load R0 module C:\Program Files\Oracle\VirtualBox/VMMR0.r0: RTCrStoreConvertToOpenSslCertStack failed: \Device\HarddiskVolume3\Program Files\Oracle\VirtualBox\VMMR0.r0 (VERR_NO_MEMORY).

Failed to load VMMR0.r0 (VERR_NO_MEMORY).

Result Code: E_FAIL (0x80004005) Component: ConsoleWrap Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}

[AZidai]

One more thing - today, after turning on my computer - VB has been upgraded to v6.1.38 - even if automatic check and upgrade is turned off!!!

[galitsyn]

Hi AZidai,

Log snippet is not enough. Could you please attach complete log?

[gombara]

Hi AZidai, How did you install/upgrade/downgrade VirtualBox? From www.virtualbox.org or you distros repo?

[fth0]

Replying to gombara:

How did you install/upgrade/downgrade VirtualBox? From www.virtualbox.org or you distros repo?

From Microsoft's repo? SCNR ;)

[gombara]

I misread Linux as host type. In any case VirtualBox does not update itself automatically thus it cannot be turned off.

[AZidai]

Replying to gombara:

I misread Linux as host type. In any case VirtualBox does not update itself automatically thus it cannot be turned off.

I haven't understood you well - in Linux as a host, VB cannot automatically upgrade itself? But, never mind, I have Windows 10 as a host and Ubuntu v20.04 as a guest OS. Everyday, when I turn on my computer, my VB is on v6.1.38 even when I downgrade it everyday to v6.1.32

[fth0]

AZidai, I'll try to sort out a few things for you:

AFAICS, VirtualBox for Windows is only provided by Oracle (1), whereas VirtualBox for Linux is provided by Oracle (2) and by many Linux distributions (3). In all three VirtualBox versions (1, 2, 3), you can configure the VirtualBox Manager to automatically check for updates, but this won't install/update any VirtualBox version that the check finds. In case of the Linux-distributed VirtualBox version (3), the Linux distribution specific package manager (e.g. apt) can be configured to automatically update all packages, including the Linux-distributed VirtualBox version (3).

That being said, I don't understand what's happening in your case, because VirtualBox cannot update itself AFAIK. Are you perhaps using a third-party package management on your Windows host like Chocolatey? How did you downgrade VirtualBox and how did you verify that the downgrade was successful?

[AZidai]

VB has been installed from virtualbox.org Any upgrade was done after getting popup message on app startup Downgrade is done with installer, downloaded from: https://www.virtualbox.org/wiki/Download_Old_Builds_6_1 Verification, if downgrade is successful, I done checking which version of VB is running, from app option "Help > About VirtualBox..." and with simple starting a session of my VM - with v6.1.32, I can use my Ubuntu VM, with any newer, I can't open it... I am not using any third-party package manager in my host Windows 10 OS. I have attached these two screenshots: https://www.virtualbox.org/attachment/ticket/21097/Current_VB_version.PNG and https://www.virtualbox.org/attachment/ticket/21097/Preferences.PNG I will downgrade VB now, screenshot again and attach it here.

[AZidai]

A version after downgrading VM: https://www.virtualbox.org/attachment/ticket/21097/Version_after_downgrading.PNG

[galitsyn]

Hi AZidai,

Thank you for logs. Could you please try VBox 6.1.38 with matching ExtensionPack version (6.1.38 as well)?

[AZidai]

Above you can see all the available log files.

[AZidai]

Could not upgrade ExtPack, after installing VB v6.1.38

[fth0]

FWIW, I noticed the following:

1) The VirtualBox version was replaced over the weekend (between 2022-09-23T20:53Z and 2022-09-26T13:07Z).

2) The VM is provided with 9.8 GB RAM, which is more than your host OS reports as available when starting the VM.

3) The guest OS uses Ubuntu kernel 5.15.0-48, which perhaps requires VirtualBox 6.1.38.

4) The guest OS has VirtualBox Guest Additions 6.1.36 installed.

[galitsyn]

From my side, I would also recommend to completely uninstall VBox (and installed ExtPack before uninstalling VBox), probably, reboot host and install fresh 6.1.38. Do you see any difference if you do so?

[AZidai]

I have uninstalled VB v6.1.38. Before that, I could not remove ExtPack v6.1.32 from Extensions. I have used CCleaner, to try clean registry after uninstalling VB (and restarting Win OS), then restarted again and install fresh VB v6.1.38. I have started VB, gone to Preferences > Extensions and seen that I have already installed ExtPack v6.1.32 !!! Is there a better way to remove ExtPack from my computer (before and/or after unistalling VB)??? Thx

[galitsyn]

and seen that I have already installed ExtPack v6.1.32

This is expected, since it was not previously removed.

Normal way to uninstall Extension Pack is to go to Preferences -> Extensions and select uninstall option. What was the issue with uninstalling it?

[AZidai]

This is the outcome: Failed to uninstall the Extension Pack Oracle VM VirtualBox Extension Pack.

The installer failed with exit code 1: VBoxExtPackHelperApp.exe: error: RTCrStoreConvertToOpenSslCertStack failed: \Device\HarddiskVolume3\Program Files\Oracle\VirtualBox\VBoxExtPackHelperApp.exe.

Result Code: E_FAIL (0x80004005) Component: ExtPackManagerWrap Interface: IExtPackManager {70401eef-c8e9-466b-9660-45cb3e9979e4}

[AZidai]

I have tried this command (from PowerShell): PS C:\Program Files\Oracle\VirtualBox> .\vboxmanage extpack cleanup VBoxManage.exe: error: The installer failed with exit code 1: VBoxExtPackHelperApp.exe: error: RTCrStoreConvertToOpenSslCertStack failed: \Device\HarddiskVolume3\Program Files\Oracle\VirtualBox\VBoxExtPackHelperApp.exe VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ExtPackManagerWrap, interface IExtPackManager, callee IUnknown VBoxManage.exe: error: Context: "Cleanup()" at line 1474 of file VBoxManageMisc.cpp PS C:\Program Files\Oracle\VirtualBox>

[AZidai]

Result of listing ExtPacks command: S C:\Program Files\Oracle\VirtualBox> ./VBoxManage list extpacks Extension Packs: 1 Pack no. 0: Oracle VM VirtualBox Extension Pack Version: 6.1.32 Revision: 149290 Edition: Description: Oracle Cloud Infrastructure integration, USB 2.0 and USB 3.0 Host Controller, Host Webcam, VirtualBox RDP, PXE ROM, Disk Encryption, NVMe. VRDE Module: VBoxVRDP Usable: false Why unusable: RTCrStoreConvertToOpenSslCertStack failed: \Device\HarddiskVolume3\Program Files\Oracle\VirtualBox\ExtensionPacks\Oracle_VM_VirtualBox_Extension_Pack\win.amd64\VBoxPuelMain.dll

[AZidai]

Another attempt: PS C:\Program Files\Oracle\VirtualBox> ./VBoxManage extpack uninstall "Oracle VM VirtualBox Extension Pack" --force 0%... Progress state: E_FAIL VBoxManage.exe: error: Failed to uninstall "Oracle VM VirtualBox Extension Pack" VBoxManage.exe: error: The installer failed with exit code 1: VBoxExtPackHelperApp.exe: error: RTCrStoreConvertToOpenSslCertStack failed: \Device\HarddiskVolume3\Program Files\Oracle\VirtualBox\VBoxExtPackHelperApp.exe VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ExtPackManagerWrap, interface IExtPackManager VBoxManage.exe: error: Context: "enum RTEXITCODE cdecl handleExtPack(struct HandlerArg *)" at line 1465 of file VBoxManageMisc.cpp

[fth0]

@galitsyn:

I discovered that the error message "RTCrStoreConvertToOpenSslCertStack failed" can be a bit misleading, because it's created twice in rtCrPkcs7VerifySignedDataUsingOpenSsl(), and one of the names should end with "Store" instead of "Stack".

[AZidai]

I have tried removing any data related to VirtualBox from Registry, manually, also I have removed ".VirtualBox" folder from Users/Home, don't know if I have removed all of data, but after that and restart, newly installed VB v6.1.38 was clear (had to add existing VM, set Preferences, etc.) But attempt of installing ExtPack was unsuccessful, again... Then, I have downgraded VB again in order to work with my VM today ... Also, I've downloaded VBoxBuestAdditions.iso v6.1.32 and reinstall them on my VM (in order to remove v6.1.36)

[fth0]

FWIW, the issue probably is somehow related to (one or more) (available or not available) certificates in the Windows certificate stores.

[AZidai]

Replying to fth0:

FWIW, the issue probably is somehow related to (one or more) (available or not available) certificates in the Windows certificate stores.

Thank you for you answer, "fth0", but what does that mean? Any newer VB version, starting from v6.1.34 have an issue with certificates within Windows certificate stores? What should I do as a common VB user? BTW, today, when I have booted my Win OS, VB version is again Version 6.1.38 r153438 (Qt5.6.2). As Queen sing: "It's a kind of magic!!!" :)

[fth0]

My hint was deliberately vague, because I cannot really tell you more without analyzing and understanding the VirtualBox source code surrounding the error. And I'm just a knowledgeable VirtualBox user. ;)

VirtualBox is reading all certificates from the Windows certificate stores into an internal OpenSSL structure. I don't know (at least not yet), if the error has to do with one of the certificates from Oracle, or with one of the other certificates which are in one of the Windows certificate stores on your host.

AFAIK, you're the first user reporting this issue. Regarding what to do, I'd suggest to wait until someone can explain in detail what to look for. Otherwise, by removing certificates from the Windows certificate stores, you can easily shoot yourself in the foot ...

[fth0]

@galitsyn:

Analyzing the VirtualBox source code, I've found the following programming mistake in RTCrStoreConvertToOpenSslCertStore(), RTCrStoreConvertToOpenSslCertStack(), rtCrOpenSslConvertX509Cert() and rtCrOpenSslConvertPkcs7Attribute():

RTDECL(int) RTCrStoreConvertToOpenSslCertStore(RTCRSTORE hStore, uint32_t fFlags, void **ppvOpenSslStore, PRTERRINFO pErrInfo)
{
[...]
/* (1) */       if (   (pCertCtx->fFlags & RTCRCERTCTX_F_ENC_MASK) == RTCRCERTCTX_F_ENC_X509_DER
                    && pCertCtx->cbEncoded > 0)
                {
                    X509 *pOsslCert = NULL;
                    const unsigned char *pabEncoded = (const unsigned char *)pCertCtx->pabEncoded;
/* (2) */           if (d2i_X509(&pOsslCert, &pabEncoded, pCertCtx->cbEncoded) == pOsslCert)
                    {
/* (3) */               if (!X509_STORE_add_cert(pOsslStore, pOsslCert))
/* (4) */                   rc = VERR_NO_MEMORY;
                        X509_free(pOsslCert);
                    }
                }
[...]
    return rc;
}

When the d2i_X509() call fails, the condition (2) is true nonetheless (NULL == NULL). In consequence, any X.509 certificate from the Windows certificate store, which is rejected by OpenSSL, leads to a failing X509_STORE_add_cert() call (3), VERR_NO_MEMORY (4) and ultimately to a failure of the calling function rtCrPkcs7VerifySignedDataUsingOpenSsl().

The first condition (1) regarding RTCRCERTCTX_F_ENC_X509_DER was corrected between VirtualBox 6.1.34 and 6.1.36. In its earlier form, it always evaluated to "false" and successfully prevented the erroneous code from being used. ;)

Some of the affected users (Bugtracker and forums) had plenty of free memory, and this VERR_NO_MEMORY here is the only one that can happen without an out of memory condition.

[galitsyn]

Hi fth0,

Thank you for pointing out. We will look at it.

[bird]

Thanks for tracking this down fth0! I've corrected the woolly error checking logic for all the buggy d2i calls I could find as well as applying the same level of paranoia to the other d2i calls. Also found a i2d call with possibly incorrect error checking. Applied fixes to 6.1.x and 7.0.x (trunk).

[Schimkat]

Replying to bird:

... Applied fixes to 6.1.x and 7.0.x (trunk).

Would you know when this will be released for the 6.1.x branch? We got a bunch of users having this issue, which is preventing completion of upgrade from 6.1.34. :-)