Environment freezes

[kdegraaf]

Hi,

I installed VB 4.2.4 on Oracle Linux 5.8. I created a virtual server with 120 GB ram and 16 processers. I installed Oracle Linux 5.5 as guest system.

When I copy a lot of data (60GB total) the guest then the guest freezes after 30GB. This happens with dynamic disk and fixed disk type.

[kdegraaf]

I'm copying from the host to the guest using scp.

I also tried: login with VNC to the guest. then copy using samba. But that had the same result, the guest freezes.

[Frank Mehnert]

Does it change anything if you decrease the number of CPUs to 12?

[Frank Mehnert]

And, I would be interested in a core dump of the VM process when the guest is hanging.

[kdegraaf]

Hi Frank,

Thanks for your quick reaction. I'm testing now how it goes with 12 cpu's. If it hangs again I will try to make a core dump.

Thanks Klaas

[kdegraaf]

Hi Frank,

I started the guest like this:

[root@fusionhost Desktop]# ulimit -c unlimited

[root@fusionhost Desktop]# sudo su

[root@fusionhost Desktop]# echo -n 1 > /proc/sys/kernel/core_uses_pid

[root@fusionhost Desktop]# echo -n 1 > /proc/sys/fs/suid_dumpable

[root@fusionhost Desktop]# exit

exit

[root@fusionhost Desktop]# /usr/lib/virtualbox/VirtualBox -startvm Fusion

now where is the core dump created?

Thanks Klaas

[Frank Mehnert]

It should be created in the directory where you started VirtualBox from.

[Frank Mehnert]

But the core dump is not created automatically, it is only created if you kill the hanging VM process like described on the core dump page.

[kdegraaf]

Core dump is uploaded to the ftp.

[kdegraaf]

Hi Frank,

Did you receive the core dump?

[Frank Mehnert]

Hi Klaas, yes, thank you. Had a short look but my environment lacks a few debugging symbols from system libraries. Have to solve this first but this will probably take some time as I'm now on vacation.

[kdegraaf]

Hi Frank,

Is there someone else who can take a look at our issue? Should I maybe log a ticket at metalink?

[kdegraaf]

Hi Frank,

I also tried with 8 cpu's, but the same result. Server hangs again.

[Carmelo]

Replying to kdegraaf:

Hi Frank,

Is there someone else who can take a look at our issue? Should I maybe log a ticket at metalink?

Hi Klaas,

Please note that opening a case via My Oracle Support (metalink) implies that you have a VirtualBox Commercial Support agreement per

​http://www.oracle.com/technetwork/server-storage/virtualbox/support/index.html

If you do not have a valid agreement, then you must continue to use the Community support. Oracle has a obligation to fix issues for customers with Commercial Support.

Regards,

Carmelo

[Frank Mehnert]

Got some more information from the core dump. This 12 core guest shows no sign of a VM process hang. All threads are waiting except one thread executing guest code. There is no sign of any deadlock. I assume when the hang happens then you are still able to pause and/or terminate the VM (there is no need to kill the VM process), right?

So what happens when your guest hangs? The guest does not accept any keyboard input nor any mouse events, is that right? Can you login to the guest via SSH (enable the SSH server in the guest before)?

[kdegraaf]

Hi Frank,

I just tested it. I can pause and shutdown the VM yes. But it does not accept keyboard and mouse input. Also I can't login with SSH. (SSH is working before I start copying)

[Frank Mehnert]

Sounds like the guest kernel crashed or stopped responding. Unfortunately such crashes can be only identified if the guest screen is not switched to graphics mode. So the next you could try is to reproduce the hang with a remote session (login with ssh into the guest, do your copy jobs from ssh) and make sure that the guest is no text mode (e.g. by switching to the 1st virtual terminal with HostKey+F1). I hope that there will be some output on the guest screen when the guest hangs.

[Frank Mehnert]

As written in private E-mail we need to know more about your test setup: Which guest kernel (2.6.18-*?), do you use 'divider=10' as kernel command line parameter? Do you experience the same problem with Linux 2.6.32 (which is the default kernel for OL5) on the host and the guest?

[kdegraaf]

Hi Frank,

I started the host again and switched to text modus. I started copy with scp and the copy stalled again after 30GB.

I takes about 20-25 minutes before it hangs. With 12 and 8 cores the server also hanged after 30GB so the time is the same I think. If you need to know this I can test this again if you want.

[kdegraaf]

Hi Frank,

The kernel is 2.6.18-308.24.1.0.1.el5. It's the same for the host and guest system. I don't know anything about kernel parameters.

I installed Oracle linux 5.8 as host and Oracle linux 5.5 as guest OS. I don't know why I have kernel 2.6.18 and not kernel 2.6.32.

I did a standard installation and After that I did setup ​http://public-yum.oracle.com/ and installed all available updates.

[Frank Mehnert]

Thanks Klaas for this information. You should definitely set up the kernel parameter 'divider=10' for your GUEST kernel as suggested in the VirtualBox UserManual. I don't believe that this will fix your guest hangs but the guest should definitely behave better and consume less CPU cycles. That kernel parameter will switch the guest to use a 100Hz system timer instead of a 1000Hz timer. 1000Hz is nice when running on bare metal but not in a VM. Add 'divider=10' to the kernel command line in the /boot/grub/grub.conf file in the guest and reboot your guest.

I have to check why the default kernel is 2.6.18 and not 2.6.32 on your setup. Can you check if a 2.6.32 kernel is available in your repository?

[kdegraaf]

divider parameter

[kdegraaf]

I uploaded a screen shot of how the grub.conf file looks now. Is it changed correctly?

If I run 'yum update kernel' I get: No packages marked for update. So I guess it's not available.

[kdegraaf]

Hi Frank,

I checked the repository file and only el5U5_base was enabled. If I also enable ol5u5_base I can update to kernel 2.6.32. Should I do that?

[Frank Mehnert]

Right, the kernel parameter is correct now. And yes, please enable the ol5u5_base repository and install the 2.6.32 kernel. If possible do this for the host as well as for the guest. Could you then repeat the scp test? I'm still trying to reproduce the problem with the 2.6.18 kernel locally.

[kdegraaf]

I updated the kernel on the host and guest to 2.6.32. I also updated Virtual Box to version 4.2.6. The files are copying now...

[kdegraaf]

Hi Frank,

Server hangs again. I do see some output now in screen of the guest.

[Frank Mehnert]

Thanks for these additional tests. Could you double-check that you really run Linux 2.6.32 in the guest and on the host (uname -r)?

[Frank Mehnert]

And could you also attach the corresponding VBox.log file of this VM session to this ticket?

[kdegraaf]

I double checked the kernel version and it's 2.6.32-100.0.19.el5 for the host and guest.

[kdegraaf]

Hi Frank,

Is there something else I can try?

[Frank Mehnert]

Hi Klaas, my other suggestion would be to try to update your guest to Oracle Linux 5.8 (right now it is 5.5 as you reported above).

I was still not able to reproduce this guest hang. My environment is now very close to yours except that my box is an AMD box (Sun Fire X4600 M2 with 32 cores and 256GB RAM) while yours is an Intel box. But I doubt that this difference is important here. My guest is configured for 120GB RAM and 16 cores. I ran scp jobs in the guest copying a huge sparse file (100GB) from the host to the guest's /dev/null. These jobs stall after 64GB but the network isn't down so I assume this is a different issue. And the guest isn't crashing, I can just restart the scp job. Doing the same scp job on the host (the host is initiator), it copies the complete file to the guest.

Could you also try to copy the files from the host to /dev/null on the guest (instead of copying it to the physical disk)? Does it still crash after some GBs? I will try the opposite.

[Frank Mehnert]

Another test would be to decrease the guest RAM size from 120GB to, for instance, 8GB and check if this makes any difference: Does the guest still hang, does the hang happen much sooner or later, this could give a hint where to search for the problem.

[kdegraaf]

Hi Frank,

I updated to 5.8. I tried with 8GB RAM and that was succesfull twice. Then I tried with 30 GB RAM and the copy stalled again. And I can't reach the server with putty. So It looks like it's a memory problem.

I will now try to copy to /dev/null

[kdegraaf]

Hi Frank,

How do I copy to /dev/null?

I get an error now: scp: /dev/null: Not a directory

[Frank Mehnert]

Hi Klaas, copy to /dev/null is possible if you don't copy as directory, for example 'scp host:file /dev/null' should be possible. Related to your screenshot you could perhaps do

scp /HD4/stage/* root@10.0.0.12:/dev/null

if all files are inside the stage directory.

[kdegraaf]

Hi Frank,

I tried your command, but I get the same error message.

[Frank Mehnert]

Sorry, my fault. Actually you have to do

for i in /HD4/stage/*; do scp $i root@10.0.0.12:/dev/null; done

Don't worry about the error messsage scp: /dev/null: truncate: Invalid argument, the file is copied to the guest anyway.

[kdegraaf]

That command was working and the copy was succesful. But now it does not go in a batch but 1 by 1. After each file I have to enter the password again.

[Frank Mehnert]

Check if 'ssh-agent' is running. This should be the case if you are logged into an X session. In that case, do 'ssh-add' to add the password for your ssh key. Next time you do 'ssh or scp', it's not necessary anymore to enter the password.

[Frank Mehnert]

To make this work you also need to add the public ssh key of the client (the machine you start ssh / scp from, it normally lives in ~/.ssh/id_rsa.pub or ~¯/.ssh/id_dsa.pub) to ~/.ssh/authorized_keys of the server (the machine you do ssh / scp to). Of course you have to create the ssh keys in ~/.ssh first. If you make their passwords empty, you even don't need to add the passphrase with ssh-add.

[kdegraaf]

Hi Frank,

This only to get the copy working. But is this also needed to get clear what is causing that the server is hanging?

Do you already have a clue what the cause is of our problems? Is it the copying or is it the usage of a lot of RAM?

[Frank Mehnert]

Right, fiddling with the SSH keys will only make debugging easier but will not help finding the cause of the trouble you observe.

Sorry, still no idea. I do happily copying here. I saw that with a 2.6.18 host kernel, copying from guest to the host over bridged networking is extremely slow. Did not see such problems when copying the other direction (host => guest). But with 2.6.32 this is much better, normal speed (I get constantly ~16MB/s in both directions) so you should never use 2.6.18 again.

It could be some RAM issue on your side because here I can also copy normal files which are much bigger (using 64GB files now) and I don't see any hiccup with 16 cores and 120GB RAM.

Are you sure that your hardware is 100% working? Could you perform some RAM check (e.g. memtest) over night?

[kdegraaf]

Hi Frank,

I will try memtest then.

[kdegraaf]

memtest is still running...

[kdegraaf]

I uploaded a picture of the current status of memtest. I says pass complete but it's still running.

[kdegraaf]

Hi Frank,

anything else I can test?

[Frank Mehnert]

I'm afraid that I'm running out of ideas. Let's summarize our findings until now as I still cannot reproduce your problems:

• Your host and guest is Oracle Linux 5.8 with Linux 2.6.32 and VirtualBox 4.2.6
• You do extensive network I/O and observe guest (Oracle Linux 5.8) hangs, more specific, your guest apparently panics as can be seen from your Screenshot-1.png.

Do you always see a guest panic like in Screenshot-1.png or is this sometimes different? Is the guest always panic'd or are you able to continue to work with the guest sometimes?

• You don't observe this guest panic if you decrease the guest RAM to 8GB but with 30GB you see this panic.
• You see this guest panic with 16, 12 and 8 guest cores.

In the mean time I'm not that sure that your problem is really SMP-related. Could you try with 1 guest core (the other configuration unchanged)?

It would be still interesting if these guest hangs do also occur if you copy to /dev/null of the guest.

I do assume that your guest does not show any disk failures. Can you check if

dd if=/dev/urandom of=foo bs=1M count=1
for i in `seq 1 64`; do cat foo >> foo64G; done

works in the guest? This will create a 64GB file with random content. This will take some time.

[aeichner]

In case the kernel panic happens always it might help to get the complete stack trace and the panic message from the guest. To get it you can configure a virtual serial port in VirtualBox and let it write everything to a file. Then configure the kernel to use the virtual serial port as a console. The file will hopefully contain the complete panic message when it occurs.

[kdegraaf]

Hi Frank,

This is now running:

dd if=/dev/urandom of=foo bs=1G count=1
for i in `seq 1 64`; do cat foo >> foo64G; done

Where are the files saved to when I ran this command:

for i in /HD4/stage/*; do scp $i root@10.0.0.12:/dev/null; done

I want to delete those files now.

[Frank Mehnert]

Hi Klaas, the idea of my suggestion above was to create one big file (foo64G) so you don't have to fiddle with several smaller files (and have to re-enter the ssh password every time). So were you able to copy foo64G or the files from /HD4/stage/* to /dev/null of the guest without a guest panic?

[kdegraaf]

Hi Frank,

I ran the command on the guest and it hanged again. Did you mean to run it on the host and copy to the guest?

I did ran this command "for i in /HD4/stage/*; do scp $i root@10.0.0.12:/dev/null; done" but where did I copy the files to? I can't find them on the guest system.

[Frank Mehnert]

Which IP has your host and which IP has your guest? Your 'scp' command line contains an IP address + file name (root@10.0.0.12:/dev/null) as target, therefore I would assume that you ran this command on the host.

[kdegraaf]

The host is 10.0.0.10 and the guest is 10.0.0.12.

[Frank Mehnert]

Therefore I would assume that you ran 'for i in /HD4/stage/*; do scp $i root@10.0.0.12:/dev/null' on the host.

Otherwise the files must be already available on the guest in the /HD4/stage/* directory and the scp command does not involve any network traffic between the host and the guest.

[kdegraaf]

Thanks for explaining.

How can I configure a virtual serial port?

[Frank Mehnert]

Just enable a virtual serial port in the VM settings of that VM, preferably by using the GUI but this works also with VBoxManage. Appending console=ttyS0 to the guest kernel command line (grub), the guest kernel should print kernel messages to the first serial port (IRQ 4, Port 0x3F8).

[kdegraaf]

I found the option 'enable serial port'.

What port mode should I select? Must 'create pipe' be selected?

[Frank Mehnert]

Use file, then the stuff will be written on a file on the host.

[kdegraaf]

Hi Frank,

I uploaded the output from the serial port. I hope it gives you some information.

[Frank Mehnert]

Unfortunately this log does not contain enough information. I would expect that it contains some information like you posted in Screenshot-1.png. The messages from output.txt show only some crash inside print_context_stack() but the context how we got there is missing. Was the log file truncated for some reason?

Also, is there any chance that your guest is damaged by an earlier experiment? How smooth went the original guest installation, did you need some tweaks? Did you have to resize the virtual guest hard disk? Did you see any other problems after you installed the guest except the guest panics you reported here?

[kdegraaf]

Hi Frank,

I don't know if the log is truncated.

I installed the guest without any problems, no tweaks where needed. I did not resize the hd. It's still the size it was created initially. I did not see any other problems.

[Frank Mehnert]

Ok. Did you see a similar guest screen like Screenshot-1.png during your last tests (make sure you switch the guest to text as you did before)?

And how about copying a big file from the host to /dev/null of the guest? This is what I wanted to know as well but from your last comments I don't know if you actually tested this or not. I still want to know if these guest panics are triggered by guest network activity, guest hard disk activity or if that does not matter. Copying to the guests /dev/null should not utilize the guest hard disk.

[kdegraaf]

Hi Frank,

I did not switch to text... I will test it again and check the output in text mode.

I just copied a file of 66G with scp from the host to the guest and that was successful.

[kdegraaf]

Copy of the zip files stalled again. No output on the console.

[Frank Mehnert]

So when you copied your zip files from the host to the guests hard disk, your guest panics but if you copy the zip files or the big 64G file from the host to the guests /dev/null device, the guest does not panic, is that right? Sorry, I want to be very sure here.

[kdegraaf]

Hi Frank,

You are correct.

[Frank Mehnert]

Could you attach the file /var/log/messages from your guest to this ticket? Please don't forget to compress this file.

[Frank Mehnert]

And please also attach /var/log/dmesg (just in case this information is not included in the messages file).

[kdegraaf]

I uploaded the files in the zip file log.zip.

[kdegraaf]

Hi Frank,

Do you have an update for us?

Thanks.

[Frank Mehnert]

The analysis of your log files didn't show any problem. To me this still looks like some hardware problem, either with memory or with the CPU / chipset. I did a lot of local tests and was not able to trigger any similar guest panic. Did you ever run any other big load (not VirtualBox VMs) on the machine you used for testing?

It could be also part of the problem that my big test box is an AMD machine while your box is Intel. I have to check if I can get some access to a comparable Intel box. If you have time, two additional tests would make sense:

1. Try with nested paging disabled. If you cannot reproduce the problem with a lot of RAM but nested paging disabled, this would give a hint where to search for a bug.
2. Try to narrow the interval 8GB (works) and 30GB (does not work). If we would need less RAM to reproduce this problem this would also help.

But still, we need to reproduce the problem.

[kdegraaf]

Hi Frank,

1. I tried copying with the option nested paging disabled and it was successful twice! The server only booted very slowly and copy started slow (1.5MB/s). But after a minute or so the copy was fast again (30MB/s).

2. I'm going to enable nested paging again. And then I'm going to narrow the interval.

[Frank Mehnert]

Interesting! I have access to an Intel box now as well and trying to reproduce your problem there.

[kdegraaf]

Hi Frank,

Until 30GB it is working fine.

[Frank Mehnert]

Thanks for testing. So far I don't need more tests from your side as I'm able to reproduce the problem here.

[kdegraaf]

Hi Frank,

That sounds good that you can reproduce the problem! Is it because of the Intel processor that you can reproduce it now?

[Frank Mehnert]

Yes, I see problem on the Intel box but not on the AMD box.

[kdegraaf]

Hi Frank,

when can we expect a solution to our problem?

[Frank Mehnert]

I cannot give an estimate yet. We found the problem but I don't know how difficult the fix will be.

[Frank Mehnert]

We found another workaround. The following command will disable VPIDs for your VM:

VBoxManage modifyvm VM_NAME --vtxvpid off

This will decrease the performance marginally but works around the problem as well (here it does). Don't forget to re-enable nested paging. We still don't fully understand the cause of this bug.

[kdegraaf]

Hi Frank,

The workaround is working!

[Frank Mehnert]

Hi Klaas, we found a better fix which makes it superfluous to disable VPIDs. If you have time, I would appreciate if you could test ​this test build. It should cleanly replace VirtualBox 4.2.6. It does not show the problem on my test box anymore.

[kdegraaf]

Hi Frank should I turn vtxpid on again?

With this command: VBoxManage modifyvm VM_NAME --vtxvpid on?

Thanks Klaas

[Frank Mehnert]

Hi Klaas, yes, please do so.

[kdegraaf]

Hi Frank,

Fix is applied. We had some performance issues with vtxvpid off, so hopefully that is gone now.

[kdegraaf]

Hi Frank,

We are still having some problems, Oracle installation teams tell me: CPU load goes high when there is huge IO.

Is there maybe a setting to fix this?

[kdegraaf]

Extra info: I also realize the CPU load going high when the RAM usage crosses above 40 GB.

[Frank Mehnert]

Could you attach a VBox.log file of such a VM session? With huge I/O, an increasing CPU load is expected, but of course the question is by which amount. What did you in your tests to trigger the high CPU load and in which amount did the CPU load increase?

[kdegraaf]

Hi Frank,

Logfiles are uploaded of two machines. Fusion is the machine with the high CPU. But both servers crashed this morning I think... They have the status aborted. Does that status means that they crashed?

This is the answer to your question for the Oracle installation team:

I didn't do anything manually to test or trigger the CPU load, but FA provisioning will trigger the high CPU usage as it bring up more than 30 weblogic server (Managed & Admin server) and does the configuration of all the individual application that's being part of this FA. It also loads the data into the IDM database as well as Fusion database.

We cannot estimate the CPU load as the FA configuration are in parallel at component level and have no control on it.

[Frank Mehnert]

Status 'aborted' most likely means that the VMs aborted. A core dump should help debugging.

Regarding the CPU load, of course if you start a lot of applications then the CPU load is expected to be high. So far I don't see the problem here. The CPU load might be also higher than on bare-metal because there is always some virtualization overhead. And the more virtual CPUs are assigned to the guest, the higher the synchronization overhead.

[kdegraaf]

Hi Frank,

Are there maybe some settings we can change on the guest that we can try?

Thanks Klaas

[Frank Mehnert]

To maximize the performance you should always assign no more virtual processors to the guest than necessary. Same for the virtual RAM. You should prefer to run several VMs in parallel instead of running one big VM with all resources assigned. Also, don't treat hyperthreads as real cores. Your host has 24 hyperthreads and 12 real cores. Therefore the sum of all virtual cores of all VMs should better not exceed 12. Better keep one extra host core. Same for the RAM: It's better to keep some RAM for the host for caching than distributing the available RAM between the guests.

[kdegraaf]

Hi Frank,

I don't understand the part about the cores. So I should not assing more cores than there are real cores you say.

Yes In virtual box I see 24 cores. But If I only use 12 of them, then the other 12 are not used. And how I see it then half of the capacity is not used. And that is not what you want.

Or is that not correct how I see it?

[Frank Mehnert]

I think your host has 12 cores with 2 hyperthreads per core. This is an Intel Xeon X5650 with 2.67GHz. The technical overview of this CPU can be found ​here. I assume that your host has has 2 of these processors. Each processor has 6 cores, so 12 threads per processor or 12 cores which is equal to 24 threads for the host.

The hyperthreads of a core have only a separate register set but everything else (instruction decoder, caches) is shared. Therefore extra hyperthreads will only marginally increase the performance of an application, and for virtualization, hyperthreads should be really not counted as real cores. See also ​here for an explanation of hyper-threading. Up to 30% performance gain is only possible for certain applications.

[kdegraaf]

Hi Frank,

Thanks for explaining. So when I use 12 cores of the 24 in virtual box it's not like don't use halve of the available capacity.

Can you please conform this?

[Frank Mehnert]

Yes, better set up your VMs that the sum of all virtual guest processors of all VMs you execute in parallel is not bigger than 12 on your host.

[kdegraaf]

Hi Frank,

We are still having problems with the installation of Oracle Fusion. The problem is the perfomance. Can you reply on the email that Richard van Maaren has send to you?

Thanks Klaas

[Frank Mehnert]

I think the original problem is fixed. Closing this ticket.