kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

[digea]

Greetings,

I have Nagios Enterprise Edition Unlimited Node running on a Virtual Machine on ESXi 5.5 for my company.
The VM has the following seup: CentOS 7 64-bit, 12GB memory, 8 vCores and 160GB storage, VM version 8.

Output of hostnamectl

Code: Select all

   Static hostname: localhost.localdomain
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 644ee04d4f2441618ed5648aa9be6090
           Boot ID: 84496b3e999c4a67852760a61830b7f8
    Virtualization: vmware
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.36.2.el7.x86_64
      Architecture: x86-64

Recently I started getting the following messages:

Code: Select all

[root@localhost ~]# grep 'soft lockup' /var/log/messages*
--- messages ommited ---
/var/log/messages-20161113:Nov 12 15:45:25 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 15:45:52 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 15:46:24 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 15:52:17 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 15:52:44 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 16:02:52 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/0:18]
/var/log/messages-20161113:Nov 12 16:03:21 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/0:18]
/var/log/messages-20161113:Nov 12 16:04:00 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/0:18]
/var/log/messages-20161113:Nov 12 16:04:28 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/0:18]
/var/log/messages-20161113:Nov 12 16:13:40 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/5:23]
/var/log/messages-20161113:Nov 12 16:14:08 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/5:23]
/var/log/messages-20161113:Nov 12 16:21:44 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 16:22:13 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 16:22:44 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 16:43:28 localhost kernel: BUG: soft lockup - CPU#3 stuck for 21s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 16:43:57 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 16:44:36 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 17:01:00 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 17:10:57 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
/var/log/messages-20161113:Nov 12 17:16:25 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 12 17:29:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 12 18:21:36 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 18:42:25 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 12 18:42:53 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 12 18:51:08 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/6:24]
/var/log/messages-20161113:Nov 12 18:51:37 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/6:24]
/var/log/messages-20161113:Nov 12 18:52:09 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/6:24]
/var/log/messages-20161113:Nov 12 19:01:24 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 12 19:30:28 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/0:18]
/var/log/messages-20161113:Nov 12 19:30:56 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/0:18]
/var/log/messages-20161113:Nov 12 19:47:04 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 20:16:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/0:18]
/var/log/messages-20161113:Nov 12 20:33:32 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 20:34:00 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 20:34:40 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 20:40:45 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 20:41:12 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 20:41:44 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/7:25]
/var/log/messages-20161113:Nov 12 20:42:21 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
--messages ommited --
/var/log/messages-20161113:Nov 13 02:16:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:26:40 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:27:09 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/4:22]
/var/log/messages-20161113:Nov 13 02:46:28 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 02:46:56 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 02:54:04 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 02:54:32 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:00:25 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:00:52 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:01:24 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:01:53 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:02:21 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:02:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:03:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/2:20]
/var/log/messages-20161113:Nov 13 03:09:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:10:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:10:56 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:11:24 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:11:53 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:12:20 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:32:48 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:33:16 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:33:52 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/7:25]
/var/log/messages-20161113:Nov 13 03:40:37 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:41:05 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:41:40 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:42:09 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:42:36 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:43:05 localhost kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]
/var/log/messages-20161113:Nov 13 03:43:32 localhost kernel: BUG: soft lockup - CPU#3 stuck for 22s! [rcuos/1:19]
--messages ommited --

and Nagios is responding very slowly to all actions and navigation/browsing.

What info would you like me to provide to help resolve this issue?

Thank you in advance.

[avandemore]

The only time I've ever seen such a bug, it's been related to the hypervisor and it's almost certainly not a Nagios bug. Please see these issues:

https://kb.vmware.com/selfservice/micro ... Id=1009996
http://unix.stackexchange.com/questions ... -x-seconds
https://access.redhat.com/solutions/21849

[digea]

Hello again,

I would like to know if there is a procedure to transfer an active Nagios license from the VM I mentioned above, to another Nagios installation, be it a new VM or a dedicated server (with the same IP address of course).
Also, is there a way to transfer all the settings, database etc so I don't have to reconfigure everything from the scratch?

Thank you in advance.

[avandemore]

Backing Up And Restoring Your Nagios XI System

Licensing questions are best directed at [email protected]. In general, you can activate up to 3 times per license. If you already have a key for an IP, you should be able to reuse it.