Nagios server getting hung

[bosecorp]

I was trying to see what caused the server to go in the hung state.
We can see soft lockups on the CPUs and this may be one of the reason why the server was not responsive.

Let us know if you need any further logs to investigate this issue further.

# egrep -i "Bug: soft lockup" /var/log/messages
Jul 10 09:28:09 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 75s! [sshd:21756]
Jul 11 13:44:38 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 65s! [httpd:24135]
Jul 11 13:44:38 nagmonus1 kernel: BUG: soft lockup - CPU#8 stuck for 66s! [automount:30611]
Jul 11 13:44:38 nagmonus1 kernel: BUG: soft lockup - CPU#19 stuck for 66s! [httpd:30394]
Jul 11 13:44:39 nagmonus1 kernel: BUG: soft lockup - CPU#17 stuck for 66s! [httpd:30427]
Jul 11 13:44:39 nagmonus1 kernel: BUG: soft lockup - CPU#10 stuck for 66s! [sshd:30601]
Jul 11 13:44:39 nagmonus1 kernel: BUG: soft lockup - CPU#8 stuck for 67s! [crond:30640]
Jul 11 13:44:39 nagmonus1 kernel: BUG: soft lockup - CPU#19 stuck for 67s! [events/19:118]
Jul 11 13:46:51 nagmonus1 kernel: BUG: soft lockup - CPU#1 stuck for 66s! [php:30743]
Jul 11 13:46:51 nagmonus1 kernel: BUG: soft lockup - CPU#2 stuck for 66s! [php:30758]
Jul 11 13:47:50 nagmonus1 kernel: BUG: soft lockup - CPU#4 stuck for 67s! [httpd:30961]
Jul 11 13:48:15 nagmonus1 kernel: BUG: soft lockup - CPU#1 stuck for 68s! [events/1:100]
Jul 11 13:49:38 nagmonus1 kernel: BUG: soft lockup - CPU#1 stuck for 67s! [check_ifopersta:1297]
Jul 11 13:51:21 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 318s! [php:30798]
Jul 11 13:54:55 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 105s! [php:1988]
Jul 13 13:06:41 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 111s! [flush-253:7:1932]
Jul 13 13:09:17 nagmonus1 kernel: BUG: soft lockup - CPU#3 stuck for 67s! [nagios:32195]
Jul 13 13:09:17 nagmonus1 kernel: BUG: soft lockup - CPU#23 stuck for 66s! [mysqld:13905]
Jul 13 13:09:38 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 95s! [httpd:14670]
Jul 13 13:09:38 nagmonus1 kernel: BUG: soft lockup - CPU#10 stuck for 86s! [auditd:3057]
Jul 13 13:09:39 nagmonus1 kernel: BUG: soft lockup - CPU#14 stuck for 84s! [mod_gearman2_wo:3883]
Jul 13 13:09:39 nagmonus1 kernel: BUG: soft lockup - CPU#8 stuck for 95s! [php:13964]
Jul 13 13:09:39 nagmonus1 kernel: BUG: soft lockup - CPU#2 stuck for 95s! [mod_gearman2_wo:24943]
Jul 13 13:09:39 nagmonus1 kernel: BUG: soft lockup - CPU#5 stuck for 87s! [mysqld:13907]
Jul 13 13:09:39 nagmonus1 kernel: BUG: soft lockup - CPU#13 stuck for 65s! [postmaster:16639]
Jul 13 13:09:39 nagmonus1 kernel: BUG: soft lockup - CPU#6 stuck for 87s! [postmaster:13888]
Jul 13 13:09:40 nagmonus1 kernel: BUG: soft lockup - CPU#9 stuck for 88s! [check_icmp:16633]
Jul 13 13:09:40 nagmonus1 kernel: BUG: soft lockup - CPU#20 stuck for 88s! [sshd:16641]
Jul 13 13:09:40 nagmonus1 kernel: BUG: soft lockup - CPU#17 stuck for 88s! [sed:16629]
Jul 13 13:09:40 nagmonus1 kernel: BUG: soft lockup - CPU#4 stuck for 88s! [mod_gearman2_wo:31225]
Jul 13 13:09:40 nagmonus1 kernel: BUG: soft lockup - CPU#12 stuck for 88s! [sshd:16642]
Jul 13 13:10:53 nagmonus1 kernel: BUG: soft lockup - CPU#2 stuck for 69s! [mod_gearman2_wo:14183]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#16 stuck for 87s! [httpd:22730]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#18 stuck for 87s! [mrtg:25806]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#3 stuck for 87s! [httpd:22286]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#1 stuck for 87s! [check_ifopersta:27620]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#2 stuck for 87s! [check_wmi_plus.:27105]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#7 stuck for 87s! [httpd:23819]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#5 stuck for 87s! [httpd:30211]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#6 stuck for 87s! [wmic:27622]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#8 stuck for 87s! [httpd:22785]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#11 stuck for 87s! [php:22344]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#13 stuck for 87s! [httpd:22192]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#17 stuck for 87s! [check_ifopersta:27617]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#4 stuck for 87s! [httpd:22851]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#9 stuck for 87s! [httpd:16721]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#10 stuck for 87s! [httpd:23609]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#19 stuck for 87s! [check_cloudwatc:27547]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#22 stuck for 87s! [httpd:13643]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#23 stuck for 87s! [httpd:23138]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#21 stuck for 87s! [php:18978]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#20 stuck for 87s! [statusjson.cgi:23562]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#15 stuck for 87s! [mysqld:14625]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#14 stuck for 87s! [check_ifopersta:27616]
Jul 13 13:16:41 nagmonus1 kernel: BUG: soft lockup - CPU#12 stuck for 87s! [httpd:13881]
Jul 13 13:23:11 nagmonus1 kernel: BUG: soft lockup - CPU#1 stuck for 61s! [httpd:22286]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#16 stuck for 76s! [wmic:28621]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 84s! [kblockd/0:154]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#6 stuck for 66s! [php:28364]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#3 stuck for 66s! [mrtg:25805]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#5 stuck for 66s! [httpd:23610]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#4 stuck for 66s! [httpd:27747]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#9 stuck for 66s! [httpd:22901]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#13 stuck for 66s! [php:28594]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#15 stuck for 66s! [httpd:22219]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#14 stuck for 66s! [php:27973]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#11 stuck for 66s! [mod_gearman2_wo:28653]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#12 stuck for 66s! [httpd:23818]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#8 stuck for 66s! [php:27803]
Jul 13 13:23:17 nagmonus1 kernel: BUG: soft lockup - CPU#10 stuck for 66s! [php:27947]
Jul 13 13:23:18 nagmonus1 kernel: BUG: soft lockup - CPU#7 stuck for 66s! [php:27942]
Jul 13 13:23:18 nagmonus1 kernel: BUG: soft lockup - CPU#2 stuck for 66s! [httpd:23607]
Jul 13 13:23:18 nagmonus1 kernel: BUG: soft lockup - CPU#21 stuck for 66s! [httpd:14214]
Jul 13 13:23:18 nagmonus1 kernel: BUG: soft lockup - CPU#20 stuck for 66s! [httpd:22788]
Jul 13 13:23:19 nagmonus1 kernel: BUG: soft lockup - CPU#17 stuck for 66s! [php:28237]
Jul 13 13:23:19 nagmonus1 kernel: BUG: soft lockup - CPU#18 stuck for 66s! [httpd:27900]
Jul 13 13:23:20 nagmonus1 kernel: BUG: soft lockup - CPU#23 stuck for 66s! [check_wmi_plus.:28034]
Jul 13 13:23:20 nagmonus1 kernel: BUG: soft lockup - CPU#22 stuck for 66s! [httpd:27785]
Jul 13 13:26:24 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 74s! [httpd:27889]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#8 stuck for 71s! [php:28749]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#9 stuck for 71s! [httpd:28657]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#16 stuck for 71s! [sshd:29345]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#21 stuck for 71s! [php:28802]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#23 stuck for 71s! [php:28324]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#20 stuck for 71s! [httpd:28853]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#12 stuck for 71s! [httpd:13507]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#18 stuck for 71s! [httpd:27900]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#17 stuck for 71s! [httpd:23817]
Jul 13 13:26:36 nagmonus1 kernel: BUG: soft lockup - CPU#1 stuck for 71s! [httpd:30211]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#13 stuck for 71s! [postmaster:29271]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#10 stuck for 71s! [php:22416]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#7 stuck for 70s! [php:29195]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#2 stuck for 71s! [httpd:23607]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#3 stuck for 71s! [httpd:22192]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#5 stuck for 71s! [php:28829]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#6 stuck for 71s! [httpd:22901]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#4 stuck for 71s! [php:28735]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#14 stuck for 71s! [check_wmi_plus.:29255]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#15 stuck for 71s! [httpd:28848]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#19 stuck for 71s! [httpd:27871]
Jul 13 13:26:37 nagmonus1 kernel: BUG: soft lockup - CPU#22 stuck for 71s! [mrtg:28562]
Jul 13 13:28:45 nagmonus1 kernel: BUG: soft lockup - CPU#0 stuck for 70s! [httpd:14650]
Jul 13 13:28:50 nagmonus1 kernel: BUG: soft lockup - CPU#9 stuck for 61s! [httpd:28657]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#2 stuck for 69s! [postmaster:28445]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#3 stuck for 69s! [mrtg:25805]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#5 stuck for 69s! [php:28744]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#1 stuck for 69s! [httpd:29321]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#6 stuck for 69s! [httpd:22730]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#7 stuck for 69s! [httpd:27783]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#8 stuck for 69s! [check_wmi_plus.:29367]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#15 stuck for 69s! [httpd:29315]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#12 stuck for 69s! [httpd:13507]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#14 stuck for 69s! [httpd:27784]
Jul 13 13:28:59 nagmonus1 kernel: BUG: soft lockup - CPU#10 stuck for 69s! [httpd:22851]

[bolson]

Is your Nagios server running in a virtual environment? If so, we often see these soft lockups when CPU resources on the host are overcomitted. If this is the case you'll want to talk to your VMWare admin about CPU reservation for the VM.

[bosecorp]

Yes. we are are running on VMware environment. We need to know whats occupying the CPUs here? Can you please help us get that data?

[bolson]

The first step would probably be to work with your VMWare admin to see if you're overcomitting CPU. If so, you can reserve CPU for your Nagios server which should mitigate these soft lockups.

[bosecorp]

We are all set here

[kyang]

Closing this thread.

Feel free to post another thread if you have more questions!
Thanks for using the Nagios Support Forums!