Page 1 of 1
Checks perpetually pending
Posted: Wed Jul 11, 2018 11:07 pm
by linuxnerd
Some of my checks are in a perpetually pending state.
Even when I force a check, nothing happens.
when I force the check, i sometimes see in the log
[1531367787] EXTERNAL COMMAND: SCHEDULE_FORCED_HOST_SVC_CHECKS;xxxxxxx;1531367811
but nothing else pertaining to that host.
My config:
https://support.nagios.com/forum/viewto ... 14#p255589
Re: Checks perpetually pending
Posted: Thu Jul 12, 2018 2:49 pm
by scottwilkerson
Make sure active checks are enabled for the failing services
Go to Service Information page for the service
Bottom center it should say
Active Checks: ENABLED
Re: Checks perpetually pending
Posted: Thu Jul 12, 2018 9:49 pm
by linuxnerd
they are active.
the checks did eventually run, but over 20 hours later.
Re: Checks perpetually pending
Posted: Fri Jul 13, 2018 2:42 pm
by tgriep
Can you run the following commands as root on your Nagios server and post the output?
Code: Select all
/usr/local/nagios/bin/nagiostats
ps -ef
I want to see the Nagios Stats and the processes that are running on the server.
Thanks
Re: Checks perpetually pending
Posted: Fri Jul 13, 2018 8:44 pm
by linuxnerd
Code: Select all
Nagios Stats 3.5.1
Copyright (c) 2003-2008 Ethan Galstad (www.nagios.org)
Last Modified: 08-30-2013
License: GPL
CURRENT STATUS DATA
------------------------------------------------------
Status File: /var/cache/nagios3/status.dat
Status File Age: 0d 0h 0m 8s
Status File Version: 3.5.1
Program Running Time: 1d 15h 28m 38s
Nagios PID: 3064
Used/High/Total Command Buffers: 0 / 2 / 16384
Total Services: 1613
Services Checked: 1613
Services Scheduled: 1613
Services Actively Checked: 1613
Services Passively Checked: 0
Total Service State Change: 0.000 / 12.370 / 0.125 %
Active Service Latency: 0.002 / 5.224 / 0.140 sec
Active Service Execution Time: 0.009 / 23.779 / 1.213 sec
Active Service State Change: 0.000 / 12.370 / 0.125 %
Active Services Last 1/5/15/60 min: 55 / 549 / 1371 / 1390
Passive Service Latency: 0.000 / 0.000 / 0.000 sec
Passive Service State Change: 0.000 / 0.000 / 0.000 %
Passive Services Last 1/5/15/60 min: 0 / 0 / 0 / 0
Services Ok/Warn/Unk/Crit: 1611 / 0 / 0 / 2
Services Flapping: 0
Services In Downtime: 0
Total Hosts: 237
Hosts Checked: 237
Hosts Scheduled: 237
Hosts Actively Checked: 237
Host Passively Checked: 0
Total Host State Change: 0.000 / 0.000 / 0.000 %
Active Host Latency: 0.000 / 5.604 / 3.273 sec
Active Host Execution Time: 0.007 / 0.144 / 0.049 sec
Active Host State Change: 0.000 / 0.000 / 0.000 %
Active Hosts Last 1/5/15/60 min: 0 / 30 / 237 / 237
Passive Host Latency: 0.000 / 0.000 / 0.000 sec
Passive Host State Change: 0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0
Hosts Up/Down/Unreach: 237 / 0 / 0
Hosts Flapping: 0
Hosts In Downtime: 0
Active Host Checks Last 1/5/15 min: 210 / 237 / 695
Scheduled: 210 / 237 / 694
On-demand: 0 / 0 / 1
Parallel: 210 / 237 / 696
Serial: 0 / 0 / 0
Cached: 0 / 0 / 0
Passive Host Checks Last 1/5/15 min: 0 / 0 / 0
Active Service Checks Last 1/5/15 min: 115 / 689 / 2125
Scheduled: 115 / 689 / 2125
On-demand: 0 / 0 / 0
Cached: 0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 0 / 0 / 0
External Commands Last 1/5/15 min: 0 / 0 / 0
Code: Select all
root 1 0 0 Jul03 ? 00:02:38 /sbin/init text text
root 2 0 0 Jul03 ? 00:00:00 [kthreadd]
root 3 2 0 Jul03 ? 00:00:10 [ksoftirqd/0]
root 5 2 0 Jul03 ? 00:00:00 [kworker/0:0H]
root 7 2 0 Jul03 ? 00:07:17 [rcu_sched]
root 8 2 0 Jul03 ? 00:00:00 [rcu_bh]
root 9 2 0 Jul03 ? 00:00:33 [migration/0]
root 10 2 0 Jul03 ? 00:00:03 [watchdog/0]
root 11 2 0 Jul03 ? 00:00:03 [watchdog/1]
root 12 2 0 Jul03 ? 00:00:34 [migration/1]
root 13 2 0 Jul03 ? 00:00:12 [ksoftirqd/1]
root 15 2 0 Jul03 ? 00:00:00 [kworker/1:0H]
root 16 2 0 Jul03 ? 00:00:02 [watchdog/2]
root 17 2 0 Jul03 ? 00:00:31 [migration/2]
root 18 2 0 Jul03 ? 00:00:13 [ksoftirqd/2]
root 20 2 0 Jul03 ? 00:00:00 [kworker/2:0H]
root 21 2 0 Jul03 ? 00:00:03 [watchdog/3]
root 22 2 0 Jul03 ? 00:00:33 [migration/3]
root 23 2 0 Jul03 ? 00:00:18 [ksoftirqd/3]
root 25 2 0 Jul03 ? 00:00:00 [kworker/3:0H]
root 26 2 0 Jul03 ? 00:00:00 [kdevtmpfs]
root 27 2 0 Jul03 ? 00:00:00 [netns]
root 28 2 0 Jul03 ? 00:00:00 [perf]
root 29 2 0 Jul03 ? 00:00:00 [khungtaskd]
root 30 2 0 Jul03 ? 00:00:00 [writeback]
root 31 2 0 Jul03 ? 00:00:00 [ksmd]
root 32 2 0 Jul03 ? 00:00:13 [khugepaged]
root 33 2 0 Jul03 ? 00:00:00 [crypto]
root 34 2 0 Jul03 ? 00:00:00 [kintegrityd]
root 35 2 0 Jul03 ? 00:00:00 [bioset]
root 36 2 0 Jul03 ? 00:00:00 [kblockd]
root 37 2 0 Jul03 ? 00:00:00 [ata_sff]
root 38 2 0 Jul03 ? 00:00:00 [md]
root 39 2 0 Jul03 ? 00:00:00 [devfreq_wq]
root 45 2 0 Jul03 ? 00:00:10 [kswapd0]
root 46 2 0 Jul03 ? 00:00:00 [vmstat]
root 47 2 0 Jul03 ? 00:00:00 [fsnotify_mark]
root 48 2 0 Jul03 ? 00:00:00 [ecryptfs-kthrea]
root 64 2 0 Jul03 ? 00:00:00 [kthrotld]
root 65 2 0 Jul03 ? 00:00:00 [acpi_thermal_pm]
root 66 2 0 Jul03 ? 00:00:00 [bioset]
root 67 2 0 Jul03 ? 00:00:00 [bioset]
root 68 2 0 Jul03 ? 00:00:00 [bioset]
root 69 2 0 Jul03 ? 00:00:00 [bioset]
root 70 2 0 Jul03 ? 00:00:00 [bioset]
root 71 2 0 Jul03 ? 00:00:00 [bioset]
root 72 2 0 Jul03 ? 00:00:00 [bioset]
root 73 2 0 Jul03 ? 00:00:00 [bioset]
root 74 2 0 Jul03 ? 00:00:00 [scsi_eh_0]
root 75 2 0 Jul03 ? 00:00:00 [scsi_tmf_0]
root 76 2 0 Jul03 ? 00:00:00 [scsi_eh_1]
root 77 2 0 Jul03 ? 00:00:00 [scsi_tmf_1]
root 83 2 0 Jul03 ? 00:00:00 [ipv6_addrconf]
root 96 2 0 Jul03 ? 00:00:00 [deferwq]
root 97 2 0 Jul03 ? 00:00:00 [charger_manager]
root 98 2 0 Jul03 ? 00:00:00 [bioset]
root 143 2 0 Jul03 ? 00:00:00 [bioset]
root 144 2 0 Jul03 ? 00:00:00 [bioset]
root 145 2 0 Jul03 ? 00:00:00 [bioset]
root 146 2 0 Jul03 ? 00:00:00 [bioset]
root 147 2 0 Jul03 ? 00:00:00 [bioset]
root 148 2 0 Jul03 ? 00:00:00 [bioset]
root 149 2 0 Jul03 ? 00:00:00 [bioset]
root 150 2 0 Jul03 ? 00:00:00 [bioset]
root 151 2 0 Jul03 ? 00:00:00 [scsi_eh_2]
root 152 2 0 Jul03 ? 00:00:00 [scsi_tmf_2]
root 153 2 0 Jul03 ? 00:00:00 [vmw_pvscsi_wq_2]
root 154 2 0 Jul03 ? 00:00:00 [bioset]
root 155 2 0 Jul03 ? 00:00:00 [bioset]
root 172 2 0 Jul03 ? 00:00:00 [kpsmoused]
root 173 2 0 Jul03 ? 00:00:00 [kworker/2:1H]
root 174 2 0 Jul03 ? 00:00:00 [ttm_swap]
root 204 2 0 Jul03 ? 00:00:00 [raid5wq]
root 221 2 0 Jul03 ? 00:00:00 [kdmflush]
root 224 2 0 Jul03 ? 00:00:00 [bioset]
root 233 2 0 Jul03 ? 00:00:00 [kworker/0:1H]
root 250 2 0 Jul03 ? 00:00:00 [jbd2/dm-0-8]
root 251 2 0 Jul03 ? 00:00:00 [ext4-rsv-conver]
root 294 2 0 Jul03 ? 00:00:00 [kworker/1:1H]
root 295 1 0 Jul03 ? 00:01:01 /lib/systemd/systemd-journald
root 315 2 0 Jul03 ? 00:00:00 [kauditd]
root 329 2 0 Jul03 ? 00:00:00 [rpciod]
root 340 1 0 Jul03 ? 00:00:00 /sbin/lvmetad -f
root 347 1 0 Jul03 ? 00:00:01 /lib/systemd/systemd-udevd
root 440 2 0 Jul03 ? 00:00:00 [kdmflush]
root 441 2 0 Jul03 ? 00:00:00 [bioset]
root 445 2 0 Jul03 ? 00:00:00 [kdmflush]
root 446 2 0 Jul03 ? 00:00:00 [bioset]
root 545 2 0 Jul03 ? 00:00:00 [jbd2/sda1-8]
root 546 2 0 Jul03 ? 00:00:00 [ext4-rsv-conver]
root 551 2 0 Jul03 ? 00:00:36 [jbd2/dm-2-8]
root 552 2 0 Jul03 ? 00:00:00 [ext4-rsv-conver]
root 573 1 0 Jul03 ? 00:11:01 /usr/bin/vmtoolsd
root 752 1 0 Jul03 ? 00:00:17 /lib/systemd/systemd-logind
message+ 754 1 0 Jul03 ? 00:00:35 /usr/bin/dbus-daemon --system --address=systemd: --nofork -
root 781 1 0 Jul03 ? 00:00:27 /usr/lib/accountsservice/accounts-daemon
root 795 1 0 Jul03 ? 00:00:05 /usr/sbin/cron -f
root 802 1 0 Jul03 ? 00:00:00 /usr/bin/VGAuthService
root 842 1 0 Jul03 ? 00:00:03 /usr/sbin/sshd -D
root 891 1 0 Jul03 ? 00:00:39 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
root 941 1 0 Jul03 tty1 00:00:00 /sbin/agetty --noclear tty1 linux
ntp 962 1 0 Jul03 ? 00:00:45 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 109:116
root 963 1 0 Jul03 ? 00:00:08 /usr/lib/policykit-1/polkitd --no-debug
root 987 2 0 01:06 ? 00:00:00 [kworker/3:2]
snmp 1003 1 0 Jul03 ? 00:06:19 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux
root 1085 2 0 Jul03 ? 00:00:01 [kworker/3:1H]
root 1091 1 0 Jul03 ? 00:00:07 /usr/lib/postfix/sbin/master
postfix 1093 1091 0 Jul03 ? 00:00:02 qmgr -l -t unix -u
root 1361 1 44 01:06 ? 00:14:00 /bin/bash /usr/local/bin/bulk_ocsp
root 1363 2 0 01:06 ? 00:00:00 [kworker/1:1]
nagios 3064 1 0 Jul11 ? 00:26:17 /usr/sbin/nagios3 -x -d /etc/nagios3/nagios.cfg
root 4191 17757 0 01:33 ? 00:00:00 sleep 764s
root 6503 2 0 Jul13 ? 00:00:00 [kworker/2:0]
root 8111 842 0 01:36 ? 00:00:00 sshd: XXX [priv]
XXX 8113 1 0 01:36 ? 00:00:00 /lib/systemd/systemd --user
root 8114 2 0 01:36 ? 00:00:00 [kworker/2:1]
root 8118 2 0 01:36 ? 00:00:00 [kworker/1:2]
XXX 8120 8113 0 01:36 ? 00:00:00 (sd-pam)
XXX 8301 8111 0 01:36 ? 00:00:00 sshd: XXX@pts/0
XXX 8320 8301 0 01:36 pts/0 00:00:00 -bash
root 8525 2 0 Jul13 ? 00:00:00 [kworker/1:0]
root 10062 2 0 01:38 ? 00:00:00 [kworker/u8:2]
XXX 10374 8320 0 01:38 pts/0 00:00:00 ps -ef
postfix 12994 1091 0 00:51 ? 00:00:00 pickup -l -t unix -u -c
root 16201 2 0 Jul13 ? 00:00:02 [kworker/3:1]
root 16203 2 0 Jul13 ? 00:00:00 [kworker/0:2]
syslog 16231 1 0 Jul12 ? 00:00:12 /usr/sbin/rsyslogd -n
root 20220 1 0 Jul08 ? 00:00:17 /usr/sbin/apache2 -k start
www-data 23553 20220 0 Jul13 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 23554 20220 0 Jul13 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 23555 20220 0 Jul13 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 23556 20220 0 Jul13 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 23557 20220 0 Jul13 ? 00:00:00 /usr/sbin/apache2 -k start
root 28603 2 0 01:27 ? 00:00:00 [kworker/u8:0]
www-data 28655 20220 0 Jul13 ? 00:00:00 /usr/sbin/apache2 -k start
root 29174 2 0 Jul13 ? 00:00:00 [kworker/2:2]
root 29175 2 0 Jul13 ? 00:00:00 [kworker/0:1]
root 31196 2 0 01:05 ? 00:00:00 [kworker/u8:1]
nagios 32513 1 0 Jul09 ? 00:00:02 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
Re: Checks perpetually pending
Posted: Sun Jul 15, 2018 8:35 pm
by linuxnerd
i think i forgot to reload the service.
the service check wasn't present into the objects.dat file.
after repushing my config / reloading, it seems to be running.
Re: Checks perpetually pending
Posted: Mon Jul 16, 2018 2:37 pm
by jomann
So are the checks happening now? Is this issue resolved?
Re: Checks perpetually pending
Posted: Wed Jul 18, 2018 7:13 am
by linuxnerd
i suppose so.
Re: Checks perpetually pending
Posted: Wed Jul 18, 2018 9:39 am
by tmcdonald
Did you have further (related) questions or are we good to lock this up?