Page 1 of 1

Checks perpetually pending

Posted: Wed Jul 11, 2018 11:07 pm
by linuxnerd
Some of my checks are in a perpetually pending state.
Even when I force a check, nothing happens.

when I force the check, i sometimes see in the log
[1531367787] EXTERNAL COMMAND: SCHEDULE_FORCED_HOST_SVC_CHECKS;xxxxxxx;1531367811
but nothing else pertaining to that host.

My config:
https://support.nagios.com/forum/viewto ... 14#p255589

Re: Checks perpetually pending

Posted: Thu Jul 12, 2018 2:49 pm
by scottwilkerson
Make sure active checks are enabled for the failing services

Go to Service Information page for the service
Bottom center it should say
Active Checks: ENABLED

Re: Checks perpetually pending

Posted: Thu Jul 12, 2018 9:49 pm
by linuxnerd
they are active.
the checks did eventually run, but over 20 hours later.

Re: Checks perpetually pending

Posted: Fri Jul 13, 2018 2:42 pm
by tgriep
Can you run the following commands as root on your Nagios server and post the output?

Code: Select all

/usr/local/nagios/bin/nagiostats
ps -ef
I want to see the Nagios Stats and the processes that are running on the server.

Thanks

Re: Checks perpetually pending

Posted: Fri Jul 13, 2018 8:44 pm
by linuxnerd

Code: Select all

Nagios Stats 3.5.1
Copyright (c) 2003-2008 Ethan Galstad (www.nagios.org)
Last Modified: 08-30-2013
License: GPL

CURRENT STATUS DATA
------------------------------------------------------
Status File:                            /var/cache/nagios3/status.dat
Status File Age:                        0d 0h 0m 8s
Status File Version:                    3.5.1

Program Running Time:                   1d 15h 28m 38s
Nagios PID:                             3064
Used/High/Total Command Buffers:        0 / 2 / 16384

Total Services:                         1613
Services Checked:                       1613
Services Scheduled:                     1613
Services Actively Checked:              1613
Services Passively Checked:             0
Total Service State Change:             0.000 / 12.370 / 0.125 %
Active Service Latency:                 0.002 / 5.224 / 0.140 sec
Active Service Execution Time:          0.009 / 23.779 / 1.213 sec
Active Service State Change:            0.000 / 12.370 / 0.125 %
Active Services Last 1/5/15/60 min:     55 / 549 / 1371 / 1390
Passive Service Latency:                0.000 / 0.000 / 0.000 sec
Passive Service State Change:           0.000 / 0.000 / 0.000 %
Passive Services Last 1/5/15/60 min:    0 / 0 / 0 / 0
Services Ok/Warn/Unk/Crit:              1611 / 0 / 0 / 2
Services Flapping:                      0
Services In Downtime:                   0

Total Hosts:                            237
Hosts Checked:                          237
Hosts Scheduled:                        237
Hosts Actively Checked:                 237
Host Passively Checked:                 0
Total Host State Change:                0.000 / 0.000 / 0.000 %
Active Host Latency:                    0.000 / 5.604 / 3.273 sec
Active Host Execution Time:             0.007 / 0.144 / 0.049 sec
Active Host State Change:               0.000 / 0.000 / 0.000 %
Active Hosts Last 1/5/15/60 min:        0 / 30 / 237 / 237
Passive Host Latency:                   0.000 / 0.000 / 0.000 sec
Passive Host State Change:              0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min:       0 / 0 / 0 / 0
Hosts Up/Down/Unreach:                  237 / 0 / 0
Hosts Flapping:                         0
Hosts In Downtime:                      0

Active Host Checks Last 1/5/15 min:     210 / 237 / 695
   Scheduled:                           210 / 237 / 694
   On-demand:                           0 / 0 / 1
   Parallel:                            210 / 237 / 696
   Serial:                              0 / 0 / 0
   Cached:                              0 / 0 / 0
Passive Host Checks Last 1/5/15 min:    0 / 0 / 0
Active Service Checks Last 1/5/15 min:  115 / 689 / 2125
   Scheduled:                           115 / 689 / 2125
   On-demand:                           0 / 0 / 0
   Cached:                              0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 0 / 0 / 0

External Commands Last 1/5/15 min:      0 / 0 / 0

Code: Select all

root         1     0  0 Jul03 ?        00:02:38 /sbin/init text text
root         2     0  0 Jul03 ?        00:00:00 [kthreadd]
root         3     2  0 Jul03 ?        00:00:10 [ksoftirqd/0]
root         5     2  0 Jul03 ?        00:00:00 [kworker/0:0H]
root         7     2  0 Jul03 ?        00:07:17 [rcu_sched]
root         8     2  0 Jul03 ?        00:00:00 [rcu_bh]
root         9     2  0 Jul03 ?        00:00:33 [migration/0]
root        10     2  0 Jul03 ?        00:00:03 [watchdog/0]
root        11     2  0 Jul03 ?        00:00:03 [watchdog/1]
root        12     2  0 Jul03 ?        00:00:34 [migration/1]
root        13     2  0 Jul03 ?        00:00:12 [ksoftirqd/1]
root        15     2  0 Jul03 ?        00:00:00 [kworker/1:0H]
root        16     2  0 Jul03 ?        00:00:02 [watchdog/2]
root        17     2  0 Jul03 ?        00:00:31 [migration/2]
root        18     2  0 Jul03 ?        00:00:13 [ksoftirqd/2]
root        20     2  0 Jul03 ?        00:00:00 [kworker/2:0H]
root        21     2  0 Jul03 ?        00:00:03 [watchdog/3]
root        22     2  0 Jul03 ?        00:00:33 [migration/3]
root        23     2  0 Jul03 ?        00:00:18 [ksoftirqd/3]
root        25     2  0 Jul03 ?        00:00:00 [kworker/3:0H]
root        26     2  0 Jul03 ?        00:00:00 [kdevtmpfs]
root        27     2  0 Jul03 ?        00:00:00 [netns]
root        28     2  0 Jul03 ?        00:00:00 [perf]
root        29     2  0 Jul03 ?        00:00:00 [khungtaskd]
root        30     2  0 Jul03 ?        00:00:00 [writeback]
root        31     2  0 Jul03 ?        00:00:00 [ksmd]
root        32     2  0 Jul03 ?        00:00:13 [khugepaged]
root        33     2  0 Jul03 ?        00:00:00 [crypto]
root        34     2  0 Jul03 ?        00:00:00 [kintegrityd]
root        35     2  0 Jul03 ?        00:00:00 [bioset]
root        36     2  0 Jul03 ?        00:00:00 [kblockd]
root        37     2  0 Jul03 ?        00:00:00 [ata_sff]
root        38     2  0 Jul03 ?        00:00:00 [md]
root        39     2  0 Jul03 ?        00:00:00 [devfreq_wq]
root        45     2  0 Jul03 ?        00:00:10 [kswapd0]
root        46     2  0 Jul03 ?        00:00:00 [vmstat]
root        47     2  0 Jul03 ?        00:00:00 [fsnotify_mark]
root        48     2  0 Jul03 ?        00:00:00 [ecryptfs-kthrea]
root        64     2  0 Jul03 ?        00:00:00 [kthrotld]
root        65     2  0 Jul03 ?        00:00:00 [acpi_thermal_pm]
root        66     2  0 Jul03 ?        00:00:00 [bioset]
root        67     2  0 Jul03 ?        00:00:00 [bioset]
root        68     2  0 Jul03 ?        00:00:00 [bioset]
root        69     2  0 Jul03 ?        00:00:00 [bioset]
root        70     2  0 Jul03 ?        00:00:00 [bioset]
root        71     2  0 Jul03 ?        00:00:00 [bioset]
root        72     2  0 Jul03 ?        00:00:00 [bioset]
root        73     2  0 Jul03 ?        00:00:00 [bioset]
root        74     2  0 Jul03 ?        00:00:00 [scsi_eh_0]
root        75     2  0 Jul03 ?        00:00:00 [scsi_tmf_0]
root        76     2  0 Jul03 ?        00:00:00 [scsi_eh_1]
root        77     2  0 Jul03 ?        00:00:00 [scsi_tmf_1]
root        83     2  0 Jul03 ?        00:00:00 [ipv6_addrconf]
root        96     2  0 Jul03 ?        00:00:00 [deferwq]
root        97     2  0 Jul03 ?        00:00:00 [charger_manager]
root        98     2  0 Jul03 ?        00:00:00 [bioset]
root       143     2  0 Jul03 ?        00:00:00 [bioset]
root       144     2  0 Jul03 ?        00:00:00 [bioset]
root       145     2  0 Jul03 ?        00:00:00 [bioset]
root       146     2  0 Jul03 ?        00:00:00 [bioset]
root       147     2  0 Jul03 ?        00:00:00 [bioset]
root       148     2  0 Jul03 ?        00:00:00 [bioset]
root       149     2  0 Jul03 ?        00:00:00 [bioset]
root       150     2  0 Jul03 ?        00:00:00 [bioset]
root       151     2  0 Jul03 ?        00:00:00 [scsi_eh_2]
root       152     2  0 Jul03 ?        00:00:00 [scsi_tmf_2]
root       153     2  0 Jul03 ?        00:00:00 [vmw_pvscsi_wq_2]
root       154     2  0 Jul03 ?        00:00:00 [bioset]
root       155     2  0 Jul03 ?        00:00:00 [bioset]
root       172     2  0 Jul03 ?        00:00:00 [kpsmoused]
root       173     2  0 Jul03 ?        00:00:00 [kworker/2:1H]
root       174     2  0 Jul03 ?        00:00:00 [ttm_swap]
root       204     2  0 Jul03 ?        00:00:00 [raid5wq]
root       221     2  0 Jul03 ?        00:00:00 [kdmflush]
root       224     2  0 Jul03 ?        00:00:00 [bioset]
root       233     2  0 Jul03 ?        00:00:00 [kworker/0:1H]
root       250     2  0 Jul03 ?        00:00:00 [jbd2/dm-0-8]
root       251     2  0 Jul03 ?        00:00:00 [ext4-rsv-conver]
root       294     2  0 Jul03 ?        00:00:00 [kworker/1:1H]
root       295     1  0 Jul03 ?        00:01:01 /lib/systemd/systemd-journald
root       315     2  0 Jul03 ?        00:00:00 [kauditd]
root       329     2  0 Jul03 ?        00:00:00 [rpciod]
root       340     1  0 Jul03 ?        00:00:00 /sbin/lvmetad -f
root       347     1  0 Jul03 ?        00:00:01 /lib/systemd/systemd-udevd
root       440     2  0 Jul03 ?        00:00:00 [kdmflush]
root       441     2  0 Jul03 ?        00:00:00 [bioset]
root       445     2  0 Jul03 ?        00:00:00 [kdmflush]
root       446     2  0 Jul03 ?        00:00:00 [bioset]
root       545     2  0 Jul03 ?        00:00:00 [jbd2/sda1-8]
root       546     2  0 Jul03 ?        00:00:00 [ext4-rsv-conver]
root       551     2  0 Jul03 ?        00:00:36 [jbd2/dm-2-8]
root       552     2  0 Jul03 ?        00:00:00 [ext4-rsv-conver]
root       573     1  0 Jul03 ?        00:11:01 /usr/bin/vmtoolsd
root       752     1  0 Jul03 ?        00:00:17 /lib/systemd/systemd-logind
message+   754     1  0 Jul03 ?        00:00:35 /usr/bin/dbus-daemon --system --address=systemd: --nofork -
root       781     1  0 Jul03 ?        00:00:27 /usr/lib/accountsservice/accounts-daemon
root       795     1  0 Jul03 ?        00:00:05 /usr/sbin/cron -f
root       802     1  0 Jul03 ?        00:00:00 /usr/bin/VGAuthService
root       842     1  0 Jul03 ?        00:00:03 /usr/sbin/sshd -D
root       891     1  0 Jul03 ?        00:00:39 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
root       941     1  0 Jul03 tty1     00:00:00 /sbin/agetty --noclear tty1 linux
ntp        962     1  0 Jul03 ?        00:00:45 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 109:116
root       963     1  0 Jul03 ?        00:00:08 /usr/lib/policykit-1/polkitd --no-debug
root       987     2  0 01:06 ?        00:00:00 [kworker/3:2]
snmp      1003     1  0 Jul03 ?        00:06:19 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux
root      1085     2  0 Jul03 ?        00:00:01 [kworker/3:1H]
root      1091     1  0 Jul03 ?        00:00:07 /usr/lib/postfix/sbin/master
postfix   1093  1091  0 Jul03 ?        00:00:02 qmgr -l -t unix -u
root      1361     1 44 01:06 ?        00:14:00 /bin/bash /usr/local/bin/bulk_ocsp
root      1363     2  0 01:06 ?        00:00:00 [kworker/1:1]
nagios    3064     1  0 Jul11 ?        00:26:17 /usr/sbin/nagios3 -x -d /etc/nagios3/nagios.cfg
root      4191 17757  0 01:33 ?        00:00:00 sleep 764s
root      6503     2  0 Jul13 ?        00:00:00 [kworker/2:0]
root      8111   842  0 01:36 ?        00:00:00 sshd: XXX [priv]
XXX       8113     1  0 01:36 ?        00:00:00 /lib/systemd/systemd --user
root      8114     2  0 01:36 ?        00:00:00 [kworker/2:1]
root      8118     2  0 01:36 ?        00:00:00 [kworker/1:2]
XXX       8120  8113  0 01:36 ?        00:00:00 (sd-pam)
XXX       8301  8111  0 01:36 ?        00:00:00 sshd: XXX@pts/0
XXX       8320  8301  0 01:36 pts/0    00:00:00 -bash
root      8525     2  0 Jul13 ?        00:00:00 [kworker/1:0]
root     10062     2  0 01:38 ?        00:00:00 [kworker/u8:2]
XXX      10374  8320  0 01:38 pts/0    00:00:00 ps -ef
postfix  12994  1091  0 00:51 ?        00:00:00 pickup -l -t unix -u -c
root     16201     2  0 Jul13 ?        00:00:02 [kworker/3:1]
root     16203     2  0 Jul13 ?        00:00:00 [kworker/0:2]
syslog   16231     1  0 Jul12 ?        00:00:12 /usr/sbin/rsyslogd -n
root     20220     1  0 Jul08 ?        00:00:17 /usr/sbin/apache2 -k start
www-data 23553 20220  0 Jul13 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 23554 20220  0 Jul13 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 23555 20220  0 Jul13 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 23556 20220  0 Jul13 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 23557 20220  0 Jul13 ?        00:00:00 /usr/sbin/apache2 -k start
root     28603     2  0 01:27 ?        00:00:00 [kworker/u8:0]
www-data 28655 20220  0 Jul13 ?        00:00:00 /usr/sbin/apache2 -k start
root     29174     2  0 Jul13 ?        00:00:00 [kworker/2:2]
root     29175     2  0 Jul13 ?        00:00:00 [kworker/0:1]
root     31196     2  0 01:05 ?        00:00:00 [kworker/u8:1]
nagios   32513     1  0 Jul09 ?        00:00:02 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

Re: Checks perpetually pending

Posted: Sun Jul 15, 2018 8:35 pm
by linuxnerd
i think i forgot to reload the service.
the service check wasn't present into the objects.dat file.
after repushing my config / reloading, it seems to be running.

Re: Checks perpetually pending

Posted: Mon Jul 16, 2018 2:37 pm
by jomann
So are the checks happening now? Is this issue resolved?

Re: Checks perpetually pending

Posted: Wed Jul 18, 2018 7:13 am
by linuxnerd
i suppose so.

Re: Checks perpetually pending

Posted: Wed Jul 18, 2018 9:39 am
by tmcdonald
Did you have further (related) questions or are we good to lock this up?