Nagios 4.1.1 shows wrong status on Web-Interface

[amitw]

Hi guys,
I am currently monitoring a small environment (6 servers, mainly Windows) and i have a strange problem.
every time im manually refreshing the page or the cgi doing it automatically, the Web-Interface doesnt shows consistently the right data.
it can show me that all my servers are up and running, on Status Up and all services are OK
and on the other hand it can show me that all my hosts are on Pending Mode and some services are missing.

attached are print screens with 2 different status Nagios Web-interface shows me in 2 seconds different between the refresh of the page.

what can be the problem?

[rkennedy]

Perhaps you have multiple processes running, what is the full output of ps -ef?

Also, do you notice any errors in your /usr/local/nagios/var/nagios.log file? Please post this for us to look at.

[amitw]

ps -ef output:

Code: Select all

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jun27 ?        00:00:04 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root         2     0  0 Jun27 ?        00:00:00 [kthreadd]
root         3     2  0 Jun27 ?        00:00:00 [ksoftirqd/0]
root         7     2  0 Jun27 ?        00:00:00 [migration/0]
root         8     2  0 Jun27 ?        00:00:00 [rcu_bh]
root         9     2  0 Jun27 ?        00:00:00 [rcuob/0]
root        10     2  0 Jun27 ?        00:00:00 [rcuob/1]
root        11     2  0 Jun27 ?        00:00:03 [rcu_sched]
root        12     2  0 Jun27 ?        00:00:02 [rcuos/0]
root        13     2  0 Jun27 ?        00:00:02 [rcuos/1]
root        14     2  0 Jun27 ?        00:00:00 [watchdog/0]
root        15     2  0 Jun27 ?        00:00:00 [watchdog/1]
root        16     2  0 Jun27 ?        00:00:00 [migration/1]
root        17     2  0 Jun27 ?        00:00:00 [ksoftirqd/1]
root        20     2  0 Jun27 ?        00:00:00 [khelper]
root        21     2  0 Jun27 ?        00:00:00 [kdevtmpfs]
root        22     2  0 Jun27 ?        00:00:00 [netns]
root        23     2  0 Jun27 ?        00:00:00 [perf]
root        24     2  0 Jun27 ?        00:00:00 [writeback]
root        25     2  0 Jun27 ?        00:00:00 [kintegrityd]
root        26     2  0 Jun27 ?        00:00:00 [bioset]
root        27     2  0 Jun27 ?        00:00:00 [kblockd]
root        28     2  0 Jun27 ?        00:00:00 [md]
root        33     2  0 Jun27 ?        00:00:00 [khungtaskd]
root        34     2  0 Jun27 ?        00:00:00 [kswapd0]
root        35     2  0 Jun27 ?        00:00:00 [ksmd]
root        36     2  0 Jun27 ?        00:00:00 [khugepaged]
root        37     2  0 Jun27 ?        00:00:00 [fsnotify_mark]
root        38     2  0 Jun27 ?        00:00:00 [crypto]
root        46     2  0 Jun27 ?        00:00:00 [kthrotld]
root        48     2  0 Jun27 ?        00:00:00 [kmpath_rdacd]
root        50     2  0 Jun27 ?        00:00:00 [kpsmoused]
root        51     2  0 Jun27 ?        00:00:00 [ipv6_addrconf]
root        71     2  0 Jun27 ?        00:00:00 [deferwq]
root       101     2  0 Jun27 ?        00:00:00 [kauditd]
root       280     2  0 Jun27 ?        00:00:00 [mpt_poll_0]
root       281     2  0 Jun27 ?        00:00:00 [ata_sff]
root       282     2  0 Jun27 ?        00:00:00 [events_power_ef]
root       283     2  0 Jun27 ?        00:00:00 [mpt/0]
root       291     2  0 Jun27 ?        00:00:00 [scsi_eh_0]
root       292     2  0 Jun27 ?        00:00:00 [scsi_tmf_0]
root       295     2  0 Jun27 ?        00:00:00 [scsi_eh_1]
root       297     2  0 Jun27 ?        00:00:00 [scsi_tmf_1]
root       298     2  0 Jun27 ?        00:00:00 [scsi_eh_2]
root       299     2  0 Jun27 ?        00:00:00 [scsi_tmf_2]
root       301     2  0 Jun27 ?        00:00:00 [ttm_swap]
root       370     2  0 Jun27 ?        00:00:00 [kdmflush]
root       371     2  0 Jun27 ?        00:00:00 [bioset]
root       381     2  0 Jun27 ?        00:00:00 [kdmflush]
root       382     2  0 Jun27 ?        00:00:00 [bioset]
root       396     2  0 Jun27 ?        00:00:00 [xfsalloc]
root       397     2  0 Jun27 ?        00:00:00 [xfs_mru_cache]
root       398     2  0 Jun27 ?        00:00:00 [xfs-buf/dm-0]
root       399     2  0 Jun27 ?        00:00:00 [xfs-data/dm-0]
root       400     2  0 Jun27 ?        00:00:00 [xfs-conv/dm-0]
root       401     2  0 Jun27 ?        00:00:00 [xfs-cil/dm-0]
root       402     2  0 Jun27 ?        00:00:14 [xfsaild/dm-0]
root       475     1  0 Jun27 ?        00:00:13 /usr/lib/systemd/systemd-journald
root       487     1  0 Jun27 ?        00:00:00 /usr/sbin/lvmetad -f
root       497     1  0 Jun27 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root       548     2  0 Jun27 ?        00:00:00 [xfs-buf/sda1]
root       550     2  0 Jun27 ?        00:00:00 [xfs-data/sda1]
root       552     2  0 Jun27 ?        00:00:00 [xfs-conv/sda1]
root       553     2  0 Jun27 ?        00:00:00 [xfs-cil/sda1]
root       554     2  0 Jun27 ?        00:00:00 [xfsaild/sda1]
root       594     1  0 Jun27 ?        00:00:00 /sbin/auditd -n
root       619     1  0 Jun27 ?        00:00:03 /usr/sbin/rsyslogd -n
root       621     1  0 Jun27 ?        00:00:00 /usr/lib/systemd/systemd-logind
root       622     1  0 Jun27 ?        00:00:03 /usr/sbin/irqbalance --foreground
dbus       623     1  0 Jun27 ?        00:00:01 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       631     1  0 Jun27 ?        00:00:00 /usr/sbin/crond -n
root       634     1  0 Jun27 ?        00:00:00 login -- root
root       685     2  0 Jun27 ?        00:00:00 [kworker/0:1H]
root       743     1  0 Jun27 ?        00:00:46 /usr/sbin/vmtoolsd
root      1006     1  0 Jun27 ?        00:00:11 /usr/bin/python -Es /usr/sbin/tuned -l -P
root      1007     1  0 Jun27 ?        00:00:00 /usr/sbin/sshd -D
root      6527     2  0 Jun28 ?        00:00:00 [kworker/1:2H]
polkitd  10432     1  0 Jun27 ?        00:00:00 /usr/lib/polkit-1/polkitd --no-debug
root     11785     2  0 Jun28 ?        00:00:04 [kworker/0:0]
nagios   12641     1  0 Jun28 ?        00:00:11 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   12643 12641  0 Jun28 ?        00:00:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   12644 12641  0 Jun28 ?        00:00:09 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   12645 12641  0 Jun28 ?        00:00:09 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   12646 12641  0 Jun28 ?        00:00:09 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   12691 12641  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root     15144     2  0 07:16 ?        00:00:00 [kworker/u4:0]
root     16067     2  0 Jun28 ?        00:00:00 [kworker/u4:2]
root     16104     2  0 Jun28 ?        00:00:00 [kworker/0:2H]
root     16983     2  0 07:48 ?        00:00:00 [kworker/0:1]
root     17600     2  0 07:58 ?        00:00:00 [kworker/0:2]
root     17630  1007  0 07:58 ?        00:00:00 sshd: root@pts/0
root     17639 17630  0 07:58 pts/0    00:00:00 -bash
nagios   17663 12646  0 07:58 ?        00:00:00 /usr/local/nagios/libexec/check_nt -H 10.50.112.12 -p 12489 -v UPTIME
nagios   17665 21128  0 07:58 ?        00:00:00 /usr/local/nagios/libexec/check_ping -H 10.50.113.11 -w 100.0,20% -c 500.0,60% -p 5
nagios   17666 21127  0 07:58 ?        00:00:00 /usr/local/nagios/libexec/check_ping -H 10.50.114.11 -w 100.0,20% -c 500.0,60% -p 5
nagios   17667 17665  0 07:58 ?        00:00:00 /usr/bin/ping -n -U -w 10 -c 5 10.50.113.11
nagios   17668 17666  0 07:58 ?        00:00:00 /usr/bin/ping -n -U -w 10 -c 5 10.50.114.11
root     17670 17639  0 07:59 pts/0    00:00:00 ps -ef
root     18805     2  0 Jun28 ?        00:00:00 [kworker/1:0]
root     18826     1  0 Jun28 ?        00:00:01 /usr/sbin/httpd -DFOREGROUND
apache   18827 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   18828 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   18829 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   18830 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   18831 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   18837 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   18841 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
nagios   21123     1  0 Jun28 ?        00:00:08 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   21125 21123  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   21126 21123  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   21127 21123  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   21128 21123  0 Jun28 ?        00:00:02 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   21133 21123  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
apache   21255 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   21818 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root     22857     2  0 Jun28 ?        00:00:14 [kworker/1:2]
apache   23237 18826  0 Jun28 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root     25534     2  0 Jun28 ?        00:00:00 [kworker/1:0H]
root     32730   634  0 Jun28 tty1     00:00:00 -bash

nagios.log output:

Code: Select all

SERVICE NOTIFICATION: ;Memory Usage;CRITICAL;notify-service-by-email;CRITICAL - Socket timeout after 10 seconds
[1467176320] SERVICE NOTIFICATION: ;PING;CRITICAL;notify-service-by-email;PING CRITICAL - Packet loss = 100%
[1467176350] SERVICE NOTIFICATION: ;Uptime;CRITICAL;notify-service-by-email;CRITICAL - Socket timeout after 10 seconds
[1467176354] SERVICE NOTIFICATION: ;CPU Load;CRITICAL;notify-service-by-email;CRITICAL - Socket timeout after 10 seconds
[1467176430] SERVICE NOTIFICATION: ;C:\ Drive Space;CRITICAL;notify-service-by-email;CRITICAL - Socket timeout after 10 seconds

Just some service notification. nothing interesting.
i just dont understand how to Web interface 1 sec shows the right data and after 1 sec shows lack of data.

Thanks for the help

[rkennedy]

It looks like you have multiple processes running, you'll want to kill both of them and then start nagios as you normal would. At this point they are both conflicting with each other.

Code: Select all

nagios   12641     1  0 Jun28 ?        00:00:11 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   12643 12641  0 Jun28 ?        00:00:08 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   12644 12641  0 Jun28 ?        00:00:09 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   12645 12641  0 Jun28 ?        00:00:09 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   12646 12641  0 Jun28 ?        00:00:09 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   12691 12641  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

nagios   21123     1  0 Jun28 ?        00:00:08 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   21125 21123  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   21126 21123  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   21127 21123  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   21128 21123  0 Jun28 ?        00:00:02 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   21133 21123  0 Jun28 ?        00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

[amitw]

Hi,
i have rebooted the server and that did the trick
Thanks

[mcapra]

Is it alright if we lock this thread and mark the issue as resolved?