perfdata is not taking new data

[pmithil7]

The newest log entries in the perfdata.log file are still from the day the crash happened and has not updated since the last 3 weeks. They are as below:

2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/swpoeb2f4-sjc-01/_HOST_.rrd: found extra data on update argument: 4.239:2.111
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/vi-sjc-06/_HOST_.rrd 1479840713:2.326:0:2.456:2.279
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/vi-sjc-06/_HOST_.rrd: found extra data on update argument: 2.456:2.279
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/dc-sjc-01/_HOST_.rrd 1479840713:2.293:0:2.350:2.256
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/dc-sjc-01/_HOST_.rrd: found extra data on update argument: 2.350:2.256
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/swconsole03/_HOST_.rrd 1479840713:2.696:0:5.059:2.054
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/swconsole03/_HOST_.rrd: found extra data on update argument: 5.059:2.054
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/vi-sjc-03/_HOST_.rrd 1479840713:2.695:0:4.186:2.232
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/vi-sjc-03/_HOST_.rrd: found extra data on update argument: 4.186:2.232
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/swb2f1-sjc-01/_HOST_.rrd 1479840713:2.399:0:2.439:2.359
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/swb2f1-sjc-01/_HOST_.rrd: found extra data on update argument: 2.439:2.359
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/swconsole01/_HOST_.rrd 1479840713:3.080:0:4.540:2.675
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/swconsole01/_HOST_.rrd: found extra data on update argument: 4.540:2.675
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/swb2f2-sjc-01/_HOST_.rrd 1479840713:2.520:0:2.776:2.376
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/swb2f2-sjc-01/_HOST_.rrd: found extra data on update argument: 2.776:2.376
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/vi-sjc-10/_HOST_.rrd 1479840715:3.240:0:3.961:2.896
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/vi-sjc-10/_HOST_.rrd: found extra data on update argument: 3.961:2.896
2016-11-22 10:51:58 [1040] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/vmnode-maa-09/_HOST_.rrd 1479840715:252.032:0:252.377:251.873
2016-11-22 10:51:58 [1040] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/vmnode-maa-09/_HOST_.rrd: found extra data on update argument: 252.377:251.873

Attaching the files that you requested.

Thanks,
Mithil

[tgriep]

Something doesn't match up with the nagios.cfg file that you posted earlier.
Some of the entries do match up with where you said the commands.cfg file resides.

I looked at the PNP4NAGIOS documentation and it looks like you are missing the following commands from your commands.cfg file which could be the cause of the issue.

Add the following to your commands.cfg file and restart the nagios processes.

Code: Select all

define command{
       command_name    process-service-perfdata-file
       command_line    /bin/mv /usr/local/pnp4nagios/var/service-perfdata /usr/local/pnp4nagios/var/spool/service-perfdata.$TIMET$
}

define command{
       command_name    process-host-perfdata-file
       command_line    /bin/mv /usr/local/pnp4nagios/var/host-perfdata /usr/local/pnp4nagios/var/spool/host-perfdata.$TIMET$
}

Also, can you run the following on the server and post the output?

Code: Select all

find / -name nagios.cfg
ps -ef --cols=300

[pmithil7]

I have added the lines to the commands.cfg file.

Ouptut for ps -ef --cols=300:

[root@nagios-eqx-01 ~]# ps -ef --cols=300
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 2016 ? 00:00:01 /sbin/init
root 2 0 0 2016 ? 00:00:00 [kthreadd]
root 3 2 0 2016 ? 00:19:05 [migration/0]
root 4 2 0 2016 ? 00:00:17 [ksoftirqd/0]
root 5 2 0 2016 ? 00:00:00 [stopper/0]
root 6 2 0 2016 ? 00:00:06 [watchdog/0]
root 7 2 0 2016 ? 00:18:26 [migration/1]
root 8 2 0 2016 ? 00:00:00 [stopper/1]
root 9 2 0 2016 ? 00:00:15 [ksoftirqd/1]
root 10 2 0 2016 ? 00:00:06 [watchdog/1]
root 11 2 0 2016 ? 00:18:19 [migration/2]
root 12 2 0 2016 ? 00:00:00 [stopper/2]
root 13 2 0 2016 ? 00:00:14 [ksoftirqd/2]
root 14 2 0 2016 ? 00:00:09 [watchdog/2]
root 15 2 0 2016 ? 00:18:25 [migration/3]
root 16 2 0 2016 ? 00:00:00 [stopper/3]
root 17 2 0 2016 ? 00:00:14 [ksoftirqd/3]
root 18 2 0 2016 ? 00:00:36 [watchdog/3]
root 19 2 0 2016 ? 00:02:48 [events/0]
root 20 2 0 2016 ? 00:02:56 [events/1]
root 21 2 0 2016 ? 00:02:44 [events/2]
root 22 2 0 2016 ? 00:04:27 [events/3]
root 23 2 0 2016 ? 00:00:00 [cgroup]
root 24 2 0 2016 ? 00:00:00 [khelper]
root 25 2 0 2016 ? 00:00:00 [netns]
root 26 2 0 2016 ? 00:00:00 [async/mgr]
root 27 2 0 2016 ? 00:00:00 [pm]
root 28 2 0 2016 ? 00:00:09 [sync_supers]
root 29 2 0 2016 ? 00:00:10 [bdi-default]
root 30 2 0 2016 ? 00:00:00 [kintegrityd/0]
root 31 2 0 2016 ? 00:00:00 [kintegrityd/1]
root 32 2 0 2016 ? 00:00:00 [kintegrityd/2]
root 33 2 0 2016 ? 00:00:00 [kintegrityd/3]
root 34 2 0 2016 ? 00:45:44 [kblockd/0]
root 35 2 0 2016 ? 00:46:28 [kblockd/1]
root 36 2 0 2016 ? 00:46:01 [kblockd/2]
root 37 2 0 2016 ? 00:46:40 [kblockd/3]
root 38 2 0 2016 ? 00:00:00 [kacpid]
root 39 2 0 2016 ? 00:00:00 [kacpi_notify]
root 40 2 0 2016 ? 00:00:00 [kacpi_hotplug]
root 41 2 0 2016 ? 00:00:00 [ata_aux]
root 42 2 0 2016 ? 00:00:00 [ata_sff/0]
root 43 2 0 2016 ? 00:00:00 [ata_sff/1]
root 44 2 0 2016 ? 00:00:00 [ata_sff/2]
root 45 2 0 2016 ? 00:00:00 [ata_sff/3]
root 46 2 0 2016 ? 00:00:00 [ksuspend_usbd]
root 47 2 0 2016 ? 00:00:00 [khubd]
root 48 2 0 2016 ? 00:00:00 [kseriod]
root 49 2 0 2016 ? 00:00:00 [md/0]
root 50 2 0 2016 ? 00:00:00 [md/1]
root 51 2 0 2016 ? 00:00:00 [md/2]
root 52 2 0 2016 ? 00:00:00 [md/3]
root 53 2 0 2016 ? 00:00:00 [md_misc/0]
root 54 2 0 2016 ? 00:00:00 [md_misc/1]
root 55 2 0 2016 ? 00:00:00 [md_misc/2]
root 56 2 0 2016 ? 00:00:00 [md_misc/3]
root 57 2 0 2016 ? 00:00:00 [linkwatch]
root 59 2 0 2016 ? 00:00:01 [khungtaskd]
root 60 2 0 2016 ? 00:00:00 [kswapd0]
root 61 2 0 2016 ? 00:00:00 [ksmd]
root 62 2 0 2016 ? 00:00:18 [khugepaged]
root 63 2 0 2016 ? 00:00:00 [aio/0]
root 64 2 0 2016 ? 00:00:00 [aio/1]
root 65 2 0 2016 ? 00:00:00 [aio/2]
root 66 2 0 2016 ? 00:00:00 [aio/3]
root 67 2 0 2016 ? 00:00:00 [crypto/0]
root 68 2 0 2016 ? 00:00:00 [crypto/1]
root 69 2 0 2016 ? 00:00:00 [crypto/2]
root 70 2 0 2016 ? 00:00:00 [crypto/3]
root 78 2 0 2016 ? 00:00:00 [kthrotld/0]
root 79 2 0 2016 ? 00:00:00 [kthrotld/1]
root 80 2 0 2016 ? 00:00:00 [kthrotld/2]
root 81 2 0 2016 ? 00:00:00 [kthrotld/3]
root 82 2 0 2016 ? 00:00:00 [pciehpd]
root 84 2 0 2016 ? 00:00:00 [kpsmoused]
root 85 2 0 2016 ? 00:00:00 [usbhid_resumer]
root 86 2 0 2016 ? 00:00:00 [deferwq]
root 118 2 0 2016 ? 00:00:00 [kdmremove]
root 119 2 0 2016 ? 00:00:00 [kstriped]
root 204 2 0 2016 ? 00:00:00 [scsi_eh_0]
root 205 2 0 2016 ? 00:00:00 [scsi_eh_1]
root 271 2 0 2016 ? 00:00:00 [scsi_eh_2]
root 272 2 0 2016 ? 00:00:00 [vmw_pvscsi_wq_2]
root 368 2 0 2016 ? 00:00:00 [kdmflush]
root 370 2 0 2016 ? 00:00:00 [kdmflush]
root 389 2 0 2016 ? 00:03:38 [flush-253:0]
root 390 2 0 2016 ? 00:02:17 [jbd2/dm-0-8]
root 391 2 0 2016 ? 00:00:00 [ext4-dio-unwrit]
apache 428 2039 0 Jan15 ? 00:05:27 /usr/sbin/httpd
apache 433 2039 0 Jan15 ? 00:05:07 /usr/sbin/httpd
apache 434 2039 0 Jan15 ? 00:05:36 /usr/sbin/httpd
root 476 1 0 2016 ? 00:00:00 /sbin/udevd -d
root 691 2 0 2016 ? 00:00:33 [vmmemctl]
root 836 2 0 2016 ? 00:00:00 [kdmflush]
root 854 2 0 2016 ? 00:32:05 [flush-253:2]
root 878 2 0 2016 ? 00:00:00 [jbd2/sda1-8]
root 879 2 0 2016 ? 00:00:00 [ext4-dio-unwrit]
root 880 2 0 2016 ? 00:25:51 [jbd2/dm-2-8]
root 881 2 0 2016 ? 00:00:00 [ext4-dio-unwrit]
root 1000 2 0 2016 ? 00:00:00 [kauditd]
root 1004 1 0 2016 ? 00:00:00 /bin/sh /command/svscanboot
root 1015 1004 0 2016 ? 00:00:30 svscan /service
root 1016 1004 0 2016 ? 00:00:00 readproctitle service errors: ..............................................................................................................................................................................................................................
root 1393 1 0 2016 ? 00:41:57 /usr/sbin/vmtoolsd
root 1518 1 0 2016 ? 00:00:00 /sbin/portreserve
nslcd 1526 1 0 2016 ? 00:00:00 /usr/sbin/nslcd
root 1540 1 0 2016 ? 00:07:43 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
root 1571 1 0 2016 ? 00:10:49 irqbalance --pid=/var/run/irqbalance.pid
rpc 1587 1 0 2016 ? 00:00:07 rpcbind
rpcuser 1607 1 0 2016 ? 00:00:00 rpc.statd
apache 1675 2039 0 Jan16 ? 00:02:04 /usr/sbin/httpd
apache 1704 2039 0 Jan16 ? 00:02:20 /usr/sbin/httpd
dbus 1720 1 0 2016 ? 00:00:00 dbus-daemon --system
root 1763 1 0 2016 ? 00:00:12 /usr/sbin/ypbind
root 1804 1 0 2016 ? 00:00:00 /usr/sbin/acpid
nagios 1822 1 6 09:51 ? 00:03:38 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 1824 1822 0 09:51 ? 00:00:07 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
root 1825 1 0 2016 ? 00:01:28 automount --pid-file /var/run/autofs.pid
nagios 1828 1822 0 09:51 ? 00:00:07 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 1829 1822 0 09:51 ? 00:00:07 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 1831 1822 0 09:51 ? 00:00:07 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 1832 1822 0 09:51 ? 00:00:07 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 1836 1822 0 09:51 ? 00:00:07 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nscd 1845 1 0 2016 ? 00:33:38 /usr/sbin/nscd
nagios 1853 1822 0 09:51 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 1854 1 0 2016 ? 00:00:00 /usr/sbin/mcelog --daemon
root 1873 1 0 2016 ? 00:17:27 /usr/sbin/snmptrapd -Lsd -p /var/run/snmptrapd.pid
root 1889 1 0 2016 ? 00:00:00 /usr/sbin/sshd
root 1898 1 0 2016 ? 00:00:11 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp 1907 1 0 2016 ? 00:00:32 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
nobody 1916 1 0 2016 ? 00:15:16 /usr/sbin/gmond
root 1997 1 0 2016 ? 00:00:28 /usr/libexec/postfix/master
postfix 2006 1997 0 2016 ? 00:00:38 qmgr -l -t fifo -u
root 2022 1 0 2016 ? 00:00:00 /usr/sbin/abrtd
root 2030 1 0 2016 ? 00:00:50 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages
root 2039 1 0 2016 ? 00:01:21 /usr/sbin/httpd
root 2049 1 0 2016 ? 00:00:25 crond
root 2062 1 0 2016 ? 00:00:00 /usr/sbin/atd
nagios 2099 1 0 2016 ? 00:23:13 /usr/bin/python /usr/bin/mkeventd --syslog --syslog-fd 3 -C /etc/check_mk --statedir /var/lib/mkeventd --logdir /var/lib/mkeventd -P /var/log/nagios/rw/mkeventd/events -E /var/log/nagios/rw/mkeventd/eventsocket -S /var/log/nagios/rw/mkeventd/status --p
root 2121 1 0 2016 ? 00:02:06 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent
root 2146 1 0 2016 ? 00:01:20 /opt/puppetlabs/puppet/bin/pxp-agent
root 2162 1 0 2016 ? 00:00:01 /usr/sbin/certmonger -S -p /var/run/certmonger.pid
nagios 2236 1 0 2016 ? 00:00:18 /usr/local/pnp4nagios/bin/npcd -d -f /usr/local/pnp4nagios/etc/npcd.cfg
root 2244 1 0 2016 tty1 00:00:00 /sbin/mingetty /dev/tty1
root 2246 1 0 2016 tty2 00:00:00 /sbin/mingetty /dev/tty2
root 2248 1 0 2016 tty3 00:00:00 /sbin/mingetty /dev/tty3
root 2250 1 0 2016 tty4 00:00:00 /sbin/mingetty /dev/tty4
root 2252 1 0 2016 tty5 00:00:00 /sbin/mingetty /dev/tty5
root 2254 476 0 2016 ? 00:00:00 /sbin/udevd -d
root 2255 476 0 2016 ? 00:00:00 /sbin/udevd -d
root 2256 1 0 2016 tty6 00:00:00 /sbin/mingetty /dev/tty6
root 2377 2 0 2016 ? 00:00:00 [rpciod/0]
root 2378 2 0 2016 ? 00:00:00 [rpciod/1]
root 2379 2 0 2016 ? 00:00:00 [rpciod/2]
root 2380 2 0 2016 ? 00:00:00 [rpciod/3]
root 2421 2 0 2016 ? 00:00:00 [kslowd000]
root 2422 2 0 2016 ? 00:00:00 [kslowd001]
root 2423 2 0 2016 ? 00:00:00 [nfsiod]
root 2424 2 0 2016 ? 00:00:00 [lockd]
apache 6717 2039 0 Jan15 ? 00:05:08 /usr/sbin/httpd
apache 6718 2039 0 Jan15 ? 00:05:31 /usr/sbin/httpd
apache 6719 2039 0 Jan15 ? 00:05:12 /usr/sbin/httpd
apache 6720 2039 0 Jan15 ? 00:05:06 /usr/sbin/httpd
apache 6722 2039 0 Jan15 ? 00:05:27 /usr/sbin/httpd
apache 6723 2039 0 Jan15 ? 00:05:13 /usr/sbin/httpd
apache 6725 2039 0 Jan15 ? 00:05:36 /usr/sbin/httpd
apache 6726 2039 0 Jan15 ? 00:05:30 /usr/sbin/httpd
root 6738 1 0 Jan15 ? 00:02:53 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived --pid=/var/run/puppetlabs/mcollectived.pid --config=/etc/puppetlabs/mcollective/server.cfg --daemonize
root 13810 1889 0 09:48 ? 00:00:00 sshd: root@pts/0
root 13844 13810 0 09:48 pts/0 00:00:00 -bash
postfix 14111 1997 0 10:27 ? 00:00:00 pickup -l -t fifo -u
nagios 26186 1836 0 10:43 ? 00:00:00 python /var/lib/check_mk/precompiled/login-eqx-02
nagios 26224 1829 0 10:43 ? 00:00:00 python /var/lib/check_mk/precompiled/login-eqx-03
nagios 26768 1832 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_ssh 10.16.127.56
nagios 26857 1836 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swlab01-sjc-01.force10networks.com
nagios 26943 1836 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/login-eqx-09
nagios 27102 1831 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_icmp -n 6 -t 20 -w 1500.00 80.00 -c 3000.00 100.00 10.16.127.171
nagios 27118 1828 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swoob-dfw-01
nagios 27204 1831 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_icmp -n 6 -t 20 -w 1500.00 80.00 -c 3000.00 100.00 10.16.127.110
nagios 27392 1832 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swlabcore1-dfw-01
nagios 27919 1824 1 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swpoeb2f3-sjc-01
nagios 27982 1832 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/login-eqx-10
nagios 27989 1836 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swdist1-sjc-03
nagios 28037 1836 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_ssh 10.16.127.205
nagios 28154 1836 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swidf-dfw-01
nagios 28443 1824 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swcore2-dfw-01
nagios 28444 1828 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_icmp -n 6 -t 20 -w 1500.00 80.00 -c 3000.00 100.00 10.16.3.48
nagios 28560 1832 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swperim2-eqx-01
nagios 28932 1836 1 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swb2f3-sjc-01
nagios 29269 1828 0 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/swdist-dfw-01
nagios 29577 1832 1 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/cdu-b2lab2-0412
nagios 29916 1831 1 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/cdu-b1lab3-0305
nagios 30151 1831 3 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/cdu-b2lab2-1615
nagios 30208 28154 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 192.168.3.11 .1.3.6.1.2.1.2.2.1.8
nagios 30308 27392 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 192.168.3.252 .1.3.6.1.2.1.2.2.1.13
nagios 30326 1829 4 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/itmgmt-maa-02
nagios 30329 29269 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 192.168.3.100 .1.3.6.1.2.1.31.1.1.1.15
nagios 30338 27989 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 10.11.3.91 .1.3.6.1.2.1.2.2.1.19
nagios 30349 1832 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_icmp -w 400.00 80.00 -c 500.00 100.00 10.11.104.251
nagios 30360 1824 3 10:44 ? 00:00:00 python /var/lib/check_mk/precompiled/odlcont-maa-01
nagios 30361 1828 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_icmp -n 6 -t 20 -w 1500.00 80.00 -c 3000.00 100.00 10.16.127.56
nagios 30372 1836 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_icmp -n 6 -t 20 -w 1500.00 80.00 -c 3000.00 100.00 10.16.126.32
nagios 30373 1831 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_icmp -n 6 -t 20 -w 1500.00 80.00 -c 3000.00 100.00 10.16.127.39
nagios 30374 1829 0 10:44 ? 00:00:00 /usr/local/nagios/libexec/check_icmp -n 6 -t 20 -w 1500.00 80.00 -c 3000.00 100.00 10.16.126.3
nagios 30377 27118 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 192.168.126.101 .1.3.6.1.2.1.2.2.1.6
nagios 30378 28932 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 10.11.3.62 .1.3.6.1.2.1.31.1.1.1.13
nagios 30379 30151 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 10.11.149.251 .1.3.6.1.2.1.2.2.1.11
nagios 30380 28443 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 192.168.3.253 .1.3.6.1.2.1.31.1.1.1.7
nagios 30384 26857 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 10.11.3.67 .1.3.6.1.2.1.31.1.1.1.13
nagios 30390 29916 0 10:44 ? 00:00:00 snmpbulkwalk -v2c -c -m -M -Cc -OQ -OU -On -Ot 10.11.194.251 .1.3.6.1.2.1.2.2.1.20
root 30396 13844 0 10:44 pts/0 00:00:00 ps -ef --cols=300

Output for find -name nagios.cfg:

[root@nagios-eqx-01 ~]# find -name nagios.cfg
./nagios/nagios-4.0.8/sample-config/nagios.cfg
./nagios/nagios-4.0.8/t-tap/smallconfig/nagios.cfg
./nagios/nagios-4.0.8/t/etc/nagios.cfg

[pmithil7]

Sorry for the late reply to this post as i was on a break. Meanwhile I changed the 'process_performance_data=0' from 1 so that the data doesn't accumulate again and break the server's memory size again while i'm gone. I have turned it back to 1 but i still don't see graphs on the GUI or new data in the perfdata folder.

[tgriep]

Are you still seeing the following errors in the perfdata log file?
RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd: found extra data on update argument: 3.172:2.226

If so, I would delete all of the rrd and xml files for all of the host and services and see if they get recreated and the graphs start to populate.

[pmithil7]

Yes the perfdata.log file still has the same errors and nothing has changed. The perfdata has about 900 hosts, how do i go and delete all the rrd and xml files for all of them?

[tgriep]

Warning, this will delete all of the performance graph / data for every host and service on your server.
Make sure this is what you want to do.

First change to the folder where all of the performance data is stored by running the following as root on the Nagios server.

Code: Select all

cd /usr/local/pnp4nagios/var/perfdata/

Then run the next 2 commands to delete the xml and rrd files

Code: Select all

find . -name "*.xml" -type f|xargs rm -f
find . -name "*.rrd" -type f|xargs rm -f

Wait for about 15 to 20 minutes for the graphs to regenerate and if there are no more errors in the perfdata.log file, then check the GUI to see if they are populated with data.

[pmithil7]

Yes. This deleted all the data and also the graphs for every host and service. Now i'm not able to see any graph in the Nagios. No changes have been seen in the 'perfdata' directory but the size of 'host-perfdata' & 'service-perfdata' keeps on growing and data is being continuously written there. Also 'spool' directory is still empty and perfdata.log file still has the same old contents of errors. As the .xml & .rrd files aren't being created in perfdata, graphs have disappeared.

[pmithil7]

Any help on how to solve this? I'm not sure why data is not being spooled or generated in the perfdata directory.

[tgriep]

Is this a Nagios XI server or a Nagios Core system?
What version of Nagios are you running on this server?

Can you post these files so I can review them?

Code: Select all

/usr/local/nagios/etc/nagios.cfg
/usr/local/nagios/etc/commands.cfg
/usr/local/pnp4nagios/etc/npcd.cfg
/usr/local/pnp4nagios/etc/process_perfdata.cfg

Also , post and errors in the following log files.

Code: Select all

/usr/local/pnp4nagios/var/perfdata.log
/usr/local/pnp4nagios/var/npcd.log

You also should take a look at the PNP4Nagios troubleshooting guide at this link for more help on your issue.
https://docs.pnp4nagios.org/pnp-0.6/verify