Some of the checks are missing perfdata

[Sampath.Basireddy]

Hello There,

I just noticed some of the checks are missing perfdata and also perfdata.log stopped updating since last 3 days.

We are are NagiosXI 5.5.2. No changes were made on the day when log file stopped updating.

Code: Select all

[root@srvnxi1 var]# tail -f perfdata.log
2018-12-15 07:04:01 [10483] [0] *** TIMEOUT: Please check your npcd.cfg
2018-12-15 07:04:01 [10483] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1544875326.perfdata.service-PID-10483 deleted
2018-12-15 07:04:01 [10483] [0] *** Timeout while processing Host: "prd01aumspxy04.ux.corp.local" Service: "HAProxy_-_MSBEN"
2018-12-15 07:04:01 [10483] [0] *** process_perfdata.pl terminated on signal ALRM
2018-12-15 07:04:01 [10527] [0] *** TIMEOUT: Timeout after 40 secs. ***
2018-12-15 07:04:01 [10527] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2018-12-15 07:04:01 [10527] [0] *** TIMEOUT: Please check your npcd.cfg
2018-12-15 07:04:01 [10527] [0] *** TIMEOUT: /var/nagiosramdisk/spool/perfdata//1544875355.perfdata.service-PID-10527 deleted
2018-12-15 07:04:01 [10527] [0] *** Timeout while processing Host: "usfmsidf224" Service: "GigabitEthernet1_0_17_Bandwidth"
2018-12-15 07:04:01 [10527] [0] *** process_perfdata.pl terminated on signal ALRM

[npolovenko]

@Sampath.Basireddy, Your server load could be higher then the npcd load threshold causing it to not process the perfdata on time.
Please follow this article and increase the NPCD timeout time to 40, and double the load_threshold value. https://support.nagios.com/kb/article.php?id=9
Then go ahead and restart the npcd with

service npcd restart

[Sampath.Basireddy]

NCPD Timeout and load_threshold levels are at 40 already.

rrd files seems to be up to date, but not seeing graphs in the console.

[npolovenko]

@Sampath.Basireddy, Could you PM me a fresh copy of your system profile?

[Sampath.Basireddy]

@npolovenko, PM'd you system profile.

[npolovenko]

@Sampath.Basireddy, I'm seeing this error message in the perfdata log:

found extra data on update argument

Have you modified the services that are not returning the data recently? Have you added an extra data source? If that is the case Nagios will fail to update the RRD files properly and you'd need to follow the procedure to add a new datasource to the rrd.

I also noticed that you have 2 nagios NDO message queues. This is not related to graphs but should be fixed.
Please run these commands:

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

[Sampath.Basireddy]

@ npolovenko, No changes were made to the service. Well, this service is on our passive server and the server recently became active.

But otherwise, no changes were made to the service. Only change is it starting getting data for the check and ideally should be showing graph.

I ran all the commands.

[Sampath.Basireddy]

@ npolovenko,

Looks like now the complete graphs are missing after running those commands. I created a support request. Please help fix this ASAP.

[lmiltchev]

Go to Home > Details > Service Status, click on a "problem" service, go to the Advanced tab and show us a screenshot of the page.

What is your server load currently? Run the command below and show the output:

Code: Select all

uptime

Restart ncpd service:

Code: Select all

systemctl restart npcd.service

make sure it started:

Code: Select all

systemctl status npcd.service

then post the following files on the forum:

- /usr/local/nagios/etc/pnp/npcd.cfg
- /usr/local/nagios/etc/pnp/process_perfdata.cfg
- /usr/local/nagios/var/npcd.log
- /usr/local/nagios/var/perfdata.log

[Sampath.Basireddy]

@ lmiltchev, Please go ahead and close this topic. This is resolved.