perfdata is not taking new data

[pmithil7]

I'm facing an issue where data is not seen in the graphs of pnp4nagios after a certain day. On seeing the perfdata contents it only has data till that day. It has been 2 weeks till then and it has not written new data in the perfdata folder. Apart from this the 'service-perfdata' file is increasing everyday and has reached to 57G currently. This would fill up my /var folder folder diskspace soon.

I need help in getting perfdata being written and is there a way to roll off data in service-perfdata file?

Any help is much appreciated.

Thanks,
Mithil

[tmcdonald]

There are a few things to look into, so we compiled them into this KB article:

https://support.nagios.com/kb/article.php?id=9

Give it a read and if none of that fixes it, let us know so we can troubleshoot further.

[pmithil7]

Thank you for your feedback.

Hi,

In my directory /usr/local/pnp4nagios/var/ there are three files 'perfdata', 'host-perfdata' & 'service-perfdata'. I had an issue where my var folder got completely full and nagios crashed as it had no space to store or process files. I increased the size of it and then it started to work fine. Now from that particular day, i don't see any data in the graphs.

Digging deep into these files i can see that 'perfdata' doesn't have any data after that day when it crashed and all the data in 'host-perfdata' & 'service-perfdata' is only after that particular day and it is growing in size.

I found this via issuing commands: tail --lines=10 service-perfdata & head --lines=10 service-perfdata and seeing the UNIX time in both outputs to be from the day it crashed to my current day. This file is increasing in an alarming manner and is already 60G. I need to stop it from growing.

I sense perfdata isn't receiving the data currently as it doesn't have any data from the day nagios crashed.

My perfdata.log is full of the below logs:
File truncated2016-11-22 08:46:04 [29619] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd: found extra data on update argument: 1.587:0.198
2016-11-22 08:46:04 [29619] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd 1479833153:2.505:0:5.384:1.747
2016-11-22 08:46:04 [29619] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd: found extra data on update argument: 5.384:1.747
2016-11-22 08:46:04 [29619] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd 1479833153:1.192:0:1.229:1.135
2016-11-22 08:46:04 [29619] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd: found extra data on update argument: 1.229:1.135
2016-11-22 08:46:04 [29619] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd 1479833153:2.070:0:2.134:1.989
2016-11-22 08:46:04 [29619] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd: found extra data on update argument: 2.134:1.989
2016-11-22 08:46:04 [29619] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd 1479833153:2.435:0:3.172:2.226
2016-11-22 08:46:04 [29619] [0] RRDs::update ERROR /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd: found extra data on update argument: 3.172:2.226
2016-11-22 08:46:04 [29619] [0] RRDs::update /usr/local/pnp4nagios/var/perfdata/xxxx/_HOST_.rrd 1479833153:2.283:0:2.313:2.237

where xxxx is all my different hostnames.

I can share further details if required.

Thanks,
Mithil

[tgriep]

We would need to see the following files to help out.
Can you post them?

Code: Select all

process_perfdata.cfg
npcd.cfg
nagios.cfg

Also, run the following command and post the output.

Code: Select all

ps -ef --cols=300

Since the host-perfdata and the service-perfdata are so large, the system may not ever have the time to process them, try clearing them out and restart the following processed to see if that starts the perfdata.

Code: Select all

service npcd restart
service nagios restart

[pmithil7]

Attached are the configuration files. I will attached the output of 'ps -ef --cols=300' in the next comment.

Should i clear up all the data in host-perfdata & service-perfdata or wait for any configuration issue feedback from you all ?

Thank you for the feedback.

[tgriep]

Yes, you should clear out the host and service perfdata files and see if that fixes the issue.

[pmithil7]

Attaching output for 'ps -ef --cols=300'.

I will remove the contents of host-perfdata & service-perfdata and restart 'npcd' and 'nagios' to see if it works.

Thank you.

-Mithil

[pmithil7]

Is there an easy way to cleanup all the data in the huge 61G 'service-perfdata' file? The data seems to be huge.

Thanks,
Mithil

[tgriep]

Login as root to the server and run the following to clear out all of the data in the service-perfdata file. This will clear it all out but leave the file intact.

Code: Select all

> /usr/local/pnp4nagios/var/service-perfdata

[ssax]

It looks like someone manually upgraded pnp4nagios, you will need to revert your install:

Download the same version of XI that you have to /tmp:

https://assets.nagios.com/downloads/nag ... rsions.php

Then run these commands:
- Make sure to change YOURFILENAME

Code: Select all

cd /tmp
tar zxf YOURFILENAME.tar.gz
cd /tmp/nagiosxi/subcomponents/pnp
./install

Then test it out and let us know the results.


Thank you