Custom SNMP check

[ssax]

Did you originally have the service set as some other OID?

You might try deleting the SERVICE.rrd and SERVICE.xml files for your host/service under /usr/local/nagios/share/perfdata/HOST/ so that they can get rebuilt.

^ Make sure to change SERVICE and HOST to your service name and hostname for the problem service.

[angus]

That did not seem to work. There are other services with the same issue for the host. Is there anything else i can try?

[abrist]

That did not seem to work. There are other services with the same issue for the host. Is there anything else i can try?

Removing the RRDs and XMLs should force the services to rebuild the performance data RRDs on the next check. If you are having issues with other services, you may have an issue with the npcd daemon or perfdata processing. What is the output of:

Code: Select all

ls /usr/local/nagios/var/spool/xidpe/ | wc -l
ls /usr/local/nagios/var/spool/checkresults/ | wc -l
ls /usr/local/nagios/var/spool/perfdata/ | wc -l
ls -la /usr/local/nagios/var/*-perfdata
tail -25 /usr/local/nagios/var/npcd.log
tail -25 /usr/local/nagios/var/perfdata.log
service npcd status
umask

[angus]

ls /usr/local/nagios/var/spool/xidpe/ | wc -l
0
ls /usr/local/nagios/var/spool/checkresults/ | wc -l
0

ls /usr/local/nagios/var/spool/perfdata/ | wc -l
2

ls -la /usr/local/nagios/var/*-perfdata
-rw-r--r-- 1 nagios nagios 286 Jul 2 09:35 /usr/local/nagios/var/host-perfdata
-rw-r--r-- 1 nagios nagios 3986 Jul 2 09:35 /usr/local/nagios/var/service-perfdata


tail -25 /usr/local/nagios/var/npcd.log
[05-01-2015 10:20:08] NPCD: npcd Daemon (0.4.14) started with PID=1426
[05-01-2015 10:20:08] NPCD: Please have a look at 'npcd -V' to get license information
[05-01-2015 10:20:08] NPCD: HINT: load_threshold is enabled - ('10.000000')
[05-01-2015 10:23:18] NPCD: Caught Termination Signal - Hasta la vista... baby
[05-01-2015 10:23:40] NPCD: npcd Daemon (0.4.14) started with PID=1420
[05-01-2015 10:23:40] NPCD: Please have a look at 'npcd -V' to get license information
[05-01-2015 10:23:40] NPCD: HINT: load_threshold is enabled - ('10.000000')
[05-01-2015 13:20:15] NPCD: Caught Termination Signal - Hasta la vista... baby
[05-01-2015 13:20:35] NPCD: npcd Daemon (0.4.14) started with PID=1423
[05-01-2015 13:20:35] NPCD: Please have a look at 'npcd -V' to get license information
[05-01-2015 13:20:35] NPCD: HINT: load_threshold is enabled - ('10.000000')
[05-08-2015 08:50:17] NPCD: Caught Termination Signal - Hasta la vista... baby
[05-08-2015 08:54:54] NPCD: npcd Daemon (0.4.14) started with PID=1532
[05-08-2015 08:54:54] NPCD: Please have a look at 'npcd -V' to get license information
[05-08-2015 08:54:54] NPCD: HINT: load_threshold is enabled - ('10.000000')
[05-20-2015 16:12:34] NPCD: Caught Termination Signal - Hasta la vista... baby
[05-21-2015 09:28:42] NPCD: npcd Daemon (0.4.14) started with PID=1527
[05-21-2015 09:28:42] NPCD: Please have a look at 'npcd -V' to get license information
[05-21-2015 09:28:42] NPCD: HINT: load_threshold is enabled - ('10.000000')
[06-17-2015 15:06:47] NPCD: Caught Termination Signal - Hasta la vista... baby
[06-17-2015 15:07:19] NPCD: npcd Daemon (0.4.14) started with PID=1524
[06-17-2015 15:07:19] NPCD: Please have a look at 'npcd -V' to get license information
[06-17-2015 15:07:19] NPCD: HINT: load_threshold is enabled - ('10.000000')
[06-30-2015 14:40:29] NPCD: ERROR: Executed command exits with return code '7'
[06-30-2015 14:40:29] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1435689608.perfdata.service'


tail -25 /usr/local/nagios/var/perfdata.log
2015-06-30 14:40:29 [31969] [0] *** TIMEOUT: Timeout after 5 secs. ***
2015-06-30 14:40:29 [31969] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2015-06-30 14:40:29 [31969] [0] *** TIMEOUT: Please check your npcd.cfg
2015-06-30 14:40:29 [31969] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1435689608.perfdata.service-PID-31969 deleted
2015-06-30 14:40:29 [31969] [0] *** Timeout while processing Host: "Usn003" Service: "Disk_Usage_usn002_backup_11"
2015-06-30 14:40:29 [31969] [0] *** process_perfdata.pl terminated on signal ALRM


service npcd status
NPCD running (pid 1524).


umask
0022

[ssax]

Looks like you are hitting a timeout.

Edit this file:

Code: Select all

/usr/local/nagios/etc/pnp/process_perfdata.cfg

Change:

Code: Select all

TIMEOUT = 5

To:

Code: Select all

TIMEOUT = 20

Now restart npcd:

Code: Select all

service npcd restart

[abrist]

Are there only a few similar services defected (like all ping/host graphs), or does it seem a bit more random?

[angus]

I will make those changes after lunch. There are only two or three services giving values that a misrepresented on the graph. They are all very similar to the one we are dealing with. Each service is a single SNMP check to an OID on the same server as the one we have been referencing. The results for these services are numeric followed by a "c" just like the service we are working on. As an example the out put would be 37c.

[angus]

The change did not resolve the counter mismatch issues.

Thanks.

[ssax]

Can you PM one of us your .rrd and .xml files for the service that is having the issue (under /usr/local/nagios/share/perfdata/HOST/), we want to dig into them and check out the values.

[angus]

I sent that over.