Nagios Perf Data not working for particular service

[philip.ianni]

Hey all,

Currently having an issue with perf data on a particular host.

The service checks disk space on a variety of hosts and graphs properly, however 1/9 hosts does not seem to like to graph.

The output when I run the check from the command line and via "test command" is identical and looks, at least to me, the same as the other 8 hosts:

Code: Select all

Disk Space/iNode Warning!! -  /dev/mapper/vg_sandboxdb-lv_root (86%) /dev/mapper/vg_sandboxdb_data-lv_opt (84%) /dev/mapper/vg_sandboxdb-lv_root (IUse 22%) /dev/mapper/vg_sandboxdb_data-lv_opt (IUse 1%) | /dev/mapper/vg_sandboxdb-lv_root=86% /dev/mapper/vg_sandboxdb_data-lv_opt=84% /dev/mapper/vg_sandboxdb-lv_root_iNode=22% /dev/mapper/vg_sandboxdb_data-lv_opt_iNode=1%

Despite this, perfdata not appearing in graph

PerfData.JPG

I have checked what I think are the revelant logs related to perfdata but do not see anything out of the ordinary

/var/log/messages
/usr/local/nagiosxi/var/perfdataproc.log


Thanks for your help!

[tmcdonald]

First, just to confirm, is this for Nagios Core or Nagios XI? You posted in the Core section but there are XI paths in your post. I want to move this to the proper section.

Next, assuming this is XI, please run the following as root and post back:

Code: Select all

service npcd status
ls -l /usr/local/nagios/var/spool/xidpe | wc -l
ls -l /usr/local/nagios/var/spool/perfdata | wc -l
find /usr/local/nagios/share/perfdata -name "*HOSTNAME*"

making sure to replace HOSTNAME in that last command with the name of the host having issues.

[philip.ianni]

Seems like perfdata writing to a file that has the same hostname, but no domain suffix. ( I think this is what host was originally called)

[root@vfmsrv107 perfdata]# ll | grep dev-current-db01
drwxrwxr-x 2 nagios nagios 4096 Jul  7  2015 dev-current-db01
drwxrwxr-x 2 nagios nagios 4096 Apr  8 10:26 dev-current-db01.vfmltech.com


[root@vfmsrv107 ~]# ls -l /usr/local/nagios/var/spool/xidpe | wc -l
1
[root@vfmsrv107 ~]# ls -l /usr/local/nagios/var/spool/perfdata | wc -l
3

[tmcdonald]

First, just to confirm, is this for Nagios Core or Nagios XI? You posted in the Core section but there are XI paths in your post. I want to move this to the proper section.

Can you please address this? Your graphs are showing the old PNP4Nagios style which might imply Core, or possibly an older version of XI, but it will help to know for sure what product you are using. It is most likely XI - please let us know which version you are on.

[philip.ianni]

Apologies, this is indeed XI.

[ssax]

I don't think that will work, the perfdata output doesn't conform to our API standards:

Code: Select all

https://nagios-plugins.org/doc/guidelines.html#AEN200

What plugin are you using? Please provide a link if possible.

[philip.ianni]

The output is fine for the other hosts which is weird

Code: Select all

#!/bin/bash

runDir="/usr/lib64/nagios/plugins"
warn="$1"
critical="$2"
iNodesWarn="$3"
iNodesCritical="$4"
count="2"
sedType="p"
diskAll=`df -Plh | grep -v tmpfs | grep -v nas`
lineCount=`echo "$diskAll" | wc -l`
iNodesTmpFile="$runDir/tmp/.check_disk_io_inodes2"
inodes=$(df -lPi | grep -v "tmpfs" | grep -v nas > $iNodesTmpFile)



exitCode="0"
criticalExit="2"
warnExit="1"


if [ $# -lt 4 ]
then
        echo "Command Line Arguments Missing!!!"
        echo "Arg 1: Disk Space Warning Value"
        echo "Arg 2: Disk Space Critical Value"
        echo "Arg 3: Disk Inode Warning Value"
        echo "Arg 4: Disk Inode Critical Value"

        exit 3
fi
while [ $count -le $lineCount ]
do
        diskName=`echo "$diskAll" | sed -n $count$sedType | awk '{print $1}'`
        diskPercent=`echo "$diskAll" | sed -n $count$sedType | awk '{print $5}' | sed 's/%$//'`

        if [ $diskPercent -ge $critical ]
        then
                if [ $exitCode -lt $criticalExit ]
                then
                        exitCode=2
                fi
        elif [ $diskPercent -ge $warn ]
        then
                if [ $exitCode -lt $warnExit ]
                then
                        exitCode=1
                fi
        fi
        msg="$msg $diskName ($diskPercent%)"
        
                 perfData="$perfData$diskName=$diskPercent% "
        let count=count+1
done
count="2"
while [ $count -le $lineCount ]
do
        iNodeDisk=$(cat $iNodesTmpFile | sed -n $count$sedType |awk '{print $1}')
        iNodePercent=$(cat $iNodesTmpFile | sed -n $count$sedType |awk '{print $5}' | sed 's/%$//')

        if [ $iNodePercent -ge $iNodesCritical ]
        then
                if [ $exitCode -lt $criticalExit ]
                then
                        exitCode=2
                fi
        elif [ $iNodePercent -ge $iNodesWarn ]
        then
                if [ $exitCode -lt $warnExit ]
                then
                        exitCode=1
                fi
        fi
        msg="$msg $iNodeDisk (IUse $iNodePercent%)"
        perfData="$perfData${iNodeDisk}_iNode=$iNodePercent% "
        let count=count+1
done


if [ $exitCode -eq $criticalExit ]
then
        echo "Disk Space/iNode Not Ok!!! - $msg | $perfData"
        echo "<pre>$diskAll</pre>"
        echo "<pre>$(cat $iNodesTmpFile)</pre>"
        exit $criticalExit
elif [ $exitCode -eq $warnExit ]
then
        echo "Disk Space/iNode Warning!! - $msg | $perfData"
        echo "<pre>$diskAll</pre>"
        echo "<pre>$(cat $iNodesTmpFile)</pre>"
        exit $warnExit
else
        echo "Disk Space/iNode OK. $msg | $perfData"
        echo "<pre>$diskAll</pre>"
        echo "<pre>$(cat $iNodesTmpFile)</pre>"
        exit $exitCode
fi

[ssax]

Please go to one of the working services, click on the advanced tab, and paste the perfdata output for one of them as well just so we can compare.

You could always try deleting your XML and RRD file for this particular service in /usr/local/nagios/share/perfdata/HOSTNAME so that it can be rebuilt, maybe something is wrong with the RRD.

[philip.ianni]

That is correct!

I just blew away the directory and Nagios started graphing it properly

Thank you

[rkennedy]

Great to hear!

Marking this as resolved, and closing it out.