No output to perfdata.log of Total Processes

[draakje]

Hi all,

I am pretty new on nagios and I need some help chasing a problem.

I have just setup a freshly installed Ubuntu server LTS 12.04 And installed Nagios 3.4.1 with nagiosgraph 1.4.4.
Everything seems to be running ok. aka: I have my localhost monitored everything is green (Current Load, current users, http, ping root partition, ssh, swap and total processes) And on all checks data is returned on the web.

Now I would like some to be logged in a graph. This works fine for all services except for the Total Processes.

As I do not see any data appearing in the perfdata.log file I suspect there is a misconfig in nagios.
I do not have debugging enable on nagios. If I should enable this at which level should I do this?s

When I check the graph I only have 1 error : "No Data available: host=localhost service=Total Processes db="

Any idea's?

Kind regards,

Ronald

[jsmurphy]

I think the heart of the problem may be that not all plugins return performance data. If you were to run a plugin on the linux command line that has perf data you will usually see something to the affect of:
Disk OK | /=10644MB;29876;29866;0;29886 /dev/shm=0MB;1953;1943;0;1963, etc, etc

The pipe symbol (the one that looks like | ) designates that everything after it is performance information. Not every Nagios plugin has performance information available and for some checks there is a switch to enable or disable it.

Try it out yourself, run one of the commands that has perf data working and then run the one that doesn't. I am 80% sure that the check_procs plugin doesn't support perfdata.

[draakje]

Hi,

Thanks for your reply.

I believe I do get perfdata returned from the plugin

This is the output when I run the command manually:
PROCS OK: 48 processes with STATE = RSZDT

This is an output of the load check (which is generated into a graph)
OK - load average: 0.02, 0.04, 0.05|load1=0.020;5.000;10.000;0; load5=0.040;4.000;6.000;0; load15=0.050;3.000;4.000;0;

And here an output of the HTTP check (which is also generated into a graph)
HTTP OK: HTTP/1.1 200 OK - 453 bytes in 0.002 second response time |time=0.001585s;;;0.000000 size=453B;;;0

[draakje]

Also,

I am running with debug = 256
I do see the the command executed

Code: Select all

[1340023807.004493] [2320.2] [pid=27964] Raw Command Input: $USER1$/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
[1340023807.004575] [2320.2] [pid=27964] Expanded Command Output: $USER1$/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
[1340023820.011375] [2320.2] [pid=27964] Raw Command Input: /usr/local/nagiosgraph/bin/insert.pl
[1340023820.011490] [2320.2] [pid=27964] Expanded Command Output: /usr/local/nagiosgraph/bin/insert.pl
[1340023820.011529] [256.1] [pid=27964] Running command '/usr/local/nagiosgraph/bin/insert.pl'...
[1340023820.097496] [256.1] [pid=27964] Execution time=0.085 sec, early timeout=0, result=0, output=(null)

What I don't see is the output appearing in the perfdata.log.

So what is responsible for the output to perfdata?

[sebastiaopburnay]

This is the output when I run the command manually:
PROCS OK: 48 processes with STATE = RSZDT

Yes, you are right, that check plugin does not give perf_data.

One thing you could do would be to develop your own plugin.

....

Except, I developed one just now, I was going to do it anyway, here:

Code: Select all

#!/bin/bash

# Determine memory usage percentage on Linux servers.


# Usage:  check_mem.sh WARNING CRITICAL
#         Where WARNING and CRITICAL are the integer only portions of the
#         percentage for the level desired.

# Define Levels based on input
#
WARNLEVEL=$1
CRITLEVEL=$2

# Setup standard Nagios/NRPE return codes
#
UNKNOWN_STATE=3
CRITICAL_STATE=2
WARNING_STATE=1
OK_STATE=0

# Give full paths to commands - Nagios can't determine location otherwise
# You should adjust these vars' values to the ones on your system
#
# (i.e. Run 'root@server:# whereis <command-name>' and check the full path)
# 
PS=/bin/ps
WC=/usr/bin/wc

BC=/usr/bin/bc
GREP=/bin/grep
AWK=/bin/awk
FREE=/usr/bin/free
TAIL=/usr/bin/tail
HEAD=/usr/bin/head

# Get amount of processes running into variable NRUNNINGPROCS
#
NRUNNINGPROCS=`ps aux | wc -l`


# Compare the NRUNNINGPROCS to the Warning and Critical therhsolds input at
# command line.  Issue message and set return code as appropriate for each
# level.  Nagios web page will use these to determine alarm level and message.
# 
# As you can see, it is designed to output the perf_data :)
#

if [ `echo "$NRUNNINGPROCS > $CRITLEVEL" |bc` -eq 1 ]
then echo "CRITICAL - ${NRUNNINGPROCS} Running Processes | 'Processes'=${NRUNNINGPROCS};${WARNLEVEL};${CRITLEVEL};"
     exit ${CRITICAL_STATE}
elif [ `echo "$NRUNNINGPROCS > $WARNLEVEL" |bc` -eq 1 ]
then echo "WARNING - ${NRUNNINGPROCS} Running Processes | 'Processes'=${NRUNNINGPROCS};${WARNLEVEL};${CRITLEVEL};"
     exit ${WARNING_STATE}
elif [ `echo "$NRUNNINGPROCS < $WARNLEVEL" |bc` -eq 1 ]
then echo "OK -  ${NRUNNINGPROCS} Running Processes | 'Processes'=${NRUNNINGPROCS};${WARNLEVEL};${CRITLEVEL};"
     exit ${OK_STATE}
else echo "Unable to determine number of running processes."
     exit ${UNKNOWN_STATE}
fi
echo "Unable to determine number of running processes."
exit ${UNKNOWN_STATE}

The output would be something like:

root@zion:/usr/local/nagios/libexec# ./check_running_procs.sh 140 180
WARNING - 160 Running Processes | 'Processes'=160;140;180;

Kind regards,
sebastiaopburnay

[sebastiaopburnay]

I forgot to mention,

To process my perfdata and produce graphs, I use pnp4nagios.

This link describes a very smple walk-trough to install/configure it:

http://www.techienote.com/2011/02/pnp4n ... buntu.html

Kind regards,
sebastiaopburnay

[draakje]

I have done some more checking..

I have now install nagios v3.2.3 with nagiosgraph 1.4.4 and in this combination I do get the graphs working for the total processes.

So the question remains:
What has changed since ver3.3.1 (and higher) what made the perfdata not working anymore ?


Thanks again

Edit:

The output of the same command is also the same (
/check_procs -w 250 -c 400 -s RSZDT
PROCS OK: 43 processes with STATE = RSZDT
So how can you tell that this pluging does not export the data??

Edit#2: when monitoring the perfdata.log I do see the exported data (1340089552||localhost||Total Processes||PROCS OK: 45 processes with STATE = RSZDT||) so again what has changed since 3.3.1 ?

Regards

[jsmurphy]

Except, I developed one just now, I was going to do it anyway, here:

That's pretty cool, you should put that on Nagios Exchange

have now install nagios v3.2.3 with nagiosgraph 1.4.4 and in this combination I do get the graphs working for the total processes.
So the question remains:
What has changed since ver3.3.1 (and higher) what made the perfdata not working anymore ?

Nothing will have changed in Nagios (in terms of altering this scenario), something will have changed in the process check plugin (maybe you replaced it?). The output you showed us from your earlier post revealed that the process check was not outputting performance data, where as the other checks you posted were outputting performance data. Glad you found a solution that got it working though!

[draakje]

Nothing will have changed in Nagios (in terms of altering this scenario), something will have changed in the process check plugin (maybe you replaced it?). The output you showed us from your earlier post revealed that the process check was not outputting performance data, where as the other checks you posted were outputting performance data. Glad you found a solution that got it working though!

So you think nothing has changed in nagios. Can you somehow confirm this?
I ask this because I do think something has changed.
This because of my testing.

On version 3.2.3 it works fine. And from version 3.3.1 it doesn't.

The output of the same command is also the same (
/check_procs -w 250 -c 400 -s RSZDT
PROCS OK: 43 processes with STATE = RSZDT
When monitoring the perfdata.log I do see the exported data (1340089552||localhost||Total Processes||PROCS OK: 45 processes with STATE = RSZDT||)
So my conclusion is that there has been some kind of change. (Maybe just a bug?)

Regards

[draakje]

Further more.

When running a debug (2048) It all seems the same (ver 3.2.3 vs ver3.4.1)
Except for this part.

Code: Select all

[1340090461.153834] [2048.1] [pid=27962] **** BEGIN MACRO PROCESSING ***********
[1340090461.153982] [2048.1] [pid=27962] Processing: '$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$'
[1340090461.154030] [2048.2] [pid=27962]   Processing part: ''
[1340090461.154074] [2048.2] [pid=27962]   Not currently in macro.  Running output (0): ''
[1340090461.154118] [2048.2] [pid=27962]   Processing part: 'LASTSERVICECHECK'
[1340090461.154162] [2048.2] [pid=27962]   macro_x[13] (LASTSERVICECHECK) match.
[1340090461.154208] [2048.2] [pid=27962]   Processed 'LASTSERVICECHECK', Clean Options: 0, Free: 1
[1340090461.154252] [2048.2] [pid=27962]   Processed 'LASTSERVICECHECK', Clean Options: 0, Free: 1
[1340090461.154296] [2048.2] [pid=27962]   Cleaning options: global=0, local=0, effective=0
[1340090461.154339] [2048.2] [pid=27962]   Uncleaned macro.  Running output (10): '1340090452'
[1340090461.154383] [2048.2] [pid=27962]   Just finished macro.  Running output (10): '1340090452'
[1340090461.154421] [2048.2] [pid=27962]   Processing part: '||'
[1340090461.154457] [2048.2] [pid=27962]   Not currently in macro.  Running output (12): '1340090452||'
[1340090461.154493] [2048.2] [pid=27962]   Processing part: 'HOSTNAME'
[1340090461.154529] [2048.2] [pid=27962]   macro_x[0] (HOSTNAME) match.
[1340090461.154565] [2048.2] [pid=27962]   Processed 'HOSTNAME', Clean Options: 0, Free: 1
[1340090461.154601] [2048.2] [pid=27962]   Processed 'HOSTNAME', Clean Options: 0, Free: 1
[1340090461.154637] [2048.2] [pid=27962]   Cleaning options: global=0, local=0, effective=0
[1340090461.154673] [2048.2] [pid=27962]   Uncleaned macro.  Running output (21): '1340090452||localhost'
[1340090461.154709] [2048.2] [pid=27962]   Just finished macro.  Running output (21): '1340090452||localhost'
[1340090461.154746] [2048.2] [pid=27962]   Processing part: '||'
[1340090461.154813] [2048.2] [pid=27962]   Not currently in macro.  Running output (23): '1340090452||localhost||'
[1340090461.154871] [2048.2] [pid=27962]   Processing part: 'SERVICEDESC'
[1340090461.154907] [2048.2] [pid=27962]   macro_x[3] (SERVICEDESC) match.
[1340090461.154943] [2048.2] [pid=27962]   Processed 'SERVICEDESC', Clean Options: 0, Free: 1
[1340090461.154979] [2048.2] [pid=27962]   Processed 'SERVICEDESC', Clean Options: 0, Free: 1
[1340090461.155015] [2048.2] [pid=27962]   Cleaning options: global=0, local=0, effective=0
[1340090461.155055] [2048.2] [pid=27962]   Uncleaned macro.  Running output (38): '1340090452||localhost||Total Processes'
[1340090461.155092] [2048.2] [pid=27962]   Just finished macro.  Running output (38): '1340090452||localhost||Total Processes'
[1340090461.155129] [2048.2] [pid=27962]   Processing part: '||'
[1340090461.155164] [2048.2] [pid=27962]   Not currently in macro.  Running output (40): '1340090452||localhost||Total Processes||'
[1340090461.155202] [2048.2] [pid=27962]   Processing part: 'SERVICEOUTPUT'
[1340090461.155238] [2048.2] [pid=27962]   macro_x[17] (SERVICEOUTPUT) match.
[1340090461.155274] [2048.2] [pid=27962]   New clean options: 3
[1340090461.155309] [2048.2] [pid=27962]   Processed 'SERVICEOUTPUT', Clean Options: 3, Free: 1
[1340090461.155429] [2048.2] [pid=27962]   Processed 'SERVICEOUTPUT', Clean Options: 3, Free: 1
[1340090461.155521] [2048.2] [pid=27962]   Cleaning options: global=0, local=3, effective=3
[1340090461.155566] [2048.2] [pid=27962]   Cleaned macro.  Running output (81): '1340090452||localhost||Total Processes||PROCS OK: 49 processes with STATE = RSZDT'
[1340090461.155621] [2048.2] [pid=27962]   Just finished macro.  Running output (81): '1340090452||localhost||Total Processes||PROCS OK: 49 processes with STATE = RSZDT'
[1340090461.155666] [2048.2] [pid=27962]   Processing part: '||'
[1340090461.155709] [2048.2] [pid=27962]   Not currently in macro.  Running output (83): '1340090452||localhost||Total Processes||PROCS OK: 49 processes with STATE = RSZDT||'
[1340090461.155753] [2048.2] [pid=27962]   Processing part: 'SERVICEPERFDATA'
[1340090461.155826] [2048.2] [pid=27962]   macro_x[19] (SERVICEPERFDATA) match.
[1340090461.155896] [2048.2] [pid=27962]   New clean options: 3
[1340090461.155938] [2048.2] [pid=27962]   Processed 'SERVICEPERFDATA', Clean Options: 3, Free: 1
[1340090461.155981] [2048.2] [pid=27962]   Processing part: ''
[1340090461.156023] [2048.2] [pid=27962]   Not currently in macro.  Running output (83): '1340090452||localhost||Total Processes||PROCS OK: 49 processes with STATE = RSZDT||'
[1340090461.156106] [2048.1] [pid=27962]   Done.  Final output: '1340090452||localhost||Total Processes||PROCS OK: 49 processes with STATE = RSZDT||'
[1340090461.156150] [2048.1] [pid=27962] **** END MACRO PROCESSING *************

This is not done in the new versions. Any idea why not?