Nagios performance data not working

[vuduops]

MOD NOTE: Split from https://support.nagios.com/forum/viewto ... 16&t=44950

I ran the install_ramdisk.sh. That seemed to have fixed the performance grapher crashing issue but now I donot see any RRD file being created. Can you please help.

Thanks
Krishna

[dwhitfield]

Please take a look at https://support.nagios.com/kb/article.php?id=9, but be sure to adjust your file paths to the ramdisk file paths.

Additionally, can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

You can also generate a profile manually using the script at /usr/local/nagiosxi/html/includes/components/profile/getprofile.sh

That should generate a profile in /usr/local/nagiosxi/var/components/ which you can get off the server with an application such as FileZilla.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

If you get an error that PROFILE BUILD FAILED, please see https://support.nagios.com/kb/article.p ... ategory=44

[vuduops]

Please find the system profile zip attached

[dwasswa]

Hi @ vuduops

Nagios spools small files that are moved around and processed. Sometimes these spooled files get filled up.

You will need to count the amount of these spooled files. The standard amount should be number <=20,000 which healthy.

If any number comes up above 20,000,you will need to delete some because some processes will get caught up in a loop.


Please follow the steps below to resolve this issue.

1.Count the amount of these spooled files by running the command below:

The output for these commands will be a number

Code: Select all

ls /usr/local/nagios/var/spool/perfdata/ | wc -l

Code: Select all

ls /usr/local/nagios/var/spool/xidpe/ | wc -l

Remember,if you get a large number above 20,000 it means problems.

2.If it is a large number above 20,000 those files need to be deleted so please run the command below to delete them.

Code: Select all

find /usr/local/nagios/var/spool/perfdata/ -type f -delete

3.PLEASE WAIT 30 MINUTES TO SEE IF THE PERFORMANCE GRAPHS ARE WORKING AGAIN.


Please try that and let me know if it worked or if you have any questions.

Otherwise if we will have to change verbosity for performance data logging to a higher lever in performance.cfg and npcd.cfg files.

Note: NPCD deamon is responsible for processing performance data. That is why i mentioned its configuration file(npcd.cfg ) above in regards to increasing logging verbosity.

However,don't worry about that for now,first try what i mentioned in the first three steps above.

[vuduops]

This is what I see. Still no rrd file being generated

Code: Select all

[[email protected] tmp]#     ls /usr/local/nagios/var/spool/perfdata/ | wc -l
1
[[email protected] tmp]#     ls /usr/local/nagios/var/spool/xidpe/ | wc -l
6855

[npolovenko]

@vuduops, Can you check the contents of this folder:

Code: Select all

/var/nagiosramdisk/spool/perfdata/

Also based on the log files you need to increase resource load threshold in order for npcd to run correctly:

Code: Select all

 Open the npcd config file:
/usr/local/nagios/etc/pnp/npcd.cfg
And change this line:
load_threshold = 10.0
to this:
load_threshold = 20.0
Save and exit

When that is done please restart npcd daemon:
service npcd restart

[vuduops]

Here is the output

Code: Select all

[email protected] tmp]# cd     /var/nagiosramdisk/spool/perfdata/
[[email protected] perfdata]# ls
./  ../
[[email protected] perfdata]# vi /usr/local/nagios/etc/pnp/npcd.cfg
[[email protected] perfdata]# grep load_threshold /usr/local/nagios/etc/pnp/npcd.cfg
# use_load_threshold - enables/disables load watching
# use_load_threshold = <0 / 1> (default: 0)
#use_load_threshold = 0
# load_threshold - npcd won't start new threads
# load_threshold = <float value> (default: 10.0)
load_threshold = 20.0
[[email protected] perfdata]# service npcd restart
NPCD Stopped.
DEBUG: Config File = /usr/local/nagios/etc/pnp/npcd.cfg
CONFIG_OPT_LOGTYPE = file
CONFIG_OPT_LOGFILE = /usr/local/nagios/var/npcd.log
CONFIG_OPT_LOGFILESIZE = 10485760
CONFIG_OPT_LOGLEVEL = -1
CONFIG_OPT_SCANDIR = /var/nagiosramdisk/spool/perfdata/
CONFIG_OPT_RUNCMD = /usr/local/nagios/libexec/process_perfdata.pl
CONFIG_OPT_RUNCMD_ARG = -b
CONFIG_OPT_MAXTHREADS = 5
CONFIG_OPT_LOAD = 20.0
CONFIG_OPT_USER = nagios
CONFIG_OPT_GROUP = nagios
CONFIG_OPT_PIDFILE = /usr/local/nagiosxi/var/subsys/npcd.pid
CONFIG_OPT_SLEEPTIME = 15
CONFIG_OPT_IDENTMYSELF = (null)
---------------------------
DEBUG: load_threshold is enabled - ('20.000000')
NPCD started.

[dwasswa]

Hi @vuduops,

Please run the the following command to ensure the crond system daemon is running:

Code: Select all

service crond status

Run the command below and post the error messages from cron jobs.

Code: Select all

tail -100 /var/log/cron

Lets also check if the

Code: Select all

perfdataproc.php

cron job is running by running the command below and post output.

Code: Select all

ps axuw | grep perfdataproc.php

Check for errors and post output.

Code: Select all

/usr/local/nagiosxi/var/perfdataproc.log

[vuduops]

Hi @vuduops,

1. service crond status

Code: Select all

[[email protected] perfdata]#     service crond status
crond (pid  4848) is running...

2 . Run the command below and post the error messages from cron jobs./var/log/cron

Code: Select all

[[email protected] perfdata]# ls -ltr /var/log/cron
-rw-r--r-- 1 root root 6587255 Oct 25 18:24 /var/log/cron
[[email protected] perfdata]# chmod 775 /var/log/cron
[[email protected] perfdata]# ls -ltr /var/log/cron
-rwxrwxr-x 1 root root 6587255 Oct 25 18:24 /var/log/cron*
[[email protected] perfdata]#     service crond restart
Stopping crond:                                            [  OK  ]
Starting crond:                                            [  OK  ]

[[email protected] perfdata]# /var/log/cron
-bash: /var/log/cron: Text file busy

3. Lets also check if the perfdataproc.php cron job is running by running the command below and post output ps axuw | grep perfdataproc.php

Code: Select all

[[email protected] perfdata]# grep -i error /usr/local/nagiosxi/var/perfdataproc.log
[[email protected] perfdata]#     ps axuw | grep perfdataproc.php
nagios   23601  0.0  0.0 106072  1280 ?        Ss   18:29   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios   23607  0.2  0.0 352048 23244 ?        S    18:29   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
root     30995  0.0  0.0 103320   968 pts/0    S+   18:29   0:00 grep perfdataproc.php

4. Check for errors and post output.

Code: Select all

[[email protected] perfdata]# grep -i error /usr/local/nagiosxi/var/perfdataproc.log
[[email protected] perfdata]# 

[tgriep]

I took a look at your profile and it looks like the install_ramdisk.sh script did not change all of the files so that is the reason the RRD files are not getting created / updated.
You can try and run the script again and see if the rest of the files get updated or you can manually update the files be doing the manual process in this PDF file.
http://assets.nagios.com/downloads/nagi ... giosXI.pdf