Home » Categories » Products » Nagios XI » Troubleshooting » Common Problems

Nagios XI - Performance Graph Problems

Problem Description

Performance graphs not displaying data when their checks are returning true performance data.

The following troubleshooting steps may help you to find the root cause of the issue, and resolve it.

 

 

Editing Files

In many steps of this article you will be required to edit files. This documentation will use the vi text editor. When using the vi
editor:

  • To make changes press i on the keyboard first to enter insert mode
  • Press Esc to exit insert mode
  • When you have finished, save the changes in vi by typing :wq and press Enter

 

Check that Performance Data is enabled

Navigate to Admin > System Information > Monitoring Engine Status

Ensure that the Performance Data process is green.

 

 

Count The Amount Of Spooled Files

Nagios spools performance data into small files which get moved around and processed. Sometimes a problem can arise which stops the files from being processed and they begin to spool up. The following commands will count the amount of files in these locations:

ls /usr/local/nagios/var/spool/perfdata/ | wc -l
ls /usr/local/nagios/var/spool/xidpe/ | wc -l

Note: The pipe | symbol is used before the wc -l command.

 

If the commands return a number greater than 20,000 then it is likely you will need to delete the files as the processes will get caught in a loop and not be able to process them. In the logging steps further on in this article, if there are too many files you'll find that nothing gets logged, hence checking the amount of files first can save a lot of time.

To delete a large amount of files in a directory, execute this command:

find /usr/local/nagios/var/spool/perfdata/ -type f -delete

 

Note: This command can take a long time to execute, simply due to the volume of files. You can open another SSH session and observe the count of files in the folder with this command:

watch 'ls /usr/local/nagios/var/spool/perfdata/ | wc -l'

 

When the files are finally deleted, wait approximately thirty minutes to see if performance graphs start to work. If they do not, proceed with the next troubleshooting steps in this guide.

 

 

Increase Performance Data Logging Verbosity

Edit the following file from an SSH session:

/usr/local/nagios/etc/pnp/process_perfdata.cfg

Change:

LOG_LEVEL = 0

To:

LOG_LEVEL = 2

The process_perfdata.pl script should now log all errors and debug information to the file /usr/local/nagios/var/perfdata.log which can be watched using this command:

tail -f /usr/local/nagios/var/perfdata.log

 

Look for any errors, incorrect exit codes, and/or timeouts.

Remember to return this value to it's default setting when troubleshooting is completed.

One of the top problem causes you can find in this log is a typical timeout error, to resolve that temporarily, you can increase the performance data processor's timeout range by editing:

/usr/local/nagios/etc/pnp/process_perfdata.cfg

Change:

TIMEOUT = 5

To:

TIMEOUT = 20

Or a greater number if this continues to be a issue.

 

 

Increase NPCD Logging Verbosity

NPCD is a bulk processing tool which reaps and processes your performance data once received. To increase it's logging verbosity edit the following file in an SSH session:

/usr/local/nagios/etc/pnp/npcd.cfg

Change:

log_level = 0

To:

log_level = -1

 

Yes, that -1 is not a typing mistake, it will increase the verbosity. Now, restart the NPCD service using one of the commands below:

RHEL 6 | CentOS 6 | Oracle Linux 6 | Ubuntu 14

service npcd restart

 

RHEL 7 | CentOS 7 | Oracle Linux 7 | Debian | Ubuntu 16/18

systemctl restart npcd.service

 

Remember to return this value to it's default setting when troubleshooting is completed.

NPCD should now log all errors and debug information to the file /usr/local/nagios/var/npcd.log which can be watched using this command:

tail -f /usr/local/nagios/var/npcd.log

 

One of the top problem causes to look for in the above log is lines indicating that you are hitting a load threshold, this is common if you are either receiving too much data for NPCD to keep up with the current system's load, or that it is trying to crunch through stacked up performance data.

You can increase this threshold by editing the following file:

/usr/local/nagios/etc/pnp/npcd.cfg

Change:

load_threshold = 10.0

To a value greater than your system's current load. Use this with caution however, as the NPCD process will eat as much load as you give it, so watch your resources!

 

 

Check Nagios User Account

In some situations the nagios user account can expire causing issues like this to occur. Run this command to see if the nagios user is expired:

chage -l nagios

 

If it is, run this command to enable the expired nagios user:

chage -I -1 -m 0 -M 99999 -E -1 nagios

 

Check Number of Metrics Gathered

It is also possible that the number of metrics gathered by RRD has changed. If you compare what metrics are shown on the graphs to what is shown at the bottom of the service screen, and they're different, you will need to run the RRD conversion script.

 

Final Thoughts

For any support related questions please visit the Nagios Support Forums at:

http://support.nagios.com/forum/

1 (1)
Article Rating (1 Votes)
Rate this article
  • Icon PDFExport to PDF
  • Icon MS-WordExport to MS Word
Attachments Attachments
There are no attachments for this article.
Related Articles RSS Feed
Nagios XI - Defining Global Environment Variables
Viewed 3529 times since Thu, Mar 17, 2016
Nagios XI - Resetting The nagiosadmin Password
Viewed 19968 times since Mon, Jan 25, 2016
NDOUtils - Message Queue Exceeded
Viewed 6225 times since Thu, Jan 21, 2016
Nagios XI - CentOS 6 Installation Problems XI 2011R1.7 2011R1.8
Viewed 1576 times since Tue, Feb 2, 2016
Nagios XI - NSP: Sorry Dave, I can’t let you do that
Viewed 8668 times since Tue, Jan 27, 2015
Nagios XI - SNMPTT Service generates Cannot find module errors
Viewed 1907 times since Sun, Apr 9, 2017
Disabling Port 113 IDENT Requests
Viewed 1735 times since Tue, Mar 12, 2019
Nagios XI - MSSQL Wizards - Adaptive Server connection failed
Viewed 3723 times since Thu, Aug 3, 2017
Slack Notifications Stopped Working
Viewed 945 times since Wed, Mar 18, 2020
Nagios XI - ICMP and Ping Checks Stopped Graphing
Viewed 4304 times since Mon, Jan 25, 2016