Some Service Performance Graphs Not Showing

[chrisp]

Hi,

I've just gone through the FAQ relating to "Performance Graphs Are Missing Or Not Displayed" but my issue remains unresolved.

Although some of our performance graphs are displaying, some are not.

For example, in the attached screenshot, as far as I understand, there should be graphs for all listed services. I have configured them to be enabled, I can see the data files are in /usr/local/nagios/share/perfdata/mdfmta012.xxx.xxxx.xxx/ as follows: -

Code: Select all

[root@Nagios ~]# ll -a /usr/local/nagios/share/perfdata/mdfmta012.xxx.xxxx.xxx/        
drwxrwxr-x  2 nagios nagios   4096 Jan  3 17:16 .
drwxrwxr-x 96 nagios nagios   4096 Jan  3 13:22 ..
-rwxrwxr-x  1 nagios nagios 768224 Jan  3 17:01 _HOST_.rrd
-rw-rw-rw-  1 nagios nagios   2702 Jan  3 17:16 _HOST_.xml
-rwxrwxr-x  1 nagios nagios 384952 Jan  3 17:15 SMTP_25.rrd
-rw-rw-rw-  1 nagios nagios   2116 Jan  3 17:15 SMTP_25.xml
-rwxrwxr-x  1 nagios nagios 384952 Dec 21 16:14 SMTP_Submission_587.rrd
-rwxrwxr-x  1 nagios nagios   2196 Dec 21 16:14 SMTP_Submission_587.xml
-rwxrwxr-x  1 nagios nagios 384952 Jan  3 17:01 XXXXXSMTP_25.rrd
-rw-rw-rw-  1 nagios nagios   2270 Jan  3 17:16 XXXXXSMTP_25.xml
-rwxrwxr-x  1 nagios nagios 384952 Jan  3 17:11 XXXXXSMTPS_465.rrd
-rw-rw-rw-  1 nagios nagios   2358 Jan  3 17:16 XXXXXSMTPS_465.xml
-rwxrwxr-x  1 nagios nagios 384952 Jan  3 17:09 XXXXXSMTP_Submission_587.rrd
-rw-rw-rw-  1 nagios nagios   2344 Jan  3 17:15 XXXXXSMTP_Submission_587.xml

Can you see anything I'm doing wrong? Is there something I can submit, which will give you the detail you need to see the root cause?

Cheers,
--
ChrisP

Linux Distro: CentOS release 6.3 (Final)
Architechure: x86_64
Nagios Core 3.4.1
Installation: Manual
SSL Enabled: Yes

###################################
# Nagios XI version information
###################################
full=2012R1.3
major=2012
minor=R1.3
releasedate=2012-12-05
release=303

[abrist]

Could you post a tail of the log /usr/local/nagios/var/perfdata.log?

root@localhost$ tail -50 /usr/local/nagios/var/perfdata.log

[chrisp]

Code: Select all

[root@Nagios ~]# tail -50 /usr/local/nagios/var/perfdata.log
2012-12-27 14:49:54 [8337] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-27 14:49:54 [8337] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-27 14:49:54 [8337] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-27 14:49:54 [8337] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1356619776.perfdata.service-PID-8337 deleted
2012-12-27 14:49:54 [8337] [0] *** Timeout while processing Host: "cas03-eq.xxx.xxxx.xxx" Service: "THUS_POP3S"
2012-12-27 14:49:54 [8337] [0] *** process_perfdata.pl terminated on signal ALRM
2012-12-28 01:01:30 [19510] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-28 01:01:30 [19510] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-28 01:01:30 [19511] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-28 01:01:30 [19513] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-28 01:01:30 [19510] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-28 01:01:30 [19511] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-28 01:01:30 [19513] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-28 01:01:30 [19511] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-28 01:01:30 [19510] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1356656464.perfdata.host-PID-19510 deleted
2012-12-28 01:01:30 [19513] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-28 01:01:30 [19511] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1356656479.perfdata.service-PID-19511 deleted
2012-12-28 01:01:30 [19513] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1356656464.perfdata.service-PID-19513 deleted
2012-12-28 01:01:30 [19510] [0] *** Timeout while processing Host: "mdfmta009.xxx.xxxx.xxx" Service: "_HOST_"
2012-12-28 01:01:30 [19510] [0] *** process_perfdata.pl terminated on signal ALRM
2012-12-28 01:01:30 [19511] [0] *** Timeout while processing Host: "mdfmta004.xxx.xxxx.xxx" Service: "THUS_SMTP_25"
2012-12-28 01:01:30 [19513] [0] *** Timeout while processing Host: "cas09-tmg.xxx.xxxx.xxx" Service: "THUS_IMAP"
2012-12-28 01:01:30 [19511] [0] *** process_perfdata.pl terminated on signal ALRM
2012-12-28 01:01:30 [19513] [0] *** process_perfdata.pl terminated on signal ALRM
2012-12-28 01:01:30 [19512] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-28 01:01:30 [19512] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-28 01:01:30 [19512] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-28 01:01:30 [19512] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1356656479.perfdata.host-PID-19512:No such file or directory
2012-12-28 01:01:30 [19512] [0] *** Timeout while processing Host: ".pnp-internal" Service: "runtime"
2012-12-28 01:01:30 [19512] [0] *** process_perfdata.pl terminated on signal ALRM

[abrist]

How many checks are you running and how long are the ones that are failing to report perfdata taking to run?

You should try increasing the timeout in the process_perfdata.cfg file.

Edit: /usr/local/nagios/etc/pnp/process_perfdata.cfg
change "TIMEOUT = 5" to "TIMEOUT = 10"

Restart npcd:

root@localhost$ service npcd restart

Give it about 15 minutes and then check your logs and graphs once again.

[Gavin]

I work with Chris, and have also been looking at this problem. The same service is showing up to date data via. highcharts, but nothing comes up for the normal charts. It's trying to load the URL: nagios/nagiosxi/includes/components/perfdata/graphApi.php?host=mdfmta012.xxx.xxx.xxx&service=SMTP_25&source=1&view=1&start=&end=&rand=1357401766

I noticed last night that this error was being written to /var/log/messages:

Code: Select all

Jan  5 01:40:23 Nagios rrdcached[2874]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/mdfmta012.xxx.xxx.xxx/SMTP_25.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/mdfmta012.xxx.xxx.xxx/SMTP_25.rrd: illegal attempt to update using time 1357348695 when last update time is 1357349955 (minimum one second step))

I then disabled 'rrdcached' by commenting out 'RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock' in '/usr/local/nagios/etc/pnp/process_perfdata.cfg' and restarting npcd. This didn't help, and the graphs still weren't appearing so, several hours later, I re-enabled 'rrdcached'. Interestingly, the error above is no longer being written to the logs. We installed rrdcached according to the instruction on your website.

Any ideas where we can go next? The data is clearly there, it's just playing hard to get!

Thanks,

Gavin

[scottwilkerson]

Can you check the httpd error log while accessing one of the pages.

Code: Select all

tail -f /var/log/httpd/error_log

to see if you are getting errors there...

[Gavin]

Nope. We're getting some entries from opscreen.php (unrelated), but nothing else.

Thanks,

Gavin

[mguthrie]

Try loading the following URL and see if you have a corrupt image:

nagios/nagiosxi/includes/components/perfdata/graphApi.php?host=localhost

[Gavin]

We get 'No performance data available' back for that. If I change 'localhost' to the hostname we've changed Nagios to, I get a graph back.

Thanks,

Gavin

[mguthrie]

Hmm, just for fun can you try it once more with the additional arguments in place:

nagios/nagiosxi/includes/components/perfdata/graphApi.php?host=mdfmta012.xxx.xxx.xxx&service=SMTP_25&source=1&view=1&start=&end=&rand=1357401766

Are any of the rrdtool graphs working, or are they all broken?