Page 1 of 2
Some Service Performance Graphs Not Showing
Posted: Fri Jan 04, 2013 5:14 am
by chrisp
Hi,
I've just gone through the
FAQ relating to "Performance Graphs Are Missing Or Not Displayed" but my issue remains unresolved.
Although some of our performance graphs
are displaying, some are not.
For example, in the attached screenshot, as far as I understand, there should be graphs for all listed services. I have configured them to be enabled, I can see the data files are in
/usr/local/nagios/share/perfdata/mdfmta012.xxx.xxxx.xxx/ as follows: -
Code: Select all
[root@Nagios ~]# ll -a /usr/local/nagios/share/perfdata/mdfmta012.xxx.xxxx.xxx/
drwxrwxr-x 2 nagios nagios 4096 Jan 3 17:16 .
drwxrwxr-x 96 nagios nagios 4096 Jan 3 13:22 ..
-rwxrwxr-x 1 nagios nagios 768224 Jan 3 17:01 _HOST_.rrd
-rw-rw-rw- 1 nagios nagios 2702 Jan 3 17:16 _HOST_.xml
-rwxrwxr-x 1 nagios nagios 384952 Jan 3 17:15 SMTP_25.rrd
-rw-rw-rw- 1 nagios nagios 2116 Jan 3 17:15 SMTP_25.xml
-rwxrwxr-x 1 nagios nagios 384952 Dec 21 16:14 SMTP_Submission_587.rrd
-rwxrwxr-x 1 nagios nagios 2196 Dec 21 16:14 SMTP_Submission_587.xml
-rwxrwxr-x 1 nagios nagios 384952 Jan 3 17:01 XXXXXSMTP_25.rrd
-rw-rw-rw- 1 nagios nagios 2270 Jan 3 17:16 XXXXXSMTP_25.xml
-rwxrwxr-x 1 nagios nagios 384952 Jan 3 17:11 XXXXXSMTPS_465.rrd
-rw-rw-rw- 1 nagios nagios 2358 Jan 3 17:16 XXXXXSMTPS_465.xml
-rwxrwxr-x 1 nagios nagios 384952 Jan 3 17:09 XXXXXSMTP_Submission_587.rrd
-rw-rw-rw- 1 nagios nagios 2344 Jan 3 17:15 XXXXXSMTP_Submission_587.xml
Can you see anything I'm doing wrong? Is there something I can submit, which will give you the detail you need to see the root cause?
Cheers,
--
ChrisP
Linux Distro: CentOS release 6.3 (Final)
Architechure: x86_64
Nagios Core 3.4.1
Installation: Manual
SSL Enabled: Yes
###################################
# Nagios XI version information
###################################
full=2012R1.3
major=2012
minor=R1.3
releasedate=2012-12-05
release=303
Re: Some Service Performance Graphs Not Showing
Posted: Fri Jan 04, 2013 11:35 am
by abrist
Could you post a tail of the log /usr/local/nagios/var/perfdata.log?
root@localhost$ tail -50 /usr/local/nagios/var/perfdata.log
Re: Some Service Performance Graphs Not Showing
Posted: Fri Jan 04, 2013 12:14 pm
by chrisp
Code: Select all
[root@Nagios ~]# tail -50 /usr/local/nagios/var/perfdata.log
2012-12-27 14:49:54 [8337] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-27 14:49:54 [8337] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-27 14:49:54 [8337] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-27 14:49:54 [8337] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1356619776.perfdata.service-PID-8337 deleted
2012-12-27 14:49:54 [8337] [0] *** Timeout while processing Host: "cas03-eq.xxx.xxxx.xxx" Service: "THUS_POP3S"
2012-12-27 14:49:54 [8337] [0] *** process_perfdata.pl terminated on signal ALRM
2012-12-28 01:01:30 [19510] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-28 01:01:30 [19510] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-28 01:01:30 [19511] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-28 01:01:30 [19513] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-28 01:01:30 [19510] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-28 01:01:30 [19511] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-28 01:01:30 [19513] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-28 01:01:30 [19511] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-28 01:01:30 [19510] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1356656464.perfdata.host-PID-19510 deleted
2012-12-28 01:01:30 [19513] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-28 01:01:30 [19511] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1356656479.perfdata.service-PID-19511 deleted
2012-12-28 01:01:30 [19513] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1356656464.perfdata.service-PID-19513 deleted
2012-12-28 01:01:30 [19510] [0] *** Timeout while processing Host: "mdfmta009.xxx.xxxx.xxx" Service: "_HOST_"
2012-12-28 01:01:30 [19510] [0] *** process_perfdata.pl terminated on signal ALRM
2012-12-28 01:01:30 [19511] [0] *** Timeout while processing Host: "mdfmta004.xxx.xxxx.xxx" Service: "THUS_SMTP_25"
2012-12-28 01:01:30 [19513] [0] *** Timeout while processing Host: "cas09-tmg.xxx.xxxx.xxx" Service: "THUS_IMAP"
2012-12-28 01:01:30 [19511] [0] *** process_perfdata.pl terminated on signal ALRM
2012-12-28 01:01:30 [19513] [0] *** process_perfdata.pl terminated on signal ALRM
2012-12-28 01:01:30 [19512] [0] *** TIMEOUT: Timeout after 5 secs. ***
2012-12-28 01:01:30 [19512] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2012-12-28 01:01:30 [19512] [0] *** TIMEOUT: Please check your npcd.cfg
2012-12-28 01:01:30 [19512] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1356656479.perfdata.host-PID-19512:No such file or directory
2012-12-28 01:01:30 [19512] [0] *** Timeout while processing Host: ".pnp-internal" Service: "runtime"
2012-12-28 01:01:30 [19512] [0] *** process_perfdata.pl terminated on signal ALRM
Re: Some Service Performance Graphs Not Showing
Posted: Fri Jan 04, 2013 12:22 pm
by abrist
How many checks are you running and how long are the ones that are failing to report perfdata taking to run?
You should try increasing the timeout in the process_perfdata.cfg file.
Edit: /usr/local/nagios/etc/pnp/process_perfdata.cfg
change "TIMEOUT = 5" to "TIMEOUT = 10"
Restart npcd:
root@localhost$ service npcd restart
Give it about 15 minutes and then check your logs and graphs once again.
Re: Some Service Performance Graphs Not Showing
Posted: Sat Jan 05, 2013 11:16 am
by Gavin
I work with Chris, and have also been looking at this problem. The same service is showing up to date data via. highcharts, but nothing comes up for the normal charts. It's trying to load the URL:
nagios/nagiosxi/includes/components/perfdata/graphApi.php?host=mdfmta012.xxx.xxx.xxx&service=SMTP_25&source=1&view=1&start=&end=&rand=1357401766
I noticed last night that this error was being written to
/var/log/messages:
Code: Select all
Jan 5 01:40:23 Nagios rrdcached[2874]: queue_thread_main: rrd_update_r (/usr/local/nagios/share/perfdata/mdfmta012.xxx.xxx.xxx/SMTP_25.rrd) failed with status -1. (/usr/local/nagios/share/perfdata/mdfmta012.xxx.xxx.xxx/SMTP_25.rrd: illegal attempt to update using time 1357348695 when last update time is 1357349955 (minimum one second step))
I then disabled 'rrdcached' by commenting out
'RRD_DAEMON_OPTS = unix:/var/rrdtool/rrdcached/rrdcached.sock' in
'/usr/local/nagios/etc/pnp/process_perfdata.cfg' and restarting npcd. This didn't help, and the graphs still weren't appearing so, several hours later, I re-enabled 'rrdcached'. Interestingly, the error above is no longer being written to the logs. We installed rrdcached according to the instruction on your website.
Any ideas where we can go next? The data is clearly there, it's just playing hard to get!
Thanks,
Gavin
Re: Some Service Performance Graphs Not Showing
Posted: Mon Jan 07, 2013 10:11 am
by scottwilkerson
Can you check the httpd error log while accessing one of the pages.
to see if you are getting errors there...
Re: Some Service Performance Graphs Not Showing
Posted: Mon Jan 07, 2013 11:18 am
by Gavin
Nope. We're getting some entries from opscreen.php (unrelated), but nothing else.
Thanks,
Gavin
Re: Some Service Performance Graphs Not Showing
Posted: Mon Jan 07, 2013 11:28 am
by mguthrie
Try loading the following URL and see if you have a corrupt image:
nagios/nagiosxi/includes/components/perfdata/graphApi.php?host=localhost
Re: Some Service Performance Graphs Not Showing
Posted: Mon Jan 07, 2013 11:31 am
by Gavin
We get 'No performance data available' back for that. If I change 'localhost' to the hostname we've changed Nagios to, I get a graph back.
Thanks,
Gavin
Re: Some Service Performance Graphs Not Showing
Posted: Tue Jan 08, 2013 10:32 am
by mguthrie
Hmm, just for fun can you try it once more with the additional arguments in place:
nagios/nagiosxi/includes/components/perfdata/graphApi.php?host=mdfmta012.xxx.xxx.xxx&service=SMTP_25&source=1&view=1&start=&end=&rand=1357401766
Are any of the rrdtool graphs working, or are they all broken?