Perfdata is being collected but performance graph is empty

[joe.ward]

When viewing hostgroup status (summary, grid, etc, or BBmap) there are disk monitoring services (C and D drive) that show status. If I drill down for a host, the performance chart of the disk space is empty. I've found one host out of about 25 that has a graph with data for the C drive. The graph starts earlier today when I restarted the Nagios XI services (Admin Home - Performance Grapher) but this seems to be only for one host-service pair.

A separate service is defined for each disk drive then the servers are assigned to the service through hostgroups.

Perflog says data is being found for hosts that have empty graphs (axis, etc display but no data in the graph).

CPU data is graphing OK. The issue seems somehow related to the service, but works for one host-service...

Thank you

[lmiltchev]

When you click on the service in question (Home->Service Detail) and go to the "Advanced" tab, do you see anything next to "Performance Data" under the "Advanced Status Details" section? What is the output of the following command?

Code: Select all

ll /usr/local/nagios/share/perfdata/

[joe.ward]

When you click on the service in question (Home->Service Detail) and go to the "Advanced" tab, do you see anything next to "Performance Data" under the "Advanced Status Details" section?

Performance Data shows:
Service State: Warning
Duration: 8d 0h 55m 54s
State Type: Hard
Current Check: 2 of 2
Last Check: 2013-12-04 16:16:16
Next Check: 2013-12-04 16:46:16
Last State Change: 2013-11-26 15:48:46
Last Notification: 2013-12-04 15:46:25
Check Type: Active
Check Latency: 0.139 seconds
Execution Time: 0.008 seconds
State Change: 0%
Performance Data: 'D:\ Used Space'=122.77Gb;120.00;135.00;0.00;150.00

What is the output of the following command?

There is a list of directories, each is owned by the nagios user and group, mode rwx, with current time stamps.

Thank you

[abrist]

Lets check the logs:

Code: Select all

tail -25 /usr/local/nagios/var/perfdata.log
tail -25 /usr/local/nagios/var/npcd.log

[joe.ward]

The log files look "normal".

I had the disk check set for every 240 minutes. I set it to 30 minutes so I wouldn't have to wait so long to see if it was working. Seems OK since then. I'll set it back and see if it stays good. Strange...

[sreinhardt]

can you please post the output of the commands lmiltchev posted. It may seem normal, but there could be something additional that jumps out to us.

[tmcdonald]

We'll keep this thread open for you.

[joe.ward]

OK - if I set the disk check interval to 240 minutes, it stops showing in the perf chart. If I set it to 30 minutes, the perf chart shows data.

tail -25 perfdata.log

2013-12-06 14:45:19 [6881] [1] Found Performance Data for www4stl / APC_Uptime (time=0.429332s;;;0.000000 size=28583B;;;0)
2013-12-06 14:45:19 [6881] [1] Found Performance Data for www3stl / APC_Uptime (time=0.459559s;;;0.000000 size=28583B;;;0)
2013-12-06 14:45:19 [6881] [1] Found Performance Data for vmapcccrm / CPU_Use (5 min avg Load=2%;80;90;0;100)
2013-12-06 14:45:19 [6881] [1] 11 lines processed
2013-12-06 14:45:19 [6881] [1] /usr/local/nagios/var/spool/perfdata//1386359103.perfdata.service-PID-6881 deleted
2013-12-06 14:45:19 [6881] [1] PNP exiting (runtime 0.008936s) ...
2013-12-06 14:45:34 [7020] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2013-12-06 14:45:34 [7019] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2013-12-06 14:45:34 [7019] [1] Found Performance Data for www5stl / APC_Uptime (time=0.404564s;;;0.000000 size=28583B;;;0)
2013-12-06 14:45:34 [7020] [1] Found Performance Data for vmapcccrm / _HOST_ (rta=1.948ms;3000.000;5000.000;0; pl=0%;80;100;;)
2013-12-06 14:45:34 [7020] [1] Found Performance Data for www1stl / _HOST_ (rta=1.046ms;3000.000;5000.000;0; pl=0%;80;100;;)
2013-12-06 14:45:34 [7019] [1] Found Performance Data for wvusstlopsfo1 / CPU_Use (5 min avg Load=4%;80;90;0;100)
2013-12-06 14:45:34 [7019] [1] Found Performance Data for SE_Prod_2 / Disk_C__80-90 (C:\ Used Space=23.26Gb;32.00;36.00;0.00;40.00)
2013-12-06 14:45:34 [7020] [1] Found Performance Data for www5stl / _HOST_ (rta=1.625ms;3000.000;5000.000;0; pl=0%;80;100;;)
2013-12-06 14:45:34 [7020] [1] Found Performance Data for http://www.sereply.com / _HOST_ (time=1.323247s;;;0.000000 size=118944B;;;0)
2013-12-06 14:45:34 [7020] [1] 4 lines processed
2013-12-06 14:45:34 [7020] [1] /usr/local/nagios/var/spool/perfdata//1386359118.perfdata.host-PID-7020 deleted
2013-12-06 14:45:34 [7020] [1] PNP exiting (runtime 0.005259s) ...
2013-12-06 14:45:34 [7019] [1] Found Performance Data for www3stl / CPU_Use (5 min avg Load=14%;80;90;0;100)
2013-12-06 14:45:34 [7019] [1] Found Performance Data for vmspectradev / CPU_Use (5 min avg Load=1%;80;90;0;100)
2013-12-06 14:45:34 [7019] [1] Found Performance Data for vmapccdev / CPU_Use (5 min avg Load=0%;80;90;0;100)
2013-12-06 14:45:34 [7019] [1] Found Performance Data for SE_Prod_1 / CPU_Use (5 min avg Load=7%;80;90;0;100)
2013-12-06 14:45:34 [7019] [1] 10 lines processed
2013-12-06 14:45:34 [7019] [1] /usr/local/nagios/var/spool/perfdata//1386359118.perfdata.service-PID-7019 deleted
2013-12-06 14:45:34 [7019] [1] PNP exiting (runtime 0.020058s) ...

tail -25 npcd.log

[12-06-2013 14:48:35] NPCD: No more files to process... waiting for 15 seconds
[12-06-2013 14:48:50] NPCD: Processing file '1386359313.perfdata.host'
[12-06-2013 14:48:50] NPCD: Processing file '1386359313.perfdata.service'
[12-06-2013 14:48:51] NPCD: No more files to process... waiting for 15 seconds
[12-06-2013 14:49:06] NPCD: Processing file '1386359328.perfdata.host'
[12-06-2013 14:49:06] NPCD: Processing file '1386359328.perfdata.service'
[12-06-2013 14:49:06] NPCD: No more files to process... waiting for 15 seconds
[12-06-2013 14:49:21] NPCD: Processing file '1386359343.perfdata.host'
[12-06-2013 14:49:21] NPCD: Processing file '1386359343.perfdata.service'
[12-06-2013 14:49:21] NPCD: No more files to process... waiting for 15 seconds
[12-06-2013 14:49:36] NPCD: Processing file '1386359358.perfdata.service'
[12-06-2013 14:49:36] NPCD: Processing file '1386359358.perfdata.host'
[12-06-2013 14:49:36] NPCD: No more files to process... waiting for 15 seconds
[12-06-2013 14:49:51] NPCD: Processing file '1386359373.perfdata.host'
[12-06-2013 14:49:51] NPCD: Processing file '1386359373.perfdata.service'
[12-06-2013 14:49:51] NPCD: No more files to process... waiting for 15 seconds
[12-06-2013 14:50:06] NPCD: Processing file '1386359388.perfdata.service'
[12-06-2013 14:50:06] NPCD: Processing file '1386359388.perfdata.host'
[12-06-2013 14:50:06] NPCD: No more files to process... waiting for 15 seconds
[12-06-2013 14:50:21] NPCD: Processing file '1386359403.perfdata.host'
[12-06-2013 14:50:21] NPCD: Processing file '1386359403.perfdata.service'
[12-06-2013 14:50:21] NPCD: No more files to process... waiting for 15 seconds
[12-06-2013 14:50:36] NPCD: Processing file '1386359418.perfdata.host'
[12-06-2013 14:50:36] NPCD: Processing file '1386359418.perfdata.service'
[12-06-2013 14:50:36] NPCD: No more files to process... waiting for 15 seconds

[sreinhardt]

This is most likely due to the rrd file expecting information every 5 minutes. They can handle information being put in at slightly lesser intervals, but once you reach a critical point, sometime after one hour, the majority of stored results are nans and will break graphing. Otherwise, I would agree your logs look nice and clean!

[joe.ward]

Is the expectation for 5 minute intervals, and breaking after 60 minutes, documented somewhere? It would be nice if the CCM would give a warning, had a footnote, pop up when saving, or something to warn a fella!

Thanks for the explanation.