Performance graphs not showing for a specific user

[IkaIka]

Hello, I've been contacted by a colleague whose unable to display performance graphs in XI. He says they used to render fine for him and only stopped working recently. I'm at loss as to what might be causing this. Our XI version is 5.5.7 (an update to current version is in the works).

[scottwilkerson]

Is this just one user? If so it could be a browser cache issue, I would test a different browser, or if you are an Admin you can go to Admin -> Manage Users and Masquerade as the user to see if you can see the performance graphs.

[IkaIka]

Thanks your your response, @scottwilkerson. I did some more testing and seems like this problem is affecting a specific host rather than a specific user. I can't display the graphs using my own admin account or masquerading as the user who reported this to me. The problem also manifests itself when using incognito mode and a different browser. I tried reconfiguring its host groups and service groups according to a different hosts whose graphs show correctly, but I still can't get it to work. I'd appreciate any suggestions on what might be causing this and how to fix it.

[scottwilkerson]

Can you show a screenshot of the advanced tab for the host so we can see if performance data is being generated by the checks?

Also, I would like to see the permissions of the files in the following location replacing HOST with the hostname

Code: Select all

ls -la /usr/local/nagios/share/perfdata/HOST/

[IkaIka]

Please note, I'm about 90% sure the ls output below pertains to the host in question. Seems like its name in XI was changed at some point but the directory structure remained old. Could this be the reason for the problem?

Code: Select all

drwxrwxr-x    2 nagios nagios    4096 May 20 21:21 .
drwxrwxr-x 3486 nagios nagios  151552 Jun  3 10:10 ..
-rw-rw-r--    1 nagios nagios 1534776 May 20 21:21 _HOST_.rrd
-rw-rw-r--    1 nagios nagios    4069 May 20 21:21 _HOST_.xml
-rw-rw-r--    1 nagios nagios  384960 May 20 21:20 Physical_and_SWAP_Memory.rrd
-rw-rw-r--    1 nagios nagios    2382 May 20 21:20 Physical_and_SWAP_Memory.xml
-rw-rw-r--    1 nagios nagios  384960 May 20 21:21 System_upTime.rrd
-rw-rw-r--    1 nagios nagios    2272 May 20 21:21 System_upTime.xml

Also attaching a screencap of the advanced tab for the host.

[scottwilkerson]

I see the .rrd's haven't updated since May 20, does this date correspond to an upgrade on the system or anything?

What kind of output do you see running these commands?

Code: Select all

tail -50 /usr/local/nagios/var/npcd.log
tail -50 /usr/local/nagios/var/perfdata.log

[IkaIka]

As far as I can tell, this XI instance hasn't been updated in at least 6 months. The reporting user claims the graphs worked as of May 23rd, but it's best to take it with a grain of salt.

[scottwilkerson]

What kind of output do you see running these commands?

Code: Select all

tail -50 /usr/local/nagios/var/npcd.log
tail -50 /usr/local/nagios/var/perfdata.log

[IkaIka]

Code: Select all

tail -50 /usr/local/nagios/var/npcd.log
[10-15-2018 16:54:00] NPCD: WARN: MAX load reached: load 165.420000/50.000000 at i=1
[10-15-2018 16:54:15] NPCD: WARN: MAX load reached: load 129.060000/50.000000 at i=1
[10-15-2018 16:54:30] NPCD: WARN: MAX load reached: load 100.690000/50.000000 at i=1
[10-15-2018 16:54:45] NPCD: WARN: MAX load reached: load 78.700000/50.000000 at i=1
[10-15-2018 16:55:00] NPCD: WARN: MAX load reached: load 61.640000/50.000000 at i=1
[11-08-2018 12:30:51] NPCD: npcd Daemon (0.4.14) started with PID=2139
[11-08-2018 12:30:51] NPCD: Please have a look at 'npcd -V' to get license information
[11-08-2018 12:30:51] NPCD: HINT: load_threshold is enabled - ('50.000000')
[11-30-2018 12:11:48] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-30-2018 12:11:48] NPCD: npcd Daemon (0.4.14) started with PID=20010
[11-30-2018 12:11:48] NPCD: Please have a look at 'npcd -V' to get license information
[11-30-2018 12:11:48] NPCD: HINT: load_threshold is enabled - ('50.000000')
[12-21-2018 13:39:31] NPCD: ERROR: Executed command exits with return code '7'
[12-21-2018 13:39:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1545395762.perfdata.host'
[12-21-2018 13:40:06] NPCD: ERROR: Executed command exits with return code '7'
[12-21-2018 13:40:06] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1545395762.perfdata.service'
[12-21-2018 13:40:21] NPCD: WARN: MAX load reached: load 178.120000/50.000000 at i=0
[12-21-2018 13:40:36] NPCD: WARN: MAX load reached: load 281.800000/50.000000 at i=1
[12-21-2018 13:40:51] NPCD: WARN: MAX load reached: load 220.600000/50.000000 at i=1
[12-21-2018 13:41:06] NPCD: WARN: MAX load reached: load 172.330000/50.000000 at i=1
[12-21-2018 13:41:21] NPCD: WARN: MAX load reached: load 134.670000/50.000000 at i=1
[12-21-2018 13:41:36] NPCD: WARN: MAX load reached: load 105.130000/50.000000 at i=1
[12-21-2018 13:41:51] NPCD: WARN: MAX load reached: load 82.360000/50.000000 at i=1
[12-21-2018 13:42:06] NPCD: WARN: MAX load reached: load 64.350000/50.000000 at i=1
[12-21-2018 13:42:21] NPCD: WARN: MAX load reached: load 50.840000/50.000000 at i=1
[03-20-2019 08:21:30] NPCD: ERROR: Executed command exits with return code '7'
[03-20-2019 08:21:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1553066285.perfdata.host'
[03-20-2019 08:21:57] NPCD: ERROR: Executed command exits with return code '7'
[03-20-2019 08:21:57] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1553066285.perfdata.service'
[03-20-2019 08:22:13] NPCD: WARN: MAX load reached: load 161.370000/50.000000 at i=0
[03-20-2019 08:22:28] NPCD: WARN: MAX load reached: load 160.660000/50.000000 at i=1
[03-20-2019 08:22:43] NPCD: WARN: MAX load reached: load 155.620000/50.000000 at i=1
[03-20-2019 08:22:58] NPCD: WARN: MAX load reached: load 150.150000/50.000000 at i=1
[03-20-2019 08:23:14] NPCD: WARN: MAX load reached: load 146.440000/50.000000 at i=1
[03-20-2019 08:23:29] NPCD: WARN: MAX load reached: load 132.560000/50.000000 at i=1
[03-20-2019 08:23:45] NPCD: WARN: MAX load reached: load 171.160000/50.000000 at i=1
[03-20-2019 08:24:00] NPCD: WARN: MAX load reached: load 282.860000/50.000000 at i=1
[03-20-2019 08:24:15] NPCD: WARN: MAX load reached: load 505.390000/50.000000 at i=1
[03-20-2019 08:24:30] NPCD: WARN: MAX load reached: load 507.520000/50.000000 at i=1
[03-20-2019 08:24:45] NPCD: WARN: MAX load reached: load 445.630000/50.000000 at i=1
[03-20-2019 08:25:00] NPCD: WARN: MAX load reached: load 347.430000/50.000000 at i=1
[03-20-2019 08:25:15] NPCD: WARN: MAX load reached: load 271.200000/50.000000 at i=1
[03-20-2019 08:25:30] NPCD: WARN: MAX load reached: load 211.270000/50.000000 at i=1
[03-20-2019 08:25:45] NPCD: WARN: MAX load reached: load 164.760000/50.000000 at i=1
[03-20-2019 08:26:00] NPCD: WARN: MAX load reached: load 128.480000/50.000000 at i=1
[03-20-2019 08:26:15] NPCD: WARN: MAX load reached: load 100.320000/50.000000 at i=1
[03-20-2019 08:26:30] NPCD: WARN: MAX load reached: load 78.170000/50.000000 at i=1
[03-20-2019 08:26:45] NPCD: WARN: MAX load reached: load 60.990000/50.000000 at i=1
[04-18-2019 08:56:42] NPCD: ERROR: Executed command exits with return code '7'
[04-18-2019 08:56:43] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /var/nagiosramdisk/spool/perfdata//1555570470.perfdata.service'

Code: Select all

tail -50 /usr/local/nagios/var/perfdata.log
2019-06-05 16:18:50 [569] [2] Processing Line 530
2019-06-05 16:18:50 [569] [2] Datatype set to 'SERVICEPERFDATA'
2019-06-05 16:18:50 [569] [1] Found Performance Data for XXX / Disk_Space___ (/=6311MB;40316;45355;0;50395)
2019-06-05 16:18:50 [569] [2] No Custom Template found for check_nrpe (/usr/local/nagios/etc/pnp/check_commands/check_nrpe.cfg)
2019-06-05 16:18:50 [569] [2] Template is check_nrpe.php
2019-06-05 16:18:50 [569] [2] data2rrd called
2019-06-05 16:18:50 [569] [2] RRDs Perl Modules are not installed. Falling back to rrdtool system call.
2019-06-05 16:18:50 [569] [2] /usr/bin/rrdtool update /usr/local/nagios/share/perfdata/XXX/Disk_Space___.rrd 1559744307:6311
2019-06-05 16:18:50 [569] [1] rrdtool update returns 0
2019-06-05 16:18:50 [569] [2] Processing Line 531
2019-06-05 16:18:50 [569] [2] Datatype set to 'SERVICEPERFDATA'
2019-06-05 16:18:50 [569] [1] Found Performance Data for XXX / Alive (rta=56.335ms;3000.000;5000.000;0; pl=0%;80;100;; rtmax=59.583ms;;;; rtmin=52.485ms;;;;)
2019-06-05 16:18:50 [569] [2] No Custom Template found for check-host-alive (/usr/local/nagios/etc/pnp/check_commands/check-host-alive.cfg)
2019-06-05 16:18:50 [569] [2] Template is check-host-alive.php
2019-06-05 16:18:50 [569] [2] No Custom Template found for check-host-alive (/usr/local/nagios/etc/pnp/check_commands/check-host-alive.cfg)
2019-06-05 16:18:50 [569] [2] Template is check-host-alive.php
2019-06-05 16:18:50 [569] [2] No Custom Template found for check-host-alive (/usr/local/nagios/etc/pnp/check_commands/check-host-alive.cfg)
2019-06-05 16:18:50 [569] [2] Template is check-host-alive.php
2019-06-05 16:18:50 [569] [2] No Custom Template found for check-host-alive (/usr/local/nagios/etc/pnp/check_commands/check-host-alive.cfg)
2019-06-05 16:18:50 [569] [2] Template is check-host-alive.php
2019-06-05 16:18:50 [569] [2] data2rrd called
2019-06-05 16:18:50 [569] [2] RRDs Perl Modules are not installed. Falling back to rrdtool system call.
2019-06-05 16:18:50 [569] [2] /usr/bin/rrdtool update /usr/local/nagios/share/perfdata/XXX/Alive.rrd 1559744307:56.335:0:59.583:52.485
2019-06-05 16:18:50 [569] [1] rrdtool update returns 0
2019-06-05 16:18:50 [569] [2] Processing Line 532
2019-06-05 16:18:50 [569] [2] Datatype set to 'SERVICEPERFDATA'
2019-06-05 16:18:50 [569] [1] Found Performance Data for XXX / Physical_Memory_-_RAM ('Physical_Memory'=6041MB;7949;8030;0;8111)
2019-06-05 16:18:50 [569] [2] No Custom Template found for check_snmp_storage_cc (/usr/local/nagios/etc/pnp/check_commands/check_snmp_storage_cc.cfg)
2019-06-05 16:18:50 [569] [2] Template is check_snmp_storage_cc.php
2019-06-05 16:18:50 [569] [2] data2rrd called
2019-06-05 16:18:50 [569] [2] RRDs Perl Modules are not installed. Falling back to rrdtool system call.
2019-06-05 16:18:50 [569] [2] /usr/bin/rrdtool update /usr/local/nagios/share/perfdata/XXX/Physical_Memory_-_RAM.rrd 1559744307:6041
2019-06-05 16:18:50 [569] [1] rrdtool update returns 0
2019-06-05 16:18:50 [569] [2] Processing Line 533
2019-06-05 16:18:50 [569] [2] Datatype set to 'SERVICEPERFDATA'
2019-06-05 16:18:50 [569] [1] Found Performance Data for XXX / eblade-disk_util (occupation_time=0.100000003)
2019-06-05 16:18:50 [569] [2] No Custom Template found for netdata_to_nagios (/usr/local/nagios/etc/pnp/check_commands/netdata_to_nagios.cfg)
2019-06-05 16:18:50 [569] [2] Template is netdata_to_nagios.php
2019-06-05 16:18:50 [569] [2] data2rrd called
2019-06-05 16:18:50 [569] [2] RRDs Perl Modules are not installed. Falling back to rrdtool system call.
2019-06-05 16:18:50 [569] [2] /usr/bin/rrdtool update /usr/local/nagios/share/perfdata/XXX/eblade-disk_util.rrd 1559744307:0.100000003
2019-06-05 16:18:50 [569] [1] rrdtool update returns 0
2019-06-05 16:18:50 [569] [2] Processing Line 534
2019-06-05 16:18:50 [569] [2] Datatype set to 'SERVICEPERFDATA'
2019-06-05 16:18:50 [569] [1] Found Performance Data for XXX / c281-a002 (rta=32.570000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0)
2019-06-05 16:18:50 [569] [2] No Custom Template found for check-spec-host (/usr/local/nagios/etc/pnp/check_commands/check-spec-host.cfg)
2019-06-05 16:18:50 [569] [2] Template is check-spec-host.php
2019-06-05 16:18:50 [569] [2] No Custom Template found for check-spec-host (/usr/local/nagios/etc/pnp/check_commands/check-spec-host.cfg)
2019-06-05 16:18:50 [569] [2] Template is check-spec-host.php
2019-06-05 16:18:50 [569] [2] data2rrd called

Please note that XXX denotes redacted server names. The affected host does not appear in the output.

[scottwilkerson]

So the last log is a pretty short period of time(all within 1 second), can you run the following replacing XXX withthe host name so we can see what's around those messages

Code: Select all

grep -B 5 -A 5 "XXX" /usr/local/nagios/var/perfdata.log