No performance graphs on installation. Data is being sent.

[dflick]

NagiosXI 2012R2.8 installed via VMware image
/usr/local/nagios/var/perfdata.log does not exist.
If I look at the host on the advanced tab I see data collected:
Performance Data: rta=8.995ms;3000.000;5000.000;0; pl=0%;80;100;;

All of my hosts have data collected but none have graphs. I have hit a dead end and don't know where to go next.

[slansing]

What is the output of the following:

Code: Select all

tail -50 /usr/local/nagios/var/npcd.log

Code: Select all

ls -1 /usr/local/nagios/var/spool/checkresults/ | wc -l
ls -1 /usr/local/nagios/var/spool/perfdata/ | wc -l
ls -1 /usr/local/nagios/var/spool/xidpe | wc -l

[dflick]

[root@caecd-buc-xi ~]# tail -50 /usr/local/nagios/var/npcd.log
[01-03-2014 22:25:56] NPCD: npcd Daemon (0.4.14) started with PID=1495
[01-03-2014 22:25:56] NPCD: Please have a look at 'npcd -V' to get license information
[01-03-2014 22:25:56] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-03-2014 22:40:15] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-03-2014 23:03:54] NPCD: npcd Daemon (0.4.14) started with PID=1494
[01-03-2014 23:03:54] NPCD: Please have a look at 'npcd -V' to get license information
[01-03-2014 23:03:54] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-03-2014 23:14:28] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-03-2014 23:14:55] NPCD: npcd Daemon (0.4.14) started with PID=1480
[01-03-2014 23:14:55] NPCD: Please have a look at 'npcd -V' to get license information
[01-03-2014 23:14:55] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-04-2014 13:20:40] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-04-2014 13:38:12] NPCD: npcd Daemon (0.4.14) started with PID=1481
[01-04-2014 13:38:12] NPCD: Please have a look at 'npcd -V' to get license information
[01-04-2014 13:38:12] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-04-2014 13:55:47] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-04-2014 13:57:48] NPCD: npcd Daemon (0.4.14) started with PID=1535
[01-04-2014 13:57:48] NPCD: Please have a look at 'npcd -V' to get license information
[01-04-2014 13:57:48] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-04-2014 14:02:53] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-04-2014 14:03:14] NPCD: npcd Daemon (0.4.14) started with PID=1398
[01-04-2014 14:03:14] NPCD: Please have a look at 'npcd -V' to get license information
[01-04-2014 14:03:14] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-04-2014 14:17:54] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-04-2014 14:18:39] NPCD: npcd Daemon (0.4.14) started with PID=1495
[01-04-2014 14:18:39] NPCD: Please have a look at 'npcd -V' to get license information
[01-04-2014 14:18:39] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-04-2014 14:23:13] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-09-2014 12:30:48] NPCD: npcd Daemon (0.4.14) started with PID=1494
[01-09-2014 12:30:48] NPCD: Please have a look at 'npcd -V' to get license information
[01-09-2014 12:30:48] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-09-2014 12:42:52] NPCD: npcd Daemon (0.4.14) started with PID=1392
[01-09-2014 12:42:52] NPCD: Please have a look at 'npcd -V' to get license information
[01-09-2014 12:42:52] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-13-2014 21:03:38] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-13-2014 21:04:07] NPCD: npcd Daemon (0.4.14) started with PID=1467
[01-13-2014 21:04:07] NPCD: Please have a look at 'npcd -V' to get license information
[01-13-2014 21:04:07] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-13-2014 21:22:13] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-13-2014 21:22:37] NPCD: npcd Daemon (0.4.14) started with PID=1470
[01-13-2014 21:22:37] NPCD: Please have a look at 'npcd -V' to get license information
[01-13-2014 21:22:37] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-16-2014 21:50:21] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-16-2014 21:50:21] NPCD: npcd Daemon (0.4.14) started with PID=25567
[01-16-2014 21:50:21] NPCD: Please have a look at 'npcd -V' to get license information
[01-16-2014 21:50:21] NPCD: HINT: load_threshold is enabled - ('20.000000')
[01-16-2014 21:55:34] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-16-2014 21:56:00] NPCD: npcd Daemon (0.4.14) started with PID=1582
[01-16-2014 21:56:00] NPCD: Please have a look at 'npcd -V' to get license information
[01-16-2014 21:56:00] NPCD: HINT: load_threshold is enabled - ('20.000000')

[root@caecd-buc-xi ~]# ls -1 /usr/local/nagios/var/spool/checkresults/ | wc -l
6
[root@caecd-buc-xi ~]# ls -1 /usr/local/nagios/var/spool/perfdata/ | wc -l
0
[root@caecd-buc-xi ~]# ls -1 /usr/local/nagios/var/spool/xidpe | wc -l
148724

[slansing]

What is the current load on your XI server? You can view this with:

Code: Select all

TOP

It looks like you might be hitting the NPCD max load threshold, or your were doing so recently. To increase this threshold edit:

Code: Select all

/usr/local/nagios/etc/pnp/npcd.cfg

And change:

Code: Select all

load_threshold = 10.0

To:

Code: Select all

load_threshold = 20.0

Save, and quit the file, then stop npcd:

Code: Select all

service npcd stop

Now, this next step will require that you remove the trapped performance data from the XIDPE directory, once you do this you won't be able to recover it, but it is really of no use to you now anyways as it cannot be processed by Nagios XI and it is clogging the pipe.:

Code: Select all

find /usr/local/nagios/var/spool/xidpe/ . -type d -print -delete

Start NPCD again:

Code: Select all

service npcd start

And after about 10 or so minutes run these commands again and share the output, you should start seeing graphs if there are not a large number of files piling up "100+ usually.":

Code: Select all

ls -1 /usr/local/nagios/var/spool/checkresults/ | wc -l
ls -1 /usr/local/nagios/var/spool/perfdata/ | wc -l
ls -1 /usr/local/nagios/var/spool/xidpe | wc -l

[dflick]

Still nothing but I got an error on that find command.

[root@caecd-buc-xi ~]# top
top - 10:04:50 up 5 days, 12:09, 2 users, load average: 0.44, 0.79, 1.12
Tasks: 134 total, 2 running, 132 sleeping, 0 stopped, 0 zombie
Cpu(s): 47.3%us, 3.3%sy, 0.0%ni, 47.3%id, 1.7%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 3923756k total, 1436100k used, 2487656k free, 206248k buffers
Swap: 2064376k total, 0k used, 2064376k free, 492712k cached

I already set the load threshold to 20 per another post and it unfortunately did not help. I have not seen any "threshold reached" messages in the logs.


[root@caecd-buc-xi ~]# find /usr/local/nagios/var/spool/xidpe . -type d -print -delete
/usr/local/nagios/var/spool/xidpe
find: cannot delete `/usr/local/nagios/var/spool/xidpe': Directory not empty
./.pki/nssdb
./.pki
./scripts
find: cannot delete `./scripts': Directory not empty

[root@caecd-buc-xi ~]# ls -1 /usr/local/nagios/var/spool/checkresults/ | wc -l
16
You have new mail in /var/spool/mail/root
[root@caecd-buc-xi ~]# ls -1 /usr/local/nagios/var/spool/perfdata/ | wc -l
0
[root@caecd-buc-xi ~]# ls -1 /usr/local/nagios/var/spool/xidpe | wc -l
148994

[slansing]

Please run:

Code: Select all

ls /usr/local/nagios/var/spool/xidpe | wc -l

service npcd stop

cd /usr/local/nagios/var/spool/xidpe

find . -type f -delete

service npcd start

After a few minutes:

Code: Select all

ls /usr/local/nagios/var/spool/xidpe | wc -l

Also, is cron running?

Code: Select all

service crond status

If it is not:

Code: Select all

service crond start

[dflick]

Completed your request. Here is the needed output.

[root@caecd-buc-xi ~]# ls /usr/local/nagios/var/spool/xidpe | wc -l
149458
[root@caecd-buc-xi xidpe]# service crond status
crond (pid 1571) is running...
[root@caecd-buc-xi xidpe]# ls /usr/local/nagios/var/spool/xidpe | wc -l
2

[dflick]

That last set of commands seems to have fixed it! I am getting the graphs now. I will wait to confirm that data is coming into those. Can you tell me what that last set of commands did to fix it?

[slansing]

It looks like at some point you must have hit the load threshold on the system, this likely caused the scraped performance data to pile up in XIDPE. The reason I had you check cron is because cron is responsible for moving the files from the xidpe directory to the perfdata directory where perfdataproc processes the data, and dumps it to the RRD's that your graphs on the front end pull their data from.

[dflick]

Thanks for the info and the help to resolve this issue!