Page 1 of 1
Restored backup to vmware centos6, no graphs
Posted: Wed Mar 06, 2013 5:17 pm
by r.jaynes
Hello,
We did a backup from the NagiosXI 2012R1.6/Centos5.9 OVF (32bit) and restored it to a new NagiosXI 2012R1.6/Centos6.3 OVF (64 bit) today. After working out a few bugs and getting custom plugins back working, the only issue we are experiencing are the graphs. I have restored previous graph data from the 32bit server following this guide (
http://support.nagios.com/wiki/index.ph ... Install.3F), and I can see previous data fine. The issue is all new data. Currently from the time I took our server down to now every graph is blank (roughly from 12PM forward).
I found this thread:
http://support.nagios.com/forum/viewtop ... aph#p44897 , and have followed some of the troubleshooting steps through there, but still no dice.
I can confirm that we are seeing "Performance Data" when I go to the advanced tab for a particular service.
I will attach any logs/output/etc that is required to troubleshoot this. Thank you!
Re: Restored backup to vmware centos6, no graphs
Posted: Wed Mar 06, 2013 5:20 pm
by r.jaynes
/usr/local/nagios/var/npcd.log
Code: Select all
[root@monitor ~]# tail -50 /usr/local/nagios/var/npcd.log
[03-06-2013 16:17:14] NPCD: Regular File: 1362608223.perfdata.host
[03-06-2013 16:17:14] NPCD: A thread was started on thread_counter = 0
[03-06-2013 16:17:14] NPCD: DEBUG: load 1.850000/15.000000
[03-06-2013 16:17:14] NPCD: ThreadCounter 1/5 File is 1362608223.perfdata.service
[03-06-2013 16:17:14] NPCD: Regular File: 1362608223.perfdata.service
[03-06-2013 16:17:14] NPCD: A thread was started on thread_counter = 1
[03-06-2013 16:17:14] NPCD: Have to wait: Filecounter = 2 - thread_counter = 2
[03-06-2013 16:17:14] NPCD: Processing file 1362608223.perfdata.service with ID 140126960289536 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1362608223.perfdata.service
[03-06-2013 16:17:14] NPCD: Processing file '1362608223.perfdata.service'
[03-06-2013 16:17:14] NPCD: Processing file 1362608223.perfdata.host with ID 140126970779392 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1362608223.perfdata.host
[03-06-2013 16:17:14] NPCD: Processing file '1362608223.perfdata.host'
[03-06-2013 16:17:14] NPCD: No more files to process... waiting for 15 seconds
[03-06-2013 16:17:29] NPCD: Found 4 files in /usr/local/nagios/var/spool/perfdata/
[03-06-2013 16:17:29] NPCD: DEBUG: load 1.440000/15.000000
[03-06-2013 16:17:29] NPCD: ThreadCounter 0/5 File is .
[03-06-2013 16:17:29] NPCD: DEBUG: load 1.440000/15.000000
[03-06-2013 16:17:29] NPCD: ThreadCounter 0/5 File is ..
[03-06-2013 16:17:29] NPCD: DEBUG: load 1.440000/15.000000
[03-06-2013 16:17:29] NPCD: ThreadCounter 0/5 File is 1362608238.perfdata.host
[03-06-2013 16:17:29] NPCD: Regular File: 1362608238.perfdata.host
[03-06-2013 16:17:29] NPCD: A thread was started on thread_counter = 0
[03-06-2013 16:17:29] NPCD: DEBUG: load 1.440000/15.000000
[03-06-2013 16:17:29] NPCD: ThreadCounter 1/5 File is 1362608238.perfdata.service
[03-06-2013 16:17:29] NPCD: Regular File: 1362608238.perfdata.service
[03-06-2013 16:17:29] NPCD: A thread was started on thread_counter = 1
[03-06-2013 16:17:29] NPCD: Have to wait: Filecounter = 2 - thread_counter = 2
[03-06-2013 16:17:29] NPCD: Processing file 1362608238.perfdata.service with ID 140126960289536 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1362608238.perfdata.service
[03-06-2013 16:17:29] NPCD: Processing file '1362608238.perfdata.service'
[03-06-2013 16:17:29] NPCD: Processing file 1362608238.perfdata.host with ID 140126970779392 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1362608238.perfdata.host
[03-06-2013 16:17:29] NPCD: Processing file '1362608238.perfdata.host'
[03-06-2013 16:17:29] NPCD: No more files to process... waiting for 15 seconds
[03-06-2013 16:17:44] NPCD: Found 4 files in /usr/local/nagios/var/spool/perfdata/
[03-06-2013 16:17:44] NPCD: DEBUG: load 1.270000/15.000000
[03-06-2013 16:17:44] NPCD: ThreadCounter 0/5 File is .
[03-06-2013 16:17:44] NPCD: DEBUG: load 1.270000/15.000000
[03-06-2013 16:17:44] NPCD: ThreadCounter 0/5 File is ..
[03-06-2013 16:17:44] NPCD: DEBUG: load 1.270000/15.000000
[03-06-2013 16:17:44] NPCD: ThreadCounter 0/5 File is 1362608253.perfdata.host
[03-06-2013 16:17:44] NPCD: Regular File: 1362608253.perfdata.host
[03-06-2013 16:17:44] NPCD: A thread was started on thread_counter = 0
[03-06-2013 16:17:44] NPCD: DEBUG: load 1.270000/15.000000
[03-06-2013 16:17:44] NPCD: ThreadCounter 1/5 File is 1362608253.perfdata.service
[03-06-2013 16:17:44] NPCD: Regular File: 1362608253.perfdata.service
[03-06-2013 16:17:44] NPCD: A thread was started on thread_counter = 1
[03-06-2013 16:17:44] NPCD: Have to wait: Filecounter = 2 - thread_counter = 2
[03-06-2013 16:17:44] NPCD: Processing file 1362608253.perfdata.service with ID 140126960289536 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1362608253.perfdata.service
[03-06-2013 16:17:44] NPCD: Processing file '1362608253.perfdata.service'
[03-06-2013 16:17:44] NPCD: Processing file 1362608253.perfdata.host with ID 140126970779392 - going to exec /usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1362608253.perfdata.host
[03-06-2013 16:17:44] NPCD: Processing file '1362608253.perfdata.host'
[03-06-2013 16:17:44] NPCD: No more files to process... waiting for 15 seconds
Re: Restored backup to vmware centos6, no graphs
Posted: Wed Mar 06, 2013 5:21 pm
by r.jaynes
/usr/local/nagios/var/perfdata.log
I haven't seen any new entries in this log for some time.
Code: Select all
[root@monitor ~]# tail -50 /usr/local/nagios/var/perfdata.log
2013-03-06 10:20:21 [26180] [0] *** Timeout while processing Host: "MG-ME-873-WPWAN" Service: "Ping"
2013-03-06 10:20:21 [26180] [0] *** process_perfdata.pl terminated on signal ALRM
2013-03-06 10:24:09 [27706] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-03-06 10:24:10 [27706] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-03-06 10:24:10 [27706] [0] *** TIMEOUT: Please check your npcd.cfg
2013-03-06 10:24:10 [27706] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1362586864.perfdata.service-PID-27706 deleted
2013-03-06 10:24:10 [27706] [0] *** Timeout while processing Host: "OMX5A_Gulfport" Service: "Ping"
2013-03-06 10:24:10 [27706] [0] *** process_perfdata.pl terminated on signal ALRM
2013-03-06 10:24:17 [27736] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-03-06 10:24:17 [27736] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-03-06 10:24:17 [27736] [0] *** TIMEOUT: Please check your npcd.cfg
2013-03-06 10:24:17 [27736] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1362586904.perfdata.service-PID-27736 deleted
2013-03-06 10:24:17 [27736] [0] *** Timeout while processing Host: "APC-RR102.10-A12" Service: "Total_Load"
2013-03-06 10:24:17 [27736] [0] *** process_perfdata.pl terminated on signal ALRM
2013-03-06 10:24:17 [27741] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-03-06 10:24:17 [27741] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-03-06 10:24:17 [27741] [0] *** TIMEOUT: Please check your npcd.cfg
2013-03-06 10:24:17 [27741] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1362586919.perfdata.service-PID-27741 deleted
2013-03-06 10:24:17 [27741] [0] *** Timeout while processing Host: "cp2.megagate.com" Service: "Swap_Usage"
2013-03-06 10:24:17 [27741] [0] *** process_perfdata.pl terminated on signal ALRM
2013-03-06 10:24:17 [27740] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-03-06 10:24:17 [27740] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-03-06 10:24:17 [27740] [0] *** TIMEOUT: Please check your npcd.cfg
2013-03-06 10:24:17 [27740] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1362586919.perfdata.host-PID-27740 deleted
2013-03-06 10:24:17 [27740] [0] *** Timeout while processing Host: "APC-RR102.11-A14" Service: "_HOST_"
2013-03-06 10:24:17 [27740] [0] *** process_perfdata.pl terminated on signal ALRM
2013-03-06 10:24:24 [27788] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-03-06 10:24:24 [27788] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-03-06 10:24:24 [27788] [0] *** TIMEOUT: Please check your npcd.cfg
2013-03-06 10:24:24 [27788] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1362586964.perfdata.service-PID-27788 deleted
2013-03-06 10:24:24 [27788] [0] *** Timeout while processing Host: "MegaServer" Service: "Memory_Usage"
2013-03-06 10:24:24 [27788] [0] *** process_perfdata.pl terminated on signal ALRM
2013-03-06 10:24:30 [27828] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-03-06 10:24:30 [27828] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-03-06 10:24:30 [27828] [0] *** TIMEOUT: Please check your npcd.cfg
2013-03-06 10:24:30 [27828] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1362586979.perfdata.service-PID-27828 deleted
2013-03-06 10:24:30 [27828] [0] *** Timeout while processing Host: "SQL3" Service: "Drive_C__Disk_Usage"
2013-03-06 10:24:30 [27828] [0] *** process_perfdata.pl terminated on signal ALRM
2013-03-06 10:26:00 [28691] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-03-06 10:26:00 [28691] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-03-06 10:26:00 [28691] [0] *** TIMEOUT: Please check your npcd.cfg
2013-03-06 10:26:00 [28691] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1362587099.perfdata.service-PID-28691 deleted
2013-03-06 10:26:00 [28691] [0] *** Timeout while processing Host: "Site132SQL" Service: "Drive_E__SQL_Data_Disk_Usage"
2013-03-06 10:26:00 [28691] [0] *** process_perfdata.pl terminated on signal ALRM
2013-03-06 10:26:01 [28695] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-03-06 10:26:01 [28695] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-03-06 10:26:01 [28695] [0] *** TIMEOUT: Please check your npcd.cfg
2013-03-06 10:26:01 [28695] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1362587115.perfdata.service-PID-28695 deleted
2013-03-06 10:26:01 [28695] [0] *** Timeout while processing Host: "janus.megagate.com" Service: "MySQL_Long_Running_Processes"
2013-03-06 10:26:01 [28695] [0] *** process_perfdata.pl terminated on signal ALRM
Re: Restored backup to vmware centos6, no graphs
Posted: Wed Mar 06, 2013 5:25 pm
by abrist
Verify that 1 and only 1 npcd parent process is running, and that it actually is running:
Code: Select all
ps -aef | grep npcd
service npcd stop
killall npcd
service npcd start
Re: Restored backup to vmware centos6, no graphs
Posted: Wed Mar 06, 2013 5:28 pm
by r.jaynes
Looks like only one was running. I have killed and restarted it per your instructions.
ps -aef | grep npcd
Code: Select all
[root@monitor ~]# ps -aef | grep npcd
nagios 1776 1 0 15:52 ? 00:00:00 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
root 19122 2267 0 16:25 pts/0 00:00:00 grep npcd
after service npcd stop/killall npcd:
Code: Select all
[root@monitor ~]# ps -aef | grep npcd
nagios 19766 1 0 16:26 ? 00:00:00 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
root 19811 2267 0 16:26 pts/0 00:00:00 grep npcd
abrist wrote:Verify that 1 and only 1 npcd parent process is running, and that it actually is running:
Code: Select all
ps -aef | grep npcd
service npcd stop
killall npcd
service npcd start
Re: Restored backup to vmware centos6, no graphs
Posted: Wed Mar 06, 2013 5:30 pm
by lmiltchev
Also, run the following commands and show us the output:
Code: Select all
ll /usr/local/nagios/var/spool/xidpe/|wc -l
ll /usr/local/nagios/var/spool/perfdata/|wc -l
Re: Restored backup to vmware centos6, no graphs
Posted: Wed Mar 06, 2013 5:33 pm
by r.jaynes
ll /usr/local/nagios/var/spool/xidpe/|wc -l
Code: Select all
[root@monitor ~]# ll /usr/local/nagios/var/spool/xidpe/|wc -l
1
ll /usr/local/nagios/var/spool/perfdata/|wc -l
Code: Select all
[root@monitor ~]# ll /usr/local/nagios/var/spool/perfdata/|wc -l
3
lmiltchev wrote:Also, run the following commands and show us the output:
Code: Select all
ll /usr/local/nagios/var/spool/xidpe/|wc -l
ll /usr/local/nagios/var/spool/perfdata/|wc -l
Re: Restored backup to vmware centos6, no graphs
Posted: Wed Mar 06, 2013 5:54 pm
by r.jaynes
Side note - I have rebooted a couple of times during all of this.
Re: Restored backup to vmware centos6, no graphs
Posted: Thu Mar 07, 2013 9:26 am
by r.jaynes
I don't know why, but the graphs started worked last night around ~9:35PM. Everything is reporting correctly as far as I can tell.
Re: Restored backup to vmware centos6, no graphs
Posted: Thu Mar 07, 2013 10:59 am
by abrist
Let us know if the problem recurs.