Page 2 of 3
Re: Performance data missing after changing nagios MySQL pas
Posted: Tue Nov 10, 2015 10:38 am
by eloyd
The configuration files you mention have not been modified from their original install:
Code: Select all
67504209 4 -rw-rw-r-- 1 apache nagios 793 Apr 23 2015 ./nagios/etc/pnp/process_perfdata.cfg
67504206 4 -rw-rw-r-- 1 apache nagios 3090 Apr 23 2015 ./nagios/etc/pnp/npcd.cfg
Both of them have log_level type entries set to zero. So I don't think that's the problem, since performance data has continued to be collected up until Oct 9 despite these settings. I did get them to reboot the box and tail the log files. Pertinent bits follow:
Code: Select all
tail -100 /usr/local/nagios/var/npcd.log
<snip>
[11-10-2015 10:10:07] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-10-2015 10:10:29] NPCD: npcd Daemon (0.4.14) started with PID=1588
[11-10-2015 10:10:29] NPCD: Please have a look at 'npcd -V' to get license information
[11-10-2015 10:10:29] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-10-2015 10:11:14] NPCD: WARN: MAX load reached: load 11.790000/10.000000 at i=0
[11-10-2015 10:11:29] NPCD: WARN: MAX load reached: load 10.980000/10.000000 at i=1
Code: Select all
tail -100 /usr/local/nagios/var/perfdata.log
<snip>
2015-07-08 00:01:18 [5851] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1436328024.perfdata.service-PID-5851 deleted
2015-07-08 00:01:18 [5852] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1436328025.perfdata.service-PID-5852 deleted
2015-07-08 00:01:18 [5848] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1436327995.perfdata.host-PID-5848 deleted
2015-07-08 00:01:18 [5849] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1436327995.perfdata.service-PID-5849 deleted
2015-07-08 00:01:18 [5849] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5849] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-08 00:01:18 [5848] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5848] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-08 00:01:18 [5852] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5852] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-08 00:01:18 [5851] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5851] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-08 00:01:18 [5850] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5850] [0] *** process_perfdata.pl terminated on signal ALRM
Note that perfdata is from JULY, which is way before October, and so I'm not sure that there's any value in that log, either. And a reboot did not seem to create any positive results, as things are still showing "no data" in the performance graphs. I've had them up the load_threshold to 100.0 (instead of the default 10.0) even though use_load_threshold is commented out and therefore should default to 0 (though it seems like it does not). Will advise if this makes a difference, but the load was 2.0 when I had them check.
Re: Performance data missing after changing nagios MySQL pas
Posted: Tue Nov 10, 2015 4:35 pm
by ssax
Are you seeing anything in /var/log/cron (can you see perfdataproc being run? any errors?) or in /usr/local/nagiosxi/var/perfdataproc.log that give any indication?
Re: Performance data missing after changing nagios MySQL pas
Posted: Tue Nov 17, 2015 2:21 pm
by eloyd
Sorry, I just saw this message today. I now have full access to the box.
Code: Select all
ls -l /usr/local/nagiosxi/var/perfdataproc.log
-rw-r--r-- 1 nagios nagios 81 Nov 17 14:20 /usr/local/nagiosxi/var/perfdataproc.log
cat /usr/local/nagiosxi/var/perfdataproc.log
Outbound data DISABLED Tue, 17 Nov 2015 14:19:01 -0500
DONE. Processed 0 files.
and
Code: Select all
# grep perf /var/log/cron | tail -10
Nov 17 14:12:02 nagiosxipoc CROND[30360]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:13:01 nagiosxipoc CROND[31232]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:14:01 nagiosxipoc CROND[32099]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:15:01 nagiosxipoc CROND[602]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:16:01 nagiosxipoc CROND[1581]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:17:01 nagiosxipoc CROND[2470]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:18:01 nagiosxipoc CROND[3357]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:19:01 nagiosxipoc CROND[4239]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:20:01 nagiosxipoc CROND[5189]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:21:01 nagiosxipoc CROND[6123]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
I really need to figure out what happened here, as this is stopping this system from going live.
Re: Performance data missing after changing nagios MySQL pas
Posted: Tue Nov 17, 2015 2:33 pm
by eloyd
As an aside, I tore apart /usr/local/nagiosxi/cron/perfdataproc.php briefly to do some debugging and if I execute it manually, I get this:
Code: Select all
Outbound data DISABLED Tue, 17 Nov 2015 14:32:50 -0500
Might that be stopping it from parsing data?
Re: Performance data missing after changing nagios MySQL pas
Posted: Wed Nov 18, 2015 11:45 am
by tgriep
Can you run the following and post the output?
Code: Select all
ls -la /usr/local/nagios/var/spool
Also, run the following tail for a few minutes on that system and post the output here.
Code: Select all
tail -f /usr/local/nagiosxi/var/perfdataproc.log
Re: Performance data missing after changing nagios MySQL pas
Posted: Wed Nov 18, 2015 12:35 pm
by eloyd
Looks to me like it's not even trying.
Code: Select all
# ls -la /usr/local/nagios/var/spool
total 4
drwxr-xr-x 5 nagios nagios 52 Apr 23 2015 .
drwxrwxr-x 6 nagios nagios 4096 Nov 18 12:31 ..
drwxrwsr-x 2 nagios nagcmd 6 Nov 17 16:37 checkresults
drwxr-xr-x 2 nagios nagios 6 Nov 18 12:31 perfdata
drwxr-xr-x 2 nagios nagios 6 Nov 18 12:31 xidpe
Even though perfdata gets files in it from time to time:
Code: Select all
ls -lsa
total 28
0 drwxr-xr-x 2 nagios nagios 71 Nov 18 12:34 .
0 drwxr-xr-x 5 nagios nagios 52 Apr 23 2015 ..
8 -rw-r--r-- 1 nagios nagios 4907 Nov 18 12:34 1447868081.perfdata.host
20 -rw-r--r-- 1 nagios nagios 17762 Nov 18 12:34 1447868081.perfdata.servic
Code: Select all
]# tail -f /usr/local/nagiosxi/var/perfdataproc.log
Outbound data DISABLED Wed, 18 Nov 2015 12:32:01 -0500
DONE. Processed 0 files.
tail: /usr/local/nagiosxi/var/perfdataproc.log: file truncated
Outbound data DISABLED Wed, 18 Nov 2015 12:33:01 -0500
DONE. Processed 0 files.
tail: /usr/local/nagiosxi/var/perfdataproc.log: file truncated
Outbound data DISABLED Wed, 18 Nov 2015 12:34:01 -0500
DONE. Processed 0 files.
Re: Performance data missing after changing nagios MySQL pas
Posted: Wed Nov 18, 2015 5:44 pm
by tgriep
Can you PM the System Profile?
Re: Performance data missing after changing nagios MySQL pas
Posted: Thu Nov 19, 2015 5:41 pm
by tgriep
Thanks for the PM. I am going to need the profile.zip file.
To send us your system profile
Click "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the file and email it back to us.
Re: Performance data missing after changing nagios MySQL pas
Posted: Mon Nov 23, 2015 10:27 am
by tgriep
Can you edit the /usr/local/nagios/etc/pnp/process_perfdata.cfg file
Change the LOG_LEVEL from
to
Save it and let the system run for around 30 minutes and then run the following and post the output here?
Code: Select all
tail -200 /usr/local/nagios/var/perfdata.log
Re: Performance data missing after changing nagios MySQL pas
Posted: Mon Nov 23, 2015 10:38 am
by eloyd
Is 3 minutes long enough?
Code: Select all
2015-11-23 10:36:52 [12967] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:36:52 [12967] [1] 137 lines processed
2015-11-23 10:36:52 [12967] [1] /usr/local/nagios/var/spool/perfdata//1448293003.perfdata.service-PID-12967 deleted
2015-11-23 10:36:52 [12967] [1] PNP exiting (runtime 0.001734s) ...
2015-11-23 10:36:52 [12968] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:36:52 [12968] [1] 26 lines processed
2015-11-23 10:36:52 [12968] [1] /usr/local/nagios/var/spool/perfdata//1448293003.perfdata.host-PID-12968 deleted
2015-11-23 10:36:52 [12968] [1] PNP exiting (runtime 0.000417s) ...
2015-11-23 10:37:07 [13259] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:07 [13260] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:07 [13260] [1] 27 lines processed
2015-11-23 10:37:07 [13260] [1] /usr/local/nagios/var/spool/perfdata//1448293019.perfdata.host-PID-13260 deleted
2015-11-23 10:37:07 [13259] [1] 122 lines processed
2015-11-23 10:37:07 [13259] [1] /usr/local/nagios/var/spool/perfdata//1448293019.perfdata.service-PID-13259 deleted
2015-11-23 10:37:07 [13259] [1] PNP exiting (runtime 0.0031s) ...
2015-11-23 10:37:07 [13260] [1] PNP exiting (runtime 0.001203s) ...
2015-11-23 10:37:37 [13630] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:37 [13629] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:37 [13628] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:37 [13631] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:37 [13629] [1] 24 lines processed
2015-11-23 10:37:37 [13628] [1] 120 lines processed
2015-11-23 10:37:37 [13630] [1] 116 lines processed
2015-11-23 10:37:37 [13629] [1] /usr/local/nagios/var/spool/perfdata//1448293049.perfdata.host-PID-13629 deleted
2015-11-23 10:37:37 [13629] [1] PNP exiting (runtime 0.005579s) ...
2015-11-23 10:37:37 [13630] [1] /usr/local/nagios/var/spool/perfdata//1448293033.perfdata.service-PID-13630 deleted
2015-11-23 10:37:37 [13631] [1] 29 lines processed
2015-11-23 10:37:37 [13631] [1] /usr/local/nagios/var/spool/perfdata//1448293033.perfdata.host-PID-13631 deleted
2015-11-23 10:37:37 [13631] [1] PNP exiting (runtime 0.003732s) ...
2015-11-23 10:37:37 [13630] [1] PNP exiting (runtime 0.009288s) ...
2015-11-23 10:37:37 [13628] [1] /usr/local/nagios/var/spool/perfdata//1448293049.perfdata.service-PID-13628 deleted
2015-11-23 10:37:37 [13628] [1] PNP exiting (runtime 0.007977s) ...
2015-11-23 10:37:53 [13846] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:53 [13847] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:53 [13847] [1] 24 lines processed
2015-11-23 10:37:53 [13847] [1] /usr/local/nagios/var/spool/perfdata//1448293063.perfdata.host-PID-13847 deleted
2015-11-23 10:37:53 [13847] [1] PNP exiting (runtime 0.001084s) ...
2015-11-23 10:37:53 [13846] [1] 124 lines processed
2015-11-23 10:37:53 [13846] [1] /usr/local/nagios/var/spool/perfdata//1448293063.perfdata.service-PID-13846 deleted
2015-11-23 10:37:53 [13846] [1] PNP exiting (runtime 0.00287s) ...
2015-11-23 10:38:08 [14147] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:38:08 [14146] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:38:08 [14147] [1] 32 lines processed
2015-11-23 10:38:08 [14146] [1] 108 lines processed
2015-11-23 10:38:08 [14147] [1] /usr/local/nagios/var/spool/perfdata//1448293078.perfdata.host-PID-14147 deleted
2015-11-23 10:38:08 [14146] [1] /usr/local/nagios/var/spool/perfdata//1448293078.perfdata.service-PID-14146 deleted
2015-11-23 10:38:08 [14146] [1] PNP exiting (runtime 0.002738s) ...
2015-11-23 10:38:08 [14147] [1] PNP exiting (runtime 0.002059s) ...