Performance data missing after changing nagios MySQL pasword

[eloyd]

The configuration files you mention have not been modified from their original install:

Code: Select all

67504209    4 -rw-rw-r--   1 apache   nagios        793 Apr 23  2015 ./nagios/etc/pnp/process_perfdata.cfg
67504206    4 -rw-rw-r--   1 apache   nagios       3090 Apr 23  2015 ./nagios/etc/pnp/npcd.cfg

Both of them have log_level type entries set to zero. So I don't think that's the problem, since performance data has continued to be collected up until Oct 9 despite these settings. I did get them to reboot the box and tail the log files. Pertinent bits follow:

Code: Select all

tail -100 /usr/local/nagios/var/npcd.log
<snip>
[11-10-2015 10:10:07] NPCD: Caught Termination Signal - Hasta la vista... baby
[11-10-2015 10:10:29] NPCD: npcd Daemon (0.4.14) started with PID=1588
[11-10-2015 10:10:29] NPCD: Please have a look at 'npcd -V' to get license information
[11-10-2015 10:10:29] NPCD: HINT: load_threshold is enabled - ('10.000000')
[11-10-2015 10:11:14] NPCD: WARN: MAX load reached: load 11.790000/10.000000 at i=0
[11-10-2015 10:11:29] NPCD: WARN: MAX load reached: load 10.980000/10.000000 at i=1

Code: Select all

tail -100 /usr/local/nagios/var/perfdata.log
<snip>
2015-07-08 00:01:18 [5851] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1436328024.perfdata.service-PID-5851 deleted
2015-07-08 00:01:18 [5852] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1436328025.perfdata.service-PID-5852 deleted
2015-07-08 00:01:18 [5848] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1436327995.perfdata.host-PID-5848 deleted
2015-07-08 00:01:18 [5849] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1436327995.perfdata.service-PID-5849 deleted
2015-07-08 00:01:18 [5849] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5849] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-08 00:01:18 [5848] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5848] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-08 00:01:18 [5852] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5852] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-08 00:01:18 [5851] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5851] [0] *** process_perfdata.pl terminated on signal ALRM
2015-07-08 00:01:18 [5850] [0] *** Timeout while processing Host: "" Service: ""
2015-07-08 00:01:18 [5850] [0] *** process_perfdata.pl terminated on signal ALRM

Note that perfdata is from JULY, which is way before October, and so I'm not sure that there's any value in that log, either. And a reboot did not seem to create any positive results, as things are still showing "no data" in the performance graphs. I've had them up the load_threshold to 100.0 (instead of the default 10.0) even though use_load_threshold is commented out and therefore should default to 0 (though it seems like it does not). Will advise if this makes a difference, but the load was 2.0 when I had them check.

[ssax]

Are you seeing anything in /var/log/cron (can you see perfdataproc being run? any errors?) or in /usr/local/nagiosxi/var/perfdataproc.log that give any indication?

[eloyd]

Sorry, I just saw this message today. I now have full access to the box.

Code: Select all

ls -l /usr/local/nagiosxi/var/perfdataproc.log
-rw-r--r-- 1 nagios nagios 81 Nov 17 14:20 /usr/local/nagiosxi/var/perfdataproc.log

cat /usr/local/nagiosxi/var/perfdataproc.log
Outbound data DISABLED Tue, 17 Nov 2015 14:19:01 -0500

DONE. Processed 0 files.

and

Code: Select all

# grep perf /var/log/cron | tail -10
Nov 17 14:12:02 nagiosxipoc CROND[30360]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:13:01 nagiosxipoc CROND[31232]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:14:01 nagiosxipoc CROND[32099]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:15:01 nagiosxipoc CROND[602]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:16:01 nagiosxipoc CROND[1581]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:17:01 nagiosxipoc CROND[2470]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:18:01 nagiosxipoc CROND[3357]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:19:01 nagiosxipoc CROND[4239]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:20:01 nagiosxipoc CROND[5189]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Nov 17 14:21:01 nagiosxipoc CROND[6123]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1)

I really need to figure out what happened here, as this is stopping this system from going live.

[eloyd]

As an aside, I tore apart /usr/local/nagiosxi/cron/perfdataproc.php briefly to do some debugging and if I execute it manually, I get this:

Code: Select all

Outbound data DISABLED Tue, 17 Nov 2015 14:32:50 -0500

Might that be stopping it from parsing data?

[tgriep]

Can you run the following and post the output?

Code: Select all

ls -la /usr/local/nagios/var/spool

Also, run the following tail for a few minutes on that system and post the output here.

Code: Select all

tail -f /usr/local/nagiosxi/var/perfdataproc.log

[eloyd]

Looks to me like it's not even trying.

Code: Select all

# ls -la /usr/local/nagios/var/spool
total 4
drwxr-xr-x 5 nagios nagios   52 Apr 23  2015 .
drwxrwxr-x 6 nagios nagios 4096 Nov 18 12:31 ..
drwxrwsr-x 2 nagios nagcmd    6 Nov 17 16:37 checkresults
drwxr-xr-x 2 nagios nagios    6 Nov 18 12:31 perfdata
drwxr-xr-x 2 nagios nagios    6 Nov 18 12:31 xidpe

Even though perfdata gets files in it from time to time:

Code: Select all

ls -lsa
total 28
 0 drwxr-xr-x 2 nagios nagios    71 Nov 18 12:34 .
 0 drwxr-xr-x 5 nagios nagios    52 Apr 23  2015 ..
 8 -rw-r--r-- 1 nagios nagios  4907 Nov 18 12:34 1447868081.perfdata.host
20 -rw-r--r-- 1 nagios nagios 17762 Nov 18 12:34 1447868081.perfdata.servic

Code: Select all

]# tail -f /usr/local/nagiosxi/var/perfdataproc.log
Outbound data DISABLED Wed, 18 Nov 2015 12:32:01 -0500

DONE. Processed 0 files.
tail: /usr/local/nagiosxi/var/perfdataproc.log: file truncated
Outbound data DISABLED Wed, 18 Nov 2015 12:33:01 -0500

DONE. Processed 0 files.
tail: /usr/local/nagiosxi/var/perfdataproc.log: file truncated
Outbound data DISABLED Wed, 18 Nov 2015 12:34:01 -0500

DONE. Processed 0 files.

[tgriep]

Can you PM the System Profile?

[tgriep]

Thanks for the PM. I am going to need the profile.zip file.
To send us your system profile
Click "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the file and email it back to us.

[tgriep]

Can you edit the /usr/local/nagios/etc/pnp/process_perfdata.cfg file
Change the LOG_LEVEL from

Code: Select all

LOG_LEVEL = 0

to

Code: Select all

LOG_LEVEL = 1

Save it and let the system run for around 30 minutes and then run the following and post the output here?

Code: Select all

tail -200 /usr/local/nagios/var/perfdata.log

[eloyd]

Is 3 minutes long enough?

Code: Select all

2015-11-23 10:36:52 [12967] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:36:52 [12967] [1] 137 lines processed
2015-11-23 10:36:52 [12967] [1] /usr/local/nagios/var/spool/perfdata//1448293003.perfdata.service-PID-12967 deleted
2015-11-23 10:36:52 [12967] [1] PNP exiting (runtime 0.001734s) ...
2015-11-23 10:36:52 [12968] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:36:52 [12968] [1] 26 lines processed
2015-11-23 10:36:52 [12968] [1] /usr/local/nagios/var/spool/perfdata//1448293003.perfdata.host-PID-12968 deleted
2015-11-23 10:36:52 [12968] [1] PNP exiting (runtime 0.000417s) ...
2015-11-23 10:37:07 [13259] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:07 [13260] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:07 [13260] [1] 27 lines processed
2015-11-23 10:37:07 [13260] [1] /usr/local/nagios/var/spool/perfdata//1448293019.perfdata.host-PID-13260 deleted
2015-11-23 10:37:07 [13259] [1] 122 lines processed
2015-11-23 10:37:07 [13259] [1] /usr/local/nagios/var/spool/perfdata//1448293019.perfdata.service-PID-13259 deleted
2015-11-23 10:37:07 [13259] [1] PNP exiting (runtime 0.0031s) ...
2015-11-23 10:37:07 [13260] [1] PNP exiting (runtime 0.001203s) ...
2015-11-23 10:37:37 [13630] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:37 [13629] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:37 [13628] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:37 [13631] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:37 [13629] [1] 24 lines processed
2015-11-23 10:37:37 [13628] [1] 120 lines processed
2015-11-23 10:37:37 [13630] [1] 116 lines processed
2015-11-23 10:37:37 [13629] [1] /usr/local/nagios/var/spool/perfdata//1448293049.perfdata.host-PID-13629 deleted
2015-11-23 10:37:37 [13629] [1] PNP exiting (runtime 0.005579s) ...
2015-11-23 10:37:37 [13630] [1] /usr/local/nagios/var/spool/perfdata//1448293033.perfdata.service-PID-13630 deleted
2015-11-23 10:37:37 [13631] [1] 29 lines processed
2015-11-23 10:37:37 [13631] [1] /usr/local/nagios/var/spool/perfdata//1448293033.perfdata.host-PID-13631 deleted
2015-11-23 10:37:37 [13631] [1] PNP exiting (runtime 0.003732s) ...
2015-11-23 10:37:37 [13630] [1] PNP exiting (runtime 0.009288s) ...
2015-11-23 10:37:37 [13628] [1] /usr/local/nagios/var/spool/perfdata//1448293049.perfdata.service-PID-13628 deleted
2015-11-23 10:37:37 [13628] [1] PNP exiting (runtime 0.007977s) ...
2015-11-23 10:37:53 [13846] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:53 [13847] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:37:53 [13847] [1] 24 lines processed
2015-11-23 10:37:53 [13847] [1] /usr/local/nagios/var/spool/perfdata//1448293063.perfdata.host-PID-13847 deleted
2015-11-23 10:37:53 [13847] [1] PNP exiting (runtime 0.001084s) ...
2015-11-23 10:37:53 [13846] [1] 124 lines processed
2015-11-23 10:37:53 [13846] [1] /usr/local/nagios/var/spool/perfdata//1448293063.perfdata.service-PID-13846 deleted
2015-11-23 10:37:53 [13846] [1] PNP exiting (runtime 0.00287s) ...
2015-11-23 10:38:08 [14147] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:38:08 [14146] [1] process_perfdata.pl-0.6.11 starting in BULK Mode called by NPCD
2015-11-23 10:38:08 [14147] [1] 32 lines processed
2015-11-23 10:38:08 [14146] [1] 108 lines processed
2015-11-23 10:38:08 [14147] [1] /usr/local/nagios/var/spool/perfdata//1448293078.perfdata.host-PID-14147 deleted
2015-11-23 10:38:08 [14146] [1] /usr/local/nagios/var/spool/perfdata//1448293078.perfdata.service-PID-14146 deleted
2015-11-23 10:38:08 [14146] [1] PNP exiting (runtime 0.002738s) ...
2015-11-23 10:38:08 [14147] [1] PNP exiting (runtime 0.002059s) ...