Performance data missing after changing nagios MySQL pasword

[eloyd]

We have a customer that had an overzealous DBA that changed the nagiosxi user's MySQL password in an effort to be more secure. After realizing their mistake, they did this:

Code: Select all

service mysqld stop
for file in `ls /var/lib/mysql/nagios/ | grep "MYI"`; do myisamchk -r -f /var/lib/mysql/nagios/$file; done
service mysqld start

And then changed the password back to the default.

Since then (October 9) they have had no performance data. I cannot log in directly to their Nagios server as they are on a restricted network, but they will type whatever commands I ask them to as root.

Any thoughts on what might have happened or how to get performance data back?

[tgriep]

Sound like the npcd daemon isn't running. Lets check and see if there are a lot of files built up before it is restarted.
Have them run this on the server and post back.

Code: Select all

cd /usr/local/nagios/var/spool/xidpe
ls | wc -l

You can have them look at the WIKI link too for some pointers.
https://support.nagios.com/wiki/index.p ... h_Problems

[eloyd]

Awaiting results. Thanks.

[lmiltchev]

Actually, ask them to run:

Code: Select all

ls /usr/local/nagios/var/spool/xidpe | wc -l
ls /usr/local/nagios/var/spool/perfdata | wc -l
ls /usr/local/nagios/var/spool/checkresults | wc -l

This way, we will know what the number of perfdata files is in each of the three directories (xidpe, perfdata & checkresults). Thanks!

[eloyd]

Nothing in any of those directories:

Code: Select all

# ls /usr/local/nagios/var/spool/xidpe
# ls /usr/local/nagios/var/spool/perfdata
# ls /usr/local/nagios/var/spool/checkresults

And before you ask:

Code: Select all

# ps -aef | grep npcd
nagios    1588     1  0 Oct19 ?        00:00:10 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg

[jdalrymple]

The only thing that correlates performance data to the mysql db (that I can think of) are the process_perfdata_bulk_stuff commands in nagiosql.

Code: Select all

ls -lh /usr/local/nagios/var | grep perfdata
echo "select command_line from tbl_command where command_name like "%bulk%";" | mysql -u whatever -pwhatever nagiosql

Obviously ask their overzealous dbas for "whatever"

[eloyd]

Sorry for the late reply:

Code: Select all

]# ls -l /usr/local/nagios/var | grep perfdata
-rw-r--r-- 1 nagios nagios    4198 Nov  9 13:43 host-perfdata
-rw-rw-r-- 1 nagios nagios    5198 Jul  8 00:01 perfdata.log
-rw-r--r-- 1 nagios nagios   13024 Nov  9 13:43 service-perfdata

And, since they changed the username/password back to defaults,

Code: Select all

mysql> select command_line from tbl_command where command_name like "%bulk%";
+--------------------------------------------------------------------------------------------------------------+
| command_line                                                                                                 |
+--------------------------------------------------------------------------------------------------------------+
| /bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/perfdata/service-perfdata.$TIMET$ |
| /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/perfdata/host-perfdata.$TIMET$       |
| /bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.service    |
| /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.host          |
+--------------------------------------------------------------------------------------------------------------+

So it looks like it should be gathering data, but nothing has shown in the reports since Oct 9.

busted.png

[lmiltchev]

Do you see any errors/timeouts in logs?

Code: Select all

tail -100 /usr/local/nagios/var/npcd.log
tail -100 /usr/local/nagios/var/perfdata.log

[eloyd]

Will need to wait until tomorrow to get through firewall (I have to email them commands to type).

[lmiltchev]

Sure. We will wait for your input.

One more thing - make sure that logging is enabled in the npcd.cfg:

Code: Select all

log_level = 2

and in the process_perfdata.cfg

Code: Select all

LOG_LEVEL = 2

and the npcd has been restarted prior to tailing the logs:

Code: Select all

service npcd restart