Page 1 of 3
Performance data missing after changing nagios MySQL pasword
Posted: Tue Oct 20, 2015 1:44 pm
by eloyd
We have a customer that had an overzealous DBA that changed the nagiosxi user's MySQL password in an effort to be more secure. After realizing their mistake, they did this:
Code: Select all
service mysqld stop
for file in `ls /var/lib/mysql/nagios/ | grep "MYI"`; do myisamchk -r -f /var/lib/mysql/nagios/$file; done
service mysqld start
And then changed the password back to the default.
Since then (October 9) they have had no performance data. I cannot log in directly to their Nagios server as they are on a restricted network, but they will type whatever commands I ask them to as root.
Any thoughts on what might have happened or how to get performance data back?
Re: Performance data missing after changing nagios MySQL pas
Posted: Tue Oct 20, 2015 4:47 pm
by tgriep
Sound like the npcd daemon isn't running. Lets check and see if there are a lot of files built up before it is restarted.
Have them run this on the server and post back.
Code: Select all
cd /usr/local/nagios/var/spool/xidpe
ls | wc -l
You can have them look at the WIKI link too for some pointers.
https://support.nagios.com/wiki/index.p ... h_Problems
Re: Performance data missing after changing nagios MySQL pas
Posted: Wed Oct 21, 2015 5:43 pm
by eloyd
Awaiting results. Thanks.
Re: Performance data missing after changing nagios MySQL pas
Posted: Thu Oct 22, 2015 8:53 am
by lmiltchev
Actually, ask them to run:
Code: Select all
ls /usr/local/nagios/var/spool/xidpe | wc -l
ls /usr/local/nagios/var/spool/perfdata | wc -l
ls /usr/local/nagios/var/spool/checkresults | wc -l
This way, we will know what the number of perfdata files is in each of the three directories (xidpe, perfdata & checkresults). Thanks!
Re: Performance data missing after changing nagios MySQL pas
Posted: Thu Oct 22, 2015 9:03 am
by eloyd
Nothing in any of those directories:
Code: Select all
# ls /usr/local/nagios/var/spool/xidpe
# ls /usr/local/nagios/var/spool/perfdata
# ls /usr/local/nagios/var/spool/checkresults
And before you ask:
Code: Select all
# ps -aef | grep npcd
nagios 1588 1 0 Oct19 ? 00:00:10 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
Re: Performance data missing after changing nagios MySQL pas
Posted: Thu Oct 22, 2015 11:18 am
by jdalrymple
The only thing that correlates performance data to the mysql db (that I can think of) are the process_perfdata_bulk_stuff commands in nagiosql.
Code: Select all
ls -lh /usr/local/nagios/var | grep perfdata
echo "select command_line from tbl_command where command_name like "%bulk%";" | mysql -u whatever -pwhatever nagiosql
Obviously ask their overzealous dbas for "whatever"
Re: Performance data missing after changing nagios MySQL pas
Posted: Mon Nov 09, 2015 1:47 pm
by eloyd
Sorry for the late reply:
Code: Select all
]# ls -l /usr/local/nagios/var | grep perfdata
-rw-r--r-- 1 nagios nagios 4198 Nov 9 13:43 host-perfdata
-rw-rw-r-- 1 nagios nagios 5198 Jul 8 00:01 perfdata.log
-rw-r--r-- 1 nagios nagios 13024 Nov 9 13:43 service-perfdata
And, since they changed the username/password back to defaults,
Code: Select all
mysql> select command_line from tbl_command where command_name like "%bulk%";
+--------------------------------------------------------------------------------------------------------------+
| command_line |
+--------------------------------------------------------------------------------------------------------------+
| /bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/perfdata/service-perfdata.$TIMET$ |
| /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/perfdata/host-perfdata.$TIMET$ |
| /bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.service |
| /bin/mv /usr/local/nagios/var/host-perfdata /usr/local/nagios/var/spool/xidpe/$TIMET$.perfdata.host |
+--------------------------------------------------------------------------------------------------------------+
So it looks like it should be gathering data, but nothing has shown in the reports since Oct 9.
busted.png
Re: Performance data missing after changing nagios MySQL pas
Posted: Mon Nov 09, 2015 5:54 pm
by lmiltchev
Do you see any errors/timeouts in logs?
Code: Select all
tail -100 /usr/local/nagios/var/npcd.log
tail -100 /usr/local/nagios/var/perfdata.log
Re: Performance data missing after changing nagios MySQL pas
Posted: Mon Nov 09, 2015 7:15 pm
by eloyd
Will need to wait until tomorrow to get through firewall (I have to email them commands to type).
Re: Performance data missing after changing nagios MySQL pas
Posted: Tue Nov 10, 2015 10:12 am
by lmiltchev
Sure. We will wait for your input.
One more thing - make sure that logging is enabled in the npcd.cfg:
and in the process_perfdata.cfg
and the npcd has been restarted prior to tailing the logs: