Page 1 of 2
NagiosXI Mysql DB table data got delted
Posted: Thu Oct 04, 2012 8:14 am
by Satyam
Hi,
We are running NgaiosXI but suddenly one day we found that Nagvis is showing error(hosts or services not found) on the NOC screen, but there was no impact on Nagios/XI, when i checked the root cause, i came to know the data from the Mysqldb tables, running on a remote server(we have offloadeed mysql to another server) got delted automatically. I checked various logs but i couldn't understand how it got deleted automatically (or it has been delted manually by someone by mistake) whatever but two tables nagios_hosts & Nagios_services had 0 rows.
To fix it once i have restored data in these tables with the sql backup, nagvis started showing correctly.
Can someone help me in finding root cause for this.
Re: NagiosXI Mysql DB table data got delted
Posted: Thu Oct 04, 2012 9:24 am
by scottwilkerson
Did the mysql logs show the database corrupted (before your restore)??
Re: NagiosXI Mysql DB table data got delted
Posted: Fri Oct 05, 2012 3:46 am
by Satyam
Hi Scott,
Yes i have seen in the Nagios logs that some tables got crashed but not exactly these tables nagios_hosts & nagios_services, so after that i repaired the tables using nagiosxi script repairmysql.sh
Actually what happened is that the whole space on the mysqld server got filled that day with gearmand log which I had put in debug mode and I forgot to make it in least logging mode and that log file size grew in GBs. So I have cleared the space then I repaired few tables which were crashed as shown in the Nagios log and then still Nagvis was showing error, so i found particular these two tables(nagios_hosts & nagios_services) were empty and once i restored them it was working fine. I am not sure how data got deleted from particularly these two tables though these tables were not shown as crashed in nagios logs.
Re: NagiosXI Mysql DB table data got delted
Posted: Fri Oct 05, 2012 10:23 am
by scottwilkerson
This explains a lot.
Sometime if you run out of disk space not only can the table get corrupt, it can lose the formatting all together...
In this case, after making space, restore a backup is usually your best bet.
Re: NagiosXI Mysql DB table data got delted
Posted: Wed Oct 17, 2012 1:26 am
by Satyam
Hi Scott,
This time again it happened and data from the same two tables nagios_hosts & nagios_services in nagios db got deleted automatically, though this time there are no logs for tables crashing or any other problem in nagios log, also disk space is more than enough. Its difficult for me to understand why its happenning after running fine for a week or so, since we have automated the db backup, so we have the backups from which we are trying to restore now.
Also when i checked mysqld.log i found few warning lines like these:-
120418 12:28:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
120418 12:28:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
Any idea what can be the problem???
Re: NagiosXI Mysql DB table data got delted
Posted: Wed Oct 17, 2012 11:10 am
by scottwilkerson
This is strange behavior.
What version of XI are you running?
Re: NagiosXI Mysql DB table data got delted
Posted: Thu Oct 18, 2012 2:20 am
by Satyam
Hi Scott,
We are running "Nagios XI 2011R1.7 Copyright © 2008-2011 Nagios Enterprises, LLC."
Please let me know if you want any further info on the same.
Re: NagiosXI Mysql DB table data got delted
Posted: Thu Oct 18, 2012 10:05 am
by scottwilkerson
The more I look at this it seems that there may be an ndo2db problem
Are you seeing any errors in the syslog relating to ndo2db?
We may also want to try to reset nagios and ndo2db
Code: Select all
service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start
Re: NagiosXI Mysql DB table data got delted
Posted: Fri Oct 19, 2012 3:34 am
by Satyam
Hi Scott,
I have checked the system logs (/var/log/messages) for any ndo2db errors, but I couldn't find any. Also checked in the ndo2db.debug but without any success.
Also, killed and restarted nagios & ndo2db on nagios servers. I am observing it, and let me post here, once i get the same problem again.
Re: NagiosXI Mysql DB table data got delted
Posted: Fri Oct 19, 2012 10:34 am
by mguthrie
Do you have any DB maintenance scripts of your own running on that remote server? It's acting as though those tables are being truncated...