NagiosXI Mysql DB table data got delted

[Satyam]

Hi,

We are running NgaiosXI but suddenly one day we found that Nagvis is showing error(hosts or services not found) on the NOC screen, but there was no impact on Nagios/XI, when i checked the root cause, i came to know the data from the Mysqldb tables, running on a remote server(we have offloadeed mysql to another server) got delted automatically. I checked various logs but i couldn't understand how it got deleted automatically (or it has been delted manually by someone by mistake) whatever but two tables nagios_hosts & Nagios_services had 0 rows.

To fix it once i have restored data in these tables with the sql backup, nagvis started showing correctly.

Can someone help me in finding root cause for this.

[scottwilkerson]

Did the mysql logs show the database corrupted (before your restore)??

[Satyam]

Hi Scott,

Yes i have seen in the Nagios logs that some tables got crashed but not exactly these tables nagios_hosts & nagios_services, so after that i repaired the tables using nagiosxi script repairmysql.sh

Actually what happened is that the whole space on the mysqld server got filled that day with gearmand log which I had put in debug mode and I forgot to make it in least logging mode and that log file size grew in GBs. So I have cleared the space then I repaired few tables which were crashed as shown in the Nagios log and then still Nagvis was showing error, so i found particular these two tables(nagios_hosts & nagios_services) were empty and once i restored them it was working fine. I am not sure how data got deleted from particularly these two tables though these tables were not shown as crashed in nagios logs.

[scottwilkerson]

This explains a lot.

Sometime if you run out of disk space not only can the table get corrupt, it can lose the formatting all together...

In this case, after making space, restore a backup is usually your best bet.

[Satyam]

Hi Scott,

This time again it happened and data from the same two tables nagios_hosts & nagios_services in nagios db got deleted automatically, though this time there are no logs for tables crashing or any other problem in nagios log, also disk space is more than enough. Its difficult for me to understand why its happenning after running fine for a week or so, since we have automated the db backup, so we have the backups from which we are trying to restore now.

Also when i checked mysqld.log i found few warning lines like these:-

120418 12:28:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
120418 12:28:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295


Any idea what can be the problem???

[scottwilkerson]

This is strange behavior.

What version of XI are you running?

[Satyam]

Hi Scott,

We are running "Nagios XI 2011R1.7 Copyright © 2008-2011 Nagios Enterprises, LLC."

Please let me know if you want any further info on the same.

[scottwilkerson]

The more I look at this it seems that there may be an ndo2db problem

Are you seeing any errors in the syslog relating to ndo2db?

Code: Select all

tail -f /var/log/messages

We may also want to try to reset nagios and ndo2db

Code: Select all

service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start

[Satyam]

Hi Scott,

I have checked the system logs (/var/log/messages) for any ndo2db errors, but I couldn't find any. Also checked in the ndo2db.debug but without any success.

Also, killed and restarted nagios & ndo2db on nagios servers. I am observing it, and let me post here, once i get the same problem again.

[mguthrie]

Do you have any DB maintenance scripts of your own running on that remote server? It's acting as though those tables are being truncated...