Page 1 of 1
My Nagios lost all of its hosts after sudden power down
Posted: Thu Sep 23, 2021 1:58 am
by cairnsd
Hi
My NagiosXI version 5.8.6 lost all hosts after a sudden power down. Any ideas how to restore? Tried to restore configuration from 2 days ago, but sitll my hosts are empty. I received some emails about my hosts coming back to life, but the hosts do not display anymore in the host status.
Thanks in advance
Diego
Re: My Nagios lost all of its hosts after sudden power down
Posted: Thu Sep 23, 2021 1:50 pm
by pbroste
Hello @cairnsd
Thanks for reaching out, and want to go through and verify that the hosts and services are accounted for, and also verify that the system date/time matches the database.
Code: Select all
date
echo 'select now();' | mysql -t -u root -pnagiosxi
echo 'select count(*) from nagios_hosts;' | mysql -t -u root -pnagiosxi nagios
echo 'select count(*) from nagios_services;' | mysql -t -u root -pnagiosxi nagios
If the hosts and service counts look good then let's go ahead and reindex the Core Configs from the database by:
Code: Select all
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
Then bounce the nagios.service by:
Please let us know how things look.
Thanks,
Perry
Re: My Nagios lost all of its hosts after sudden power down
Posted: Fri Sep 24, 2021 2:31 am
by cairnsd
Hi Perry
Run the commands, all looked well but
+----------+
| count(*) |
+----------+
| 94 |
+----------+
+----------+
| count(*) |
+----------+
| 280 |
+----------+
Total Warnings: 1
Total Errors: 0
Things look okay - No serious problems were detected during the pre-flight check
> Return Code: 0
When I login to my Nagios I got the following notices, and hosts are still not there....see attach. I could also try to restore from Backup if it seems too complicated to solve this issue.
Thank for letting me know
Regards
Diego
Re: My Nagios lost all of its hosts after sudden power down
Posted: Fri Sep 24, 2021 7:04 am
by danniiffxi
Hi @cairnsd
I am not a Nagios Dev here, but I have had similar issues in the past where when rebooting the Nagios server you are presented with SQL errors. Running the command below will repair your databases, it will take a few mins to run through, but everything should start to appear again.
This is based on CentOS 7/8, it stops the Nagios and SQL services, runs the DB repair script that is installed with Nagios XI, after it is complete it will restart the services.
Code: Select all
systemctl stop nagios.service
systemctl stop mysql.service
systemctl stop mysqld.service
cd /usr/local/nagiosxi/scripts
./repair_databases.sh
systemctl start mysql.service
systemctl start mysqld.service
systemctl start nagios.service
Let me know how you get on
Re: My Nagios lost all of its hosts after sudden power down
Posted: Fri Sep 24, 2021 1:09 pm
by pbroste
Hello @cairnsd
Please follow up when you get a chance and let us know if the details @danniiffxi provided resolved your issue.
Thanks,
Perry
Re: My Nagios lost all of its hosts after sudden power down
Posted: Mon Sep 27, 2021 3:22 am
by cairnsd
Hi all
Thanks for your replies...still the same issue, please find below the output for the posted commands:
Code: Select all
]# systemctl stop nagios.service
[root@localhost ~]# systemctl stop mysql.service
Failed to stop mysql.service: Unit mysql.service not loaded.
[root@localhost ~]# systemctl stop mysqld.service
Failed to stop mysqld.service: Unit mysqld.service not loaded.
[root@localhost ~]# cd /usr/local/nagiosxi/scripts
[root@localhost scripts]# ./repair_databases.sh
DATABASE: nagios
TABLE:
/var/lib/mysql/nagios /usr/local/nagiosxi/scripts
- recovering (with sort) MyISAM-table 'nagios_acknowledgements'
Data records: 22
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_commands'
Data records: 167
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_commenthistory'
Data records: 852
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_comments'
Data records: 2
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_configfiles'
Data records: 1
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_configfilevariables'
Data records: 131
- Fixing index 1
---------
- recovering (with sort) MyISAM-table 'nagios_conninfo'
Data records: 301
- Fixing index 1
---------
- recovering (with sort) MyISAM-table 'nagios_contact_addresses'
Data records: 10
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_contactgroup_members'
Data records: 17
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_contactgroups'
Data records: 5
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_contact_notificationcommands'
Data records: 20
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_contactnotificationmethods'
Data records: 2237
- Fixing index 1
- Fixing index 2
- Fixing index 3
---------
- recovering (with sort) MyISAM-table 'nagios_contactnotifications'
Data records: 2237
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
---------
- recovering (with sort) MyISAM-table 'nagios_contacts'
Data records: 10
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_contactstatus'
Data records: 10
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_customvariables'
Data records: 170
- Fixing index 1
- Fixing index 2
- Fixing index 3
---------
- recovering (with sort) MyISAM-table 'nagios_customvariablestatus'
Data records: 170
- Fixing index 1
- Fixing index 2
- Fixing index 3
---------
- recovering (with keycache) MyISAM-table 'nagios_dbversion'
Data records: 1
---------
- recovering (with sort) MyISAM-table 'nagios_downtimehistory'
Data records: 121
- Fixing index 1
- Fixing index 2
---------
myisamchk: error: 'nagios_eventhandlers' is not a MyISAM-table
---------
- recovering (with sort) MyISAM-table 'nagios_externalcommands'
Data records: 0
- Fixing index 1
---------
- recovering (with sort) MyISAM-table 'nagios_flappinghistory'
Data records: 980
- Fixing index 1
---------
myisamchk: error: 'nagios_hostchecks' is not a MyISAM-table
---------
- recovering (with sort) MyISAM-table 'nagios_host_contactgroups'
Data records: 48
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_host_contacts'
Data records: 89
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_hostdependencies'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_hostescalation_contactgroups'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_hostescalation_contacts'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_hostescalations'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_hostgroup_members'
Data records: 95
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_hostgroups'
Data records: 17
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_host_parenthosts'
Data records: 107
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_hosts'
Data records: 94
- Fixing index 1
- Fixing index 2
- Fixing index 3
---------
myisamchk: error: 'nagios_hoststatus' is not a MyISAM-table
---------
- recovering (with sort) MyISAM-table 'nagios_instances'
Data records: 1
- Fixing index 1
---------
myisamchk: error: 'nagios_logentries' is not a MyISAM-table
---------
- recovering (with sort) MyISAM-table 'nagios_notifications'
Data records: 866
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
---------
- recovering (with sort) MyISAM-table 'nagios_objects'
Data records: 881
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
---------
- recovering (with sort) MyISAM-table 'nagios_processevents'
Data records: 991
- Fixing index 1
---------
- recovering (with sort) MyISAM-table 'nagios_programstatus'
Data records: 1
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_runtimevariables'
Data records: 18
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_scheduleddowntime'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
myisamchk: error: 'nagios_servicechecks' is not a MyISAM-table
---------
- recovering (with sort) MyISAM-table 'nagios_service_contactgroups'
Data records: 175
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_service_contacts'
Data records: 211
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_servicedependencies'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contactgroups'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contacts'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_serviceescalations'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_servicegroup_members'
Data records: 5
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_servicegroups'
Data records: 1
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_service_parentservices'
Data records: 0
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_services'
Data records: 280
- Fixing index 1
- Fixing index 2
- Fixing index 3
---------
myisamchk: error: 'nagios_servicestatus' is not a MyISAM-table
---------
myisamchk: error: 'nagios_statehistory' is not a MyISAM-table
---------
myisamchk: error: 'nagios_systemcommands' is not a MyISAM-table
---------
- recovering (with sort) MyISAM-table 'nagios_timedeventqueue'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
---------
- recovering (with sort) MyISAM-table 'nagios_timedevents'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
---------
- recovering (with sort) MyISAM-table 'nagios_timeperiods'
Data records: 8
- Fixing index 1
- Fixing index 2
---------
- recovering (with sort) MyISAM-table 'nagios_timeperiod_timeranges'
Data records: 33
- Fixing index 1
- Fixing index 2
/usr/local/nagiosxi/scripts
===============
REPAIR COMPLETE
===============
DATABASE: nagiosql
TABLE:
/var/lib/mysql/nagiosql /usr/local/nagiosxi/scripts
DATABASE: nagiosxi
TABLE:
/var/lib/mysql/nagiosxi /usr/local/nagiosxi/scripts
=======================
nagios database repair FAILED, please check output above!
[root@localhost scripts]# systemctl start mysql.service
Failed to start mysql.service: Unit not found.
[root@localhost scripts]# systemctl start mysqld.service
Failed to start mysqld.service: Unit not found.
[root@localhost scripts]# systemctl start nagios.service
[root@localhost scripts]#
Re: My Nagios lost all of its hosts after sudden power down
Posted: Mon Sep 27, 2021 4:30 am
by cairnsd
Hi there
Ok, my supplier helped me find the issue....
Run this to find where the error was
systemctl restart nagios
grep NDO /usr/local/nagios/var/nagios.log
I got plenty of errors like this one
NDO-3: Query failed in ndo_empty_queue_service_status
NDO-3: ndo_return = 1 (Statement not prepared)
So found in the forum
https://support.nagios.com/forum/viewto ... 16&t=61416
and run this
mysqlcheck -f -r -u root -pnagiosxi --databases nagios --use-frm
systemctl restart nagios
And my nagios is back displaying all my hosts !
Thanks everyone for your time
regards
Diego
Re: My Nagios lost all of its hosts after sudden power down
Posted: Mon Sep 27, 2021 10:56 am
by pbroste
Hello @cairnsd
Excellent, thanks for providing the resolution. I will go ahead and lock this.
Thanks,
Perry