My Nagios lost all of its hosts after sudden power down

[cairnsd]

Hi

My NagiosXI version 5.8.6 lost all hosts after a sudden power down. Any ideas how to restore? Tried to restore configuration from 2 days ago, but sitll my hosts are empty. I received some emails about my hosts coming back to life, but the hosts do not display anymore in the host status.

Thanks in advance
Diego

[pbroste]

Hello @cairnsd

Thanks for reaching out, and want to go through and verify that the hosts and services are accounted for, and also verify that the system date/time matches the database.

Code: Select all

date
echo 'select now();' | mysql -t -u root -pnagiosxi
echo 'select count(*) from nagios_hosts;' | mysql -t -u root -pnagiosxi nagios
echo 'select count(*) from nagios_services;' | mysql -t -u root -pnagiosxi nagios

If the hosts and service counts look good then let's go ahead and reindex the Core Configs from the database by:

Code: Select all

/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

Then bounce the nagios.service by:

Code: Select all

systemctl restart nagios

Please let us know how things look.

Thanks,
Perry

[cairnsd]

Hi Perry

Run the commands, all looked well but

+----------+
| count(*) |
+----------+
| 94 |
+----------+
+----------+
| count(*) |
+----------+
| 280 |
+----------+
Total Warnings: 1
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
> Return Code: 0

When I login to my Nagios I got the following notices, and hosts are still not there....see attach. I could also try to restore from Backup if it seems too complicated to solve this issue.

Thank for letting me know
Regards
Diego

[danniiffxi]

Hi @cairnsd

I am not a Nagios Dev here, but I have had similar issues in the past where when rebooting the Nagios server you are presented with SQL errors. Running the command below will repair your databases, it will take a few mins to run through, but everything should start to appear again.

This is based on CentOS 7/8, it stops the Nagios and SQL services, runs the DB repair script that is installed with Nagios XI, after it is complete it will restart the services.

Code: Select all

systemctl stop nagios.service
systemctl stop mysql.service
systemctl stop mysqld.service
cd /usr/local/nagiosxi/scripts
./repair_databases.sh
systemctl start mysql.service
systemctl start mysqld.service
systemctl start nagios.service

Let me know how you get on

[pbroste]

Hello @cairnsd

Please follow up when you get a chance and let us know if the details @danniiffxi provided resolved your issue.

Thanks,
Perry

[cairnsd]

Hi all

Thanks for your replies...still the same issue, please find below the output for the posted commands:

Code: Select all

]# systemctl stop nagios.service
[root@localhost ~]# systemctl stop mysql.service
Failed to stop mysql.service: Unit mysql.service not loaded.
[root@localhost ~]# systemctl stop mysqld.service
Failed to stop mysqld.service: Unit mysqld.service not loaded.
[root@localhost ~]# cd /usr/local/nagiosxi/scripts
[root@localhost scripts]# ./repair_databases.sh
DATABASE: nagios
TABLE:
/var/lib/mysql/nagios /usr/local/nagiosxi/scripts
- recovering (with sort) MyISAM-table 'nagios_acknowledgements'
Data records: 22
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_commands'
Data records: 167
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_commenthistory'
Data records: 852
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_comments'
Data records: 2
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_configfiles'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_configfilevariables'
Data records: 131
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_conninfo'
Data records: 301
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_contact_addresses'
Data records: 10
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactgroup_members'
Data records: 17
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactgroups'
Data records: 5
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contact_notificationcommands'
Data records: 20
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactnotificationmethods'
Data records: 2237
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_contactnotifications'
Data records: 2237
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4

---------

- recovering (with sort) MyISAM-table 'nagios_contacts'
Data records: 10
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_contactstatus'
Data records: 10
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_customvariables'
Data records: 170
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with sort) MyISAM-table 'nagios_customvariablestatus'
Data records: 170
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

- recovering (with keycache) MyISAM-table 'nagios_dbversion'
Data records: 1

---------

- recovering (with sort) MyISAM-table 'nagios_downtimehistory'
Data records: 121
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_eventhandlers' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_externalcommands'
Data records: 0
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_flappinghistory'
Data records: 980
- Fixing index 1

---------

myisamchk: error: 'nagios_hostchecks' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_host_contactgroups'
Data records: 48
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_host_contacts'
Data records: 89
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostdependencies'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalation_contactgroups'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalation_contacts'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostescalations'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostgroup_members'
Data records: 95
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hostgroups'
Data records: 17
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_host_parenthosts'
Data records: 107
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_hosts'
Data records: 94
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

myisamchk: error: 'nagios_hoststatus' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_instances'
Data records: 1
- Fixing index 1

---------

myisamchk: error: 'nagios_logentries' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_notifications'
Data records: 866
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4

---------

- recovering (with sort) MyISAM-table 'nagios_objects'
Data records: 881
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5

---------

- recovering (with sort) MyISAM-table 'nagios_processevents'
Data records: 991
- Fixing index 1

---------

- recovering (with sort) MyISAM-table 'nagios_programstatus'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_runtimevariables'
Data records: 18
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_scheduleddowntime'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_servicechecks' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_service_contactgroups'
Data records: 175
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_service_contacts'
Data records: 211
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicedependencies'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contactgroups'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contacts'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalations'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroup_members'
Data records: 5
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroups'
Data records: 1
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_service_parentservices'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_services'
Data records: 280
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

myisamchk: error: 'nagios_servicestatus' is not a MyISAM-table

---------

myisamchk: error: 'nagios_statehistory' is not a MyISAM-table

---------

myisamchk: error: 'nagios_systemcommands' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timedeventqueue'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

- recovering (with sort) MyISAM-table 'nagios_timedevents'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiods'
Data records: 8
- Fixing index 1
- Fixing index 2

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiod_timeranges'
Data records: 33
- Fixing index 1
- Fixing index 2
/usr/local/nagiosxi/scripts

===============
REPAIR COMPLETE
===============
DATABASE: nagiosql
TABLE:
/var/lib/mysql/nagiosql /usr/local/nagiosxi/scripts
DATABASE: nagiosxi
TABLE:
/var/lib/mysql/nagiosxi /usr/local/nagiosxi/scripts

=======================
nagios database repair FAILED, please check output above!

[root@localhost scripts]# systemctl start mysql.service
Failed to start mysql.service: Unit not found.
[root@localhost scripts]# systemctl start mysqld.service
Failed to start mysqld.service: Unit not found.
[root@localhost scripts]# systemctl start nagios.service
[root@localhost scripts]#

[cairnsd]

Hi there

Ok, my supplier helped me find the issue....

Run this to find where the error was

systemctl restart nagios
grep NDO /usr/local/nagios/var/nagios.log

I got plenty of errors like this one

NDO-3: Query failed in ndo_empty_queue_service_status
NDO-3: ndo_return = 1 (Statement not prepared)

So found in the forum https://support.nagios.com/forum/viewto ... 16&t=61416

and run this
mysqlcheck -f -r -u root -pnagiosxi --databases nagios --use-frm
systemctl restart nagios


And my nagios is back displaying all my hosts !

Thanks everyone for your time
regards
Diego

[pbroste]

Hello @cairnsd

Excellent, thanks for providing the resolution. I will go ahead and lock this.

Thanks,
Perry