Waiting for Database Startup

[jabi27]

Hi,

When accessing out nagios logserver it says "Waiting for Database Startup" and have done this for many hours. How can I verify speed up the process ?

Best
/Jan

[cdienger]

It should be a relatively quick thing and there may be issues elsewhere. Did anything change recently? Any crashes or problems? What's disk space look like with a "df -h" ?

Check out the elasticsearch logs in /var/log/elasticsearch/<cluster_uuid>.log

What version of NLS is this? You can find it on the command line with:

cat /var/www/html/nagioslogserver/lsversion

[jabi27]

The root disk was full. But it has been cleared but still nok luck.

cat /var/www/html/nagioslogserver/lsversion
VERSION=2002[root@nagios-logserver nagioslogserver]#

In the logfile here is a *lot* of this kind of errors:
------------------------------------------------------------------------
[2018-12-06 08:54:03,459][WARN ][cluster.action.shard ] [18198326-e85b-4f58-ba6f-f7544233e813] [nagioslogserver][0] received shard failed for [nagioslogserver][0], node[KeSQFl_tS9iS7BbqyshWlA], [P], s[INITIALIZING], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-12-06T07:54:03.372Z], details[shard failure [failed recovery][IndexShardGatewayRecoveryException[[nagioslogserver][0] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchException[failed to read [commands][run_all_alerts]]; nested: ElasticsearchIllegalArgumentException[No version type match [109]]; ]]], indexUUID [aXP3BohWQgOvQkuzzkr3fQ], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[nagioslogserver][0] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchException[failed to read [commands][run_all_alerts]]; nested: ElasticsearchIllegalArgumentException[No version type match [109]]; ]]
---------------------------------------------------------------

Thanks and best regards

/Jan

[cdienger]

There appears to be some corruption with the nagioslogserver indice which contains a lot of the NLS configuration. Do you know when the disk filed up? NLS will take backups of indices that contain configuration information which we can try to restore to.

You can get a list of these backups:

ll /store/backups/nagioslogserver/

and then restore with:

cd /usr/local/nagioslogserver/scripts/
./restore_backup.sh /store/backups/nagioslogserver/<backup_name>