Hung at Waiting for Database Startup

[sgiworks]

Hello Team,

We had encountered disk space issue on our Nagios Log Server, however we have now fixed it by increasing the disk space.

However the server console isn't coming up as normal, we see Waiting for Database Startup.

I have already tried restarted the logtash and elasticsearcg service, but still no luck.


Regards,
Swapnil

[scottwilkerson]

How long has it been since elasticsearch restarted? It can take a significant amount of time to repair the indexes.

How much free space do you currently have on each of your instances?

Code: Select all

df -h

Can you run the following and post the output

Code: Select all

curl -XGET 'localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason'|grep -v STARTED

and

Code: Select all

curl -XGET 'localhost:9200/_cat/health?v'

and

Code: Select all

curl -XGET 'localhost:9200/_cat/nodes?v'

[sgiworks]

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 350G 250G 100G 72% /
devtmpfs 6.9G 0 6.9G 0% /dev
tmpfs 6.9G 0 6.9G 0% /dev/shm
tmpfs 6.9G 697M 6.2G 10% /run
tmpfs 6.9G 0 6.9G 0% /sys/fs/cgroup
/dev/sda1 497M 103M 394M 21% /boot
/dev/sdb1 28G 2.1G 24G 8% /mnt/resource
tmpfs 1.4G 0 1.4G 0% /run/user/1000
tmpfs 1.4G 0 1.4G 0% /run/user/1001


[root@IWKSEASPAINLS01 tmp]# curl -XGET 'localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason'|grep -v STARTED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to localhost:9200; Connection refused


[root@IWKSEASPAINLS01 tmp]# curl -XGET 'localhost:9200/_cat/health?v'
curl: (7) Failed connect to localhost:9200; Connection refused
[root@IWKSEASPAINLS01 tmp]# curl -XGET 'localhost:9200/_cat/nodes?v'
curl: (7) Failed connect to localhost:9200; Connection refused

[sgiworks]

Server is back online after taking a reboot.

thanks!

[scottwilkerson]

great to hear!

[sgiworks]

Needed one recommendation on what could be the issue to raise in index size?

[scottwilkerson]

That index for today's date is broken (this can happen after a disk failure), click the delete button, and let it re-create a new one

[scottwilkerson]

Needed one recommendation on what could be the issue to raise in index size?

For the size, this could simply mean you added configuration to a server that has a LOT of logs, I would search in those time periods to see which host is producing all the logs