Nagios Support Forum

Hello Team,

We had encountered disk space issue on our Nagios Log Server, however we have now fixed it by increasing the disk space.

However the server console isn't coming up as normal, we see Waiting for Database Startup.

I have already tried restarted the logtash and elasticsearcg service, but still no luck.


Regards,
Swapnil

How long has it been since elasticsearch restarted? It can take a significant amount of time to repair the indexes.

How much free space do you currently have on each of your instances? Can you run the following and post the output and and

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 350G 250G 100G 72% /
devtmpfs 6.9G 0 6.9G 0% /dev
tmpfs 6.9G 0 6.9G 0% /dev/shm
tmpfs 6.9G 697M 6.2G 10% /run
tmpfs 6.9G 0 6.9G 0% /sys/fs/cgroup
/dev/sda1 497M 103M 394M 21% /boot
/dev/sdb1 28G 2.1G 24G 8% /mnt/resource
tmpfs 1.4G 0 1.4G 0% /run/user/1000
tmpfs 1.4G 0 1.4G 0% /run/user/1001


[root@IWKSEASPAINLS01 tmp]# curl -XGET 'localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason'|grep -v STARTED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to localhost:9200; Connection refused


[root@IWKSEASPAINLS01 tmp]# curl -XGET 'localhost:9200/_cat/health?v'
curl: (7) Failed connect to localhost:9200; Connection refused
[root@IWKSEASPAINLS01 tmp]# curl -XGET 'localhost:9200/_cat/nodes?v'
curl: (7) Failed connect to localhost:9200; Connection refused

Server is back online after taking a reboot.

thanks!

great to hear!

Needed one recommendation on what could be the issue to raise in index size?

That index for today's date is broken (this can happen after a disk failure), click the delete button, and let it re-create a new one

sgiworks wrote:Needed one recommendation on what could be the issue to raise in index size?

For the size, this could simply mean you added configuration to a server that has a LOT of logs, I would search in those time periods to see which host is producing all the logs