Page 1 of 1
NLS Elastic Search Status going into Critical very often
Posted: Fri Jul 12, 2019 10:30 am
by Sampath.Basireddy
Hello There,
Since last couple weeks I started noticing Elastic Search Status going into Critical state very often like more than 10-15 times every day and it clears automatically.
Setup is clustered and we have 2 instances in the cluster and both instances are in same data center.
One instance reports "Info: CRITICAL - Status is yellow, 204 Unassigned Shards"
And the other instance reports "Info: CRITICAL - Status is 503"
Recycling services did not help.
NLS Version: 2.0.4
Re: NLS Elastic Search Status going into Critical very often
Posted: Fri Jul 12, 2019 2:08 pm
by scottwilkerson
How much free disk space do you have available on each instance?
Nagios Log Server requires greater than 20% free before it can move replica shards around properly
Re: NLS Elastic Search Status going into Critical very often
Posted: Fri Jul 12, 2019 6:22 pm
by Sampath.Basireddy
More than 40% free space available:
1st Instance:
Code: Select all
[root@nlsp1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-lv_root 9.5G 3.9G 5.2G 43% /
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 24K 32G 1% /dev/shm
tmpfs 32G 3.1G 29G 10% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda1 969M 142M 777M 16% /boot
/dev/mapper/vg00-lv_home 1.9G 6.6M 1.8G 1% /home
/dev/mapper/vg00-lv_opt 969M 610M 293M 68% /opt
/dev/mapper/vg00-lv_var 5.7G 2.0G 3.4G 38% /var
/dev/mapper/vg00-lv_tmp 4.7G 21M 4.5G 1% /tmp
/dev/mapper/vg_apps-lvol0 2.2T 1.3T 996G 56% /usr/local/nagioslogserver
tmpfs 6.3G 0 6.3G 0% /run/user/13802
tmpfs 6.3G 0 6.3G 0% /run/user/1384400234
[root@nlsp1 ~]$
2nd Instance:
Code: Select all
[root@nlsp2 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 6.7G 34G 17% /
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 24K 32G 1% /dev/shm
tmpfs 32G 3.2G 29G 11% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/vg_apps-lvol0 2.2T 1.3T 995G 56% /usr/local/nagioslogserver
tmpfs 6.3G 0 6.3G 0% /run/user/993
tmpfs 6.3G 0 6.3G 0% /run/user/1384400234
[root@nlsp2 ~]$
Re: NLS Elastic Search Status going into Critical very often
Posted: Mon Jul 15, 2019 11:54 am
by cdienger
Please PM me a profile from each system. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:
/usr/local/nagioslogserver/scripts/profile.sh
This will create /tmp/system-profile.tar.gz.
Note that this file can be very large and may not be able to be uploaded through the system. This is usually due to the logs in the Logstash and/or Elasticseach directories found in it. If it is too large, please open the profile, extract these directories/files and send them separately.
Re: NLS Elastic Search Status going into Critical very often
Posted: Tue Jul 16, 2019 9:01 am
by Sampath.Basireddy
pm'd the profile downloaded from web console.
Re: NLS Elastic Search Status going into Critical very often
Posted: Tue Jul 16, 2019 2:24 pm
by cdienger
The profile provided shows everything in a green status. How frequently does this occur? When was the last date and time it happened? Where are you seeing these info messages exactly?
Please also provide a profile from the second machine in the cluster.
Re: NLS Elastic Search Status going into Critical very often
Posted: Wed Jul 17, 2019 3:54 pm
by Sampath.Basireddy
There is no exact trend as such, but am attaching the screenshot of alerts from both Server 1 & 2.
I pm'd the profile from second server.
Re: NLS Elastic Search Status going into Critical very often
Posted: Thu Jul 18, 2019 1:59 pm
by cdienger
I'd like to take a closer look via a remote. Can you open a ticket for this?
Re: NLS Elastic Search Status going into Critical very often
Posted: Fri Jul 19, 2019 1:13 pm
by Sampath.Basireddy
Done, opened #545896
Re: NLS Elastic Search Status going into Critical very often
Posted: Fri Jul 19, 2019 1:21 pm
by cdienger
Received. Locking this thread.