NLS Elastic Search Status going into Critical very often

[Sampath.Basireddy]

Hello There,

Since last couple weeks I started noticing Elastic Search Status going into Critical state very often like more than 10-15 times every day and it clears automatically.

Setup is clustered and we have 2 instances in the cluster and both instances are in same data center.

One instance reports "Info: CRITICAL - Status is yellow, 204 Unassigned Shards"
And the other instance reports "Info: CRITICAL - Status is 503"

Recycling services did not help.

NLS Version: 2.0.4

[scottwilkerson]

How much free disk space do you have available on each instance?

Code: Select all

df -h

Nagios Log Server requires greater than 20% free before it can move replica shards around properly

[Sampath.Basireddy]

More than 40% free space available:

1st Instance:

Code: Select all

[root@nlsp1 ~]$ df -h
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vg00-lv_root   9.5G  3.9G  5.2G  43% /
devtmpfs                    32G     0   32G   0% /dev
tmpfs                       32G   24K   32G   1% /dev/shm
tmpfs                       32G  3.1G   29G  10% /run
tmpfs                       32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                  969M  142M  777M  16% /boot
/dev/mapper/vg00-lv_home   1.9G  6.6M  1.8G   1% /home
/dev/mapper/vg00-lv_opt    969M  610M  293M  68% /opt
/dev/mapper/vg00-lv_var    5.7G  2.0G  3.4G  38% /var
/dev/mapper/vg00-lv_tmp    4.7G   21M  4.5G   1% /tmp
/dev/mapper/vg_apps-lvol0  2.2T  1.3T  996G  56% /usr/local/nagioslogserver
tmpfs                      6.3G     0  6.3G   0% /run/user/13802
tmpfs                      6.3G     0  6.3G   0% /run/user/1384400234
[root@nlsp1 ~]$

2nd Instance:

Code: Select all

[root@nlsp2 ~]$ df -h
Filesystem                 Size  Used Avail Use% Mounted on
/dev/vda1                   40G  6.7G   34G  17% /
devtmpfs                    32G     0   32G   0% /dev
tmpfs                       32G   24K   32G   1% /dev/shm
tmpfs                       32G  3.2G   29G  11% /run
tmpfs                       32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/vg_apps-lvol0  2.2T  1.3T  995G  56% /usr/local/nagioslogserver
tmpfs                      6.3G     0  6.3G   0% /run/user/993
tmpfs                      6.3G     0  6.3G   0% /run/user/1384400234
[root@nlsp2 ~]$

[cdienger]

Please PM me a profile from each system. It can be gathered under Admin > System > System Status > Download System Profile or from the command line with:

/usr/local/nagioslogserver/scripts/profile.sh

This will create /tmp/system-profile.tar.gz.

Note that this file can be very large and may not be able to be uploaded through the system. This is usually due to the logs in the Logstash and/or Elasticseach directories found in it. If it is too large, please open the profile, extract these directories/files and send them separately.

[Sampath.Basireddy]

pm'd the profile downloaded from web console.

[cdienger]

The profile provided shows everything in a green status. How frequently does this occur? When was the last date and time it happened? Where are you seeing these info messages exactly?

Please also provide a profile from the second machine in the cluster.

[Sampath.Basireddy]

There is no exact trend as such, but am attaching the screenshot of alerts from both Server 1 & 2.

I pm'd the profile from second server.

[cdienger]

I'd like to take a closer look via a remote. Can you open a ticket for this?

[Sampath.Basireddy]

Done, opened #545896

[cdienger]

Received. Locking this thread.