Nagios cluster status RED and getting error viewing dashboar

[anish]

Hi ,

The Nagios Cluster status is in RED. And while viewing the dashboard there is an error " FacetPhaseExecutionException[Facet [0]: (key) field [@timestamp] not found]"

Logs are not getting collected for the Newly added source .

df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 99G 91G 7.0G 93% /
devtmpfs 3.9G 148K 3.9G 1% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 99G 91G 7.0G 93% /

Need your inputs for further trouble shooting

[rkennedy]

Can you post a profile for us to review? (Admin -> System Status -> Download Profile)

This will have a bit of information for us to get started with.

[anish]

Hi ,

Please find the attached system profile of our environment

[mcapra]

Your root partition is at 93% usage:

Code: Select all

rootfs 99G 91G 7.0G 93% /

I would suggest expanding the disk, restarting the elasticsearch service, allowing the cluster to quiesce, and seeing if that resolves the issue.

If that doesn't resolve the issue, can you share the contents of your Elasticsearch logs? They should be located in /var/log/elasticsearch.

[anish]

Hi ,
Increased the Disk space ,created the repository file . Now the disk space is free . Please check the contents located in elastic search.

[root@SESKLNGLSIDV01 elasticsearch]# ls
18454681-1b03-46d0-ae65-bf12905ec57d_index_indexing_slowlog.log
18454681-1b03-46d0-ae65-bf12905ec57d_index_search_slowlog.log
18454681-1b03-46d0-ae65-bf12905ec57d.log
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170203.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170204.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170205.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170206.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170207.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170208.gz


Disk space


Filesystem Size Used Avail Use% Mounted on
rootfs 99G 8.8G 89G 9% /
devtmpfs 2.0G 160K 2.0G 1% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sda1 99G 8.8G 89G 9% /
/dev/mapper/vg_app-lv_app
197G 188M 187G 1% /repository

Still the cluster status is red

[rkennedy]

Can you post the elasticsearch + logstash logs for us to review?

Also, what happens if you attempt to start them? service elasticsearch start and service logstash start?

Generally the red status means your shards are not starting properly. What is the full output of curl 'localhost:9200/_cluster/health?level=indices&pretty'?

[anish]

Hi ,

Please find the attached attachment

Output of curl 'localhost:9200/_cluster/health?level=indices&pret

service elasticsearch start,service logstash start.

[rkennedy]

Code: Select all

INFO: [974db6d2-928a-4ddf-88ee-94467cd26836] loaded [], sites []
Feb 15, 2017 9:23:01 AM org.elasticsearch.common.network.NetworkUtils <clinit>
WARNING: failed to resolve local host, fallback to loopback
java.net.UnknownHostException: SESKLNGLSIDV01: SESKLNGLSIDV01: Name or service not known

I noticed that in the log file for logstash. Can you add to your /etc/hosts so that SESKLNGLSIDV01 resolves to 127.0.0.1? I haven't seen this in the past, but I imagine this will fix it. Then try starting logstash again.

Your elastic cluster looks to be in good health, we just need to get logstash working at this point.

Code: Select all

[root@SESKLNGLSIDV01 ~]# curl 'localhost:9200/_cluster/health?level=indices&pret                                                                                                                     ty'
{
  "cluster_name" : "18454681-1b03-46d0-ae65-bf12905ec57d",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,

[anish]

Thanks ,the issue got fixed after updating etc\ hosts