Page 1 of 1
Nagios cluster status RED and getting error viewing dashboar
Posted: Fri Feb 03, 2017 10:04 am
by anish
Hi ,
The Nagios Cluster status is in RED. And while viewing the dashboard there is an error " FacetPhaseExecutionException[Facet [0]: (key) field [@timestamp] not found]"
Logs are not getting collected for the Newly added source .
df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 99G 91G 7.0G 93% /
devtmpfs 3.9G 148K 3.9G 1% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 99G 91G 7.0G 93% /
Need your inputs for further trouble shooting
Re: Nagios cluster status RED and getting error viewing dash
Posted: Fri Feb 03, 2017 10:20 am
by rkennedy
Can you post a profile for us to review? (Admin -> System Status -> Download Profile)
This will have a bit of information for us to get started with.
Re: Nagios cluster status RED and getting error viewing dash
Posted: Mon Feb 06, 2017 6:22 am
by anish
Hi ,
Please find the attached system profile of our environment
Re: Nagios cluster status RED and getting error viewing dash
Posted: Mon Feb 06, 2017 10:18 am
by mcapra
Your root partition is at 93% usage:
I would suggest expanding the disk, restarting the elasticsearch service, allowing the cluster to quiesce, and seeing if that resolves the issue.
If that doesn't resolve the issue, can you share the contents of your Elasticsearch logs? They should be located in
/var/log/elasticsearch.
Re: Nagios cluster status RED and getting error viewing dash
Posted: Wed Feb 08, 2017 6:44 am
by anish
Hi ,
Increased the Disk space ,created the repository file . Now the disk space is free . Please check the contents located in elastic search.
[root@SESKLNGLSIDV01 elasticsearch]# ls
18454681-1b03-46d0-ae65-bf12905ec57d_index_indexing_slowlog.log
18454681-1b03-46d0-ae65-bf12905ec57d_index_search_slowlog.log
18454681-1b03-46d0-ae65-bf12905ec57d.log
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170203.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170204.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170205.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170206.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170207.gz
18454681-1b03-46d0-ae65-bf12905ec57d.log-20170208.gz
Disk space
Filesystem Size Used Avail Use% Mounted on
rootfs 99G 8.8G 89G 9% /
devtmpfs 2.0G 160K 2.0G 1% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sda1 99G 8.8G 89G 9% /
/dev/mapper/vg_app-lv_app
197G 188M 187G 1% /repository
Still the cluster status is red
Re: Nagios cluster status RED and getting error viewing dash
Posted: Wed Feb 08, 2017 9:58 am
by rkennedy
Can you post the elasticsearch + logstash logs for us to review?
Also, what happens if you attempt to start them? service elasticsearch start and service logstash start?
Generally the red status means your shards are not starting properly. What is the full output of curl 'localhost:9200/_cluster/health?level=indices&pretty'?
Re: Nagios cluster status RED and getting error viewing dash
Posted: Wed Feb 15, 2017 9:53 am
by anish
Hi ,
Please find the attached attachment
Output of curl 'localhost:9200/_cluster/health?level=indices&pret
service elasticsearch start,service logstash start.
Re: Nagios cluster status RED and getting error viewing dash
Posted: Wed Feb 15, 2017 10:17 am
by rkennedy
Code: Select all
INFO: [974db6d2-928a-4ddf-88ee-94467cd26836] loaded [], sites []
Feb 15, 2017 9:23:01 AM org.elasticsearch.common.network.NetworkUtils <clinit>
WARNING: failed to resolve local host, fallback to loopback
java.net.UnknownHostException: SESKLNGLSIDV01: SESKLNGLSIDV01: Name or service not known
I noticed that in the log file for logstash. Can you add to your /etc/hosts so that SESKLNGLSIDV01 resolves to 127.0.0.1? I haven't seen this in the past, but I imagine this will fix it. Then try starting logstash again.
Your elastic cluster looks to be in good health, we just need to get logstash working at this point.
Code: Select all
[root@SESKLNGLSIDV01 ~]# curl 'localhost:9200/_cluster/health?level=indices&pret ty'
{
"cluster_name" : "18454681-1b03-46d0-ae65-bf12905ec57d",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
Re: Nagios cluster status RED and getting error viewing dash
Posted: Wed Mar 01, 2017 8:14 am
by anish
Thanks ,the issue got fixed after updating etc\ hosts