Instance status different in console and GUI

[pccwglobalit]

in GUI, two of cluster status are red for both elasticsearch and logstash but in cluster status, i can see all 4 nodes are running.
also, in system status, all four nodes are processing and cannot know the nodes status then cannot restart nodes.

[jolson]

From: http://www.elastic.co/guide/en/elastics ... ealth.html

On the shard level, a red status indicates that the specific shard is not allocated in the cluster, yellow means that the primary shard is allocated but replicas are not, and green means that all shards are allocated. The index level status is controlled by the worst shard status. The cluster status is controlled by the worst index status.

Since this is a cluster health alert, it is being triggered by the worst index - which in turn is triggered by the worst shard. We need to figure out which shard(s) are causing this issue.

To see which shard are in the 'red' health status, let's run the following command on the CLI:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'

The results are going to be sloppy, but you will be able to find any shard with a status of 'red' and identify where they reside. Please post the results of the bad shards here, and we'll come up with a plan on how to get your cluster green again.

[pccwglobalit]

this is status now. also i attached shard status.
the green status on top is 192.168.82.240. but in the below, it shows red and i check in server that elasticsearch and logstash are both running. same situation on nls1-be.

[root@nls1-be ~]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "25248501-4aa5-4697-bfaa-5af953305a1f",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 471,
"active_shards" : 942,
"relocating_shards" : 2,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

[jolson]

Could you please re-attach those screenshots? You may have to hit the 'add the file' button after selecting them. Maybe the screenshots are too big for attachment?

I would like to see precisely what you're talking about so that we can troubleshoot this in an efficient way.

[pccwglobalit]

when i posted jpeg, it said "Sorry, the board attachment quota has been reached."

[pccwglobalit]

this is instance status

[pccwglobalit]

this is system status, you can see it keeps trying to get the status.

[pccwglobalit]

shard status

[jolson]

Please log into the nodes which are displaying as red in the GUI and run the commands:

Code: Select all

service elasticsearch status

Code: Select all

service logstash status

Code: Select all

chkconfig --list | egrep "logst|elas"

[pccwglobalit]

nls1

elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: active (running) since Wed 2015-03-18 09:07:46 GMT; 7h ago
Process: 29433 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 29435 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─29450 java -Xms8g -Xmx8g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatin...

Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.apache.log4j.PropertyConfigurator.parseAppender(Prop...842)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.apache.log4j.PropertyConfigurator.parseCategory(Prop...768)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.apache.log4j.PropertyConfigurator.parseCatsAndRender...672)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.apache.log4j.PropertyConfigurator.doConfigure(Proper...516)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.apache.log4j.PropertyConfigurator.configure(Property...440)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.elasticsearch.common.logging.log4j.LogConfigurator.c...101)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Boots...:94)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticse...:32)
Mar 18 09:07:46 nls1-tmh2.it.pccwglobal.com elasticsearch[29435]: log4j:ERROR Either File or DatePattern options are not set ...le].
Hint: Some lines were ellipsized, use -l to show in full.

Logstash Daemonlogstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Wed 2015-03-18 13:37:32 GMT; 2h 41min ago
Process: 19079 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 19091 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
├─19097 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserve...
└─19099 java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx1024m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava...

Mar 18 13:37:31 nls1-tmh2.it.pccwglobal.com runuser[19097]: pam_unix(runuser:session): session opened for user root by (uid=0)
Mar 18 13:37:32 nls1-tmh2.it.pccwglobal.com logstash[19091]: Starting Logstash Daemon: [ OK ]
Mar 18 13:37:32 nls1-tmh2.it.pccwglobal.com systemd[1]: Started LSB: Logstash.

Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.

If you want to list systemd services use 'systemctl list-unit-files'.
To see services enabled on particular target use
'systemctl list-dependencies [target]'.

elasticsearch 0:off 1:off 2:off 3:on 4:off 5:on 6:off
logstash 0:off 1:off 2:off 3:on 4:off 5:on 6:off

nls-be
elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: active (running) since Wed 2015-03-18 09:11:28 UTC; 7h ago
Process: 25872 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 25946 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─25960 java -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatin...

Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.apache.log4j.PropertyConfigurator.parseAppender(Proper...842)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.apache.log4j.PropertyConfigurator.parseCategory(Proper...768)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderer...672)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.apache.log4j.PropertyConfigurator.doConfigure(Property...516)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.apache.log4j.PropertyConfigurator.configure(PropertyCo...440)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.elasticsearch.common.logging.log4j.LogConfigurator.con...101)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstr...:94)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsear...:32)
Mar 18 09:11:29 nls1-be.it.pccwglobal.com elasticsearch[25946]: log4j:ERROR Either File or DatePattern options are not set fo...le].
Hint: Some lines were ellipsized, use -l to show in full.

Logstash Daemonlogstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Wed 2015-03-18 09:25:35 UTC; 6h ago
Process: 26181 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 26194 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
├─26200 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserve...
└─26202 java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava....

Mar 18 09:25:35 nls1-be.it.pccwglobal.com runuser[26200]: pam_unix(runuser:session): session opened for user root by (uid=0)
Mar 18 09:25:35 nls1-be.it.pccwglobal.com logstash[26194]: Starting Logstash Daemon: [ OK ]
Mar 18 09:25:35 nls1-be.it.pccwglobal.com systemd[1]: Started LSB: Logstash.

Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.

If you want to list systemd services use 'systemctl list-unit-files'.
To see services enabled on particular target use
'systemctl list-dependencies [target]'.

elasticsearch 0:off 1:off 2:off 3:on 4:off 5:on 6:off
logstash 0:off 1:off 2:off 3:on 4:off 5:on 6:off