Exact same. These two servers were built same time and I did a full yum update before I installed NLS. I ran the command to be sure, and yes, exact same output.mcapra wrote:Cool, send it on over if/when it crashes. Can I also see the java -version output on the other node that is functional?
New install Logstash failing
Re: New install Logstash failing
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
Re: New install Logstash failing
Seeing this in the screen session now:
Code: Select all
Sep 21, 2016 3:31:21 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] loaded [], sites []
Sep 21, 2016 3:31:33 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] loaded [], sites []
Sep 21, 2016 3:31:33 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] loaded [], sites []
Sep 21, 2016 3:31:33 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] loaded [], sites []
Sep 21, 2016 3:31:33 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] loaded [], sites []
Sep 21, 2016 3:41:22 PM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [442] handler org.elasticsearch.action.TransportActionNodeProxy$1@17c1dd3, error [true], resetting
Sep 21, 2016 3:41:22 PM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [478] handler org.elasticsearch.action.TransportActionNodeProxy$1@32737a46, error [true], resetting
Sep 21, 2016 3:41:23 PM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [427] handler org.elasticsearch.action.TransportActionNodeProxy$1@5a329f3b, error [true], resetting
Sep 21, 2016 3:41:23 PM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [443] handler org.elasticsearch.action.TransportActionNodeProxy$1@2473a70, error [true], resetting
Sep 21, 2016 3:41:23 PM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [480] handler org.elasticsearch.action.TransportActionNodeProxy$1@eb168bf, error [true], resetting
Sep 21, 2016 3:41:24 PM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [429] handler org.elasticsearch.action.TransportActionNodeProxy$1@5c76bf7b, error [true], resetting
Sep 21, 2016 3:41:24 PM org.elasticsearch.transport.netty.MessageChannelHandler messageReceived
WARNING: [f5c0ae96-4dd5-4f16-bd5a-a73b245f7694] Message not fully read (response) for [445] handler org.elasticsearch.action.TransportActionNodeProxy$1@2de6d497, error [true], resetting
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
Re: New install Logstash failing
From all nodes, can you share the output of:
Code: Select all
curl 'localhost:9200/_nodes/jvm?pretty'Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: New install Logstash failing
Bad Node
Good Node
EDIT: the hostname with "365" is the bad node
Code: Select all
{
"cluster_name" : "1c6d88c6-e9ca-480d-8c85-8bd97ef41adf",
"nodes" : {
"XpBH4IG2S4aMC8qR-gl3Dg" : {
"name" : "91c49ceb-0d7c-4e27-a80a-b9036f0d3375",
"transport_address" : "inet[/172.25.14.21:9300]",
"host" : "hdfs-chi-nag01.hdfs.itciss.com",
"ip" : "172.25.14.21",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[localhost/127.0.0.1:9200]",
"attributes" : {
"max_local_storage_nodes" : "1"
},
"jvm" : {
"pid" : 6763,
"version" : "1.7.0_111",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.111-b01",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1474470457548,
"mem" : {
"heap_init_in_bytes" : 8362393600,
"heap_max_in_bytes" : 8328577024,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 8328577024
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
}
},
"JJlVSkeLSjCWkQJ8t9TT4g" : {
"name" : "f5c0ae96-4dd5-4f16-bd5a-a73b245f7694",
"transport_address" : "inet[/172.25.12.28:9300]",
"host" : "hdfs-365-nag01.hdfs.itciss.com",
"ip" : "172.25.12.28",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[localhost/127.0.0.1:9200]",
"attributes" : {
"max_local_storage_nodes" : "1"
},
"jvm" : {
"pid" : 28233,
"version" : "1.7.0_111",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.111-b01",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1474470455064,
"mem" : {
"heap_init_in_bytes" : 8360296448,
"heap_max_in_bytes" : 8326479872,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 8326479872
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
}
}
}
}
Code: Select all
{
"cluster_name" : "1c6d88c6-e9ca-480d-8c85-8bd97ef41adf",
"nodes" : {
"XpBH4IG2S4aMC8qR-gl3Dg" : {
"name" : "91c49ceb-0d7c-4e27-a80a-b9036f0d3375",
"transport_address" : "inet[/172.25.14.21:9300]",
"host" : "hdfs-chi-nag01.hdfs.itciss.com",
"ip" : "172.25.14.21",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[localhost/127.0.0.1:9200]",
"attributes" : {
"max_local_storage_nodes" : "1"
},
"jvm" : {
"pid" : 6763,
"version" : "1.7.0_111",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.111-b01",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1474470457548,
"mem" : {
"heap_init_in_bytes" : 8362393600,
"heap_max_in_bytes" : 8328577024,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 8328577024
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
}
},
"JJlVSkeLSjCWkQJ8t9TT4g" : {
"name" : "f5c0ae96-4dd5-4f16-bd5a-a73b245f7694",
"transport_address" : "inet[/172.25.12.28:9300]",
"host" : "hdfs-365-nag01.hdfs.itciss.com",
"ip" : "172.25.12.28",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[localhost/127.0.0.1:9200]",
"attributes" : {
"max_local_storage_nodes" : "1"
},
"jvm" : {
"pid" : 28233,
"version" : "1.7.0_111",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.111-b01",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1474470455064,
"mem" : {
"heap_init_in_bytes" : 8360296448,
"heap_max_in_bytes" : 8326479872,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 8326479872
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
}
}
}
}
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
Re: New install Logstash failing
Is the cluster behind a load-balancer or RRDNS of any sort?
There could also be corrupted shards within the cluster causing this. From "Good Node" share the outputs of:
There could also be corrupted shards within the cluster causing this. From "Good Node" share the outputs of:
Code: Select all
curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
curl -XGET http://localhost:9200/_cat/shards?vFormer Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: New install Logstash failing
I didn't start sending to it until today(except for itself), and was crashed before adding anything today, so sure shouldn't have anything corrupt. Also, no, no RRDNS or F5(yet).
Command output:
And then command output after I deleted the Indices as I cared 0% about the data and re-ran the commands and let the cluster get back to green(about 30 seconds)
This was all from the "good" node
Command output:
Code: Select all
[root@hdfs-chi-nag01 ~]# curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
{"cluster_name":"1c6d88c6-e9ca-480d-8c85-8bd97ef41adf","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":81,"active_shards":162,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"indices":{"logstash-2016.09.17":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.08":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.09":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.18":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.19":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.13":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"nagioslogserver":{"status":"green","number_of_shards":1,"number_of_replicas":1,"active_primary_shards":1,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.14":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.15":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.16":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.10":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.11":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.12":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"nagioslogserver_log":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.21":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.20":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"kibana-int":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}}}}[root@hdfs-chi-nag01 ~]# curl -XGET http://localhost:9200/_cat/shards?v
index shard prirep state docs store ip node
logstash-2016.09.17 2 p STARTED 5876 867.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.17 2 r STARTED 5876 867.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.17 0 p STARTED 5765 890.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.17 0 r STARTED 5765 890.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.17 3 p STARTED 5865 857.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.17 3 r STARTED 5865 857.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.17 1 p STARTED 5715 880kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.17 1 r STARTED 5715 880kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.17 4 p STARTED 5923 911.8kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.17 4 r STARTED 5923 911.8kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.08 4 p STARTED 1061 219kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.08 4 r STARTED 1061 219kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.08 0 p STARTED 1063 217kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.08 0 r STARTED 1063 217kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.08 3 p STARTED 1084 225kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.08 3 r STARTED 1084 225kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.08 1 p STARTED 1054 223.7kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.08 1 r STARTED 1054 223.7kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.08 2 p STARTED 1108 229kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.08 2 r STARTED 1108 229kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.09 4 p STARTED 4287 720.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.09 4 r STARTED 4287 720.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.09 0 p STARTED 4280 728.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.09 0 r STARTED 4280 728.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.09 3 p STARTED 4175 699.2kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.09 3 r STARTED 4175 699.2kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.09 1 p STARTED 4239 719.8kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.09 1 r STARTED 4239 719.8kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.09 2 p STARTED 4387 745.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.09 2 r STARTED 4387 745.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.18 4 p STARTED 5809 892.9kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.18 4 r STARTED 5809 892.9kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.18 0 p STARTED 5814 853.2kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.18 0 r STARTED 5814 853.2kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.18 3 p STARTED 5882 907.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.18 3 r STARTED 5882 907.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.18 1 p STARTED 5751 891.5kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.18 1 r STARTED 5751 891.5kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.18 2 p STARTED 5793 889.9kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.18 2 r STARTED 5793 889.9kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.19 2 p STARTED 5872 896.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.19 2 r STARTED 5872 896.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.19 0 p STARTED 5827 897kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.19 0 r STARTED 5827 897kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.19 3 p STARTED 5896 908.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.19 3 r STARTED 5896 908.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.19 1 p STARTED 5901 910.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.19 1 r STARTED 5901 910.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.19 4 p STARTED 5632 853kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.19 4 r STARTED 5632 853kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.13 4 p STARTED 5816 856.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.13 4 r STARTED 5816 856.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.13 0 p STARTED 5776 873.7kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.13 0 r STARTED 5776 873.7kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.13 3 p STARTED 5852 889.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.13 3 r STARTED 5852 889.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.13 1 p STARTED 5792 891kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.13 1 r STARTED 5792 891kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.13 2 p STARTED 5845 874.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.13 2 r STARTED 5845 874.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver 0 p STARTED 29 71.9kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver 0 r STARTED 29 80kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.14 4 p STARTED 5869 909.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.14 4 r STARTED 5869 909.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.14 0 p STARTED 5816 891.9kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.14 0 r STARTED 5816 891.9kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.14 3 p STARTED 5913 908.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.14 3 r STARTED 5913 908.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.14 1 p STARTED 5784 897.7kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.14 1 r STARTED 5784 897.7kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.14 2 p STARTED 5849 907.1kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.14 2 r STARTED 5849 907.1kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.15 2 p STARTED 5867 902.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.15 2 r STARTED 5867 902.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.15 0 p STARTED 6004 905.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.15 0 r STARTED 6004 905.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.15 3 p STARTED 5648 830.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.15 3 r STARTED 5648 830.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.15 1 p STARTED 5845 860kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.15 1 r STARTED 5845 860kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.15 4 p STARTED 5855 888.1kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.15 4 r STARTED 5855 888.1kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.16 4 p STARTED 5784 892.2kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.16 4 r STARTED 5784 892.2kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.16 0 p STARTED 5863 862kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.16 0 r STARTED 5863 862kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.16 3 p STARTED 5855 865kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.16 3 r STARTED 5855 865kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.16 1 p STARTED 5775 842.8kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.16 1 r STARTED 5775 842.8kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.16 2 p STARTED 5827 895.2kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.16 2 r STARTED 5827 895.2kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.10 2 p STARTED 5910 879.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.10 2 r STARTED 5910 879.4kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.10 0 p STARTED 5792 848.7kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.10 0 r STARTED 5792 848.7kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.10 3 p STARTED 5913 861.1kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.10 3 r STARTED 5913 861.1kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.10 1 p STARTED 5824 889.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.10 1 r STARTED 5824 889.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.10 4 p STARTED 5799 871.9kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.10 4 r STARTED 5799 871.9kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.11 4 p STARTED 5855 857.7kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.11 4 r STARTED 5855 857.7kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.11 0 p STARTED 5814 851.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.11 0 r STARTED 5814 851.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.11 3 p STARTED 5818 845.5kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.11 3 r STARTED 5818 845.5kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.11 1 p STARTED 5858 902kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.11 1 r STARTED 5858 902kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.11 2 p STARTED 5927 918.1kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.11 2 r STARTED 5927 918.1kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.12 4 p STARTED 5826 854.8kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.12 4 r STARTED 5826 854.8kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.12 0 p STARTED 5875 864.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.12 0 r STARTED 5875 864.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.12 3 p STARTED 5795 899.1kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.12 3 r STARTED 5795 899.1kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.12 1 p STARTED 5840 906.9kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.12 1 r STARTED 5840 906.9kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.12 2 p STARTED 5774 863.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.12 2 r STARTED 5774 863.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 2 p STARTED 11262 1.8mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 2 r STARTED 11262 1.9mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 0 p STARTED 11318 1.9mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 0 r STARTED 11318 1.9mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 3 p STARTED 11423 1.9mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 3 r STARTED 11423 1.9mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 1 p STARTED 11284 1.9mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 1 r STARTED 11284 1.9mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 4 p STARTED 11350 1.9mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 4 r STARTED 11350 1.9mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 2 p STARTED 24191 9.2mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 2 r STARTED 24191 9.2mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 0 p STARTED 24093 9.1mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 0 r STARTED 24093 9.2mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 3 p STARTED 23979 9.1mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 3 r STARTED 23979 9.2mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 1 p STARTED 23984 9.2mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 1 r STARTED 23984 9.2mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 4 p STARTED 23883 9.2mb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 4 r STARTED 23883 9.2mb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.20 4 p STARTED 5892 988.7kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.20 4 r STARTED 5892 988.7kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.20 0 p STARTED 5871 883.5kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.20 0 r STARTED 5871 883.5kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.20 3 p STARTED 5775 963.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.20 3 r STARTED 5775 949.7kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.20 1 p STARTED 5806 846kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.20 1 r STARTED 5806 846kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.20 2 p STARTED 5834 874.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.20 2 r STARTED 5834 874.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 4 p STARTED 0 144b 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 4 r STARTED 0 144b 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 0 p STARTED 2 27.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 0 r STARTED 2 27.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 3 p STARTED 3 21.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 3 r STARTED 3 21.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 1 p STARTED 0 144b 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 1 r STARTED 0 144b 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 2 p STARTED 1 11.8kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 2 r STARTED 1 11.8kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
Code: Select all
[root@hdfs-chi-nag01 ~]# curl -XGET http://localhost:9200/_cat/shards?v
index shard prirep state docs store ip node
nagioslogserver_log 4 p STARTED 11351 973.2kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 4 r STARTED 11351 991.8kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 0 p STARTED 11318 998kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 0 r STARTED 11318 991.2kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 3 p STARTED 11425 1008.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 3 r STARTED 11425 997.2kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 1 p STARTED 11286 976.8kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 1 r STARTED 11286 989.7kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver_log 2 p STARTED 11263 973.8kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver_log 2 r STARTED 11263 988.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 2 r STARTED 8 31.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 2 p STARTED 9 31.5kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 0 r STARTED 12 46.5kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 0 p STARTED 14 46.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 3 r STARTED 4 28.4kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 3 p STARTED 6 28.5kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 1 r STARTED 2 8.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 1 p STARTED 2 8.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 4 r STARTED 2 14.1kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
logstash-2016.09.21 4 p STARTED 2 14.2kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
nagioslogserver 0 p STARTED 29 76.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
nagioslogserver 0 r STARTED 29 128kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 2 p STARTED 1 11.8kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 2 r STARTED 1 11.8kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 0 p STARTED 2 27.6kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 0 r STARTED 2 27.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 3 p STARTED 3 21.3kb 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 3 r STARTED 3 21.3kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 1 p STARTED 0 144b 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 1 r STARTED 0 144b 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
kibana-int 4 p STARTED 0 144b 172.25.14.21 91c49ceb-0d7c-4e27-a80a-b9036f0d3375
kibana-int 4 r STARTED 0 144b 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
[root@hdfs-chi-nag01 ~]# curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
{"cluster_name":"1c6d88c6-e9ca-480d-8c85-8bd97ef41adf","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":16,"active_shards":32,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"indices":{"nagioslogserver_log":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2016.09.21":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"nagioslogserver":{"status":"green","number_of_shards":1,"number_of_replicas":1,"active_primary_shards":1,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"kibana-int":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}}}}2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
Re: New install Logstash failing
So we're allocating shards to the "Bad Node" just fine, but unable to write data to them. Very strange. Have you tried removing "Bad Node" from the cluster and adding a fresh node?
Can I see the contents of the logstash/elasticsearch configurations on the "Bad Node"?
Can I see the contents of the logstash/elasticsearch configurations on the "Bad Node"?
Code: Select all
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*.conf
cat /usr/local/nagioslogserver/elasticsearch/config/*.yml
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: New install Logstash failing
Wait, what? Did I explain somethign poorly in my last post or did you get that from looking at my pasted stuff?mcapra wrote:So we're allocating shards to the "Bad Node" just fine, but unable to write data to them.
new output requested:
Code: Select all
[root@hdfs-365-nag01 tmp]# cat /usr/local/nagioslogserver/logstash/etc/conf.d/*.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 21 Sep 2016 07:59:41 -0700
#
#
# Global inputs
#
input {
syslog {
type => 'syslog'
port => 5544
}
tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}
tcp {
type => 'import_raw'
tags => 'import_raw'
port => 2056
}
tcp {
type => 'import_json'
tags => 'import_json'
port => 2057
codec => json
}
}
#
# Local inputs
#
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 21 Sep 2016 07:59:41 -0700
#
#
# Global filters
#
filter {
if [program] == 'apache_access' {
grok {
match => [ 'message', '%{COMBINEDAPACHELOG}']
}
date {
match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
}
mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
}
}
if [program] == 'apache_error' {
grok {
match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
}
mutate {
replace => [ 'type', 'apache_error' ]
}
}
dns {
reverse => [ "host" ]
action => [ "replace" ]
add_tag => [ "dns" ]
}
if ([message] == "default send string") {
drop{}
}
}
#
# Local filters
#
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 21 Sep 2016 07:59:42 -0700
#
#
# Required output for Nagios Log Server
#
output {
elasticsearch {
cluster => '1c6d88c6-e9ca-480d-8c85-8bd97ef41adf'
host => 'localhost'
document_type => '%{type}'
node_name => 'f5c0ae96-4dd5-4f16-bd5a-a73b245f7694'
protocol => 'transport'
workers => 4
}
}
#
# Global outputs
#
#
# Local outputs
#
[root@hdfs-365-nag01 tmp]# cat /usr/local/nagioslogserver/elasticsearch/config/*.yml
##################### Elasticsearch Configuration Example #####################
# This file contains an overview of various configuration settings,
# targeted at operations staff. Application developers should
# consult the guide at <http://elasticsearch.org/guide>.
#
# The installation procedure is covered at
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.
#
# Elasticsearch comes with reasonable defaults for most settings,
# so you can try it out without bothering with configuration.
#
# Most of the time, these defaults are just fine for running a production
# cluster. If you're fine-tuning your cluster, or wondering about the
# effect of certain configuration option, please _do ask_ on the
# mailing list or IRC channel [http://elasticsearch.org/community].
# Any element in the configuration can be replaced with environment variables
# by placing them in ${...} notation. For example:
#
# node.rack: ${RACK_ENV_VAR}
# For information on supported formats and syntax for the config file, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>
################################### Cluster ###################################
# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: nagios_elasticsearch
#################################### Node #####################################
# Node names are generated dynamically on startup, so you're relieved
# from configuring them manually. You can tie this node to a specific name:
#
# node.name: "Franz Kafka"
# Every node can be configured to allow or deny being eligible as the master,
# and to allow or deny to store the data.
#
# Allow this node to be eligible as a master node (enabled by default):
#
# node.master: true
#
# Allow this node to store data (enabled by default):
#
# node.data: true
# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
# This will be the "workhorse" of your cluster.
#
# node.master: false
# node.data: true
#
# 2. You want this node to only serve as a master: to not store any data and
# to have free resources. This will be the "coordinator" of your cluster.
#
# node.master: true
# node.data: false
#
# 3. You want this node to be neither master nor data node, but
# to act as a "search load balancer" (fetching data from nodes,
# aggregating results, etc.)
#
# node.master: false
# node.data: false
# Use the Cluster Health API [http://localhost:9200/_cluster/health], the
# Node Info API [http://localhost:9200/_nodes] or GUI tools
# such as <http://www.elasticsearch.org/overview/marvel/>,
# <http://github.com/karmi/elasticsearch-paramedic>,
# <http://github.com/lukas-vlcek/bigdesk> and
# <http://mobz.github.com/elasticsearch-head> to inspect the cluster state.
# A node can have generic attributes associated with it, which can later be used
# for customized shard allocation filtering, or allocation awareness. An attribute
# is a simple key value pair, similar to node.key: value, here is an example:
#
# node.rack: rack314
# By default, multiple nodes are allowed to start from the same installation location
# to disable it, set the following:
node.max_local_storage_nodes: 1
#################################### Index ####################################
# You can set a number of options (such as shard/replica options, mapping
# or analyzer definitions, translog settings, ...) for indices globally,
# in this file.
#
# Note, that it makes more sense to configure index settings specifically for
# a certain index, either when creating it or by using the index templates API.
#
# See <http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html> and
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html>
# for more information.
# Set the number of shards (splits) of an index (5 by default):
#
# index.number_of_shards: 5
# Set the number of replicas (additional copies) of an index (1 by default):
#
# index.number_of_replicas: 1
# Note, that for development on a local machine, with small indices, it usually
# makes sense to "disable" the distributed features:
#
# index.number_of_shards: 1
# index.number_of_replicas: 0
# These settings directly affect the performance of index and search operations
# in your cluster. Assuming you have enough machines to hold shards and
# replicas, the rule of thumb is:
#
# 1. Having more *shards* enhances the _indexing_ performance and allows to
# _distribute_ a big index across machines.
# 2. Having more *replicas* enhances the _search_ performance and improves the
# cluster _availability_.
#
# The "number_of_shards" is a one-time setting for an index.
#
# The "number_of_replicas" can be increased or decreased anytime,
# by using the Index Update Settings API.
#
# Elasticsearch takes care about load balancing, relocating, gathering the
# results from nodes, etc. Experiment with different settings to fine-tune
# your setup.
# Use the Index Status API (<http://localhost:9200/A/_status>) to inspect
# the index status.
#################################### Paths ####################################
# Path to directory containing configuration (this file and logging.yml):
#
# path.conf: /path/to/conf
# Path to directory where to store index data allocated for this node.
#
# path.data: /path/to/data
#
# Can optionally include more than one location, causing data to be striped across
# the locations (a la RAID 0) on a file level, favouring locations with most free
# space on creation. For example:
#
# path.data: /path/to/data1,/path/to/data2
# Path to temporary files:
#
# path.work: /path/to/work
# Path to log files:
#
# path.logs: /path/to/logs
# Path to where plugins are installed:
#
# path.plugins: /path/to/plugins
#################################### Plugin ###################################
# If a plugin listed here is not installed for current node, the node will not start.
#
# plugin.mandatory: mapper-attachments,lang-groovy
################################### Memory ####################################
# Elasticsearch performs poorly when JVM starts swapping: you should ensure that
# it _never_ swaps.
#
# Set this property to true to lock the memory:
#
bootstrap.mlockall: true
# Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set
# to the same value, and that the machine has enough memory to allocate
# for Elasticsearch, leaving enough memory for the operating system itself.
#
# You should also make sure that the Elasticsearch process is allowed to lock
# the memory, eg. by using `ulimit -l unlimited`.
############################## Network And HTTP ###############################
# Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens
# on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node
# communication. (the range means that if the port is busy, it will automatically
# try the next port).
# Set the bind address specifically (IPv4 or IPv6):
#
# network.bind_host: 192.168.0.1
# Set the address other nodes will use to communicate with this node. If not
# set, it is automatically derived. It must point to an actual IP address.
#
# network.publish_host: 192.168.0.1
# Set both 'bind_host' and 'publish_host':
#
# network.host: 192.168.0.1
# Set a custom port for the node to node communication (9300 by default):
#
# transport.tcp.port: 9300
# Enable compression for all communication between nodes (disabled by default):
#
transport.tcp.compress: true
# Set a custom port to listen for HTTP traffic:
#
# http.port: 9200
# Set a custom allowed content length:
#
# http.max_content_length: 100mb
# Disable HTTP completely:
#
# http.enabled: false
# Set the HTTP host to listen to
#
http.host: "localhost"
################################### Gateway ###################################
# The gateway allows for persisting the cluster state between full cluster
# restarts. Every change to the state (such as adding an index) will be stored
# in the gateway, and when the cluster starts up for the first time,
# it will read its state from the gateway.
# There are several types of gateway implementations. For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html>.
# The default gateway type is the "local" gateway (recommended):
#
# gateway.type: local
# Settings below control how and when to start the initial recovery process on
# a full cluster restart (to reuse as much local data as possible when using shared
# gateway).
# Allow recovery process after N nodes in a cluster are up:
#
# gateway.recover_after_nodes: 1
# Set the timeout to initiate the recovery process, once the N nodes
# from previous setting are up (accepts time value):
#
# gateway.recover_after_time: 5m
# Set how many nodes are expected in this cluster. Once these N nodes
# are up (and recover_after_nodes is met), begin recovery process immediately
# (without waiting for recover_after_time to expire):
#
# gateway.expected_nodes: 2
############################# Recovery Throttling #############################
# These settings allow to control the process of shards allocation between
# nodes during initial recovery, replica allocation, rebalancing,
# or when adding and removing nodes.
# Set the number of concurrent recoveries happening on a node:
#
# 1. During the initial recovery
#
# cluster.routing.allocation.node_initial_primaries_recoveries: 4
#
# 2. During adding/removing nodes, rebalancing, etc
#
# cluster.routing.allocation.node_concurrent_recoveries: 2
# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
#
# indices.recovery.max_bytes_per_sec: 20mb
# Set to limit the number of open concurrent streams when
# recovering a shard from a peer:
#
# indices.recovery.concurrent_streams: 5
################################## Discovery ##################################
# Discovery infrastructure ensures nodes can be found within a cluster
# and master node is elected. Multicast discovery is the default.
# Set to ensure a node sees N other master eligible nodes to be considered
# operational within the cluster. Its recommended to set it to a higher value
# than 1 when running more than 2 nodes in the cluster.
#
# discovery.zen.minimum_master_nodes: 1
# Set the time to wait for ping responses from other nodes when discovering.
# Set this option to a higher value on a slow or congested network
# to minimize discovery failures:
#
# discovery.zen.ping.timeout: 3s
# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>
# Unicast discovery allows to explicitly control which nodes will be used
# to discover the cluster. It can be used when multicast is not present,
# or to restrict the cluster communication-wise.
#
# 1. Disable multicast discovery (enabled by default):
#
discovery.zen.ping.multicast.enabled: false
#
# 2. Configure an initial list of master nodes in the cluster
# to perform discovery when new nodes (master or data) are started:
#
discovery.zen.ping.unicast.hosts: ["localhost"]
# EC2 discovery allows to use AWS EC2 API in order to perform discovery.
#
# You have to install the cloud-aws plugin for enabling the EC2 discovery.
#
# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>
#
# See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>
# for a step-by-step tutorial.
# GCE discovery allows to use Google Compute Engine API in order to perform discovery.
#
# You have to install the cloud-gce plugin for enabling the GCE discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-gce>.
# Azure discovery allows to use Azure API in order to perform discovery.
#
# You have to install the cloud-azure plugin for enabling the Azure discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-azure>.
################################## Slow Log ##################################
# Shard level query and fetch threshold logging.
#index.search.slowlog.threshold.query.warn: 10s
#index.search.slowlog.threshold.query.info: 5s
#index.search.slowlog.threshold.query.debug: 2s
#index.search.slowlog.threshold.query.trace: 500ms
#index.search.slowlog.threshold.fetch.warn: 1s
#index.search.slowlog.threshold.fetch.info: 800ms
#index.search.slowlog.threshold.fetch.debug: 500ms
#index.search.slowlog.threshold.fetch.trace: 200ms
#index.indexing.slowlog.threshold.index.warn: 10s
#index.indexing.slowlog.threshold.index.info: 5s
#index.indexing.slowlog.threshold.index.debug: 2s
#index.indexing.slowlog.threshold.index.trace: 500ms
################################## GC Logging ################################
#monitor.jvm.gc.young.warn: 1000ms
#monitor.jvm.gc.young.info: 700ms
#monitor.jvm.gc.young.debug: 400ms
#monitor.jvm.gc.old.warn: 10s
#monitor.jvm.gc.old.info: 5s
#monitor.jvm.gc.old.debug: 2s
# you can override this using by setting a system property, for example -Des.logger.level=DEBUG
es.logger.level: INFO
rootLogger: ${es.logger.level}, console, file
logger:
# log action execution errors for easier debugging
action: DEBUG
# reduce the logging for aws, too much is logged under the default INFO
com.amazonaws: WARN
# gateway
#gateway: DEBUG
#index.gateway: DEBUG
# peer shard recovery
#indices.recovery: DEBUG
# discovery
#discovery: TRACE
index.search.slowlog: TRACE, index_search_slow_log_file
index.indexing.slowlog: TRACE, index_indexing_slow_log_file
additivity:
index.search.slowlog: false
index.indexing.slowlog: false
appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
file:
type: file
file: ${path.logs}/${cluster.name}.log
# datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
index_search_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_search_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
index_indexing_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
Re: New install Logstash failing
The output of curl -XGET http://localhost:9200/_cat/shards?v shows shards being assigned to 172.25.12.28 (hdfs-365-nag01.hdfs.itciss.com):BanditBBS wrote: Wait, what? Did I explain somethign poorly in my last post or did you get that from looking at my pasted stuff?
Code: Select all
logstash-2016.09.21 2 p STARTED 9 31.5kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
logstash-2016.09.21 0 p STARTED 14 46.6kb 172.25.12.28 f5c0ae96-4dd5-4f16-bd5a-a73b245f7694
Can we see the contents of your elasticsearch log on the "Bad Node"? Should be located at /var/log/elasticsearch/f5c0ae96-4dd5-4f16-bd5a-a73b245f7694.log
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: New install Logstash failing
Umm
Code: Select all
[root@hdfs-365-nag01 ~]# ll /var/log/elasticsearch/
total 64
-rw-r--r--. 1 nagios users 0 Sep 8 15:48 043f21b0-1d53-4881-8f3e-9c58be028b8d_index_indexing_slowlog.log
-rw-r--r--. 1 nagios users 0 Sep 8 15:48 043f21b0-1d53-4881-8f3e-9c58be028b8d_index_search_slowlog.log
-rw-r--r--. 1 nagios users 0 Sep 10 06:07 043f21b0-1d53-4881-8f3e-9c58be028b8d.log
-rw-r--r--. 1 nagios users 691 Sep 10 06:07 043f21b0-1d53-4881-8f3e-9c58be028b8d.log-20160910.gz
-rw-r--r--. 1 nagios users 0 Sep 8 15:50 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf_index_indexing_slowlog.log
-rw-r--r--. 1 nagios users 0 Sep 8 15:50 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf_index_search_slowlog.log
-rw-r--r--. 1 nagios users 26834 Sep 21 18:00 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log
-rw-r--r--. 1 nagios users 2412 Sep 15 07:44 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160915.gz
-rw-r--r--. 1 nagios users 2251 Sep 16 06:33 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160916.gz
-rw-r--r--. 1 nagios users 2347 Sep 17 06:31 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160917.gz
-rw-r--r--. 1 nagios users 3388 Sep 18 06:12 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160918.gz
-rw-r--r--. 1 nagios users 2503 Sep 19 06:29 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160919.gz
-rw-r--r--. 1 nagios users 3737 Sep 20 06:18 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160920.gz
-rw-r--r--. 1 nagios users 2323 Sep 21 06:50 1c6d88c6-e9ca-480d-8c85-8bd97ef41adf.log-20160921.gz
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github