The username specified does not exist

[muratKURU]

Hi All
I get the error "The username specified does not exist." on Nagios Log Server.
I'm 100% sure that the username nagiosadmin and the password is correct.
Why do I get this error?

[jolson]

How long has this box been running, and have you had any recent trouble with it?

Check if selinux is running:

Code: Select all

sestatus

If so, please turn it off and try again.

Attempt to use this script to change the password for user 'nagiosadmin':

Code: Select all

/usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh -p examplepassword

Does it throw any errors?

I have had this problem before when I was using a 2 instance cluster and they busted apart - please make sure your cluster is healthy:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

What is the 'status'?

We can also take a look at your http error log:

Code: Select all

tail /var/log/httpd/error_log

Let me know if any of the above is helpful. Thanks.

Best,


Jesse

[muratKURU]

Hi
Thank you for the quick reply.
The root disk was full on both server.
I have extended the disks but the Login on one Instanee did not work anymore.
I have removed instance with the problem from the GUI but don't know hat to add it back.

When I run the command "/usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh -p examplepassword" it just hangs. Not feedback at all.

selinux seems to be off.
[root@cnalg346 ~]# sestatus
SELinux status: disabled
[root@cnalg346 ~]#

My cluster is not healthy
How to make it healthy again?

[jolson]

Well, first let's collect some information from both of your instances to try and get them to re-join one another. Please run the following on both nodes:

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_hosts

Code: Select all

cat /usr/local/nagioslogserver/var/node_uuid

Code: Select all

curl 'localhost:9200/_cat/master?v'

The plan of action will be to try and re-make the cluster via the CLI. After that, we'll see what happens to the health and the login.

Get the above information back to me and we'll go from there.

[muratKURU]

cat /usr/local/nagioslogserver/var/cluster_hosts
[root@cnalg347 ~]# cat /usr/local/nagioslogserver/var/cluster_hosts
xx.xx.xx.xx[root@cnalg347 ~]#
[root@cnalg346 ~]# cat /usr/local/nagioslogserver/var/cluster_hosts
xx.xx.xx.xx[root@cnalg346 ~]#

cat /usr/local/nagioslogserver/var/node_uuid
[root@cnalg347 ~]# cat /usr/local/nagioslogserver/var/node_uuid
ceaf4ab8-f364-4049-b1ba-c999163ff36d
[root@cnalg346 ~]# cat /usr/local/nagioslogserver/var/node_uuid
1abaec90-4a41-4231-b96b-6cf796e1b1bb

curl 'localhost:9200/_cat/master?v'
[root@cnalg347 ~]# curl 'localhost:9200/_cat/master?v'
id host ip node
4I9sbVrySRichpGfeeWqjw cnalg347 xx.xx.xx.xx ceaf4ab8-f364-4049-b1ba-c999163ff36d
[root@cnalg346 ~]# curl 'localhost:9200/_cat/master?v'
id host ip node
7z_eZ4xOQkK-CmXJvJdA0g cnalg346 xx.xx.xx.xx 1abaec90-4a41-4231-b96b-6cf796e1b1bb

[jolson]

Right now, each of your instances thinks it's the master of itself. You will need to edit the cluster_hosts file on each node to read as follows:

Code: Select all

Node 1: cat /usr/local/nagioslogserver/var/cluster_hosts
localhost
Node1IPAddress
Node2IPAddress

Code: Select all

Node 2: cat /usr/local/nagioslogserver/var/cluster_hosts
localhost
Node2IPAddress
Node1IPAddress

After these modifications are made, please restart elasticsearch on both hosts - they should attempt to join one another.

Code: Select all

service elasticsearch restart

[muratKURU]

Hi jolson
It worked.
But the "Cluster Health" is still red.
Will it take a while until it turns to green?

[jolson]

Glad to hear that worked - are you able to log into the GUI?

Please run the following commands - using the output we can work out which indices may be corrupted.

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
curl -XGET http://localhost:9200/_cat/shards

Also, I would like a screenshot of your 'Administration -> Cluster Status' page please. If any of the information is too sensitive, feel free to PM it to me. Thanks!

[muratKURU]

[root@cnalg346 ~]# curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
{"cluster_name":"f10efa0a-a8e4-4c2a-8979-95c9ff99fdca","status":"red","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":41,"active_shards":82,"relocating_shards":0,"initializing_shards":4,"unassigned_shards":46,"indices":{"logstash-2015.04.21":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.29":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":10,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"nagioslogserver_log":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.12.11":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.20":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.28":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":10,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"logstash-2015.04.23":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":1,"active_shards":2,"relocating_shards":0,"initializing_shards":2,"unassigned_shards":6,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.22":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":2,"active_shards":4,"relocating_shards":0,"initializing_shards":2,"unassigned_shards":4,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1}}},"logstash-2015.04.25":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.24":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"nagioslogserver":{"status":"green","number_of_shards":1,"number_of_replicas":1,"active_primary_shards":1,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.27":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":10,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"logstash-2015.04.26":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":2,"active_shards":4,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":6,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"kibana-int":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}}}}

[root@cnalg346 ~]url -XGET http://localhost:9200/_cat/shards
logstash-2015.04.29 4 p UNASSIGNED
logstash-2015.04.29 4 r UNASSIGNED
logstash-2015.04.29 0 p UNASSIGNED
logstash-2015.04.29 0 r UNASSIGNED
logstash-2015.04.29 3 p UNASSIGNED
logstash-2015.04.29 3 r UNASSIGNED
logstash-2015.04.29 1 p UNASSIGNED
logstash-2015.04.29 1 r UNASSIGNED
logstash-2015.04.29 2 p UNASSIGNED
logstash-2015.04.29 2 r UNASSIGNED
logstash-2015.04.21 4 r STARTED 263409 270.9mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 4 p STARTED 263409 270.9mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.21 0 r STARTED 263870 269.5mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 0 p STARTED 263870 269.5mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.21 3 r STARTED 263833 271mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 3 p STARTED 263833 271mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.21 1 r STARTED 264501 272.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 1 p STARTED 264501 272.1mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.21 2 r STARTED 264483 271.2mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 2 p STARTED 264483 271.2mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 2 r STARTED 272654 278.4mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 2 p STARTED 272654 278.4mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 0 r STARTED 272692 278.2mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 0 p STARTED 272692 278.2mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 3 r STARTED 273105 279.5mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 3 p STARTED 273105 279.5mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 1 r STARTED 272555 277.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 1 p STARTED 272555 277.1mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 4 r STARTED 272291 278mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 4 p STARTED 272291 278mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.28 4 p UNASSIGNED
logstash-2015.04.28 4 r UNASSIGNED
logstash-2015.04.28 0 p UNASSIGNED
logstash-2015.04.28 0 r UNASSIGNED
logstash-2015.04.28 3 p UNASSIGNED
logstash-2015.04.28 3 r UNASSIGNED
logstash-2015.04.28 1 p UNASSIGNED
logstash-2015.04.28 1 r UNASSIGNED
logstash-2015.04.28 2 p UNASSIGNED
logstash-2015.04.28 2 r UNASSIGNED
logstash-2015.04.23 2 p INITIALIZING 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.23 2 r UNASSIGNED
logstash-2015.04.23 0 p INITIALIZING 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.23 0 r UNASSIGNED
logstash-2015.04.23 3 p UNASSIGNED
logstash-2015.04.23 3 r UNASSIGNED
logstash-2015.04.23 1 p UNASSIGNED
logstash-2015.04.23 1 r UNASSIGNED
logstash-2015.04.23 4 r STARTED 382 570.2kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.23 4 p STARTED 382 570.2kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 4 p INITIALIZING 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 4 r UNASSIGNED
logstash-2015.04.22 0 p INITIALIZING 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 0 r UNASSIGNED
logstash-2015.04.22 3 r STARTED 132540 135.2mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.22 3 p STARTED 132540 135.2mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 1 p INITIALIZING 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.22 1 r UNASSIGNED
logstash-2015.04.22 2 r STARTED 131500 134.2mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.22 2 p STARTED 131500 134.2mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 4 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 4 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 0 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 0 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 3 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 3 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 1 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 1 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 2 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 2 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 2 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 2 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 0 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 0 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 3 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 3 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 1 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 1 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 4 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 4 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver 0 r STARTED 56 114.9kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver 0 p STARTED 56 114.9kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.27 2 p UNASSIGNED
logstash-2015.04.27 2 r UNASSIGNED
logstash-2015.04.27 0 p UNASSIGNED
logstash-2015.04.27 0 r UNASSIGNED
logstash-2015.04.27 3 p UNASSIGNED
logstash-2015.04.27 3 r UNASSIGNED
logstash-2015.04.27 1 p UNASSIGNED
logstash-2015.04.27 1 r UNASSIGNED
logstash-2015.04.27 4 p UNASSIGNED
logstash-2015.04.27 4 r UNASSIGNED
logstash-2015.04.26 2 p UNASSIGNED
logstash-2015.04.26 2 r UNASSIGNED
logstash-2015.04.26 0 r STARTED 1 7.7kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.26 0 p STARTED 1 7.7kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.26 3 p UNASSIGNED
logstash-2015.04.26 3 r UNASSIGNED
logstash-2015.04.26 1 r STARTED 3 9.2kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.26 1 p STARTED 3 9.2kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.26 4 p UNASSIGNED
logstash-2015.04.26 4 r UNASSIGNED
nagioslogserver_log 2 r STARTED 79925 18.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 2 p STARTED 79925 18.1mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver_log 0 r STARTED 79984 18.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 0 p STARTED 79984 39.7mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver_log 3 r STARTED 80069 18.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 3 p STARTED 80069 18.1mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver_log 1 r STARTED 79464 18mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 1 p STARTED 79464 18mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver_log 4 r STARTED 79667 18mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 4 p STARTED 79667 18mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 4 r STARTED 84 91.4kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 4 p STARTED 84 91.4kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 0 r STARTED 78 89.4kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 0 p STARTED 78 89.4kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 3 r STARTED 62 82.2kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 3 p STARTED 62 82.2kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 1 r STARTED 77 83.4kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 1 p STARTED 77 83.4kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 2 r STARTED 79 90.8kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 2 p STARTED 79 90.8kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 2 r STARTED 5 60.9kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 2 p STARTED 5 60.9kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 0 r STARTED 4 50.8kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 0 p STARTED 4 50.8kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 3 r STARTED 5 48.7kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 3 p STARTED 5 48.8kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 1 r STARTED 4 48.5kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 1 p STARTED 4 48.5kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 4 r STARTED 3 35.9kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 4 p STARTED 3 35.9kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d

[root@cnalg347 ~]# curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards'
{"cluster_name":"f10efa0a-a8e4-4c2a-8979-95c9ff99fdca","status":"red","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":41,"active_shards":82,"relocating_shards":0,"initializing_shards":4,"unassigned_shards":46,"indices":{"nagioslogserver_log":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.29":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":10,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"logstash-2015.04.21":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.28":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":10,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"logstash-2015.04.20":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.12.11":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.23":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":1,"active_shards":2,"relocating_shards":0,"initializing_shards":2,"unassigned_shards":6,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.22":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":2,"active_shards":4,"relocating_shards":0,"initializing_shards":2,"unassigned_shards":4,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"logstash-2015.04.25":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"nagioslogserver":{"status":"green","number_of_shards":1,"number_of_replicas":1,"active_primary_shards":1,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.24":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"logstash-2015.04.27":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":10,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"1":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"logstash-2015.04.26":{"status":"red","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":2,"active_shards":4,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":6,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"3":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2},"4":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}},"kibana-int":{"status":"green","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":10,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}}}}

[root@cnalg347 ~]url -XGET http://localhost:9200/_cat/shards
logstash-2015.04.21 4 r STARTED 263409 270.9mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 4 p STARTED 263409 270.9mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.21 0 r STARTED 263870 269.5mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 0 p STARTED 263870 269.5mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.21 3 r STARTED 263833 271mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 3 p STARTED 263833 271mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.21 1 r STARTED 264501 272.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 1 p STARTED 264501 272.1mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.21 2 r STARTED 264483 271.2mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.21 2 p STARTED 264483 271.2mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.29 4 p UNASSIGNED
logstash-2015.04.29 4 r UNASSIGNED
logstash-2015.04.29 0 p UNASSIGNED
logstash-2015.04.29 0 r UNASSIGNED
logstash-2015.04.29 3 p UNASSIGNED
logstash-2015.04.29 3 r UNASSIGNED
logstash-2015.04.29 1 p UNASSIGNED
logstash-2015.04.29 1 r UNASSIGNED
logstash-2015.04.29 2 p UNASSIGNED
logstash-2015.04.29 2 r UNASSIGNED
logstash-2015.04.28 4 p UNASSIGNED
logstash-2015.04.28 4 r UNASSIGNED
logstash-2015.04.28 0 p UNASSIGNED
logstash-2015.04.28 0 r UNASSIGNED
logstash-2015.04.28 3 p UNASSIGNED
logstash-2015.04.28 3 r UNASSIGNED
logstash-2015.04.28 1 p UNASSIGNED
logstash-2015.04.28 1 r UNASSIGNED
logstash-2015.04.28 2 p UNASSIGNED
logstash-2015.04.28 2 r UNASSIGNED
logstash-2015.04.20 2 r STARTED 272654 278.4mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 2 p STARTED 272654 278.4mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 0 r STARTED 272692 278.2mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 0 p STARTED 272692 278.2mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 3 r STARTED 273105 279.5mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 3 p STARTED 273105 279.5mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 1 r STARTED 272555 277.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 1 p STARTED 272555 277.1mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.20 4 r STARTED 272291 278mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.20 4 p STARTED 272291 278mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.23 2 p UNASSIGNED
logstash-2015.04.23 2 r UNASSIGNED
logstash-2015.04.23 0 p INITIALIZING 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.23 0 r UNASSIGNED
logstash-2015.04.23 3 p UNASSIGNED
logstash-2015.04.23 3 r UNASSIGNED
logstash-2015.04.23 1 p UNASSIGNED
logstash-2015.04.23 1 r UNASSIGNED
logstash-2015.04.23 4 r STARTED 382 570.2kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.23 4 p STARTED 382 570.2kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 4 p INITIALIZING 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 4 r UNASSIGNED
logstash-2015.04.22 0 p INITIALIZING 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 0 r UNASSIGNED
logstash-2015.04.22 3 r STARTED 132540 135.2mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.22 3 p STARTED 132540 135.2mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 1 p INITIALIZING 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.22 1 r UNASSIGNED
logstash-2015.04.22 2 r STARTED 131500 134.2mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.22 2 p STARTED 131500 134.2mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 4 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 4 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 0 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 0 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 3 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 3 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 1 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 1 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.25 2 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.25 2 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 2 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 2 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 0 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 0 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 3 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 3 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 1 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 1 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.24 4 r STARTED 0 79b 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.24 4 p STARTED 0 115b 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver 0 r STARTED 56 114.9kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver 0 p STARTED 56 114.9kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.27 4 p UNASSIGNED
logstash-2015.04.27 4 r UNASSIGNED
logstash-2015.04.27 0 p UNASSIGNED
logstash-2015.04.27 0 r UNASSIGNED
logstash-2015.04.27 3 p UNASSIGNED
logstash-2015.04.27 3 r UNASSIGNED
logstash-2015.04.27 1 p UNASSIGNED
logstash-2015.04.27 1 r UNASSIGNED
logstash-2015.04.27 2 p UNASSIGNED
logstash-2015.04.27 2 r UNASSIGNED
logstash-2015.04.26 4 p UNASSIGNED
logstash-2015.04.26 4 r UNASSIGNED
logstash-2015.04.26 0 r STARTED 1 7.7kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.26 0 p STARTED 1 7.7kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.26 3 p UNASSIGNED
logstash-2015.04.26 3 r UNASSIGNED
logstash-2015.04.26 1 r STARTED 3 9.2kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.04.26 1 p STARTED 3 9.2kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.04.26 2 p UNASSIGNED
logstash-2015.04.26 2 r UNASSIGNED
nagioslogserver_log 2 r STARTED 79926 18.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 2 p STARTED 79926 18.1mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver_log 0 r STARTED 79986 18.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 0 p STARTED 79986 39.7mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver_log 3 r STARTED 80072 18.1mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 3 p STARTED 80072 18.1mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver_log 1 r STARTED 79464 18mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 1 p STARTED 79464 18mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
nagioslogserver_log 4 r STARTED 79669 18mb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
nagioslogserver_log 4 p STARTED 79669 18mb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 4 r STARTED 84 91.4kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 4 p STARTED 84 91.4kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 0 r STARTED 78 89.4kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 0 p STARTED 78 89.4kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 3 r STARTED 62 82.2kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 3 p STARTED 62 82.2kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 1 r STARTED 77 83.4kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 1 p STARTED 77 83.4kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
logstash-2015.12.11 2 r STARTED 79 90.8kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
logstash-2015.12.11 2 p STARTED 79 90.8kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 2 r STARTED 5 60.9kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 2 p STARTED 5 60.9kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 0 r STARTED 4 50.8kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 0 p STARTED 4 50.8kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 3 r STARTED 5 48.7kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 3 p STARTED 5 48.8kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 1 r STARTED 4 48.5kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 1 p STARTED 4 48.5kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d
kibana-int 4 r STARTED 3 35.9kb 10.20.98.18 1abaec90-4a41-4231-b96b-6cf796e1b1bb
kibana-int 4 p STARTED 3 35.9kb 10.20.98.25 ceaf4ab8-f364-4049-b1ba-c999163ff36d

[jolson]

There are quite a few indices that have issues. We will need to remove them to get this cluster back to a green state. If you have a backup in place, you are free to restore the backups after the removal procedure. If you don't have a backup system in place, consider setting it up: http://assets.nagios.com/downloads/nagi ... enance.pdf

The bad indices are as follows:
logstash-2015.04.29
logstash-2015.04.28
logstash-2015.04.27
logstash-2015.04.26
logstash-2015.04.23
logstash-2015.04.22

We can either wait for them to *hopefully* re-initialize themselves, or we can remove them. I recommend waiting a day or so, and removing them if the health is not improving:

Code: Select all

curl -XDELETE 'http://localhost:9200/logstash-2015.04.29/'
curl -XDELETE 'http://localhost:9200/logstash-2015.04.28/'
curl -XDELETE 'http://localhost:9200/logstash-2015.04.27/'
curl -XDELETE 'http://localhost:9200/logstash-2015.04.26/'
curl -XDELETE 'http://localhost:9200/logstash-2015.04.23/'
curl -XDELETE 'http://localhost:9200/logstash-2015.04.22/'