Shards stay unassigned after yum update

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:

Shards stay unassigned after yum update

Post by WillemDH »

Hello,

I'm having issues with my NLS cluster after a full yum update on one of the nodes. The shards of one of my nodes stay unassigned. Check the screenshot. Tried another reboot, tried restarting logstash and elasticsearch services.

This is the list of updates I installed:

Code: Select all

Updated:
bind-libs.x86_64 32:9.8.2-0.37.rc1.el6_7.4   
bind-utils.x86_64 32:9.8.2-0.37.rc1.el6_7.4  
gdk-pixbuf2.x86_64 0:2.24.1-6.el6_7          
httpd.x86_64 0:2.2.15-47.el6.centos                  
httpd-tools.x86_64 0:2.2.15-47.el6.centos 
kernel-firmware.noarch 0:2.6.32-573.3.1.el6  
kernel-headers.x86_64 0:2.6.32-573.3.1.el6  
libXfont.x86_64 0:1.4.5-5.el6_7              
net-snmp.x86_64 1:5.5-54.el6_7.1                    
net-snmp-libs.x86_64 1:5.5-54.el6_7.1     
net-snmp-utils.x86_64 1:5.5-54.el6_7.1       
nss-softokn.x86_64 0:3.14.3-23.el6_7         
nss-softokn-freebl.x86_64 0:3.14.3-23.el6_7  
openssh.x86_64 0:5.3p1-112.el6_7                     
openssh-clients.x86_64 0:5.3p1-112.el6_7  
openssh-server.x86_64 0:5.3p1-112.el6_7      
pam.x86_64 0:1.1.1-20.el6_7.1                
selinux-policy.noarch 0:3.7.19-279.el6_7.5   
selinux-policy-targeted.noarch 0:3.7.19-279.el6_7.5  
sqlite.x86_64 0:3.6.20-1.el6_7.2          
sudo.x86_64 0:1.8.6p3-20.el6_7               
tzdata.noarch 0:2015f-1.el6                  
tzdata-java.noarch 0:2015f-1.el6            
I have a snapshot I could restore, but as it seems the logs are still showing and handled 'normally', I decided to not restore the snapshot for now. What can I do to make NLS re-assign the unassigned shards?

Code: Select all

curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "ee9e60a0-f4cb-41ec-a97f-8f17434b748e",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 171,
  "active_shards" : 171,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 171,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}
Both nodes are still on R2.1. The second node has not been patched yet, as I would like to solve the issue on the first node before that. I also noticed that when I make configuration changes, the Save and Apply button stay greyed out.

Grtz

Willem
You do not have the required permissions to view the files attached to this post.
Nagios XI 5.8.1
https://outsideit.net
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Shards stay unassigned after yum update

Post by jolson »

I have suspicion about the following packages:

Code: Select all

kernel-headers.x86_64 0:2.6.32-573.3.1.el6
sqlite.x86_64 0:3.6.20-1.el6_7.2
tzdata.noarch 0:2015f-1.el6  
tzdata-java.noarch 0:2015f-1.el6

First, let's get your node information:

Code: Select all

curl 'localhost:9200/_cat/nodes?v'
Let's see which nodes the shards have settled on:

Code: Select all

curl -s localhost:9200/_cat/shards
Any errors in elasticsearch's log?

Code: Select all

tail -n200 /var/log/elasticsearch/*.log
Let me know. It's possible that an elasticsearch restart may be enough to resolve whatever is going on here.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:

Re: Shards stay unassigned after yum update

Post by WillemDH »

Hello,

Thanks for the quick reply.

Code: Select all

curl 'localhost:9200/_cat/nodes?v'
host        ip        heap.percent ram.percent load node.role master name
log02 127.0.0.1           40          65 0.07 d         *      95f9ab14-da22-4144-bb0b-6bbc5662115c
log01 127.0.0.1            5          68 0.00 d         m      c4d16075-9bc2-4095-9f00-e7de7f96930c

Code: Select all

 curl -s localhost:9200/_cat/shards
logstash-2015.08.10 2 p STARTED    1165502  792.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.10 2 r UNASSIGNED
logstash-2015.08.10 0 p STARTED    1165299  791.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.10 0 r UNASSIGNED
logstash-2015.08.10 3 p STARTED    1165922  792.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.10 3 r UNASSIGNED
logstash-2015.08.10 1 p STARTED    1165703  792.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.10 1 r UNASSIGNED
logstash-2015.08.10 4 p STARTED    1166171    793mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.10 4 r UNASSIGNED
nagioslogserver     0 p STARTED         61  117.2kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
nagioslogserver     0 r UNASSIGNED
logstash-2015.08.12 2 p STARTED    1093921  704.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.12 2 r UNASSIGNED
logstash-2015.08.12 0 p STARTED    1093246  702.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.12 0 r UNASSIGNED
logstash-2015.08.12 3 p STARTED    1093632  704.3mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.12 3 r UNASSIGNED
logstash-2015.08.12 1 p STARTED    1092759  703.3mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.12 1 r UNASSIGNED
logstash-2015.08.12 4 p STARTED    1092760  703.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.12 4 r UNASSIGNED
logstash-2015.08.11 2 p STARTED    1125068  727.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.11 2 r UNASSIGNED
logstash-2015.08.11 0 p STARTED    1125634  730.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.11 0 r UNASSIGNED
logstash-2015.08.11 3 p STARTED    1125880  730.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.11 3 r UNASSIGNED
logstash-2015.08.11 1 p STARTED    1125285    731mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.11 1 r UNASSIGNED
logstash-2015.08.11 4 p STARTED    1126006    732mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.11 4 r UNASSIGNED
logstash-2015.08.31 4 p STARTED    1393556  905.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.31 4 r UNASSIGNED
logstash-2015.08.31 0 p STARTED    1393654  903.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.31 0 r UNASSIGNED
logstash-2015.08.31 3 p STARTED    1394072  906.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.31 3 r UNASSIGNED
logstash-2015.08.31 1 p STARTED    1393846  907.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.31 1 r UNASSIGNED
logstash-2015.08.31 2 p STARTED    1394551  906.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.31 2 r UNASSIGNED
logstash-2015.08.14 4 p STARTED    1054802  672.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.14 4 r UNASSIGNED
logstash-2015.08.14 0 p STARTED    1054968  673.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.14 0 r UNASSIGNED
logstash-2015.08.14 3 p STARTED    1054418  671.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.14 3 r UNASSIGNED
logstash-2015.08.14 1 p STARTED    1055415  672.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.14 1 r UNASSIGNED
logstash-2015.08.14 2 p STARTED    1054694  673.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.14 2 r UNASSIGNED
logstash-2015.08.13 2 p STARTED    1254145  860.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.13 2 r UNASSIGNED
logstash-2015.08.13 0 p STARTED    1255424  861.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.13 0 r UNASSIGNED
logstash-2015.08.13 3 p STARTED    1253972  860.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.13 3 r UNASSIGNED
logstash-2015.08.13 1 p STARTED    1255103  860.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.13 1 r UNASSIGNED
logstash-2015.08.13 4 p STARTED    1254016  858.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.13 4 r UNASSIGNED
logstash-2015.08.16 2 p STARTED     736031  442.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.16 2 r UNASSIGNED
logstash-2015.08.16 0 p STARTED     735923  441.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.16 0 r UNASSIGNED
logstash-2015.08.16 3 p STARTED     736114  442.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.16 3 r UNASSIGNED
logstash-2015.08.16 1 p STARTED     735020  441.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.16 1 r UNASSIGNED
logstash-2015.08.16 4 p STARTED     736192  442.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.16 4 r UNASSIGNED
logstash-2015.08.15 4 p STARTED     728285  431.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.15 4 r UNASSIGNED
logstash-2015.08.15 0 p STARTED     728675  431.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.15 0 r UNASSIGNED
logstash-2015.08.15 3 p STARTED     729281  432.3mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.15 3 r UNASSIGNED
logstash-2015.08.15 1 p STARTED     728080    431mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.15 1 r UNASSIGNED
logstash-2015.08.15 2 p STARTED     728251  432.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.15 2 r UNASSIGNED
logstash-2015.08.18 2 p STARTED    1236181  799.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.18 2 r UNASSIGNED
logstash-2015.08.18 0 p STARTED    1237371  801.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.18 0 r UNASSIGNED
logstash-2015.08.18 3 p STARTED    1235577  800.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.18 3 r UNASSIGNED
logstash-2015.08.18 1 p STARTED    1235654  800.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.18 1 r UNASSIGNED
logstash-2015.08.18 4 p STARTED    1236821  798.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.18 4 r UNASSIGNED
logstash-2015.08.17 4 p STARTED    1348180  916.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.17 4 r UNASSIGNED
logstash-2015.08.17 0 p STARTED    1348457  917.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.17 0 r UNASSIGNED
logstash-2015.08.17 3 p STARTED    1349188  917.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.17 3 r UNASSIGNED
logstash-2015.08.17 1 p STARTED    1348276  917.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.17 1 r UNASSIGNED
logstash-2015.08.17 2 p STARTED    1348695  917.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.17 2 r UNASSIGNED
logstash-2015.08.19 2 p STARTED    1158190  764.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.19 2 r UNASSIGNED
logstash-2015.08.19 0 p STARTED    1157885  764.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.19 0 r UNASSIGNED
logstash-2015.08.19 3 p STARTED    1157891  763.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.19 3 r UNASSIGNED
logstash-2015.08.19 1 p STARTED    1158194  764.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.19 1 r UNASSIGNED
logstash-2015.08.19 4 p STARTED    1157691  764.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.19 4 r UNASSIGNED
logstash-2015.08.30 2 p STARTED     655381  388.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.30 2 r UNASSIGNED
logstash-2015.08.30 0 p STARTED     654883  388.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.30 0 r UNASSIGNED
logstash-2015.08.30 3 p STARTED     654654  389.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.30 3 r UNASSIGNED
logstash-2015.08.30 1 p STARTED     654303  388.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.30 1 r UNASSIGNED
logstash-2015.08.30 4 p STARTED     654540  388.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.30 4 r UNASSIGNED
nagioslogserver_log 4 p STARTED     216427   21.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
nagioslogserver_log 4 r UNASSIGNED
nagioslogserver_log 0 p STARTED     215918   21.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
nagioslogserver_log 0 r UNASSIGNED
nagioslogserver_log 3 p STARTED     216240   21.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
nagioslogserver_log 3 r UNASSIGNED
nagioslogserver_log 1 p STARTED     216289   21.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
nagioslogserver_log 1 r UNASSIGNED
nagioslogserver_log 2 p STARTED     215913   21.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
nagioslogserver_log 2 r UNASSIGNED
logstash-2015.12.11 4 p STARTED         88   85.7kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.12.11 4 r UNASSIGNED
logstash-2015.12.11 0 p STARTED         90   88.8kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.12.11 0 r UNASSIGNED
logstash-2015.12.11 3 p STARTED         72   82.2kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.12.11 3 r UNASSIGNED
logstash-2015.12.11 1 p STARTED         57   77.3kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.12.11 1 r UNASSIGNED
logstash-2015.12.11 2 p STARTED         73   83.3kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.12.11 2 r UNASSIGNED
logstash-2015.08.23 4 p STARTED     682160    429mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.23 4 r UNASSIGNED
logstash-2015.08.23 0 p STARTED     681525  428.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.23 0 r UNASSIGNED
logstash-2015.08.23 3 p STARTED     682061  429.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.23 3 r UNASSIGNED
logstash-2015.08.23 1 p STARTED     682568  429.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.23 1 r UNASSIGNED
logstash-2015.08.23 2 p STARTED     681816  428.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.23 2 r UNASSIGNED
logstash-2015.08.22 2 p STARTED     697583  423.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.22 2 r UNASSIGNED
logstash-2015.08.22 0 p STARTED     696115  423.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.22 0 r UNASSIGNED
logstash-2015.08.22 3 p STARTED     696715  423.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.22 3 r UNASSIGNED
logstash-2015.08.22 1 p STARTED     697006  425.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.22 1 r UNASSIGNED
logstash-2015.08.22 4 p STARTED     697135  423.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.22 4 r UNASSIGNED
logstash-2015.08.21 2 p STARTED    1053984  688.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.21 2 r UNASSIGNED
logstash-2015.08.21 0 p STARTED    1054500  690.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.21 0 r UNASSIGNED
logstash-2015.08.21 3 p STARTED    1054130    689mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.21 3 r UNASSIGNED
logstash-2015.08.21 1 p STARTED    1053987  689.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.21 1 r UNASSIGNED
logstash-2015.08.21 4 p STARTED    1053906  687.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.21 4 r UNASSIGNED
logstash-2015.08.20 2 p STARTED    1122628  734.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.20 2 r UNASSIGNED
logstash-2015.08.20 0 p STARTED    1123609  736.3mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.20 0 r UNASSIGNED
logstash-2015.08.20 3 p STARTED    1123398  737.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.20 3 r UNASSIGNED
logstash-2015.08.20 1 p STARTED    1122970  737.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.20 1 r UNASSIGNED
logstash-2015.08.20 4 p STARTED    1123190  733.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.20 4 r UNASSIGNED
logstash-2015.08.09 4 p STARTED     640673    390mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.09 4 r UNASSIGNED
logstash-2015.08.09 0 p STARTED     640845  389.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.09 0 r UNASSIGNED
logstash-2015.08.09 3 p STARTED     641222  390.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.09 3 r UNASSIGNED
logstash-2015.08.09 1 p STARTED     640505  390.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.09 1 r UNASSIGNED
logstash-2015.08.09 2 p STARTED     640625  390.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.09 2 r UNASSIGNED
logstash-2015.08.27 2 p STARTED    1379306  961.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.27 2 r UNASSIGNED
logstash-2015.08.27 0 p STARTED    1379088  962.3mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.27 0 r UNASSIGNED
logstash-2015.08.27 3 p STARTED    1378697  962.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.27 3 r UNASSIGNED
logstash-2015.08.27 1 p STARTED    1378830  963.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.27 1 r UNASSIGNED
logstash-2015.08.27 4 p STARTED    1378747  961.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.27 4 r UNASSIGNED
logstash-2015.08.26 4 p STARTED    1207818  797.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.26 4 r UNASSIGNED
logstash-2015.08.26 0 p STARTED    1208575  797.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.26 0 r UNASSIGNED
logstash-2015.08.26 3 p STARTED    1207843  795.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.26 3 r UNASSIGNED
logstash-2015.08.26 1 p STARTED    1207886    797mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.26 1 r UNASSIGNED
logstash-2015.08.26 2 p STARTED    1207976  796.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.26 2 r UNASSIGNED
logstash-2015.08.25 4 p STARTED    1284759  839.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.25 4 r UNASSIGNED
logstash-2015.08.25 0 p STARTED    1284700  838.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.25 0 r UNASSIGNED
logstash-2015.08.25 3 p STARTED    1285274  837.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.25 3 r UNASSIGNED
logstash-2015.08.25 1 p STARTED    1284586  838.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.25 1 r UNASSIGNED
logstash-2015.08.25 2 p STARTED    1285251  840.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.25 2 r UNASSIGNED
logstash-2015.08.24 4 p STARTED    1397295    964mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.24 4 r UNASSIGNED
logstash-2015.08.24 0 p STARTED    1397276  963.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.24 0 r UNASSIGNED
logstash-2015.08.24 3 p STARTED    1397370  965.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.24 3 r UNASSIGNED
logstash-2015.08.24 1 p STARTED    1396666    965mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.24 1 r UNASSIGNED
logstash-2015.08.24 2 p STARTED    1396541  962.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.24 2 r UNASSIGNED
logstash-2015.08.29 2 p STARTED     691189  408.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.29 2 r UNASSIGNED
logstash-2015.08.29 0 p STARTED     691042  408.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.29 0 r UNASSIGNED
logstash-2015.08.29 3 p STARTED     690860    409mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.29 3 r UNASSIGNED
logstash-2015.08.29 1 p STARTED     691503  408.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.29 1 r UNASSIGNED
logstash-2015.08.29 4 p STARTED     691622  408.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.29 4 r UNASSIGNED
logstash-2015.08.28 2 p STARTED    1271951  884.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.28 2 r UNASSIGNED
logstash-2015.08.28 0 p STARTED    1272100  884.3mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.28 0 r UNASSIGNED
logstash-2015.08.28 3 p STARTED    1271167  882.2mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.28 3 r UNASSIGNED
logstash-2015.08.28 1 p STARTED    1271658  885.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.28 1 r UNASSIGNED
logstash-2015.08.28 4 p STARTED    1271940  884.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.08.28 4 r UNASSIGNED
logstash-2015.09.01 2 p STARTED    1784449    1.1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.01 2 r UNASSIGNED
logstash-2015.09.01 0 p STARTED    1784698    1.1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.01 0 r UNASSIGNED
logstash-2015.09.01 3 p STARTED    1784423    1.1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.01 3 r UNASSIGNED
logstash-2015.09.01 1 p STARTED    1784423    1.1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.01 1 r UNASSIGNED
logstash-2015.09.01 4 p STARTED    1785830    1.1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.01 4 r UNASSIGNED
logstash-2015.09.02 2 p STARTED    1685534      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.02 2 r UNASSIGNED
logstash-2015.09.02 0 p STARTED    1685511      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.02 0 r UNASSIGNED
logstash-2015.09.02 3 p STARTED    1685733      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.02 3 r UNASSIGNED
logstash-2015.09.02 1 p STARTED    1685078      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.02 1 r UNASSIGNED
logstash-2015.09.02 4 p STARTED    1684751      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.02 4 r UNASSIGNED
kibana-int          4 p STARTED          5   60.2kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
kibana-int          4 r UNASSIGNED
kibana-int          0 p STARTED         10   38.4kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
kibana-int          0 r UNASSIGNED
kibana-int          3 p STARTED         13   72.6kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
kibana-int          3 r UNASSIGNED
kibana-int          1 p STARTED         14  104.9kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
kibana-int          1 r UNASSIGNED
kibana-int          2 p STARTED         14   84.4kb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
kibana-int          2 r UNASSIGNED
logstash-2015.09.07 4 p STARTED    1692408      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.07 4 r UNASSIGNED
logstash-2015.09.07 0 p STARTED    1692694      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.07 0 r UNASSIGNED
logstash-2015.09.07 3 p STARTED    1692082      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.07 3 r UNASSIGNED
logstash-2015.09.07 1 p STARTED    1692722      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.07 1 r UNASSIGNED
logstash-2015.09.07 2 p STARTED    1692296      1gb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.07 2 r UNASSIGNED
logstash-2015.09.08 2 p STARTED    1282176  854.5mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.08 2 r UNASSIGNED
logstash-2015.09.08 0 p STARTED    1281991  850.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.08 0 r UNASSIGNED
logstash-2015.09.08 3 p STARTED    1282427  854.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.08 3 r UNASSIGNED
logstash-2015.09.08 1 p STARTED    1281824  858.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.08 1 r UNASSIGNED
logstash-2015.09.08 4 p STARTED    1281008  858.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.08 4 r UNASSIGNED
logstash-2015.09.05 4 p STARTED     936757  569.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.05 4 r UNASSIGNED
logstash-2015.09.05 0 p STARTED     936785  569.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.05 0 r UNASSIGNED
logstash-2015.09.05 3 p STARTED     937257  570.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.05 3 r UNASSIGNED
logstash-2015.09.05 1 p STARTED     937546  568.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.05 1 r UNASSIGNED
logstash-2015.09.05 2 p STARTED     937566  567.7mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.05 2 r UNASSIGNED
logstash-2015.09.06 4 p STARTED     871125  522.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.06 4 r UNASSIGNED
logstash-2015.09.06 0 p STARTED     871201  521.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.06 0 r UNASSIGNED
logstash-2015.09.06 3 p STARTED     871280  520.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.06 3 r UNASSIGNED
logstash-2015.09.06 1 p STARTED     871784  523.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.06 1 r UNASSIGNED
logstash-2015.09.06 2 p STARTED     871232  522.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.06 2 r UNASSIGNED
logstash-2015.09.03 2 p STARTED    1582362 1021.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.03 2 r UNASSIGNED
logstash-2015.09.03 0 p STARTED    1582761 1020.9mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.03 0 r UNASSIGNED
logstash-2015.09.03 3 p STARTED    1582592 1022.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.03 3 r UNASSIGNED
logstash-2015.09.03 1 p STARTED    1583108 1021.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.03 1 r UNASSIGNED
logstash-2015.09.03 4 p STARTED    1582519 1020.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.03 4 r UNASSIGNED
logstash-2015.09.04 4 p STARTED    1547097    995mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.04 4 r UNASSIGNED
logstash-2015.09.04 0 p STARTED    1546846  993.4mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.04 0 r UNASSIGNED
logstash-2015.09.04 3 p STARTED    1546870  994.8mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.04 3 r UNASSIGNED
logstash-2015.09.04 1 p STARTED    1546213  994.6mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.04 1 r UNASSIGNED
logstash-2015.09.04 2 p STARTED    1546630  990.1mb 127.0.0.1 95f9ab14-da22-4144-bb0b-6bbc5662115c
logstash-2015.09.04 2 r UNASSIGNED
I will send you a pm with the elasticsearch logs. I already tried restarting the elasticsearch service, reboot, etc.

Grtz
Nagios XI 5.8.1
https://outsideit.net
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Shards stay unassigned after yum update

Post by jolson »

In your elasticsearch logs:

[2015-09-08 10:09:10,753][INFO ][cluster.routing.allocation.decider] [xxxxxx] low disk watermark [85%] exceeded on [xxxxxx][xxxxxx] free: 22.1gb[14.9%], replicas will not be assigned to this node
cluster.routing.allocation.disk.watermark.low controls the low watermark for disk usage. It defaults to 85%, meaning ES will not allocate new shards to nodes once they have more than 85% disk used. It can also be set to an absolute byte value (like 500mb) to prevent ES from allocating shards if less than the configured amount of space is available.
It looks like your non-master node (c4d16075-9bc2-4095-9f00-e7de7f96930c) has 85% or more of its disk space taken up currently - meaning that elasticsearch refuses to assign your replica nodes over to it. You'll need to either increase the disk space, or remove files that are currently taking up space on that node. My guess is that there is still a bunch of log information hanging out on the node that will be overwritten when it re-joins your cluster.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:

Re: Shards stay unassigned after yum update

Post by WillemDH »

Jesse, well I was in the process of adding a new disk. Check https://support.nagios.com/forum/viewto ... 38&t=34216 I'd rather not extend the existing one or do I not have any other choice?
Nagios XI 5.8.1
https://outsideit.net
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Shards stay unassigned after yum update

Post by jolson »

Well, we have two options:
1. We could attempt removing the current shard information on your remaining instance (since technically no shards are assigned to it). This may make data replication take a long time, since the replica shards will all need to be written back to the secondary instance.

2. Extend the hard disk of your secondary node.

If we want to pursue option 1, I'd like to see what kind of data is still sitting on your secondary instance.

Verify that we're working on the secondary instance:

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_uuid
The output of the above command should be c4d16075-9bc2-4095-9f00-e7de7f96930c.

Check the size of your elasticsearch data store:

Code: Select all

du -h /usr/local/nagioslogserver/elasticsearch/data
You may need to change the above command if you have specified a particular data storage location.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:

Re: Shards stay unassigned after yum update

Post by WillemDH »

Jesse,

TO be honest I was hoping it was possible to just change the threshold of 85% to 95 % or sth similar.. :)
But as you did not suggest that option, I suppose that is not possible.
On both nodes same result:

Code: Select all

cat /usr/local/nagioslogserver/var/cluster_uuid
ee9e60a0-f4cb-41ec-a97f-8f17434b748e
Which is not the result you seemed to be expecting?

I was not able to post the output of the du her, as:
Your message contains 197551 characters. The maximum number of allowed characters is 60000.
So I zipped it and attached it to this thread.

Grtz
You do not have the required permissions to view the files attached to this post.
Nagios XI 5.8.1
https://outsideit.net
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Shards stay unassigned after yum update

Post by jolson »

Which is not the result you seemed to be expecting?
My apologies - the command that I had you run was entered improperly. The following command is the one I meant to post:

Code: Select all

cat /usr/local/nagioslogserver/var/node_uuid
This should match the expected result.

If possible, I highly recommend taking a snapshot of your server before removing all of the data from it. You should be free to remove *all* of the data in your /usr/local/nagioslogserver/elasticsearch/data directory. I tested the procedure in-house of course, and the procedure I used is below.

First, stop elasticsearch on the problem node:

Code: Select all

service elasticsearch stop
Next, ensure that all of your data still exists on the master node:

Code: Select all

curl -s localhost:9200/_cat/shards
Your current cluster health should be yellow:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
If everything looks good, let's go ahead and remove the data from your data directory on the problem node.
Change to the data directory:

Code: Select all

cd /usr/local/nagioslogserver/elasticsearch/data
Be *certain* that you're now in the data directory so that we don't accidentally wipe the system completely. Next, remove everything:

Code: Select all

rm -rf *
At this point we can restart elasticsearch on the problem node:

Code: Select all

service elasticsearch start
You should see data start to populate in the /usr/local/nagioslogserver/elasticsearch/data directory at this point. This will continue until all primary and replica shards have been distributed appropriately.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
User avatar
WillemDH
Posts: 2320
Joined: Wed Mar 20, 2013 5:49 am
Location: Ghent
Contact:

Re: Shards stay unassigned after yum update

Post by WillemDH »

Thank you Jesse,

Node 01:

Code: Select all

cat /usr/local/nagioslogserver/var/node_uuid
c4d16075-9bc2-4095-9f00-e7de7f96930c
Node 02:

Code: Select all

cat /usr/local/nagioslogserver/var/node_uuid
95f9ab14-da22-4144-bb0b-6bbc5662115c
I'm a bit confused about the concept of 'master node'. Is this something that can be changed? Looking at the output of

Code: Select all

curl 'localhost:9200/_cat/nodes?v'
host        ip        heap.percent ram.percent load node.role master name
log02 127.0.0.1           40          65 0.07 d         *      95f9ab14-da22-4144-bb0b-6bbc5662115c
log01 127.0.0.1            5          68 0.00 d         m      c4d16075-9bc2-4095-9f00-e7de7f96930c
shows an 'm' for node role for node 1. Checking https://www.elastic.co/guide/en/elastic ... nodes.html implies 'm' means node 1 is the master node?

But then you are saying:
Verify that we're working on the secondary instance: The output of the above command should be c4d16075-9bc2-4095-9f00-e7de7f96930c.
So do you understand my confusion? c4d16075-9bc2-4095-9f00-e7de7f96930c is the id of the node having problems, but is also the master node. :?:

Another thing I noticed today is that it seems like some things changed during the noght. Apparently NLS was able to assign some unassigned shards, as yesterday I had 171 unassigned shards. Today it seem I have only 8 unassigned shards.

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "ee9e60a0-f4cb-41ec-a97f-8f17434b748e",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 171,
  "active_shards" : 334,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 8,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}
Apart from the above considerations, I was also thinking. As said in previous post I was in the process of adding extra disks to both NLS nodes. I already added a disk of 250 GB to each node and mounted them to /mnt/data
If I would finish this procedure by adding editing the data paths from

Code: Select all

DATA_DIR="$ES_HOME/data"
to

Code: Select all

DATA_DIR="$ES_HOME/data,/mnt/data"
Is there a chance elasticsearch would be so intelligent to start replicating to the second data path?

(Sorry for all the questions)

Grtz

Willem
Nagios XI 5.8.1
https://outsideit.net
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Shards stay unassigned after yum update

Post by jolson »

So do you understand my confusion? c4d16075-9bc2-4095-9f00-e7de7f96930c is the id of the node having problems, but is also the master node. :?:
This is where the confusion lies - the 'm' does not stand for 'master node' - it stands for 'master eligible node' AKA a node that is *capable* of becoming the master node. The asterisk (*) marks the current master node.

You can verify this with the a master curl:

Code: Select all

curl 'localhost:9200/_cat/master?v'
Is there a chance elasticsearch would be so intelligent to start replicating to the second data path?
If you define multiple data dirs, you can think of it like RAID-0 striping. Elasticsearch will begin to balance data between the two data directories.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Locked