Update from 2.1.4 to 2.1.6 stuck after Kibana upgrade

[rferebee]

Hello,

I am upgrading my Log Server environment to version 2.1.6 and I'm running into an issue I've never encountered before.

Two of my three nodes are stuck at a point after upgrading Kibana. They're just sitting there on the update script not progressing. It's been roughly 30 minutes since I started the process which typically takes me 15 minutes total.

I cannot access the GUI of my third node even though the upgrade seemed to go fine on that one.

I'm not sure how to proceed. Any guidance is appreciated!

[cdienger]

If it is still stuck on this Kibana upgraded message, go ahead and terminate it with "CTRL+C" and then run the upgrade again:

Code: Select all

bash -x ./upgrade

[rferebee]

I'm going to have to give you folks a call in the morning. It did finally finish, but my environment is hosed ATM.

I can't login and the CPU activity spikes up to normal levels after a reboot, but then immediately falls down to almost no activity. Logstash and elasticsearch appear to be running on all three nodes. Apache is running.

I have no idea what's going on, but it's not happy.

[rferebee]

I left a voicemail with Nagios tech support about 30 minutes ago. I will need someone to give me a call back ASAP. Thank you!

[cdienger]

Just updating the thread to clean up our dashboards

[rferebee]

So, all the shards/indices were about to be done... I was literally on 1 shard left and the entire system took a nose dive again.

I don't know what's going on, but I need more help again tomorrow please.

[cdienger]

I'll reach out in a moment.

[rferebee]

As far as I can tell, one of the other nodes decided it wanted to be master all of a sudden which caused a bunch of the shards to switch back to initializing. I rebooted one of the nodes that thought it was master and now I'm slowing bringing the shards back online.

I just hope what happened last night doesn't happen again. Is there anyway to assign a master?

[cdienger]

Spoke to @rferebee. Keep us posted

[rferebee]

The same exact thing happened again. I need assistance ASAP please!