Upgrade and status staying yellow

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Upgrade and status staying yellow

Post by BanditBBS »

I just upgraded from 1.4.1 to 1.4.2 and my cluster status won't leave yellow! Looks the same on both instances:
Capture.PNG
You do not have the required permissions to view the files attached to this post.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Upgrade and status staying yellow

Post by BanditBBS »

It may have been yellow already.....followed link on https://support.nagios.com/kb/article.php?id=469 to increase low disk watermark and things are changing....

Having an issue with the clustered URL now though, getting "Could not authenticate, invalid token given" when hovering the red icons on top. Or, it looks like it is working then goes back to login page.....waiting for shards to finish allocating....will report back in a few.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Upgrade and status staying yellow

Post by mcapra »

Let us know when there's an update! I'd be interested to see if the Javascript console contains anything meaningful for the "Could not authenticate" issue.

The elasticsearch logs might provide some insights on the previous issue (/var/log/elasticsearch/*.log)
Former Nagios employee
https://www.mcapra.com/
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Upgrade and status staying yellow

Post by BanditBBS »

Ok, cluster status is back to green! Still having issue with the cluster URL though :(

Now when I try and login it just bounces right back to the login screen.

I checked http error and access logs, no errors in either.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Upgrade and status staying yellow

Post by mcapra »

Can you share a screenshot of what you're looking at? The previously mentioned error of Could not authenticate. Invalid token given. pretty exclusively pertains to incorrect user permissions or an invalid API key. If the login process is now just booting you back to the main login screen without any sort of error message, I would like to see the content of the elasticsearch logs at /var/log/elasticsearch/*.log
Former Nagios employee
https://www.mcapra.com/
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Upgrade and status staying yellow

Post by BanditBBS »

I can login to either node directly and it works great. It is only the cluster(F5 Load balancer) URL that I keep getting looped back to the login screen. Every so often it does pretend to login but then gives the "Could not authenticate. Invalid token given." error and in the iframe of the number of events the login screen appears and if I click anything it reloads the login page. This happens maybe 1 out of 100 tries to login. The elasticsearch log is basically empty except for the issue when I was crossing the watermark levels. Nothing else written since the cluster has been turned green.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Upgrade and status staying yellow

Post by mcapra »

We have seen some intermittent Same-origin policy violations when accessing a cluster behind a load balancer. I would check the Javascript console from your preferred browser when the problem happens to see if any useful debug information is in there.
Former Nagios employee
https://www.mcapra.com/
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Upgrade and status staying yellow

Post by BanditBBS »

I messages my networking team to assist here, in case any changes made recently. That being said, only change on my part was upgrading from 1.4.1 to 1.4.2. I'm very certain that the cluster domain name was working for browsing just last week. Also, no ports are changed so it *shouldn't* be a same origin policy issue, F5 and nodes' ports for browsing are the same.
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Upgrade and status staying yellow

Post by mcapra »

It could be other Javascript related issues. If the apache/elasticsearch logs aren't shedding any light, the Javascript console would be the next place I would check.
Former Nagios employee
https://www.mcapra.com/
User avatar
BanditBBS
Posts: 2474
Joined: Tue May 31, 2011 12:57 pm
Location: Scio, OH
Contact:

Re: Upgrade and status staying yellow

Post by BanditBBS »

Error in java console:

Code: Select all

Navigated to http://naglog.itciss.com/nagioslogserver/login?redirect=
Navigated to http://naglog.itciss.com/nagioslogserver/
jquery-1.10.1.min.js:5 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.
/nagioslogserver/ZeroClipboard.swf?noCache=1473708621106:1 GET http://naglog.itciss.com/nagioslogserver/ZeroClipboard.swf?noCache=1473708621106 
(program):73 ZeroClipboard error of type "flash-deactivated": Flash is too outdated for your browser and/or is configured as click-to-activate
Navigated to http://naglog.itciss.com/nagioslogserver/login?redirect=dashboard/fetch_rss/marketing
That happened when I disabled catch in the java console. I also saw another issue related to the fetch-rss for marketing stuff. Could that be the cause(and possible same-origin policy issue?).
2 of XI5.6.14 Prod/DR/DEV - Nagios LogServer 2 Nodes
See my projects on the Exchange at BanditBBS - Also check out my Nagios stuff on my personal page at Bandit's Home and at github
Locked