Page 3 of 4
Re: adding extra NLS datapath
Posted: Fri Nov 15, 2019 3:47 pm
by rocheryderm
I appreciate your speedy replies, really.
I think we're getting close to the end here....
I can't logon to the GUI. At first I thought the timeouts were because the cluster was busy assigning shards.
However, now it's quite idle, and still refuses logon to the gui with "The username specified does not exist."
I've tried using curl to reset the password of "nagiosadmin" with "/usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh" and also as specified here:
https://support.nagios.com/forum/viewto ... 37&t=36158
but timeout expires and it fails.
I've also tried using curl to create a new admin account using curl with these instructions:
https://support.nagios.com/forum/viewto ... 06#p203705
But that fails, too, with this:
Code: Select all
{"error":"UnavailableShardsException[[nagioslogserver][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: index {[nagioslogserver][user][AW5wzJa92rRukiroeT00], source[{\"username\":\"someuser\",\"password\":\"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19\",\"auth_type\":\"admin\",\"email\":\"[email protected]\",\"language\":\"default\",\"apiaccess\":\"1\",\"apikey\":\"1396e08757545557073844695e5b64caa0bd3ad3\",\"created\":\"2015-01-23 10:00:00\",\"created_by\":0,\"default_dashboard\":\"/dashboard/elasticsearch/default\"}]}]","status":503}
What are my options?
I'm not feeling good about this. Can you believe this all started just because I added another datapath?
Re: adding extra NLS datapath
Posted: Fri Nov 15, 2019 3:55 pm
by scottwilkerson
This looks like the
nagioslogserver index isn't active which is generally pretty bad, however, there may be a saving grace, there are system backups that are automatically taken that are located in
/store/backups/nagioslogserver/
See the "Restore the System Backup" section on the bottom of page 7 - top of page 8 of the following document on the procedure the repair this
https://assets.nagios.com/downloads/nag ... Server.pdf
It basically will looks something like this (from one of the nodes)
Code: Select all
cd /usr/local/nagioslogserver/scripts/
./restore_backup.sh /store/backups/nagioslogserver/nagioslogserver.2019-05-10.1494373596.tar.gz
Re: adding extra NLS datapath
Posted: Fri Nov 15, 2019 4:35 pm
by rocheryderm
Unfortunately cd /usr/local/nagioslogserver/scripts/ was empty.
I had to bite the bullet. Indices with unassignable shards deleted. Was able to create a new admin user, logon. Cluster is up, is most definitely collecting data from logstash but...
GUI shows this... (attached)
hovering over either red excalamation notes that either "Elasticsearch" or "Logstash" is not running.
This is fatiguing.
Re: adding extra NLS datapath
Posted: Fri Nov 15, 2019 4:41 pm
by scottwilkerson
/usr/local/nagioslogserver/scripts/ was empty?
2 posts ago you said you did this
rocheryderm wrote:I've tried using curl to reset the password of "nagiosadmin" with "/usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh" and also as specified here: viewtopic.php?f=37&t=36158
That is in the directory you say was empty
Re: adding extra NLS datapath
Posted: Fri Nov 15, 2019 5:02 pm
by rocheryderm
<sigh>
I'm sorry.
"/store/backups/nagioslogserver/" was empty.
Re: adding extra NLS datapath
Posted: Mon Nov 18, 2019 9:49 am
by rocheryderm
So... I'm still having this problem...
https://support.nagios.com/forum/viewto ... 20#p297815
And also can't see any data when in the dashboard. I know Elasticsearch is ingesting data because I have Nagios queries against ES that continue to work.
When I attempt to choose a specific query, I get this message in the dashboard
"No time filter Timestamped indices are configured without a failover. Waiting for time filter."
This goes away after a few seconds, and I'm left with
"No filters available"
An ideas?
Re: adding extra NLS datapath
Posted: Mon Nov 18, 2019 10:30 am
by scottwilkerson
Can you go to Admin -> System Status
Click Download System profile and PM it to me?
rocheryderm wrote:And also can't see any data when in the dashboard. I know Elasticsearch is ingesting data because I have Nagios queries against ES that continue to work.
When I attempt to choose a specific query, I get this message in the dashboard
"No time filter Timestamped indices are configured without a failover. Waiting for time filter."
This goes away after a few seconds, and I'm left with
"No filters available"
I suspect this is because you no longer have the default dashboards if kibana-int index was removed , you should be able to select a time period from the drop-down in the upper right but this is going to be a pain.
rocheryderm wrote:<sigh>
I'm sorry.
"/store/backups/nagioslogserver/" was empty.
Was this really empty on all servers?
Re: adding extra NLS datapath
Posted: Mon Nov 18, 2019 11:22 am
by rocheryderm
Is there anything you can do to help me restore the default dashboards?
And yes, "/store/backups/nagioslogserver/" is empty on 4/4 instances.
Re: adding extra NLS datapath
Posted: Mon Nov 18, 2019 3:53 pm
by scottwilkerson
This should restore the global dashboards
Code: Select all
/usr/bin/php /var/www/html/nagioslogserver/www/index.php install/upgrade
Re: adding extra NLS datapath
Posted: Mon Nov 18, 2019 4:31 pm
by rocheryderm
Excellent, thank you, now I can do ad-hoc queries!
Do you think I should try to do a shutdown of the cluster and bring it back up? Would still love to make those other erroneous states of logstash and elasticsearch status go away.