adding extra NLS datapath

[rocheryderm]

I appreciate your speedy replies, really.

I think we're getting close to the end here....

I can't logon to the GUI. At first I thought the timeouts were because the cluster was busy assigning shards.

However, now it's quite idle, and still refuses logon to the gui with "The username specified does not exist."

I've tried using curl to reset the password of "nagiosadmin" with "/usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh" and also as specified here: https://support.nagios.com/forum/viewto ... 37&t=36158

but timeout expires and it fails.

I've also tried using curl to create a new admin account using curl with these instructions: https://support.nagios.com/forum/viewto ... 06#p203705

But that fails, too, with this:

Code: Select all

{"error":"UnavailableShardsException[[nagioslogserver][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: index {[nagioslogserver][user][AW5wzJa92rRukiroeT00], source[{\"username\":\"someuser\",\"password\":\"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19\",\"auth_type\":\"admin\",\"email\":\"[email protected]\",\"language\":\"default\",\"apiaccess\":\"1\",\"apikey\":\"1396e08757545557073844695e5b64caa0bd3ad3\",\"created\":\"2015-01-23 10:00:00\",\"created_by\":0,\"default_dashboard\":\"/dashboard/elasticsearch/default\"}]}]","status":503}

What are my options?

I'm not feeling good about this. Can you believe this all started just because I added another datapath?

[scottwilkerson]

This looks like the nagioslogserver index isn't active which is generally pretty bad, however, there may be a saving grace, there are system backups that are automatically taken that are located in /store/backups/nagioslogserver/

See the "Restore the System Backup" section on the bottom of page 7 - top of page 8 of the following document on the procedure the repair this
https://assets.nagios.com/downloads/nag ... Server.pdf

It basically will looks something like this (from one of the nodes)

Code: Select all

cd /usr/local/nagioslogserver/scripts/
./restore_backup.sh /store/backups/nagioslogserver/nagioslogserver.2019-05-10.1494373596.tar.gz

[rocheryderm]

Unfortunately cd /usr/local/nagioslogserver/scripts/ was empty.



I had to bite the bullet. Indices with unassignable shards deleted. Was able to create a new admin user, logon. Cluster is up, is most definitely collecting data from logstash but...

GUI shows this... (attached)

hovering over either red excalamation notes that either "Elasticsearch" or "Logstash" is not running.

This is fatiguing.

[scottwilkerson]

/usr/local/nagioslogserver/scripts/ was empty?

2 posts ago you said you did this

I've tried using curl to reset the password of "nagiosadmin" with "/usr/local/nagioslogserver/scripts/reset_nagiosadmin_password.sh" and also as specified here: viewtopic.php?f=37&t=36158

That is in the directory you say was empty

[rocheryderm]

<sigh>

I'm sorry.

"/store/backups/nagioslogserver/" was empty.

[rocheryderm]

So... I'm still having this problem...

https://support.nagios.com/forum/viewto ... 20#p297815

And also can't see any data when in the dashboard. I know Elasticsearch is ingesting data because I have Nagios queries against ES that continue to work.

When I attempt to choose a specific query, I get this message in the dashboard

"No time filter Timestamped indices are configured without a failover. Waiting for time filter."
This goes away after a few seconds, and I'm left with

"No filters available"

An ideas?

[scottwilkerson]

Can you go to Admin -> System Status
Click Download System profile and PM it to me?

And also can't see any data when in the dashboard. I know Elasticsearch is ingesting data because I have Nagios queries against ES that continue to work.

When I attempt to choose a specific query, I get this message in the dashboard

"No time filter Timestamped indices are configured without a failover. Waiting for time filter."
This goes away after a few seconds, and I'm left with

"No filters available"

I suspect this is because you no longer have the default dashboards if kibana-int index was removed , you should be able to select a time period from the drop-down in the upper right but this is going to be a pain.

<sigh>

I'm sorry.

"/store/backups/nagioslogserver/" was empty.

Was this really empty on all servers?

[rocheryderm]

Is there anything you can do to help me restore the default dashboards?

And yes, "/store/backups/nagioslogserver/" is empty on 4/4 instances.

[scottwilkerson]

This should restore the global dashboards

Code: Select all

/usr/bin/php /var/www/html/nagioslogserver/www/index.php install/upgrade

[rocheryderm]

Excellent, thank you, now I can do ad-hoc queries!

Do you think I should try to do a shutdown of the cluster and bring it back up? Would still love to make those other erroneous states of logstash and elasticsearch status go away.