All my data is gone, I cannot see anything

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
bkeating178
Posts: 8
Joined: Thu May 09, 2019 11:40 am

All my data is gone, I cannot see anything

Post by bkeating178 »

All,

I had an issue where my log analyzer server would just say “Waiting for Database to start”, so I rebooted and it came back. Now when I login to the portal, I see absolutely nothing, regardless of what I search for or what filters are applied or not applied.

I can see from the “admin” tab that my documents are still there and being logged (according the stats of that tab) but nothing is being displayed and I cannot search for anything, thus, my data may not actually be there anymore and I do not know why.

I attempted to restart the “elasticsearch” service from the System Status page but I get an error message there as well (Picture below).
When I go to the forum for the answer to that error I get the message “You are not authorized to view this topic” (Picture Below), I assume this is because we are still in the trial phase and that is a paid customer forum.

I need some help here on what might have happened and where to turn next.
You do not have the required permissions to view the files attached to this post.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: All my data is gone, I cannot see anything

Post by scottwilkerson »

How much disk space is availaboe on the server?

Code: Select all

df -h
Can you show a screenshot fo
Admin -> Index Status

and the Cluster Health at
Admin -> Cluster Status
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
bkeating178
Posts: 8
Joined: Thu May 09, 2019 11:40 am

Re: All my data is gone, I cannot see anything

Post by bkeating178 »

Plenty of disk space, although I see the status as "RED". Not sure why or what that means.
Any thoughts?
You do not have the required permissions to view the files attached to this post.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: All my data is gone, I cannot see anything

Post by scottwilkerson »

It looks like you have a corrupt index for today, on the
Admin -> Index Status

Click Delete on the line next to logstash-2019.06.13
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
bkeating178
Posts: 8
Joined: Thu May 09, 2019 11:40 am

Re: All my data is gone, I cannot see anything

Post by bkeating178 »

Looks like my information is slowly coming back, could this be due to the fact something was up with my database from earlier today?

How does the database get corrupted like that after being up for so long?
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: All my data is gone, I cannot see anything

Post by scottwilkerson »

bkeating178 wrote:Looks like my information is slowly coming back, could this be due to the fact something was up with my database from earlier today?

How does the database get corrupted like that after being up for so long?
Excellent.

Yes is for any reason something goes wrong at the exact time an index is first trying to be created it can cause this.

It is rare, but I have seen it on a few occasions, usually related to the server losing write capability on the drive.
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
bkeating178
Posts: 8
Joined: Thu May 09, 2019 11:40 am

Re: All my data is gone, I cannot see anything

Post by bkeating178 »

It is definitely odd, but it is working now.

Thank you for your help, I will note what we did so if it comes up again, we know what to do and where to look.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: All my data is gone, I cannot see anything

Post by scottwilkerson »

bkeating178 wrote:It is definitely odd, but it is working now.

Thank you for your help, I will note what we did so if it comes up again, we know what to do and where to look.
Glad to hear it is working again!

Locking thread
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
Locked