Administration Page won't load - 500 Internal Server Error

[MikeScott]

Greetings!
Had a bit of a power outage a while ago, and ever since, I have been unable to load the Administration page. Clicking on it either gets "page can not be displayed, a blank page, or a 500 Internal Server Error. Everything else seems to work fine.

I'm running the latest version of Nagios Log Server, on a dedicated CentOS machine.

What I've tried:
- Ran a fsck
- resintalled nagios log server

I would be willing to reload the machine, but would like to move my Elasticsearch indexes somewhere first.

Any pointers / help is appreciated.

Thanks!
Mike Scott

[tmcdonald]

What LS version is this? How much RAM is on the system? What do you have set for ES_HEAP_SIZE under /etc/sysconfig/elasticsearch?

[MikeScott]

What LS version is this? How much RAM is on the system? What do you have set for ES_HEAP_SIZE under /etc/sysconfig/elasticsearch?

I'm not sure what you mean by the LS version, let me know how to find out, and I'll do so.

The system has 20G of RAM - currently 4.6G available. ES_HEAP_SIZE is left at the default, "$(free -m|awk '/^Mem:/{print $2}') / 2 )m", so it should be at the 1G max.

[tmcdonald]

LS version = Logserver version, and you can find it in the bottom left of the web interface.

[MikeScott]

LS version = Logserver version, and you can find it in the bottom left of the web interface.

Thanks! It's 2015R1.4

[jolson]

My guess is that some indices of yours are currently busted. The quickest way to resolve this would likely be to run a restore.

You will need to restore from a backup. Please check the following location for them:

Code: Select all

ls -l /store/backups/nagioslogserver

After you locate an appropriate backup, please download my attached script and put it on your Nagios Log Server box. The attached script is a revised version of our original restore script.

Put the script in /usr/local/nagioslogserver/scripts. You will need to overwrite the old script - please do so.

After you verify that the new script is in place, run it against the backup that you have picked out.

Code: Select all

/usr/local/nagioslogserver/restore_backup.sh /store/backups/nagioslogserver/xxxxx.tar.gz

Once the restore is complete, your administrative panel should be visible again.

[MikeScott]

My guess is that some indices of yours are currently busted. The quickest way to resolve this would likely be to run a restore.

Well... funny story there. I don't have any backups just yet. I've only recently set up Nagios Log Server, and haven't had a chance to set up a store and get backups going. Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?

[jolson]

Can you check that location anyway? Backups stored in /store/backups/nagioslogserver are automatic, and should be generated daily.

Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?

I have a few ideas, but probably setting up a repository.

[MikeScott]

Can you check that location anyway? Backups stored in /store/backups/nagioslogserver are automatic, and should be generated daily.

Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?

I have a few ideas, but probably setting up a repository.

/store/backups/nagioslogserver is empty.

[jolson]

Alright, let's run a couple of commands and see if we can identify what the problem is:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
curl -s localhost:9200/_cat/shards