Page 1 of 2

Administration Page won't load - 500 Internal Server Error

Posted: Mon Jun 15, 2015 10:51 am
by MikeScott
Greetings!
Had a bit of a power outage a while ago, and ever since, I have been unable to load the Administration page. Clicking on it either gets "page can not be displayed, a blank page, or a 500 Internal Server Error. Everything else seems to work fine.

I'm running the latest version of Nagios Log Server, on a dedicated CentOS machine.

What I've tried:
- Ran a fsck
- resintalled nagios log server

I would be willing to reload the machine, but would like to move my Elasticsearch indexes somewhere first.

Any pointers / help is appreciated.

Thanks!
Mike Scott

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 10:57 am
by tmcdonald
What LS version is this? How much RAM is on the system? What do you have set for ES_HEAP_SIZE under /etc/sysconfig/elasticsearch?

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 11:30 am
by MikeScott
tmcdonald wrote:What LS version is this? How much RAM is on the system? What do you have set for ES_HEAP_SIZE under /etc/sysconfig/elasticsearch?
I'm not sure what you mean by the LS version, let me know how to find out, and I'll do so.

The system has 20G of RAM - currently 4.6G available. ES_HEAP_SIZE is left at the default, "$(free -m|awk '/^Mem:/{print $2}') / 2 )m", so it should be at the 1G max.

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 11:33 am
by tmcdonald
LS version = Logserver version, and you can find it in the bottom left of the web interface.

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 11:36 am
by MikeScott
tmcdonald wrote:LS version = Logserver version, and you can find it in the bottom left of the web interface.
Thanks! It's 2015R1.4

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 11:37 am
by jolson
My guess is that some indices of yours are currently busted. The quickest way to resolve this would likely be to run a restore.

You will need to restore from a backup. Please check the following location for them:

Code: Select all

ls -l /store/backups/nagioslogserver
After you locate an appropriate backup, please download my attached script and put it on your Nagios Log Server box. The attached script is a revised version of our original restore script.

Put the script in /usr/local/nagioslogserver/scripts. You will need to overwrite the old script - please do so.

After you verify that the new script is in place, run it against the backup that you have picked out.

Code: Select all

/usr/local/nagioslogserver/restore_backup.sh /store/backups/nagioslogserver/xxxxx.tar.gz
Once the restore is complete, your administrative panel should be visible again.

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 11:45 am
by MikeScott
jolson wrote:My guess is that some indices of yours are currently busted. The quickest way to resolve this would likely be to run a restore.
Well... funny story there. I don't have any backups just yet. I've only recently set up Nagios Log Server, and haven't had a chance to set up a store and get backups going. Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 11:50 am
by jolson
Can you check that location anyway? Backups stored in /store/backups/nagioslogserver are automatic, and should be generated daily.
Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?
I have a few ideas, but probably setting up a repository. :)

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 11:54 am
by MikeScott
jolson wrote:Can you check that location anyway? Backups stored in /store/backups/nagioslogserver are automatic, and should be generated daily.
Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?
I have a few ideas, but probably setting up a repository. :)
/store/backups/nagioslogserver is empty. :-(

Re: Administration Page won't load - 500 Internal Server Err

Posted: Mon Jun 15, 2015 12:22 pm
by jolson
Alright, let's run a couple of commands and see if we can identify what the problem is:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
curl -s localhost:9200/_cat/shards