Page 1 of 2
Administration Page won't load - 500 Internal Server Error
Posted: Mon Jun 15, 2015 10:51 am
by MikeScott
Greetings!
Had a bit of a power outage a while ago, and ever since, I have been unable to load the Administration page. Clicking on it either gets "page can not be displayed, a blank page, or a 500 Internal Server Error. Everything else seems to work fine.
I'm running the latest version of Nagios Log Server, on a dedicated CentOS machine.
What I've tried:
- Ran a fsck
- resintalled nagios log server
I would be willing to reload the machine, but would like to move my Elasticsearch indexes somewhere first.
Any pointers / help is appreciated.
Thanks!
Mike Scott
Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 10:57 am
by tmcdonald
What LS version is this? How much RAM is on the system? What do you have set for ES_HEAP_SIZE under /etc/sysconfig/elasticsearch?
Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 11:30 am
by MikeScott
tmcdonald wrote:What LS version is this? How much RAM is on the system? What do you have set for ES_HEAP_SIZE under /etc/sysconfig/elasticsearch?
I'm not sure what you mean by the LS version, let me know how to find out, and I'll do so.
The system has 20G of RAM - currently 4.6G available. ES_HEAP_SIZE is left at the default, "$(free -m|awk '/^Mem:/{print $2}') / 2 )m", so it should be at the 1G max.
Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 11:33 am
by tmcdonald
LS version = Logserver version, and you can find it in the bottom left of the web interface.
Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 11:36 am
by MikeScott
tmcdonald wrote:LS version = Logserver version, and you can find it in the bottom left of the web interface.
Thanks! It's 2015R1.4
Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 11:37 am
by jolson
My guess is that some indices of yours are currently busted. The quickest way to resolve this would likely be to run a restore.
You will need to restore from a backup. Please check the following location for them:
Code: Select all
ls -l /store/backups/nagioslogserver
After you locate an appropriate backup, please download my attached script and put it on your Nagios Log Server box. The attached script is a revised version of our original restore script.
Put the script in /usr/local/nagioslogserver/scripts. You will need to overwrite the old script - please do so.
After you verify that the new script is in place, run it against the backup that you have picked out.
Code: Select all
/usr/local/nagioslogserver/restore_backup.sh /store/backups/nagioslogserver/xxxxx.tar.gz
Once the restore is complete, your administrative panel should be visible again.
Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 11:45 am
by MikeScott
jolson wrote:My guess is that some indices of yours are currently busted. The quickest way to resolve this would likely be to run a restore.
Well... funny story there. I don't have any backups just yet. I've only recently set up Nagios Log Server, and haven't had a chance to set up a store and get backups going. Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?
Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 11:50 am
by jolson
Can you check that location anyway? Backups stored in /store/backups/nagioslogserver are automatic, and should be generated daily.
Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?
I have a few ideas, but probably setting up a repository.

Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 11:54 am
by MikeScott
jolson wrote:Can you check that location anyway? Backups stored in /store/backups/nagioslogserver are automatic, and should be generated daily.
Can you guess what the second thing I'll be doing if I have to wipe it out and start over is?
I have a few ideas, but probably setting up a repository.

/store/backups/nagioslogserver is empty.

Re: Administration Page won't load - 500 Internal Server Err
Posted: Mon Jun 15, 2015 12:22 pm
by jolson
Alright, let's run a couple of commands and see if we can identify what the problem is:
Code: Select all
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
curl -s localhost:9200/_cat/shards