Nagios Support Forum

Okay, so, for fun, what happens if you reboot the box and try to log in?

For Eloyd

[root@IWKSEUWPSSNLS01 ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 619273992 428618868 159205396 73% /
udev 15390940 112 15390828 1% /dev
tmpfs 15411524 0 15411524 0% /dev/shm
/dev/xvde1 619273992 428618868 159205396 73% /
none 15411524 0 15411524 0% /dev/shm
[root@IWKSEUWPSSNLS01 ~]# free -k
total used free shared buffers cached
Mem: 30823048 30674140 148908 204 129196 12346204
-/+ buffers/cache: 18198740 12624308
Swap: 262136 1780 260356

Reboot did not help, still same issue. Unable to login

I'd still be interested to see the full contents of this file: Something definitely busted the nagioslogserver index, which is where user information (among other things) is stored. It can be rebuilt from scratch with a new user; It's just a huge pain in the butt.

You might try to restore from a previous backup using this script: The backups are typically kept somewhere in the /store path. Basically, you provide a backup to that script (the full path to the backup) as an argument and execute it.

Assuming you want to go the restore route...

Basically, it should look something like the following: Change the file name and file path accordingly.

Before we restore, I would like to tell you that the disk space was increased on this server by remounting the root volume from 400Gb to 600Gb and then resize2fs /dev/xvde1

[root@IWKSEUWPSSNLS01 ec2-user]# resize2fs /dev/xvde1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/xvde1 is mounted on /; on-line resizing required
old desc_blocks = 25, new_desc_blocks = 38
Performing an on-line resize of /dev/xvde1 to 157286400 (4k) blocks.

If you wanted to wait to do the restore you could attach /var/log/elasticsearch/0418383b-07ee-4161-ad5b-e7d7e3b1e9d8.log.

Otherwise, let us know if the restore resolves your issue.

Log attached for review

Attached here