Logs not reporting again

[Box293]

1. What do I put for the node?

ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c

2. What about logstash?

They may also need to be done, or they may auto-correct themselves. Fixing the one I highlighted should allow you to log into the interface.

[kwhogster]

Troy

ran this

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ { "allocate" : { "index" : "nagioslogserver", "shard" : 0, "node" : "ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c", "allow_primary" : true } } ] }'

A ton of messages came out

Trying to access the site takes a very long time now.

Ran this
[root@TGCS018 /]# curl -s -XGET http://localhost:9200/_cat/shards?v | egrep 'UNASSIGNED|INITIALIZING' logstash-2016.09.17 4 p UNASSIGNED
logstash-2016.09.17 4 r UNASSIGNED
logstash-2016.09.17 0 r UNASSIGNED
logstash-2016.09.17 3 p UNASSIGNED
logstash-2016.09.17 3 r UNASSIGNED
logstash-2016.09.17 1 r UNASSIGNED
logstash-2016.09.17 2 p UNASSIGNED
logstash-2016.09.17 2 r UNASSIGNED
logstash-2016.09.13 2 r UNASSIGNED
logstash-2016.09.13 0 r UNASSIGNED
logstash-2016.09.13 3 p UNASSIGNED
logstash-2016.09.13 3 r UNASSIGNED
logstash-2016.09.13 1 r UNASSIGNED
logstash-2016.09.13 4 r UNASSIGNED
nagioslogserver 0 r UNASSIGNED
nagioslogserver 0 p UNASSIGNED
logstash-2016.09.15 2 p UNASSIGNED
logstash-2016.09.15 2 r UNASSIGNED
logstash-2016.09.15 0 p UNASSIGNED
logstash-2016.09.15 0 r UNASSIGNED
logstash-2016.09.15 3 r UNASSIGNED
logstash-2016.09.15 1 r UNASSIGNED
logstash-2016.09.15 4 r UNASSIGNED
logstash-2016.09.12 4 r UNASSIGNED
logstash-2016.09.12 0 r UNASSIGNED
logstash-2016.09.12 3 r UNASSIGNED
logstash-2016.09.12 1 p UNASSIGNED
logstash-2016.09.12 1 r UNASSIGNED
logstash-2016.09.12 2 r UNASSIGNED
nagioslogserver_log 4 p UNASSIGNED
nagioslogserver_log 4 r UNASSIGNED
nagioslogserver_log 0 p UNASSIGNED
nagioslogserver_log 0 r UNASSIGNED
nagioslogserver_log 3 r UNASSIGNED
nagioslogserver_log 1 r UNASSIGNED
nagioslogserver_log 2 p UNASSIGNED
nagioslogserver_log 2 r UNASSIGNED
logstash-2016.08.25 4 r UNASSIGNED
logstash-2016.08.25 0 p UNASSIGNED
logstash-2016.08.25 0 r UNASSIGNED
logstash-2016.08.25 3 p UNASSIGNED
logstash-2016.08.25 3 r UNASSIGNED
logstash-2016.08.25 1 r UNASSIGNED
logstash-2016.08.25 2 r UNASSIGNED
logstash-2016.08.26 2 r UNASSIGNED
logstash-2016.08.26 0 r UNASSIGNED
logstash-2016.08.26 3 p UNASSIGNED
logstash-2016.08.26 3 r UNASSIGNED
logstash-2016.08.26 1 r UNASSIGNED
logstash-2016.08.26 4 r UNASSIGNED
logstash-2016.08.23 2 r UNASSIGNED
logstash-2016.08.23 0 r UNASSIGNED
logstash-2016.08.23 3 r UNASSIGNED
logstash-2016.08.23 1 r UNASSIGNED
logstash-2016.08.23 4 r UNASSIGNED
logstash-2016.08.24 2 p UNASSIGNED
logstash-2016.08.24 2 r UNASSIGNED
logstash-2016.08.24 0 r UNASSIGNED
logstash-2016.08.24 3 r UNASSIGNED
logstash-2016.08.24 1 r UNASSIGNED
logstash-2016.08.24 4 r UNASSIGNED
logstash-2016.08.29 2 r UNASSIGNED
logstash-2016.08.29 0 r UNASSIGNED
logstash-2016.08.29 3 r UNASSIGNED
logstash-2016.08.29 1 p UNASSIGNED
logstash-2016.08.29 1 r UNASSIGNED
logstash-2016.08.29 4 r UNASSIGNED
logstash-2016.08.27 4 p UNASSIGNED
logstash-2016.08.27 4 r UNASSIGNED
logstash-2016.08.27 0 p UNASSIGNED
logstash-2016.08.27 0 r UNASSIGNED
logstash-2016.08.27 3 r UNASSIGNED
logstash-2016.08.27 1 p UNASSIGNED
logstash-2016.08.27 1 r UNASSIGNED
logstash-2016.08.27 2 r UNASSIGNED
logstash-2016.08.28 2 p UNASSIGNED
logstash-2016.08.28 2 r UNASSIGNED
logstash-2016.08.28 0 r UNASSIGNED
logstash-2016.08.28 3 r UNASSIGNED
logstash-2016.08.28 1 r UNASSIGNED
logstash-2016.08.28 4 r UNASSIGNED
logstash-2016.08.21 2 p UNASSIGNED
logstash-2016.08.21 2 r UNASSIGNED
logstash-2016.08.21 0 r UNASSIGNED
logstash-2016.08.21 3 r UNASSIGNED
logstash-2016.08.21 1 r UNASSIGNED
logstash-2016.08.21 4 r UNASSIGNED
logstash-2016.08.22 2 p UNASSIGNED
logstash-2016.08.22 2 r UNASSIGNED
logstash-2016.08.22 0 r UNASSIGNED
logstash-2016.08.22 3 p UNASSIGNED
logstash-2016.08.22 3 r UNASSIGNED
logstash-2016.08.22 1 r UNASSIGNED
logstash-2016.08.22 4 r UNASSIGNED
logstash-2016.08.20 4 p UNASSIGNED
logstash-2016.08.20 4 r UNASSIGNED
logstash-2016.08.20 0 r UNASSIGNED
logstash-2016.08.20 3 p UNASSIGNED
logstash-2016.08.20 3 r UNASSIGNED
logstash-2016.08.20 1 r UNASSIGNED
logstash-2016.08.20 2 r UNASSIGNED
logstash-2016.09.06 2 p UNASSIGNED
logstash-2016.09.06 2 r UNASSIGNED
logstash-2016.09.06 0 r UNASSIGNED
logstash-2016.09.06 3 r UNASSIGNED
logstash-2016.09.06 1 r UNASSIGNED
logstash-2016.09.06 4 p UNASSIGNED
logstash-2016.09.06 4 r UNASSIGNED
logstash-2016.09.04 2 r UNASSIGNED
logstash-2016.09.04 0 r UNASSIGNED
logstash-2016.09.04 3 r UNASSIGNED
logstash-2016.09.04 1 r UNASSIGNED
logstash-2016.09.04 4 r UNASSIGNED
logstash-2016.09.05 4 p UNASSIGNED
logstash-2016.09.05 4 r UNASSIGNED
logstash-2016.09.05 0 p UNASSIGNED
logstash-2016.09.05 0 r UNASSIGNED
logstash-2016.09.05 3 r UNASSIGNED
logstash-2016.09.05 1 r UNASSIGNED
logstash-2016.09.05 2 r UNASSIGNED
logstash-2016.09.02 4 r UNASSIGNED
logstash-2016.09.02 0 r UNASSIGNED
logstash-2016.09.02 3 p UNASSIGNED
logstash-2016.09.02 3 r UNASSIGNED
logstash-2016.09.02 1 r UNASSIGNED
logstash-2016.09.02 2 r UNASSIGNED
logstash-2016.09.03 4 r UNASSIGNED
logstash-2016.09.03 0 r UNASSIGNED
logstash-2016.09.03 3 r UNASSIGNED
logstash-2016.09.03 1 r UNASSIGNED
logstash-2016.09.03 2 r UNASSIGNED
logstash-2016.09.01 4 r UNASSIGNED
logstash-2016.09.01 0 p UNASSIGNED
logstash-2016.09.01 0 r UNASSIGNED
logstash-2016.09.01 3 r UNASSIGNED
logstash-2016.09.01 1 p UNASSIGNED
logstash-2016.09.01 1 r UNASSIGNED
logstash-2016.09.01 2 r UNASSIGNED
logstash-2016.08.09 2 p UNASSIGNED
logstash-2016.08.09 2 r UNASSIGNED
logstash-2016.08.09 0 r UNASSIGNED
logstash-2016.08.09 3 r UNASSIGNED
logstash-2016.08.09 1 r UNASSIGNED
logstash-2016.08.09 4 r UNASSIGNED
logstash-2016.08.12 4 r UNASSIGNED
logstash-2016.08.12 0 r UNASSIGNED
logstash-2016.08.12 3 r UNASSIGNED
logstash-2016.08.12 1 p UNASSIGNED
logstash-2016.08.12 1 r UNASSIGNED
logstash-2016.08.12 2 r UNASSIGNED
logstash-2016.08.13 2 p UNASSIGNED
logstash-2016.08.13 2 r UNASSIGNED
logstash-2016.08.13 0 r UNASSIGNED
logstash-2016.08.13 3 r UNASSIGNED
logstash-2016.08.13 1 r UNASSIGNED
logstash-2016.08.13 4 p UNASSIGNED
logstash-2016.08.13 4 r UNASSIGNED
logstash-2016.08.14 2 r UNASSIGNED
logstash-2016.08.14 0 r UNASSIGNED
logstash-2016.08.14 3 r UNASSIGNED
logstash-2016.08.14 1 p UNASSIGNED
logstash-2016.08.14 1 r UNASSIGNED
logstash-2016.08.14 4 r UNASSIGNED
logstash-2016.08.15 2 p UNASSIGNED
logstash-2016.08.15 2 r UNASSIGNED
logstash-2016.08.15 0 r UNASSIGNED
logstash-2016.08.15 3 r UNASSIGNED
logstash-2016.08.15 1 p UNASSIGNED
logstash-2016.08.15 1 r UNASSIGNED
logstash-2016.08.15 4 r UNASSIGNED
logstash-2016.08.16 2 p UNASSIGNED
logstash-2016.08.16 2 r UNASSIGNED
logstash-2016.08.16 0 r UNASSIGNED
logstash-2016.08.16 3 r UNASSIGNED
logstash-2016.08.16 1 r UNASSIGNED
logstash-2016.08.16 4 r UNASSIGNED
logstash-2016.08.17 4 p UNASSIGNED
logstash-2016.08.17 4 r UNASSIGNED
logstash-2016.08.17 0 r UNASSIGNED
logstash-2016.08.17 3 r UNASSIGNED
logstash-2016.08.17 1 r UNASSIGNED
logstash-2016.08.17 2 r UNASSIGNED
logstash-2016.08.18 2 p UNASSIGNED
logstash-2016.08.18 2 r UNASSIGNED
logstash-2016.08.18 0 r UNASSIGNED
logstash-2016.08.18 3 r UNASSIGNED
logstash-2016.08.18 1 r UNASSIGNED
logstash-2016.08.18 4 r UNASSIGNED
logstash-2016.08.19 2 p UNASSIGNED
logstash-2016.08.19 2 r UNASSIGNED
logstash-2016.08.19 0 p UNASSIGNED
logstash-2016.08.19 0 r UNASSIGNED
logstash-2016.08.19 3 r UNASSIGNED
logstash-2016.08.19 1 r UNASSIGNED
logstash-2016.08.19 4 r UNASSIGNED
logstash-2016.08.30 2 r UNASSIGNED
logstash-2016.08.30 0 r UNASSIGNED
logstash-2016.08.30 3 p UNASSIGNED
logstash-2016.08.30 3 r UNASSIGNED
logstash-2016.08.30 1 r UNASSIGNED
logstash-2016.08.30 4 r UNASSIGNED
logstash-2016.08.31 4 r UNASSIGNED
logstash-2016.08.31 0 p UNASSIGNED
logstash-2016.08.31 0 r UNASSIGNED
logstash-2016.08.31 3 r UNASSIGNED
logstash-2016.08.31 1 r UNASSIGNED
logstash-2016.08.31 2 r UNASSIGNED
kibana-int 4 r UNASSIGNED
kibana-int 0 r UNASSIGNED
kibana-int 3 p UNASSIGNED
kibana-int 3 r UNASSIGNED
kibana-int 1 r UNASSIGNED
kibana-int 2 r UNASSIGNED
logstash-2016.08.10 4 p UNASSIGNED
logstash-2016.08.10 4 r UNASSIGNED
logstash-2016.08.10 0 r UNASSIGNED
logstash-2016.08.10 3 p UNASSIGNED
logstash-2016.08.10 3 r UNASSIGNED
logstash-2016.08.10 1 p UNASSIGNED
logstash-2016.08.10 1 r UNASSIGNED
logstash-2016.08.10 2 r UNASSIGNED
logstash-2016.08.11 4 r UNASSIGNED
logstash-2016.08.11 0 r UNASSIGNED
logstash-2016.08.11 3 p UNASSIGNED
logstash-2016.08.11 3 r UNASSIGNED
logstash-2016.08.11 1 r UNASSIGNED
logstash-2016.08.11 2 r UNASSIGNED
[root@TGCS018 /]#


Do I need to re assign all of these? by the number after the name ???

UPDATE UPDATE

I restarted the server

I got the same error as I first reported in the issue. Finally the signon screen came back after about 10 minutes.

Now I can not logon error could not get user specified id maybe wrong


using nagiosadmin

What is up with this product. seems very unstable to me

Help

[Box293]

I suggest adding another 8GB memory to the server and see if that helps.

[kwhogster]

Troy

increased ram to 16 GB which is double from 8 GB

Same results still can not access logserver

Could not get user. Specified ID may be wrong.

Also extremely slow to access takes long time to present the logon screen

[kwhogster]

To All

I think I know what the problem is now.

[root@TGCS018 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 50G 20K 100% /
devtmpfs 7.7G 0 7.7G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 8.6M 7.7G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/centos-home 26G 33M 26G 1% /home
/dev/sda1 497M 123M 375M 25% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/0
tmpfs 1.6G 0 1.6G 0% /run/user/1001
[root@TGCS018 ~]#


Look at root file system it is 100 %

What did logserver do to fill up this file system???

[rkennedy]

Log server requires disk space to operate. I'm guessing you dumped quite a bit of logs into it?

Running out of disk space is catastrophic in NLS terms, and some times unrecoverable. You'll want to increase the disk space as soon as possible to correct this.

[Box293]

Here's how to add disk space if you've used the VM provided by us.

https://support.nagios.com/kb/article.php?id=486

[kwhogster]

Troy,

Two things.

1. What VM provide by us? please provide link to that and I will rebuild that VM

2. The instructions were good but the VM when I go to Edit Settings with the VM powered off the Disk provisioning settings are greyed out I can not modify them


One Last thing

Where are all the files that NLS stores located? I can delete them

Is there a way to tell NLS where to put the files?

I can add additional storage on this VM just can not modify the original disk

Thanks

[Box293]

1. What VM provide by us? please provide link to that and I will rebuild that VM

The VM that can be downloaded here.
http://library.nagios.com/library/produ ... /downloads

2. The instructions were good but the VM when I go to Edit Settings with the VM powered off the Disk provisioning settings are greyed out I can not modify them

Perhaps your VM has an active snapshot, you cannot increase the size of a virtual disk if a snapshot currently exists.

Where are all the files that NLS stores located? I can delete them

/usr/local/nagioslogserver/elasticsearch/
I would not try and delete them, they are part of the elasticsearch database.

Is there a way to tell NLS where to put the files?

You could move /usr/local to another partition. This guide explains how to move /var/log and could be easily adapted to move /usr/local
https://support.nagios.com/kb/article.php?id=473

[mcapra]

Split post into the following thread: https://support.nagios.com/forum/viewto ... =6&t=39489