On NLS client rsyslog/python/httpd processing stopped

[GhostRider2110]

Interesting, open files limit is showing 1024.

[tmcdonald]

Might not be a bad idea to at least double that.

How many open files does the nagios user have?

Code: Select all

lsof -u nagios | wc -l

[GhostRider2110]

On the Nagios server? or the NLS? It's the clients that are hanging.

See-ya
Mitch

[tmcdonald]

Right, sorry. I've got my mind stuck on Logserver. I believe your rsyslog should be running on the clients as root, so:

Code: Select all

lsof -u root | wc -l

And to display the actual hard limit to compare against (as opposed to just the soft limit) run:

Code: Select all

ulimit -aH

as root as well.

[GhostRider2110]

Code: Select all

[root@igapubrep01 lsaadmin]# lsof -u root | wc -l
1436
[root@igapubrep01 lsaadmin]# ulimit -aH
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30460
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30460
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[jolson]

I restarted electicsearch and everything started working again.

Anything in the elasticsearch logs?

Code: Select all

cat /var/log/elasticsearch/*.log

Be sure to check some of the compressed logs in /var/log/elasticsearch - there may be a hint in there somewhere.
It sounds to me like logstash was having trouble with outputting logs to elasticsearch. This could be caused by a host of things - but since restarting logstash did not help, we have to look at elasticsearch.

Regarding your client machines that are getting bogged down - let us know if upping the file limit helps prevent the slowdown at all.

[GhostRider2110]

Looking through them.... I did figure out why my snapshots were not processing as I had mentioned a while back, getting permission denied. I must have not changed the ownership of the datastore to nagios when I created it. Will keep looking through the logs and see if I see anything out of place...

[jolson]

Let us know what you find out - thanks Mitch.