Page 4 of 4
Re: On NLS client rsyslog/python/httpd processing stopped
Posted: Mon Jun 01, 2015 7:53 am
by GhostRider2110
Interesting, open files limit is showing 1024.
Re: On NLS client rsyslog/python/httpd processing stopped
Posted: Mon Jun 01, 2015 9:02 am
by tmcdonald
Might not be a bad idea to at least double that.
How many open files does the nagios user have?
Re: On NLS client rsyslog/python/httpd processing stopped
Posted: Mon Jun 01, 2015 9:11 am
by GhostRider2110
On the Nagios server? or the NLS? It's the clients that are hanging.
See-ya
Mitch
Re: On NLS client rsyslog/python/httpd processing stopped
Posted: Mon Jun 01, 2015 9:20 am
by tmcdonald
Right, sorry. I've got my mind stuck on Logserver. I believe your rsyslog should be running on the clients as root, so:
And to display the actual hard limit to compare against (as opposed to just the soft limit) run:
as root as well.
Re: On NLS client rsyslog/python/httpd processing stopped
Posted: Mon Jun 01, 2015 9:23 am
by GhostRider2110
Code: Select all
[root@igapubrep01 lsaadmin]# lsof -u root | wc -l
1436
[root@igapubrep01 lsaadmin]# ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30460
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 30460
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Re: On NLS client rsyslog/python/httpd processing stopped
Posted: Mon Jun 01, 2015 4:06 pm
by jolson
I restarted electicsearch and everything started working again.
Anything in the elasticsearch logs?
Be sure to check some of the compressed logs in /var/log/elasticsearch - there may be a hint in there somewhere.
It sounds to me like logstash was having trouble with outputting logs to elasticsearch. This could be caused by a host of things - but since restarting logstash did not help, we have to look at elasticsearch.
Regarding your client machines that are getting bogged down - let us know if upping the file limit helps prevent the slowdown at all.
Re: On NLS client rsyslog/python/httpd processing stopped
Posted: Tue Jun 02, 2015 3:54 pm
by GhostRider2110
Looking through them.... I did figure out why my snapshots were not processing as I had mentioned a while back, getting permission denied. I must have not changed the ownership of the datastore to nagios when I created it. Will keep looking through the logs and see if I see anything out of place...
Re: On NLS client rsyslog/python/httpd processing stopped
Posted: Tue Jun 02, 2015 4:43 pm
by jolson
Let us know what you find out - thanks Mitch.