Page 1 of 1
Logstash "Failover/Loadbalancing" on many Nagios Instances
Posted: Wed Oct 07, 2015 8:38 am
by milan
I have a question regarding the possibility (settings?) of Nagios Log Server to receive/filter/output Log-Messages using Logstash process on many nodes within one cluster, as logstash is effectively processing logs only on the "primary node".
As I want to have Failover/Loadbalancing on many Instances (using Loadbalancer) I would like to set Nagios Log Server to receive Logs over any Instance in the cluster.
I have tried to set "per instance" inputs and outputs, but it did not help.
Right now I can only receive Logs on the "primary node", whereas logstash service is active on all nodes, but they are just "pending" and cannot receive/process any logs....
Does someone had experience on that?
Is
Thanks and best regards
Milan
Re: Logstash "Failover/Loadbalancing" on many Nagios Instanc
Posted: Wed Oct 07, 2015 10:42 am
by jolson
What do you mean they are 'pending' on your other nodes? By default, any node on your cluster should be capable of accepting logs and exporting them to your cluster.
Try running the following on each node of yours:
Code: Select all
service logstash restart
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
cat /var/log/logstash/logstash.log
cat /var/log/messages
The above should help us figure out what's going on here.
Jesse
Re: Logstash "Failover/Loadbalancing" on many Nagios Instanc
Posted: Thu Oct 08, 2015 2:27 am
by milan
Hi Jesse
"Pending" means that the logstash process on the other (slave) nodes do not do any job when the logs are pointed to go on thier IP Address.
Example: I have 4 Nagios Log Server Nodes with IP's: 192.168.1.100, 192.168.1.101, 192.168.1.102 and 192.168.1.103. Node with IP 192.168.1.100 is master node.
The logstash process is running on all 4 nodes ("green" chekmarks under Administration->Instance Status).
For testing purpose I use "Snare" App to Windows Event (Audit) Logs to Nagios Log Server. I'm using UDP, Port 514 (Nagios Log Server is configured listening on priviledged Port 514...)
If I configure "Snare" to send on the IP 192.168.1.100, everything goes perfect, but when I configure "Snare" to send Logs on the other IP's from stack, it does not function...
I tried all possible combinations with setting global and "per instance" inputs/filters/outputs but it did not helped. All inputs/filters/outputs are correct and logstash process does not have any problem (no errors in the logstash log).
I have tried to check the UDP traffic using "tcpdump" and saw the logs are comming indeed over UDP/514 to the Nodes, but the logstash is doing nothing with them (except on the master node).
Best Regards
Milan
Re: Logstash "Failover/Loadbalancing" on many Nagios Instanc
Posted: Thu Oct 08, 2015 1:04 pm
by jolson
I do understand - if you could run the commands that I posted above it would help us figure out if there are any configuration problems.
For reference:
Code: Select all
service logstash restart
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
cat /var/log/logstash/logstash.log
cat /var/log/messages
Also, are you certain that port 514 is open on all of your nodes, and that logstash is configured to listen in privileged mode?
https://assets.nagios.com/downloads/nag ... Server.pdf