Page 2 of 5
Re: Trying to figure out why logstash changed to active (exi
Posted: Fri Nov 15, 2019 3:56 pm
by mbellerue
Just the TCP connections of the node that fails.
Re: Trying to figure out why logstash changed to active (exi
Posted: Fri Nov 15, 2019 4:10 pm
by rferebee
Ok, I'll have my WAN team generate a report for you. It'll probably have to wait until next week.
Re: Trying to figure out why logstash changed to active (exi
Posted: Fri Nov 15, 2019 5:20 pm
by mbellerue
Okay. We'll keep the thread open and wait to hear back.
Re: Trying to figure out why logstash changed to active (exi
Posted: Mon Nov 18, 2019 2:21 pm
by rferebee
Hello,
Unfortunately, TCP connections are not something we log (we only log denials), so I am unable to generate a report with that information.
Is there anything else you'd like to look at? I will say that we went all weekend without the logstash service exiting on any of the nodes, so that's good. Also, all of the snapshots completed successfully.
Re: Trying to figure out why logstash changed to active (exi
Posted: Mon Nov 18, 2019 5:10 pm
by cdienger
We can get this information from the a plugin like
https://exchange.nagios.org/directory/P ... es/details and using the NCPA agent.
https://support.nagios.com/kb/article/n ... i-857.html goes over setting it up. Would this be possible? If so, there is one tweak that needs to be made to this plugin - Edit it to remove the comments at the top so that the first line is "#!/bin/bash". Then the plugin would need to be copied to /usr/local/ncpa/plugins/ on the NLS machine(set permissions to "chmod 755 check_tcp_connections") and then you can run from XI.
The command would look something like:
Code: Select all
./check_ncpa.py -H NLS_IP -t '<your token>' -M 'plugins/check_tcp_connections' -q 'args=-s a -w 99998 -c 99999'
Re: Trying to figure out why logstash changed to active (exi
Posted: Mon Nov 18, 2019 6:00 pm
by rferebee
We haven't begun using the NCPA agent yet. I'll need to install and configure that first. Of course, that will require approval.
I'll get back to you.
Re: Trying to figure out why logstash changed to active (exi
Posted: Tue Nov 19, 2019 10:27 am
by cdienger
Sounds good. Keep us posted.
Re: Trying to figure out why logstash changed to active (exi
Posted: Tue Nov 19, 2019 10:30 am
by cdienger
Also, are all your clients pointing to a single NLS instance are you doing any load balancing? Balancing the incoming data among multiple NLS instances can help prevent overloading a single logstash process.
Re: Trying to figure out why logstash changed to active (exi
Posted: Tue Nov 19, 2019 10:51 am
by rferebee
We point all of our clients at a DNS name that comprises the IP addresses of all the Log Server nodes.
I've been trying to get load balancing to work in our environment for Log Server, but I haven't had much luck. I know it's not load balancing right now and I've opened support tickets in the past concerning that issue, but I haven't got a lot of direction from Nagios.
It's something we need to figure out internally. I just don't know the "best" way to approach it.
Re: Trying to figure out why logstash changed to active (exi
Posted: Tue Nov 19, 2019 2:21 pm
by cdienger
It's probably not the "best" way, but even configuring half the clients to go to one NLS machine and the other half to go to another would probably help. Maybe setting a couple DNS records so one hostname can resolve to two of the machines and another resolving to the other two. You'd have a bit of redundancy with that as well.