Trying to figure out why logstash changed to active (exited)

[mbellerue]

Just the TCP connections of the node that fails.

[rferebee]

Ok, I'll have my WAN team generate a report for you. It'll probably have to wait until next week.

[mbellerue]

Okay. We'll keep the thread open and wait to hear back.

[rferebee]

Hello,

Unfortunately, TCP connections are not something we log (we only log denials), so I am unable to generate a report with that information.

Is there anything else you'd like to look at? I will say that we went all weekend without the logstash service exiting on any of the nodes, so that's good. Also, all of the snapshots completed successfully.

[cdienger]

We can get this information from the a plugin like https://exchange.nagios.org/directory/P ... es/details and using the NCPA agent. https://support.nagios.com/kb/article/n ... i-857.html goes over setting it up. Would this be possible? If so, there is one tweak that needs to be made to this plugin - Edit it to remove the comments at the top so that the first line is "#!/bin/bash". Then the plugin would need to be copied to /usr/local/ncpa/plugins/ on the NLS machine(set permissions to "chmod 755 check_tcp_connections") and then you can run from XI.

The command would look something like:

Code: Select all

./check_ncpa.py -H NLS_IP -t '<your token>' -M 'plugins/check_tcp_connections' -q 'args=-s a -w 99998 -c 99999'

[rferebee]

We haven't begun using the NCPA agent yet. I'll need to install and configure that first. Of course, that will require approval.

I'll get back to you.

[cdienger]

Sounds good. Keep us posted.

[cdienger]

Also, are all your clients pointing to a single NLS instance are you doing any load balancing? Balancing the incoming data among multiple NLS instances can help prevent overloading a single logstash process.

[rferebee]

We point all of our clients at a DNS name that comprises the IP addresses of all the Log Server nodes.

I've been trying to get load balancing to work in our environment for Log Server, but I haven't had much luck. I know it's not load balancing right now and I've opened support tickets in the past concerning that issue, but I haven't got a lot of direction from Nagios.

It's something we need to figure out internally. I just don't know the "best" way to approach it.

[cdienger]

It's probably not the "best" way, but even configuring half the clients to go to one NLS machine and the other half to go to another would probably help. Maybe setting a couple DNS records so one hostname can resolve to two of the machines and another resolving to the other two. You'd have a bit of redundancy with that as well.