Nagios Log Server Logstash Inputs Monitoring

[madhukarneelaiahgari]

Hi Team,

We use gelf and nxlog inputs for receiving logs from different applications.

We had an issue yesterday with logstash not processing the logs that are sent from multiple services ex: using gelf plugins and nxlog log shipper.

I had to restart the logstash service on the nagios log servers to make it work again.

Are there any monitoring checks available to check these collectors/logstash/inputs, so that we will be notified of the issue when something is broken ?

The logstash service was running on nagios log server but logstash was not processing the logs from the input plugins. Below are the input configs

UPD Input:

udp {
type => 'plain_udp'
port => 1514
codec => plain
}

Gelf:

gelf { }

Above inputs were in active state when the issue happened.

Regards,
Madhu.

[jdunitz]

Hello Madhu!

I'm assuming you also have an XI server in addition to your Nagios log server.

There are a few ways you could approach monitoring the health and status of logstash.

One would be to use the config wizard for "log server". Go to Configure->Configuration Wizards and search for "log server" (with the space).
From there, you can set up a query against your log server that can look for a particular string that would let you know things are working, and alert you if something isn't right.

Another approach would be to find or write a script to run on the log server itself that would interrogate the logstash API and provide status.
You could do this by setting up an NCPA agent on the log server, writing a plugin script for it (just put it in /usr/local/ncpa/plugins), and use the NCPA config wizard on your XI server to monitor the status of your plugin. You could write a script that would be as simple or complex as you need.

We have a guide on our knowledgebase with directions for monitoring Log Server with NCPA.
Nagios Log Server - Monitoring Using NCPA + Nagios XI

Hopefully this will get you pointed in the right direction, and let us know if you need more specific detail.

[mcapra]

You could also configure a file input to process Nagios Log Server's local Logstash log for errors, assuming Logstash isn't crashing and is just junking logs on certain inputs.

Most Logstash input plugins also allow you to tag messages that pass through the input:
https://www.elastic.co/guide/en/logstas ... -gelf-tags

I forget how alerts in NLS work, but I'd think you could wire up an alert to fire when the doc count for your "gelf" or "udp" tagged messages goes below a certain threshold.

[benjaminsmith]

Thanks @mcapra!

Let us know if you need clarification on anything.

[madhukarneelaiahgari]

Sure let me configure the check and get back to you.

[benjaminsmith]

Hi,

Sounds good. We'll keep this open.