Monitoring Nagios Log Server Processes

[sgiworks]

Hello,

Recently we came across a problem with logstash service going down. Is there a plugin to configure monitoring of following Log Server processes/ services from Nagios XI?

elasticsearch status and logstash status

Regards,
Swapnil

[avandemore]

You can install something like NCPA or NRPE to check and report on the service status. These types of checks work against arbitrary services. It is also possible to monitor by process, but that can be much more complicated.

Most of this depends on how in-depth you want to go here. A simple check_by_ssh could also work.

[sgiworks]

I want to monitor the disk space, because recently we have a problem with disk getting exhausted which broke the entire server.

[mcapra]

Both the "Linux Server" and "NCPA" configuration wizards should be able to do that if the respective agents are installed in the NLS machine. Here's documentation for setting them up:

https://assets.nagios.com/downloads/nag ... _Agent.pdf
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

[sgiworks]

Great, this was helpful.

Now next is to have the core services monitored.
Elasticsearch Database
Logstash Collector

[mcapra]

Depending on whether or not you're using NRPE/NCPA, there are respective commands for checking to see if a particular process is running. Which agent are you using?

[sgiworks]

NRPE

[mcapra]

If my Nagios Log Server machine was located at 192.168.67.4, and I wanted to use check_nrpe to see if the Logstash process was running, I could do the following:

Code: Select all

[root@xi-stable ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.67.4 -c check_procs -a '-c 1: -w 1: -C runuser'
PROCS OK: 1 process with command name 'runuser' | procs=1;1:;1:;0;

This takes advantage of the fact that Logstash lives as a "runuser" process. Like so:

Code: Select all

root     19830  0.0  0.0 144248  1012 ?        SN   Mar29   0:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log  -w 4 nagios

Though this assumes you are using the check_procs plugin and have it defined in your NRPE configuration set as the check_procs command like so:

Code: Select all

command[check_procs]=/usr/local/nagios/libexec/check_procs $ARG1$

Or maybe I want to use the check_services plugin to more generally see if logstash/elasticsearch is in my ps output. I could use a NRPE command like this:

Code: Select all

command[check_services]=/usr/local/nagios/libexec/check_services -p $ARG1$

Executed from my Nagios XI machine like this:

Code: Select all

[root@xi-stable ~]# /usr/local/nagios/libexec/check_nrpe -H 192.168.67.4 -c check_services -a 'logstash'
  logstash: 3

So there's a few options. You'll need to decide which one makes more sense for your use case.