Assistance with Nagios Logging

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
mmccaugh
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Assistance with Nagios Logging

Post by mmccaugh »

I apologize if this has been asked, I have been doing some reading but cannot seem to find an answer.

When service, or host notifications are disabled should notifications still be writing to nagios.log? And if so is there a way to prevent it?

I am using nagios.log for some of my alerting, as the alerting out of Nagios core itself is shall we say, sometimes a bit chatty.. But the issue I am faced with now is that if I disable notifications it has no affect on this pipe as the notifications still write out to the log, so I am looking for either a way to stop this (Only when notifications are disabled) or at least identify somehow that the log entry was generated by a disabled host or service so I can programatically disregard it..

Is this possible?
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Assistance with Nagios Logging

Post by cdienger »

Setting:

Code: Select all

log_notifications=0
should do the trick.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
mmccaugh
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Post by mmccaugh »

I think this will actually break what I am doing though, note I DO want these to go to the logfile if notifications are enabled, I just want them to stop if i disable notifications for a service or host.. This change looks like it would actually disable them globally.
mmccaugh
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Post by mmccaugh »

OK the more reading I do the more I am thinking that what I want to do might not actually be possible with how I want to do it. But I do have an idea I think might work.

I will need to do a lot more reading, but basically looking at contacts.cfg where we define the notification actions (Currently notify-by-email or a variant) which appears to be defined in misccommands.cfg

The thought I had was to simply write a new function into the misccommands.cfg that will (If possible) generate an output line close to, or identical to what we see in nagios.log, and just write that to a new logfile (nagiosalerts.log for instance?) I could splunk THAT logfile instead and do what I need to do, and as we would be using the built in notification heirarchy what I am doing would be properly toggled on and off when notifications were toggled.

If anyone is familiar with the structure of these commands (Or rather the variables Nagios has available for me to pass as they look like simple printf's) that would be VERY helpful to me..
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Assistance with Nagios Logging

Post by mcapra »

At a high level, what I think you want to do is have a custom notification handler -- one that both handles your emailing needs, and one that could dump the notifications to some other service. In this case your service is Splunk, but it could be a generic logfile to be consumed by a Splunk forwarder, a syslog agent, fluentd, etc. A notification handler is a type of event handler.

The default Nagios Core event handlers looks like this, one for host objects and the other for service objects (as of 4.2.x anyway):

Code: Select all

define command{
        command_name    notify-host-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
        }

# 'notify-service-by-email' command definition
define command{
        command_name    notify-service-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
        }
These notification handlers do some string formatting with Nagios macros, and hand the result off to the /bin/mail command passing in the $CONTACTEMAIL$ macro. You could just as easily pass an echo command with some of these macros into a logfile to be consumed by your Splunk forwarder. Or append an echo command onto the existing handler. Or create a new handler and add it to a particular contact's host_notification_commands or service_notification_commands directive -- lots of options.

If this were my problem to solve, I'd just dump all the macros I care about for each host/service notification into some comma-separated format via an echo. Might look like this:

Code: Select all

define command{
        command_name    notify-service-to-csv
        command_line    echo "$SERVICEDISPLAYNAME$,$SERVICEOUTPUT$,$SERVICESTATE$" >> /path/to/logfile.log
        }
Then have your Splunk forwarder, syslog agent, etc, handle consuming/truncating that file and mapping those csv values into whatever you need. Or handle it in ingest.
Former Nagios employee
https://www.mcapra.com/
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Assistance with Nagios Logging

Post by cdienger »

Thanks for the input, @mcapra.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
mmccaugh
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Post by mmccaugh »

I'm going to play with this (Probably next week at this point) but I'll post what I end up with!

Biggest issue is that I do not have a dev nagios box built currently, so I have to generate alerts to test currently. I will probably spin up a test box for this so I can play without limits.

Thanks for the feedback all!
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Assistance with Nagios Logging

Post by ssax »

No problem, we'll keep an eye out, let us know if you have any related questions.
mmccaugh
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Post by mmccaugh »

define command {
command_name notify-service-by-logfile
command_line /usr/bin/printf "%b" "$LONGDATETIME$ nagiosdev nagios: SERVICE ALERT: $HOSTNAME$~$SERVICEDESC$~$SERVICESTATE$~$SERVICESTATETYPE$~$SERVICEATTEMPT$~$SERVICEOUTPUT$\n" >>/var/log/nagios/nagioscustom2.log
}

Biggest issue I had was semi colons, there was no escaping them that I found, so I ended up just swapping them for tildes for now, the above generates the same line that goes to syslog (Or Nagios.log), I opted for long date rather than epoch time but either is possible.

I can expand on this next (As I don't really need to tie myself to just what nagios.log captures now) but it works, and should I suspect resolve my issue!
mmccaugh
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Post by mmccaugh »

[root@nagiosdev nagios]# cat nagioscustom2.log
Mon Jul 8 18:16:18 EDT 2019 nagiosdev nagios: SERVICE ALERT: DEV-X6565-lin~PING~CRITICAL~HARD~3~CRITICAL - Host Unreachable (10.6.20.97)
Mon Jul 8 18:31:22 EDT 2019 nagiosdev nagios: SERVICE ALERT: DEV-X6565-lin~PING~OK~HARD~3~PING OK - Packet loss = 0%, RTA = 1.18 ms

This is output written to the new logfile (Logfile name and some other stuff will change, but this works!)

Note I only set up a ping check for the new host as I am testing, but this will work for all my checks.
Locked