Assistance with Nagios Logging

An open discussion forum for obtaining help with Nagios Core. Nagios Core users of all experience levels are welcome here. Subforum have been created for the discussion of Nagios Core and Nagios Plugin development.

NOTE: The SourceForge.net mailing lists have been deprecated in favor of this forum in order to expedite support and provide additional features not available on the old mailing list.

Assistance with Nagios Logging

Postby mmccaugh » Mon Jun 24, 2019 3:23 pm

I apologize if this has been asked, I have been doing some reading but cannot seem to find an answer.

When service, or host notifications are disabled should notifications still be writing to nagios.log? And if so is there a way to prevent it?

I am using nagios.log for some of my alerting, as the alerting out of Nagios core itself is shall we say, sometimes a bit chatty.. But the issue I am faced with now is that if I disable notifications it has no affect on this pipe as the notifications still write out to the log, so I am looking for either a way to stop this (Only when notifications are disabled) or at least identify somehow that the log entry was generated by a disabled host or service so I can programatically disregard it..

Is this possible?
mmccaugh
 
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Postby cdienger » Mon Jun 24, 2019 4:57 pm

Setting:

Code: Select all
log_notifications=0

should do the trick.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
cdienger
Support Tech
 
Posts: 3791
Joined: Tue Feb 07, 2017 11:26 am

Re: Assistance with Nagios Logging

Postby mmccaugh » Wed Jun 26, 2019 11:41 am

I think this will actually break what I am doing though, note I DO want these to go to the logfile if notifications are enabled, I just want them to stop if i disable notifications for a service or host.. This change looks like it would actually disable them globally.
mmccaugh
 
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Postby mmccaugh » Wed Jun 26, 2019 5:06 pm

OK the more reading I do the more I am thinking that what I want to do might not actually be possible with how I want to do it. But I do have an idea I think might work.

I will need to do a lot more reading, but basically looking at contacts.cfg where we define the notification actions (Currently notify-by-email or a variant) which appears to be defined in misccommands.cfg

The thought I had was to simply write a new function into the misccommands.cfg that will (If possible) generate an output line close to, or identical to what we see in nagios.log, and just write that to a new logfile (nagiosalerts.log for instance?) I could splunk THAT logfile instead and do what I need to do, and as we would be using the built in notification heirarchy what I am doing would be properly toggled on and off when notifications were toggled.

If anyone is familiar with the structure of these commands (Or rather the variables Nagios has available for me to pass as they look like simple printf's) that would be VERY helpful to me..
mmccaugh
 
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Postby mcapra » Thu Jun 27, 2019 2:19 pm

At a high level, what I think you want to do is have a custom notification handler -- one that both handles your emailing needs, and one that could dump the notifications to some other service. In this case your service is Splunk, but it could be a generic logfile to be consumed by a Splunk forwarder, a syslog agent, fluentd, etc. A notification handler is a type of event handler.

The default Nagios Core event handlers looks like this, one for host objects and the other for service objects (as of 4.2.x anyway):
Code: Select all
define command{
        command_name    notify-host-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
        }

# 'notify-service-by-email' command definition
define command{
        command_name    notify-service-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
        }


These notification handlers do some string formatting with Nagios macros, and hand the result off to the /bin/mail command passing in the $CONTACTEMAIL$ macro. You could just as easily pass an echo command with some of these macros into a logfile to be consumed by your Splunk forwarder. Or append an echo command onto the existing handler. Or create a new handler and add it to a particular contact's host_notification_commands or service_notification_commands directive -- lots of options.

If this were my problem to solve, I'd just dump all the macros I care about for each host/service notification into some comma-separated format via an echo. Might look like this:
Code: Select all
define command{
        command_name    notify-service-to-csv
        command_line    echo "$SERVICEDISPLAYNAME$,$SERVICEOUTPUT$,$SERVICESTATE$" >> /path/to/logfile.log
        }


Then have your Splunk forwarder, syslog agent, etc, handle consuming/truncating that file and mapping those csv values into whatever you need. Or handle it in ingest.
Former Nagios employee
http://www.mcapra.com/
User avatar
mcapra
 
Posts: 3587
Joined: Thu May 05, 2016 3:54 pm

Re: Assistance with Nagios Logging

Postby cdienger » Thu Jun 27, 2019 4:26 pm

Thanks for the input, @mcapra.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
cdienger
Support Tech
 
Posts: 3791
Joined: Tue Feb 07, 2017 11:26 am

Re: Assistance with Nagios Logging

Postby mmccaugh » Fri Jun 28, 2019 11:03 am

I'm going to play with this (Probably next week at this point) but I'll post what I end up with!

Biggest issue is that I do not have a dev nagios box built currently, so I have to generate alerts to test currently. I will probably spin up a test box for this so I can play without limits.

Thanks for the feedback all!
mmccaugh
 
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Postby ssax » Fri Jun 28, 2019 2:41 pm

No problem, we'll keep an eye out, let us know if you have any related questions.
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
ssax
Dreams In Code
 
Posts: 4396
Joined: Wed Feb 11, 2015 12:54 pm

Re: Assistance with Nagios Logging

Postby mmccaugh » Mon Jul 08, 2019 5:32 pm

define command {
command_name notify-service-by-logfile
command_line /usr/bin/printf "%b" "$LONGDATETIME$ nagiosdev nagios: SERVICE ALERT: $HOSTNAME$~$SERVICEDESC$~$SERVICESTATE$~$SERVICESTATETYPE$~$SERVICEATTEMPT$~$SERVICEOUTPUT$\n" >>/var/log/nagios/nagioscustom2.log
}

Biggest issue I had was semi colons, there was no escaping them that I found, so I ended up just swapping them for tildes for now, the above generates the same line that goes to syslog (Or Nagios.log), I opted for long date rather than epoch time but either is possible.

I can expand on this next (As I don't really need to tie myself to just what nagios.log captures now) but it works, and should I suspect resolve my issue!
mmccaugh
 
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Re: Assistance with Nagios Logging

Postby mmccaugh » Mon Jul 08, 2019 5:35 pm

[root@nagiosdev nagios]# cat nagioscustom2.log
Mon Jul 8 18:16:18 EDT 2019 nagiosdev nagios: SERVICE ALERT: DEV-X6565-lin~PING~CRITICAL~HARD~3~CRITICAL - Host Unreachable (10.6.20.97)
Mon Jul 8 18:31:22 EDT 2019 nagiosdev nagios: SERVICE ALERT: DEV-X6565-lin~PING~OK~HARD~3~PING OK - Packet loss = 0%, RTA = 1.18 ms

This is output written to the new logfile (Logfile name and some other stuff will change, but this works!)

Note I only set up a ping check for the new host as I am testing, but this will work for all my checks.
mmccaugh
 
Posts: 8
Joined: Fri Sep 28, 2018 11:49 am

Next

Return to Nagios Core

Who is online

Users browsing this forum: Google [Bot] and 19 guests