Integration with 3rd party Applications like SAP Solution Ma

[cdienger]

Nagios doesn't monitor incoming emails like this natively. A custom plugin would be needed to monitor incoming emails and generate the necessary Nagios friendly exit codes based on criteria programed into the script. Plugin development guidelines are covered here:

https://nagios-plugins.org/doc/guidelines.html

Integrating a plugin into XI is covered here:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

[SteveBeauchemin]

Just to encourage you here, this can be done. I do it... I have more than 20 'Appliances' where there is no agent, and no SNMP. But they can send emails. Also I have vCenter send ESX stuff this way since the vCenter and ESX stuff knows all about itself.

My system is running postfix.
I use a POP mail account that the systems email to, something like nagiosalerts @ domain.com
Then I use fetchmail on the Nagios core system to get the emails.
Then use procmail to decide what to do with the emails once I get them. It basically matches a Subject: or From: and then sends the email to a script.
Then the script pulls in the email, reads it, and makes decisions. I actually have it send to a Nagios service in almost all cases.
I like to see it in the GUI.
Then Nagios deals with sending the alert.

The script that reads the email may need to decode it. Such as base64, or quoted printable coding.
Then read it, make it human readable, and send it to the Nagios pipe.

I'm including a bunch of example fetchmail and procmail configuration info to help get you jump started.
Also a small example of perl code I use to read the email and send to Nagios. Slightly sanitized.

I do all this as the Nagios user. So these files are in /home/nagios

So, in no particular order, here are the files and code that will get you 95% of the way.

Start by getting the mail
cat fetch.sh

Code: Select all

#!/bin/sh
# First check the log status and rotate if needed
/usr/sbin/logrotate -s /home/nagios/.logrotate.status /home/nagios/.logrotate.conf
# Then check for POP mail
/usr/bin/fetchmail

Local logrotate used by the nagios user - we cannot let these files get too
large or the fetchmail process stops reading from POP server
cat .logrotate.conf

Code: Select all

# Nagios user logrotate configuration file for Procmail and Fetchmail
# Old versions of log files are compressed with gzip by default.
compress
# Archive old versions of log files adding a daily extension like YYYYMMDD instead of simply adding a number.
dateext
dateformat -%Y%m%d%H
# Log files are rotated 5 times before being removed
rotate 5
# When the files exceed this size do the rotation
size 4500k
# Make sure the new file exists after rotate
create 0600 nagios nagios
# Do these files
/home/nagios/procmailrc.log { }
/home/nagios/fetchmail.log { copytruncate }

cat .fetchmailrc

Code: Select all

# Custom POP Mail for Nagios
# this file name is .fetchmailrc
# For the cron entry use the following
# */3 * * * * /home/nagios/fetch.sh &> /dev/null
set logfile /home/nagios/fetchmail.log
poll popserver.domain.com protocol POP3
user nagiosalerts
pass NaGiOsPaSSWord-orSomething
no keep
no rewrite

Procmail
cat .procmailrc

Code: Select all

# this file name is .procmailrc
#
HOME=/home/nagios
LOGFILE=$HOME/procmailrc.log
DEFAULT=$HOME/Mail/
MAILDIR=$HOME/Mail/
VERBOSE=yes
PATH=/usr/bin
VCENTERAPPLIANCE=/usr/local/nagios/libexec/eventhandlers/mailproc-vcenter-appliance.pl
# example of filter for To: From: Subject: - each can be used individually per filter
# * ^To:.*[email protected]
# * ^Subject:.*ack.*PROBLEM~alert
# * ^From:.*[email protected]
# end of examples
# Messages we do not want
# Out of Office
:0
* ^Subject:.*Out.*of.*Office
/dev/null
# Messages we do not want
# We do not send messages to ourself
:0
* ^From:.*[email protected]
/dev/null
# vCenters - Hardware Alerts
:0 b
* ^From:.*[email protected]
| $VCENTERAPPLIANCE
# After your setup works you can uncomment the below 
# items to dump any bogus email to /dev/null
#================================= End of normal stuff
# Catch all for other RC emails - send to Mail/new directory
# Catch all for other RC emails - send to Dev Null
#:0
#* ^From:.*MY-Domain.com
#/dev/null
# You will find unmatched emails in the ~/Mail/new directory. 
# Read them and use them to construct other .procmail entries

I run this as a cron every 3 minutes
- depending on how much system load you can tolerate you can change this
# crontab -l | grep -i fetch

Code: Select all

# Nagios fetchmail every 3 minutes
*/3 * * * * /home/nagios/fetch.sh &> /dev/null

for Postfix
edit /etc/aliases
This is a smal portion...

Code: Select all

#       >>>>>>>>>>      The program "newaliases" must be run after
#       >> NOTE >>      this file is updated for any changes to
#       >>>>>>>>>>      show through to sendmail.
# added to the bottom
# Redirect Nagios alerting emails to nagios user
nagiosalerts:   nagios

and Remember to run the newaliases command after you add the alias!!

edit /etc/postfix/main.cf

Code: Select all

#I have changed the following for my site - find them in the file as they are pre-configured
# set the myhostname
myhostname = my-nagios-host-getting-the-email.com
# set the mydomain
mydomain = yeah-my-domain-name.com
# set the relayhost
relayhost = my-smtp-relay-host-in-case-I-email-alerts-from-nagios-smtp.domain.com

# this is important
mailbox_command = /usr/bin/procmail -a "$EXTENSION" DEFAULT=$HOME/Mail/ MAILDIR=$HOME/Mail

# Also, some messages could be large and still matter so I do this at the bottom of the file
# let large messages be sent, such as those with PDF file attachments
#
message_size_limit = 20480000

Now the best part. A perl script that will read the email body and make sense out of it for a human and Nagios.
I have all of these perl in /usr/local/nagios/libexec/eventhandlers/ owned by nagios user and executable.
Some use SQL to get data from the Nagios DB, some send emails, some send to Servicenow, and some are silmpy to get the message into Nagios GUI and have Nagios do the rest. I added some hints to the code for decode.

Code: Select all

#!/usr/bin/perl
use strict;
use warnings;
use Date::Format;
use MIME::QuotedPrint;
# use MIME::Base64;

# set to 0 zero to stop logging
my $DEBUG = 1;
# the next line will not work unless you change it
my $LOG = "/usr/local/var/log/postfix/postfix-[my custom app name here].log";
open(OUTPUT, '>>', $LOG) ;

my $host = "Host-Placeholder-in-nagios";
my $service = "Service-placeholder-in-nagios-for-email";
#my $state = 0; # OK - Green
my $state = 1; # WARNING - Yellow
#my $state = 2; # CRITICAL - Red
#my $state = 3; # UNKNOWN - Orange
my $message = "";
my $encoded = "";
my $decoded = "";

# get a UNIX timestamp
my $dtNow = time();
chomp $dtNow;

# read input to the perl script - this should only be the body of the email
while ( <> ) {
  $encoded .= $_;
}
# Magic happens here...  Decode the encoded body
$decoded = decode_qp($encoded);
# $decoded = decode_base64($encoded);

$message = "WARNING: ";

my @lines = split /\n/, $decoded;
foreach my $line (@lines) {
  chomp $line;
  $line =~ s/\s+/ /g;

  # just testing...
  if ($line =~ m/^test$/) {
    print "testing\n";
    exit;
  }

  # how to ignore lines we do not care about
  next if ($line =~ m/^$/);
  next if ($line =~ m/^\s+$/);
  next if ($line =~ m/^E-Mail.*To.*Address/);
  # add other tweaking of mail input data here

  $message .= $line . "<BR>";
}

# Location of Nagios Pipe
my $strCommandPipe = "/usr/local/nagios/var/rw/nagios.cmd";

#Create the completed syntax that will go to the Nagios Pipe
my $result = "PROCESS_SERVICE_CHECK_RESULT;$host;$service;$state;$message";

# Open the pipe and send the properly formatted text
open(F, ">>$strCommandPipe") or die print "Cannot write to Nagios Pipe - $!\n";
print F "[$dtNow] $result\n";
close F;

# save some data for posterity if the Debug is set up top
if ($DEBUG =~ '1') {
  my $eventtime = time2str("%Y-%m-%d %I:%M %p %Z", $dtNow);
  open(OUTPUT, '>>', $LOG) ;
  print OUTPUT "---------------------------------------------------------" .  $eventtime . " \n" ;
  print OUTPUT " Input Decoded:\n$decoded\n";
  close OUTPUT;
}
__END__

Final tid-bits to think about - the service definition needs to be a passive test, and in order to treat
each email as a new message you have to set is_volatile to 1 in the service definition.
In Nagios XI CCM Service definition on the Check Settings tab, set 'Is Volatile' to On.
Here is a possible Service Template definition for Passive tests that pull in Log data
or Email data where each item needs to be treated as a new alert.
Notice the is_volatile 1 and Freshness interval settings.

Code: Select all

define service {
    name                            Service-Template-Generic-Passive-Email-and-Logs
    service_description             Send to the Service Now tool
    check_command                   check-dummy-service!0!No messages received in the last 4 hours!!!!!!
    is_volatile                     1
    initial_state                   o
    max_check_attempts              1
    active_checks_enabled           1
    passive_checks_enabled          1
    check_period                    24x7
    parallelize_check               1
    obsess_over_service             0
    check_freshness                 1
    freshness_threshold             14400
    event_handler_enabled           0
    flap_detection_enabled          0
    process_perf_data               1
    retain_status_information       1
    retain_nonstatus_information    1
    notification_interval           604800
    notification_options            w,c,
    notifications_enabled           1
    register                        0
}

So many tiny pieces hook together to make this work. I have been getting emails into Nagios XI for so long I forgot all these. This has been a good review for me to put this together. I hope it doesn't scare you off. Each piece can be put together and tested manually. If you do this, take your time and get each piece working individually before you chain them all together.

Good Luck

Steve B

If I have mistakes in the stuff above, remember, Google is your friend.

[cdienger]

Thanks for the input @SteveBeauchemin !