Page 1 of 2

How to do checks on PRI lines

Posted: Mon Jul 31, 2017 9:50 am
by donnyforbes
Right now we use a system called dozer and we are in the process of migrating over to "Nagios". We do checks for PRI lines that we have here at our location and need to know when they go down and come backup. I have placed the script in here now, however I need to know how I can implement this into Nagios. Please advise and help.

Code: Select all

#!/usr/bin/perl
use warnings;
use strict;

use Asterisk::AMI;
use Data::Dumper;
use XML::Dumper;

my @ALERT_EMAIL = ('[email protected]');
my @EMAIL = ('[email protected]'); my @SYSOP_PHONE = ('8300326', # justin work cell
                   '4535893', # Mike work cell
                   '7578034', # Jim work cell
                );

#my @ALERT_EMAIL = ('[email protected]');
#my @EMAIL = ('[email protected]');
#my @SYSOP_PHONE = ('8300326');

my $PERSIST_FILE = '/var/check_pri_persist.xml';

my $telco_contact_info = "Telepacific Premium Support: 888-611-8722, Acct# 134607"; my $aggplant_info = "Agg Plant: (951) 736-7680, backup: (951) 736-7661";

my $CALLFILE_TEMP_DIR = '/var/tmp';
my $ASTERISK_OUTGOING_DIR = '/var/spool/asterisk/outgoing';

my $CALLALERT = 0;

# ALERT:
# 1 = ten minute alert to ALERT_EMAIL
# 2 = one minute alert to EMAIL
# 3 = 1 and 2

# VARIABLES YOU SHOULD EDIT.
my %dests = (
	'Hq Span 7' => 			{ ALERT => 3, SERVER => 'tank',      USER => 'pri_checker', SECRET => 'Wjm8A4C2n2379kq3', SPAN => 7, ALERTINTERVAL => 10, ALERTTHRESHOLD => 5, CIRCUIT => '842641', PHONE => '', FAX => '' },
	'Hq Span 8' => 			{ ALERT => 3, SERVER => 'tank',      USER => 'pri_checker', SECRET => 'Wjm8A4C2n2379kq3', SPAN => 8, ALERTINTERVAL => 10, ALERTTHRESHOLD => 5, CIRCUIT => '842635', PHONE => '', FAX => '' },
	'Yard Span 7' => 		{ ALERT => 3, SERVER => 'ernestine', USER => 'pri_checker', SECRET => 'Wjm8A4C2n2379kq3', SPAN => 7, ALERTINTERVAL => 10, ALERTTHRESHOLD => 5, CIRCUIT => '13/HCGS/848840//PT', PHONE => '', FAX => '' },
	'Yard Span 8' => 		{ ALERT => 3, SERVER => 'ernestine', USER => 'pri_checker', SECRET => 'Wjm8A4C2n2379kq3', SPAN => 8, ALERTINTERVAL => 10, ALERTTHRESHOLD => 5, CIRCUIT => '537994003.1 ????', PHONE => '', FAX => '' },
	);

my $persist = ();

if(-e $PERSIST_FILE)
{
	$persist = xml2pl($PERSIST_FILE);
}

foreach((keys %dests))
{
	my $dest = $_;

	my $astman = Asterisk::AMI->new(PeerAddr => $dests{$dest}{SERVER},
					Username => $dests{$dest}{USER},
					Secret   => $dests{$dest}{SECRET},
					Events   => 'off',
			);

	my $response = $astman->action({ Action => 'Command',
					 Command => 'pri show span ' . $dests{$dest}{SPAN},
					});
	
#	print Dumper(\$response);

	my $subject = '';
	my $body = '';

	my $var = undef;

	my @cmdnode = $response->{CMD};

	foreach my $cmds (@cmdnode)
	{
		foreach my $cmd (@{$cmds})
		{
#			print "cmd: $cmd\n";

			if($cmd =~ /Status: (.*)/) {
				$var = $1;
			}
		}
	}

#	print "dest: $dest, status: $var\n";

	unless(defined $var) {
		$var = 'Error: Unable to parse response';
		$subject = "Subject: Script Error - Pri Outage $dest\n\n";
		$body = localtime(time) .  " Unable to parse response.\nLooking for 'Status:' in: \n\n" . Dumper(@cmdnode);
	}
	else
	{
		if($var ne 'Up, Active')
		{
			$subject = "Subject: Pri Outage $dest\n\n";
			$body = localtime(time) . " $dest (SPAN: $dests{$dest}{SPAN}) = $var\n\n";
		}
	}

	if($subject ne '' && $dests{$dest}{ALERT} & 2)
	{
  		open(SENDMAIL, "|/usr/lib/sendmail -oi -t -odq") or die "Can't fork for sendmail: $!\n";
		print SENDMAIL "From: Dozer Pri checker <prichecker\@allamericanasphalt.com>\n";
		foreach my $EMAIL (@EMAIL)
		{
			print SENDMAIL "To: $EMAIL\n";
		}
		print SENDMAIL $subject;
		print SENDMAIL $body;
		print SENDMAIL $telco_contact_info;
		print SENDMAIL "\ntelco circuit ID: $dests{$dest}{CIRCUIT}";
		close(SENDMAIL) or warn "sendmail didn't close nicely";
	}


	push(@{$persist->{$dest}->{'StatusArray'}}, $var);

	my $arraylength = @{$persist->{$dest}->{'StatusArray'}};

	while($arraylength > $dests{$dest}{ALERTINTERVAL})
	{
		shift(@{$persist->{$dest}->{'StatusArray'}});
		$arraylength = @{$persist->{$dest}->{'StatusArray'}};
	}

	if($dests{$dest}{ALERT} & 1)
	{
		my $last_email = $persist->{$dest}->{LastEmail};

		if(!defined($last_email))
		{
			$last_email = 0;
		}
	
		my $cutoff_time = $dests{$dest}{'ALERTINTERVAL'} * 60 + $last_email;
	
		if(time() >= $cutoff_time)
		{
			# see if it's time to send an alert email...
			my $non_up_count = 0;
			foreach my $status (@{$persist->{$dest}->{'StatusArray'}}) 
			{
				if($status ne 'Up, Active' && $status ne 'testing')
				{
					$non_up_count++;
				}
			}
	
			if($non_up_count >= $dests{$dest}{ALERTTHRESHOLD})
			{
				# send email...
		  		open(SENDMAIL, "|/usr/lib/sendmail -oi -t -odq") or die "Can't fork for sendmail: $!\n";
				print SENDMAIL "From: Dozer Pri checker <prichecker\@allamericanasphalt.com>\n";
				foreach my $EMAIL (@ALERT_EMAIL)
				{
					print SENDMAIL "To: $EMAIL\n";
				}
				print SENDMAIL "Subject: Quantum Pri Outage $dest\n\n";
				print SENDMAIL localtime(time) . "\n$dest span $dests{$dest}{SPAN} has been down >= $dests{$dest}{ALERTTHRESHOLD} times in the last $dests{$dest}{ALERTINTERVAL}\n\n" . Dumper(\$persist->{$dest});
				print SENDMAIL $telco_contact_info;
				print SENDMAIL "\ntelco circuit ID: $dests{$dest}{CIRCUIT}";
				close(SENDMAIL) or warn "sendmail didn't close nicely";
				
				$persist->{$dest}->{'LastEmail'} = time();


				if($CALLALERT == 1)
				{
 					# create a call file...
					foreach my $sysop (@SYSOP_PHONE)
					{
						my $tmp_call_file = "$CALLFILE_TEMP_DIR/$dest-$sysop";
						$tmp_call_file =~ s/ /_/g;
						
						open(TMPFILE, ">", $tmp_call_file) or warn "couldn't open temp call file: $!";
						
						print TMPFILE "Channel: Local/$sysop\@mis-phone/n\n";
						print TMPFILE "MaxRetries: 50\n";
						print TMPFILE "RetryTime: 5\n";
						print TMPFILE "WaitTime: 5\n";
						print TMPFILE "Archive: yes\n";
						print TMPFILE "Extension: Dozer Pri monitor failure for $dest\n";
						print TMPFILE "Context: outbound-swift\n";
		
						close(TMPFILE);

						`mv $tmp_call_file $ASTERISK_OUTGOING_DIR`;
					}
				}
			}
		}
	}
}

# serialize this structure to disk...
pl2xml($persist, $PERSIST_FILE);

exit;

Re: How to do checks on PRI lines

Posted: Mon Jul 31, 2017 10:59 am
by lmiltchev
The first step is to test the script from the command line. If it works, you can create a new command and service check in XI by following the steps, outlined in the document below:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Let us know if you have issues with following the document or if you get stuck on something.

Re: How to do checks on PRI lines

Posted: Mon Jul 31, 2017 11:38 am
by eloyd
We run a VoIP shop and I'd be happy to help you offline if you can't figure it out. Is this a Perl script of your own or something you grabbed online?

Re: How to do checks on PRI lines

Posted: Mon Jul 31, 2017 1:09 pm
by donnyforbes
Hi This script does work. We had issue the other day with our PRI line and I was getting emails / notification.. so what do you suggest I do to get it to work with Nagios?

Re: How to do checks on PRI lines

Posted: Mon Jul 31, 2017 1:17 pm
by eloyd
Honestly, my one suggestion would be to install Nagios Log Server, send your Asterisk logs over to it, and use NLS to search for your errors and then send back alerts through NRDP to Nagios. SO much easier, SO much less management, SO fewer passwords/secrets, and NLS is going to be free for monitoring such little data, actually. You can also use it to monitor your Nagios log file and look for errors there, as well.

If that's not an option, then you might want to do something similar, but in reverse. Simply have a cron job on your Asterisk box that does the pri show span command and dump the output to a file. Then have Nagios look at the file for the appropriate strings and take whatever alerting action is required.

The way you're doing it now, you're essentially just using Nagios as a cron to execute your service check, but all of the smarts as to what to look for, how to process it, what action to take, and whom to notify are in your script, not Nagios.

Re: How to do checks on PRI lines

Posted: Mon Jul 31, 2017 1:58 pm
by tacolover101
donnyforbes wrote:Hi This script does work. We had issue the other day with our PRI line and I was getting emails / notification.. so what do you suggest I do to get it to work with Nagios?
all of your logic to determine if it should alert is self contained, so as @eloyd pointed out, you're simply using Nagios to execute. you would need to re-write this, to hand off logic to nagios to handle. this would require outputs and exit codes.

Re: How to do checks on PRI lines

Posted: Mon Jul 31, 2017 2:44 pm
by tgriep
To make that script work with Nagios XI, it would have to be re-written to output the correct information that Nagios uses to generate alerts, etc...
Take a look at the plugin guideline for some examples and what the plugin needs to output.
https://nagios-plugins.org/doc/guidelines.html

Re: How to do checks on PRI lines

Posted: Mon Jul 31, 2017 6:12 pm
by donnyforbes
eloyd wrote:We run a VoIP shop and I'd be happy to help you offline if you can't figure it out. Is this a Perl script of your own or something you grabbed online?
This perl script is something one of our developers wrote before my time at the company.

Re: How to do checks on PRI lines

Posted: Mon Jul 31, 2017 6:16 pm
by donnyforbes
Honestly, my one suggestion would be to install Nagios Log Server, send your Asterisk logs over to it, and use NLS to search for your errors and then send back alerts through NRDP to Nagios. SO much easier, SO much less management, SO fewer passwords/secrets, and NLS is going to be free for monitoring such little data, actually. You can also use it to monitor your Nagios log file and look for errors there, as well.
How would I go about doing this? This seems like something easier to do...

Can you give me information on getting this started? Do I install the Nagios Log Server on my Nagios server or a different one?

Re: How to do checks on PRI lines

Posted: Tue Aug 01, 2017 11:49 am
by donnyforbes
@eloyd I am in the process of getting a vm fired up with RHEL 7.3 and setting up the log server. Once I get the log server setup. I may need some assistance with this
send your Asterisk logs over to it, and use NLS to search for your errors and then send back alerts through NRDP to Nagios
Please maybe you can advise on this.

Thanks @eloyd