How to do checks on PRI lines

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

How to do checks on PRI lines

Post by donnyforbes »

Right now we use a system called dozer and we are in the process of migrating over to "Nagios". We do checks for PRI lines that we have here at our location and need to know when they go down and come backup. I have placed the script in here now, however I need to know how I can implement this into Nagios. Please advise and help.

Code: Select all

#!/usr/bin/perl
use warnings;
use strict;

use Asterisk::AMI;
use Data::Dumper;
use XML::Dumper;

my @ALERT_EMAIL = ('[email protected]');
my @EMAIL = ('[email protected]'); my @SYSOP_PHONE = ('8300326', # justin work cell
                   '4535893', # Mike work cell
                   '7578034', # Jim work cell
                );

#my @ALERT_EMAIL = ('[email protected]');
#my @EMAIL = ('[email protected]');
#my @SYSOP_PHONE = ('8300326');

my $PERSIST_FILE = '/var/check_pri_persist.xml';

my $telco_contact_info = "Telepacific Premium Support: 888-611-8722, Acct# 134607"; my $aggplant_info = "Agg Plant: (951) 736-7680, backup: (951) 736-7661";

my $CALLFILE_TEMP_DIR = '/var/tmp';
my $ASTERISK_OUTGOING_DIR = '/var/spool/asterisk/outgoing';

my $CALLALERT = 0;

# ALERT:
# 1 = ten minute alert to ALERT_EMAIL
# 2 = one minute alert to EMAIL
# 3 = 1 and 2

# VARIABLES YOU SHOULD EDIT.
my %dests = (
	'Hq Span 7' => 			{ ALERT => 3, SERVER => 'tank',      USER => 'pri_checker', SECRET => 'Wjm8A4C2n2379kq3', SPAN => 7, ALERTINTERVAL => 10, ALERTTHRESHOLD => 5, CIRCUIT => '842641', PHONE => '', FAX => '' },
	'Hq Span 8' => 			{ ALERT => 3, SERVER => 'tank',      USER => 'pri_checker', SECRET => 'Wjm8A4C2n2379kq3', SPAN => 8, ALERTINTERVAL => 10, ALERTTHRESHOLD => 5, CIRCUIT => '842635', PHONE => '', FAX => '' },
	'Yard Span 7' => 		{ ALERT => 3, SERVER => 'ernestine', USER => 'pri_checker', SECRET => 'Wjm8A4C2n2379kq3', SPAN => 7, ALERTINTERVAL => 10, ALERTTHRESHOLD => 5, CIRCUIT => '13/HCGS/848840//PT', PHONE => '', FAX => '' },
	'Yard Span 8' => 		{ ALERT => 3, SERVER => 'ernestine', USER => 'pri_checker', SECRET => 'Wjm8A4C2n2379kq3', SPAN => 8, ALERTINTERVAL => 10, ALERTTHRESHOLD => 5, CIRCUIT => '537994003.1 ????', PHONE => '', FAX => '' },
	);

my $persist = ();

if(-e $PERSIST_FILE)
{
	$persist = xml2pl($PERSIST_FILE);
}

foreach((keys %dests))
{
	my $dest = $_;

	my $astman = Asterisk::AMI->new(PeerAddr => $dests{$dest}{SERVER},
					Username => $dests{$dest}{USER},
					Secret   => $dests{$dest}{SECRET},
					Events   => 'off',
			);

	my $response = $astman->action({ Action => 'Command',
					 Command => 'pri show span ' . $dests{$dest}{SPAN},
					});
	
#	print Dumper(\$response);

	my $subject = '';
	my $body = '';

	my $var = undef;

	my @cmdnode = $response->{CMD};

	foreach my $cmds (@cmdnode)
	{
		foreach my $cmd (@{$cmds})
		{
#			print "cmd: $cmd\n";

			if($cmd =~ /Status: (.*)/) {
				$var = $1;
			}
		}
	}

#	print "dest: $dest, status: $var\n";

	unless(defined $var) {
		$var = 'Error: Unable to parse response';
		$subject = "Subject: Script Error - Pri Outage $dest\n\n";
		$body = localtime(time) .  " Unable to parse response.\nLooking for 'Status:' in: \n\n" . Dumper(@cmdnode);
	}
	else
	{
		if($var ne 'Up, Active')
		{
			$subject = "Subject: Pri Outage $dest\n\n";
			$body = localtime(time) . " $dest (SPAN: $dests{$dest}{SPAN}) = $var\n\n";
		}
	}

	if($subject ne '' && $dests{$dest}{ALERT} & 2)
	{
  		open(SENDMAIL, "|/usr/lib/sendmail -oi -t -odq") or die "Can't fork for sendmail: $!\n";
		print SENDMAIL "From: Dozer Pri checker <prichecker\@allamericanasphalt.com>\n";
		foreach my $EMAIL (@EMAIL)
		{
			print SENDMAIL "To: $EMAIL\n";
		}
		print SENDMAIL $subject;
		print SENDMAIL $body;
		print SENDMAIL $telco_contact_info;
		print SENDMAIL "\ntelco circuit ID: $dests{$dest}{CIRCUIT}";
		close(SENDMAIL) or warn "sendmail didn't close nicely";
	}


	push(@{$persist->{$dest}->{'StatusArray'}}, $var);

	my $arraylength = @{$persist->{$dest}->{'StatusArray'}};

	while($arraylength > $dests{$dest}{ALERTINTERVAL})
	{
		shift(@{$persist->{$dest}->{'StatusArray'}});
		$arraylength = @{$persist->{$dest}->{'StatusArray'}};
	}

	if($dests{$dest}{ALERT} & 1)
	{
		my $last_email = $persist->{$dest}->{LastEmail};

		if(!defined($last_email))
		{
			$last_email = 0;
		}
	
		my $cutoff_time = $dests{$dest}{'ALERTINTERVAL'} * 60 + $last_email;
	
		if(time() >= $cutoff_time)
		{
			# see if it's time to send an alert email...
			my $non_up_count = 0;
			foreach my $status (@{$persist->{$dest}->{'StatusArray'}}) 
			{
				if($status ne 'Up, Active' && $status ne 'testing')
				{
					$non_up_count++;
				}
			}
	
			if($non_up_count >= $dests{$dest}{ALERTTHRESHOLD})
			{
				# send email...
		  		open(SENDMAIL, "|/usr/lib/sendmail -oi -t -odq") or die "Can't fork for sendmail: $!\n";
				print SENDMAIL "From: Dozer Pri checker <prichecker\@allamericanasphalt.com>\n";
				foreach my $EMAIL (@ALERT_EMAIL)
				{
					print SENDMAIL "To: $EMAIL\n";
				}
				print SENDMAIL "Subject: Quantum Pri Outage $dest\n\n";
				print SENDMAIL localtime(time) . "\n$dest span $dests{$dest}{SPAN} has been down >= $dests{$dest}{ALERTTHRESHOLD} times in the last $dests{$dest}{ALERTINTERVAL}\n\n" . Dumper(\$persist->{$dest});
				print SENDMAIL $telco_contact_info;
				print SENDMAIL "\ntelco circuit ID: $dests{$dest}{CIRCUIT}";
				close(SENDMAIL) or warn "sendmail didn't close nicely";
				
				$persist->{$dest}->{'LastEmail'} = time();


				if($CALLALERT == 1)
				{
 					# create a call file...
					foreach my $sysop (@SYSOP_PHONE)
					{
						my $tmp_call_file = "$CALLFILE_TEMP_DIR/$dest-$sysop";
						$tmp_call_file =~ s/ /_/g;
						
						open(TMPFILE, ">", $tmp_call_file) or warn "couldn't open temp call file: $!";
						
						print TMPFILE "Channel: Local/$sysop\@mis-phone/n\n";
						print TMPFILE "MaxRetries: 50\n";
						print TMPFILE "RetryTime: 5\n";
						print TMPFILE "WaitTime: 5\n";
						print TMPFILE "Archive: yes\n";
						print TMPFILE "Extension: Dozer Pri monitor failure for $dest\n";
						print TMPFILE "Context: outbound-swift\n";
		
						close(TMPFILE);

						`mv $tmp_call_file $ASTERISK_OUTGOING_DIR`;
					}
				}
			}
		}
	}
}

# serialize this structure to disk...
pl2xml($persist, $PERSIST_FILE);

exit;
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: How to do checks on PRI lines

Post by lmiltchev »

The first step is to test the script from the command line. If it works, you can create a new command and service check in XI by following the steps, outlined in the document below:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Let us know if you have issues with following the document or if you get stuck on something.
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
eloyd
Cool Title Here
Posts: 2190
Joined: Thu Sep 27, 2012 9:14 am
Location: Rochester, NY
Contact:

Re: How to do checks on PRI lines

Post by eloyd »

We run a VoIP shop and I'd be happy to help you offline if you can't figure it out. Is this a Perl script of your own or something you grabbed online?
Image
Eric Loyd • http://everwatch.global • 844.240.EVER • @EricLoyd
I'm a Nagios Fanatic! • Join our public Nagios Discord Server!
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: How to do checks on PRI lines

Post by donnyforbes »

Hi This script does work. We had issue the other day with our PRI line and I was getting emails / notification.. so what do you suggest I do to get it to work with Nagios?
User avatar
eloyd
Cool Title Here
Posts: 2190
Joined: Thu Sep 27, 2012 9:14 am
Location: Rochester, NY
Contact:

Re: How to do checks on PRI lines

Post by eloyd »

Honestly, my one suggestion would be to install Nagios Log Server, send your Asterisk logs over to it, and use NLS to search for your errors and then send back alerts through NRDP to Nagios. SO much easier, SO much less management, SO fewer passwords/secrets, and NLS is going to be free for monitoring such little data, actually. You can also use it to monitor your Nagios log file and look for errors there, as well.

If that's not an option, then you might want to do something similar, but in reverse. Simply have a cron job on your Asterisk box that does the pri show span command and dump the output to a file. Then have Nagios look at the file for the appropriate strings and take whatever alerting action is required.

The way you're doing it now, you're essentially just using Nagios as a cron to execute your service check, but all of the smarts as to what to look for, how to process it, what action to take, and whom to notify are in your script, not Nagios.
Last edited by eloyd on Mon Jul 31, 2017 2:48 pm, edited 1 time in total.
Image
Eric Loyd • http://everwatch.global • 844.240.EVER • @EricLoyd
I'm a Nagios Fanatic! • Join our public Nagios Discord Server!
User avatar
tacolover101
Posts: 432
Joined: Mon Apr 10, 2017 11:55 am

Re: How to do checks on PRI lines

Post by tacolover101 »

donnyforbes wrote:Hi This script does work. We had issue the other day with our PRI line and I was getting emails / notification.. so what do you suggest I do to get it to work with Nagios?
all of your logic to determine if it should alert is self contained, so as @eloyd pointed out, you're simply using Nagios to execute. you would need to re-write this, to hand off logic to nagios to handle. this would require outputs and exit codes.
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: How to do checks on PRI lines

Post by tgriep »

To make that script work with Nagios XI, it would have to be re-written to output the correct information that Nagios uses to generate alerts, etc...
Take a look at the plugin guideline for some examples and what the plugin needs to output.
https://nagios-plugins.org/doc/guidelines.html
Be sure to check out our Knowledgebase for helpful articles and solutions!
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: How to do checks on PRI lines

Post by donnyforbes »

eloyd wrote:We run a VoIP shop and I'd be happy to help you offline if you can't figure it out. Is this a Perl script of your own or something you grabbed online?
This perl script is something one of our developers wrote before my time at the company.
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: How to do checks on PRI lines

Post by donnyforbes »

Honestly, my one suggestion would be to install Nagios Log Server, send your Asterisk logs over to it, and use NLS to search for your errors and then send back alerts through NRDP to Nagios. SO much easier, SO much less management, SO fewer passwords/secrets, and NLS is going to be free for monitoring such little data, actually. You can also use it to monitor your Nagios log file and look for errors there, as well.
How would I go about doing this? This seems like something easier to do...

Can you give me information on getting this started? Do I install the Nagios Log Server on my Nagios server or a different one?
donnyforbes
Posts: 357
Joined: Tue Jun 13, 2017 2:17 pm

Re: How to do checks on PRI lines

Post by donnyforbes »

@eloyd I am in the process of getting a vm fired up with RHEL 7.3 and setting up the log server. Once I get the log server setup. I may need some assistance with this
send your Asterisk logs over to it, and use NLS to search for your errors and then send back alerts through NRDP to Nagios
Please maybe you can advise on this.

Thanks @eloyd
Locked