Page 1 of 2

DB Alert Not Generated

Posted: Wed Jul 02, 2014 1:54 am
by wiproltdwiv
Dear Team,
We are monitoring Tablespace usage for few DBs in Nagios, we have set 30 mins check interval, retry interval of 1 mins and 5 check attempts. notification period & check period is 24/7.

the issue is like, 1 tablespace usage of the DB has crossed the defined treshold at a particular time but we have not received any alert for it in a whole day. we used to get alerts for the same table earlier. but now its not.
please confirm what may be the issue. configuration part screenshots are attached.

Re: DB Alert Not Generated

Posted: Wed Jul 02, 2014 9:07 am
by tmcdonald
As shown in your first screenshot you have notifications disabled. Is there a reason that is off?

Re: DB Alert Not Generated

Posted: Thu Jul 03, 2014 12:03 am
by wiproltdwiv
Dear Team,
We are using external handler for host and services, we are not using any mail or sms notification.
hence the notifications are disabled.
please confirm what may be the issue for alert not generated even though respective DB table space usage was high.

Regards,

Re: DB Alert Not Generated

Posted: Thu Jul 03, 2014 9:05 am
by tmcdonald
wiproltdwiv wrote: We are using external handler for host and services, we are not using any mail or sms notification.
That would be my first guess as to where to look. What changes did you make to Nagios? What external handler are you using? If not email or SMS, what is used to send notifications?

Re: DB Alert Not Generated

Posted: Fri Jul 04, 2014 5:36 am
by wiproltdwiv
Hi,
we did some customization as per need, for that
1) we created script for service and host check i.e. alertinserhost.sh.
2) created 2 commands i.e check_event_host & service with below syntax
$USER1$/alertinserthost.sh $HOSTSTATE$ $HOSTSTATETYPE$ $HOSTATTEMPT$ $HOSTNAME$ "$HOSTGROUPNAMES$" $HOSTSTATEID$ $HOSTOUTPUT$
$USER1$/alertinsertservice.sh $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $HOSTNAME$ "$HOSTGROUPNAMES$" "$SERVICEDESC$" $SERVICESTATEID$ "$SERVICEOUTPUT$"
3) created database i.e alertintegration.

i have attached host script, please check based on it is working and we are getting notification.

Re: DB Alert Not Generated

Posted: Mon Jul 07, 2014 9:34 am
by tmcdonald
wiproltdwiv wrote:i have attached host script, please check based on it is working and we are getting notification.
So wait, are you getting alerts or not? Looking through your script there is nothing in there that directly sends an alert. From my understanding all it does is insert a new record into your alerttrack database. If the entries are properly being inserted into the database then it is not a problem with Nagios.

Re: DB Alert Not Generated

Posted: Tue Jul 08, 2014 1:39 am
by wiproltdwiv
this script inserting alert into alertrack db. but we did not get check result, so that my concern what can be issue for check not happened.

Re: DB Alert Not Generated

Posted: Tue Jul 08, 2014 10:04 am
by tmcdonald
Your checks are running check_oracle_health but you are saying you have issues with your custom alertinserthost.sh script. Where are you running alertinserthost.sh in this scenario?

Re: DB Alert Not Generated

Posted: Wed Jul 09, 2014 11:53 pm
by wiproltdwiv
my concern is when any check happen that status should be inserted in nagios db. after that only our script will start work. so may be check not happened and cause of script could not found any status in nagios db. so is it any possibility to happen like this incident.

Re: DB Alert Not Generated

Posted: Thu Jul 10, 2014 4:35 pm
by sreinhardt
What actually calls your alert script to run? It seems like you intend it to be a event handler, but this service has none defined, not that it couldn't be in a template. At this point it sounds like the flow of events would be:

service check returns NOT ok.
calls event handler assigned to service, to insert into alertdb
nagios returns to normal work and continues checking services.

Does that sound correct? We really need an outline of how you expect this to work, what is happening presently, and what is not happening.