Weird notification issue!

[BanditBBS]

Ok, I've got a weird one here that I can not figure out!

I have two services on the same host:

Code: Select all

define service {
        host_name                       extn-chi-pdb02
        service_description             PRODEBS - Database Alert Log Errors
        use                             extn_generic-service-5
        servicegroups                   extn_prodebs
        check_command                   check_by_ssh_dblog!/us1001/app/oracle/diag/rdbms/prodebs/PRODEBS2/trace/alert_PRODEBS2.log!ORA!-c 1!!!!!
        max_check_attempts              1
        check_period                    xi_timeperiod_24x7
        flap_detection_enabled          0
        notification_period             xi_timeperiod_24x7
        register                        1
        }
		
define service {
        host_name                       extn-chi-pdb02
        service_description             PRODEBS - WF Deferred Queue
        use                             extn_generic-service-5
        servicegroups                   extn_prodebs
        check_command                   check_xi_oraclequery2!1522!PRODEBS!xxxxxxx!xxxxxxx!"Select count(*) from apps.wf_deferred where state=0"!"WF Deferred Queue"!--warning=100 --critical=125!
        check_period                    xi_timeperiod_24x7
        notification_period             xi_timeperiod_24x7
        _xiwizard                       oraclequery
        register                        1
        }

When a notification goes out for the top one it is only sent to the escalations and not the inherited contacts from the host. When a notification is sent for the bottom one it goes to the escalations and the inherited contacts from the host. WTF am I missing here?


EDIT #1: Now this is even weirder. I have another host with the identical setup(and I mean identical) and the service is the same as well, and it alerts to everyone it should. So now I am even more confused

[cmerchant]

You haven't shown us your service escalation, or the extn_generic-service-5 service template. In your first service definition you set the max_check_attempts to 1. Which if you don't escalate until say the third check with the changed state, you won't ever reach escalation.

[BanditBBS]

As I stated, the escalations are working perfectly, but here it is regardless:

Code: Select all

define serviceescalation {
        hostgroup_name                          EXTN-Prod-Servers
        service_description                     *
        contacts                                matmakuru,pushwinder,sasaf
        first_notification                      1
        last_notification                       1
        notification_interval                   15
        escalation_period                       24x7
        escalation_options                      w,u,c,r,
        }

The inherited ones from the HOST are the ones not working for every service, just some. The services I listed above, one works and one doesn't.
Here is the service template:

Code: Select all

define service {
       name                                     extn_generic-service-5
       service_description                      Generic Exterran Service(5 Min)
       is_volatile                              0
       max_check_attempts                       3
       check_interval                           5
       retry_interval                           2
       active_checks_enabled                    1
       passive_checks_enabled                   1
       check_period                             24x7
       parallelize_check                        1
       obsess_over_service                      1
       check_freshness                          0
       event_handler_enabled                    1
       flap_detection_enabled                   1
       process_perf_data                        1
       retain_status_information                1
       retain_nonstatus_information             1
       notification_interval                    0
       notification_period                      24x7
       notification_options                     w,c,u,r,
       notifications_enabled                    1
       register                                 0

}

And here is the host:

Code: Select all

define host {
        host_name                       extn-chi-pdb02
        use                             extn_generic_host
        alias                           extn-chi-pdb02(PRODEBS2)
        address                         10.xx.xx.xx
        contacts                        +extn_nagios_db
        contact_groups                  +EXTN_NAGIOS_ALL_CG
        register                        1
        }

and before you ask:

Code: Select all

define host {
       name                                     extn_generic_host
       check_command                            check_xi_host_ping!3000.0!80%!5000.0!100%!!!!
       max_check_attempts                       2
       check_interval                           5
       retry_interval                           1
       check_period                             xi_timeperiod_24x7
       event_handler_enabled                    1
       flap_detection_enabled                   1
       process_perf_data                        1
       retain_status_information                1
       retain_nonstatus_information             1
       contact_groups                           +EXTN_NAGIOS_ALL_CG
       notification_interval                    0
       notification_period                      xi_timeperiod_24x7
       notifications_enabled                    1
       register                                 0

}

[BanditBBS]

I'm beginning to wonder if the escalations are messing with the inheritence from host to service....but if so, why isn't it everywhere for every service. Very confusing.

[scottwilkerson]

Once an escalation is triggered, original contacts are not notified any longer unless they are part of the escalation

[BanditBBS]

Once an escalation is triggered, original contacts are not notified any longer unless they are part of the escalation

Scott(and everyone),

Please trust me that these two hosts are setup identical.
This is from this problem host when the service fails:

Capture2.PNG

This is from a working host when it fails. Setup identical(I PROMISE )

Capture.PNG

[scottwilkerson]

"Setup identical" including, the same contacts on all of the hosts, services, escalations ? Are all of the contacts set the same, same time periods, etc?

[BanditBBS]

"Setup identical" including, the same contacts on all of the hosts, services, escalations ? Are all of the contacts set the same, same time periods, etc?

Yes sir. The only difference is the IP on the host, the hostname and the slight difference in service description. Besides that, everything is identical. I'll go ahead and show, maybe I am missing something so I will edit this post in a bit and paste comparisons of the 2 services that are identical yet behave differently.

Hosts:

Code: Select all

define host {
        host_name                       extn-chi-pdb02
        use                             extn_generic_host
        alias                           extn-chi-pdb02(PRODEBS2)
        address                         10.160.101.17
        contacts                        +extn_nagios_db
        contact_groups                  +EXTN_NAGIOS_ALL_CG
        register                        1
        }
define host {
        host_name                       extn-chi-pdb05
        use                             extn_generic_host
        alias                           extn-chi-pdb05(ASCP)
        address                         10.160.101.20
        contacts                        +extn_nagios_db
        contact_groups                  +EXTN_NAGIOS_ALL_CG
        register                        1
        }

Services:

Code: Select all

define service {
        host_name                       extn-chi-pdb02
        service_description             PRODEBS - Database Alert Log Errors
        use                             extn_generic-service-5
        servicegroups                   extn_prodebs
        check_command                   check_by_ssh_dblog!/us1001/app/oracle/diag/rdbms/prodebs/PRODEBS2/trace/alert_PRODEBS2.log!ORA!-c 1!!!!!
        max_check_attempts              1
        check_period                    xi_timeperiod_24x7
        flap_detection_enabled          0
        notification_period             xi_timeperiod_24x7
        register                        1
        }

define service {
        host_name                       extn-chi-pdb05
        service_description             PRODASCP - Database Alert Log Errors
        use                             extn_generic-service-5
        servicegroups                   extn_prodascp
        check_command                   check_by_ssh_dblog!/us1001/oracle/db/tech_st/11.1.0/admin/PRODASCP_extn-chi-pdb05/diag/rdbms/prodascp/PRODASCP/trace/alert_PRODASCP.log!ORA!-c 1!!!!!
        max_check_attempts              1
        check_period                    xi_timeperiod_24x7
        flap_detection_enabled          0
        notification_period             xi_timeperiod_24x7
        register                        1
        }

The templates and other stuff have already been pasted earlier in thread. Also the images in my previous post show the difference in notifications sent.

[Box293]

Can you run these two commands and post the output:

Code: Select all

cat /usr/local/nagios/var/objects.cache | sed -rn "/define host \{/{:a;N;/}/{/.host_name.extn-chi-pdb02/p;d};ba}"
cat /usr/local/nagios/var/objects.cache | sed -rn "/define host \{/{:a;N;/}/{/.host_name.extn-chi-pdb05/p;d};ba}"

Can you run these two commands and post the output for the "PRODEBS - Database Alert Log Errors" and "PRODASCP - Database Alert Log Errors" services:

Code: Select all

cat /usr/local/nagios/var/objects.cache | sed -rn "/define service \{/{:a;N;/}/{/.host_name.extn-chi-pdb02/p;d};ba}"
cat /usr/local/nagios/var/objects.cache | sed -rn "/define service \{/{:a;N;/}/{/.host_name.extn-chi-pdb05/p;d};ba}"

[BanditBBS]

Here you go sir:

First two commands:

Code: Select all

[jclark@iss-chi-nag05 ~]$ cat /usr/local/nagios/var/objects.cache | sed -rn "/define host \{/{:a;N;/}/{/.host_name.extn-chi-pdb02/p;d};ba}"
define host {
        host_name       extn-chi-pdb02
        alias   extn-chi-pdb02(PRODEBS2)
        address 10.160.101.17
        check_period    xi_timeperiod_24x7
        check_command   check_xi_host_ping!3000.0!80%!5000.0!100%!!!!
        contact_groups  EXTN_NAGIOS_ALL_CG
        notification_period     xi_timeperiod_24x7
        initial_state   o
        importance      0
        check_interval  5.000000
        retry_interval  1.000000
        max_check_attempts      2
        active_checks_enabled   1
        passive_checks_enabled  1
        obsess  1
        event_handler_enabled   1
        low_flap_threshold      0.000000
        high_flap_threshold     0.000000
        flap_detection_enabled  1
        flap_detection_options  a
        freshness_threshold     0
        check_freshness 0
        notification_options    a
        notifications_enabled   1
        notification_interval   0.000000
        first_notification_delay        0.000000
        stalking_options        n
        process_perf_data       1
        retain_status_information       1
        retain_nonstatus_information    1
        }
[jclark@iss-chi-nag05 ~]$ cat /usr/local/nagios/var/objects.cache | sed -rn "/define host \{/{:a;N;/}/{/.host_name.extn-chi-pdb05/p;d};ba}"
define host {
        host_name       extn-chi-pdb05
        alias   extn-chi-pdb05(ASCP)
        address 10.160.101.20
        check_period    xi_timeperiod_24x7
        check_command   check_xi_host_ping!3000.0!80%!5000.0!100%!!!!
        contact_groups  EXTN_NAGIOS_ALL_CG
        notification_period     xi_timeperiod_24x7
        initial_state   o
        importance      0
        check_interval  5.000000
        retry_interval  1.000000
        max_check_attempts      2
        active_checks_enabled   1
        passive_checks_enabled  1
        obsess  1
        event_handler_enabled   1
        low_flap_threshold      0.000000
        high_flap_threshold     0.000000
        flap_detection_enabled  1
        flap_detection_options  a
        freshness_threshold     0
        check_freshness 0
        notification_options    a
        notifications_enabled   1
        notification_interval   0.000000
        first_notification_delay        0.000000
        stalking_options        n
        process_perf_data       1
        retain_status_information       1
        retain_nonstatus_information    1
        }

Second two:
Umm, those commands don't seem to show all the services that they should be and neither show the one we are looking for. Also, why do the host ones above only list the contact groups in that output and not the contacts? And the services that do show show "contact_groups admins" and that's no where on them.

EDIT: I'm an idiot. My stuff is offloaded. I'll redo the commands within the hour and repost!