RESOLVED: Custom Alert Not firing

[story]

Ive written a custom alert script that fires to a webhook style url. Currently i cannot get the alert to even attempt to execute.

when i enable logging in nagios.cfg i can see additional email based alerts going, but this custom one is never even attempted. Im not sure what im missing..

This is based on the following code --> https://github.com/mrmikee/nagios-notify-msteams

I have checked the commands below using the nagios user and they hit the url. I have also enabled email notifications on the same service and host and the email alerts work as expected. This custom alert is never attempted..

Nagios.log

Code: Select all

[1775488618] SERVICE NOTIFICATION: nagiosadmin;PB1;netbackup;CRITICAL;xi_service_notification_handler;CRITICAL: netbackup is stopped (should be running)
[1775488618] SERVICE ALERT: PB1;netbackup;CRITICAL;HARD;5;CRITICAL: netbackup is stopped (should be running)
[1775488912] SERVICE NOTIFICATION: nagiosadmin;PB1;netbackup;CRITICAL;xi_service_notification_handler;CRITICAL: netbackup is stopped (should be running)
[1775488912] SERVICE ALERT: PB1;netbackup;CRITICAL;HARD;5;CRITICAL: netbackup is stopped (should be running)
[1775489210] SERVICE NOTIFICATION: nagiosadmin;PB1;netbackup;CRITICAL;xi_service_notification_handler;CRITICAL: netbackup is stopped (should be running)
[1775489210] SERVICE ALERT: PB1;netbackup;CRITICAL;HARD;5;CRITICAL: netbackup is stopped (should be running)
[1775489508] SERVICE NOTIFICATION: nagiosadmin;PB1;netbackup;CRITICAL;xi_service_notification_handler;CRITICAL: netbackup is stopped (should be running)
[1775489508] SERVICE ALERT: PB1;netbackup;CRITICAL;HARD;5;CRITICAL: netbackup is stopped (should be running)
[1775489807] SERVICE NOTIFICATION: nagiosadmin;PB1;netbackup;CRITICAL;xi_service_notification_handler;CRITICAL: netbackup is stopped (should be running)
[1775489807] SERVICE ALERT: PB1;netbackup;CRITICAL;HARD;5;CRITICAL: netbackup is stopped (should be running)
[1775490106] SERVICE NOTIFICATION: nagiosadmin;PB1;netbackup;CRITICAL;xi_service_notification_handler;CRITICAL: netbackup is stopped (should be running)
[1775490106] SERVICE ALERT: PB1;netbackup;CRITICAL;HARD;5;CRITICAL: netbackup is stopped (should be running)

Code: Select all

define service {
    host_name                   PB1
    service_description         netbackup
    use                         xiwizard_ncpa_service
    check_command               check_xi_ncpa!-t 0REDACTEDa -P 5693  -M 'services' -q 'service=netbackup,status=running'!!!!!!!
    max_check_attempts          5
    check_interval              5
    retry_interval              1
    check_period                xi_timeperiod_24x7
    flap_detection_enabled      0
    notification_interval       1
    first_notification_delay    1
    notification_period         xi_timeperiod_24x7
    notification_options        w,c,u,r,f,s,
    notifications_enabled       1
    contacts                    IOC-OUA,nagiosadmin
    contact_groups              IOC-CONTACT-GROUP
    stalking_options            o,w,c,u,N,
    _xiwizard                   ncpa
    register                    1
}

Code: Select all

define contact {
    contact_name                     IOC-OUA
    alias                            IOC OUA User
    contactgroups                    *
    host_notifications_enabled       1
    service_notifications_enabled    1
    host_notification_period         24x7
    service_notification_period      24x7
    host_notification_options        d,u,r,f,s,n,
    service_notification_options     w,u,c,r,f,s,n,
    host_notification_commands       Soco-Notify-Ioc-Host
    service_notification_commands    Soco-Notify-Ioc-Service
    retain_status_information        1
    retain_nonstatus_information     1
    email                            [email protected]

Code: Select all

define command {
    command_name    Soco-Notify-Ioc-Host
    command_line    /home/nagios/scripts/to-isa-nagios/notify-nagios-ioc.py "HOST"
}

define command {
    command_name    Soco-Notify-Ioc-Service
    command_line    /home/nagios/scripts/to-isa-nagios/notify-nagios-ioc.py "SERVICE"
}

[cdietsch]

Hey @story,

In your command definitions I think you need to need to change the "HOST" and "SERVICE" quoted strings to be environment variables sent from Nagios.

So $HOSTNAME$ instead of "HOST" and $SERVICEDESC$ instead of "SERVICE".

Here is a full list of the Nagios Macros in case you need other information from Nagios:
https://assets.nagios.com/downloads/nag ... olist.html

I found a similar post to this one where the issue turned out to be related to permissions, so it might be worth looking at too if this doesn't completely solve the problem:
viewtopic.php?t=48234

Good Luck!

[DoubleDoubleA]

Hi @story,

Thanks for your post.

In your case, I would use debug logging and verbosity on Nagios, and I would also leverage the debug on the custom script.

See https://assets.nagios.com/downloads/nag ... gmain.html and ctl-f for "debug".

set debug_level to 289 (or some other value, feel free to experiment) and debug_verbosity to 2.

Restart the nagios process.

Don't forget to change your command to leverage your script's debugging as well.

[story]

Hey @story,

In your command definitions I think you need to need to change the "HOST" and "SERVICE" quoted strings to be environment variables sent from Nagios.

So $HOSTNAME$ instead of "HOST" and $SERVICEDESC$ instead of "SERVICE".

Here is a full list of the Nagios Macros in case you need other information from Nagios:
https://assets.nagios.com/downloads/nag ... olist.html

I found a similar post to this one where the issue turned out to be related to permissions, so it might be worth looking at too if this doesn't completely solve the problem:
viewtopic.php?t=48234

Good Luck!

the HOST and SERVICE are keywords to indicate to the code to use the host logic or the service logic, they are not the actual hostname or the service name. I am also already using the macro env vars in this script.

enable_environment_macros=1 is also set

[story]

Hi @story,

Thanks for your post.

In your case, I would use debug logging and verbosity on Nagios, and I would also leverage the debug on the custom script.

See https://assets.nagios.com/downloads/nag ... gmain.html and ctl-f for "debug".

set debug_level to 289 (or some other value, feel free to experiment) and debug_verbosity to 2.

Restart the nagios process.

Don't forget to change your command to leverage your script's debugging as well.

debug value in nagios.cfg is set to:
debug_level=4095
debug_verbosity=2

the script debugging is showing nothing its not being called at all. The command file entries dont show the debug values because i wanted to show a normal config.

[story]

Here is the config with the debug info included.. nothing is written to /tmp

Code: Select all

define command {
    command_name    Soco-Notify-Ioc-Host
    command_line    /home/nagios/scripts/to-isa-nagios/notify-nagios-ioc.py HOST --debug >> /tmp/nag-teams-debug.log 2>&1
}

define command {
    command_name    Soco-Notify-Ioc-Service
    command_line    /home/nagios/scripts/to-isa-nagios/notify-nagios-ioc.py SERVICE --debug >> /tmp/nag-teams-debug.log 2>&1
}

[story]

Figured it out...

Changed the nagios.cfg debug settings to focus on only notifications:
debug_level=32
debug_verbosity=2

Then saw this..

Code: Select all

[1775524294.780015] [032.2] [pid=3936122] ** Checking service notification viability for contact 'IOC-OUA'...
[1775524294.780027] [032.2] [pid=3936122] We shouldn't notify this contact about CRITICAL service states.
[1775524294.780032] [032.2] [pid=3936122] Not adding contact 'IOC-OUA'

It didnt add the contact to the list but was told to alert on critical.

All the entries in the alert settings were selected:
host_notification_options d,u,r,f,s,n,
service_notification_options w,u,c,r,f,s,n,

Howerver, the N in the above config snippet, nullifies everything before it, it told it to not send the alert, un selected "None" and it worked..


REQUEST: clear the selections in NagiosXI webui if the NONE is selected... its easy to miss.

[DoubleDoubleA]

Hi @story,

Good to hear it is resolved. The 'n' is easy to miss!

As far as your request, I'll take a look at that.

Aaron