Automatic Notifications not working, only broadcast works

[ramuvr]

Hi,

I have setup the latest version of Nagios Core on Ubuntu 14.04 and postfix for notifications.

My Nagios is working fine, but the notifications are not being sent automatically. It is shown as enabled but does not sent (not time related).
If I try and send a manual notification, I can send email only if I select the broadcast option.
I can use the CLI to send emails fine.

Any suggestions where the problem might be?

Regards,
Ramu

[tmcdonald]

My guess would be with the configs. Can you post the host, service, and associated contact configs for an example service that is not alerting?

[rhassing]

Maybe you could also share some log files (maillog)?

[ramuvr]

Looks like config, its only happening on linux (localhost.cfg) files. I added all Windows Severs to Windows.cfg and the automatic emails and custom ones go fine.
But when I send a custom one for linux it fails to get to the mail (nothing in the mail logs)

Localhost.cfg :

define host{
use linux-server
host_name abcd.com
alias abcd.com
address <ip.add.of.host>
}


contacts.cfg

define contact{
contact_name nagiosadmin
use generic-contact
alias Nagios Admin
email [email protected]
}

This is for host custom notification, so I am not copying service definition.

When I send a custom notification (or automatic one)

Nagios.log shows

[1415659852] EXTERNAL COMMAND: SEND_CUSTOM_HOST_NOTIFICATION;abcd.com;0;Nagios Admin;test custom notify by ramu for linux

But Mail.log shows nothing, so its apparently not reaching there (my interference!)

Many thanks,
Ramu

[rhassing]

Could you also send the linux-server host template?

[ramuvr]

From templates.cfg, these are the ones used:

# Generic host definition template - This is NOT a real host, just a template!

define host{
name generic-host ; The name of this host template
notifications_enabled 1 ; Host notifications are enabled
event_handler_enabled 1 ; Host event handler is enabled
flap_detection_enabled 1 ; Flap detection is enabled
process_perf_data 1 ; Process performance data
retain_status_information 1 ; Retain status information across program restarts
retain_nonstatus_information 1 ; Retain non-status information across program restarts
notification_period 24x7 ; Send host notifications at any time
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
}


# Linux host definition template - This is NOT a real host, just a template!

define host{
name linux-server ; The name of this host template
use generic-host ; This template inherits other values from the generic-host template
check_period 24x7 ; By default, Linux hosts are checked round the clock
check_interval 1 ; Actively check the host every 5 minutes
retry_interval 1 ; Schedule host check retries at 1 minute intervals
max_check_attempts 5 ; Check each Linux host 10 times (max)
check_command check-host-alive ; Default command to check Linux hosts
notification_period workhours ; Linux admins hate to be woken up, so we only notify during the day
; Note that the notification_period variable is being overridden from
; the value that is inherited from the generic-host template!
notification_interval 60 ; Resend notifications every 2 hours
notification_options d,u,r ; Only send notifications for specific host states
contact_groups admins ; Notifications get sent to the admins by default
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
}



# Windows host definition template - This is NOT a real host, just a template!

define host{
name windows-server ; The name of this host template
use generic-host ; Inherit default values from the generic-host template
check_period 24x7 ; By default, Windows servers are monitored round the clock
check_interval 5 ; Actively check the server every 5 minutes
retry_interval 1 ; Schedule host check retries at 1 minute intervals
max_check_attempts 10 ; Check each server 10 times (max)
check_command check-host-alive ; Default command to check if servers are "alive"
notification_period 24x7 ; Send notification out at any time - day or night
notification_interval 30 ; Resend notifications every 30 minutes
notification_options d,r ; Only send notifications for specific host states
contact_groups admins ; Notifications get sent to the admins by default
hostgroups windows-servers ; Host groups that Windows servers should be a member of
register 0 ; DONT REGISTER THIS - ITS JUST A TEMPLATE
}


Sorry, I am a noob when it comes to Linux, if you need anything else, please let me know.

[Box293]

Can you please find the host definition in /usr/local/nagios/var/objects.cache and paste it here. It will show us the host object definition final config.

[ramuvr]

########################################
# NAGIOS OBJECT CACHE FILE
#
# THIS FILE IS AUTOMATICALLY GENERATED
# BY NAGIOS. DO NOT MODIFY THIS FILE!
#
# Created: Mon Nov 10 21:42:11 2014
########################################

define timeperiod {
timeperiod_name 24x7
alias 24 Hours A Day, 7 Days A Week
sunday 00:00-24:00
monday 00:00-24:00
tuesday 00:00-24:00
wednesday 00:00-24:00
thursday 00:00-24:00
friday 00:00-24:00
saturday 00:00-24:00
}

define timeperiod {
timeperiod_name 24x7_sans_holidays
alias 24x7 Sans Holidays
december 25 00:00-00:00
july 4 00:00-00:00
january 1 00:00-00:00
thursday 4 november 00:00-00:00
monday 1 september 00:00-00:00
monday -1 may 00:00-00:00
sunday 00:00-24:00
monday 00:00-24:00
tuesday 00:00-24:00
wednesday 00:00-24:00
thursday 00:00-24:00
friday 00:00-24:00
saturday 00:00-24:00
}

define timeperiod {
timeperiod_name none
alias No Time Is A Good Time
}

define timeperiod {
timeperiod_name us-holidays
alias U.S. Holidays
january 1 00:00-00:00
july 4 00:00-00:00
december 25 00:00-00:00
monday -1 may 00:00-00:00
monday 1 september 00:00-00:00
thursday 4 november 00:00-00:00
}

define timeperiod {
timeperiod_name workhours
alias Normal Work Hours
monday 09:00-17:00
tuesday 09:00-17:00
wednesday 09:00-17:00
thursday 09:00-17:00
friday 09:00-17:00
}

define command {
command_name check-host-alive
command_line $USER1$/check_ping -H $HOSTADDRESS$ -w 3000.0,80% -c 5000.0,100% -p 5
}

define command {
command_name check_dhcp
command_line $USER1$/check_dhcp $ARG1$
}

define command {
command_name check_ftp
command_line $USER1$/check_ftp -H $HOSTADDRESS$ $ARG1$
}

define command {
command_name check_hpjd
command_line $USER1$/check_hpjd -H $HOSTADDRESS$ $ARG1$
}

define command {
command_name check_http
command_line $USER1$/check_http -I $HOSTADDRESS$ $ARG1$
}

define command {
command_name check_https
command_line $USER1$/check_https -I $HOSTADDRESS$ $ARG1$
}

define command {
command_name check_imap
command_line $USER1$/check_imap -H $HOSTADDRESS$ $ARG1$
}

define command {
command_name check_local_disk
command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
}

define command {
command_name check_local_load
command_line $USER1$/check_load -w $ARG1$ -c $ARG2$
}

define command {
command_name check_local_mrtgtraf
command_line $USER1$/check_mrtgtraf -F $ARG1$ -a $ARG2$ -w $ARG3$ -c $ARG4$ -e $ARG5$
}

define command {
command_name check_local_procs
command_line $USER1$/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
}

define command {
command_name check_local_swap
command_line $USER1$/check_swap -w $ARG1$ -c $ARG2$
}

define command {
command_name check_local_users
command_line $USER1$/check_users -w $ARG1$ -c $ARG2$
}

define command {
command_name check_nt
command_line $USER1$/check_nt -H $HOSTADDRESS$ -p 12489 -v $ARG1$ $ARG2$
}

define command {
command_name check_ping
command_line $USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5
}

define command {
command_name check_pop
command_line $USER1$/check_pop -H $HOSTADDRESS$ $ARG1$
}

define command {
command_name check_smtp
command_line $USER1$/check_smtp -H $HOSTADDRESS$ $ARG1$
}

define command {
command_name check_snmp
command_line $USER1$/check_snmp -H $HOSTADDRESS$ $ARG1$
}

define command {
command_name check_ssh
command_line $USER1$/check_ssh $ARG1$ $HOSTADDRESS$
}

define command {
command_name check_tcp
command_line $USER1$/check_tcp -H $HOSTADDRESS$ -p $ARG1$ $ARG2$
}

define command {
command_name check_udp
command_line $USER1$/check_udp -H $HOSTADDRESS$ -p $ARG1$ $ARG2$
}

define command {
command_name notify-host-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
}

define command {
command_name notify-service-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}

define command {
command_name process-host-perfdata
command_line /usr/bin/printf "%b" "$LASTHOSTCHECK$\t$HOSTNAME$\t$HOSTSTATE$\t$HOSTATTEMPT$\t$HOSTSTATETYPE$\t$HOSTEXECUTIONTIME$\t$HOSTOUTPUT$\t$HOSTPERFDATA$\n" >> /usr/local/nagios/var/host-perfdata.out
}

define command {
command_name process-service-perfdata
command_line /usr/bin/printf "%b" "$LASTSERVICECHECK$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEATTEMPT$\t$SERVICESTATETYPE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$\n" >> /usr/local/nagios/var/service-perfdata.out
}

define contactgroup {
contactgroup_name admins
alias Nagios Administrators
members nagiosadmin
}

define hostgroup {
hostgroup_name linux-servers
alias Linux Servers
members
}

define hostgroup {
hostgroup_name switches
alias Network Switches
members
}

define hostgroup {
hostgroup_name windows-servers
alias Windows Servers
members
}

define contact {
contact_name nagiosadmin
alias Nagios Admin
service_notification_period 24x7
host_notification_period 24x7
service_notification_options r,w,u,c,f,s
host_notification_options r,d,u,f,s
service_notification_commands notify-service-by-email
host_notification_commands notify-host-by-email
email [email protected]
minimum_importance 0
host_notifications_enabled 1
service_notifications_enabled 1
can_submit_commands 1
retain_status_information 1
retain_nonstatus_information 1
}

define host {
host_name abcdefgh
alias abcdefgh
address ip.add.of.host
check_period 24x7
check_command check-host-alive
contact_groups admins
notification_period 24x7
initial_state o
importance 0
check_interval 5.000000
retry_interval 1.000000
max_check_attempts 10
active_checks_enabled 1
passive_checks_enabled 1
obsess 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 1
flap_detection_options a
freshness_threshold 0
check_freshness 0
notification_options r,d
notifications_enabled 1
notification_interval 30.000000
first_notification_delay 0.000000
stalking_options n
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
}


define service {
host_name abcdfgdft
service_description PING
check_period 24x7
check_command check_ping!100.0,20%!500.0,60%
contact_groups admins
notification_period 24x7
initial_state o
importance 0
check_interval 5.000000
retry_interval 1.000000
max_check_attempts 4
is_volatile 0
parallelize_check 1
active_checks_enabled 1
passive_checks_enabled 1
obsess 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 1
flap_detection_options a
freshness_threshold 0
check_freshness 0
notification_options r,w,u,c
notifications_enabled 1
notification_interval 60.000000
first_notification_delay 0.000000
stalking_options n
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
}

Removed my host and service information and used only couple of samples from it. Hope this is ok. Do you need the actual file?

[ramuvr]

Bugger me ! That file, objects.cache does not have the details of the Linux hosts that I am monitoring ! What next? Hows this file being populated?

[ramuvr]

Oh that was easy, selected the group and 'enabled notification' although the individual hosts themselves had notification enabled, and tested and it worked now! How strange

Thanks guys,