Page 1 of 1
Unable to send notification emails
Posted: Thu May 02, 2019 1:30 pm
by jjeremydiaz
I have been having issues with receiving email alerts on Nagios XI. Emails do not get sent even though a notification gets visibly triggered and seen in nagios.log.
I tried the following:
* Made sure emails could be sent by Nagios through send a test email within the Email Settings. Test emails work
* Attached 2 contact object, 1 nagios user, 1 normal contact to a test service and enabled ALL notifications.
* Double checked the nagios user and made sure notifications were enabled for that user
* Lowered attempt thresholds to 1 to go immediately into a hard state.
* Tried attaching the contacts to a host and tried getting notifications to go through hosts instead
* Tried sending forced custom notifications
* Made sure the contact objects have notifications enabled. Made sure the notification commands were set accurately
According to the nagios.log, the notifications are being triggered but according to phpmailer.log nothing is getting sent.
What else could be preventing emails notifications from being sent?
Re: Unable to send notification emails
Posted: Thu May 02, 2019 2:16 pm
by benjaminsmith
Hi
@jjeremydiaz,
What version of Nagios XI are you using? We had an issue with notifications in 5.6, but it is now corrected in 5.6.1.
1. Please go to Admin > Manage Email Settings:
- Make sure Debug Log is checked
- Click the Update Settings button
- Take a screenshot of the settings here and send it to me
2. Then run this tail command (and leave it running):
Code: Select all
tail -F /var/log/maillog /usr/local/nagiosxi/tmp/phpmailer.log /usr/local/nagiosxi/var/eventman.log
Then force a notification (see below) to be sent then send me the full output of the tail command above and any errors you see on the screen.
How to submit passive results for testing:
For Hosts
------------
Go to Home > Service Status:
- Find the Service and click on it
- Click the + tab
- Click the "Submit passive check result" link
- Select the Check Result and type in some text for the Check Output
- Click the Submit button
NOTE: By default, passive_host_checks_are_soft=0 is set in your /usr/local/nagios/etc/nagios.cfg, this differs from services in that when you submit a passive host check result it immeditely goes into a HARD state (and should send a notification if configured to) instead of going into a SOFT state like services do. (notifications are only sent on HARD states).
3. After you've done the steps above attach a FRESH copy of your profile, you can download it from Admin > System Profile > Download Profile.
Re: Unable to send notification emails
Posted: Mon May 06, 2019 3:33 pm
by jjeremydiaz
Hi benjaminsmith,
I sent over the info you asked for in a pm but will also post here for future reference.
Thanks for the help!
Re: Unable to send notification emails
Posted: Tue May 07, 2019 11:09 am
by benjaminsmith
Hi Jeremy,
I reviewed your logs and configuration files with other members of the support team, and I have a few suggestion and hopefully we can get to the bottom of this for you.
1. The additional test user (jd_test) is using the Nagios core contact handlers (notify-service-by-email) which uses the default Linux MTA (sendmail), and this is not working. It's not the cause of the issue, but I wanted to point that out.
==> /var/log/maillog <==
May 2 13:52:58 xi01 postfix/error[98627]: 1A7DFA213B: to=<
[email protected]>, relay=none, delay=17380, delays=17380/0.52/0/0.08, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to host-pc : Connection timed out)
May 2 13:52:58 xi01 postfix/error[98625]: 386C5A200C: to=
[email protected]
2. It's likely the reason you are not receiving notifications, is that the service, check_litecoin_prices , is in a flapping state and Nagios will suppress notifications when a host or service is flapping. From the Nagios log:
[1556829925] SERVICE NOTIFICATION: jd;host1- Crypto Checks;check_litecoin_prices;FLAPPINGSTOP (CRITICAL);xi_service_notification_handler;CRITICAL LTC @ 73.76 USD
[1556829925] SERVICE NOTIFICATION: jd_test;host1 - Crypto Checks;check_litecoin_prices;FLAPPINGSTOP (CRITICAL);notify-service-by-email_custom;CRITICAL LTC @ 73.76 USD
I would recommend disabling flap detection, and test notifications by sending passive checks to change the service state. Let us know if that works or not.
Go to Admin > System Information > Monitoring Engine Status to enable or disable. Alternatively, this can be disabled in the main Nagios configuration file, and then re-start Nagios Core.
Detection and Handling of State Flapping
Re: Unable to send notification emails
Posted: Tue May 07, 2019 2:52 pm
by jjeremydiaz
Hi benjaminsmith,
Thanks for the reply. Unfortunately, disabling flapping on the litecoin check did not work. Ultimately what I am attempting to accomplish is to create custom email notification methods for Nagios core contact objects. To do so I cloned the notify-host-by-email and notify-service-by-email commands and set them to the jd_test contact. I then added both the jd XI user and the jd_test core contact to both the litecoin service check and the crypto host object. Initially the crypto host object was the only one receiving email alerts but only for the jd XI user, not the jd_test core contact with the custom notify-host-by-email_custom command.
I next (after your flapping advice) tried to configure the "Notification Management" under admin for the jd XI user and now I am receiving 0 email notifications again!
Attached are some pictures of the configuration:
Re: Unable to send notification emails
Posted: Tue May 07, 2019 5:02 pm
by tgriep
With the changes that you have done, we will need to get a new System Profile from the server.
What I did find it that the jd_test contact was using the the old style email command and that uses the Postfix mailer daemon to forward the emails.
I found in the old profile that the postfix daemon was getting blocked from forwarding emails.
See the /var/log/maillog file for those errors.
Once I get a new profile, I can check the settings.
BTW, are you seeing the notifications in the Home > Notification menu?
Re: Unable to send notification emails
Posted: Tue May 07, 2019 5:09 pm
by benjaminsmith
Hi Jeremy,
Ultimately what I am attempting to accomplish is to create custom email notification methods for Nagios core contact objects. To do so I cloned the notify-host-by-email and notify-service-by-email commands and set them to the jd_test contact.
In Nagios XI, by default xi users use different notification commands than xi users. The mail settings (from your last post) will only work if you are using xi_host_notification_handler and xi_service_notification_handler notification handlers to send notifications. ( See:
Configuring Core Contacts to Use Nagios XI PHPMailer SMTP Settings )
Also, from the mail log, the notify-host-by-email and notify-service-by-email messages are not being delivered due to connection timed out error. Generally, the process for troubleshooting notifications is to:
1. Force a state change on the host or service so Nagios generates a notification
2. Confirm the notification in the nagios log
3. Examine the mail logs to make sure the notification was successfully sent
Please perform the following steps and send over the mail logs, notification report and a fresh system profile.
1. Tail the mail logs again
Code: Select all
tail -F /var/log/maillog /usr/local/nagiosxi/tmp/phpmailer.log /usr/local/nagiosxi/var/eventman.log
2. Go to Home > Service Status > Crypto Checks and select the advanced tab and Submit Passive Check Results to change the service state to hard critical ( or send a custom notification)
3. Go to Home > Incident Management > Notifications ... and verify that Nagios did send a notification.
4. Review the mail logs to see if the message was delivered.
Re: Unable to send notification emails
Posted: Tue May 07, 2019 7:29 pm
by jjeremydiaz
Sent you the updated profile. As for the notifications, I can see the crypto host notifications but I can not see the litecoin service check notifications. I am receiving the crypto host email alerts but still no alerts for the litecoin check.
Re: Unable to send notification emails
Posted: Wed May 08, 2019 8:50 am
by tgriep
The cp user account has their notifications disabled in the user account so it will have to be enabled to receive emails.
Login to the XI GUI as the cp user and click on the user name in the top right corner.
Click on the Notification Preferences menu and make sure the notifications are enabled and the alert types.
Click on the Notification Methods menu and make sure the Receive alerts via email. is checked.
Save any changes.
In the profile, I verified that the the Host for the Crypto check is down and sending notifications.
FYI, when Nagios detects that the host is down, it will not send the notifications for the services assigned to that host so that is the reason you are not receiving the service notifications.
The service called check_litecoin_prices, can you go to the Reports Menu and run a State History report and verify that the check went in to a Hard non-OK state?
Login as root to the nagios server, run the following commands and post the output so we can view it.
Code: Select all
ls -l /usr/local/nagios/
ls -l /usr/local/nagios/libexec.rex
Re: Unable to send notification emails
Posted: Wed May 08, 2019 8:57 am
by lmiltchev
I didn't see any entries for the "check_litecoin_prices" service in the nagios.log... Was the service in a hard non-OK state? Also, was it at the time, when the "xi01 - Crypto Checks" host was in a hard DOWN state? I see the following in the log:
[Tue May 7 19:14:28 2019] HOST NOTIFICATION: cp;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:14:28 2019] HOST NOTIFICATION: jd;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:15:28 2019] HOST NOTIFICATION: cp;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:15:28 2019] HOST NOTIFICATION: jd;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:17:12 2019] HOST NOTIFICATION: cp;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:17:12 2019] HOST NOTIFICATION: jd;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:18:26 2019] HOST NOTIFICATION: cp;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:18:26 2019] HOST NOTIFICATION: jd;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:19:26 2019] HOST NOTIFICATION: cp;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:19:26 2019] HOST NOTIFICATION: jd;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:20:26 2019] HOST NOTIFICATION: cp;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:20:26 2019] HOST NOTIFICATION: jd;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:21:26 2019] HOST NOTIFICATION: cp;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:21:26 2019] HOST NOTIFICATION: jd;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:23:24 2019] HOST NOTIFICATION: cp;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
[Tue May 7 19:23:24 2019] HOST NOTIFICATION: jd;xi01 - Crypto Checks;DOWN;xi_host_notification_handler;Jello 5
It perfectly normal not to receive notifications on a service, if the host this service is attached to is down. This is by design.
If the host is in hard non-OK state, notifications for services on this host won't be sent out.
https://assets.nagios.com/downloads/nag ... tions.html