Nagios Support Forum

Hello Nagios XI Support Team,

I'm just now noticing that Nagios XI email notifications have not worked since July 24th on my platform. And no one reported it!

We've upgraded to 5.7.X (currently 5.7.2) over the summer, but I'm not sure which dates we did which version.

I can send emails via the command line with or without a -r. Weekly scheduled reports are working. Also, sending test notification emails & SMSs from the user's notifications preferenceswork as well.

Here's what's in our phpmailer.log from when it stopped working:

/usr/local/nagiosxi/tmp/phpmailer.log:
[07-24-2020 07:41:31] Message sent! (method=sendmail), Referer: includes/components/xicore/xicore.inc.php > Event Handler Notification Email
[07-24-2020 07:41:32] Message sent! (method=sendmail), Referer: includes/components/xicore/xicore.inc.php > Event Handler Notification Mobile Text
[07-24-2020 08:50:44] Message sent! (method=sendmail), Referer: includes/components/xicore/xicore.inc.php > Event Handler Notification Email
[07-24-2020 08:50:44] Message sent! (method=sendmail), Referer: includes/components/xicore/xicore.inc.php > Event Handler Notification Mobile Text
[07-24-2020 09:51:47] Message sent! (method=sendmail), Referer: includes/components/xicore/xicore.inc.php > Event Handler Notification Email
[07-24-2020 09:51:47] Message sent! (method=sendmail), Referer: includes/components/xicore/xicore.inc.php > Event Handler Notification Mobile Text
[07-24-2020 22:26:54] Message sent! (method=sendmail), Referer: admin/users.php > Account Creation
[07-27-2020 01:00:47] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[07-27-2020 02:00:41] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
...
...
...
[09-07-2020 08:00:08] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-07-2020 12:00:16] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-07-2020 12:00:16] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-08-2020 19:12:01] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-09-2020 02:15:03] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-09-2020 16:43:28] Message sent! (method=sendmail), Referer: admin/users.php > Clone User - Account Creation
[09-10-2020 03:23:45] Message sent! (method=sendmail), Referer: account/testnotification.php > PHPmailer Test
[09-10-2020 03:23:46] Message sent! (method=sendmail), Referer: account/testnotification.php > Mobile Test
[09-10-2020 03:41:13] Message sent! (method=sendmail), Referer: send_user_phone_vkey()
[09-10-2020 03:41:45] Message sent! (method=sendmail), Referer: verify_phone_vkey()
[09-10-2020 09:52:54] Message sent! (method=sendmail), Referer: login.php > Password Reset Email
[09-10-2020 10:44:21] Message sent! (method=sendmail), Referer: login.php > Password Reset Email
[09-11-2020 15:55:08] Message sent! (method=sendmail), Referer: admin/users.php > Administrator Password Reset
[09-14-2020 01:00:48] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-14-2020 01:00:48] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-14-2020 02:00:45] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-14-2020 02:00:45] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-14-2020 08:00:09] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-14-2020 12:00:05] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-14-2020 12:00:05] Message sent! (method=sendmail), Referer: includes/components/scheduledreporting > Scheduled Report
[09-15-2020 15:43:06] Message sent! (method=sendmail), Referer: account/testnotification.php > PHPmailer Test
[09-15-2020 15:43:07] Message sent! (method=sendmail), Referer: account/testnotification.php > Mobile Test
[09-15-2020 18:46:33] Message sent! (method=sendmail), Referer: send_user_phone_vkey()
[09-15-2020 18:46:52] Message sent! (method=sendmail), Referer: send_user_phone_vkey()
[09-15-2020 18:47:15] Message sent! (method=sendmail), Referer: verify_phone_vkey()

Here's what's in the syslogs, maillogs after testing a services notification on one of my servers:

root@nagios-b log]# tail -f messages | grep luczy
Sep 15 19:10:31 nagios-b nagios: SERVICE NOTIFICATION: luczynj;GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;xi_service_notification_handler;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:10:31 nagios-b nagios: SERVICE NOTIFICATION: luczynjROTA;GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;xi_service_notification_handler;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):


Sep 15 19:07:19 nagios-b nagios: SERVICE ALERT: GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;SOFT;1;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:07:31 nagios-b nagios: SERVICE ALERT: GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;SOFT;2;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:08:33 nagios-b nagios: SERVICE ALERT: GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;SOFT;3;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:09:33 nagios-b nagios: SERVICE ALERT: GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;SOFT;4;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:10:31 nagios-b nagios: SERVICE NOTIFICATION: youngr;GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;xi_service_notification_handler;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:10:31 nagios-b nagios: SERVICE NOTIFICATION: toolea;GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;xi_service_notification_handler;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:10:31 nagios-b nagios: SERVICE NOTIFICATION: luczynj;GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;xi_service_notification_handler;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:10:31 nagios-b nagios: SERVICE NOTIFICATION: luczynjROTA;GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;xi_service_notification_handler;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):
Sep 15 19:10:31 nagios-b nagios: SERVICE ALERT: GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;HARD;5;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):


[root@nagios-b log]# grep luczynj maillog
Sep 15 15:49:37 nagios-b sendmail[7404]: 08FFnaFd007404: from=luczynj, size=31809, class=0, nrcpts=1, msgid=<[email protected]>, relay=root@localhost
Sep 15 15:49:37 nagios-b sendmail[7550]: 08FFnbSn007550: from=<[email protected]>, size=32104, class=0, nrcpts=1, msgid=<[email protected]>, proto=ESMTP, daemon=MTA, relay=localhost [127.0.0.1] (may be forged)
Sep 15 15:49:37 nagios-b sendmail[7404]: 08FFnaFd007404: to=[email protected], ctladdr=luczynj (501/506), delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=61809, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (08FFnbSn007550 Message accepted for delivery)
Sep 15 15:49:39 nagios-b sendmail[7793]: 08FFnbSn007550: to=<[email protected]>, ctladdr=<[email protected]> (501/506), delay=00:00:02, xdelay=00:00:02, mailer=relay, pri=152104, relay=rspsmtp2.reston.concert.com. [168.125.8.217], dsn=2.0.0, stat=Sent ( <[email protected]> Queued mail for delivery)


There's not an etry in maillog for the service clearing, and that is in my preferences.

Thank you for your support. Please help!

Regards,
JL
Sep 15 19:13:58 nagios-b nagios: SERVICE NOTIFICATION: luczynj;GMV-OSS-RC-TPOSSA01;/ Disk Usage;OK;xi_service_notification_handler;DISK OK - free space: / 9039 MB (45% inode=89%):
Sep 15 19:13:58 nagios-b nagios: SERVICE NOTIFICATION: luczynjROTA;GMV-OSS-RC-TPOSSA01;/ Disk Usage;OK;xi_service_notification_handler;DISK OK - free space: / 9039 MB (45% inode=89%):

Hi,

Thank you for contacting the Nagios Support Team.

Just wondering if you continued to receive the scheduled reports the entire time? Also, is there nothing in the phpmailerlog between 7/27 and 9/7?

Looking at the output the logs posted, the contact is configured to use the XI mailer handler or the phpmailer library. Did you receive this notification? I don't see the corresponding entry in the log. Also, can you send over the system profile and I'll check the configurations on this contact/service?

Regards,
Benjamin

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.

Reference:
Nagios XI - Notification Problems

Hello,

Thanks for the resposne.

Yes, the scheduled reports have been working.

No, I didn't receive the email for this notification:
Sep 15 19:10:31 nagios-b nagios: SERVICE NOTIFICATION: luczynj;GMV-OSS-RC-TPOSSA01;/ Disk Usage;CRITICAL;xi_service_notification_handler;DISK CRITICAL - free space: / 1867 MB (9% inode=89%):

The /usr/local/nagiosxi/tmp/phpmailer.log shows the last Event Handler notification on [07-24-2020 09:51:47].
After that, it's password resets, PHPmailer tests, Mobile Tests, Account Creation, etc.. No more Event Handlers.

Regards,
Joe

Your contact is set to use the backend MTA:

define contact {
contact_name l*****j
alias J***********i
service_notification_period 24x7
host_notification_period 24x7
service_notification_options r,w,u,c,f,s
host_notification_options r,d,u,f,s
service_notification_commands notify-service-by-email
host_notification_commands notify-host-by-email

Go to Configure > Core Config Manager > Contacts:
- Edit your contact
- Click the Alert Settings tab
- Click the Host Notification Commands button and REMOVE notify-host-by-email and ADD xi_host_notification_handler
- Click OK
- Click the Service Notification Commands button and REMOVE notify-service-by-email and ADD xi_service_notification_handler
- Click OK
- Click Save and Apply Configuration

Now it will use the proper XI settings in Admin > Email Settings instead of the backend MTA.

You can either do that OR If you really want it to use the backend MTA (postfix likely in this case) you would need to add a relayhost to your /etc/postfix/main.cf and restart the postfix service so it will relay through the server you want (there are a ton of guides on google for this). Then it should relay properly as well. I would probably do this anyways (but still change your contact) because distribution lists won't have users either and those will be required to use notify-host/service-by-email (because they don't have an associated user in Admin > Manage Users with the same name which is a requirement of the xi_host/service_notification_handlers).

Hello.

Over the weekend a colleague found that /usr/local/nagiosxi and /usr/local/nagiosxi/var were set to 755, which stopped the event handler from sending the mail through.

It appears that this happened ~around~ the time of our 5.7.2 upgrade. Is this a known issue? What should the permissions of this file system be?

[root@nagios-b /]# ls -lsa /usr/local
4 drwxr-xr-x. 20 root root 4096 Apr 25 2019 .
4 drwxr-xr-x. 16 root root 4096 Oct 3 2018 ..
4 drwxr-xr-x. 2 root root 4096 Jun 18 12:47 bin
4 drwxr-xr-x. 2 root root 4096 Sep 23 2011 etc
4 drwxr-xr-x. 2 root root 4096 Sep 23 2011 games
4 drwxr-xr-x. 3 root root 4096 Aug 23 2018 include
4 drwxr-xr-x. 3 root root 4096 Aug 23 2018 lib
4 drwxr-xr-x. 3 root root 4096 Jul 10 2017 lib64
4 drwxr-xr-x. 2 root root 4096 Sep 23 2011 libexec
4 drwxr-xr-x 9 root root 4096 Feb 22 2018 nagios
4 drwxr-xr-x. 6 root root 4096 Jul 11 2017 nagiosmobile
4 drwxr-xr-x 12 root nagios 4096 May 5 2016 nagiosxi

If you would like all of his troubleshooting notes, let me know.

Regards,
Joe

I haven't seen this reported other than from your other post on this, here is what mine shows:
Are you just notifying us again on this or do you still have issues?

Hey there. Thanks for correcting this.

I had mistakenly posted on the other forum post, thinking it was this one.

The issue is fixed now due to the group write permission on /usr/local/nagiosxi/var wasn't set.

I'm curious if this occurred during our 5.7.2 upgrade, since it occurred on both our primary and backup server around the same time that we upgraded from 5.7.1 to 5.7.2.

Thanks again.

JLu

I haven't heard of anyone else reporting this, this also didn't happen on my system when testing.

I checked the code and didn't see any permissions changes that should impact that.

What is the output of these commands as root?