Email Notifications seem to be working intermittently

[bostidr]

I am running Nagios XI 5.3.0 on CentOS. This is an enterprise edition for VMWare 64 Bit that was originally OVA download. It has been through a few upgrades at this point for Yum and for nagiosxi.

About 10 days ago, some (not all, just some) of the emails for issues quit working. It appears that the emails go out for Problems, but many times not for when the problem Resolves. It also seems to be getting worse over time. Now were are starting to see times where neither the problem nor resolution emails are sent.

I can look at the Notifications, and Nagios says that it sent the emails out.
I can test from the user and I get test notifications.
I can look at the Event Log and I see where Nagios says it sent out the email.
I have run ./repair_databases.sh nagios, no issues.
I have run tcpdump and do not see these emails sent when Nagios reports that it did in the above logs.

We send out our messages via a smtp relay at the site. I have had those guys looking at this for over an hour and they have many thousands of emails going through without issue. Nagios server will "say" in the logs that an email will have been sent and sometimes it is in the smtp and sometimes not.

So far, I have not been able to get reliability for the past 10 days for the messages. I also do not seem to be able to find a common reason.

[avandemore]

Depends on your setup: Emails will never show as sent in the SMTP log unless they are actually confirmed received by the remote side. This is a foundation principle of SMTP so if it's in the log, Nagios has done all it can do. On RHEL/CentOS this file is /var/log/maillog.

If you're using phpmailer, that's a different story. A screenshot of XI > Admin > Manage Email Settings would be helpful.

Also can you upgrade to 5.3.1? This is a bug fix only release taking care of a number of issues. Instructions for a manual update can be found here.

[bostidr]

Upgraded to 5.3.1, same issues.

Depends on your setup: Emails will never show as sent in the SMTP log unless they are actually confirmed received by the remote side. This is a foundation principle of SMTP so if it's in the log, Nagios has done all it can do. On RHEL/CentOS this file is /var/log/maillog.

Trying to explain this better. For two years this has worked without any real issues, so just saying that because this is not a new install or something we have copied, etc. There are two admins on the box, and we "believe" that we have not changed anything in these settings.

I look at these logs and I see no difference between ones that go through and ones that do not. Nagios notifier will say it went through, event log will say it went through, but a tcpdump has no record of the box trying to talk to the smtp server. When it does, the emails go through.

It "feels" like a database corruption of some type, but that is a feeling and nothing based in fact.

Hopefully that gives more information.....

[ssax]

Are you seeing anything in your /usr/local/nagiosxi/tmp/phpmailer.log? If so, please post them.

Also, run this command and generate a notification that will fail:

Code: Select all

tail -f /usr/local/nagiosxi/var/eventman.log

Send us the entire sanitized output.

Thank you

[avandemore]

There must be some kind foul up here. That screenshot contains text from pre_5.3.1. Can you ensure 5.3.1 is installed?

This is what a 5.3.1 Mail Settings page looks like:

mail_settings.png

[bostidr]

Here are the two requested files.

I upgrade to 5.3.1 after I took the picture of setting and nothing changed, so I just kept it, here is the picture again, but from now.

Thanks,
Dana

[avandemore]

XI > Admin > System Profile > Download Profile

Please include the zip file in your response. You can PM myself or another tech for privacy if you like.

EDIT: Opps, you already sent it. We are looking into this. A