Nagios Mailer delayed/dropping outgoing mail

[Dusan.Mandic]

Hello all,

I have recently migrated our NagiosXI 5.6.x RHEL 6 to a NagiosXI 5.7.1 RHEL7 build. We are having some recurring issues with contacts/users receiving email notifications. I went down the inheritance/permissions tree and found that Nagios states in Notification Reports that a user was sent an email, but the user did not receive the email notification. I am on the same service and did recieve the notification. The user both receives other service alerts from the same host with the same notification/alert settings and other hosts using the same service with same notification/alert settings. I checked our Exchange SMTP server and it looked like the user did not receive mail.

We also had a maintenance event this past weekend, in which there were thousands of hosts/services in downtime. When the downtime over notification sent, they were sent over a period of 4 hours, which strikes me as something with the mailer service being incorrect. I would like assistance in the matter as soon as possible please

Best,

Dusan

[ssax]

Send a screenshot of Admin > Email Settings.

Please follow these instructions:

Please go to Admin > Manage Email Settings:
- Make sure Logging is checked
- Click the Update Settings button

Then run this tail command (and leave it running):

Code: Select all

tail -Fn0 /var/log/maillog /usr/local/nagiosxi/tmp/phpmailer.log /usr/local/nagiosxi/var/eventman.log

Then force a notification to be sent then send me the full output of the tail command above and any errors you see on the screen (please test via the instructions in bold at the bottom exactly).

After you've done the steps above attach a FRESH copy of your profile.zip, you can download it from Admin > System Profile > Download Profile.

Additionally, please send the output of these commands (as root) so that I can check for some other common issues that could be affecting notifications:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

How to submit passive results for testing:

For Hosts
------------

Go to Home > Host Status:
- Find the Host and click on it
- Click the + tab
- Click the "Submit passive check result" link
- Select the Check Result and type in some text for the Check Output
- Click the Submit button

NOTE: By default, passive_host_checks_are_soft=0 is set in your /usr/local/nagios/etc/nagios.cfg, this differs from services in that when you submit a passive host check result it immeditely goes into a HARD state (and should send a notification if configured to) instead of going into a SOFT state like services do. (notifications are only sent on HARD states).

For Services
----------------

Go to Home > Service Status:
- Find the Service and click on it
- Click the + tab
- Note these two rows:

State Type: Hard
Current Check: 1 of 4

Those columns tell the current State Type and the Current Check number. In order to generate a notification for a service you will need to submit MULTIPLE problem check results (the number that you need to submit is determined by the last number in the Current Check column, that is the max_check_attempts setting). For services, when you submit a passive check result, each result that you submit will be a SOFT state until you submit enough to hit the Max Check Attempts setting that you've defined on the service, only then will the service enter a HARD problem state which will generate the notification (just remember, notifications are only sent on HARD states).

- Click the "Submit passive check result" link
- Select the Check Result and type in some text for the Check Output
- Click the Submit button
- Submit as many as you need, right after another, until the service enters the HARD state so that a notification will be sent

NOTE: When coming from a HARD problem state (whether we are talking about hosts or services) if you submit an OK passive result it should fire off a recovery notification after a single passive result has been submitted.

[Dusan.Mandic]

Thanks SSAX,

I noticed that our host passive checks were disabled. I turned them on for the Notification testing.

[ssax]

I'm seeing this:

Code: Select all

PHP Warning:  file_put_contents(/usr/local/nagiosxi/tmp/phpmailer.log): failed to open stream: Permission denied

Please run these commands:

Code: Select all

chown nagios.nagios /usr/local/nagiosxi/tmp/phpmailer.log
chmod 0774 /usr/local/nagiosxi/tmp/phpmailer.log

Then do the testing again and resend the output.

Can you send a screenshot of the whole line in the notification report so I can see the contact name and the dispatcher that it used?

[Dusan.Mandic]

Can you send a screenshot of the whole line in the notification report so I can see the contact name and the dispatcher that it used?

Not sure what you mean by this. The text file is the whole terminal output of the tail command.

This is the service notification email that purportedly is not being sent:

Email Notification Data:

Array
(
[meta] => Array
(
[notification-type] => service
[contact] => iseriesadmins
[contactemail] => XXXXXXXXXX
[type] => PROBLEM
[escalated] => 0
[author] =>
[comments] =>
XXXXX
XXXXXXXXXXXXX
[service] => check_DISK
[hoststate] => UP
[hoststateid] => 0
[servicestate] => CRITICAL
[servicestateid] => 2
[lastservicestate] => CRITICAL
[lastservicestateid] => 2
[servicestatetype] => HARD
[currentattempt] => 5
[maxattempts] => 5
[serviceeventid] => 895066
[serviceproblemid] => 245101
[serviceoutput] => NAGIOS SUPPORT TESTING
[longserviceoutput] =>
[datetime] => Thu Feb 25 11:18:08 CST 2021
)

[referer] => includes/components/xicore/xicore.inc.php > Event Handler Notification Email
[to] => ST-iSXXXXXXXXXXXXXX
[subject] => PROBLEM Service Alert - XXXXXXXXXX/check_DISK is CRITICAL
[high_priority] => 0
[message] => ***** Nagios XI Alert *****


Moderator's Note: The files has been shared with the support team but has been removed from the public forum.

[ssax]

From your logs we see it sent to the mail server right after that, you may want reach out to your mail admins and have them investigate as I've never seen it say sent and not have made it to the mail server:

[02-25-2021 11:18:22] Message sent! (method=smtp;host=XXXX.XXXXX.XXXXX;port=25;security=none), Referer: includes/components/xicore/xicore.inc.php > Event Handler Notification Email

[Dusan.Mandic]

Thanks SSAX,

I think this one may have been the autofill on SMTP. We didnt run the logs, but it seems like notifications are going out now.

[scottwilkerson]

Thanks SSAX,

I think this one may have been the autofill on SMTP. We didnt run the logs, but it seems like notifications are going out now.

Great!

Locking thread