Stop Receiving Nagios Notification Lately

[StephMengMeng]

Hi,

I've run into some weird behavior related to Nagios3 lately that bothered me quite a lot. The thing is that, we stopped receiving email notification for the past week. I went through the following steps trying to diagnosis and still not figuring it out. Would you please give me some suggestions?

1. Checked Nagios configuration files and contact files, basically all these files are not updated since Nagios works fine.
2. We are using PostFix on Ubuntu. PostFix works fine sending email to my inbox.
3. Checked the /var/log/mail.log, and actually found email sent out from nagios server and the format is the same as the test email in #2.
4. Checked /var/log/nagios/nagios.log and found notify-service-by-email services are issued.
5. 'Send custom service notification' manually from 'Send custom service notification' page on Nagios UI. NOT RECEIVING ANYTHING tho.

Would you please give me some idea based on my checks describe above?

Thank you.
Stephanie

[tmcdonald]

Did the mail log reveal anything about a possible blacklist or rejected email? Often if you don't go through "trustworthy" relays to send mail your messages can be picked up as spam or blocked entirely.

[StephMengMeng]

Hi tmcdonald,

I don't see any blacklist or rejection related to the nagios actions. All parameters from postfix/pickup-cleanup-qmgr-smtp-gmgr looks clean to me. Is there anyway I can test if it's Nagios not sending email by Linux CLI instead of from UI (since it doesn't work for me)… Or any possible issue?

Thanks!
Stephanie

[slansing]

Do you have notifications enabled in the interface? If so, does your (/var/log/mail.log) show standard notifications being sent out? Perhaps you could give us a snippet from that log, of an entire email chain being sent be sure to remove the sent to address if it is external.

[StephMengMeng]

Hi slansing,

I think we have notification enabled in the interface. Below are logs from both mail log and nagios mail log for sending out custom message. (I replace the domain name with example'.

'Apr 30 10:23:55 server1 postfix/pickup[10359]: 8E3D53FE02E6: uid=107 from=<nagios>
Apr 30 10:23:55 server1 postfix/cleanup[18836]: 8E3D53FE02E6: message-id=<20140430152355.8E3D53FE02E6@server1>
Apr 30 10:23:55 server1 postfix/qmgr[12552]: 8E3D53FE02E6: from=<nagios@server1>, size=617, nrcpt=1 (queue active)
Apr 30 10:23:56 server1 postfix/smtp[18838]: 8E3D53FE02E6: to=<etl@example.com>, relay=example.com.s6a1.psmtp.com[64.18.5.10]:25, delay=1.3, delays=0.01/0/0.31/0.98, dsn=2.0.0, status=sent (250 Thanks)
Apr 30 10:23:56 server1 postfix/qmgr[12552]: 8E3D53FE02E6: removed
'

'[1398871435] SERVICE NOTIFICATION: ETL;etlfarme;ETL Frame queue check;CUSTOM (CRITICAL);notify-service-by-email;multiple instances are scheduled to run for the following job_id:;nagiosadmin;this is a test'

Thanks for the help.

Stephanie

Do you have notifications enabled in the interface? If so, does your (/var/log/mail.log) show standard notifications being sent out? Perhaps you could give us a snippet from that log, of an entire email chain being sent be sure to remove the sent to address if it is external.

[slansing]

So it certainly looks like they are leaving the Nagios server, I would start looking at your relay/smtp server or anything between nagios and your inbox as to why they are getting blocked or dropped.

[StephMengMeng]

Hi slansing ,

You are right. It's something configuration changed on the mail server side and now it's fixed. Thanks a lot for the help!

Stephanie

So it certainly looks like they are leaving the Nagios server, I would start looking at your relay/smtp server or anything between nagios and your inbox as to why they are getting blocked or dropped.