Verifiy Text alerts are sent from the Handler

[isadmin]

We are trying to track intermittent failures in alerting. Some people are only getting text alerts here and there.
The /var/log/messages shows the alerts sent to the users and the xi_host_notification_handler....
When I try and track them in postfix by viewing /var/log/maillog/ I cant only see message.id=<some uniquenumber@our smtp server domain.com
How can I track and verify the alerts are making it to the Exchange servers....the admins say the server is not receiving them?

[isadmin]

To add to this when I view the maillog I only see messages sent from=root and to-root. I dont see any messages sent to our alert contacts? but obviously we are receiving some of them?

[Box293]

Let's check a few things.

When a user is logged into XI, in the top right corner it says "Logged in as: user"
Click on the User to take us to the user preferences
Under Notifications Options click Notification Preferences
Is "Enable Notifications" ticked and the relevant notification options selected?

From Core Configuration Mananger:
Find a service that is supposed to send notifications
On the Alert Settings tab of the service
By clicking Manage Contacts or Manage Contactgroups are there valid contacts selected?

Also while on this screen you'll notice Notification options and Notifications enabled might not be selected or skip is selected. This is OK as they may be inherriting these settings from templates.
Go to the Common Settings tab of the service
Click the Managed Templates button
Make a note of the name of the Assigned template(s)
Close this and the service being edited
Go to Templates > Service templates
Check the Alert Settings tab to ensure Notification options and Notifications enabled
NOTE: This template might also be using a template, so you may need to follow the template chain

Finally, Under Alerting > Contancts / Contact Groups
Check the relevant contacts or contact groups to ensure alerting has been enabled.

Let us know if this identified anything which could be causing your issue.

[isadmin]

Thanks Box293
Two of us have traced all alerting from the service/host to the notifications to the contact templates and groups and everything is good.
From /var/log/messages we see the alerts sent to all the contacts via the the handler for the users but sometimes they are not all received by our mail server to send out? I want to track them some how from the handler to postfix to our exchange servers and
see where they are getting dropped if that makes sense?
I dont see any in /var/maillog so I am not sure. I just see message id's and root sending to and from root?
what happens to them once the handler receives the alert notification?
Just for testing yesterday we downed a service for 3 hrs.....6 people should have been notified every 30 min.
4 users on different ISP's for cell service received all the alerts....one user didnt receive the first or second alert but did receive the 3rd...the second user received the first alert and then the 3rd alert....its like the handler or NDO
is dropping something? /var/log/messages shows all users being sent the alerts via the handler.

Heres the messages from /var/log/ but we never see the missing user alerts make it to the exchange server...
we have crossed referenced this with our mail server logs...their getting dropped between the handler and the exchange somewhere?
Looking into this deeper I assume we dont see anything in postfix because we use the stmp setting...so the handler should be sending the alert emails driectly to our Exchange servers correct?

Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION:user1;test-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION: user2;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION: helpdesk;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION: user3;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION: helpdesk_user;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION: user4;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION:user5;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION: user6;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION: user7;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%
Jun 9 02:08:27 our_servername nagios: HOST NOTIFICATION: user8;portage-router;DOWN;xi_host_notification_handler;PING CRITICAL - Packet loss = 100%

heres an example of the entries in /var/maillog
Jun 10 06:40:22 our_servername postfix/qmgr[30409]: 564E91D4D: removed
Jun 10 06:45:06 our_servername postfix/pickup[5567]: 54DB1152F: uid=0 from=<root>
Jun 10 06:45:06 our_servername postfix/cleanup[21260]: 54DB1152F: message-id=<20140610114506.54DB1152F@our_servername.our_domain.com>
Jun 10 06:45:06 our_servername postfix/qmgr[30409]: 54DB1152F: from=<root@our_servername.our_domain.com>, size=3226, nrcpt=1 (queue active)
Jun 10 06:45:06 our_servername postfix/local[21262]: 54DB1152F: to=<root@our_servername.our_domain.com>, orig_to=<root>, relay=local, delay=0.14, delays=0.12/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)

[sreinhardt]

Just for testing yesterday we downed a service for 3 hrs.....6 people should have been notified every 30 min.
4 users on different ISP's for cell service received all the alerts....one user didnt receive the first or second alert but did receive the 3rd...the second user received the first alert and then the 3rd alert

So these are being tested as sent to a email2sms gateway? Are you using the carriers gateways, and what carriers are you using?

Looking into this deeper I assume we dont see anything in postfix because we use the stmp setting...so the handler should be sending the alert emails driectly to our Exchange servers correct?

Correct since these are using the XI notification handler if you have smtp settings configured all messages should use that. and not postfix\sendmail unless that is your configured mail sending option.

Have you tried tcpdumping when a set of notifications are set to come out? This would absolutely confirm if they are at least leaving the nagios system and being stopped somewhere else.

[Box293]

Just going through what you have explained and did some testing on system here.

but we never see the missing user alerts make it to the exchange server...
we have crossed referenced this with our mail server logs...their getting dropped between the handler and the exchange somewhere?
Looking into this deeper I assume we dont see anything in postfix because we use the stmp setting...so the handler should be sending the alert emails driectly to our Exchange servers correct?

Correct. When using the SMTP method, Sendmail is not responsible for sending the alerts

heres an example of the entries in /var/maillog
Jun 10 06:40:22 our_servername postfix/qmgr[30409]: 564E91D4D: removed
Jun 10 06:45:06 our_servername postfix/pickup[5567]: 54DB1152F: uid=0 from=<root>
Jun 10 06:45:06 our_servername postfix/cleanup[21260]: 54DB1152F: message-id=<20140610114506.54DB1152F@our_servername.our_domain.com>
Jun 10 06:45:06 our_servername postfix/qmgr[30409]: 54DB1152F: from=<root@our_servername.our_domain.com>, size=3226, nrcpt=1 (queue active)
Jun 10 06:45:06 our_servername postfix/local[21262]: 54DB1152F: to=<root@our_servername.our_domain.com>, orig_to=<root>, relay=local, delay=0.14, delays=0.12/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)

These messages are not going to be related to the alerts. When I was doing some testing I was watching /var/log/maillog and /var/log/messages. I could see the alerts appear in /var/log/messages and nothing would appear in /var/log/maillog (which is expected when using SMTP method). These messages you are seeing in /var/log/maillog are system messages being sent to your root account. Have a look at /var/spool/mail/root to see what they are.

I've just noticed that sreinhardt has replied as well. Follow his advice and I'm going to do a little more research.

[Box293]

I have been looking through some code and it appears that emails are sent using PHPMailer.

However I cannot seem to find any documentation that explains how to turn on logging so we can track what happens to each message that is sent.

[isadmin]

Yes the alerts are being sent via smtp to our servers then relayed to [email protected] @mms.att.net etc etc....strange thing is if this was a service provider issue why does someone get alert 1 and alert 3 ? but 2 gets lost
Again last night alerts went off I did not receive an alert even though /var/log/messages shows the alert being send to me via the handler....I will look at tcpdump to see if we can catch them leaving

[isadmin]

I am using tcpdump to capture the packets and I do see something going to the exchange server but how can I view the
capture in text form to see the actual email requests being sent?
Im using tcpdump -nXXv -A -s0 -tttt -w handler-alert.pcap dst [ip address of smtp server]
then tcpdump -tttt -r handler-alert.pcap to view

[isadmin]

found a great command for the tool box
tcpdump -l -s0 -w - tcp dst port 25 | strings | grep -i 'MAIL FROM\|RCPT TO'
shows when the emails are sent from nagios to smtp