Problems with emails after database crash

[jorgeaaq]

I have a powerfailure, after that nagios database complain about some tables crashed

after run the script everything appears to be working but I do not receive emails

I can send a Test Email and works

If I check /var/log/maillog there are no new logs of mail generated by nagios

The configuration appears to be the same for 9months ( always working and sending emails normally)

if I check /usr/local/nagios/var/nagios.log appears lines like this

SERVICE NOTIFICATION: nagiosadmin;CiscoPBX;Gi0/1 Status;CRITICAL;xi_service_notification_handler;CRITICAL: Interface GigabitEthernet0/1 (index 5) is down.

If I check /var/log/messages appear lines like this

Nov 14 18:25:28 cobra nagios: SERVICE NOTIFICATION: nagiosadmin;CiscoPBX;Gi0/1 Status;CRITICAL;xi_service_notification_handler;CRITICAL: Interface GigabitEthernet0/1 (index 5) is down.

but not emails are generated

command xi_service_notification_handler looks like this

xi_service_notification_handler

/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php --notification-type=service --contact="$CONTACTNAME$" --contactemail="$CONTACTEMAIL$" --type=$NOTIFICATIONTYPE$ --escalated="$NOTIFICATIONISESCALATED$" --author="$NOTIFICATIONAUTHOR$" --comments="$NOTIFICATIONCOMMENT$" --host="$HOSTNAME$" --hostaddress="$HOSTADDRESS$" --hostalias="$HOSTALIAS$" --hostdisplayname="$HOSTDISPLAYNAME$" --service="$SERVICEDESC$" --hoststate=$HOSTSTATE$ --hoststateid=$HOSTSTATEID$ --servicestate=$SERVICESTATE$ --servicestateid=$SERVICESTATEID$ --lastservicestate=$LASTSERVICESTATE$ --lastservicestateid=$LASTSERVICESTATEID$ --servicestatetype=$SERVICESTATETYPE$ --currentattempt=$SERVICEATTEMPT$ --maxattempts=$MAXSERVICEATTEMPTS$ --serviceeventid=$SERVICEEVENTID$ --serviceproblemid=$SERVICEPROBLEMID$ --serviceoutput="$SERVICEOUTPUT$" --longserviceoutput="$LONGSERVICEOUTPUT$" --datetime="$LONGDATETIME$"



My system was created using the OVA for VMWARE

Nagios XI - System Info
System:
Nagios XI Version : 5.4.11
cobra.csags.com.mx 2.6.32-696.13.2.el6.x86_64 x86_64
CentOS release 6.9 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Server Name: cobra.csags.com.mx
Server Address: 10.0.0.12
Server Port: 443
Date/Time
PHP Timezone: America/Mexico_City
PHP Time: Tue, 14 Nov 2017 18:20:19 -0600
System Time: Tue, 14 Nov 2017 18:20:19 -0600

how can I debug something more to find why no emails are generated?

thanks in advance

Jorge Arenas

[dwasswa]

Hi @jorgeaaq

What methods are you using for sending email; SMTP or Sendmail?

Please send a snapshot of your Admin mail settings by going to Admin >> Mail Settings.

Example:

Admin_Mail_settings.PNG

Secondly, please attach a file of your phpmailer logs located at /usr/local/nagiosxi/tmp/phpmailer.log

[jorgeaaq]

Attached you will find the image of Mail Settings

there is no file with such name in that path ( that maybe part of the problem?) or any other path ( i search all the system for that name)

Thanks in advance

Jorge

[jorgeaaq]

ok

after send a test email and enable el phpmailer.log
this is the content

[root@cobra tmp]# more phpmailer.log
[11-16-2017 11:03:47] Message sent! (method=sendmail), Referer: admin/testemail.php


i restart the nagios and wait to some notifications generated in the nagios.log ... and this file remains intact ... no logs , only the test

Thanks in advance

[dwasswa]

Hi @jorgeaaq,

You need to double check your notification or alert settings.

To do that, please follow the steps below:

Step 1: Check user e.g nagiosadmin Notification Methods.

Go to nagiosadmin >> Notification Methods
Make sure Receive alerts via email is checked.

Notification_methods.PNG

Step 2:Check contact settings

Go to CCM (Core Config Manager) >> Contacts
Click on Contact Name

Ccm_contacts.PNG

On the Contact Management page, click on Alert Settings


Go to Manage Service Notification Commands and Manage Host Notification Commands
Make sure you have xi_service_notification_handler for service and xi_host_notification_handler for host are Assigned.
Example:

xi_service_notification_handler.PNG

Please double check those settings and let me know if that solves your issue.

[jorgeaaq]

Hi:

attached you will find the screenshots everything is correct

on /usr/local/nagios/var/nagios.log

appear logs of the call for xi_service_notification_handler

but no emails are generated and /var/log/maillog not show any information and phpmailer.log only show the test email calls


1510862326] SERVICE NOTIFICATION: nagiosadmin;CiscoPBX;Gi0/1 Status;CRITICAL;xi_service_notification_handler;CRITICAL: Interface GigabitEthernet0/1 (index 5) is down.
[1510862366] SERVICE NOTIFICATION: nagiosadmin;mckinley;Windows Services;CRITICAL;xi_service_notification_handler;CRITICAL: TrustedInstaller=stopped (auto), delayed ()
[1510862389] SERVICE NOTIFICATION: nagiosadmin;puma;Yum Updates;WARNING;xi_service_notification_handler;YUM WARNING: O/S requires an update.
[1510862641] SERVICE NOTIFICATION: nagiosadmin;kibo;Windows Services;CRITICAL;xi_service_notification_handler;CRITICAL: TrustedInstaller=stopped (auto), delayed ()
[1510862799] SERVICE NOTIFICATION: nagiosadmin;leon;Check Mail Queue;WARNING;xi_service_notification_handler;WARNING: mailq is 96 (threshold w = 95)
[1510862922] SERVICE NOTIFICATION: nagiosadmin;CiscoPBX;Gi0/1 Status;CRITICAL;xi_service_notification_handler;CRITICAL: Interface GigabitEthernet0/1 (index 5) is down.
[1510863519] SERVICE NOTIFICATION: nagiosadmin;CiscoPBX;Gi0/1 Status;CRITICAL;xi_service_notification_handler;CRITICAL: Interface GigabitEthernet0/1 (index 5) is down.

[dwasswa]

Hi @jorgeaaq,

Please attach your System Profile by following the steps below:

Go to Admin >> Sytem Profile >> Download Profile

Download_system_profile.PNG

Secondly, please run the following command...tail -f /usr/local/nagiosxi/var/eventman.log

This command can be used to watch the logs related to events and notifications in Nagios XI.

As you run that command, force an immediate check by clicking on service that is in an Warning or Critical state, then watch the logs and post the output.

See steps below:

Step 1: Establish terminal session and run command tail -f /usr/local/nagiosxi/var/eventman.log


Step 2: Go to Service Detail menu and click on a Warning or Critical service.

Select_service.PNG

Step 3: On the Service Status Detail page for that service, click on Force an immediate check to the right under the Quick Actions options.

Force_check.PNG

Step 4: Watch logs and post the output.

[jorgeaaq]

Attached you will find profile


this is the output of tail to eventman.log while I was forcing the check of all warning and critical monitors


[root@cobra ~]# tail -f /usr/local/nagiosxi/var/eventman.log
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS
.....................

[npolovenko]

@jorgeaaq, Please connect to your Nagios server via ssh and run the following command:

Code: Select all

mailq

Does that show any queued up mail? If yes, please run the following command to flush it:

Code: Select all

sudo /usr/sbin/postsuper -d ALL

Please make sure that the queue is empty now. When done please run:

Code: Select all

service nagios restart

and

Code: Select all

service httpd restart

let us know if that helped.

[jorgeaaq]

Hi:

the return of every command was

[root@cobra ~]# mailq
Mail queue is empty

[root@cobra ~]# postsuper -d ALL

[root@cobra ~]# service nagios restart
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.

[root@cobra ~]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]


nothing change after this, I even restart the VM completely and still no emails generated