Stopped receiving email notifications

[yeahMon]

Recently we had a lot of building maintenance going on, so i disabled notifications under process monitoring, but left everything else running. After re-enabling the notification process, we lo longer receive email notifications for any service or host. We are running Nagios 5.5.7 on redhat.

-email is done using sendmail
-test emails work fine
-i tried rebooting the nagios server which did not help
-i ran the database repair script which successfully made some repairs, but still no email notifications
-no changes to nagios other than stopping/starting the notification process
-/var/log/maillog does not seem to show any errors

Please let me if i can provide any other information.

[benjaminsmith]

Hello @yeahMon,

Is cron running on the server:

Code: Select all

systemctl status crond.service

If that checks out, please go to Admin > Manage Email Settings:
- Make sure Debug Log is checked
- Click the Update Settings button

Then run this tail command (and leave it running):

Code: Select all

tail -F /var/log/maillog /usr/local/nagiosxi/tmp/phpmailer.log /usr/local/nagiosxi/var/eventman.log

Then force a notification to be sent then send the the full output of the tail command above and any errors you see on the screen.

Thanks.

[yeahMon]

Thanks for the response.

I confirmed that crond is running, but i dont see any cron jobs listed for root or nagios. Nor do i see any jobs in /etc/crontab. Heres the log output.

/var/mailog:
0/1.2/0.09, dsn=2.0.0, status=sent (250 2.0.0 x3MEJHjG011291 Message accepted for delivery)
Apr 22 10:19:17 zeus postfix/qmgr[2951]: D65345B80564: removed
Apr 22 10:41:33 zeus postfix/pickup[7699]: A8DC65B80564: uid=48 from=<root@zeus>
Apr 22 10:41:33 zeus postfix/cleanup[18581]: A8DC65B80564: message-id=<[email protected]>
Apr 22 10:41:33 zeus postfix/qmgr[2951]: A8DC65B80564: from=<[email protected]>, size=924, nrcpt=1 (queue active)
Apr 22 10:41:34 zeus postfix/smtp[18584]: A8DC65B80564: to=<[email protected]>, relay=mailhost.xxxxxxx.com[135.5.2.65]:25, delay=1.3, delays=0.02/0.01/1.2/0.1, dsn=2.0.0, status=sent (250 2.0.0 x3MEfYD9031174 Message accepted for delivery)
Apr 22 10:41:34 zeus postfix/qmgr[2951]: A8DC65B80564: removed

eventman.log is full of of these these types of things:
)
PHP Notice: Undefined variable: settings in /usr/local/nagiosxi/html/includes/components/mtr/mtr.inc.php on line 529
*** GLOBAL HANDLER (snmptrapsender)...
Array
(
[event_id] => 3723024
[event_source] => 2
[event_type] => 1
[event_time] => 2019-04-13 22:27:02
[event_meta] => Array
(
[handler-type] => service
[host] => nspregr-compute-42-iLo
[service] => Storage
[hostaddress] => xxx.xxx.xxx.xxx
[hoststate] => DOWN
[hoststateid] => 1
[hosteventid] => 1576199
[hostproblemid] => 759978
[servicestate] => UNKNOWN
[servicestateid] => 3
[lastservicestate] => UNKNOWN
[lastservicestateid] => 3
[servicestatetype] => SOFT
[currentattempt] => 3
[maxattempts] => 3
[serviceeventid] => 1576985
[serviceproblemid] => 760754
[serviceoutput] => Timeout - Probable Cause might be SNMP Community string is incorrect or not configured.
[longserviceoutput] =>
[servicedowntime] => 0
)
[logging_enabled] => 1
)

[benjaminsmith]

Hi @yeahMon,

Apr 22 10:41:33 zeus postfix/qmgr[2951]: A8DC65B80564: from=<[email protected]>, size=924, nrcpt=1 (queue active)
Apr 22 10:41:34 zeus postfix/smtp[18584]: A8DC65B80564: to=<[email protected]>, relay=mailhost.xxxxxxx.com[135.5.2.65]:25, delay=1.3, delays=0.02/0.01/1.2/0.1, dsn=2.0.0, status=sent (250 2.0.0 x3MEfYD9031174 Message accepted for delivery)

Reviewing the log file above, it does look like the message was sent by postfix and accepted. Have you checked the spam/junk folder for the message as it might be getting blocked or filtered? This is often the case for email messages from [email protected].

If the message is not there, can you confirm the relay settings are correct and going to the right place. To further troubleshoot, can you upload the logs from your last post as it's helpful to view the entire evetman.log and send us your system profile.

Thanks.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket.

[yeahMon]

Hi Benjamin,

Nothing in junk mail. I have PM'd you my profile.zip. eventman log was 200MB and failed to upload.

I checked the mailer and it seems to be working fine. Through the operating system i can sendmail as well as through nagios test mail.

When i tail maillog and go pull a cable somewhere to trigger an event/notification, the maillog shows nothing at all. It looks like something is disconnected between receiving the event in nagios and nagios sending the notification.

I also notice that i can see notifications DO show up from home->notifications on bottom left panel. Just no emails.

[benjaminsmith]

Hi @yeahMon,

As you can see, there are definitely some database issues on the system.

/usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_systemcommands.MYI'; try to repair it

Run the following from the command to force a repair of the tables:

Code: Select all

mysqlcheck -r -f -uroot -pnagiosxi --all-databases	

Please post the output to the following command (it will show tables by size):

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table

[yeahMon]

Thanks Benjamin,

The repair looked OK, except for some of these statements:

nagiosxi.xi_auth_tokens
note : The storage engine for the table doesn't support repair
nagiosxi.xi_cmp_trapdata
note : The storage engine for the table doesn't support repair
nagiosxi.xi_cmp_trapdata_log
note : The storage engine for the table doesn't support repair

I have uploaded the output for the table sizes.

[benjaminsmith]

Hello @yeahMon,

One thing to check is to make sure all the Nagios XI subsystems are up and running (see image). Go to Admin > System Information > System Status

Next, the meta table is large, so run the following command from the terminal to truncate the tables:

Code: Select all

echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi

Then run the following command:

Code: Select all

tail -F /var/log/maillog /usr/local/nagiosxi/tmp/phpmailer.log /usr/local/nagiosxi/var/eventman.log

Now go to Home > Details > Host Status, then select a host and click on the advanced tab. Under the Commands tab, select Send Custom Notification, fill in the form and submit.

Do you receive the message? If no, are there any errors in the log output.

[yeahMon]

Hey Benjamin,

The process's are all running as per your screen cap.

I ran the commands to truncate the meta table and sent you the logs via PM.
I didn't see an obvious error, but mabey there's information you could find useful.

Still not receiving email notifications.

Thanks.

[yeahMon]

OK, this is weird.

I received the custom email notification at 9:17, about 20 minutes after i submitted the form!
I also received a couple notifications from a couple other monitored hosts that have been flapping.

But nothing after that. Even after pulling some cables to trigger alerts.

Test email continues to work as expected.