notifications not going out

[jvestrum]

Our host and service notification emails seem to have stopped working, as of the latest upgrade to XI 5.4.8 (possibly before that). When I check the notifications log in the web interface, it claims to have sent a notification at 14:14 today, but the email never went through (and nothing in /var/log/maillog indicating an attempt). Here's what I see in /usr/local/nagios/var/nagios.log:

Code: Select all

[1502909080] SERVICE NOTIFICATION: a5cltzz;usatcad1.mmm.com;/tsd Disk Usage;WARNING;xi_service_notification_handler;DISK WARNING - free space: /tsd 4710 MB (5% inode=99%):
[1502909080] SERVICE NOTIFICATION: us264366;usatcad1.mmm.com;/tsd Disk Usage;WARNING;xi_service_notification_handler;DISK WARNING - free space: /tsd 4710 MB (5% inode=99%):
[1502909080] SERVICE NOTIFICATION: us268072;usatcad1.mmm.com;/tsd Disk Usage;WARNING;xi_service_notification_handler;DISK WARNING - free space: /tsd 4710 MB (5% inode=99%):
[1502909080] wproc: NOTIFY job 433251 from worker Core Worker 21341 is a non-check helper but exited with return code 255
[1502909080] wproc:   host=usatcad1.mmm.com; service=/tsd Disk Usage; contact=a5cltzz
[1502909080] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1502909080] wproc:   stderr line 01: No entry for terminal type "unknown";
[1502909080] wproc:   stderr line 02: using dumb terminal settings.
[1502909080] wproc: NOTIFY job 433252 from worker Core Worker 21340 is a non-check helper but exited with return code 255
[1502909080] wproc:   host=usatcad1.mmm.com; service=/tsd Disk Usage; contact=us268072
[1502909080] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1502909080] wproc:   stderr line 01: No entry for terminal type "unknown";
[1502909080] wproc:   stderr line 02: using dumb terminal settings.
[1502909080] wproc: NOTIFY job 433252 from worker Core Worker 21342 is a non-check helper but exited with return code 255
[1502909080] wproc:   host=usatcad1.mmm.com; service=/tsd Disk Usage; contact=us264366
[1502909080] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1502909080] wproc:   stderr line 01: No entry for terminal type "unknown";
[1502909080] wproc:   stderr line 02: using dumb terminal settings.

Perhaps something going wrong in xi_service_notification_handler? I can successfully send a "Test Email" via the web interface.

[dwhitfield]

Please post a screenshot of http://YOURSERVER/nagiosxi/admin/mailsettings.php (replace YOURSERVER), also, please post your /usr/local/nagiosxi/tmp/phpmailer.log , if you have it.

For clarity, there are no individuals receiving email, correct? Did you check your spam?

I assume so, but if you go to http://YOURSERVER/nagiosxi/admin/syssta ... ringengine are notifications on?

Can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

If you can't send the entire profile, please post your commands.cfg file.

UPDATE: Profile shared with techs.

[jvestrum]

I've attached the mail settings. It's set to Sendmail. There is no phpmailer.log.

None of the contacts are receiving email, including in spam folders.

Notifications are enabled.

I'll PM you the profile in a minute.

[jvestrum]

To clarify, here is a screenshot of the Notifications log from the web gui. As you can see, it claims to have sent several notifications after 15:00 today. However the most recent line in my /var/log/maillog is at 13:44:

Aug 16 13:44:32 gtcs-nxi01 sendmail[13625]: v7GIiWhO013623: to=<[email protected]>, delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=121032, relay=mailserv.mmm.com [192.28.32.144], dsn=2.0.0, stat=Sent (ok: Message 828818127 accepted)

Which corresponds to the last time I clicked "Send a Test Email" in the Mail Settings screen (and I did receive the test email).

[mcapra]

I would double-check the settings mentioned under the "User Notification Configuration" section (Page 2 into Page 3) of this document:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Making sure both "Enable Notifications" is checked, as well as the specific types of alerts you'd like to receive emails for.

[jvestrum]

I would double-check the settings mentioned under the "User Notification Configuration" section (Page 2 into Page 3) of this document:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Making sure both "Enable Notifications" is checked, as well as the specific types of alerts you'd like to receive emails for.

I checked, and all the settings are right, Enable Notifications is checked, all the types are checked except Host/Service Downtime, time periods are all 00:00 - 24:00, email address is correct.

[dwhitfield]

You have a couple of things going on in your profile. I am hoping the below will take care of the issue at hand. It also looks like your perfdata (graphs) are not working. If you haven't started a thread about that, I would check to see if your graphs are working and if not post a separate thread.

Regarding the instructions below, if you do not have killall, you can install it via the following command:
# yum install psmisc

If psmisc is not in your repos, then instead you can check to make sure nagios is not running with
# ps -aef | grep nagios

If that document does not resolve your issue, please run the following commands in order and report any errors. You ***must*** use mariadb instead of mysqld in the commands below, ***if*** you have mariadb.
# service nagios stop
# service ndo2db stop
# service mysqld stop
# service crond stop
# service httpd stop
# killall -9 nagios
# killall -9 ndo2db
# rm -rf /usr/local/nagios/var/rw/nagios.cmd
# rm -rf /usr/local/nagios/var/nagios.lock
# rm -f /usr/local/nagios/var/ndo.sock
# rm -f /usr/local/nagios/var/ndo2db.lock
# rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
# for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
# service ndo2db start
# service nagios start
# service mysqld start
# service crond start
# service httpd start

[jvestrum]

I ran those command and there were no errors (output is below). However the behaviour is the same, still no notifications. You are correct, the graphs are broken too.

Code: Select all

[root@gtcs-nxi01 ~]# serivice nagios stop
Stopping nagios:. done.
[root@gtcs-nxi01 ~]# service ndo2db stop
Stopping ndo2db: done.
[root@gtcs-nxi01 ~]# service mysqld stop
Stopping mysqld:                                           [  OK  ]
[root@gtcs-nxi01 ~]# service crond stop
Stopping crond:                                            [  OK  ]
[root@gtcs-nxi01 ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@gtcs-nxi01 ~]# killall -9 nagios
nagios: no process killed
[root@gtcs-nxi01 ~]# killall -9 ndo2db
ndo2db: no process killed
[root@gtcs-nxi01 ~]# rm -rf /usr/local/nagios/var/rw/nagios.cmd
[root@gtcs-nxi01 ~]# rm -rf /usr/local/nagios/var/nagios.lock
[root@gtcs-nxi01 ~]# rm -f /usr/local/nagios/var/ndo.sock
[root@gtcs-nxi01 ~]# rm -f /usr/local/nagios/var/ndo2db.lock
[root@gtcs-nxi01 ~]# rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
[root@gtcs-nxi01 ~]# for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
[root@gtcs-nxi01 ~]# service ndo2db start
Starting ndo2db: done.
[root@gtcs-nxi01 ~]# service nagios start
Starting nagios: done.
[root@gtcs-nxi01 ~]# service mysqld start
Starting mysqld:                                           [  OK  ]
[root@gtcs-nxi01 ~]# service crond start
Starting crond:                                            [  OK  ]
[root@gtcs-nxi01 ~]# service httpd start
Starting httpd:                                            [  OK  ]

[tgriep]

When XI was updated on your system, it should of updated 2 entries in the /usr/local/nagiosxi/html/config.inc.php file and your system, those updates are missing.
Can you edit that file and change these 2 lines from

Code: Select all

require_once(dirname(__FILE__) . '/includes/db.inc.php');
require_once(dirname(__FILE__) . '/db/common.inc.php');

to

Code: Select all

if (!defined('CFG_ONLY')) { require_once(dirname(__FILE__) . '/includes/db.inc.php'); }
if (!defined('CFG_ONLY')) { require_once(dirname(__FILE__) . '/db/common.inc.php'); }

Save the file and restart the following processes

Code: Select all

service nagios restart
service apache restart

Then see if the email notifications are sent the next time they are scheduled.

[jvestrum]

That worked! Thanks.