Page 1 of 2
notifications not going out
Posted: Wed Aug 16, 2017 2:51 pm
by jvestrum
Our host and service notification emails seem to have stopped working, as of the latest upgrade to XI 5.4.8 (possibly before that). When I check the notifications log in the web interface, it claims to have sent a notification at 14:14 today, but the email never went through (and nothing in /var/log/maillog indicating an attempt). Here's what I see in /usr/local/nagios/var/nagios.log:
Code: Select all
[1502909080] SERVICE NOTIFICATION: a5cltzz;usatcad1.mmm.com;/tsd Disk Usage;WARNING;xi_service_notification_handler;DISK WARNING - free space: /tsd 4710 MB (5% inode=99%):
[1502909080] SERVICE NOTIFICATION: us264366;usatcad1.mmm.com;/tsd Disk Usage;WARNING;xi_service_notification_handler;DISK WARNING - free space: /tsd 4710 MB (5% inode=99%):
[1502909080] SERVICE NOTIFICATION: us268072;usatcad1.mmm.com;/tsd Disk Usage;WARNING;xi_service_notification_handler;DISK WARNING - free space: /tsd 4710 MB (5% inode=99%):
[1502909080] wproc: NOTIFY job 433251 from worker Core Worker 21341 is a non-check helper but exited with return code 255
[1502909080] wproc: host=usatcad1.mmm.com; service=/tsd Disk Usage; contact=a5cltzz
[1502909080] wproc: early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1502909080] wproc: stderr line 01: No entry for terminal type "unknown";
[1502909080] wproc: stderr line 02: using dumb terminal settings.
[1502909080] wproc: NOTIFY job 433252 from worker Core Worker 21340 is a non-check helper but exited with return code 255
[1502909080] wproc: host=usatcad1.mmm.com; service=/tsd Disk Usage; contact=us268072
[1502909080] wproc: early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1502909080] wproc: stderr line 01: No entry for terminal type "unknown";
[1502909080] wproc: stderr line 02: using dumb terminal settings.
[1502909080] wproc: NOTIFY job 433252 from worker Core Worker 21342 is a non-check helper but exited with return code 255
[1502909080] wproc: host=usatcad1.mmm.com; service=/tsd Disk Usage; contact=us264366
[1502909080] wproc: early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1502909080] wproc: stderr line 01: No entry for terminal type "unknown";
[1502909080] wproc: stderr line 02: using dumb terminal settings.
Perhaps something going wrong in xi_service_notification_handler? I can successfully send a "Test Email" via the web interface.
Re: notifications not going out
Posted: Wed Aug 16, 2017 3:01 pm
by dwhitfield
Please post a screenshot of
http://YOURSERVER/nagiosxi/admin/mailsettings.php (replace YOURSERVER), also, please post your /usr/local/nagiosxi/tmp/phpmailer.log , if you have it.
For clarity, there are no individuals receiving email, correct? Did you check your spam?
I assume so, but if you go to
http://YOURSERVER/nagiosxi/admin/syssta ... ringengine are notifications on?
Can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.
After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.
If you can't send the entire profile, please post your commands.cfg file.
UPDATE: Profile shared with techs.
Re: notifications not going out
Posted: Wed Aug 16, 2017 3:25 pm
by jvestrum
I've attached the mail settings. It's set to Sendmail. There is no phpmailer.log.
None of the contacts are receiving email, including in spam folders.
Notifications are enabled.
I'll PM you the profile in a minute.
Re: notifications not going out
Posted: Wed Aug 16, 2017 3:49 pm
by jvestrum
To clarify, here is a screenshot of the Notifications log from the web gui. As you can see, it claims to have sent several notifications after 15:00 today. However the most recent line in my /var/log/maillog is at 13:44:
Aug 16 13:44:32 gtcs-nxi01 sendmail[13625]: v7GIiWhO013623: to=<
[email protected]>, delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=121032, relay=mailserv.mmm.com [192.28.32.144], dsn=2.0.0, stat=Sent (ok: Message 828818127 accepted)
Which corresponds to the last time I clicked "Send a Test Email" in the Mail Settings screen (and I did receive the test email).
Re: notifications not going out
Posted: Wed Aug 16, 2017 3:52 pm
by mcapra
I would double-check the settings mentioned under the "User Notification Configuration" section (Page 2 into Page 3) of this document:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
Making sure both "Enable Notifications" is checked, as well as the specific types of alerts you'd like to receive emails for.
Re: notifications not going out
Posted: Wed Aug 16, 2017 4:01 pm
by jvestrum
mcapra wrote:I would double-check the settings mentioned under the "User Notification Configuration" section (Page 2 into Page 3) of this document:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
Making sure both "Enable Notifications" is checked, as well as the specific types of alerts you'd like to receive emails for.
I checked, and all the settings are right, Enable Notifications is checked, all the types are checked except Host/Service Downtime, time periods are all 00:00 - 24:00, email address is correct.
Re: notifications not going out
Posted: Thu Aug 17, 2017 9:38 am
by dwhitfield
You have a couple of things going on in your profile. I am hoping the below will take care of the issue at hand. It also looks like your perfdata (graphs) are not working. If you haven't started a thread about that, I would check to see if your graphs are working and if not post a separate thread.
Regarding the instructions below, if you do not have killall, you can install it via the following command:
# yum install psmisc
If psmisc is not in your repos, then instead you can check to make sure nagios is not running with
# ps -aef | grep nagios
If that document does not resolve your issue, please run the following commands in order and report any errors. You ***must*** use mariadb instead of mysqld in the commands below, ***if*** you have mariadb.
# service nagios stop
# service ndo2db stop
# service mysqld stop
# service crond stop
# service httpd stop
# killall -9 nagios
# killall -9 ndo2db
# rm -rf /usr/local/nagios/var/rw/nagios.cmd
# rm -rf /usr/local/nagios/var/nagios.lock
# rm -f /usr/local/nagios/var/ndo.sock
# rm -f /usr/local/nagios/var/ndo2db.lock
# rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
# for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
# service ndo2db start
# service nagios start
# service mysqld start
# service crond start
# service httpd start
Re: notifications not going out
Posted: Thu Aug 17, 2017 9:59 am
by jvestrum
I ran those command and there were no errors (output is below). However the behaviour is the same, still no notifications. You are correct, the graphs are broken too.
Code: Select all
[root@gtcs-nxi01 ~]# serivice nagios stop
Stopping nagios:. done.
[root@gtcs-nxi01 ~]# service ndo2db stop
Stopping ndo2db: done.
[root@gtcs-nxi01 ~]# service mysqld stop
Stopping mysqld: [ OK ]
[root@gtcs-nxi01 ~]# service crond stop
Stopping crond: [ OK ]
[root@gtcs-nxi01 ~]# service httpd stop
Stopping httpd: [ OK ]
[root@gtcs-nxi01 ~]# killall -9 nagios
nagios: no process killed
[root@gtcs-nxi01 ~]# killall -9 ndo2db
ndo2db: no process killed
[root@gtcs-nxi01 ~]# rm -rf /usr/local/nagios/var/rw/nagios.cmd
[root@gtcs-nxi01 ~]# rm -rf /usr/local/nagios/var/nagios.lock
[root@gtcs-nxi01 ~]# rm -f /usr/local/nagios/var/ndo.sock
[root@gtcs-nxi01 ~]# rm -f /usr/local/nagios/var/ndo2db.lock
[root@gtcs-nxi01 ~]# rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
[root@gtcs-nxi01 ~]# for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
[root@gtcs-nxi01 ~]# service ndo2db start
Starting ndo2db: done.
[root@gtcs-nxi01 ~]# service nagios start
Starting nagios: done.
[root@gtcs-nxi01 ~]# service mysqld start
Starting mysqld: [ OK ]
[root@gtcs-nxi01 ~]# service crond start
Starting crond: [ OK ]
[root@gtcs-nxi01 ~]# service httpd start
Starting httpd: [ OK ]
Re: notifications not going out
Posted: Thu Aug 17, 2017 1:03 pm
by tgriep
When XI was updated on your system, it should of updated 2 entries in the /usr/local/nagiosxi/html/config.inc.php file and your system, those updates are missing.
Can you edit that file and change these 2 lines from
Code: Select all
require_once(dirname(__FILE__) . '/includes/db.inc.php');
require_once(dirname(__FILE__) . '/db/common.inc.php');
to
Code: Select all
if (!defined('CFG_ONLY')) { require_once(dirname(__FILE__) . '/includes/db.inc.php'); }
if (!defined('CFG_ONLY')) { require_once(dirname(__FILE__) . '/db/common.inc.php'); }
Save the file and restart the following processes
Code: Select all
service nagios restart
service apache restart
Then see if the email notifications are sent the next time they are scheduled.
Re: notifications not going out
Posted: Thu Aug 17, 2017 1:15 pm
by jvestrum
That worked! Thanks.