Runtime Error for host and service notification

[valmberg]

Running Nagios XI 5.3.4
Have gone through the "How to Configure Email And Text Notifications" document.
Have tested a test email and successfully received it.

If I use the 'notify-host-by-email' or the 'notify-service-by-email' Notification Commands, the emails work. If I try to use the xi_host_notification_handler or the xi_service_notification_handler Notification Commands, the emails do not get sent.

The event log reports the following:

Code: Select all

[1481951226] SERVICE NOTIFICATION: bilo;Cenvrpprd1.oya.ad;Memory Usage;CUSTOM (OK);xi_service_notification_handler;OK - Physical Memory: Total: 24GB - Used: 9.813GB (41%) - Free: 14.187GB (59%);Vince A;test test test
[1481951226] SERVICE NOTIFICATION: garc;Cenvrpprd1.oya.ad;Memory Usage;CUSTOM (OK);xi_service_notification_handler;OK - Physical Memory: Total: 24GB - Used: 9.813GB (41%) - Free: 14.187GB (59%);Vince A;test test test
[1481951226] SERVICE NOTIFICATION: mosj;Cenvrpprd1.oya.ad;Memory Usage;CUSTOM (OK);xi_service_notification_handler;OK - Physical Memory: Total: 24GB - Used: 9.813GB (41%) - Free: 14.187GB (59%);Vince A;test test test
[1481951226] SERVICE NOTIFICATION: WieJ;Cenvrpprd1.oya.ad;Memory Usage;CUSTOM (OK);xi_service_notification_handler;OK - Physical Memory: Total: 24GB - Used: 9.813GB (41%) - Free: 14.187GB (59%);Vince A;test test test
[1481951226] SERVICE NOTIFICATION: harj;Cenvrpprd1.oya.ad;Memory Usage;CUSTOM (OK);xi_service_notification_handler;OK - Physical Memory: Total: 24GB - Used: 9.813GB (41%) - Free: 14.187GB (59%);Vince A;test test test
[1481951226] SERVICE NOTIFICATION: almv;Cenvrpprd1.oya.ad;Memory Usage;CUSTOM (OK);xi_service_notification_handler;OK - Physical Memory: Total: 24GB - Used: 9.813GB (41%) - Free: 14.187GB (59%);Vince A;test test test
[1481951226] SERVICE NOTIFICATION: nobj;Cenvrpprd1.oya.ad;Memory Usage;CUSTOM (OK);xi_service_notification_handler;OK - Physical Memory: Total: 24GB - Used: 9.813GB (41%) - Free: 14.187GB (59%);Vince A;test test test
[1481951226] SERVICE NOTIFICATION: sped;Cenvrpprd1.oya.ad;Memory Usage;CUSTOM (OK);xi_service_notification_handler;OK - Physical Memory: Total: 24GB - Used: 9.813GB (41%) - Free: 14.187GB (59%);Vince A;test test test
[1481951228] wproc: NOTIFY job 25 from worker Core Worker 9912 is a non-check helper but exited with return code 255
[1481951228] wproc:   host=Cenvrpprd1.oya.ad; service=Memory Usage; contact=garc
[1481951228] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1481951228] wproc:   stderr line 01: No entry for terminal type "unknown";
[1481951228] wproc:   stderr line 02: using dumb terminal settings.
[1481951228] wproc: NOTIFY job 27 from worker Core Worker 9911 is a non-check helper but exited with return code 255
[1481951228] wproc:   host=Cenvrpprd1.oya.ad; service=Memory Usage; contact=sped
[1481951228] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1481951228] wproc:   stderr line 01: No entry for terminal type "unknown";
[1481951228] wproc:   stderr line 02: using dumb terminal settings.
[1481951228] wproc: NOTIFY job 26 from worker Core Worker 9913 is a non-check helper but exited with return code 255
[1481951228] wproc:   host=Cenvrpprd1.oya.ad; service=Memory Usage; contact=mosj
[1481951228] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1481951228] wproc:   stderr line 01: No entry for terminal type "unknown";
[1481951228] wproc:   stderr line 02: using dumb terminal settings.
[1481951228] wproc: NOTIFY job 26 from worker Core Worker 9910 is a non-check helper but exited with return code 255
[1481951228] wproc:   host=Cenvrpprd1.oya.ad; service=Memory Usage; contact=harrj
[1481951228] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1481951228] wproc:   stderr line 01: No entry for terminal type "unknown";
[1481951228] wproc:   stderr line 02: using dumb terminal settings.
[1481951228] wproc: NOTIFY job 27 from worker Core Worker 9913 is a non-check helper but exited with return code 255
[1481951228] wproc:   host=Cenvrpprd1.oya.ad; service=Memory Usage; contact=nobj
[1481951228] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1481951228] wproc:   stderr line 01: No entry for terminal type "unknown";
[1481951228] wproc:   stderr line 02: using dumb terminal settings.
[1481951228] wproc: NOTIFY job 25 from worker Core Worker 9910 is a non-check helper but exited with return code 255
[1481951228] wproc:   host=Cenvrpprd1.oya.ad; service=Memory Usage; contact=bilo
[1481951228] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1481951228] wproc:   stderr line 01: No entry for terminal type "unknown";
[1481951228] wproc:   stderr line 02: using dumb terminal settings.
[1481951228] wproc: NOTIFY job 26 from worker Core Worker 9912 is a non-check helper but exited with return code 255
[1481951228] wproc:   host=Cenvrpprd1.oya.ad; service=Memory Usage; contact=almv
[1481951228] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1481951228] wproc:   stderr line 01: No entry for terminal type "unknown";
[1481951228] wproc:   stderr line 02: using dumb terminal settings.
[1481951228] wproc: NOTIFY job 26 from worker Core Worker 9911 is a non-check helper but exited with return code 255
[1481951228] wproc:   host=Cenvrpprd1.oya.ad; service=Memory Usage; contact=WieJ
[1481951228] wproc:   early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
[1481951228] wproc:   stderr line 01: No entry for terminal type "unknown";
[1481951228] wproc:   stderr line 02: using dumb terminal settings.

From the Eventman.log file:

PHP Notice: Undefined index: nagiosxi in /usr/local/nagiosxi/html/includes/db.inc.php on line 488
PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosim/nagiosim.inc.php on line 489
PHP Notice: Undefined index: nagiosxi in /usr/local/nagiosxi/html/includes/db.inc.php on line 488
PHP Notice: Undefined index: nagiosxi in /usr/local/nagiosxi/html/includes/db.inc.php on line 488
.................
PROCESSED 0 EVENTS

[rkennedy]

Can you please show us a screenshot of your 'Manage Email Settings' page? From there, you'll want to turn debugging on as well. This should help to see what's going on. Please post the related log file for the method you're using.

One more thing - does the person have a user account created, or are they only a contact?

[valmberg]

Here is a picture of the manage email settings.

Every person has a user account created. If I use the 'notify-host-by-email' or the 'notify-service-by-email' Notification Commands, the emails work.

I've had the debugging turned on. But the events don't get logged in the phpmailer.log. I'm thinking this is because the email event doesn't get triggered from the runtime event.

Here is the phpmailer.log showing only that Scheduled Reports are being mailed by the PHPmailer service.
[12-17-2016 09:00:37] Message sent! (method=smtp;host=cenvmsprd1.oya.ad;port=587;security=none), Referer: includes/components/scheduledreporting > Scheduled Report
[12-18-2016 09:00:17] Message sent! (method=smtp;host=cenvmsprd1.oya.ad;port=587;security=none), Referer: includes/components/scheduledreporting > Scheduled Report
[12-19-2016 09:00:07] Message sent! (method=smtp;host=cenvmsprd1.oya.ad;port=587;security=none), Referer: includes/components/scheduledreporting > Scheduled Report

[rkennedy]

When you log in as the user, navigate to the top right and click the username, from there look through the Notification Preferences / Notification Methods pages - is everything enabled here?

If that's not the culprit, it's possible SQL has crashed. (usually the only reason I've seen them not send) What happens if you repair SQL, does this have an affect? Just run /usr/local/nagiosxi/scripts/repair_databases.sh

[valmberg]

Yes. Everything is enabled there.

I ran the 'repair_databases.sh' script. It fixed some indexes and reported:

Code: Select all

===============
REPAIR COMPLETE
===============
Stopping ndo2db: done.
Starting ndo2db: done.
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.

=======================
nagios database repair succeeded
nagiosql database repair succeeded

However, i'm still not receiving the emails from the PHPmailer.

This keeps showing up in the event log:

Code: Select all

Runtime Error	2016-12-19 13:07:17	wproc: early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
Runtime Error	2016-12-19 13:07:17	wproc: host=Clackamas Parole; service=LAN Port Status; contact=almberv
Runtime Error	2016-12-19 13:07:17	wproc: NOTIFY job 177 from worker Core Worker 11801 is a non-check helper but exited with return code 255

[dwhitfield]

Can you post your nagios.cfg?

It seems like a bit of a long shot, but if macros are not enabled (in nagios.cfg) that could be the issue.

Also, while I do think you should enable them as a troubleshooting step, please take note:

This option determines whether or not the Nagios daemon will make all standard macros available as environment variables to your check, notification, event hander, etc. commands. In large Nagios installations this can be problematic because it takes additional memory and (more importantly) CPU to compute the values of all macros and make them available to the environment.

Based on your previous profiles, I'd say you have a medium install (on the small side of medium), but I only looked at the host number to come up with that, so if you have a lot of services, that could push you up. It's something to think about if you have performance issues and as you grow though.

[valmberg]

Are you talking about enable_environment_macros=0?

[dwhitfield]

Yes, sorry for the confusion. The large installation tweaks does disable some macros on its own, but not all of them. The large installation tweaks might end up being enough for you from a performance aspect. Let us know first if that fixes the mail issue and then please let us know if that looks like it will be a long term fix. Thanks!

[valmberg]

No it did not fix it. I changed "enable_environment_macros=1"

Still getting a runtime error when having having a service send a message:
Runtime Error 2016-12-19 15:06:38 wproc: stderr line 02: using dumb terminal settings.
Runtime Error 2016-12-19 15:06:38 wproc: stderr line 01: No entry for terminal type "unknown";
Runtime Error 2016-12-19 15:06:38 wproc: early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
Runtime Error 2016-12-19 15:06:38 wproc: host=Clackamas Parole; service=LAN Port Status; contact=almberv
Runtime Error 2016-12-19 15:06:38 wproc: NOTIFY job 11 from worker Core Worker 4958 is a non-check helper but exited with return code 255
Runtime Error 2016-12-19 15:06:38 wproc: stderr line 02: using dumb terminal settings.
Runtime Error 2016-12-19 15:06:38 wproc: stderr line 01: No entry for terminal type "unknown";
Runtime Error 2016-12-19 15:06:38 wproc: early_timeout=0; exited_ok=1; wait_status=65280; error_code=0;
Runtime Error 2016-12-19 15:06:38 wproc: host=Clackamas Parole; service=LAN Port Status; contact=almberv
Runtime Error 2016-12-19 15:06:38 wproc: NOTIFY job 12 from worker Core Worker 4961 is a non-check helper but exited with return code 255

[tgriep]

Can you post your /etc/sudoers file so we can view it?
The stderr lines in the log points to an issue for a setting in that file is missing.
Can you post the service configuration from one of the services that are not generating an email?
Also, run this on the server as root and post the output as well.

Code: Select all

ps -ef --cols-300

Thanks