All notifications not being sent!!!

[BanditBBS]

This is a huge issue! I only noticed this once we went to 5.3.0, thought it may have been a fluke, but more instances of it were caught last night and we are on 5.3.1 now.

XI shows notifications going out to our servicedesk(sd) but they are not received by the sd. The are constantly monitoring the notifications report and comparing to their inbox and last night had some say sent but not show up. sd then emails me and lets me know. I then do my research, I have mail debug on and no errors around that time and no "Message Sent" at the time either, so of course nothing in the maillog file either. Its like phpmailer just isn't being called, like the notification handler is either not being called or not running properly! There is absolutely no pattern to this that I can find.

HELP!

[tmcdonald]

Definitely aware of this, compiling a list of common tickets and threads right now since not everyone is having this issue. Devs think they know what the issue is, likely fix on the way. Stay tuned.

[jomann]

Are all notifications or just some notifications for certain services in particular? Can you send us your eventman.log with at least a bit of data in it - to see if this is related to the other issues or not.

[BanditBBS]

It is not all, it is completely random.

Code: Select all

[jclark@iss-chi-nag05 nagios]$ cat /usr/local/nagiosxi/var/eventman.log
.......PROCESS EVENT: ID=7410712, SOURCE=2, TYPE=1, TIME=2016-10-28 10:02:17
*** GLOBAL HANDLER...
Array
(
    [event_id] => 7410712
    [event_source] => 2
    [event_type] => 1
    [event_time] => 2016-10-28 10:02:17
    [event_meta] => Array
        (
            [handler-type] => service
            [host] => rvbd-chi-dap02
            [service] => Zombie Processes
            [hostaddress] => 10.160.6.23
            [hoststate] => UP
            [hoststateid] => 0
            [hosteventid] => 3092758
            [hostproblemid] => 0
            [servicestate] => OK
            [servicestateid] => 0
            [lastservicestate] => WARNING
            [lastservicestateid] => 1
            [servicestatetype] => SOFT
            [currentattempt] => 2
            [maxattempts] => 3
            [serviceeventid] => 3629066
            [serviceproblemid] => 0
            [serviceoutput] => PROCS OK: 0 processes with STATE = Z
            [longserviceoutput] =>
            [servicedowntime] => 0
        )

    [logging_enabled] => 1
)
*** GLOBAL HANDLER (snmptrapsender)...
Array
(
    [event_id] => 7410712
    [event_source] => 2
    [event_type] => 1
    [event_time] => 2016-10-28 10:02:17
    [event_meta] => Array
        (
            [handler-type] => service
            [host] => rvbd-chi-dap02
            [service] => Zombie Processes
            [hostaddress] => 10.160.6.23
            [hoststate] => UP
            [hoststateid] => 0
            [hosteventid] => 3092758
            [hostproblemid] => 0
            [servicestate] => OK
            [servicestateid] => 0
            [lastservicestate] => WARNING
            [lastservicestateid] => 1
            [servicestatetype] => SOFT
            [currentattempt] => 2
            [maxattempts] => 3
            [serviceeventid] => 3629066
            [serviceproblemid] => 0
            [serviceoutput] => PROCS OK: 0 processes with STATE = Z
            [longserviceoutput] =>
            [servicedowntime] => 0
        )

    [logging_enabled] => 1
)
SNMP TRAP SENDER NOT CONFIGURED!
.
PROCESSED 1 EVENTS

That is the entire contents of the log

[jomann]

Ah don't see the error in that log but it's like the cmdsubsys.log and it truncates every minute. Since it's random though it's likely the same issue that we saw in other systems. I'll let Trevor continue to manage this, I believe the fix should resolve this.

[WillemDH]

I believe the fix should resolve this

Has this issue been identified and resolved in 5.3.2?

[mcapra]

Essentially, events with special characters in their status information were not correctly being serialized when inserted into the database. I'm unsure if this is the exact issue you were facing, but it was an issue that affected several user's notifications and was fixed in 5.3.2.