Page 1 of 1
All notifications not being sent!!!
Posted: Fri Oct 28, 2016 8:25 am
by BanditBBS
This is a huge issue! I only noticed this once we went to 5.3.0, thought it may have been a fluke, but more instances of it were caught last night and we are on 5.3.1 now.
XI shows notifications going out to our servicedesk(sd) but they are not received by the sd. The are constantly monitoring the notifications report and comparing to their inbox and last night had some say sent but not show up. sd then emails me and lets me know. I then do my research, I have mail debug on and no errors around that time and no "Message Sent" at the time either, so of course nothing in the maillog file either. Its like phpmailer just isn't being called, like the notification handler is either not being called or not running properly! There is absolutely no pattern to this that I can find.
HELP!
Re: All notifications not being sent!!!
Posted: Fri Oct 28, 2016 9:30 am
by tmcdonald
Definitely aware of this, compiling a list of common tickets and threads right now since not everyone is having this issue. Devs think they know what the issue is, likely fix on the way. Stay tuned.
Re: All notifications not being sent!!!
Posted: Fri Oct 28, 2016 9:45 am
by jomann
Are all notifications or just some notifications for certain services in particular? Can you send us your eventman.log with at least a bit of data in it - to see if this is related to the other issues or not.
Re: All notifications not being sent!!!
Posted: Fri Oct 28, 2016 10:03 am
by BanditBBS
It is not all, it is completely random.
Code: Select all
[jclark@iss-chi-nag05 nagios]$ cat /usr/local/nagiosxi/var/eventman.log
.......PROCESS EVENT: ID=7410712, SOURCE=2, TYPE=1, TIME=2016-10-28 10:02:17
*** GLOBAL HANDLER...
Array
(
[event_id] => 7410712
[event_source] => 2
[event_type] => 1
[event_time] => 2016-10-28 10:02:17
[event_meta] => Array
(
[handler-type] => service
[host] => rvbd-chi-dap02
[service] => Zombie Processes
[hostaddress] => 10.160.6.23
[hoststate] => UP
[hoststateid] => 0
[hosteventid] => 3092758
[hostproblemid] => 0
[servicestate] => OK
[servicestateid] => 0
[lastservicestate] => WARNING
[lastservicestateid] => 1
[servicestatetype] => SOFT
[currentattempt] => 2
[maxattempts] => 3
[serviceeventid] => 3629066
[serviceproblemid] => 0
[serviceoutput] => PROCS OK: 0 processes with STATE = Z
[longserviceoutput] =>
[servicedowntime] => 0
)
[logging_enabled] => 1
)
*** GLOBAL HANDLER (snmptrapsender)...
Array
(
[event_id] => 7410712
[event_source] => 2
[event_type] => 1
[event_time] => 2016-10-28 10:02:17
[event_meta] => Array
(
[handler-type] => service
[host] => rvbd-chi-dap02
[service] => Zombie Processes
[hostaddress] => 10.160.6.23
[hoststate] => UP
[hoststateid] => 0
[hosteventid] => 3092758
[hostproblemid] => 0
[servicestate] => OK
[servicestateid] => 0
[lastservicestate] => WARNING
[lastservicestateid] => 1
[servicestatetype] => SOFT
[currentattempt] => 2
[maxattempts] => 3
[serviceeventid] => 3629066
[serviceproblemid] => 0
[serviceoutput] => PROCS OK: 0 processes with STATE = Z
[longserviceoutput] =>
[servicedowntime] => 0
)
[logging_enabled] => 1
)
SNMP TRAP SENDER NOT CONFIGURED!
.
PROCESSED 1 EVENTS
That is the entire contents of the log
Re: All notifications not being sent!!!
Posted: Fri Oct 28, 2016 10:05 am
by jomann
Ah don't see the error in that log but it's like the cmdsubsys.log and it truncates every minute. Since it's random though it's likely the same issue that we saw in other systems. I'll let Trevor continue to manage this, I believe the fix should resolve this.
Re: All notifications not being sent!!!
Posted: Thu Nov 03, 2016 12:13 pm
by WillemDH
I believe the fix should resolve this
Has this issue been identified and resolved in 5.3.2?
Re: All notifications not being sent!!!
Posted: Thu Nov 03, 2016 12:16 pm
by mcapra
Essentially, events with special characters in their status information were not correctly being serialized when inserted into the database. I'm unsure if this is the exact issue you were facing, but it was an issue that affected several user's notifications and was fixed in 5.3.2.