High Load with Global event handlers

[niebais]

Centos 6 64 bit
Nagios XI 2.7 2014

Ok, I found an issue dealing with the GLOBAL EVENT HANDLERS that are set in the nagios.cfg. Everything works fine until you get about 100-1000 errors on one system and then the load jumps tremendously high.

I'm asking this question about the handlers. 1) Can we disable them safely? 2) What are the implications of turning those off?

The problem with the process is that it spins off a PHP process for every single soft error that occurs on the system. It really does add up on even medium large systems. On an all passive system this is a problem as well because of the load it generates.

[ssax]

The short answer would be no, unless you aren't using XI user notifications or some of the other examples from below.

You CAN turn them off, however, ALL of the items that rely on these hooks will no longer work, some examples are:

XI User Email Notification (pretty important)
Global Event Handlers
SNMP Trap Senders
Nagios IM Component
Nagios Reactor Component

To Disable, comment the following in nagios.cfg ( not recommended )

global_host_event_handler=xi_host_event_handler
global_service_event_handler=xi_service_event_handler

A better solution would be to tail the /usr/local/nagiosxi/var/eventman.log to see what is all being called, maybe you have one of the above components configured that you do not need.

[Box293]

I've observed this behaviour. I estimate that approx 30MB of memory is used per instance. So that roughly means:

100 errors = about 3 GB memory

Thats assuming 100 errors are all happening at the same time.

So it's good to have plenty of spare memory available in your Nagios XI host for when things go bad.

It's best to avoid these from being triggered in the first place. Implement service dependencies which will prevent checks from being executed and hence failing.

Alternatively, look at this pending feature request:
http://support.nagios.com/forum/viewtop ... 40#p140739

It will really resolve the problem as once the host goes down the service checks won't execute as their schedule keeps getting pushed back.

[niebais]

The problem we are having is we have plenty of memory, but when you get 1000 php processes all kicking off at the same time with these large systems it causes a really big problem and would crash any system these days. I need to shut that functionality off to work with the larger systems.

[niebais]

Thanks Box, that thread has some good stuff in it for these issues as well.

[tmcdonald]

If you need any help with that patch, lemme know - I wrote it :D