Flooding - /usr/local/nagiosxi/var/event_handler.log

[saber]

Hi,

On the web interface, all the hosts are up. However we keep being flooded by

Notifications Email: "PROBLEM Host Alert.. Info=> Host check timed out after 30.00 seconds"

and the Date/Time of those notifications is the current one.

The file /usr/local/nagiosxi/var/event_handler.log

[0] => XXXX
[eventqueue_id] => XXXX
[1] => XXXXXX
[event_time] => XXXX
[2] => X
[event_source] => X
[3] => X
[event_type] => X
[4] => XXXXXX

many times... so much that the disk space gets to 100%!

The following issue happened after we did a network maintenance Sunday and the NagiosXI server did not have network for about 15mins..

Do you know what can cause a such issue?

Thank you!

[dwhitfield]

You'll likely want to run through https://support.nagios.com/kb/article.php?id=26 with diskspace being 100%

There's a link to https://support.nagios.com/kb/article.php?id=266 from the other article, but it seems worth mentioning directly.

There was a bug in the 5.3.x series that could potentially cause something like this. What version of XI are you running? (always a good piece of information to include in your first post)

What is the check time in the web UI saying the hosts are up? Do all of your time zones match (php, hwclock, data, mysql, UI)?

Can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the Download Profile button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

[saber]

Thank you for the fast answer.

We are using the latest version which is 5.4.4. And it's a dedicated server. We had event_handler.log consuming 50GB+.

Can you please PM your email, I don't feel confortable to send the logs on a forums (even with via a PM).

Best Regards,

[dwhitfield]

What is the check time in the web UI saying the hosts are up? Do all of your time zones match (php, hwclock, date, mysql, UI)? The reason I ask is I've seen a lot of issues with time and RHEV. hwclock and date should be a quick check to make sure the VM is not causing a time issue, but PHP, MySQL, and the Web UI should all be checked too.

The time information won't be in the profile, for the most part.

I will PM you.

UPDATE: profile received and shared with techs

[saber]

Thanks, hopefully we will find a solution, it really looks like a bug!

[tacolover101]

is the data you're seeing in the event_handler valid events that happen? do you have event handlers / global event handlers configured on a lot of devices?

just looking at things from an outside view, if it's not a bug after all.

[saber]

is the data you're seeing in the event_handler valid events that happen? do you have event handlers / global event handlers configured on a lot of devices?

just looking at things from an outside view, if it's not a bug after all.

In eventmap.log I keep seeing

[event_id] => XXXX
[event_source] => 2
[event_type] => 1
[event_time] => 2017-05-01 17:11:33
[event_meta] => Array
(
[handler-type] => host
[host] => XXXX
[hostaddress] => XXXX
[hoststate] => UP
[hoststateid] => 0
[lasthoststate] => DOWN
[lasthoststateid] => 1
[hoststatetype] => SOFT
[currentattempt] => 5
[maxattempts] => 5
[hosteventid] => XXXX
[hostproblemid] => 0
[hostoutput] => OK - XXXX: rta 84.808ms, lost 0%
[longhostoutput] =>
[hostdowntime] => 0
)

[logging_enabled] => 1


In a loop! It never ends and the same exact alerts (which are 24/48 hours old!) keep happening and happening!

Thank you,

[saber]

Also we are not using any special config on NagiosXI, it is mostly the defaults.. no handlers / global event..

[saber]

I even deleted those two inactive components

"SNMP Trap Sender"
"Global Event Handlers"

But the eventmap.log is now flooded by

"PROCESS EVENT: ID=XXXXX, SOURCE=2, TYPE=1, TIME=2017-05-01 00:57:25"

[dwhitfield]

What is the output of the following commands:

Code: Select all

mysql -uroot -pnagiosxi -e 'SELECT @@global.time_zone, @@session.time_zone;'
hwclock
service ntpd status

What does the UI under Admin --> System Settings say your time zone is?

What's a host that is sending emails? You certainly have plenty of host alerts in your nagios.log.