event_handler.log out of control/mysqld hogging resources

[TCBeekley]

Hello,

I seem to be experiencing the same issues as mentioned in this topic https://support.nagios.com/forum/viewto ... 16&t=44440

event_handler.log is growing incredibly fast and mysqld is using all of the memory and CPU available on the machine. It looks to me like the log is trying to work through events from a week ago.

Code: Select all

    [eventqueue_id] => 668340
    [1] => 2018-02-21 18:00:25
    [event_time] => 2018-02-21 18:00:25
    [2] => 2
    [event_source] => 2
    [3] => 1
    [event_type] => 1

[tmcdonald]

Let's get a system profile. In the XI web interface, go to Admin -> System Profile and click the blue "Download Profile" button. Send that to me via Private Message.

Also, was there any outage about a week ago that might explain the backlog? Either on your XI machine directly or within the network.

Edit: Profile.txt received

[TCBeekley]

This is a "Staging" server where we build monitoring before moving it to production so there is a lot of unknowns and critical alerts while we get everything working correctly.

The profile download wouldn't populate, but I did copy & paste. It should be in your inbox.

Thanks!

[tgriep]

It could be a bad entry in one of the SQL tables causing them to backup up and not process.

Try following through this procedure to clear out the tables and hopefully fix the issue.
Run the following as root.

Code: Select all

service nagios stop
service ndo2db stop
service crond stop
pkill -9 -u nagios

If your server is using the Postgress database, you would run this

Code: Select all

echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | psql nagiosxi nagiosxi

If you are using MYSQL, you would run this

Code: Select all

echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi

Code: Select all

service crond start
service ndo2db start
service nagios start
service npcd restart

Give the system 15 to 20 minutes to run and see if the status starts to populate with the correct data.

[TCBeekley]

Looks like that did that trick!

Everything is back to normal.

Thanks!