Page 1 of 1
event_handler.log out of control/mysqld hogging resources
Posted: Tue Feb 27, 2018 7:49 am
by TCBeekley
Hello,
I seem to be experiencing the same issues as mentioned in this topic
https://support.nagios.com/forum/viewto ... 16&t=44440
event_handler.log is growing incredibly fast and mysqld is using all of the memory and CPU available on the machine. It looks to me like the log is trying to work through events from a week ago.
Code: Select all
[eventqueue_id] => 668340
[1] => 2018-02-21 18:00:25
[event_time] => 2018-02-21 18:00:25
[2] => 2
[event_source] => 2
[3] => 1
[event_type] => 1
Re: event_handler.log out of control/mysqld hogging resource
Posted: Tue Feb 27, 2018 3:16 pm
by tmcdonald
Let's get a system profile. In the XI web interface, go to Admin -> System Profile and click the blue "Download Profile" button. Send that to me via Private Message.
Also, was there any outage about a week ago that might explain the backlog? Either on your XI machine directly or within the network.
Edit: Profile.txt received
Re: event_handler.log out of control/mysqld hogging resource
Posted: Wed Feb 28, 2018 7:24 am
by TCBeekley
This is a "Staging" server where we build monitoring before moving it to production so there is a lot of unknowns and critical alerts while we get everything working correctly.
The profile download wouldn't populate, but I did copy & paste. It should be in your inbox.
Thanks!
Re: event_handler.log out of control/mysqld hogging resource
Posted: Wed Feb 28, 2018 5:08 pm
by tgriep
It could be a bad entry in one of the SQL tables causing them to backup up and not process.
Try following through this procedure to clear out the tables and hopefully fix the issue.
Run the following as root.
Code: Select all
service nagios stop
service ndo2db stop
service crond stop
pkill -9 -u nagios
If your server is using the Postgress database, you would run this
Code: Select all
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | psql nagiosxi nagiosxi
If you are using MYSQL, you would run this
Code: Select all
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
Code: Select all
service crond start
service ndo2db start
service nagios start
service npcd restart
Give the system 15 to 20 minutes to run and see if the status starts to populate with the correct data.
Re: event_handler.log out of control/mysqld hogging resource
Posted: Thu Mar 01, 2018 7:42 am
by TCBeekley
Looks like that did that trick!
Everything is back to normal.
Thanks!