eventhandler.log started filling up like crazy

[matt.lilek]

Hello Team,

Had a Major outage today and eventhandler.log got crazy full quick. After the outage I deleted the file and ran a repair and it just started going crazy again. I have run this cron manually but doesnt seem to be truncating /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php . Well only 3GB left now and by the time i get a reply that will be gone too! Maybe ill blow the log away again to buy me more time 18GB= a few hours max

[matt.lilek]

Just noticed i got about 41GB of perfdata too, does that even sound correct?

[tacolover101]

i would expect this to happen on an outage as the queue will be out to processing. did this number work down with time?

when you mention 41GB of perfdata - where on the filesystem is this located? what folder or file? this part depends if it's pre vs. post processing, or perhaps additional verbose logging someone may have enabled.

[scottwilkerson]

one thing you can do is edit the /etc/cron.d/nagiosxi and replace >> with >

this will truncate each of the logs every minute

change this

Code: Select all

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php >> /usr/local/nagiosxi/var/dbmaint.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1
01  * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/recurring_downtime.php >> /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1

to this

Code: Select all

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01  * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/recurring_downtime.php > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1

[matt.lilek]

Hello,

Thanks for the reply. Despite my best efforts i feel like im losing the battle right now. The problem is no longer with the log filling up but the system is still consuming disk. /var/lib/pgsql/data/base/16385 has a ton of 1GB files in it, is that right? Im struggling a bit in knowing what are normal file sizes and which are not. Can i please get some help in figuring out what is taking all the space and how i can get it to stop taking everything that i am trying to free up. Earlier when i was talking about perfdata it was /usr/local/nagios/share/perfdata but that has dropped to 31GB now. Please let me what more i can do before it crashes again. Very surprised that it was not down when i logged in this morning. Thank you!

[scottwilkerson]

If your events are firing out of control you may just want to clear the full event queue with the following command:

Code: Select all

echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | psql nagiosxi nagiosxi

this will delete all notification/events in the system that have not fired yet.

[matt.lilek]

DAMN! Well Scott, that just cleared up 47GB! Think we might be good now. The outage only lasted a couple hours but the effects lasted what seemed as an eternity. I wouldn't say they were out of control anymore but maybe the event queue was full? Does that make sense that 47GB would free up by running that command? Thanks for that, if you dont hear back from me within the next day you can wrap this one up. If you dont hear from me before the weekend, have a great weekend. Look forward to hearing your response!

[matt.lilek]

oh and btw, i did run a echo "truncate table xi_events; truncate table xi_meta;" | mysql -uroot -pnagiosxi -D nagiosxi this morning before receiving your note. Looks like i was on the right track with my google searches/nagios forums. Thanks again for the guidance!

[scottwilkerson]

oh and btw, i did run a echo "truncate table xi_events; truncate table xi_meta;" | mysql -uroot -pnagiosxi -D nagiosxi this morning before receiving your note. Looks like i was on the right track with my google searches/nagios forums. Thanks again for the guidance!

Ya, on a upgraded system you need to do it in psql.

Does that make sense that 47GB would free up by running that command?

It can, especially if you had a mass outage and tons of alerts spiraling out of control.