Event_Handler.log on fire

[vAJ]

A while back, we enabled some Global Event Handlers on our dev instance. After this, we saw the event_handler.log take off in size.

Recently, I've disabled those GEHs and the log is still on fire. tail of the file shows blistering writes which look like this:

Code: Select all

Array
(
    [0] => 3568451
    [eventqueue_id] => 3568451
    [1] => 1498312163
    [event_time] => 1498312163
    [2] => 2
    [event_source] => 2
    [3] => 2
    [event_type] => 2
    [4] => YToyOTp7czoxNzoibm90aWZpY2F0aW9uLXR5cGUiO3M6Nzoic2VydmljZSI7czo3OiJjb250YWN0IjtzOjc6ImhwcmFzYWQiO3M6MTI6ImNvbnRhY3RlbWFpbCI7czoyMjoiaGFyaS5wcmFzYWRAVHJpbmV0LmNvbSI7czo0OiJ0eXBlIjtzOjc6IlBST0JMRU0iO3M6OToiZXNjYWxhdGVkIjtzOjE6IjAiO3M6NjoiYXV0aG9yIjtiOjA7czo4OiJjb21tZW50cyI7YjowO3M6NDoiaG9zdCI7czoyODoidmFsMDFicm1zY2hlMDEudXguY29ycC5sb2NhbCI7czoxMToiaG9zdGFkZHJlc3MiO3M6Mjg6InZhbDAxYnJtc2NoZTAxLnV4LmNvcnAubG9jYWwiO3M6OToiaG9zdGFsaWFzIjtzOjI4OiJ2YWwwMWJybXNjaGUwMS51eC5jb3JwLmxvY2FsIjtzOjE1OiJob3N0ZGlzcGxheW5hbWUiO3M6Mjg6InZhbDAxYnJtc2NoZTAxLnV4LmNvcnAubG9jYWwiO3M6Nzoic2VydmljZSI7czoxMzoiUmVkaXNfSGl0cmF0ZSI7czo5OiJob3N0c3RhdGUiO3M6MjoiVVAiO3M6MTE6Imhvc3RzdGF0ZWlkIjtzOjE6IjAiO3M6MTI6InNlcnZpY2VzdGF0ZSI7czo4OiJDUklUSUNBTCI7czoxNDoic2VydmljZXN0YXRlaWQiO3M6MToiMiI7czoxNjoibGFzdHNlcnZpY2VzdGF0ZSI7czo4OiJDUklUSUNBTCI7czoxODoibGFzdHNlcnZpY2VzdGF0ZWlkIjtzOjE6IjIiO3M6MTY6InNlcnZpY2VzdGF0ZXR5cGUiO3M6NDoiSEFSRCI7czoxNDoiY3VycmVudGF0dGVtcHQiO3M6MToiNSI7czoxMToibWF4YXR0ZW1wdHMiO3M6MToiNSI7czoxNDoic2VydmljZWV2ZW50aWQiO3M6NzoiMTYwNDQ5MCI7czoxNjoic2VydmljZXByb2JsZW1pZCI7czo2OiI3MjE0NTMiO3M6MTM6InNlcnZpY2VvdXRwdXQiO3M6Nzc6IkNSSVRJQ0FMIEVSUk9SIC0gQ2FuIG5vdCBjb25uZWN0IHRvIHZhbDAxYnJtc2NoZTAxLnV4LmNvcnAubG9jYWwgb24gcG9ydCA2Mzc5IjtzOjE3OiJsb25nc2VydmljZW91dHB1dCI7YjowO3M6ODoiZGF0ZXRpbWUiO3M6Mjg6IlNhdCBKdW4gMjQgMDk6NDk6MjIgRURUIDIwMTciO3M6MTM6Imhvc3Rncm91cG5hbWUiO3M6NToibXNhcHAiO3M6MTQ6Imhvc3Rncm91cG5hbWVzIjtzOjEwOToibXNhcHAsbGludXgtc2luZ2xlcGFydCxsaW51eC1zZXJ2ZXJzLFJlZGlzIC0gU0RMQyxCcmFkZW50b24tVkFMMDEsQnJhZGVudG9uLVZBTC1tc2FwcCxCcmFkZW50b24tVkFMLEJyYWRlbnRvbiI7czoxNjoiY29udGFjdGdyb3VwbmFtZSI7czoxOToieGlfY29udGFjdGdyb3VwX2FsbCI7fQ==
    [event_meta] => YToyOTp7czoxNzoibm90aWZpY2F0aW9uLXR5cGUiO3M6Nzoic2VydmljZSI7czo3OiJjb250YWN0IjtzOjc6ImhwcmFzYWQiO3M6MTI6ImNvbnRhY3RlbWFpbCI7czoyMjoiaGFyaS5wcmFzYWRAVHJpbmV0LmNvbSI7czo0OiJ0eXBlIjtzOjc6IlBST0JMRU0iO3M6OToiZXNjYWxhdGVkIjtzOjE6IjAiO3M6NjoiYXV0aG9yIjtiOjA7czo4OiJjb21tZW50cyI7YjowO3M6NDoiaG9zdCI7czoyODoidmFsMDFicm1zY2hlMDEudXguY29ycC5sb2NhbCI7czoxMToiaG9zdGFkZHJlc3MiO3M6Mjg6InZhbDAxYnJtc2NoZTAxLnV4LmNvcnAubG9jYWwiO3M6OToiaG9zdGFsaWFzIjtzOjI4OiJ2YWwwMWJybXNjaGUwMS51eC5jb3JwLmxvY2FsIjtzOjE1OiJob3N0ZGlzcGxheW5hbWUiO3M6Mjg6InZhbDAxYnJtc2NoZTAxLnV4LmNvcnAubG9jYWwiO3M6Nzoic2VydmljZSI7czoxMzoiUmVkaXNfSGl0cmF0ZSI7czo5OiJob3N0c3RhdGUiO3M6MjoiVVAiO3M6MTE6Imhvc3RzdGF0ZWlkIjtzOjE6IjAiO3M6MTI6InNlcnZpY2VzdGF0ZSI7czo4OiJDUklUSUNBTCI7czoxNDoic2VydmljZXN0YXRlaWQiO3M6MToiMiI7czoxNjoibGFzdHNlcnZpY2VzdGF0ZSI7czo4OiJDUklUSUNBTCI7czoxODoibGFzdHNlcnZpY2VzdGF0ZWlkIjtzOjE6IjIiO3M6MTY6InNlcnZpY2VzdGF0ZXR5cGUiO3M6NDoiSEFSRCI7czoxNDoiY3VycmVudGF0dGVtcHQiO3M6MToiNSI7czoxMToibWF4YXR0ZW1wdHMiO3M6MToiNSI7czoxNDoic2VydmljZWV2ZW50aWQiO3M6NzoiMTYwNDQ5MCI7czoxNjoic2VydmljZXByb2JsZW1pZCI7czo2OiI3MjE0NTMiO3M6MTM6InNlcnZpY2VvdXRwdXQiO3M6Nzc6IkNSSVRJQ0FMIEVSUk9SIC0gQ2FuIG5vdCBjb25uZWN0IHRvIHZhbDAxYnJtc2NoZTAxLnV4LmNvcnAubG9jYWwgb24gcG9ydCA2Mzc5IjtzOjE3OiJsb25nc2VydmljZW91dHB1dCI7YjowO3M6ODoiZGF0ZXRpbWUiO3M6Mjg6IlNhdCBKdW4gMjQgMDk6NDk6MjIgRURUIDIwMTciO3M6MTM6Imhvc3Rncm91cG5hbWUiO3M6NToibXNhcHAiO3M6MTQ6Imhvc3Rncm91cG5hbWVzIjtzOjEwOToibXNhcHAsbGludXgtc2luZ2xlcGFydCxsaW51eC1zZXJ2ZXJzLFJlZGlzIC0gU0RMQyxCcmFkZW50b24tVkFMMDEsQnJhZGVudG9uLVZBTC1tc2FwcCxCcmFkZW50b24tVkFMLEJyYWRlbnRvbiI7czoxNjoiY29udGFjdGdyb3VwbmFtZSI7czoxOToieGlfY29udGFjdGdyb3VwX2FsbCI7fQ==
)

Where am I missing how to tone this down? We're looking at 6GB in just a few hours.

[dwhitfield]

What version of XI are you running? 5.4.4 introduced a lot more logging. You can change it back to the way it was in /etc/cron.d/nagiosxi. Theoretically, the line is * * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1 and you can just change it to * * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1

[vAJ]

Yep, 5.4.4

If I upgrade to 5.4.6 today, will it overwrite the reverted change?

[dwhitfield]

I believe it will be overwritten.

/etc/logrotate.d/nagiosxi should set the max size to 5MB. Can you verify that's the case?

Also, what's the output of rpm -qa | grep logrotate and what OS are you running?

[vAJ]

Code: Select all

/usr/local/nagiosxi/var/*log {
    missingok
    notifempty
}

/usr/local/nagiosxi/var/xidebug.log {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    rotate 1
    compress
}

/usr/local/nagiosxi/var/xidebug.log.backtrace {
    missingok
    notifempty
    size 100M
    create 0660 apache nagios
    rotate 1
    compress
}

logrotate-3.8.6-12.el7.x86_64

on RHEL7

[dwhitfield]

That top one should look like

Code: Select all

/usr/local/nagiosxi/var/*log {
    missingok
    notifempty
    size 5M
    rotate 1
    compress
}

Why that didn't take on upgrade, I do not know. Maybe someone on your team modified it? 5M is certainly not a requirement, but it should be...something.

[vAJ]

Well, this got the file size under control, but now a DB performance issue which appears to be related to eventhandlers is more prevalent.

This weekend, the event_handlers.log caused our partition for /usr/local/nagios to fill 100%. It got too big, too fast to be dropped by logrotate.

This seems to have caused a cascade of instability issues with the box. Mostly around MariaDB taking up massive amounts of resources.

Trying to /usr/local/nagiosxi/cron/dbmaint.php manually, after a few attempts with

Code: Select all

PHP Warning:  mysqli_real_connect(): (08004/1040): Too many connections in /usr/local/nagios/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 117

It finally starts running, but hangs on

Code: Select all

CLEANING ndoutils TABLE 'eventhandlers'...
SQL: DELETE FROM nagios_eventhandlers WHERE start_time < FROM_UNIXTIME(1498682123)

I've sent an email to open a case, but haven't heard back yet.

[dwhitfield]

Seems like @ssax took your ticket. Are we ready to lock this up?

[vAJ]

You betcha. Thanks guys!