High load and a lot of event_handler.php processes

toper · Post by **toper** » Wed Feb 08, 2017 9:02 am

Hi
After a power outage 2 days ago which caused 10000's of alarms i have high cpu load and a lot of event_handler.php processes running at the same time.

How can i stop or clear this ? - reboot did not help

51 processes running

nagios 329 320 0 12:32 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 472 457 0 11:23 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 1892 1884 0 11:02 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 2310 2294 0 12:27 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 2588 2576 0 13:10 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 2914 2904 0 11:34 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 4419 4412 0 10:41 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1

We are running Nagios XI 5.4.0

profile.zip

Thanks
Peter
TDC - Denmark

Post by **mcapra** » Wed Feb 08, 2017 2:44 pm

There's probably an awful lot of events that were generated. Can you share the output of:

Code: Select all

echo "select count(*) from xi_meta;" | mysql -uroot -pnagiosxi nagiosxi
echo "select count(*) from xi_eventqueue;" | mysql -uroot -pnagiosxi nagiosxi
echo "SELECT table_schema as 'DB', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC \t;" | mysql -uroot -pnagiosxi

If the counts are crazy high, we may need to truncate those tables or at least cut them down substantially.

toper · Post by **toper** » Wed Feb 08, 2017 2:55 pm

Strange ? - Unknown database 'nagiosxi'

[root@khk26dsg5 ~]# echo "select count(*) from xi_meta;" | mysql -uroot -pnagiosxi nagiosxi
ERROR 1049 (42000): Unknown database 'nagiosxi'

[root@khk26dsg5 ~]# echo "select count(*) from xi_eventqueue;" | mysql -uroot -pnagiosxi nagiosxi
ERROR 1049 (42000): Unknown database 'nagiosxi'

[root@khk26dsg5 ~]# echo "SELECT table_schema as 'DB', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC \t;" | mysql -uroot -pnagiosxi

Outfile disabled.
DB Table Size in MB
nagios nagios_statehistory 479.11
nagios nagios_contactnotifications 81.62
nagios nagios_contactnotificationmethods 77.74
nagios nagios_logentries 4.44
nagios nagios_commenthistory 2.90
nagios nagios_flappinghistory 2.30
nagios nagios_servicestatus 1.80
nagios nagios_services 0.68
nagios nagios_objects 0.61
nagios nagios_systemcommands 0.47
mysql help_topic 0.45
nagios nagios_contactgroup_members 0.42
nagios nagios_conninfo 0.24
nagios nagios_hoststatus 0.22
nagiosql tbl_info 0.13
nagios nagios_processevents 0.13
mysql help_keyword 0.10
nagios nagios_servicegroup_members 0.10
nagios nagios_hosts 0.09
nagios nagios_hostgroup_members 0.09
nagiosql tbl_host 0.08
nagios nagios_service_contactgroups 0.06
nagios nagios_service_contacts 0.06
nagios nagios_eventhandlers 0.06
nagios nagios_contact_notificationcommands 0.04
nagiosql tbl_command 0.04
nagiosql tbl_service 0.03
nagios nagios_notifications 0.03
nagiosql tbl_hostgroup 0.03
nagios nagios_timeperiod_timeranges 0.03
mysql help_relation 0.03
nagiosql tbl_timedefinition 0.03
mysql help_category 0.03
nagios nagios_commands 0.02
nagiosql tbl_contactgroup 0.02
nagiosql tbl_lnkHostToHostgroup 0.02
nagios nagios_host_contactgroups 0.02
nagios nagios_hostgroups 0.02
nagios nagios_externalcommands 0.02
nagiosql tbl_servicetemplate 0.02
nagiosql tbl_contact 0.02
nagios nagios_contactgroups 0.01
nagiosql tbl_lnkHostToHosttemplate 0.01
nagiosql tbl_hosttemplate 0.01
nagiosql tbl_timeperiod 0.01
nagios nagios_configfilevariables 0.01
nagiosql tbl_lnkHostgroupToHost 0.01
nagiosql tbl_lnkHostToContactgroup 0.01
nagiosql tbl_servicegroup 0.01
nagiosql tbl_variabledefinition 0.01
nagios nagios_contacts 0.01
nagiosql tbl_domain 0.01
nagiosql tbl_user 0.01
nagiosql tbl_contacttemplate 0.01
nagiosql tbl_lnkServiceToHost 0.01
nagiosql tbl_lnkServiceToServicetemplate 0.01
mysql db 0.01
nagios nagios_timeperiods 0.01
nagiosql tbl_lnkHostgroupToHostgroup 0.01
nagiosql tbl_lnkContactgroupToContactgroup 0.01
mysql proc 0.01
nagios nagios_contactstatus 0.01
nagios nagios_acknowledgements 0.01
nagiosql tbl_lnkServiceToServicegroup 0.01
nagiosql tbl_lnkServiceToHostgroup 0.01
nagiosql tbl_lnkServicetemplateToServicetemplate 0.01
nagios nagios_configfiles 0.00
nagiosql tbl_lnkContactToVariabledefinition 0.00
nagios nagios_servicegroups 0.00
nagios nagios_runtimevariables 0.00
nagiosql tbl_hostdependency 0.00
mysql columns_priv 0.00
mysql tables_priv 0.00
mysql procs_priv 0.00
nagiosql tbl_serviceescalation 0.00
nagiosql tbl_servicedependency 0.00
nagiosql tbl_hostescalation 0.00
nagiosql tbl_submenu 0.00
nagiosql tbl_settings 0.00
nagios nagios_comments 0.00
nagios nagios_programstatus 0.00
nagios nagios_downtimehistory 0.00
nagiosql tbl_lnkContactToContacttemplate 0.00
nagiosql tbl_logbook 0.00
nagiosql tbl_lnkHosttemplateToHosttemplate 0.00
nagiosql tbl_lnkContactToContactgroup 0.00
mysql user 0.00
nagiosql tbl_session 0.00
nagiosql tbl_lnkServiceToContact 0.00
nagiosql tbl_mainmenu 0.00
nagiosql tbl_lnkServiceToVariabledefinition 0.00
nagiosql tbl_session_locks 0.00
nagiosql tbl_lnkServiceToContactgroup 0.00
nagios nagios_instances 0.00
nagiosql tbl_lnkHosttemplateToVariabledefinition 0.00
nagiosql tbl_lnkContacttemplateToCommandService 0.00
nagiosql tbl_lnkContacttemplateToCommandHost 0.00
nagiosql tbl_lnkContactToCommandService 0.00
nagiosql tbl_lnkContactToCommandHost 0.00
nagiosql tbl_lnkServicetemplateToVariabledefinition 0.00
mysql host 0.00
nagios nagios_customvariablestatus 0.00
nagios nagios_customvariables 0.00
mysql event 0.00
nagios nagios_dbversion 0.00
nagiosql tbl_lnkServiceescalationToService 0.00
nagios nagios_service_parentservices 0.00
nagios nagios_hostescalation_contacts 0.00
nagiosql tbl_lnkServicedependencyToHostgroup_DH 0.00
mysql time_zone_leap_second 0.00
nagiosql tbl_lnkContacttemplateToContacttemplate 0.00
nagiosql tbl_lnkHostdependencyToHost_DH 0.00
information_schema PLUGINS 0.00
mysql ndb_binlog_index 0.00
nagiosql tbl_lnkServiceescalationToHostgroup 0.00
nagios nagios_hostescalation_contactgroups 0.00
nagiosql tbl_lnkServicedependencyToHost_H 0.00
mysql time_zone 0.00
nagiosql tbl_lnkContacttemplateToContactgroup 0.00
information_schema COLUMNS 0.00
information_schema VIEWS 0.00
nagiosql tbl_lnkHostToVariabledefinition 0.00
information_schema PARTITIONS 0.00
nagios nagios_timedevents 0.00
nagiosql tbl_lnkServiceescalationToHost 0.00
nagios nagios_hostdependencies 0.00
nagiosql tbl_lnkServicedependencyToHost_DH 0.00
nagiosql tbl_lnkHostescalationToHostgroup 0.00
nagiosql tbl_lnkServiceescalationToContactgroup 0.00
nagios nagios_scheduleddowntime 0.00
nagios nagios_hostchecks 0.00
nagiosql tbl_lnkServicetemplateToHost 0.00
nagios nagios_serviceescalations 0.00
nagiosql tbl_lnkHostescalationToHost 0.00
information_schema TRIGGERS 0.00
nagiosql tbl_lnkTimeperiodToTimeperiod 0.00
nagios nagios_timedeventqueue 0.00
nagiosql tbl_lnkHosttemplateToHostgroup 0.00
nagios nagios_host_parenthosts 0.00
nagiosql tbl_lnkServicetemplateToContactgroup 0.00
nagios nagios_serviceescalation_contacts 0.00
nagiosql tbl_lnkHostescalationToContactgroup 0.00
nagiosql tbl_lnkServiceescalationToContact 0.00
information_schema ROUTINES 0.00
mysql servers 0.00
nagiosql tbl_lnkHostToHost 0.00
nagios nagios_contact_addresses 0.00
nagiosql tbl_lnkHosttemplateToHost 0.00
nagios nagios_host_contacts 0.00
nagiosql tbl_lnkServicetemplateToContact 0.00
nagios nagios_serviceescalation_contactgroups 0.00
nagiosql tbl_lnkHostescalationToContact 0.00
nagiosql tbl_lnkServicedependencyToService_S 0.00
nagiosql tbl_lnkContactgroupToContact 0.00
nagiosql tbl_serviceextinfo 0.00
mysql time_zone_transition_type 0.00
mysql func 0.00
nagiosql tbl_lnkServicegroupToServicegroup 0.00
nagios nagios_servicedependencies 0.00
nagiosql tbl_lnkHostdependencyToHostgroup_H 0.00
nagiosql tbl_lnkServicedependencyToService_DS 0.00
nagiosql tbl_lnkHosttemplateToContactgroup 0.00
mysql time_zone_transition 0.00
nagiosql tbl_lnkHostToContact 0.00
information_schema EVENTS 0.00
nagiosql tbl_hostextinfo 0.00
nagiosql tbl_lnkServicetemplateToServicegroup 0.00
nagiosql tbl_lnkHostdependencyToHostgroup_DH 0.00
nagiosql tbl_lnkServicegroupToService 0.00
nagios nagios_servicechecks 0.00
nagios nagios_hostescalations 0.00
nagiosql tbl_lnkServicedependencyToHostgroup_H 0.00
nagiosql tbl_lnkHosttemplateToContact 0.00
mysql time_zone_name 0.00
nagiosql tbl_lnkContacttemplateToVariabledefinition 0.00
nagiosql tbl_lnkServicetemplateToHostgroup 0.00
nagiosql tbl_lnkHostdependencyToHost_H 0.00
information_schema PROCESSLIST 0.00
mysql plugin 0.00
information_schema SESSION_VARIABLES 0.00
information_schema COLUMN_PRIVILEGES 0.00
information_schema SESSION_STATUS 0.00
information_schema SCHEMA_PRIVILEGES 0.00
information_schema COLLATION_CHARACTER_SET_APPLICABILITY 0.00
information_schema USER_PRIVILEGES 0.00
information_schema KEY_COLUMN_USAGE 0.00
information_schema SCHEMATA 0.00
mysql slow_log 0.00
information_schema COLLATIONS 0.00
information_schema GLOBAL_VARIABLES 0.00
information_schema CHARACTER_SETS 0.00
information_schema TABLE_PRIVILEGES 0.00
information_schema GLOBAL_STATUS 0.00
mysql general_log 0.00
information_schema REFERENTIAL_CONSTRAINTS 0.00
information_schema TABLE_CONSTRAINTS 0.00
information_schema FILES 0.00
information_schema TABLES 0.00
information_schema PROFILING 0.00
information_schema STATISTICS 0.00
information_schema ENGINES 0.00
[root@khk26dsg5 ~]#

toper · Post by **toper** » Wed Feb 08, 2017 3:02 pm

system_status.png

rkennedy · Post by **rkennedy** » Wed Feb 08, 2017 3:14 pm

toper wrote:Strange ? - Unknown database 'nagiosxi'

[root@khk26dsg5 ~]# echo "select count(*) from xi_meta;" | mysql -uroot -pnagiosxi nagiosxi
ERROR 1049 (42000): Unknown database 'nagiosxi'

[root@khk26dsg5 ~]# echo "select count(*) from xi_eventqueue;" | mysql -uroot -pnagiosxi nagiosxi
ERROR 1049 (42000): Unknown database 'nagiosxi'
[root@khk26dsg5 ~]#

The largest table size of 500M for the history doesn't sound too concerning. What does though, is that these commands will not run. They're working fine from my test system here -

Code: Select all

[root@xi libexec]# echo "select count(*) from xi_meta;" | mysql -uroot -pnagiosxi nagiosxi
count(*)
5
[root@xi libexec]# echo "select count(*) from xi_eventqueue;" | mysql -uroot -pnagiosxi nagiosxi
count(*)
0

We'll need to see what sort of count is on these tables. Try this manual route instead (this could vary if you're on an offloaded DB) -

Code: Select all

mysql -uroot -pnagiosxi
use nagiosxi;
select count(*) from xi_eventqueue;
select count(*) from xi_meta;

Could you paste everything outputted? It should be something like this (though, I suspect you'll have higher numbers) -

Code: Select all

MariaDB [(none)]> use nagiosxi;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [nagiosxi]> select count(*) from xi_eventqueue;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)

MariaDB [nagiosxi]> select count(*) from xi_meta;
+----------+
| count(*) |
+----------+
|        5 |
+----------+
1 row in set (0.00 sec)

toper wrote:
system_status.png

RE: this - I noticed in your profile things seem to be pretty chaotic right now which is probably why things are looking the way they do. Once we can clear these queue's out, it should go back down to normal.

toper · Post by **toper** » Wed Feb 08, 2017 3:32 pm

Hi,

This is an quite old installation, the reason may be, I'm still running pgsql ??

br,
Peter

[root@khk26dsg5 ~]# mysql -uroot -pnagiosxi
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1449
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use nagiosxi;
ERROR 1049 (42000): Unknown database 'nagiosxi'
mysql>
mysql> select count(*) from xi_eventqueue;
ERROR 1046 (3D000): No database selected

mysql>
mysql>
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| nagios |
| nagiosql |
| test |
+--------------------+
5 rows in set (0.00 sec)

I'm NOT a postgres expert

Post by **tgriep** » Wed Feb 08, 2017 4:55 pm

It looks like the xi_meta table it corrupted and the event_handler cron job is not processing that table.

Run the following procedure as root on the server to truncate the table and vacuum the postgres database.

Code: Select all

service nagios stop
service ndo2db stop
service crond stop
service postgresql restart
echo "TRUNCATE xi_meta CASCADE;"|psql nagiosxi nagiosxi

Then go to this link for the instructions for vaccuming the Postgres database. About half way down you will find the instructions.
https://support.nagios.com/kb/article.php?id=25

Then run the following to start up the processes.

Code: Select all

service crond start
service ndo2db start
service nagios start
service npcd restart

If the system is still loaded, can you post or PM me your System Profile?
login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and email or post it here.

toper · Post by **toper** » Thu Feb 09, 2017 2:54 am

Post by **tgriep** » Thu Feb 09, 2017 1:31 pm

Try this procedure to truncate all of the tables and see if the system starts to respond normally.

Code: Select all

service nagios stop
service ndo2db stop
service crond stop
service postgresql restart
pkill -9 -u nagios
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | psql nagiosxi nagiosxi
service crond start
service ndo2db start
service nagios start
service npcd restart

toper · Post by **toper** » Thu Feb 09, 2017 2:33 pm

Hi,

It looks fine now - low load again and tables get handled.

[root@khk26dsg5 ~]# psql nagiosxi nagiosxi
psql (8.4.20)
Type "help" for help.

nagiosxi=> SELECT COUNT(*) FROM xi_meta;
count
-------
504
(1 row)

nagiosxi=> SELECT COUNT(*) FROM xi_events;
count
-------
499
(1 row)

nagiosxi=> SELECT COUNT(*) FROM xi_eventqueue;
count
-------
0
(1 row)

I understand you have some older versions of XI running pgsql and newer versions running mysql/mariadb..
Is it possible to migrate to the newer version ?
I would also like to move our NagiosXI's from RHEL6 to CentOS7, is that a good idea with pgsql ?
Do you also in the future support the old pgsql version ?

thanks
Peter C
TDC - Denmark

Nagios Support Forum

High load and a lot of event_handler.php processes

High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes

Re: High load and a lot of event_handler.php processes