Page 4 of 6

Re: New NagiosXI server apply configuration taking hours

Posted: Wed Mar 22, 2017 10:56 am
by avandemore
Please run the following commands and share the output:

Code: Select all

# su - nagios
$ /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php

Re: New NagiosXI server apply configuration taking hours

Posted: Wed Mar 22, 2017 12:44 pm
by akepley
it is already running

Code: Select all

[email protected] ~$ /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
LOCKFILE '/usr/local/nagiosxi/var/dbmaint.lock' EXISTS - EXITING!
[email protected] ~$ ps -ef |grep dbmaint
nagios    1892  1876  0 10:45 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios    1894  1892  0 10:45 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios    3690  3649  0 06:20 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios    3694  3690  0 06:20 ?        00:00:03 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios    6162  6142  0 09:05 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios    6171  6162  0 09:05 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios    7872  7640  0 12:43 pts/0    00:00:00 grep dbmaint
nagios   10882 10843  0 07:25 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   10886 10882  0 07:25 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   11488 11474  0 11:55 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   11492 11488  0 11:55 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   12914 12879  0 10:10 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   12917 12914  0 10:10 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   17604 17577  0 08:30 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   17609 17604  0 08:30 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   22527 22491  0 11:20 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   22532 22527  0 11:20 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   24112 24072  0 09:35 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   24114 24112  0 09:35 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   25349 25327  0 06:55 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   25353 25349  0 06:55 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   29079 29058  0 12:25 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   29083 29079  0 12:25 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   31871 31847  0 08:00 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   31874 31871  0 08:00 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php

Re: New NagiosXI server apply configuration taking hours

Posted: Wed Mar 22, 2017 12:58 pm
by akepley
the log for dbmaint appeared that all of these jobs were locking each other, so I killed those and tried running as nagios. this is where it is:

Code: Select all

[email protected] ~$ /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1458668843)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1458668843)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1489600043)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1482428843)
CLEANING ndoutils TABLE 'notifications'...
SQL: DELETE FROM nagios_notifications WHERE start_time < FROM_UNIXTIME(1482428843)
CLEANING ndoutils TABLE 'contactnotifications'...
SQL: DELETE FROM nagios_contactnotifications WHERE start_time < FROM_UNIXTIME(1482428843)
CLEANING ndoutils TABLE 'contactnotificationmethods'...
SQL: DELETE FROM nagios_contactnotificationmethods WHERE start_time < FROM_UNIXTIME(1482428843)
CLEANING ndoutils TABLE 'statehistory'...
SQL: DELETE FROM nagios_statehistory WHERE state_time < FROM_UNIXTIME(1427132843)
CLEANING ndoutils TABLE 'timedevents'...
SQL: DELETE FROM nagios_timedevents WHERE event_time < FROM_UNIXTIME(1490204543)
CLEANING ndoutils TABLE 'systemcommands'...
SQL: DELETE FROM nagios_systemcommands WHERE start_time < FROM_UNIXTIME(1490204543)
CLEANING ndoutils TABLE 'servicechecks'...
SQL: DELETE FROM nagios_servicechecks WHERE start_time < FROM_UNIXTIME(1490204543)
CLEANING ndoutils TABLE 'hostchecks'...
SQL: DELETE FROM nagios_hostchecks WHERE start_time < FROM_UNIXTIME(1490204543)
CLEANING ndoutils TABLE 'eventhandlers'...
SQL: DELETE FROM nagios_eventhandlers WHERE start_time < FROM_UNIXTIME(1490204543)
LASTOPT:  1490201701
INTERVAL: 60
NOW:      1490204843
OPTTIME:  1490205301
CLEANING nagiosxi TABLE 'commands'...
SQL: DELETE FROM xi_commands WHERE processing_time < FROM_UNIXTIME(1490176043)
CLEANING nagiosxi TABLE 'events'...
SQL: DELETE FROM xi_events WHERE processing_time < FROM_UNIXTIME(1490176043)
SQL1: SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL

It is sitting at that last select statement

Re: New NagiosXI server apply configuration taking hours

Posted: Wed Mar 22, 2017 3:19 pm
by ssax
What is the output of these commands?

Code: Select all

echo "SELECT table_schema as 'DB', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC \t;" | mysql -uroot -pnagiosxi
echo "SELECT count(*) from xi_meta;" | mysql -uroot -pnagiosxi
echo "SELECT count(*) from xi_events;" | mysql -uroot -pnagiosxi
Thank you

Re: New NagiosXI server apply configuration taking hours

Posted: Wed Mar 22, 2017 4:39 pm
by akepley
Hmm...for the second time since cron started working, my server went to 100% CPU and became unusable. It appears to be kicking off cron jobs over and over on top of one another and will run 10+ of the same thing at one time.

Re: New NagiosXI server apply configuration taking hours

Posted: Wed Mar 22, 2017 5:08 pm
by akepley

Code: Select all

Outfile disabled.
+--------------------+----------------------------------------------------+------------+
| DB                 | Table                                              | Size in MB |
+--------------------+----------------------------------------------------+------------+
| nagiosxi           | xi_meta                                            |    1860.44 |
| nagiosxi           | xi_eventqueue                                      |      57.78 |
| nagiosxi           | xi_events                                          |      53.98 |
| nagios             | nagios_logentries                                  |      18.18 |
| nagios             | nagios_notifications                               |      14.71 |
| nagios             | nagios_contactnotifications                        |       5.29 |
| nagios             | nagios_contactnotificationmethods                  |       4.23 |
| mysql              | time_zone_transition                               |       4.01 |
| nagios             | nagios_statehistory                                |       1.39 |
| nagios             | nagios_servicestatus                               |       0.58 |
| mysql              | time_zone_transition_type                          |       0.43 |
| mysql              | help_topic                                         |       0.42 |
| mysql              | proc                                               |       0.40 |
| mysql              | time_zone_name                                     |       0.38 |
| nagios             | nagios_services                                    |       0.23 |
| nagiosxi           | xi_auditlog                                        |       0.21 |
| nagiosql           | tbl_service                                        |       0.19 |
| nagiosql           | tbl_serviceescalation                              |       0.17 |
| nagios             | nagios_objects                                     |       0.14 |
| nagiosql           | tbl_info                                           |       0.13 |
| nagios             | nagios_hoststatus                                  |       0.11 |
| mysql              | help_keyword                                       |       0.10 |
| nagios             | nagios_customvariablestatus                        |       0.10 |
| nagios             | nagios_customvariables                             |       0.10 |
| nagios             | nagios_serviceescalations                          |       0.07 |
| nagiosql           | tbl_variabledefinition                             |       0.06 |
| nagios             | nagios_service_contactgroups                       |       0.06 |
| nagiosql           | tbl_lnkServiceescalationToHost                     |       0.05 |
| nagios             | nagios_hosts                                       |       0.04 |
| nagios             | nagios_serviceescalation_contactgroups             |       0.04 |
| nagiosql           | tbl_host                                           |       0.04 |
| nagiosql           | tbl_lnkServiceescalationToService                  |       0.04 |
| nagiosql           | tbl_lnkServiceescalationToContactgroup             |       0.04 |
| nagiosql           | tbl_lnkServiceToServicetemplate                    |       0.04 |
| nagios             | nagios_systemcommands                              |       0.03 |
| nagiosql           | tbl_lnkServiceToContactgroup                       |       0.03 |
| nagios             | nagios_commenthistory                              |       0.03 |
| mysql              | time_zone                                          |       0.03 |
| nagiosql           | tbl_lnkServiceToHost                               |       0.03 |
| nagios             | nagios_service_contacts                            |       0.03 |
| nagiosql           | tbl_lnkServiceToVariabledefinition                 |       0.03 |
| nagiosql           | tbl_command                                        |       0.03 |
| mysql              | help_relation                                      |       0.03 |
| nagios             | nagios_serviceescalation_contacts                  |       0.02 |
| nagios             | nagios_hostgroup_members                           |       0.02 |
| nagiosql           | tbl_lnkServiceescalationToContact                  |       0.02 |
| nagiosql           | tbl_lnkServiceToContact                            |       0.02 |
| nagios             | nagios_commands                                    |       0.02 |
| nagiosql           | tbl_servicetemplate                                |       0.02 |
| mysql              | slave_master_info                                  |       0.02 |
| mysql              | rds_global_status_history                          |       0.02 |
| mysql              | rds_configuration                                  |       0.02 |
| mysql              | rds_sysinfo                                        |       0.02 |
| sys                | sys_config                                         |       0.02 |
| mysql              | rds_replication_status                             |       0.02 |
| mysql              | innodb_table_stats                                 |       0.02 |
| mysql              | rds_history                                        |       0.02 |
| mysql              | innodb_index_stats                                 |       0.02 |
| mysql              | slave_worker_info                                  |       0.02 |
| mysql              | rds_heartbeat2                                     |       0.02 |
| mysql              | slave_relay_log_info                               |       0.02 |
| mysql              | rds_global_status_history_old                      |       0.02 |
| nagios             | nagios_hostescalations                             |       0.02 |
| nagiosql           | tbl_hosttemplate                                   |       0.01 |
| nagios             | nagios_flappinghistory                             |       0.01 |
| nagios             | nagios_timeperiod_timeranges                       |       0.01 |
| nagiosxi           | xi_usermeta                                        |       0.01 |
| nagios             | nagios_hostescalation_contactgroups                |       0.01 |
| nagiosql           | tbl_lnkHostgroupToHost                             |       0.01 |
| nagios             | nagios_contact_notificationcommands                |       0.01 |
| nagiosql           | tbl_lnkHostescalationToHost                        |       0.01 |
| nagios             | nagios_host_contactgroups                          |       0.01 |
| nagios             | nagios_configfilevariables                         |       0.01 |
| nagiosql           | tbl_timeperiod                                     |       0.01 |
| nagiosql           | tbl_contact                                        |       0.01 |
| nagiosql           | tbl_timedefinition                                 |       0.01 |
| nagiosql           | tbl_hostgroup                                      |       0.01 |
| nagiosql           | tbl_hostescalation                                 |       0.01 |
| nagiosql           | tbl_contactgroup                                   |       0.01 |
| nagiosql           | tbl_domain                                         |       0.01 |
| nagiosql           | tbl_user                                           |       0.01 |
| nagiosql           | tbl_contacttemplate                                |       0.01 |
| nagiosxi           | xi_options                                         |       0.01 |
| nagiosxi           | xi_sysstat                                         |       0.01 |
| nagiosql           | tbl_lnkHostToHosttemplate                          |       0.01 |
| nagiosql           | tbl_lnkHostToContactgroup                          |       0.01 |
| nagiosql           | tbl_lnkHostToVariabledefinition                    |       0.01 |
| nagios             | nagios_host_contacts                               |       0.01 |
| nagios             | nagios_eventhandlers                               |       0.01 |
| nagiosql           | tbl_lnkHostToHostgroup                             |       0.01 |
| mysql              | db                                                 |       0.01 |
| nagios             | nagios_hostescalation_contacts                     |       0.01 |
| nagiosxi           | xi_commands                                        |       0.01 |
| nagios             | nagios_processevents                               |       0.01 |
| mysql              | proxies_priv                                       |       0.01 |
| nagiosql           | tbl_lnkServicetemplateToServicetemplate            |       0.00 |
| nagios             | nagios_configfiles                                 |       0.00 |
| nagiosql           | tbl_lnkHostToContact                               |       0.00 |
| nagios             | nagios_timeperiods                                 |       0.00 |
| mysql              | event                                              |       0.00 |
| nagios             | nagios_runtimevariables                            |       0.00 |
| mysql              | help_category                                      |       0.00 |
| nagiosql           | tbl_hostdependency                                 |       0.00 |
| nagiosql           | tbl_servicedependency                              |       0.00 |
| mysql              | procs_priv                                         |       0.00 |
| mysql              | tables_priv                                        |       0.00 |
| mysql              | columns_priv                                       |       0.00 |
| nagiosql           | tbl_servicegroup                                   |       0.00 |
| nagios             | nagios_contacts                                    |       0.00 |
| nagios             | nagios_conninfo                                    |       0.00 |
| nagios             | nagios_hostgroups                                  |       0.00 |
| nagiosql           | tbl_submenu                                        |       0.00 |
| nagios             | nagios_host_parenthosts                            |       0.00 |
| nagios             | nagios_contactstatus                               |       0.00 |
| nagiosql           | tbl_settings                                       |       0.00 |
| nagios             | nagios_contactgroups                               |       0.00 |
| nagios             | nagios_comments                                    |       0.00 |
| nagios             | nagios_contactgroup_members                        |       0.00 |
| nagiosxi           | xi_users                                           |       0.00 |
| nagios             | nagios_programstatus                               |       0.00 |
| nagios             | nagios_contact_addresses                           |       0.00 |
| nagiosql           | tbl_session                                        |       0.00 |
| mysql              | user                                               |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHosttemplate                  |       0.00 |
| nagiosql           | tbl_logbook                                        |       0.00 |
| nagiosql           | tbl_lnkHostToHost                                  |       0.00 |
| nagiosql           | tbl_mainmenu                                       |       0.00 |
| nagiosql           | tbl_session_locks                                  |       0.00 |
| nagiosql           | tbl_lnkContactToContacttemplate                    |       0.00 |
| nagios             | nagios_externalcommands                            |       0.00 |
| nagiosql           | tbl_lnkHostescalationToContactgroup                |       0.00 |
| nagiosql           | tbl_lnkHostescalationToContact                     |       0.00 |
| nagiosql           | tbl_lnkContactgroupToContact                       |       0.00 |
| nagiosql           | tbl_lnkContactToCommandService                     |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToContactgroup                  |       0.00 |
| nagiosql           | tbl_lnkContactToCommandHost                        |       0.00 |
| nagios             | nagios_instances                                   |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToCommandService             |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToCommandHost                |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToContactgroup               |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToContactgroup               |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHostgroup                     |       0.00 |
| nagiosql           | tbl_lnkContactToContactgroup                       |       0.00 |
| mysql              | host                                               |       0.00 |
| nagios             | nagios_dbversion                                   |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToHostgroup                  |       0.00 |
| nagios             | nagios_servicechecks                               |       0.00 |
| information_schema | TRIGGERS                                           |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToHostgroup                |       0.00 |
| nagiosql           | tbl_serviceextinfo                                 |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToVariabledefinition         |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHostgroup_H                 |       0.00 |
| nagiosql           | tbl_lnkServiceToServicegroup                       |       0.00 |
| nagios             | nagios_hostdependencies                            |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToContacttemplate            |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHostgroup_DH                |       0.00 |
| nagiosql           | tbl_lnkServiceToHostgroup                          |       0.00 |
| nagios             | nagios_hostchecks                                  |       0.00 |
| information_schema | ROUTINES                                           |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHost_H                      |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToService_S                |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHost_DH                     |       0.00 |
| mysql              | func                                               |       0.00 |
| nagios             | nagios_servicegroups                               |       0.00 |
| information_schema | EVENTS                                             |       0.00 |
| mysql              | time_zone_leap_second                              |       0.00 |
| mysql              | servers                                            |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToContact                       |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToHost                       |       0.00 |
| nagios             | nagios_service_parentservices                      |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToService_DS               |       0.00 |
| mysql              | plugin                                             |       0.00 |
| nagiosql           | tbl_lnkContactgroupToContactgroup                  |       0.00 |
| nagios             | nagios_servicegroup_members                        |       0.00 |
| nagiosql           | tbl_lnkHostgroupToHostgroup                        |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHostgroup_H              |       0.00 |
| mysql              | ndb_binlog_index                                   |       0.00 |
| information_schema | PROCESSLIST                                        |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToVariabledefinition            |       0.00 |
| nagiosql           | tbl_lnkTimeperiodToTimeperiod                      |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToContact                    |       0.00 |
| nagiosxi           | xi_incidents                                       |       0.00 |
| information_schema | PLUGINS                                            |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHostgroup_DH             |       0.00 |
| information_schema | COLUMNS                                            |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToVariabledefinition         |       0.00 |
| nagiosql           | tbl_lnkContactToVariabledefinition                 |       0.00 |
| nagiosql           | tbl_lnkServicegroupToServicegroup                  |       0.00 |
| nagios             | nagios_scheduleddowntime                           |       0.00 |
| nagiosql           | tbl_lnkHostescalationToHostgroup                   |       0.00 |
| information_schema | PARTITIONS                                         |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHost_H                   |       0.00 |
| information_schema | VIEWS                                              |       0.00 |
| nagiosql           | tbl_hostextinfo                                    |       0.00 |
| nagiosql           | tbl_lnkServicegroupToService                       |       0.00 |
| nagios             | nagios_timedevents                                 |       0.00 |
| information_schema | PARAMETERS                                         |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHost_DH                  |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHost                          |       0.00 |
| nagios             | nagios_downtimehistory                             |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToServicegroup               |       0.00 |
| nagios             | nagios_servicedependencies                         |       0.00 |
| nagios             | nagios_timedeventqueue                             |       0.00 |
| information_schema | OPTIMIZER_TRACE                                    |       0.00 |
| nagios             | nagios_acknowledgements                            |       0.00 |
| performance_schema | events_waits_summary_by_thread_by_event_name       |       0.00 |
| information_schema | INNODB_SYS_TABLES                                  |       0.00 |
| information_schema | CHARACTER_SETS                                     |       0.00 |
| performance_schema | socket_summary_by_instance                         |       0.00 |
| performance_schema | events_statements_summary_global_by_event_name     |       0.00 |
| information_schema | SESSION_STATUS                                     |       0.00 |
| information_schema | INNODB_BUFFER_PAGE                                 |       0.00 |
| performance_schema | events_stages_summary_by_thread_by_event_name      |       0.00 |
| information_schema | KEY_COLUMN_USAGE                                   |       0.00 |
| information_schema | INNODB_CMP_RESET                                   |       0.00 |
| performance_schema | performance_timers                                 |       0.00 |
| performance_schema | socket_summary_by_event_name                       |       0.00 |
| performance_schema | events_statements_summary_by_user_by_event_name    |       0.00 |
| information_schema | SCHEMA_PRIVILEGES                                  |       0.00 |
| information_schema | INNODB_CMP_PER_INDEX_RESET                         |       0.00 |
| information_schema | GLOBAL_VARIABLES                                   |       0.00 |
| information_schema | INNODB_METRICS                                     |       0.00 |
| performance_schema | objects_summary_global_by_type                     |       0.00 |
| performance_schema | events_stages_summary_by_host_by_event_name        |       0.00 |
| performance_schema | socket_instances                                   |       0.00 |
| performance_schema | events_statements_summary_by_thread_by_event_name  |       0.00 |
| information_schema | SCHEMATA                                           |       0.00 |
| information_schema | INNODB_SYS_FIELDS                                  |       0.00 |
| information_schema | INNODB_CMP                                         |       0.00 |
| performance_schema | mutex_instances                                    |       0.00 |
| performance_schema | events_stages_summary_by_account_by_event_name     |       0.00 |
| information_schema | GLOBAL_STATUS                                      |       0.00 |
| performance_schema | events_statements_summary_by_host_by_event_name    |       0.00 |
| information_schema | INNODB_FT_DEFAULT_STOPWORD                         |       0.00 |
| performance_schema | setup_timers                                       |       0.00 |
| performance_schema | hosts                                              |       0.00 |
| mysql              | general_log                                        |       0.00 |
| performance_schema | events_stages_history_long                         |       0.00 |
| information_schema | FILES                                              |       0.00 |
| information_schema | INNODB_SYS_TABLESTATS                              |       0.00 |
| performance_schema | events_statements_summary_by_digest                |       0.00 |
| information_schema | REFERENTIAL_CONSTRAINTS                            |       0.00 |
| information_schema | INNODB_SYS_INDEXES                                 |       0.00 |
| performance_schema | setup_objects                                      |       0.00 |
| performance_schema | host_cache                                         |       0.00 |
| performance_schema | events_stages_history                              |       0.00 |
| information_schema | INNODB_LOCK_WAITS                                  |       0.00 |
| performance_schema | events_waits_summary_by_instance                   |       0.00 |
| information_schema | TABLE_PRIVILEGES                                   |       0.00 |
| information_schema | INNODB_SYS_FOREIGN_COLS                            |       0.00 |
| information_schema | PROFILING                                          |       0.00 |
| information_schema | INNODB_SYS_COLUMNS                                 |       0.00 |
| performance_schema | setup_instruments                                  |       0.00 |
| performance_schema | events_statements_summary_by_account_by_event_name |       0.00 |
| performance_schema | file_summary_by_instance                           |       0.00 |
| performance_schema | events_stages_current                              |       0.00 |
| information_schema | ENGINES                                            |       0.00 |
| information_schema | INNODB_SYS_DATAFILES                               |       0.00 |
| performance_schema | users                                              |       0.00 |
| performance_schema | events_waits_summary_by_host_by_event_name         |       0.00 |
| information_schema | TABLE_CONSTRAINTS                                  |       0.00 |
| information_schema | INNODB_FT_INDEX_CACHE                              |       0.00 |
| information_schema | INNODB_SYS_FOREIGN                                 |       0.00 |
| performance_schema | setup_consumers                                    |       0.00 |
| performance_schema | events_statements_history_long                     |       0.00 |
| performance_schema | cond_instances                                     |       0.00 |
| information_schema | COLUMN_PRIVILEGES                                  |       0.00 |
| information_schema | INNODB_TRX                                         |       0.00 |
| performance_schema | file_summary_by_event_name                         |       0.00 |
| performance_schema | events_waits_summary_by_account_by_event_name      |       0.00 |
| information_schema | TABLESPACES                                        |       0.00 |
| information_schema | INNODB_SYS_TABLESPACES                             |       0.00 |
| performance_schema | threads                                            |       0.00 |
| performance_schema | setup_actors                                       |       0.00 |
| performance_schema | events_statements_history                          |       0.00 |
| information_schema | INNODB_BUFFER_PAGE_LRU                             |       0.00 |
| performance_schema | accounts                                           |       0.00 |
| information_schema | INNODB_LOCKS                                       |       0.00 |
| mysql              | slow_log                                           |       0.00 |
| performance_schema | file_instances                                     |       0.00 |
| performance_schema | events_waits_history_long                          |       0.00 |
| information_schema | TABLES                                             |       0.00 |
| information_schema | INNODB_FT_BEING_DELETED                            |       0.00 |
| performance_schema | table_lock_waits_summary_by_table                  |       0.00 |
| performance_schema | session_connect_attrs                              |       0.00 |
| performance_schema | events_statements_current                          |       0.00 |
| information_schema | INNODB_FT_DELETED                                  |       0.00 |
| information_schema | COLLATION_CHARACTER_SET_APPLICABILITY              |       0.00 |
| performance_schema | events_waits_summary_global_by_event_name          |       0.00 |
| information_schema | INNODB_FT_CONFIG                                   |       0.00 |
| information_schema | STATISTICS                                         |       0.00 |
| information_schema | INNODB_FT_INDEX_TABLE                              |       0.00 |
| performance_schema | table_io_waits_summary_by_table                    |       0.00 |
| performance_schema | events_waits_history                               |       0.00 |
| performance_schema | session_account_connect_attrs                      |       0.00 |
| performance_schema | events_stages_summary_global_by_event_name         |       0.00 |
| information_schema | INNODB_CMPMEM_RESET                                |       0.00 |
| information_schema | USER_PRIVILEGES                                    |       0.00 |
| performance_schema | events_waits_summary_by_user_by_event_name         |       0.00 |
| information_schema | INNODB_BUFFER_POOL_STATS                           |       0.00 |
| information_schema | COLLATIONS                                         |       0.00 |
| information_schema | INNODB_CMPMEM                                      |       0.00 |
| performance_schema | table_io_waits_summary_by_index_usage              |       0.00 |
| performance_schema | events_waits_current                               |       0.00 |
| information_schema | SESSION_VARIABLES                                  |       0.00 |
| performance_schema | events_stages_summary_by_user_by_event_name        |       0.00 |
| information_schema | INNODB_CMP_PER_INDEX                               |       0.00 |
| performance_schema | rwlock_instances                                   |       0.00 |
| sys                | statements_with_full_table_scans                   |       NULL |
| sys                | x$waits_by_host_by_latency                         |       NULL |
| sys                | host_summary_by_file_io                            |       NULL |
| sys                | x$io_global_by_file_by_latency                     |       NULL |
| sys                | x$host_summary_by_statement_latency                |       NULL |
| sys                | schema_table_statistics                            |       NULL |
| sys                | x$user_summary_by_file_io                          |       NULL |
| sys                | io_global_by_file_by_latency                       |       NULL |
| sys                | x$schema_table_statistics                          |       NULL |
| sys                | user_summary_by_statement_type                     |       NULL |
| sys                | schema_redundant_indexes                           |       NULL |
| sys                | x$user_summary                                     |       NULL |
| sys                | x$host_summary_by_stages                           |       NULL |
| sys                | io_global_by_file_by_bytes                         |       NULL |
| sys                | x$schema_index_statistics                          |       NULL |
| sys                | user_summary_by_statement_latency                  |       NULL |
| sys                | schema_object_overview                             |       NULL |
| sys                | x$statements_with_temp_tables                      |       NULL |
| sys                | x$host_summary_by_file_io_type                     |       NULL |
| sys                | x$schema_flattened_keys                            |       NULL |
| sys                | user_summary_by_stages                             |       NULL |
| sys                | io_by_thread_by_latency                            |       NULL |
| sys                | schema_index_statistics                            |       NULL |
| sys                | x$statements_with_sorting                          |       NULL |
| sys                | x$host_summary_by_file_io                          |       NULL |
| sys                | user_summary_by_file_io_type                       |       NULL |
| sys                | innodb_lock_waits                                  |       NULL |
| sys                | x$ps_schema_table_statistics_io                    |       NULL |
| sys                | schema_auto_increment_columns                      |       NULL |
| sys                | x$statements_with_runtimes_in_95th_percentile      |       NULL |
| sys                | x$host_summary                                     |       NULL |
| sys                | innodb_buffer_stats_by_table                       |       NULL |
| sys                | x$ps_digest_avg_latency_distribution               |       NULL |
| sys                | user_summary_by_file_io                            |       NULL |
| sys                | host_summary                                       |       NULL |
| sys                | statements_with_errors_or_warnings                 |       NULL |
| sys                | x$wait_classes_global_by_latency                   |       NULL |
| sys                | x$io_global_by_file_by_bytes                       |       NULL |
| sys                | ps_check_lost_instrumentation                      |       NULL |
| sys                | x$statements_with_full_table_scans                 |       NULL |
| sys                | waits_global_by_latency                            |       NULL |
| sys                | innodb_buffer_stats_by_schema                      |       NULL |
| sys                | x$ps_digest_95th_percentile_by_avg_us              |       NULL |
| sys                | user_summary                                       |       NULL |
| sys                | statement_analysis                                 |       NULL |
| sys                | x$wait_classes_global_by_avg_latency               |       NULL |
| sys                | x$io_by_thread_by_latency                          |       NULL |
| sys                | x$statements_with_errors_or_warnings               |       NULL |
| sys                | waits_by_user_by_latency                           |       NULL |
| sys                | processlist                                        |       NULL |
| sys                | host_summary_by_statement_type                     |       NULL |
| sys                | x$processlist                                      |       NULL |
| sys                | session                                            |       NULL |
| sys                | x$user_summary_by_statement_type                   |       NULL |
| sys                | x$innodb_lock_waits                                |       NULL |
| sys                | waits_by_host_by_latency                           |       NULL |
| sys                | metrics                                            |       NULL |
| sys                | x$statement_analysis                               |       NULL |
| sys                | host_summary_by_statement_latency                  |       NULL |
| sys                | x$latest_file_io                                   |       NULL |
| sys                | statements_with_temp_tables                        |       NULL |
| sys                | schema_unused_indexes                              |       NULL |
| sys                | x$user_summary_by_statement_latency                |       NULL |
| sys                | x$innodb_buffer_stats_by_table                     |       NULL |
| sys                | latest_file_io                                     |       NULL |
| sys                | x$session                                          |       NULL |
| sys                | wait_classes_global_by_latency                     |       NULL |
| sys                | host_summary_by_stages                             |       NULL |
| sys                | x$io_global_by_wait_by_latency                     |       NULL |
| sys                | statements_with_sorting                            |       NULL |
| sys                | x$waits_global_by_latency                          |       NULL |
| sys                | schema_tables_with_full_table_scans                |       NULL |
| sys                | x$user_summary_by_stages                           |       NULL |
| sys                | x$innodb_buffer_stats_by_schema                    |       NULL |
| sys                | io_global_by_wait_by_latency                       |       NULL |
| sys                | x$schema_tables_with_full_table_scans              |       NULL |
| sys                | wait_classes_global_by_avg_latency                 |       NULL |
| sys                | x$io_global_by_wait_by_bytes                       |       NULL |
| sys                | statements_with_runtimes_in_95th_percentile        |       NULL |
| sys                | x$waits_by_user_by_latency                         |       NULL |
| sys                | host_summary_by_file_io_type                       |       NULL |
| sys                | x$user_summary_by_file_io_type                     |       NULL |
| sys                | x$host_summary_by_statement_type                   |       NULL |
| sys                | schema_table_statistics_with_buffer                |       NULL |
| sys                | io_global_by_wait_by_bytes                         |       NULL |
| sys                | x$schema_table_statistics_with_buffer              |       NULL |
| sys                | version                                            |       NULL |
+--------------------+----------------------------------------------------+------------+
395 rows in set (0.71 sec)


mysql> SELECT count(*) from xi_meta;
+----------+
| count(*) |
+----------+
|  1654732 |
+----------+
1 row in set (2.37 sec)

mysql> SELECT count(*) from xi_events;
+----------+
| count(*) |
+----------+
|   391856 |
+----------+
1 row in set (0.00 sec)

Re: New NagiosXI server apply configuration taking hours

Posted: Thu Mar 23, 2017 7:15 am
by akepley
The instance died in the night when it ran out of memory.

Code: Select all

Authorized uses only. All activity may be monitored and reported.

 login: nagios01.ec2.prod.azw.nasba login: Out of memory: Kill process 19767 (php) score 20 or sacrifice child

Killed process 19767, UID 501, (php) total-vm:384864kB, anon-rss:77676kB, file-rss:52kB

Out of memory: Kill process 21514 (php) score 20 or sacrifice child

Killed process 21514, UID 501, (php) total-vm:384864kB, anon-rss:77884kB, file-rss:4kB
I have restarted this morning and 10 minutes in, these are the php processes:

Code: Select all

nagios    2508  2495  0 06:56 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios    2513  2508  0 06:56 ?        00:00:04 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios    4837  4817  0 06:59 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios    4842  4837  0 06:59 ?        00:00:03 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios    5635  5568  0 07:00 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios    5639  5635  0 07:00 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios    6773  6763  0 07:02 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios    6775  6773  0 07:02 ?        00:00:02 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios    8652  8633  0 07:05 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios    8658  8652  0 07:05 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   10314 10295  0 07:08 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   10315 10314  0 07:08 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   12127 12104  0 07:11 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   12129 12127  0 07:11 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   13254 13244  0 07:13 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios   13256 13254  0 07:13 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios   13259 13240  0 07:13 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios   13260 13259  0 07:13 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios   13264 13245  0 07:13 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios   13265 13241  0 07:13 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios   13267 13243  0 07:13 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios   13268 13264  0 07:13 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios   13269 13267  1 07:13 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios   13270 13265  0 07:13 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php

Is this normal?

Re: New NagiosXI server apply configuration taking hours

Posted: Thu Mar 23, 2017 7:48 am
by akepley
Updated look at the php process list and it continues to grow. I'm guessing this is why I'm running out of memory:

Code: Select all

nagios     956   953  0 07:47 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios     958   952  0 07:47 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios     959   956  0 07:47 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios     962   949  0 07:47 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios     963   958  0 07:47 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios     964   962  0 07:47 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios     967   950  0 07:47 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios     969   954  0 07:47 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios     971   967  0 07:47 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios     973   969  0 07:47 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios    2508  2495  0 06:56 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios    2513  2508  0 06:56 ?        00:00:05 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios    4837  4817  0 06:59 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios    4842  4837  0 06:59 ?        00:00:04 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios    5635  5568  0 07:00 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios    5639  5635  0 07:00 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios    6773  6763  0 07:02 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios    6775  6773  0 07:02 ?        00:00:03 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios    8652  8633  0 07:05 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios    8658  8652  0 07:05 ?        00:00:02 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   10314 10295  0 07:08 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   10315 10314  0 07:08 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   12127 12104  0 07:11 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   12129 12127  0 07:11 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   13891 13880  0 07:14 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   13894 13891  0 07:14 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   15593 15584  0 07:17 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   15597 15593  0 07:17 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   17450 17430  0 07:20 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   17455 17450  0 07:20 ?        00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   19164 19152  0 07:23 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   19166 19164  0 07:23 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   20935 20927  0 07:26 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   20940 20935  0 07:26 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   22709 22698  0 07:29 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   22711 22709  0 07:29 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   24410 24398  0 07:32 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   24413 24410  0 07:32 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   26478 26462  0 07:35 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   26482 26457  0 07:35 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios   26483 26478  0 07:35 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   26484 26482  0 07:35 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios   28184 28167  0 07:38 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   28190 28184  0 07:38 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   29411 29379  0 07:40 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   29412 29411  0 07:40 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   31077 31065  0 07:43 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   31078 31077  0 07:43 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios   32298 32283  0 07:45 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios   32299 32298  0 07:45 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
I see a similar issue with another ticket on the forum: https://support.nagios.com/forum/viewto ... 16&t=42357

Re: New NagiosXI server apply configuration taking hours

Posted: Thu Mar 23, 2017 9:01 am
by avandemore
Please run these commands:

Code: Select all

# mysql -uroot -pnagiosxi -e "TRUNCATE TABLE xi_events;"
# mysql -uroot -pnagiosxi -e "TRUNCATE TABLE xi_meta;"
# mysql -uroot -pnagiosxi -e "TRUNCATE TABLE xi_eventqueue;"
You can kill the hung processes and watch them to see if the issue is resolved.

Re: New NagiosXI server apply configuration taking hours

Posted: Thu Mar 23, 2017 9:34 am
by akepley
done

Code: Select all

mysql> TRUNCATE TABLE xi_events;
Query OK, 0 rows affected (0.05 sec)

mysql> TRUNCATE TABLE xi_meta;
Query OK, 0 rows affected (2.04 sec)

mysql> TRUNCATE TABLE xi_eventqueue;
Query OK, 0 rows affected (0.02 sec)

Code: Select all

mysql> select COUNT(*) from xi_eventqueue;
+----------+
| COUNT(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)

mysql> select COUNT(*) from xi_events;
+----------+
| COUNT(*) |
+----------+
|     3030 |
+----------+
1 row in set (0.00 sec)

mysql> select COUNT(*) from xi_meta;
+----------+
| COUNT(*) |
+----------+
|     4008 |
+----------+
1 row in set (0.01 sec)