Code: Select all
# su - nagios
$ /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.phpNew NagiosXI server apply configuration taking hours
-
avandemore
- Posts: 1597
- Joined: Tue Sep 27, 2016 4:57 pm
Re: New NagiosXI server apply configuration taking hours
Please run the following commands and share the output:
Previous Nagios employee
Re: New NagiosXI server apply configuration taking hours
it is already running
Code: Select all
[email protected] ~$ /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
LOCKFILE '/usr/local/nagiosxi/var/dbmaint.lock' EXISTS - EXITING!
[email protected] ~$ ps -ef |grep dbmaint
nagios 1892 1876 0 10:45 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 1894 1892 0 10:45 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 3690 3649 0 06:20 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 3694 3690 0 06:20 ? 00:00:03 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 6162 6142 0 09:05 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 6171 6162 0 09:05 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 7872 7640 0 12:43 pts/0 00:00:00 grep dbmaint
nagios 10882 10843 0 07:25 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 10886 10882 0 07:25 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 11488 11474 0 11:55 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 11492 11488 0 11:55 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 12914 12879 0 10:10 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 12917 12914 0 10:10 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 17604 17577 0 08:30 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 17609 17604 0 08:30 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 22527 22491 0 11:20 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 22532 22527 0 11:20 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 24112 24072 0 09:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 24114 24112 0 09:35 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 25349 25327 0 06:55 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 25353 25349 0 06:55 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 29079 29058 0 12:25 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 29083 29079 0 12:25 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 31871 31847 0 08:00 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 31874 31871 0 08:00 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
Re: New NagiosXI server apply configuration taking hours
the log for dbmaint appeared that all of these jobs were locking each other, so I killed those and tried running as nagios. this is where it is:
It is sitting at that last select statement
Code: Select all
[email protected] ~$ /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1458668843)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1458668843)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1489600043)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1482428843)
CLEANING ndoutils TABLE 'notifications'...
SQL: DELETE FROM nagios_notifications WHERE start_time < FROM_UNIXTIME(1482428843)
CLEANING ndoutils TABLE 'contactnotifications'...
SQL: DELETE FROM nagios_contactnotifications WHERE start_time < FROM_UNIXTIME(1482428843)
CLEANING ndoutils TABLE 'contactnotificationmethods'...
SQL: DELETE FROM nagios_contactnotificationmethods WHERE start_time < FROM_UNIXTIME(1482428843)
CLEANING ndoutils TABLE 'statehistory'...
SQL: DELETE FROM nagios_statehistory WHERE state_time < FROM_UNIXTIME(1427132843)
CLEANING ndoutils TABLE 'timedevents'...
SQL: DELETE FROM nagios_timedevents WHERE event_time < FROM_UNIXTIME(1490204543)
CLEANING ndoutils TABLE 'systemcommands'...
SQL: DELETE FROM nagios_systemcommands WHERE start_time < FROM_UNIXTIME(1490204543)
CLEANING ndoutils TABLE 'servicechecks'...
SQL: DELETE FROM nagios_servicechecks WHERE start_time < FROM_UNIXTIME(1490204543)
CLEANING ndoutils TABLE 'hostchecks'...
SQL: DELETE FROM nagios_hostchecks WHERE start_time < FROM_UNIXTIME(1490204543)
CLEANING ndoutils TABLE 'eventhandlers'...
SQL: DELETE FROM nagios_eventhandlers WHERE start_time < FROM_UNIXTIME(1490204543)
LASTOPT: 1490201701
INTERVAL: 60
NOW: 1490204843
OPTTIME: 1490205301
CLEANING nagiosxi TABLE 'commands'...
SQL: DELETE FROM xi_commands WHERE processing_time < FROM_UNIXTIME(1490176043)
CLEANING nagiosxi TABLE 'events'...
SQL: DELETE FROM xi_events WHERE processing_time < FROM_UNIXTIME(1490176043)
SQL1: SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL
Re: New NagiosXI server apply configuration taking hours
What is the output of these commands?
Thank you
Code: Select all
echo "SELECT table_schema as 'DB', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC \t;" | mysql -uroot -pnagiosxi
echo "SELECT count(*) from xi_meta;" | mysql -uroot -pnagiosxi
echo "SELECT count(*) from xi_events;" | mysql -uroot -pnagiosxi
Re: New NagiosXI server apply configuration taking hours
Hmm...for the second time since cron started working, my server went to 100% CPU and became unusable. It appears to be kicking off cron jobs over and over on top of one another and will run 10+ of the same thing at one time.
You do not have the required permissions to view the files attached to this post.
Re: New NagiosXI server apply configuration taking hours
Code: Select all
Outfile disabled.
+--------------------+----------------------------------------------------+------------+
| DB | Table | Size in MB |
+--------------------+----------------------------------------------------+------------+
| nagiosxi | xi_meta | 1860.44 |
| nagiosxi | xi_eventqueue | 57.78 |
| nagiosxi | xi_events | 53.98 |
| nagios | nagios_logentries | 18.18 |
| nagios | nagios_notifications | 14.71 |
| nagios | nagios_contactnotifications | 5.29 |
| nagios | nagios_contactnotificationmethods | 4.23 |
| mysql | time_zone_transition | 4.01 |
| nagios | nagios_statehistory | 1.39 |
| nagios | nagios_servicestatus | 0.58 |
| mysql | time_zone_transition_type | 0.43 |
| mysql | help_topic | 0.42 |
| mysql | proc | 0.40 |
| mysql | time_zone_name | 0.38 |
| nagios | nagios_services | 0.23 |
| nagiosxi | xi_auditlog | 0.21 |
| nagiosql | tbl_service | 0.19 |
| nagiosql | tbl_serviceescalation | 0.17 |
| nagios | nagios_objects | 0.14 |
| nagiosql | tbl_info | 0.13 |
| nagios | nagios_hoststatus | 0.11 |
| mysql | help_keyword | 0.10 |
| nagios | nagios_customvariablestatus | 0.10 |
| nagios | nagios_customvariables | 0.10 |
| nagios | nagios_serviceescalations | 0.07 |
| nagiosql | tbl_variabledefinition | 0.06 |
| nagios | nagios_service_contactgroups | 0.06 |
| nagiosql | tbl_lnkServiceescalationToHost | 0.05 |
| nagios | nagios_hosts | 0.04 |
| nagios | nagios_serviceescalation_contactgroups | 0.04 |
| nagiosql | tbl_host | 0.04 |
| nagiosql | tbl_lnkServiceescalationToService | 0.04 |
| nagiosql | tbl_lnkServiceescalationToContactgroup | 0.04 |
| nagiosql | tbl_lnkServiceToServicetemplate | 0.04 |
| nagios | nagios_systemcommands | 0.03 |
| nagiosql | tbl_lnkServiceToContactgroup | 0.03 |
| nagios | nagios_commenthistory | 0.03 |
| mysql | time_zone | 0.03 |
| nagiosql | tbl_lnkServiceToHost | 0.03 |
| nagios | nagios_service_contacts | 0.03 |
| nagiosql | tbl_lnkServiceToVariabledefinition | 0.03 |
| nagiosql | tbl_command | 0.03 |
| mysql | help_relation | 0.03 |
| nagios | nagios_serviceescalation_contacts | 0.02 |
| nagios | nagios_hostgroup_members | 0.02 |
| nagiosql | tbl_lnkServiceescalationToContact | 0.02 |
| nagiosql | tbl_lnkServiceToContact | 0.02 |
| nagios | nagios_commands | 0.02 |
| nagiosql | tbl_servicetemplate | 0.02 |
| mysql | slave_master_info | 0.02 |
| mysql | rds_global_status_history | 0.02 |
| mysql | rds_configuration | 0.02 |
| mysql | rds_sysinfo | 0.02 |
| sys | sys_config | 0.02 |
| mysql | rds_replication_status | 0.02 |
| mysql | innodb_table_stats | 0.02 |
| mysql | rds_history | 0.02 |
| mysql | innodb_index_stats | 0.02 |
| mysql | slave_worker_info | 0.02 |
| mysql | rds_heartbeat2 | 0.02 |
| mysql | slave_relay_log_info | 0.02 |
| mysql | rds_global_status_history_old | 0.02 |
| nagios | nagios_hostescalations | 0.02 |
| nagiosql | tbl_hosttemplate | 0.01 |
| nagios | nagios_flappinghistory | 0.01 |
| nagios | nagios_timeperiod_timeranges | 0.01 |
| nagiosxi | xi_usermeta | 0.01 |
| nagios | nagios_hostescalation_contactgroups | 0.01 |
| nagiosql | tbl_lnkHostgroupToHost | 0.01 |
| nagios | nagios_contact_notificationcommands | 0.01 |
| nagiosql | tbl_lnkHostescalationToHost | 0.01 |
| nagios | nagios_host_contactgroups | 0.01 |
| nagios | nagios_configfilevariables | 0.01 |
| nagiosql | tbl_timeperiod | 0.01 |
| nagiosql | tbl_contact | 0.01 |
| nagiosql | tbl_timedefinition | 0.01 |
| nagiosql | tbl_hostgroup | 0.01 |
| nagiosql | tbl_hostescalation | 0.01 |
| nagiosql | tbl_contactgroup | 0.01 |
| nagiosql | tbl_domain | 0.01 |
| nagiosql | tbl_user | 0.01 |
| nagiosql | tbl_contacttemplate | 0.01 |
| nagiosxi | xi_options | 0.01 |
| nagiosxi | xi_sysstat | 0.01 |
| nagiosql | tbl_lnkHostToHosttemplate | 0.01 |
| nagiosql | tbl_lnkHostToContactgroup | 0.01 |
| nagiosql | tbl_lnkHostToVariabledefinition | 0.01 |
| nagios | nagios_host_contacts | 0.01 |
| nagios | nagios_eventhandlers | 0.01 |
| nagiosql | tbl_lnkHostToHostgroup | 0.01 |
| mysql | db | 0.01 |
| nagios | nagios_hostescalation_contacts | 0.01 |
| nagiosxi | xi_commands | 0.01 |
| nagios | nagios_processevents | 0.01 |
| mysql | proxies_priv | 0.01 |
| nagiosql | tbl_lnkServicetemplateToServicetemplate | 0.00 |
| nagios | nagios_configfiles | 0.00 |
| nagiosql | tbl_lnkHostToContact | 0.00 |
| nagios | nagios_timeperiods | 0.00 |
| mysql | event | 0.00 |
| nagios | nagios_runtimevariables | 0.00 |
| mysql | help_category | 0.00 |
| nagiosql | tbl_hostdependency | 0.00 |
| nagiosql | tbl_servicedependency | 0.00 |
| mysql | procs_priv | 0.00 |
| mysql | tables_priv | 0.00 |
| mysql | columns_priv | 0.00 |
| nagiosql | tbl_servicegroup | 0.00 |
| nagios | nagios_contacts | 0.00 |
| nagios | nagios_conninfo | 0.00 |
| nagios | nagios_hostgroups | 0.00 |
| nagiosql | tbl_submenu | 0.00 |
| nagios | nagios_host_parenthosts | 0.00 |
| nagios | nagios_contactstatus | 0.00 |
| nagiosql | tbl_settings | 0.00 |
| nagios | nagios_contactgroups | 0.00 |
| nagios | nagios_comments | 0.00 |
| nagios | nagios_contactgroup_members | 0.00 |
| nagiosxi | xi_users | 0.00 |
| nagios | nagios_programstatus | 0.00 |
| nagios | nagios_contact_addresses | 0.00 |
| nagiosql | tbl_session | 0.00 |
| mysql | user | 0.00 |
| nagiosql | tbl_lnkHosttemplateToHosttemplate | 0.00 |
| nagiosql | tbl_logbook | 0.00 |
| nagiosql | tbl_lnkHostToHost | 0.00 |
| nagiosql | tbl_mainmenu | 0.00 |
| nagiosql | tbl_session_locks | 0.00 |
| nagiosql | tbl_lnkContactToContacttemplate | 0.00 |
| nagios | nagios_externalcommands | 0.00 |
| nagiosql | tbl_lnkHostescalationToContactgroup | 0.00 |
| nagiosql | tbl_lnkHostescalationToContact | 0.00 |
| nagiosql | tbl_lnkContactgroupToContact | 0.00 |
| nagiosql | tbl_lnkContactToCommandService | 0.00 |
| nagiosql | tbl_lnkHosttemplateToContactgroup | 0.00 |
| nagiosql | tbl_lnkContactToCommandHost | 0.00 |
| nagios | nagios_instances | 0.00 |
| nagiosql | tbl_lnkContacttemplateToCommandService | 0.00 |
| nagiosql | tbl_lnkContacttemplateToCommandHost | 0.00 |
| nagiosql | tbl_lnkContacttemplateToContactgroup | 0.00 |
| nagiosql | tbl_lnkServicetemplateToContactgroup | 0.00 |
| nagiosql | tbl_lnkHosttemplateToHostgroup | 0.00 |
| nagiosql | tbl_lnkContactToContactgroup | 0.00 |
| mysql | host | 0.00 |
| nagios | nagios_dbversion | 0.00 |
| nagiosql | tbl_lnkServicetemplateToHostgroup | 0.00 |
| nagios | nagios_servicechecks | 0.00 |
| information_schema | TRIGGERS | 0.00 |
| nagiosql | tbl_lnkServiceescalationToHostgroup | 0.00 |
| nagiosql | tbl_serviceextinfo | 0.00 |
| nagiosql | tbl_lnkContacttemplateToVariabledefinition | 0.00 |
| nagiosql | tbl_lnkHostdependencyToHostgroup_H | 0.00 |
| nagiosql | tbl_lnkServiceToServicegroup | 0.00 |
| nagios | nagios_hostdependencies | 0.00 |
| nagiosql | tbl_lnkContacttemplateToContacttemplate | 0.00 |
| nagiosql | tbl_lnkHostdependencyToHostgroup_DH | 0.00 |
| nagiosql | tbl_lnkServiceToHostgroup | 0.00 |
| nagios | nagios_hostchecks | 0.00 |
| information_schema | ROUTINES | 0.00 |
| nagiosql | tbl_lnkHostdependencyToHost_H | 0.00 |
| nagiosql | tbl_lnkServicedependencyToService_S | 0.00 |
| nagiosql | tbl_lnkHostdependencyToHost_DH | 0.00 |
| mysql | func | 0.00 |
| nagios | nagios_servicegroups | 0.00 |
| information_schema | EVENTS | 0.00 |
| mysql | time_zone_leap_second | 0.00 |
| mysql | servers | 0.00 |
| nagiosql | tbl_lnkHosttemplateToContact | 0.00 |
| nagiosql | tbl_lnkServicetemplateToHost | 0.00 |
| nagios | nagios_service_parentservices | 0.00 |
| nagiosql | tbl_lnkServicedependencyToService_DS | 0.00 |
| mysql | plugin | 0.00 |
| nagiosql | tbl_lnkContactgroupToContactgroup | 0.00 |
| nagios | nagios_servicegroup_members | 0.00 |
| nagiosql | tbl_lnkHostgroupToHostgroup | 0.00 |
| nagiosql | tbl_lnkServicedependencyToHostgroup_H | 0.00 |
| mysql | ndb_binlog_index | 0.00 |
| information_schema | PROCESSLIST | 0.00 |
| nagiosql | tbl_lnkHosttemplateToVariabledefinition | 0.00 |
| nagiosql | tbl_lnkTimeperiodToTimeperiod | 0.00 |
| nagiosql | tbl_lnkServicetemplateToContact | 0.00 |
| nagiosxi | xi_incidents | 0.00 |
| information_schema | PLUGINS | 0.00 |
| nagiosql | tbl_lnkServicedependencyToHostgroup_DH | 0.00 |
| information_schema | COLUMNS | 0.00 |
| nagiosql | tbl_lnkServicetemplateToVariabledefinition | 0.00 |
| nagiosql | tbl_lnkContactToVariabledefinition | 0.00 |
| nagiosql | tbl_lnkServicegroupToServicegroup | 0.00 |
| nagios | nagios_scheduleddowntime | 0.00 |
| nagiosql | tbl_lnkHostescalationToHostgroup | 0.00 |
| information_schema | PARTITIONS | 0.00 |
| nagiosql | tbl_lnkServicedependencyToHost_H | 0.00 |
| information_schema | VIEWS | 0.00 |
| nagiosql | tbl_hostextinfo | 0.00 |
| nagiosql | tbl_lnkServicegroupToService | 0.00 |
| nagios | nagios_timedevents | 0.00 |
| information_schema | PARAMETERS | 0.00 |
| nagiosql | tbl_lnkServicedependencyToHost_DH | 0.00 |
| nagiosql | tbl_lnkHosttemplateToHost | 0.00 |
| nagios | nagios_downtimehistory | 0.00 |
| nagiosql | tbl_lnkServicetemplateToServicegroup | 0.00 |
| nagios | nagios_servicedependencies | 0.00 |
| nagios | nagios_timedeventqueue | 0.00 |
| information_schema | OPTIMIZER_TRACE | 0.00 |
| nagios | nagios_acknowledgements | 0.00 |
| performance_schema | events_waits_summary_by_thread_by_event_name | 0.00 |
| information_schema | INNODB_SYS_TABLES | 0.00 |
| information_schema | CHARACTER_SETS | 0.00 |
| performance_schema | socket_summary_by_instance | 0.00 |
| performance_schema | events_statements_summary_global_by_event_name | 0.00 |
| information_schema | SESSION_STATUS | 0.00 |
| information_schema | INNODB_BUFFER_PAGE | 0.00 |
| performance_schema | events_stages_summary_by_thread_by_event_name | 0.00 |
| information_schema | KEY_COLUMN_USAGE | 0.00 |
| information_schema | INNODB_CMP_RESET | 0.00 |
| performance_schema | performance_timers | 0.00 |
| performance_schema | socket_summary_by_event_name | 0.00 |
| performance_schema | events_statements_summary_by_user_by_event_name | 0.00 |
| information_schema | SCHEMA_PRIVILEGES | 0.00 |
| information_schema | INNODB_CMP_PER_INDEX_RESET | 0.00 |
| information_schema | GLOBAL_VARIABLES | 0.00 |
| information_schema | INNODB_METRICS | 0.00 |
| performance_schema | objects_summary_global_by_type | 0.00 |
| performance_schema | events_stages_summary_by_host_by_event_name | 0.00 |
| performance_schema | socket_instances | 0.00 |
| performance_schema | events_statements_summary_by_thread_by_event_name | 0.00 |
| information_schema | SCHEMATA | 0.00 |
| information_schema | INNODB_SYS_FIELDS | 0.00 |
| information_schema | INNODB_CMP | 0.00 |
| performance_schema | mutex_instances | 0.00 |
| performance_schema | events_stages_summary_by_account_by_event_name | 0.00 |
| information_schema | GLOBAL_STATUS | 0.00 |
| performance_schema | events_statements_summary_by_host_by_event_name | 0.00 |
| information_schema | INNODB_FT_DEFAULT_STOPWORD | 0.00 |
| performance_schema | setup_timers | 0.00 |
| performance_schema | hosts | 0.00 |
| mysql | general_log | 0.00 |
| performance_schema | events_stages_history_long | 0.00 |
| information_schema | FILES | 0.00 |
| information_schema | INNODB_SYS_TABLESTATS | 0.00 |
| performance_schema | events_statements_summary_by_digest | 0.00 |
| information_schema | REFERENTIAL_CONSTRAINTS | 0.00 |
| information_schema | INNODB_SYS_INDEXES | 0.00 |
| performance_schema | setup_objects | 0.00 |
| performance_schema | host_cache | 0.00 |
| performance_schema | events_stages_history | 0.00 |
| information_schema | INNODB_LOCK_WAITS | 0.00 |
| performance_schema | events_waits_summary_by_instance | 0.00 |
| information_schema | TABLE_PRIVILEGES | 0.00 |
| information_schema | INNODB_SYS_FOREIGN_COLS | 0.00 |
| information_schema | PROFILING | 0.00 |
| information_schema | INNODB_SYS_COLUMNS | 0.00 |
| performance_schema | setup_instruments | 0.00 |
| performance_schema | events_statements_summary_by_account_by_event_name | 0.00 |
| performance_schema | file_summary_by_instance | 0.00 |
| performance_schema | events_stages_current | 0.00 |
| information_schema | ENGINES | 0.00 |
| information_schema | INNODB_SYS_DATAFILES | 0.00 |
| performance_schema | users | 0.00 |
| performance_schema | events_waits_summary_by_host_by_event_name | 0.00 |
| information_schema | TABLE_CONSTRAINTS | 0.00 |
| information_schema | INNODB_FT_INDEX_CACHE | 0.00 |
| information_schema | INNODB_SYS_FOREIGN | 0.00 |
| performance_schema | setup_consumers | 0.00 |
| performance_schema | events_statements_history_long | 0.00 |
| performance_schema | cond_instances | 0.00 |
| information_schema | COLUMN_PRIVILEGES | 0.00 |
| information_schema | INNODB_TRX | 0.00 |
| performance_schema | file_summary_by_event_name | 0.00 |
| performance_schema | events_waits_summary_by_account_by_event_name | 0.00 |
| information_schema | TABLESPACES | 0.00 |
| information_schema | INNODB_SYS_TABLESPACES | 0.00 |
| performance_schema | threads | 0.00 |
| performance_schema | setup_actors | 0.00 |
| performance_schema | events_statements_history | 0.00 |
| information_schema | INNODB_BUFFER_PAGE_LRU | 0.00 |
| performance_schema | accounts | 0.00 |
| information_schema | INNODB_LOCKS | 0.00 |
| mysql | slow_log | 0.00 |
| performance_schema | file_instances | 0.00 |
| performance_schema | events_waits_history_long | 0.00 |
| information_schema | TABLES | 0.00 |
| information_schema | INNODB_FT_BEING_DELETED | 0.00 |
| performance_schema | table_lock_waits_summary_by_table | 0.00 |
| performance_schema | session_connect_attrs | 0.00 |
| performance_schema | events_statements_current | 0.00 |
| information_schema | INNODB_FT_DELETED | 0.00 |
| information_schema | COLLATION_CHARACTER_SET_APPLICABILITY | 0.00 |
| performance_schema | events_waits_summary_global_by_event_name | 0.00 |
| information_schema | INNODB_FT_CONFIG | 0.00 |
| information_schema | STATISTICS | 0.00 |
| information_schema | INNODB_FT_INDEX_TABLE | 0.00 |
| performance_schema | table_io_waits_summary_by_table | 0.00 |
| performance_schema | events_waits_history | 0.00 |
| performance_schema | session_account_connect_attrs | 0.00 |
| performance_schema | events_stages_summary_global_by_event_name | 0.00 |
| information_schema | INNODB_CMPMEM_RESET | 0.00 |
| information_schema | USER_PRIVILEGES | 0.00 |
| performance_schema | events_waits_summary_by_user_by_event_name | 0.00 |
| information_schema | INNODB_BUFFER_POOL_STATS | 0.00 |
| information_schema | COLLATIONS | 0.00 |
| information_schema | INNODB_CMPMEM | 0.00 |
| performance_schema | table_io_waits_summary_by_index_usage | 0.00 |
| performance_schema | events_waits_current | 0.00 |
| information_schema | SESSION_VARIABLES | 0.00 |
| performance_schema | events_stages_summary_by_user_by_event_name | 0.00 |
| information_schema | INNODB_CMP_PER_INDEX | 0.00 |
| performance_schema | rwlock_instances | 0.00 |
| sys | statements_with_full_table_scans | NULL |
| sys | x$waits_by_host_by_latency | NULL |
| sys | host_summary_by_file_io | NULL |
| sys | x$io_global_by_file_by_latency | NULL |
| sys | x$host_summary_by_statement_latency | NULL |
| sys | schema_table_statistics | NULL |
| sys | x$user_summary_by_file_io | NULL |
| sys | io_global_by_file_by_latency | NULL |
| sys | x$schema_table_statistics | NULL |
| sys | user_summary_by_statement_type | NULL |
| sys | schema_redundant_indexes | NULL |
| sys | x$user_summary | NULL |
| sys | x$host_summary_by_stages | NULL |
| sys | io_global_by_file_by_bytes | NULL |
| sys | x$schema_index_statistics | NULL |
| sys | user_summary_by_statement_latency | NULL |
| sys | schema_object_overview | NULL |
| sys | x$statements_with_temp_tables | NULL |
| sys | x$host_summary_by_file_io_type | NULL |
| sys | x$schema_flattened_keys | NULL |
| sys | user_summary_by_stages | NULL |
| sys | io_by_thread_by_latency | NULL |
| sys | schema_index_statistics | NULL |
| sys | x$statements_with_sorting | NULL |
| sys | x$host_summary_by_file_io | NULL |
| sys | user_summary_by_file_io_type | NULL |
| sys | innodb_lock_waits | NULL |
| sys | x$ps_schema_table_statistics_io | NULL |
| sys | schema_auto_increment_columns | NULL |
| sys | x$statements_with_runtimes_in_95th_percentile | NULL |
| sys | x$host_summary | NULL |
| sys | innodb_buffer_stats_by_table | NULL |
| sys | x$ps_digest_avg_latency_distribution | NULL |
| sys | user_summary_by_file_io | NULL |
| sys | host_summary | NULL |
| sys | statements_with_errors_or_warnings | NULL |
| sys | x$wait_classes_global_by_latency | NULL |
| sys | x$io_global_by_file_by_bytes | NULL |
| sys | ps_check_lost_instrumentation | NULL |
| sys | x$statements_with_full_table_scans | NULL |
| sys | waits_global_by_latency | NULL |
| sys | innodb_buffer_stats_by_schema | NULL |
| sys | x$ps_digest_95th_percentile_by_avg_us | NULL |
| sys | user_summary | NULL |
| sys | statement_analysis | NULL |
| sys | x$wait_classes_global_by_avg_latency | NULL |
| sys | x$io_by_thread_by_latency | NULL |
| sys | x$statements_with_errors_or_warnings | NULL |
| sys | waits_by_user_by_latency | NULL |
| sys | processlist | NULL |
| sys | host_summary_by_statement_type | NULL |
| sys | x$processlist | NULL |
| sys | session | NULL |
| sys | x$user_summary_by_statement_type | NULL |
| sys | x$innodb_lock_waits | NULL |
| sys | waits_by_host_by_latency | NULL |
| sys | metrics | NULL |
| sys | x$statement_analysis | NULL |
| sys | host_summary_by_statement_latency | NULL |
| sys | x$latest_file_io | NULL |
| sys | statements_with_temp_tables | NULL |
| sys | schema_unused_indexes | NULL |
| sys | x$user_summary_by_statement_latency | NULL |
| sys | x$innodb_buffer_stats_by_table | NULL |
| sys | latest_file_io | NULL |
| sys | x$session | NULL |
| sys | wait_classes_global_by_latency | NULL |
| sys | host_summary_by_stages | NULL |
| sys | x$io_global_by_wait_by_latency | NULL |
| sys | statements_with_sorting | NULL |
| sys | x$waits_global_by_latency | NULL |
| sys | schema_tables_with_full_table_scans | NULL |
| sys | x$user_summary_by_stages | NULL |
| sys | x$innodb_buffer_stats_by_schema | NULL |
| sys | io_global_by_wait_by_latency | NULL |
| sys | x$schema_tables_with_full_table_scans | NULL |
| sys | wait_classes_global_by_avg_latency | NULL |
| sys | x$io_global_by_wait_by_bytes | NULL |
| sys | statements_with_runtimes_in_95th_percentile | NULL |
| sys | x$waits_by_user_by_latency | NULL |
| sys | host_summary_by_file_io_type | NULL |
| sys | x$user_summary_by_file_io_type | NULL |
| sys | x$host_summary_by_statement_type | NULL |
| sys | schema_table_statistics_with_buffer | NULL |
| sys | io_global_by_wait_by_bytes | NULL |
| sys | x$schema_table_statistics_with_buffer | NULL |
| sys | version | NULL |
+--------------------+----------------------------------------------------+------------+
395 rows in set (0.71 sec)
mysql> SELECT count(*) from xi_meta;
+----------+
| count(*) |
+----------+
| 1654732 |
+----------+
1 row in set (2.37 sec)
mysql> SELECT count(*) from xi_events;
+----------+
| count(*) |
+----------+
| 391856 |
+----------+
1 row in set (0.00 sec)Re: New NagiosXI server apply configuration taking hours
The instance died in the night when it ran out of memory.
I have restarted this morning and 10 minutes in, these are the php processes:
Is this normal?
Code: Select all
Authorized uses only. All activity may be monitored and reported.
login: nagios01.ec2.prod.azw.nasba login: Out of memory: Kill process 19767 (php) score 20 or sacrifice child
Killed process 19767, UID 501, (php) total-vm:384864kB, anon-rss:77676kB, file-rss:52kB
Out of memory: Kill process 21514 (php) score 20 or sacrifice child
Killed process 21514, UID 501, (php) total-vm:384864kB, anon-rss:77884kB, file-rss:4kBCode: Select all
nagios 2508 2495 0 06:56 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 2513 2508 0 06:56 ? 00:00:04 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 4837 4817 0 06:59 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 4842 4837 0 06:59 ? 00:00:03 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 5635 5568 0 07:00 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 5639 5635 0 07:00 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 6773 6763 0 07:02 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 6775 6773 0 07:02 ? 00:00:02 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 8652 8633 0 07:05 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 8658 8652 0 07:05 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 10314 10295 0 07:08 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 10315 10314 0 07:08 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 12127 12104 0 07:11 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 12129 12127 0 07:11 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 13254 13244 0 07:13 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 13256 13254 0 07:13 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios 13259 13240 0 07:13 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 13260 13259 0 07:13 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 13264 13245 0 07:13 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 13265 13241 0 07:13 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios 13267 13243 0 07:13 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 13268 13264 0 07:13 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 13269 13267 1 07:13 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios 13270 13265 0 07:13 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.phpIs this normal?
Re: New NagiosXI server apply configuration taking hours
Updated look at the php process list and it continues to grow. I'm guessing this is why I'm running out of memory:
I see a similar issue with another ticket on the forum: https://support.nagios.com/forum/viewto ... 16&t=42357
Code: Select all
nagios 956 953 0 07:47 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 958 952 0 07:47 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 959 956 0 07:47 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios 962 949 0 07:47 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 963 958 0 07:47 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios 964 962 0 07:47 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 967 950 0 07:47 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios 969 954 0 07:47 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 971 967 0 07:47 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios 973 969 0 07:47 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 2508 2495 0 06:56 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 2513 2508 0 06:56 ? 00:00:05 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 4837 4817 0 06:59 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 4842 4837 0 06:59 ? 00:00:04 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 5635 5568 0 07:00 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 5639 5635 0 07:00 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 6773 6763 0 07:02 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 6775 6773 0 07:02 ? 00:00:03 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 8652 8633 0 07:05 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 8658 8652 0 07:05 ? 00:00:02 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 10314 10295 0 07:08 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 10315 10314 0 07:08 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 12127 12104 0 07:11 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 12129 12127 0 07:11 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 13891 13880 0 07:14 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 13894 13891 0 07:14 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 15593 15584 0 07:17 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 15597 15593 0 07:17 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 17450 17430 0 07:20 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 17455 17450 0 07:20 ? 00:00:01 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 19164 19152 0 07:23 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 19166 19164 0 07:23 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 20935 20927 0 07:26 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 20940 20935 0 07:26 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 22709 22698 0 07:29 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 22711 22709 0 07:29 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 24410 24398 0 07:32 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 24413 24410 0 07:32 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 26478 26462 0 07:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 26482 26457 0 07:35 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 26483 26478 0 07:35 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 26484 26482 0 07:35 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
nagios 28184 28167 0 07:38 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 28190 28184 0 07:38 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 29411 29379 0 07:40 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 29412 29411 0 07:40 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 31077 31065 0 07:43 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 31078 31077 0 07:43 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
nagios 32298 32283 0 07:45 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 32299 32298 0 07:45 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
-
avandemore
- Posts: 1597
- Joined: Tue Sep 27, 2016 4:57 pm
Re: New NagiosXI server apply configuration taking hours
Please run these commands:
You can kill the hung processes and watch them to see if the issue is resolved.
Code: Select all
# mysql -uroot -pnagiosxi -e "TRUNCATE TABLE xi_events;"
# mysql -uroot -pnagiosxi -e "TRUNCATE TABLE xi_meta;"
# mysql -uroot -pnagiosxi -e "TRUNCATE TABLE xi_eventqueue;"Previous Nagios employee
Re: New NagiosXI server apply configuration taking hours
done
Code: Select all
mysql> TRUNCATE TABLE xi_events;
Query OK, 0 rows affected (0.05 sec)
mysql> TRUNCATE TABLE xi_meta;
Query OK, 0 rows affected (2.04 sec)
mysql> TRUNCATE TABLE xi_eventqueue;
Query OK, 0 rows affected (0.02 sec)
Code: Select all
mysql> select COUNT(*) from xi_eventqueue;
+----------+
| COUNT(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)
mysql> select COUNT(*) from xi_events;
+----------+
| COUNT(*) |
+----------+
| 3030 |
+----------+
1 row in set (0.00 sec)
mysql> select COUNT(*) from xi_meta;
+----------+
| COUNT(*) |
+----------+
| 4008 |
+----------+
1 row in set (0.01 sec)