Page 1 of 1
CPU utilization high
Posted: Wed Jun 02, 2021 7:15 am
by deek
Hello Team ,
Recently we see the CPU utilization getting high . The services monitored on the Nagios are not many.
We think it could be some corrupted configuration file .
Could you please help us here and share your thoughts .
Please let me know if you need Nagios profile .
Regards ,
Deek
Re: CPU utilization high
Posted: Wed Jun 02, 2021 1:47 pm
by vtrac
Hi Deek,
How are you doing?
Were there any changes made to the local Network, System or Nagios XI?
Please check the "/var/log/mariadb/mariadb.log" and see if the database has any crashed tables.
If so, please run the below to fix it:
Code: Select all
/usr/local/nagiosxi/scripts/repair_databases.sh
Please also run the below command on your XI command prompt and post the outputs here:
(Please NOTE it is a ONE line command)
Code: Select all
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
To further investigate the issue, please upload the "profile.zip".
Nagios Xi GUI > Admin > System Config > System Profile
Best Regards,
Vinh
Re: CPU utilization high
Posted: Wed Jun 16, 2021 3:50 am
by deek
Hi ,
I have messaged you the system profile .
And the output for the command is :
[root@lx******** ~]# echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
+--------------------------------------------+------------+
| Table | Size in MB |
+--------------------------------------------+------------+
| nagios_acknowledgements | 0.00 |
| nagios_commands | 0.02 |
| nagios_commenthistory | 34.98 |
| nagios_comments | 0.01 |
| nagios_configfiles | 0.00 |
| nagios_configfilevariables | 0.01 |
| nagios_conninfo | 0.24 |
| nagios_contact_addresses | 0.00 |
| nagios_contact_notificationcommands | 0.31 |
| nagios_contactgroup_members | 0.03 |
| nagios_contactgroups | 0.00 |
| nagios_contactnotificationmethods | 271.90 |
| nagios_contactnotifications | 287.94 |
| nagios_contacts | 0.08 |
| nagios_contactstatus | 0.05 |
| nagios_customvariables | 0.71 |
| nagios_customvariablestatus | 0.74 |
| nagios_dbversion | 0.00 |
| nagios_downtimehistory | 0.00 |
| nagios_eventhandlers | 0.02 |
| nagios_externalcommands | 0.00 |
| nagios_flappinghistory | 18.86 |
| nagios_host_contactgroups | 0.00 |
| nagios_host_contacts | 0.03 |
| nagios_host_parenthosts | 0.00 |
| nagios_hostchecks | 0.00 |
| nagios_hostdependencies | 0.00 |
| nagios_hostescalation_contactgroups | 0.00 |
| nagios_hostescalation_contacts | 0.00 |
| nagios_hostescalations | 0.00 |
| nagios_hostgroup_members | 0.03 |
| nagios_hostgroups | 0.01 |
| nagios_hosts | 0.21 |
| nagios_hoststatus | 0.49 |
| nagios_instances | 0.00 |
| nagios_logentries | 2318.35 |
| nagios_notifications | 513.75 |
| nagios_objects | 4.29 |
| nagios_processevents | 0.34 |
| nagios_programstatus | 0.00 |
| nagios_runtimevariables | 0.00 |
| nagios_scheduleddowntime | 0.00 |
| nagios_service_contactgroups | 0.31 |
| nagios_service_contacts | 0.41 |
| nagios_service_parentservices | 0.00 |
| nagios_servicechecks | 0.00 |
| nagios_servicedependencies | 0.00 |
| nagios_serviceescalation_contactgroups | 0.00 |
| nagios_serviceescalation_contacts | 0.00 |
| nagios_serviceescalations | 0.00 |
| nagios_servicegroup_members | 0.05 |
| nagios_servicegroups | 0.00 |
| nagios_services | 2.54 |
| nagios_servicestatus | 4.98 |
| nagios_statehistory | NULL |
| nagios_systemcommands | 0.03 |
| nagios_timedeventqueue | 0.00 |
| nagios_timedevents | 0.00 |
| nagios_timeperiod_timeranges | 0.26 |
| nagios_timeperiods | 0.06 |
| tbl_command | 0.04 |
| tbl_contact | 0.10 |
| tbl_contactgroup | 0.01 |
| tbl_contacttemplate | 0.01 |
| tbl_domain | 0.01 |
| tbl_host | 0.16 |
| tbl_hostdependency | 0.00 |
| tbl_hostescalation | 0.00 |
| tbl_hostextinfo | 0.00 |
| tbl_hostgroup | 0.01 |
| tbl_hosttemplate | 0.01 |
| tbl_info | 0.13 |
| tbl_lnkContactToCommandHost | 0.00 |
| tbl_lnkContactToCommandService | 0.00 |
| tbl_lnkContactToContactgroup | 0.00 |
| tbl_lnkContactToContacttemplate | 0.02 |
| tbl_lnkContactToVariabledefinition | 0.00 |
| tbl_lnkContactgroupToContact | 0.00 |
| tbl_lnkContactgroupToContactgroup | 0.00 |
| tbl_lnkContacttemplateToCommandHost | 0.00 |
| tbl_lnkContacttemplateToCommandService | 0.00 |
| tbl_lnkContacttemplateToContactgroup | 0.00 |
| tbl_lnkContacttemplateToContacttemplate | 0.00 |
| tbl_lnkContacttemplateToVariabledefinition | 0.00 |
| tbl_lnkHostToContact | 0.02 |
| tbl_lnkHostToContactgroup | 0.00 |
| tbl_lnkHostToHost | 0.00 |
| tbl_lnkHostToHostgroup | 0.00 |
| tbl_lnkHostToHosttemplate | 0.04 |
| tbl_lnkHostToVariabledefinition | 0.03 |
| tbl_lnkHostdependencyToHost_DH | 0.00 |
| tbl_lnkHostdependencyToHost_H | 0.00 |
| tbl_lnkHostdependencyToHostgroup_DH | 0.00 |
| tbl_lnkHostdependencyToHostgroup_H | 0.00 |
| tbl_lnkHostescalationToContact | 0.00 |
| tbl_lnkHostescalationToContactgroup | 0.00 |
| tbl_lnkHostescalationToHost | 0.00 |
| tbl_lnkHostescalationToHostgroup | 0.00 |
| tbl_lnkHostgroupToHost | 0.02 |
| tbl_lnkHostgroupToHostgroup | 0.00 |
| tbl_lnkHosttemplateToContact | 0.00 |
| tbl_lnkHosttemplateToContactgroup | 0.00 |
| tbl_lnkHosttemplateToHost | 0.00 |
| tbl_lnkHosttemplateToHostgroup | 0.00 |
| tbl_lnkHosttemplateToHosttemplate | 0.00 |
| tbl_lnkHosttemplateToVariabledefinition | 0.00 |
| tbl_lnkServiceToContact | 0.34 |
| tbl_lnkServiceToContactgroup | 0.02 |
| tbl_lnkServiceToHost | 0.27 |
| tbl_lnkServiceToHostgroup | 0.00 |
| tbl_lnkServiceToServicegroup | 0.00 |
| tbl_lnkServiceToServicetemplate | 0.42 |
| tbl_lnkServiceToVariabledefinition | 0.34 |
| tbl_lnkServicedependencyToHost_DH | 0.00 |
| tbl_lnkServicedependencyToHost_H | 0.00 |
| tbl_lnkServicedependencyToHostgroup_DH | 0.00 |
| tbl_lnkServicedependencyToHostgroup_H | 0.00 |
| tbl_lnkServicedependencyToService_DS | 0.00 |
| tbl_lnkServicedependencyToService_S | 0.00 |
| tbl_lnkServicedependencyToServicegroup_DS | 0.02 |
| tbl_lnkServicedependencyToServicegroup_S | 0.02 |
| tbl_lnkServiceescalationToContact | 0.00 |
| tbl_lnkServiceescalationToContactgroup | 0.00 |
| tbl_lnkServiceescalationToHost | 0.00 |
| tbl_lnkServiceescalationToHostgroup | 0.00 |
| tbl_lnkServiceescalationToService | 0.00 |
| tbl_lnkServiceescalationToServicegroup | 0.02 |
| tbl_lnkServicegroupToService | 0.05 |
| tbl_lnkServicegroupToServicegroup | 0.00 |
| tbl_lnkServicetemplateToContact | 0.00 |
| tbl_lnkServicetemplateToContactgroup | 0.00 |
| tbl_lnkServicetemplateToHost | 0.00 |
| tbl_lnkServicetemplateToHostgroup | 0.00 |
| tbl_lnkServicetemplateToServicegroup | 0.00 |
| tbl_lnkServicetemplateToServicetemplate | 0.01 |
| tbl_lnkServicetemplateToVariabledefinition | 0.00 |
| tbl_lnkTimeperiodToTimeperiod | 0.00 |
| tbl_logbook | 0.00 |
| tbl_mainmenu | 0.00 |
| tbl_permission | 0.02 |
| tbl_permission_inactive | 0.02 |
| tbl_service | 2.65 |
| tbl_servicedependency | 0.00 |
| tbl_serviceescalation | 0.00 |
| tbl_serviceextinfo | 0.00 |
| tbl_servicegroup | 0.01 |
| tbl_servicetemplate | 0.02 |
| tbl_session | 0.01 |
| tbl_session_locks | 0.00 |
| tbl_settings | 0.00 |
| tbl_submenu | 0.00 |
| tbl_timedefinition | 0.24 |
| tbl_timeperiod | 0.11 |
| tbl_user | 0.01 |
| tbl_variabledefinition | 0.73 |
| xi_auditlog | 3.73 |
| xi_auth_tokens | 1.59 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.00 |
| xi_eventqueue | 1.11 |
| xi_events | 0.51 |
| xi_incidents | 0.02 |
| xi_meta | 13.24 |
| xi_mibs | 0.05 |
| xi_options | 0.02 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.01 |
| xi_usermeta | 1.34 |
| xi_users | 0.08 |
+--------------------------------------------+------------+
Re: CPU utilization high
Posted: Wed Jun 16, 2021 10:03 am
by vtrac
Hi,
How are you doing?
Looks like you have a crashed table in your databased:
Code: Select all
[ERROR] mysqld: Table './nagios/nagios_statehistory' is marked as crashed and last (automatic?) repair failed
Code: Select all
Jun 2 07:46:32 abcdnagprd008 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_statehistory SET instance_id='1', state_time=FROM_UNIXTIME(1622634392), state_time_usec='866491', object_id='29285', state_change='1', state='3', state_type='1', current_check_attempt='3', max_check_attempts='3', last_state='2', last_hard_state='2', output='Remote command execution failed: ssh_exchange_identification: Connection closed by remote host', long_output='''
Please run the below command on your XI command prompt to fix the DB:
Code: Select all
/usr/local/nagiosxi/scripts/repair_databases.sh
If the above command did not fix the crashed table issue, please try the below command to force it:
Code: Select all
systemctl stop mariadb.service
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_statehistory
systemctl start mariadb.service
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php
Please also run the below command on your XI machine:
Code: Select all
ls -l /var/lib/mysql/nagios/nagios_statehistory*
Best Regards,
Vinh
Re: CPU utilization high
Posted: Tue Jun 29, 2021 2:35 am
by deek
Thanks this worked
Re: CPU utilization high
Posted: Tue Jun 29, 2021 9:47 am
by vtrac
Great!! .... locking thread ...