Page 3 of 4

Re: Host Detail Page - High CPU usage

Posted: Mon Jul 27, 2015 2:31 pm
by tgriep
Could you post the following file?

Code: Select all

/usr/local/nagios/etc/ndo2db.cfg

Re: Host Detail Page - High CPU usage

Posted: Mon Jul 27, 2015 2:38 pm
by TBT
/usr/local/nagios/etc/ndo2db.cfg

Code: Select all

#####################################################################
# NDO2DB DAEMON CONFIG FILE
#####################################################################


lock_file=/usr/local/nagios/var/ndo2db.lock

ndo2db_user=nagios
ndo2db_group=nagios

socket_type=unix

socket_name=/usr/local/nagios/var/ndo.sock

tcp_port=5668


db_servertype=mysql
db_host=xxx.xxx.xxx.xxx
db_port=3306

db_name=nagios4
db_prefix=nagios_

db_user=xxxxx
db_pass=xxxxx



## TABLE TRIMMING OPTIONS
# Several database tables containing Nagios event data can become quite large
# over time.  Most admins will want to trim these tables and keep only a
# certain amount of data in them.  The options below are used to specify the
# age (in MINUTES) that data should be allowd to remain in various tables
# before it is deleted.  Using a value of zero (0) for any value means that
# that particular table should NOT be automatically trimmed.

# Keep timed events for 24 hours
max_timedevents_age=1440

# Keep system commands for 1 week
max_systemcommands_age=10080

# Keep service checks for 1 week
max_servicechecks_age=10080

# Keep host checks for 1 week
max_hostchecks_age=10080

# Keep event handlers for 31 days
max_eventhandlers_age=44640

# DEBUG LEVEL
# This option determines how much (if any) debugging information will
# be written to the debug file.  OR values together to log multiple
# types of information.
# Values: -1 = Everything
#          0 = Nothing
#          1 = Process info
#          2 = SQL queries

debug_level=0



# DEBUG VERBOSITY
# This option determines how verbose the debug log out will be.
# Values: 0 = Brief output
#         1 = More detailed
#         2 = Very detailed

debug_verbosity=1



# DEBUG FILE
# This option determines where the daemon should write debugging information.

debug_file=/usr/local/nagios/var/ndo2db.debug



# MAX DEBUG FILE SIZE
# This option determines the maximum size (in bytes) of the debug file.  If
# the file grows larger than this size, it will be renamed with a .old
# extension.  If a file already exists with a .old extension it will
# automatically be deleted.  This helps ensure your disk space usage doesn't
# get out of control when debugging.

max_debug_file_size=1000000


Re: Host Detail Page - High CPU usage

Posted: Mon Jul 27, 2015 4:49 pm
by tgriep
Could you run the following from the Nagios XI server and post the output?

Code: Select all

mysqlcheck -f -r -u <db_user> -p<db_pass> --databases nagios -h xxx.xxx.xxx.xxx
Replace <db_user> with the username assigned to the offsite MYSQL database
Replace <db_pass> with the password assigned to the offsite MYSQL database
xxx.xxx.xxx.xxx with the IP address of the offsite MYSQL database

Re: Host Detail Page - High CPU usage

Posted: Tue Jul 28, 2015 7:28 am
by TBT
tgriep wrote:Could you run the following from the Nagios XI server and post the output?

Code: Select all

mysqlcheck -f -r -u <db_user> -p<db_pass> --databases nagios -h xxx.xxx.xxx.xxx
I assume you meant nagios4, is this safe to run on a production system?

Re: Host Detail Page - High CPU usage

Posted: Tue Jul 28, 2015 9:23 am
by jdalrymple
TBT wrote:I assume you meant nagios4, is this safe to run on a production system?
Sure he did, and while it's safe to run, it will lock tables and prevent updates while running the checks. Depending on the size of your system and whether or not there are data issues it could prevent updates for minutes.

Alternatively, I think we here are all confused as to whether the mysqld.log output you showed earlier was from the database server or the Nagios server. If we can verify that mysqld.log on the offloaded db server has no errors, then there is no need for the check. The whole scope of problem that you're having has the feeling of database problems though.

Re: Host Detail Page - High CPU usage

Posted: Tue Jul 28, 2015 9:33 am
by TBT
Output request by jdalrymple on Fri Jul 24, 2015 2:29 pm for /var/log/mysqld.log was off of the XI Server.

Output request by tgriep on Mon Jul 27, 2015 1:55 pm for /var/log/mysql.log was off of MySQL Server.

Both posts show no errors preset in either log at time of check. Please advise.

Re: Host Detail Page - High CPU usage

Posted: Tue Jul 28, 2015 5:06 pm
by tgriep
Can you run the database check to see if there is a corrupt table?

Code: Select all

mysqlcheck -f -r -u <db_user> -p<db_pass> --databases nagios4 -h xxx.xxx.xxx.xxx
Also, run the following and post the output here.

Code: Select all

ulimit -a
ipcs -q

Re: Host Detail Page - High CPU usage

Posted: Wed Jul 29, 2015 7:32 am
by TBT
mysqlcheck reports "OK" on all checks.

ulimit -a

Code: Select all

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30488
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30488
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
ipcs -q

Code: Select all

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0xf5000002 8814592    nagios     600        0            0

Re: Host Detail Page - High CPU usage

Posted: Wed Jul 29, 2015 4:10 pm
by jdalrymple
I don't know if it's going to help, but it may at least point us to or from mysql being the culprit, you can add the following you my.cnf (on the remote mysqld host)

Code: Select all

slow_query_log=1
slow_query_log_file=/var/log/mysql_slow.log
in the mysqld section of the file.

You'll have to create the file and give the mysql user rights. This will tell us if you have query problems causing the slowness.

Re: Host Detail Page - High CPU usage

Posted: Wed Jul 29, 2015 4:19 pm
by tgriep
Do you have any custom column components installed on this system that displays extra columns on the host details page?
Do you have any host checks that are returning a massive amount of data?