How to improve memory usage

[Nosairis]

Hi all,

I have a concern that Nagios is getting slow, even though the amount of host is no much. Please kindly assist for this. Thanks in advance.


Nagios XI Version : 5.2.7
Nagios 2.6.32-573.22.1.el6.x86_64 x86_64
CentOS release 6.7 (Final)
Gnome is not installed
Apache Information

PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
Server Name:
Server Address:
Server Port:
Date/Time

PHP Timezone: Asia/Kuala_Lumpur
PHP Time:
System Time:
Nagios XI Data

License ends in: VTUTOM

nagios (pid 23331) is running...
NPCD running (pid 1621).
ndo2db (pid 1645) is running...
CPU Load 15: 9.48
Total Hosts: 272
Total Services: 3828

[root@Nagios ~]# top M
top - 15:26:25 up 71 days, 16:02, 1 user, load average: 14.18, 11.57, 8.71
Tasks: 212 total, 4 running, 208 sleeping, 0 stopped, 0 zombie
Cpu(s): 57.0%us, 23.6%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 15.576G total, 11.812G used, 3855.098M free, 90.961M buffers
Swap: 2015.996M total, 47.789M used, 1968.207M free, 8758.184M cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3739 mysql 20 0 2203m 38m 4192 S 130.9 0.2 96221:34 mysqld
23346 nagios 20 0 52820 4304 1056 R 67.4 0.0 16:19.07 ndo2db
18482 nagios 20 0 141m 10m 2236 S 9.9 0.1 0:00.05 check_ifopersta
18526 nagios 20 0 141m 10m 2236 S 9.9 0.1 0:00.05 check_ifopersta
18476 nagios 20 0 141m 10m 2236 S 7.9 0.1 0:00.04 check_ifopersta
18091 nagios 20 0 141m 10m 2236 S 5.9 0.1 0:00.05 check_ifopersta
18528 nagios 20 0 137m 8928 2156 R 5.9 0.1 0:00.03 check_ifopersta
11828 root 20 0 129m 41m 1712 S 2.0 0.3 0:00.50 mrtg
1 root 20 0 19232 1024 872 S 0.0 0.0 0:04.52 init

Thanks

[bwallace]

Implementing a RAM disk will certainly help:
https://assets.nagios.com/downloads/nag ... giosXI.pdf

In the meantime, could you post a screenshots of Admin --> System Status --> Server Statistics and also for Monitoring Engine Status (curious about avg service check execution time)

Lastly, run the following and post the results:
ps -eo pid,comm,%cpu,pcpu,user,nice,cpu,pid,args | sort -rk 3|head -n 30

[rkennedy]

In addition to what @bwallace mentioned - I have seen the culprit in this be invalid SNMP checks because the script doesn't close for a bit of time while it waits for a response. Are your checks that are using check_ifoperstatus checks that finish properly, or are they failing on your machine currently?

[Nosairis]

Hi bwallace,

[root@Nagios hosts]# ps -eo pid,comm,%cpu,pcpu,user,nice,cpu,pid,args | sort -rk 3|head -n 30
PID COMMAND %CPU %CPU USER NI CPU PID COMMAND
3739 mysqld 93.9 93.9 mysql 0 - 3739 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
2347 check_ifopersta 5.0 5.0 nagios 0 - 2347 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.3.180.193 -C Myd1nM0n -k 2004 -v 1
2234 check_ifopersta 5.0 5.0 nagios 0 - 2234 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.25.161 -C Myd1nM0n -k 1 -v 1
1688 check_ifopersta 2.5 2.5 nagios 0 - 1688 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.19.129 -C Myd1nM0n -k 9 -v 1
927 check_ifopersta 1.6 1.6 nagios 0 - 927 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.21.129 -C Myd1nM0n -k 81 -v 1
331 check_ifopersta 1.6 1.6 nagios 0 - 331 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.4.161 -C Myd1nM0n -k 2004 -v 1
23331 nagios 1.6 1.6 nagios 0 - 23331 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
23346 ndo2db 12.6 12.6 nagios 0 - 23346 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
1171 flush-253:0 1.2 1.2 root 0 - 1171 [flush-253:0]
32315 check_ifopersta 1.2 1.2 nagios 0 - 32315 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.25.97 -C Myd1nM0n -k 9 -v 1
18280 httpd 1.2 1.2 apache 0 - 18280 /usr/sbin/httpd
12537 httpd 1.1 1.1 apache 0 - 12537 /usr/sbin/httpd
12534 httpd 1.1 1.1 apache 0 - 12534 /usr/sbin/httpd
2475 check_ifopersta 1.0 1.0 nagios 0 - 2475 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 198.18.0.74 -C vinx -k 399 -v 1
12535 httpd 1.0 1.0 apache 0 - 12535 /usr/sbin/httpd
12533 httpd 1.0 1.0 apache 0 - 12533 /usr/sbin/httpd
12532 httpd 1.0 1.0 apache 0 - 12532 /usr/sbin/httpd
12528 httpd 1.0 1.0 apache 0 - 12528 /usr/sbin/httpd
12536 httpd 0.9 0.9 apache 0 - 12536 /usr/sbin/httpd
12531 httpd 0.9 0.9 apache 0 - 12531 /usr/sbin/httpd
30958 check_ifopersta 0.8 0.8 nagios 0 - 30958 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.22.161 -C Myd1nM0n -k 1 -v 1
30443 check_ifopersta 0.8 0.8 nagios 0 - 30443 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.1.129 -C Myd1nM0n -k 81 -v 1
30253 check_ifopersta 0.5 0.5 nagios 0 - 30253 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.4.161 -C Myd1nM0n -k 9 -v 1
29947 check_ifopersta 0.5 0.5 nagios 0 - 29947 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.21.33 -C Myd1nM0n -k 81 -v 1
29946 check_ifopersta 0.5 0.5 nagios 0 - 29946 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.1.129 -C Myd1nM0n -k 9 -v 1
27182 check_ifopersta 0.5 0.5 nagios 0 - 27182 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.200.9.65 -C Myd1nM0n -k 9 -v 1
26524 check_ifopersta 0.4 0.4 nagios 0 - 26524 /usr/bin/perl -w /usr/local/nagios/libexec/check_ifoperstatus -H 10.3.150.225 -C Myd1nM0n -k 81 -v 1
29164 php 0.3 0.3 nagios 0 - 29164 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
29158 php 0.3 0.3 nagios 0 - 29158 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php

Hi rkennedy ,
How can I check the check_ifoperstatus is finished properly or are they failing on your machine currently?

Thanks

[rkennedy]

How can I check the check_ifoperstatus is finished properly or are they failing on your machine currently?

When you look at the Home page, are your services that are running SNMP checks in an 'UP' state?

[Nosairis]

Hi rkennedy,

sorry for late update.

the check_ifoperstatus is no more be an issues as before after restarted the service. But however the memory usage still very high. thenfore, I didn't follow the suggestion in the pdf. Is there any way to reduce the memory usage? Thanks in advance.

[root@Nagios ~]# top
top - 18:18:05 up 79 days, 18:54, 1 user, load average: 4.57, 5.38, 5.70
Tasks: 240 total, 6 running, 234 sleeping, 0 stopped, 0 zombie
Cpu(s): 57.3%us, 23.6%sy, 0.0%ni, 18.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 16333028k total, 12871324k used, 3461704k free, 97996k buffers
Swap: 2064380k total, 42192k used, 2022188k free, 9795424k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3739 mysql 20 0 2203m 40m 4124 S 160.3 0.3 108557:03 mysqld
6876 nagios 20 0 141m 10m 2236 S 9.9 0.1 0:00.05 check_ifopersta
25359 nagios 20 0 34996 13m 1284 S 9.9 0.1 23:49.40 nagios
25373 nagios 20 0 52820 4304 1032 S 9.9 0.0 175:49.87 ndo2db
6944 nagios 20 0 305m 14m 7276 R 7.9 0.1 0:00.04 php
452 root 20 0 0 0 0 S 2.0 0.0 134:22.35 jbd2/dm-0-8
25364 nagios 20 0 10016 976 664 S 2.0 0.0 2:59.23 nagios
1 root 20 0 19232 1024 872 S 0.0 0.0 0:05.08 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 46:26.59 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 2:19.65 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 1:19.26 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 36:30.67 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/1
9 root 20 0 0 0 0 S 0.0 0.0 1:59.95 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:43.75 watchdog/1

[lmiltchev]

How often do you usually check your hosts/services? Are you using the "default" 5 min check interval or use a lower value?

Can you run the following commands and show us the output?

Code: Select all

tail -30 /var/log/mysqld.log
ps -ef | grep bin/nagios
ipcs -q

In addition to utilizing RAM disk, you could also perform some other actions to "fine tune" Nagios XI, in order to improve performance. Please read our documentation on the topic here:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

[Nosairis]

Hi lmiltchev,

For all the hosts we monitor, we set 1 minute to check.


[root@Nagios ~]# tail -30 /var/log/mysqld.log
160504 12:03:19 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_commands' is marked as crashed and should be repaired
160504 12:03:20 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_commands' is marked as crashed and should be repaired
160504 12:03:20 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_commands' is marked as crashed and should be repaired
160504 12:03:21 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_commands' is marked as crashed and should be repaired
160504 12:03:21 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
160504 12:03:21 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_commands' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_commands' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_commands' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
160504 12:03:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
160504 12:03:23 [Note] /usr/libexec/mysqld: Normal shutdown

160504 12:03:23 [Note] Event Scheduler: Purging the queue. 0 events
160504 12:03:25 InnoDB: Starting shutdown...
160504 12:03:25 InnoDB: Shutdown completed; log sequence number 0 44233
160504 12:03:25 [Note] /usr/libexec/mysqld: Shutdown complete

160504 12:03:25 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160504 12:03:31 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160504 12:03:31 InnoDB: Initializing buffer pool, size = 8.0M
160504 12:03:31 InnoDB: Completed initialization of buffer pool
160504 12:03:31 InnoDB: Started; log sequence number 0 44233
160504 12:03:31 [Note] Event Scheduler: Loaded 0 events
160504 12:03:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
[root@Nagios ~]# ps -ef | grep bin/nagios
nagios 25359 1 1 Jul21 ? 00:48:43 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 25361 25359 0 Jul21 ? 00:06:18 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 25362 25359 0 Jul21 ? 00:06:18 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 25363 25359 0 Jul21 ? 00:06:15 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 25364 25359 0 Jul21 ? 00:06:09 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 25365 25359 0 Jul21 ? 00:06:10 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 25366 25359 0 Jul21 ? 00:06:22 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 25418 25359 0 Jul21 ? 00:00:07 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 26188 24696 0 18:01 pts/1 00:00:00 grep bin/nagios
[root@Nagios ~]# ipcs -q

------ Message Queues --------
key msqid owner perms used-bytes messages
0x3c000002 9076736 nagios 600 0 0

[Box293]

Implementing a RAM Disk should be the first thing on your to-do list.

Do you have a lot of hosts going into a down state frequently? You could look at implementing host_down_disable_service_checks=1 in nagios.cfg.

https://support.nagios.com/kb/article.p ... tegory=164

[Nosairis]

Hi Box293 ,

Thanks for advice, sure will implement it.

btw, for services status when host down, actually we don't have much hosts down, just the service status looks weird.