NDOUtils - Message Queue Exceeded

Problem Description

In Nagios you experience the following symptoms:

Missing hosts or services or status data
Takes a very long time to restart the Nagios process
Unusually high CPU load

A flood of messages in the /var/log/messages related to ndo2db like:

ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README.
ndo2db: Warning: queue send error, retrying...

In addition to this, you may see multiple queues for the nagios user when executing the following command:

ipcs -q

The following output is produced:

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0xee070002 1409024    nagios     600        100672512    98313
0x50070002 1441793    nagios     600        0            0

Explanation

NDOUtils uses the operating system kernel message queue. As the amount of messages increases the kernel settings need to be tuned to allow more messages to be queued and processed.

Resolving The Problem

First identify the values you are currently using:

grep 'kernel.msgmnb' /etc/sysctl.conf
grep 'kernel.msgmax' /etc/sysctl.conf
grep 'kernel.msgmni' /etc/sysctl.conf

The following output is produced (or similar):

kernel.msgmnb = 131072000
kernel.msgmax = 131072000
kernel.msgmni = 256000

If the settings are not already defined then no output will be displayed for that command and will need to be defined in the /etc/sysctl.conf file.

For msgmnb and msgmax the same value should be used for both. Recommended values are:

131072000
262144000

For msgmni the recommended value is:

512000

Values higher than these may not be the solution to your problem unless you have a high performance server.

For msgmnb and msgmax the following commands will update /etc/sysctl.conf with increased values. This example will increase them to 262144000.

sed -i 's/^kernel\.msgmnb.*/kernel\.msgmnb = 262144000/g' /etc/sysctl.conf
sed -i 's/^kernel\.msgmax.*/kernel\.msgmax = 262144000/g' /etc/sysctl.conf

The following commands are for the msgmni option. For the grep command you executed previously:

If it did not return output, this command will add the setting to the /etc/sysctl.conf file:

echo 'kernel.msgmni = 512000' >> /etc/sysctl.conf

If it did return output, this command will update the setting in the /etc/sysctl.conf file:

sed -i 's/^kernel\.msgmni.*/kernel\.msgmni = 512000/g' /etc/sysctl.conf

After making those changes, execute the following command:

sysctl -p

The following output is produced (or similar):

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 262144000
kernel.msgmax = 262144000
kernel.shmmax = 4294967295
kernel.shmall = 268435456
kernel.msgmni = 512000

You can see the increased values have been applied to the kernel.

You need to restart services using the commands below:

RHEL 6 | CentOS 6 | Oracle Linux 6 | Ubuntu 14

service nagios stop
service ndo2db restart
service nagios start

RHEL 7 | CentOS 7 | Oracle Linux 7 | Debian | Ubuntu 16/18

systemctl stop nagios.service
systemctl restart ndo2db.service
systemctl start nagios.service

Once you have completed these steps you should check the message queues by executing the following command:

ipcs -q

If you see more than one queue for the user nagios execute the following command to clear the queues:

for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done

You can then watch the queues for 10-15 minutes to ensure they are being processed:

watch ipcs -q

To stop watching the queues press Ctrl + C on the keyboard.

Other Recommendations

If you find the message queue is not being processed quickly enough the problem may be related to MySQL / MariaDB. Make sure that the DB server has enough CPU and memory resources and if the DB server is on the same server as the Nagios server you should look at offloading the DB to a dedicated server.

Final Thoughts

For any support related questions please visit the Nagios Support Forums at:

http://support.nagios.com/forum/