Problem Description
In Nagios you experience the following symptoms:
ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README.
ndo2db: Warning: queue send error, retrying...
In addition to this, you may see multiple queues for the nagios user when executing the following command:
ipcs -q
The following output is produced:
------ Message Queues --------
key msqid owner perms used-bytes messages
0xee070002 1409024 nagios 600 100672512 98313
0x50070002 1441793 nagios 600 0 0
NDOUtils uses the operating system kernel message queue. As the amount of messages increases the kernel settings need to be tuned to allow more messages to be queued and processed.
First identify the values you are currently using:
grep 'kernel.msgmnb' /etc/sysctl.conf
grep 'kernel.msgmax' /etc/sysctl.conf
grep 'kernel.msgmni' /etc/sysctl.conf
The following output is produced (or similar):
kernel.msgmnb = 131072000
kernel.msgmax = 131072000
kernel.msgmni = 256000
If the settings are not already defined then no output will be displayed for that command and will need to be defined in the /etc/sysctl.conf file.
For msgmnb and msgmax the same value should be used for both. Recommended values are:
For msgmni the recommended value is:
Values higher than these may not be the solution to your problem unless you have a high performance server.
For msgmnb and msgmax the following commands will update /etc/sysctl.conf with increased values. This example will increase them to 262144000.
sed -i 's/^kernel\.msgmnb.*/kernel\.msgmnb = 262144000/g' /etc/sysctl.conf
sed -i 's/^kernel\.msgmax.*/kernel\.msgmax = 262144000/g' /etc/sysctl.conf
The following commands are for the msgmni option. For the grep command you executed previously:
If it did not return output, this command will add the setting to the /etc/sysctl.conf file:
echo 'kernel.msgmni = 512000' >> /etc/sysctl.conf
If it did return output, this command will update the setting in the /etc/sysctl.conf file:
sed -i 's/^kernel\.msgmni.*/kernel\.msgmni = 512000/g' /etc/sysctl.conf
After making those changes, execute the following command:
sysctl -p
The following output is produced (or similar):
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 262144000
kernel.msgmax = 262144000
kernel.shmmax = 4294967295
kernel.shmall = 268435456
kernel.msgmni = 512000
You can see the increased values have been applied to the kernel.
You need to restart services using the commands below:
RHEL 6 | CentOS 6 | Oracle Linux 6 | Ubuntu 14
service nagios stop
service ndo2db restart
service nagios start
RHEL 7 | CentOS 7 | Oracle Linux 7 | Debian | Ubuntu 16/18
systemctl stop nagios.service
systemctl restart ndo2db.service
systemctl start nagios.service
Once you have completed these steps you should check the message queues by executing the following command:
ipcs -q
If you see more than one queue for the user nagios execute the following command to clear the queues:
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
You can then watch the queues for 10-15 minutes to ensure they are being processed:
watch ipcs -q
To stop watching the queues press Ctrl + C on the keyboard.
If you find the message queue is not being processed quickly enough the problem may be related to MySQL / MariaDB. Make sure that the DB server has enough CPU and memory resources and if the DB server is on the same server as the Nagios server you should look at offloading the DB to a dedicated server.
For any support related questions please visit the Nagios Support Forums at:
http://support.nagios.com/forum/
Article ID: 139
Created On: Thu, Jan 21, 2016 at 11:13 PM
Last Updated On: Tue, Dec 18, 2018 at 5:29 PM
Authored by: tlea
Online URL: https://support.nagios.com/kb/article/ndoutils-message-queue-exceeded-139.html