Data Base MariaDB Nagios queueing

[FCC_Nagios_Support]

Good morning,

Since a week when we added 200 new services, the behaviour of Database is being late. For example passive checks that use pipe nagios.cmd are delaying much time and some times the pipe is not different to 0.
I saw in messages:
Jun 19 10:00:08 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 10:00:08 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 10:00:11 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 10:00:12 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 10:00:14 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 10:00:15 a2nagio001p ndo2db: Warning: queue send error, retrying...


Any solution.
Many Thanks in advance
KR

[FCC_Nagios_Support]

More messages:
[root@a2nagio001p log]# tail -f /var/log/messages|grep queue
Jun 19 14:18:08 a2nagio001p ndo2db: Warning: Retrying message send. This can occur because you have too few messages allowed or too few total bytes allowed in message queues. You are currently using 128000 of 512000 messages and 131072000 of 131072000 bytes in the queue. See README for kernel tuning options.
Jun 19 14:18:28 a2nagio001p ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may need to be tuned. See README.
Jun 19 14:18:28 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 14:18:48 a2nagio001p ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may need to be tuned. See README.
Jun 19 14:18:48 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 14:19:08 a2nagio001p ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may need to be tuned. See README.
Jun 19 14:19:08 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 14:19:28 a2nagio001p ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may need to be tuned. See README.
Jun 19 14:19:28 a2nagio001p ndo2db: Warning: queue send error, retrying...
Jun 19 14:19:39 a2nagio001p ndo2db: Message sent to queue.

ANY WAY WE MODIFIED KERNEL EN
[root@a2nagio001p etc]# cat sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
kernel.msgmnb = 131072000
kernel.msgmax = 262144000
kernel.msgmni = 512000
kernel.shmmax = 4294967295
kernel.shmall = 268435456
[root@a2nagio001p etc]#


I NEED TO REBOOT THE SYSTEM TO TAKE EFFECT?
MANY THANKS

[FCC_Nagios_Support]

Thanks

[benjaminsmith]

Hello @FCC_Nagios_Support,

Let's optimize the kernel message settings on this server. Please follow the steps in the guide below to increase the settings.

NDOUtils - Message Queue Exceeded

Be sure to re-start both the nagios and ndo2db services, then clear out the queues (as shown in the kb article) and let me know if the issue is resolved.

Benjamin

[FCC_Nagios_Support]

Many Thanks,
It seems the changes in Kernel does not apply:
[root@a2nagio001p ~]# sysctl -a|grep kernel.msgm
kernel.msgmax = 262144000
kernel.msgmnb = 131072000
kernel.msgmni = 512000
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.ens192.stable_secret"
sysctl: reading key "net.ipv6.conf.ens224.stable_secret"
sysctl: reading key "net.ipv6.conf.ens256.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
[root@a2nagio001p ~]#
I did -p before to apply
but:
Jun 19 20:48:19 a2nagio001p ndo2db: Warning: Retrying message send. This can occur because you have too few messages allowed or too few total bytes allowed in message queues. You are currently using 128000 of 512000 messages and 131072000 of 131072000 bytes in the queue. See README for kernel tuning options.
Jun 19 20:48:39 a2nagio001p ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may need to be tuned. See README.
Jun 19 20:48:39 a2nagio001p ndo2db: Warning: queue send error, retrying...


Thansk and Regards

[benjaminsmith]

Hi,

It seems the changes in Kernel does not apply:

.
Hmm, the sed commands in the kb article should update the file. Were you logged in as root when running those? If not try once more. You can run the following to verify the settings have been updated.

Code: Select all

cat /etc/sysctl.conf

Then, try restarting the whole stack to clear the message queues. For Cent 7, the commands would be as follow (you may have to adjust these for your OS).

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl restart mariadb
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

Once you re-start everything, then let the server run for a while and send over a fresh system profile and I'll check the logs for any issues. Thanks, Benjamin

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket, and then reply to this post to bring it up in the queue.