Acknowldgement is not working

[tgriep]

The output of the ps commands looks OK.
With the added features, there are more connections to the postgres database and that is reflected in the graph so it should not be something to worry about.

[absarcompass]

I can not pin point but the issue is persistent after the new upgrade to 5.5.2
when i ever i made changes to services or host like removing or adding and restart nagios it stop responding and then after a goood 10 15 minutes come back to normality with monitoring and service checks
and at that point in time /var/log/messages has following warnings although ngios mysql and ndo2db status was ok at that time

Code: Select all

Aug 15 08:57:58 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:57:58 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:57:59 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:57:59 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:00 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:00 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:01 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:02 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:03 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:03 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:04 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:04 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:05 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:05 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:06 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:06 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:06 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:06 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:07 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:07 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:08 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:08 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:09 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:09 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:10 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:10 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:11 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:11 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:12 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:12 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:13 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:13 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:14 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:14 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:15 nagiosxi ndo2db: Message sent to queue.
Aug 15 08:58:15 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 08:58:16 nagiosxi ndo2db: Message sent to queue.

can you advise what could be an issue here

[absarcompass]

This looks like a bug to me
when ever i made changes in services or hosts threshold or any value which require nagios service restart, /var/log/messeges gets filed with following error. I have already run the scripts to repair mysql and postgresql databases as mentioned in this thread previously

Code: Select all

Aug 15 13:40:25 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:25 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:26 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:26 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:27 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:27 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:28 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:28 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:29 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:29 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:30 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:30 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:31 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:31 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:32 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:32 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:33 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:33 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:34 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:34 nagiosxi ndo2db: Warning: queue send error, retrying...
Aug 15 13:40:35 nagiosxi ndo2db: Message sent to queue.
Aug 15 13:40:35 nagiosxi ndo2db: Warning: queue send error, retrying...

[ssax]

What is the output of this command?

Code: Select all

ipcs -q | grep nagios

This is what it should show (only one kernel message queue):

Code: Select all

[root@xid nagiosxi]# ipcs -q | grep nagios
0x8a000040 196608     nagios     600        0            0

If it shows more than one then you should run these commands to see if it resolves the issue:

Code: Select all

service nagios stop
killall -9 nagios
service ndo2db stop
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
service ndo2db start
service nagios start

Thank you

[absarcompass]

Thanks for that
yes the output is same

Code: Select all

# ipcs -q

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    
0x92000002 360448     nagios     600        0            0           

it just that when i apply config and restart nagios the messages values goes to thousands and stays like that for more than 10 minutes and at that point in time /var/log/messages start flooding with this

Code: Select all

nagiosxi ndo2db: Warning: queue send error, retrying...
nagiosxi ndo2db: Message sent to queue.

this was not happening before upgrading to version 5.5.2. before upgrading to 5.5.2 restarting nagios or applying config was not an issue and never lagged or delayed the response but now services just stopped being monitor when a nagios restart or config apply command is issued.

[ssax]

This should probably be moved into a ticket so that we can do a remote session, please send in a ticket so that we can work with you from there.

Thank you!

[absarcompass]

Hi All,
I have offloaded the database but on restaring nagios and ndo2db service checks are not working
# ipcs -q

------ Message Queues --------
key msqid owner perms used-bytes messages
0xfa000002 1572864 nagios 600 262144000 256000

#tail -f /var/log/messages
Oct 3 11:11:37 nagiosxi ndo2db: Warning: queue send error, retrying...
Oct 3 11:11:38 nagiosxi ndo2db: Message sent to queue.
Oct 3 11:11:38 nagiosxi ndo2db: Warning: Retrying message send. This can occur because you have too few messages allowed or too few total bytes allowed in message queues. You are currently using 256000 of 512000 messages and 262144000 of 262144000 bytes in the queue. See README for kernel tuning options.
Oct 3 11:11:39 nagiosxi ndo2db: Message sent to queue.
Oct 3 11:11:39 nagiosxi ndo2db: Warning: queue send error, retrying...
Oct 3 11:11:40 nagiosxi ndo2db: Message sent to queue.
Oct 3 11:11:40 nagiosxi ndo2db: Warning: queue send error, retrying...
Oct 3 11:11:41 nagiosxi ndo2db: Message sent to queue.

I can login to Gui and see all the XI System Component Status as green in status But service checks are not working
am i missing something here??
have followed this doc https://assets.nagios.com/downloads/nag ... Server.pdf and can see databases, tables and values in offloaded DB

[absarcompass]

just to add the value from /etc/sysctl.conf

# grep 'kernel.msgmni' /etc/sysctl.conf
kernel.msgmni = 512000

[ssax]

Please PM me a copy of your profile so that I can look through your configuration files, you can download it from Admin > System Profile > Download Profile.

Thank you

[tacolover101]

in addition to what the Wizard @ssax mentioned, this article may apply - https://support.nagios.com/kb/article.php?id=139