ndo2db: Warning: queue send error, retrying...

[venkitesh]

hi Team,

Daily nagios xi is crashing with the error message ndo2db: Warning: queue send error, retrying...

Nagiosxi Version : 2014R2.6

OS: CentOS release 6.5 64 bit

RAM: 6GB

CPU: 4Cores.

Total number of hosts: 620
Total number of services: 300

Sysctl.conf

kernel.msgmnb = 131072000

# Controls the maximum size of a message, in bytes
kernel.msgmax = 131072000

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 268435456

kernel.msgmni = 256000

ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 46668
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 46668
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

After making changes i did a reboot also. Daily at night it is crashing with this error message "ndo2db: Warning: queue send error, retrying..."

Kindly help

[abrist]

Is XI actually crashing? Or is that error just present in the logs?
Does this happen at the same time every night? If so, are db backups/etc being run at that time?

[venkitesh]

hi abrist,

the exact requirement is mentioned in this post.

http://support.nagios.com/forum/viewtop ... 9&p=106800

Please help

[tgriep]

Could you run this and post back the results?

Code: Select all

cat /proc/sys/kernel/msgmni

Also, could you post more of the error log?

[venkitesh]

cat /proc/sys/kernel/msgmni

256000

Now the issue is without doing any thing load came back to normal.

but no service checks are happening.

also last service checks are showing last weeks date.

nagios: Auto-save of retention data completed successfully

Please help .

[tgriep]

Could you run this and post back the results?

Code: Select all

service ndo2db status

[tmcdonald]

This issue is similar to one that has been affecting several users recently. We are not yet sure of the cause, so we are gathering information to try and determine a common attribute across all affected systems.

Please send us an updated system profile if possible. Go to Admin -> System Profile and click the blue "Download Profile" button. Then attach that profile.zip file to a PM to myself.

Please also run the following commands:

Code: Select all

ipcs -q >> /tmp/nagios.txt
/usr/local/nagios/bin/ndo2db --version >> /tmp/nagios.txt
/usr/local/nagios/bin/nagios --version >> /tmp/nagios.txt
ps -ef | grep ndo >> /tmp/nagios.txt
ps -ef | grep nagios.cfg >> /tmp/nagios.txt
yum list installed >> /tmp/nagios.txt

And also send me the /tmp/nagios.txt file.

[mkbansal0588]

We are also experiencing the same issue. Do we have any ETA on fix?

[tmcdonald]

No ETA, but I have replied to the ticket you have open.