ndo2db Error: queue recv error. after logout

[pepe_carlos]

Hi,

I have a error very strange in my nagios installation, after upgrade of operative system from Ubuntu 14.04 to 16.04, appear in syslog the message:
Error: queue recv error. but only when exit the ssh session.

my sysctl.conf has the next configuration:

Code: Select all

kernel.msgmnb = 262144000
kernel.msgmax = 262144000
kernel.shmmax = 4294967295
kernel.shmall = 268435456
kernel.msgmni = 512000

the output for the "ipcs -q" command when error happen is the next:

Code: Select all

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x26000002 2359296    nagios     600        7825408      7642

when is ok is the next:

Code: Select all

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x26000002 2359296    nagios     600        0              0

And the messages in syslog are the next when I perform a logoff ssh from system:

Code: Select all

Sep  9 17:15:01 nagios_server systemd[1]: Stopping User Manager for UID 1000...
Sep  9 17:15:01 nagios_server systemd[2568]: Reached target Shutdown.
Sep  9 17:15:01 nagios_server systemd[2568]: Stopped target Default.
Sep  9 17:15:01 nagios_server systemd[2568]: Stopped target Basic System.
Sep  9 17:15:01 nagios_server systemd[2568]: Stopped target Paths.
Sep  9 17:15:01 nagios_server systemd[2568]: Stopped target Timers.
Sep  9 17:15:01 nagios_server systemd[2568]: Stopped target Sockets.
Sep  9 17:15:01 nagios_server systemd[2568]: Starting Exit the Session...
Sep  9 17:15:01 nagios_server systemd[2568]: Received SIGRTMIN+24 from PID 2855 (kill).
Sep  9 17:15:01 nagios_server systemd[1]: Stopped User Manager for UID 1000.
Sep  9 17:15:01 nagios_server systemd[1]: Removed slice User Slice of nagios.
Sep  9 17:15:01 nagios_server ndo2db: Error: queue recv error.
Sep  9 17:15:03 nagios_server ndo2db: message repeated 102984 times: [ Error: queue recv error.]

Somebody knows what could happen?

Thanks.

[mbellerue]

How many checks are running on your system?

When the error happens, what is the output of

Code: Select all

vmstat 1 5

[pepe_carlos]

I have 496 hosts and 4021 services

The ouput of vmstat when is OK:

Code: Select all

~$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  1      0 15854064 371152 15416152    0    0    34   255   79   18 17  4 78  1  0
 1  0      0 15898916 371152 15416128    0    0     0   156 2369 8260 10 11 79  0  0
 1  0      0 15899544 371152 15416092    0    0     0    80 2210 7310  8 10 82  0  0
 1  0      0 15883416 371152 15416080    0    0     0     0 2344 7679  7 11 82  0  0
 1  0      0 15895752 371152 15416068    0    0     0    88 2556 8574 13 11 76  0  0

ouput of vmstat when error happen:

Code: Select all

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 15794640 371152 15415720    0    0    33   255   86   83 17  4 78  1  0
 5  0      0 15648372 371152 15416480    0    0     0    44 16235 154562 36 16 49  0  0
 4  0      0 15754764 371152 15416380    0    0     0    56 7230 152563 29 16 55  0  0
 8  0      0 15699452 371152 15416892    0    0     0  2156 9961 157660 31 14 55  0  0
 3  1      0 15548716 371152 15417844    0    0     0  4372 17017 167178 39 15 46  0  0

The process has an average of 286 process running (in the last 24 hours), I dont see any changes in number of process when error happens, please see the image.

Thanks.

[pepe_carlos]

Hi,

I think that I found the solution, seems that if the SO upgrade is performed in remote (ssh session) is neccesary perform login in local tty console, once I perform the login in tty console, the problem has not happened again.

[mbellerue]

Very good catch! Thanks for posting it here! Do you want to leave this thread open for a bit in case the issue comes back, or should we close it?

[pepe_carlos]

Please close the post

Thanks