URGENT: Message "ndo2db: Error: queue recv error."

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
monitoreo1
Posts: 124
Joined: Wed Feb 18, 2015 10:41 am

URGENT: Message "ndo2db: Error: queue recv error."

Post by monitoreo1 »

Hi Everybody !!!!

We are viewing the next messages ( a lot ) in the "messages log":

root@XXXXXXXXXX journal]# tail -f /var/log/messages
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.


At the same time we are experimenting that the performance of the httpp interface is not good.

When we are configurating or editing services or hosts, does not take the changes.

Also some services in the minemap view, change their color in a intermitent way.

Can somebody help me?

Thanks for you support.

We are using:
Nagios XI Version : 2014R2.6
x86_64
Red Hat Enterprise Linux Server release 7.0 (Maipo)
Gnome is not installed
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by Box293 »

What is the output of:

Code: Select all

ipcs -q
Run this command

Code: Select all

tail /var/log/mysqld.log -n 100 > /tmp/mysqld_log.txt
Send us this file:
/tmp/mysqld_log.txt


I'll get you to check the amount of free disk space on your XI server. Type the following at the command prompt:

Code: Select all

df -h
df -i
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
monitoreo1
Posts: 124
Joined: Wed Feb 18, 2015 10:41 am

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by monitoreo1 »

Thanks for your response !!!!

ipcs -q

------ Message Queues --------
key msqid owner perms used-bytes messages
0x77010080 983040 nagios 600 0 0

df -h

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGSYS-LVRAIZ 3.0G 1.6G 1.5G 52% /
devtmpfs 5.8G 0 5.8G 0% /dev
tmpfs 5.8G 0 5.8G 0% /dev/shm
tmpfs 5.8G 17M 5.8G 1% /run
tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
/dev/mapper/VGSYS-LVUSR 35G 4.0G 32G 12% /usr
/dev/mapper/VGSYS-LVHOME 3.0G 33M 2.9G 2% /home
/dev/mapper/VGSYS-LVVAR 5.9G 586M 5.3G 10% /var
/dev/mapper/VGSYS-LVOPT 3.0G 48M 2.9G 2% /opt
/dev/mapper/VGSYS-LVVARLOG 15G 3.4G 12G 23% /var/log
/dev/mapper/VGPROGPROD-LVSEGURIDAD 2.0G 33M 2.0G 2% /seguridad
/dev/mapper/VGPROGPROD-LVPRODUCCION 2.0G 33M 2.0G 2% /produccion
/dev/mapper/VGSYS-LVTMP 3.0G 33M 2.9G 2% /tmp
/dev/vda1 509M 133M 376M 27% /boot


df -i

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VGSYS-LVRAIZ 3072000 2056 3069944 1% /
devtmpfs 1514609 425 1514184 1% /dev
tmpfs 1516770 1 1516769 1% /dev/shm
tmpfs 1516770 580 1516190 1% /run
tmpfs 1516770 13 1516757 1% /sys/fs/cgroup
/dev/mapper/VGSYS-LVUSR 36700160 77636 36622524 1% /usr
/dev/mapper/VGSYS-LVHOME 3072000 15 3071985 1% /home
/dev/mapper/VGSYS-LVVAR 6144000 7035 6136965 1% /var
/dev/mapper/VGSYS-LVOPT 3072000 61 3071939 1% /opt
/dev/mapper/VGSYS-LVVARLOG 15728640 175 15728465 1% /var/log
/dev/mapper/VGPROGPROD-LVSEGURIDAD 2048000 3 2047997 1% /seguridad
/dev/mapper/VGPROGPROD-LVPRODUCCION 2048000 3 2047997 1% /produccion
/dev/mapper/VGSYS-LVTMP 3072000 157 3071843 1% /tmp
/dev/vda1 524288 339 523949 1% /boot


tail /var/log/mysqld.log -n 100 > /tmp/mysqld_log.txt

I can´t find the log mysqld.log !!!!!
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by Box293 »

monitoreo1 wrote:tail /var/log/mysqld.log -n 100 > /tmp/mysqld_log.txt

I can´t find the log mysqld.log !!!!!
Sorry with Red Hat Enterprise Linux Server release 7.0 (Maipo) I believe mariadb is used.

Can you look in the config file /etc/my.cnf and see what file is used for the log file and then tail 100 lines from that please.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
monitoreo1
Posts: 124
Joined: Wed Feb 18, 2015 10:41 am

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by monitoreo1 »

The path is:

[root@XXXXXXXXXX ~]# tail /etc/my.cnf
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

and the output is:

tail /var/log/mariadb/mariadb.log -n 100 > /tmp/mariadb_log.txt

150605 18:04:46 InnoDB: highest supported file format is Barracuda.
150605 18:04:47 InnoDB: Waiting for the background threads to start
150605 18:04:48 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150605 18:04:48 [Note] Plugin 'FEEDBACK' is disabled.
150605 18:04:48 [Note] Server socket created on IP: '0.0.0.0'.
150605 18:04:48 [Note] Event Scheduler: Loaded 0 events
150605 18:04:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150610 9:21:49 [Note] /usr/libexec/mysqld: Normal shutdown

150610 9:21:49 [Note] Event Scheduler: Purging the queue. 0 events
150610 9:21:50 InnoDB: Starting shutdown...
150610 9:21:50 InnoDB: Shutdown completed; log sequence number 1597945
150610 9:21:50 [Note] /usr/libexec/mysqld: Shutdown complete

150610 09:21:51 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150610 09:22:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150610 9:22:56 InnoDB: The InnoDB memory heap is disabled
150610 9:22:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150610 9:22:56 InnoDB: Compressed tables use zlib 1.2.7
150610 9:22:56 InnoDB: Using Linux native AIO
150610 9:22:56 InnoDB: Initializing buffer pool, size = 128.0M
150610 9:22:56 InnoDB: Completed initialization of buffer pool
150610 9:22:56 InnoDB: highest supported file format is Barracuda.
150610 9:22:56 InnoDB: Waiting for the background threads to start
150610 9:22:57 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150610 9:22:57 [Note] Plugin 'FEEDBACK' is disabled.
150610 9:22:57 [Note] Server socket created on IP: '0.0.0.0'.
150610 9:22:58 [Note] Event Scheduler: Loaded 0 events
150610 9:22:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150610 22:17:28 [Note] /usr/libexec/mysqld: Normal shutdown

150610 22:17:28 [Note] Event Scheduler: Purging the queue. 0 events
150610 22:17:28 InnoDB: Starting shutdown...
150610 22:17:29 InnoDB: Shutdown completed; log sequence number 1597945
150610 22:17:29 [Note] /usr/libexec/mysqld: Shutdown complete

150610 22:17:29 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150610 22:23:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150610 22:23:32 InnoDB: The InnoDB memory heap is disabled
150610 22:23:32 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150610 22:23:32 InnoDB: Compressed tables use zlib 1.2.7
150610 22:23:32 InnoDB: Using Linux native AIO
150610 22:23:32 InnoDB: Initializing buffer pool, size = 128.0M
150610 22:23:32 InnoDB: Completed initialization of buffer pool
150610 22:23:32 InnoDB: highest supported file format is Barracuda.
150610 22:23:32 InnoDB: Waiting for the background threads to start
150610 22:23:33 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150610 22:23:33 [Note] Plugin 'FEEDBACK' is disabled.
150610 22:23:33 [Note] Server socket created on IP: '0.0.0.0'.
150610 22:23:33 [Note] Event Scheduler: Loaded 0 events
150610 22:23:33 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150611 18:36:47 [Note] /usr/libexec/mysqld: Normal shutdown

150611 18:36:47 [Note] Event Scheduler: Purging the queue. 0 events
150611 18:36:47 InnoDB: Starting shutdown...
150611 18:36:48 InnoDB: Shutdown completed; log sequence number 1597945
150611 18:36:48 [Note] /usr/libexec/mysqld: Shutdown complete

150611 18:36:48 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150611 18:46:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150611 18:46:30 InnoDB: The InnoDB memory heap is disabled
150611 18:46:30 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150611 18:46:30 InnoDB: Compressed tables use zlib 1.2.7
150611 18:46:30 InnoDB: Using Linux native AIO
150611 18:46:30 InnoDB: Initializing buffer pool, size = 128.0M
150611 18:46:30 InnoDB: Completed initialization of buffer pool
150611 18:46:30 InnoDB: highest supported file format is Barracuda.
150611 18:46:30 InnoDB: Waiting for the background threads to start
150611 18:46:31 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150611 18:46:31 [Note] Plugin 'FEEDBACK' is disabled.
150611 18:46:31 [Note] Server socket created on IP: '0.0.0.0'.
150611 18:46:31 [Note] Event Scheduler: Loaded 0 events
150611 18:46:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150611 18:47:30 [Note] /usr/libexec/mysqld: Normal shutdown

150611 18:47:30 [Note] Event Scheduler: Purging the queue. 0 events
150611 18:47:30 InnoDB: Starting shutdown...
150611 18:47:31 InnoDB: Shutdown completed; log sequence number 1597945
150611 18:47:31 [Note] /usr/libexec/mysqld: Shutdown complete

150611 18:47:31 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150611 18:49:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150611 18:49:56 InnoDB: The InnoDB memory heap is disabled
150611 18:49:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150611 18:49:56 InnoDB: Compressed tables use zlib 1.2.7
150611 18:49:56 InnoDB: Using Linux native AIO
150611 18:49:56 InnoDB: Initializing buffer pool, size = 128.0M
150611 18:49:56 InnoDB: Completed initialization of buffer pool
150611 18:49:56 InnoDB: highest supported file format is Barracuda.
150611 18:49:56 InnoDB: Waiting for the background threads to start
150611 18:49:57 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150611 18:49:57 [Note] Plugin 'FEEDBACK' is disabled.
150611 18:49:57 [Note] Server socket created on IP: '0.0.0.0'.
150611 18:49:57 [Note] Event Scheduler: Loaded 0 events
150611 18:49:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server


Today we dont have the first error reported: "Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error."

We applied two action on the server:

a) Increase the cpu and the memory:

Tasks: 440 total, 11 running, 429 sleeping, 0 stopped, 0 zombie
%Cpu0 : 37.6 us, 4.3 sy, 0.0 ni, 57.4 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu1 : 36.0 us, 8.9 sy, 0.0 ni, 54.5 id, 0.3 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu2 : 39.4 us, 4.3 sy, 0.0 ni, 56.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 35.2 us, 6.0 sy, 0.0 ni, 58.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 32.3 us, 4.3 sy, 0.0 ni, 63.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 26.0 us, 4.0 sy, 0.0 ni, 69.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 12134164 total, 3975760 used, 8158404 free, 1328 buffers
KiB Swap: 4095996 total, 0 used, 4095996 free. 2371420 cached Mem

b) We applied the db processes: "VACUUM; VACUUM ANALYZE; VACUUM FULL;"


But today , in a intermitent way:

a) When we are configurating or editing services or hosts, does not take the changes.

b) Also some services in the minemap view, change their color in a intermitent way.


Do you know how resolve this?

Do you know if exist a document that help me to understand what databases use my Nagios installation, and where are the logs that we have to see for prevent incidents ?

Thanks again.
User avatar
lmiltchev
Former Nagios Staff
Posts: 13587
Joined: Mon May 23, 2011 12:15 pm

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by lmiltchev »

The mysql (mariadb) database: ndoutils database holds all historical nagios object data, and the nagiosql database holds all object configuration data.
The postgresql database is used for XI interface specific data - dashlets, user, etc.

Logs locations are:
For mysqld: /var/log/mysqld.log
For mariadb: /var/log/mariadb/mariadb.log
For postgresql: /var/lib/pgsql/data/pg_log/postgresql-<day of the week>.log

Hope this helps.
Be sure to check out our Knowledgebase for helpful articles and solutions!
chris91
Posts: 5
Joined: Wed Jul 29, 2015 6:02 am

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by chris91 »

Hello,

I'm not replying to help original member (as I hope he/she found the solution) but to help other members (as me) that could meet the same trouble today.

I had the following behaviour each time I launched ndo2db service:
- journalctl was registering hundreds of lines with "Error: queue recv error"
- CPU was rising over 90% used, blocking connections, services restart and so on.
- other clue was in /usr/local/nagios/var/nagios.log : "ndomod: Still unable to reconnect to data sink."

I checked nearly all nagios, ndomod and ndo2db config files but found no solution.
I solved my trouble by doing 2 things :
- re-install from scratch ndoutils-2.0.0
- checking Nagvis configuration.

During configuration steps of sources, I used option --with-mysql-lib=/usr/lib/mysql
As this directory was in fact on path /usr/lib64/mysql I changed this option.
Not sure that this was the issue as the /lib/ directory was checked without error, but...

As this broker was installed to be able to use Nagvis, I also checked Nagvis configuration files. I found an error in the user/pwd stored in the config file located at /usr/local/nagvis/etc/nagvis.ini.php
The pwd was wrong, so maybe Nagvis was trying to connect to MySQL DB without any success, keeping all CPU for this action.

Final state is that errors disappeared from journalctl and nagios.log, and CPU fall under 10% used. Enough for me. :)

Software versions :
- CentOS 7
- Nagios 4.1.1
- ndoutils-2.0.0
- nagvis-1.8.2
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by mcapra »

Thanks for sharing your findings @chris91!
Former Nagios employee
https://www.mcapra.com/
Jon albert
Posts: 1
Joined: Tue Sep 13, 2016 10:47 am

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by Jon albert »

monitoreo1 wrote:Hi Everybody !!!!

We are viewing the next messages ( a lot ) in the "messages log":

root@XXXXXXXXXX journal]# tail -f /var/log/messages
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.


At the same time we are experimenting that the performance of the httpp interface is not good.

When we are configurating or editing services or hosts, does not take the changes.

Also some services in the minemap view, change their color in a intermitent way.

Can somebody help me?

Thanks for you support.

We are using:
Nagios XI Version : 2014R2.6
x86_64
Red Hat Enterprise Linux Server release 7.0 (Maipo)
Gnome is not installed
I was getting same error not exactly but something like that and i have fixed it by re-configuring all parameters.
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: URGENT: Message "ndo2db: Error: queue recv error."

Post by tmcdonald »

Appreciate all the replies, but as this thread is quite old I will be locking it.
Former Nagios employee
Locked