URGENT: Message "ndo2db: Error: queue recv error."
-
- Posts: 124
- Joined: Wed Feb 18, 2015 10:41 am
URGENT: Message "ndo2db: Error: queue recv error."
Hi Everybody !!!!
We are viewing the next messages ( a lot ) in the "messages log":
root@XXXXXXXXXX journal]# tail -f /var/log/messages
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
At the same time we are experimenting that the performance of the httpp interface is not good.
When we are configurating or editing services or hosts, does not take the changes.
Also some services in the minemap view, change their color in a intermitent way.
Can somebody help me?
Thanks for you support.
We are using:
Nagios XI Version : 2014R2.6
x86_64
Red Hat Enterprise Linux Server release 7.0 (Maipo)
Gnome is not installed
We are viewing the next messages ( a lot ) in the "messages log":
root@XXXXXXXXXX journal]# tail -f /var/log/messages
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
At the same time we are experimenting that the performance of the httpp interface is not good.
When we are configurating or editing services or hosts, does not take the changes.
Also some services in the minemap view, change their color in a intermitent way.
Can somebody help me?
Thanks for you support.
We are using:
Nagios XI Version : 2014R2.6
x86_64
Red Hat Enterprise Linux Server release 7.0 (Maipo)
Gnome is not installed
- Box293
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
- Contact:
Re: URGENT: Message "ndo2db: Error: queue recv error."
What is the output of:
Run this command
Send us this file:
/tmp/mysqld_log.txt
I'll get you to check the amount of free disk space on your XI server. Type the following at the command prompt:
Code: Select all
ipcs -q
Code: Select all
tail /var/log/mysqld.log -n 100 > /tmp/mysqld_log.txt
/tmp/mysqld_log.txt
I'll get you to check the amount of free disk space on your XI server. Type the following at the command prompt:
Code: Select all
df -h
df -i
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 124
- Joined: Wed Feb 18, 2015 10:41 am
Re: URGENT: Message "ndo2db: Error: queue recv error."
Thanks for your response !!!!
ipcs -q
------ Message Queues --------
key msqid owner perms used-bytes messages
0x77010080 983040 nagios 600 0 0
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGSYS-LVRAIZ 3.0G 1.6G 1.5G 52% /
devtmpfs 5.8G 0 5.8G 0% /dev
tmpfs 5.8G 0 5.8G 0% /dev/shm
tmpfs 5.8G 17M 5.8G 1% /run
tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
/dev/mapper/VGSYS-LVUSR 35G 4.0G 32G 12% /usr
/dev/mapper/VGSYS-LVHOME 3.0G 33M 2.9G 2% /home
/dev/mapper/VGSYS-LVVAR 5.9G 586M 5.3G 10% /var
/dev/mapper/VGSYS-LVOPT 3.0G 48M 2.9G 2% /opt
/dev/mapper/VGSYS-LVVARLOG 15G 3.4G 12G 23% /var/log
/dev/mapper/VGPROGPROD-LVSEGURIDAD 2.0G 33M 2.0G 2% /seguridad
/dev/mapper/VGPROGPROD-LVPRODUCCION 2.0G 33M 2.0G 2% /produccion
/dev/mapper/VGSYS-LVTMP 3.0G 33M 2.9G 2% /tmp
/dev/vda1 509M 133M 376M 27% /boot
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VGSYS-LVRAIZ 3072000 2056 3069944 1% /
devtmpfs 1514609 425 1514184 1% /dev
tmpfs 1516770 1 1516769 1% /dev/shm
tmpfs 1516770 580 1516190 1% /run
tmpfs 1516770 13 1516757 1% /sys/fs/cgroup
/dev/mapper/VGSYS-LVUSR 36700160 77636 36622524 1% /usr
/dev/mapper/VGSYS-LVHOME 3072000 15 3071985 1% /home
/dev/mapper/VGSYS-LVVAR 6144000 7035 6136965 1% /var
/dev/mapper/VGSYS-LVOPT 3072000 61 3071939 1% /opt
/dev/mapper/VGSYS-LVVARLOG 15728640 175 15728465 1% /var/log
/dev/mapper/VGPROGPROD-LVSEGURIDAD 2048000 3 2047997 1% /seguridad
/dev/mapper/VGPROGPROD-LVPRODUCCION 2048000 3 2047997 1% /produccion
/dev/mapper/VGSYS-LVTMP 3072000 157 3071843 1% /tmp
/dev/vda1 524288 339 523949 1% /boot
tail /var/log/mysqld.log -n 100 > /tmp/mysqld_log.txt
I can´t find the log mysqld.log !!!!!
ipcs -q
------ Message Queues --------
key msqid owner perms used-bytes messages
0x77010080 983040 nagios 600 0 0
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGSYS-LVRAIZ 3.0G 1.6G 1.5G 52% /
devtmpfs 5.8G 0 5.8G 0% /dev
tmpfs 5.8G 0 5.8G 0% /dev/shm
tmpfs 5.8G 17M 5.8G 1% /run
tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
/dev/mapper/VGSYS-LVUSR 35G 4.0G 32G 12% /usr
/dev/mapper/VGSYS-LVHOME 3.0G 33M 2.9G 2% /home
/dev/mapper/VGSYS-LVVAR 5.9G 586M 5.3G 10% /var
/dev/mapper/VGSYS-LVOPT 3.0G 48M 2.9G 2% /opt
/dev/mapper/VGSYS-LVVARLOG 15G 3.4G 12G 23% /var/log
/dev/mapper/VGPROGPROD-LVSEGURIDAD 2.0G 33M 2.0G 2% /seguridad
/dev/mapper/VGPROGPROD-LVPRODUCCION 2.0G 33M 2.0G 2% /produccion
/dev/mapper/VGSYS-LVTMP 3.0G 33M 2.9G 2% /tmp
/dev/vda1 509M 133M 376M 27% /boot
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VGSYS-LVRAIZ 3072000 2056 3069944 1% /
devtmpfs 1514609 425 1514184 1% /dev
tmpfs 1516770 1 1516769 1% /dev/shm
tmpfs 1516770 580 1516190 1% /run
tmpfs 1516770 13 1516757 1% /sys/fs/cgroup
/dev/mapper/VGSYS-LVUSR 36700160 77636 36622524 1% /usr
/dev/mapper/VGSYS-LVHOME 3072000 15 3071985 1% /home
/dev/mapper/VGSYS-LVVAR 6144000 7035 6136965 1% /var
/dev/mapper/VGSYS-LVOPT 3072000 61 3071939 1% /opt
/dev/mapper/VGSYS-LVVARLOG 15728640 175 15728465 1% /var/log
/dev/mapper/VGPROGPROD-LVSEGURIDAD 2048000 3 2047997 1% /seguridad
/dev/mapper/VGPROGPROD-LVPRODUCCION 2048000 3 2047997 1% /produccion
/dev/mapper/VGSYS-LVTMP 3072000 157 3071843 1% /tmp
/dev/vda1 524288 339 523949 1% /boot
tail /var/log/mysqld.log -n 100 > /tmp/mysqld_log.txt
I can´t find the log mysqld.log !!!!!
- Box293
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
- Contact:
Re: URGENT: Message "ndo2db: Error: queue recv error."
Sorry with Red Hat Enterprise Linux Server release 7.0 (Maipo) I believe mariadb is used.monitoreo1 wrote:tail /var/log/mysqld.log -n 100 > /tmp/mysqld_log.txt
I can´t find the log mysqld.log !!!!!
Can you look in the config file /etc/my.cnf and see what file is used for the log file and then tail 100 lines from that please.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 124
- Joined: Wed Feb 18, 2015 10:41 am
Re: URGENT: Message "ndo2db: Error: queue recv error."
The path is:
[root@XXXXXXXXXX ~]# tail /etc/my.cnf
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
and the output is:
tail /var/log/mariadb/mariadb.log -n 100 > /tmp/mariadb_log.txt
150605 18:04:46 InnoDB: highest supported file format is Barracuda.
150605 18:04:47 InnoDB: Waiting for the background threads to start
150605 18:04:48 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150605 18:04:48 [Note] Plugin 'FEEDBACK' is disabled.
150605 18:04:48 [Note] Server socket created on IP: '0.0.0.0'.
150605 18:04:48 [Note] Event Scheduler: Loaded 0 events
150605 18:04:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150610 9:21:49 [Note] /usr/libexec/mysqld: Normal shutdown
150610 9:21:49 [Note] Event Scheduler: Purging the queue. 0 events
150610 9:21:50 InnoDB: Starting shutdown...
150610 9:21:50 InnoDB: Shutdown completed; log sequence number 1597945
150610 9:21:50 [Note] /usr/libexec/mysqld: Shutdown complete
150610 09:21:51 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150610 09:22:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150610 9:22:56 InnoDB: The InnoDB memory heap is disabled
150610 9:22:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150610 9:22:56 InnoDB: Compressed tables use zlib 1.2.7
150610 9:22:56 InnoDB: Using Linux native AIO
150610 9:22:56 InnoDB: Initializing buffer pool, size = 128.0M
150610 9:22:56 InnoDB: Completed initialization of buffer pool
150610 9:22:56 InnoDB: highest supported file format is Barracuda.
150610 9:22:56 InnoDB: Waiting for the background threads to start
150610 9:22:57 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150610 9:22:57 [Note] Plugin 'FEEDBACK' is disabled.
150610 9:22:57 [Note] Server socket created on IP: '0.0.0.0'.
150610 9:22:58 [Note] Event Scheduler: Loaded 0 events
150610 9:22:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150610 22:17:28 [Note] /usr/libexec/mysqld: Normal shutdown
150610 22:17:28 [Note] Event Scheduler: Purging the queue. 0 events
150610 22:17:28 InnoDB: Starting shutdown...
150610 22:17:29 InnoDB: Shutdown completed; log sequence number 1597945
150610 22:17:29 [Note] /usr/libexec/mysqld: Shutdown complete
150610 22:17:29 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150610 22:23:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150610 22:23:32 InnoDB: The InnoDB memory heap is disabled
150610 22:23:32 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150610 22:23:32 InnoDB: Compressed tables use zlib 1.2.7
150610 22:23:32 InnoDB: Using Linux native AIO
150610 22:23:32 InnoDB: Initializing buffer pool, size = 128.0M
150610 22:23:32 InnoDB: Completed initialization of buffer pool
150610 22:23:32 InnoDB: highest supported file format is Barracuda.
150610 22:23:32 InnoDB: Waiting for the background threads to start
150610 22:23:33 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150610 22:23:33 [Note] Plugin 'FEEDBACK' is disabled.
150610 22:23:33 [Note] Server socket created on IP: '0.0.0.0'.
150610 22:23:33 [Note] Event Scheduler: Loaded 0 events
150610 22:23:33 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150611 18:36:47 [Note] /usr/libexec/mysqld: Normal shutdown
150611 18:36:47 [Note] Event Scheduler: Purging the queue. 0 events
150611 18:36:47 InnoDB: Starting shutdown...
150611 18:36:48 InnoDB: Shutdown completed; log sequence number 1597945
150611 18:36:48 [Note] /usr/libexec/mysqld: Shutdown complete
150611 18:36:48 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150611 18:46:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150611 18:46:30 InnoDB: The InnoDB memory heap is disabled
150611 18:46:30 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150611 18:46:30 InnoDB: Compressed tables use zlib 1.2.7
150611 18:46:30 InnoDB: Using Linux native AIO
150611 18:46:30 InnoDB: Initializing buffer pool, size = 128.0M
150611 18:46:30 InnoDB: Completed initialization of buffer pool
150611 18:46:30 InnoDB: highest supported file format is Barracuda.
150611 18:46:30 InnoDB: Waiting for the background threads to start
150611 18:46:31 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150611 18:46:31 [Note] Plugin 'FEEDBACK' is disabled.
150611 18:46:31 [Note] Server socket created on IP: '0.0.0.0'.
150611 18:46:31 [Note] Event Scheduler: Loaded 0 events
150611 18:46:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150611 18:47:30 [Note] /usr/libexec/mysqld: Normal shutdown
150611 18:47:30 [Note] Event Scheduler: Purging the queue. 0 events
150611 18:47:30 InnoDB: Starting shutdown...
150611 18:47:31 InnoDB: Shutdown completed; log sequence number 1597945
150611 18:47:31 [Note] /usr/libexec/mysqld: Shutdown complete
150611 18:47:31 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150611 18:49:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150611 18:49:56 InnoDB: The InnoDB memory heap is disabled
150611 18:49:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150611 18:49:56 InnoDB: Compressed tables use zlib 1.2.7
150611 18:49:56 InnoDB: Using Linux native AIO
150611 18:49:56 InnoDB: Initializing buffer pool, size = 128.0M
150611 18:49:56 InnoDB: Completed initialization of buffer pool
150611 18:49:56 InnoDB: highest supported file format is Barracuda.
150611 18:49:56 InnoDB: Waiting for the background threads to start
150611 18:49:57 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150611 18:49:57 [Note] Plugin 'FEEDBACK' is disabled.
150611 18:49:57 [Note] Server socket created on IP: '0.0.0.0'.
150611 18:49:57 [Note] Event Scheduler: Loaded 0 events
150611 18:49:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
Today we dont have the first error reported: "Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error."
We applied two action on the server:
a) Increase the cpu and the memory:
Tasks: 440 total, 11 running, 429 sleeping, 0 stopped, 0 zombie
%Cpu0 : 37.6 us, 4.3 sy, 0.0 ni, 57.4 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu1 : 36.0 us, 8.9 sy, 0.0 ni, 54.5 id, 0.3 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu2 : 39.4 us, 4.3 sy, 0.0 ni, 56.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 35.2 us, 6.0 sy, 0.0 ni, 58.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 32.3 us, 4.3 sy, 0.0 ni, 63.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 26.0 us, 4.0 sy, 0.0 ni, 69.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 12134164 total, 3975760 used, 8158404 free, 1328 buffers
KiB Swap: 4095996 total, 0 used, 4095996 free. 2371420 cached Mem
b) We applied the db processes: "VACUUM; VACUUM ANALYZE; VACUUM FULL;"
But today , in a intermitent way:
a) When we are configurating or editing services or hosts, does not take the changes.
b) Also some services in the minemap view, change their color in a intermitent way.
Do you know how resolve this?
Do you know if exist a document that help me to understand what databases use my Nagios installation, and where are the logs that we have to see for prevent incidents ?
Thanks again.
[root@XXXXXXXXXX ~]# tail /etc/my.cnf
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
and the output is:
tail /var/log/mariadb/mariadb.log -n 100 > /tmp/mariadb_log.txt
150605 18:04:46 InnoDB: highest supported file format is Barracuda.
150605 18:04:47 InnoDB: Waiting for the background threads to start
150605 18:04:48 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150605 18:04:48 [Note] Plugin 'FEEDBACK' is disabled.
150605 18:04:48 [Note] Server socket created on IP: '0.0.0.0'.
150605 18:04:48 [Note] Event Scheduler: Loaded 0 events
150605 18:04:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150610 9:21:49 [Note] /usr/libexec/mysqld: Normal shutdown
150610 9:21:49 [Note] Event Scheduler: Purging the queue. 0 events
150610 9:21:50 InnoDB: Starting shutdown...
150610 9:21:50 InnoDB: Shutdown completed; log sequence number 1597945
150610 9:21:50 [Note] /usr/libexec/mysqld: Shutdown complete
150610 09:21:51 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150610 09:22:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150610 9:22:56 InnoDB: The InnoDB memory heap is disabled
150610 9:22:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150610 9:22:56 InnoDB: Compressed tables use zlib 1.2.7
150610 9:22:56 InnoDB: Using Linux native AIO
150610 9:22:56 InnoDB: Initializing buffer pool, size = 128.0M
150610 9:22:56 InnoDB: Completed initialization of buffer pool
150610 9:22:56 InnoDB: highest supported file format is Barracuda.
150610 9:22:56 InnoDB: Waiting for the background threads to start
150610 9:22:57 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150610 9:22:57 [Note] Plugin 'FEEDBACK' is disabled.
150610 9:22:57 [Note] Server socket created on IP: '0.0.0.0'.
150610 9:22:58 [Note] Event Scheduler: Loaded 0 events
150610 9:22:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150610 22:17:28 [Note] /usr/libexec/mysqld: Normal shutdown
150610 22:17:28 [Note] Event Scheduler: Purging the queue. 0 events
150610 22:17:28 InnoDB: Starting shutdown...
150610 22:17:29 InnoDB: Shutdown completed; log sequence number 1597945
150610 22:17:29 [Note] /usr/libexec/mysqld: Shutdown complete
150610 22:17:29 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150610 22:23:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150610 22:23:32 InnoDB: The InnoDB memory heap is disabled
150610 22:23:32 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150610 22:23:32 InnoDB: Compressed tables use zlib 1.2.7
150610 22:23:32 InnoDB: Using Linux native AIO
150610 22:23:32 InnoDB: Initializing buffer pool, size = 128.0M
150610 22:23:32 InnoDB: Completed initialization of buffer pool
150610 22:23:32 InnoDB: highest supported file format is Barracuda.
150610 22:23:32 InnoDB: Waiting for the background threads to start
150610 22:23:33 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150610 22:23:33 [Note] Plugin 'FEEDBACK' is disabled.
150610 22:23:33 [Note] Server socket created on IP: '0.0.0.0'.
150610 22:23:33 [Note] Event Scheduler: Loaded 0 events
150610 22:23:33 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150611 18:36:47 [Note] /usr/libexec/mysqld: Normal shutdown
150611 18:36:47 [Note] Event Scheduler: Purging the queue. 0 events
150611 18:36:47 InnoDB: Starting shutdown...
150611 18:36:48 InnoDB: Shutdown completed; log sequence number 1597945
150611 18:36:48 [Note] /usr/libexec/mysqld: Shutdown complete
150611 18:36:48 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150611 18:46:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150611 18:46:30 InnoDB: The InnoDB memory heap is disabled
150611 18:46:30 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150611 18:46:30 InnoDB: Compressed tables use zlib 1.2.7
150611 18:46:30 InnoDB: Using Linux native AIO
150611 18:46:30 InnoDB: Initializing buffer pool, size = 128.0M
150611 18:46:30 InnoDB: Completed initialization of buffer pool
150611 18:46:30 InnoDB: highest supported file format is Barracuda.
150611 18:46:30 InnoDB: Waiting for the background threads to start
150611 18:46:31 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150611 18:46:31 [Note] Plugin 'FEEDBACK' is disabled.
150611 18:46:31 [Note] Server socket created on IP: '0.0.0.0'.
150611 18:46:31 [Note] Event Scheduler: Loaded 0 events
150611 18:46:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
150611 18:47:30 [Note] /usr/libexec/mysqld: Normal shutdown
150611 18:47:30 [Note] Event Scheduler: Purging the queue. 0 events
150611 18:47:30 InnoDB: Starting shutdown...
150611 18:47:31 InnoDB: Shutdown completed; log sequence number 1597945
150611 18:47:31 [Note] /usr/libexec/mysqld: Shutdown complete
150611 18:47:31 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
150611 18:49:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150611 18:49:56 InnoDB: The InnoDB memory heap is disabled
150611 18:49:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150611 18:49:56 InnoDB: Compressed tables use zlib 1.2.7
150611 18:49:56 InnoDB: Using Linux native AIO
150611 18:49:56 InnoDB: Initializing buffer pool, size = 128.0M
150611 18:49:56 InnoDB: Completed initialization of buffer pool
150611 18:49:56 InnoDB: highest supported file format is Barracuda.
150611 18:49:56 InnoDB: Waiting for the background threads to start
150611 18:49:57 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1597945
150611 18:49:57 [Note] Plugin 'FEEDBACK' is disabled.
150611 18:49:57 [Note] Server socket created on IP: '0.0.0.0'.
150611 18:49:57 [Note] Event Scheduler: Loaded 0 events
150611 18:49:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
Today we dont have the first error reported: "Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error."
We applied two action on the server:
a) Increase the cpu and the memory:
Tasks: 440 total, 11 running, 429 sleeping, 0 stopped, 0 zombie
%Cpu0 : 37.6 us, 4.3 sy, 0.0 ni, 57.4 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu1 : 36.0 us, 8.9 sy, 0.0 ni, 54.5 id, 0.3 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu2 : 39.4 us, 4.3 sy, 0.0 ni, 56.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 35.2 us, 6.0 sy, 0.0 ni, 58.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 32.3 us, 4.3 sy, 0.0 ni, 63.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 26.0 us, 4.0 sy, 0.0 ni, 69.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 12134164 total, 3975760 used, 8158404 free, 1328 buffers
KiB Swap: 4095996 total, 0 used, 4095996 free. 2371420 cached Mem
b) We applied the db processes: "VACUUM; VACUUM ANALYZE; VACUUM FULL;"
But today , in a intermitent way:
a) When we are configurating or editing services or hosts, does not take the changes.
b) Also some services in the minemap view, change their color in a intermitent way.
Do you know how resolve this?
Do you know if exist a document that help me to understand what databases use my Nagios installation, and where are the logs that we have to see for prevent incidents ?
Thanks again.
Re: URGENT: Message "ndo2db: Error: queue recv error."
The mysql (mariadb) database: ndoutils database holds all historical nagios object data, and the nagiosql database holds all object configuration data.
The postgresql database is used for XI interface specific data - dashlets, user, etc.
Logs locations are:
For mysqld: /var/log/mysqld.log
For mariadb: /var/log/mariadb/mariadb.log
For postgresql: /var/lib/pgsql/data/pg_log/postgresql-<day of the week>.log
Hope this helps.
The postgresql database is used for XI interface specific data - dashlets, user, etc.
Logs locations are:
For mysqld: /var/log/mysqld.log
For mariadb: /var/log/mariadb/mariadb.log
For postgresql: /var/lib/pgsql/data/pg_log/postgresql-<day of the week>.log
Hope this helps.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: URGENT: Message "ndo2db: Error: queue recv error."
Hello,
I'm not replying to help original member (as I hope he/she found the solution) but to help other members (as me) that could meet the same trouble today.
I had the following behaviour each time I launched ndo2db service:
- journalctl was registering hundreds of lines with "Error: queue recv error"
- CPU was rising over 90% used, blocking connections, services restart and so on.
- other clue was in /usr/local/nagios/var/nagios.log : "ndomod: Still unable to reconnect to data sink."
I checked nearly all nagios, ndomod and ndo2db config files but found no solution.
I solved my trouble by doing 2 things :
- re-install from scratch ndoutils-2.0.0
- checking Nagvis configuration.
During configuration steps of sources, I used option --with-mysql-lib=/usr/lib/mysql
As this directory was in fact on path /usr/lib64/mysql I changed this option.
Not sure that this was the issue as the /lib/ directory was checked without error, but...
As this broker was installed to be able to use Nagvis, I also checked Nagvis configuration files. I found an error in the user/pwd stored in the config file located at /usr/local/nagvis/etc/nagvis.ini.php
The pwd was wrong, so maybe Nagvis was trying to connect to MySQL DB without any success, keeping all CPU for this action.
Final state is that errors disappeared from journalctl and nagios.log, and CPU fall under 10% used. Enough for me.
Software versions :
- CentOS 7
- Nagios 4.1.1
- ndoutils-2.0.0
- nagvis-1.8.2
I'm not replying to help original member (as I hope he/she found the solution) but to help other members (as me) that could meet the same trouble today.
I had the following behaviour each time I launched ndo2db service:
- journalctl was registering hundreds of lines with "Error: queue recv error"
- CPU was rising over 90% used, blocking connections, services restart and so on.
- other clue was in /usr/local/nagios/var/nagios.log : "ndomod: Still unable to reconnect to data sink."
I checked nearly all nagios, ndomod and ndo2db config files but found no solution.
I solved my trouble by doing 2 things :
- re-install from scratch ndoutils-2.0.0
- checking Nagvis configuration.
During configuration steps of sources, I used option --with-mysql-lib=/usr/lib/mysql
As this directory was in fact on path /usr/lib64/mysql I changed this option.
Not sure that this was the issue as the /lib/ directory was checked without error, but...
As this broker was installed to be able to use Nagvis, I also checked Nagvis configuration files. I found an error in the user/pwd stored in the config file located at /usr/local/nagvis/etc/nagvis.ini.php
The pwd was wrong, so maybe Nagvis was trying to connect to MySQL DB without any success, keeping all CPU for this action.
Final state is that errors disappeared from journalctl and nagios.log, and CPU fall under 10% used. Enough for me.
Software versions :
- CentOS 7
- Nagios 4.1.1
- ndoutils-2.0.0
- nagvis-1.8.2
Re: URGENT: Message "ndo2db: Error: queue recv error."
Thanks for sharing your findings @chris91!
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
-
- Posts: 1
- Joined: Tue Sep 13, 2016 10:47 am
Re: URGENT: Message "ndo2db: Error: queue recv error."
I was getting same error not exactly but something like that and i have fixed it by re-configuring all parameters.monitoreo1 wrote:Hi Everybody !!!!
We are viewing the next messages ( a lot ) in the "messages log":
root@XXXXXXXXXX journal]# tail -f /var/log/messages
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
Jun 11 15:25:36 XXXXXXXXXX ndo2db: Error: queue recv error.
At the same time we are experimenting that the performance of the httpp interface is not good.
When we are configurating or editing services or hosts, does not take the changes.
Also some services in the minemap view, change their color in a intermitent way.
Can somebody help me?
Thanks for you support.
We are using:
Nagios XI Version : 2014R2.6
x86_64
Red Hat Enterprise Linux Server release 7.0 (Maipo)
Gnome is not installed
Re: URGENT: Message "ndo2db: Error: queue recv error."
Appreciate all the replies, but as this thread is quite old I will be locking it.
Former Nagios employee