ndo2db is running out of kernel queue space.

[jsmelser]

So I got this:

Mar 18 16:24:14 <servname> ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may need to be tuned. See README.
Mar 18 16:24:14 <servname> ndo2db: Warning: queue send error, retrying...

message queue, of course, is full:

------ Message Queues --------
key msqid owner perms used-bytes messages
0x8f000080 17661953 nagios 600 262144000 256000

I have the settings of:
kernel.msgmnb = 262144000
kernel.msgmax = 262144000
kernel.msgmni = 512000


I am thinking of double all the values. Is that ok or are there other settings I need to worry about?

[ssax]

Please PM me a copy of your profile, you can download it from Admin > System Profile > Download Profile.

I would only double these:

Code: Select all

kernel.msgmnb = 524288000
kernel.msgmax = 524288000

Run these commands as root as well and post the output:

Code: Select all

ulimit -a
su - nagios
ulimit -a

[jsmelser]

I submitted the profile to your inbox. Thanks for looking.

[root@prvlopsnsrv01 ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 47216
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 47216
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@prvlopsnsrv01 ~]# su - nagios
Last login: Mon Mar 18 04:34:52 MDT 2019
Last failed login: Tue Mar 19 10:13:03 MDT 2019
There were 1669 failed login attempts since the last successful login.
[nagios@prvlopsnsrv01 ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 47216
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[ssax]

Root:

Code: Select all

open files (-n) 1024

Nagios:

Code: Select all

open files (-n) 1024
max user processes (-u) 4096

Try increasing your limits to:

Code: Select all

open files 10000
max user processes 8192

You should be able to do that here:

Code: Select all

/etc/security/limits.conf

You may have nproc or other settings defined in /etc/limits.d as well that may affect your changes, make sure one of those files is not overriding your changes. Once you've made the changes, reboot and then verify, the ulimit output for both the root and nagios users.

[ssax]

After reviewing your profile, I'm seeing these errors:

Code: Select all

PHP Warning:  mysqli_real_connect(): Headers and client library minor version mismatch. Headers:50556 Library:100138 in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning:  mysqli_real_connect(): Headers and client library minor version mismatch. Headers:50556 Library:100138 in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125
PHP Warning:  mysqli_real_connect(): Headers and client library minor version mismatch. Headers:50556 Library:100138 in /usr/local/nagiosxi/html/db/adodb/drivers/adodb-mysqli.inc.php on line 125

What is the output of this command?

Code: Select all

rpm -qa | grep "httpd\|php\|mysql\|mariadb"

I see you're running XI 5.5.2 as well, once we get your version mismatch issue fixed you should upgrade to the latest for some performance improvements.

[jsmelser]

Added -i to get mariadb.

We dont use mysql, we use mariadb here.

[nagios@prvlopsnsrv01 ~]$ rpm -qa | grep -i "httpd\|php\|mysql\|mariadb"
perl-DBD-MySQL-4.023-6.el7.x86_64
MariaDB-shared-10.2.22-1.el7.centos.x86_64
php-gd-5.4.16-46.el7.x86_64
php-process-5.4.16-46.el7.x86_64
php-pdo-5.4.16-46.el7.x86_64
httpd-2.4.6-88.el7.x86_64
MySQL-python-1.2.5-1.el7.x86_64
php-pear-1.9.4-21.el7.noarch
MariaDB-client-10.2.22-1.el7.centos.x86_64
MariaDB-compat-10.2.22-1.el7.centos.x86_64
php-common-5.4.16-46.el7.x86_64
php-mysql-5.4.16-46.el7.x86_64
php-snmp-5.4.16-46.el7.x86_64
php-5.4.16-46.el7.x86_64
php-pgsql-5.4.16-46.el7.x86_64
php-mcrypt-5.4.16-7.el7.x86_64
php-ldap-5.4.16-46.el7.x86_64
php-xml-5.4.16-46.el7.x86_64
php-pecl-ssh2-0.12-1.el7.x86_64
rsyslog-mysql-8.24.0-34.el7.x86_64
php-mbstring-5.4.16-46.el7.x86_64
php-cli-5.4.16-46.el7.x86_64
php-pear-HTML-Template-IT-1.3.0-2.el5.noarch
MariaDB-devel-10.2.22-1.el7.centos.x86_64
httpd-tools-2.4.6-88.el7.x86_64
MariaDB-common-10.2.22-1.el7.centos.x86_64
php-mssql-5.4.16-7.el7.x86_64

[ssax]

Try doing this:
- Note: If it is going to remove more packages than just php-mysql and nagiosxi-deps, please send them before removing them

Code: Select all

yum remove php-mysql
yum install php-mysqlnd

Then restart apache:

Code: Select all

service httpd restart

[jsmelser]

oh sweet, I did not realize that was a thing. Done and its fine, thanks!

I cant reboot until later tonight so I will do the above tonight.

[lmiltchev]

Sounds good. Let us know if you have any more questions/issues.

[jsmelser]

To update on this:

I figured out there was an underlying mariadb issue. Well, more, I am forcing nagios to use galera cluster which technically is not supported.

Nagios runs a script for db maintenance that runs optimizes that galera cluster does NOT like. This freaks out maxscale which intern, makes nagios unhappy. By the time nagios is getting back going again, optimize runs again and its starts all over. (its take nagios 10-15 minutes for it to get things all sorted back out in the back end. This cause the message cue's to build up.

I ended up changing the db maintenance code to stop doing optimize tables and things have been very smooth since.