Apply configuration Taking Long time

[ericssonvietnam]

Apply configuration does not complete

What i need to check.

[tgriep]

Take a look at this KB article for possible solutions to the Apply Config taking a long time to run or not at all.
https://support.nagios.com/kb/article/n ... letes.html

[ericssonvietnam]

Take a look at this KB article for possible solutions to the Apply Config taking a long time to run or not at all.
https://support.nagios.com/kb/article/n ... letes.html

[root@nagiosxi ~]# service nagios restart
Running configuration check...
Stopping nagios:..........
Warning - nagios did not exit in a timely manner
Starting nagios: done.

No alarms are updating as if now can you suggest how i can fix it.

below are the message logs for the application.

Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi rsyslogd-2177: imuxsock begins to drop messages from pid 9229 due to rate-limiting
Jun 22 02:01:39 nagiosxi ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README.
Jun 22 02:01:39 nagiosxi ndo2db: Warning: queue send error, retrying...
Jun 22 02:01:40 nagiosxi ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README.
Jun 22 02:01:40 nagiosxi ndo2db: Warning: queue send error, retrying...

[tgriep]

It looks like you have a couple of issues that have to be resolved.
The did not exit in a timely manner could be fixed by this KB article
https://support.nagios.com/kb/article/n ... anner.html

And the ndo2db errors could be fixed by this KB article.
https://support.nagios.com/kb/article/n ... eeded.html

If your system has a high number of host and service checks, that is one of the causes for the errors you are having and the above fixes should help out.

[ericssonvietnam]

Take a look at this KB article for possible solutions to the Apply Config taking a long time to run or not at all.
https://support.nagios.com/kb/article/n ... letes.html

[root@nagiosxi ~]# service nagios restart
Running configuration check...
Stopping nagios:..........
Warning - nagios did not exit in a timely manner
Starting nagios: done.

No alarms are updating as if now can you suggest how i can fix it.

below are the message logs for the application.

Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi ndo2db: Error: queue recv error.
Jun 22 02:01:37 nagiosxi rsyslogd-2177: imuxsock begins to drop messages from pid 9229 due to rate-limiting
Jun 22 02:01:39 nagiosxi ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README.
Jun 22 02:01:39 nagiosxi ndo2db: Warning: queue send error, retrying...
Jun 22 02:01:40 nagiosxi ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README.
Jun 22 02:01:40 nagiosxi ndo2db: Warning: queue send error, retrying...

I have done the similar activity in past as well last month and there is no major integration after that nagios restart issue is fixed.

Below is my observation :
[root@nagiosxi var]# ipcs -q

------ Message Queues --------
key msqid owner perms used-bytes messages
0x03050002 3375104 nagios 600 35840 35
0x8f050002 3080193 nagios 600 8653824 8451
0x8a050002 3112962 nagios 600 6601728 6447
0xc4050002 3145731 nagios 600 9079808 8867
0xc1050002 3178500 nagios 600 8696832 8493
0x91050002 3211269 nagios 600 8584192 8383
0xb3050002 3244038 nagios 600 7645184 7466
0x8b050002 3276807 nagios 600 8876032 8668
0x9d050002 3309576 nagios 600 8833024 8626
0xc9050002 3342345 nagios 600 5769216 5634

And still load average is too high on the server
[root@nagiosxi var]# uptime
03:09:33 up 20 days, 1:27, 3 users, load average: 47.14, 44.17, 45.24
Which is due to nagios application can you suggest how i can reduce the load average.

[tmcdonald]

How many total hosts and services are you monitoring from this machine? With a load that high and the message queue issues you are seeing, I would imagine it is over 30k.

[ericssonvietnam]

How many total hosts and services are you monitoring from this machine? With a load that high and the message queue issues you are seeing, I would imagine it is over 30k.

I have only 15060 services in total and there are around 168 nodes.

But would like to tell you that i have observer that as i restart nagios service from the GUI after some changes in configuration i observed increase in number of message q

current outoput
[root@nagiosxi ~]# ipcs -q

------ Message Queues --------
key msqid owner perms used-bytes messages
0xf0050002 3473408 nagios 600 0 0
0xfe050002 3637249 nagios 600 0 0
0xbe050002 3604482 nagios 600 0 0

[tgriep]

The current Message Queue looks pretty good when you posted.
Can you run the following command so we can see what are the top 20 processes are so we can start to look at what is causing the higher load?

Code: Select all

top -b -n1 |head -n 28

One quick thing you can do to gain some performance it to add a RAM Disk to the server.
The following are the instructions to setup that up on the Nagios server.
https://assets.nagios.com/downloads/nag ... giosXI.pdf