Nagios Support Forum

We are at a stage where every time we apply Configuration, the scheduling screws up.
Need to Apply a few times before it is ok.

The root cause seems to be ipcq -q shows messages queue to be more than 100k.

When it stabilises after a few hours it reaches to be a few hundred to thousand only Please advice when this issue is going to be fixed.

How many hosts and services do you have total? When you apply configuration there is a lot being written to the NDO database, and when we see problems is is usually the number of messages that shoots up, not the number of bytes.

tmcdonald wrote:How many hosts and services do you have total? When you apply configuration there is a lot being written to the NDO database, and when we see problems is is usually the number of messages that shoots up, not the number of bytes.

Hosts : 2837
Services : 20854

The scheduling graphs always drops to single bar about 10 max. Need to restart a few times.
When this proble, happens, the messages queue always shows 90k plus messages.

All server performance CPU, Memory, I/O, disk storage all ok.

This may be a possible solution you could try to see if this fixes the Message Queue problem.
Please try it on a test system first if you can.

Open /etc/sysctl.conf with a text editor. Edit the file to match the following values:
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 262144000

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 262144000

## The maximum number of messages allowed in any one message queue
kernel.msgmni = 512000

Note: If you don't have these entries in the "/etc/sysctl.conf" file, just add them to the end of the file.
After these settings are saved to the file, run:
sysctl -p

Then edit /etc/my.cnf and add the following.
[mysqld]

#InnoDB Settings
innodb_buffer_pool_size=3G
innodb_flush_log_at_trx_commit=2

#Other
max_connections=500
max_connect-errors=1000
skip-name-resolve

tgriep wrote:This may be a possible solution you could try to see if this fixes the Message Queue problem.
Please try it on a test system first if you can.

Open /etc/sysctl.conf with a text editor. Edit the file to match the following values:
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 262144000

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 262144000

## The maximum number of messages allowed in any one message queue
kernel.msgmni = 512000

Note: If you don't have these entries in the "/etc/sysctl.conf" file, just add them to the end of the file.
After these settings are saved to the file, run:
sysctl -p

Then edit /etc/my.cnf and add the following.
[mysqld]

#InnoDB Settings
innodb_buffer_pool_size=3G
innodb_flush_log_at_trx_commit=2

#Other
max_connections=500
max_connect-errors=1000
skip-name-resolve

Thanks. Will update once I have tested this out.

No problem, keep us in the loop.

No positive changes noted and no error messages anywhere.
Messages still start from 100k during Apply Configuration.

Only message noted in the /var/log/messages was this repeated every minute After 30 minutes there was still 66k messages in the queue. The scheduling was hovering wildly between 200 - 1000 and finally settled on about 600 when it used to be 1000++ and a more distributed pattern Going to revert back all changes in the db configurations later.

Thanks for trying this out. The changes worked for another customer but every system is a little different.
We are still looking at the issue and will report back when there is a fix available.