Slow performance in nagios xi

[igarcia]

After apply configuration in our nagiosxi, we detect a slow performance on the ndo2db service.
The nagios core show the alert in real time, but the nagiosxi have a delay up to 20 minutes.

Our architecture of nagiosxi:

nagioscore server:

Code: Select all

Red Hat Enterprise Linux Server release 6.7 (Santiago)
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
Mem:                   8 GB
rrdcache
ramdiskbackup
offset mysql
Nagiosxi Version:	5.2.7
mod_gearman-1.5.0b1-1.el6.x86_64
gearmand-1.1.8-2.el6.x86_64
libgearman-1.1.8-2.el6.x86_64

/etc/sysctel.conf

Code: Select all

# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 1048576000
# Controls the maximum size of a message, in bytes
kernel.msgmax = 1048576000
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 17179869180
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 1073741824
## The maximum number of messages allowed in any one message queue
kernel.msgmni = 10240000

/usr/local/nagios/etc/ndo2db.cfg

Code: Select all

# Keep timed events for 24 hours
max_timedevents_age=1440
# Keep system commands for 1 week
max_systemcommands_age=10080
# Keep service checks for 1 week
max_servicechecks_age=10080
# Keep host checks for 1 week
max_hostchecks_age=10080
# Keep event handlers for 31 days
max_eventhandlers_age=44640

3 x nagiosworkers

Code: Select all

Red Hat Enterprise Linux Server release 6.7 (Santiago)
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
mem:                   8 GB
gearmand-0.25-1.x86_64
gearmand-devel-0.25-1.x86_64
libgearman-1.1.8-2.el6.x86_64
mod_gearman-1.4_nagios4-1.el6.x86_64

2 x mysqlhost in ha (corosync + pacemaker )

Code: Select all

Oracle Linux Server release 6.6
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
mem:                   8 GB
mysql-commercial-server-5.7.9-1.1.el6.x86_64

Nagios is checking about:
1500 hosts
13000 services.

Our final numbers will be double of this ones.

We have checked our network, database, and io, and all of them are working ok.
Thanks,
Regards.

[tmcdonald]

A few questions/points:

• What Nagios XI version is this?
• What is the output of ipcs -q?
• Have you implemented a RAM disk, offloaded the DB (assuming so), or done any other performance tweaks?

[igarcia]

Hi,


The output of the command:

Code: Select all

ipcs -q

------ Colas de mensajes -----
key        msqid      propietario perms      bytes utilizados mensajes    
0x06000002 327680     nagios     600        121422848    118577      

The info of the core server:

Code: Select all

....
rrdcache
ramdisk
offset mysql
Nagiosxi Version:   5.2.7
---

The df output

Code: Select all

df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/SystemGroup-lvol1
                       12G  3,7G  7,5G  33% /
tmpfs                 3,9G     0  3,9G   0% /dev/shm
/dev/sda1             477M   80M  373M  18% /boot
/dev/mapper/SystemGroup-lvol3
                      4,9G  161M  4,5G   4% /tmp
/dev/mapper/SystemGroup-lvol2
                      9,8G  156M  9,2G   2% /var/log
/dev/mapper/SystemGroup-lvol4
                       15G  123M   14G   1% /var/spool
/dev/mapper/DataGroup-lvol1
                       40G   21G   18G  55% /APPS
tmpfs                 4,0G   34M  4,0G   1% /var/nagiosramdisk

[tmcdonald]

One last thing to check: /usr/loca/nagios/bin/ndo2db --version

You are likely on 2.0.0 and there is a later version that should fix this issue. If you are comfortable recompiling yourself, you can download the 2.1b2 version here:

https://github.com/NagiosEnterprises/nd ... tils-2.1b2

and do a simple ./configure make make install loop to update. Otherwise let me know and I can provide step-by-step instructions.

[igarcia]

Ok, one question.

We will have problems with the autoupgrade after compile ndo?

Thanks,
Regards.

[tmcdonald]

I'm not sure I understand the question. These changes should survive an upgrade to XI if that is what you are asking.

[igarcia]

Hi,

We have compile the suggested version and make some test. Sadly those test where unsatisfactory, so we have return to the previous version. I include some logs and issues.

NDOMOD log showing the version in use

Code: Select all

Apr 19 09:45:58 xxxxxxxxx nagios: ndomod: NDOMOD 2.1.0b2 (12-02-2015) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
Apr 19 09:45:58 xxxxxxxxx nagios: ndomod: Successfully connected to data sink.  0 queued items to flush.
Apr 19 09:45:58 xxxxxxxxx nagios: ndomod registered for process data
Apr 19 09:45:58 xxxxxxxxx nagios: ndomod registered for log data'
Apr 19 09:45:58 cnagios: ndomod registered for system command data'

Delay between Core and NDOMOD. Keeps in an invalid amount of time

Code: Select all

​Apr 19 10:05:30 xxxxxxxxx nagios: SERVICE ALERT: yyyyyyy;ESX NET;OK;SOFT;2;CHECK_ESX3.PL OK - net receive=279.00 KBps, send=738.00 KBps, all 2 NICs are connected

An additional problem that appears is that has disappear all of our statistics


Referencing our last question, we would like to know the impact, in future upgrades from the official nagiosxi package, if we use such a self compiled module.

[ssax]

Please PM one of us your profile so that we may look through your configuration. You can download it by going to Admin > System Config > System Profile and click the Download Profile button in the top right corner.

Thank you

[igarcia]

Please PM one of us your profile so that we may look through your configuration. You can download it by going to Admin > System Config > System Profile and click the Download Profile button in the top right corner.

Thank you

I have sent the pm.

Thanks.

[tmcdonald]

It looks like you forgot the attachment - please re-send the profile making sure to attach it in your reply.

Update: File received and shared with team.