Page 1 of 2
Slow performance in nagios xi
Posted: Mon Apr 18, 2016 5:24 am
by igarcia
After apply configuration in our nagiosxi, we detect a slow performance on the ndo2db service.
The nagios core show the alert in real time, but the nagiosxi have a delay up to 20 minutes.
Our architecture of nagiosxi:
nagioscore server:
Code: Select all
Red Hat Enterprise Linux Server release 6.7 (Santiago)
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
Mem: 8 GB
rrdcache
ramdiskbackup
offset mysql
Nagiosxi Version: 5.2.7
mod_gearman-1.5.0b1-1.el6.x86_64
gearmand-1.1.8-2.el6.x86_64
libgearman-1.1.8-2.el6.x86_64
/etc/sysctel.conf
Code: Select all
# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 1048576000
# Controls the maximum size of a message, in bytes
kernel.msgmax = 1048576000
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 17179869180
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 1073741824
## The maximum number of messages allowed in any one message queue
kernel.msgmni = 10240000
/usr/local/nagios/etc/ndo2db.cfg
Code: Select all
# Keep timed events for 24 hours
max_timedevents_age=1440
# Keep system commands for 1 week
max_systemcommands_age=10080
# Keep service checks for 1 week
max_servicechecks_age=10080
# Keep host checks for 1 week
max_hostchecks_age=10080
# Keep event handlers for 31 days
max_eventhandlers_age=44640
3 x nagiosworkers
Code: Select all
Red Hat Enterprise Linux Server release 6.7 (Santiago)
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
mem: 8 GB
gearmand-0.25-1.x86_64
gearmand-devel-0.25-1.x86_64
libgearman-1.1.8-2.el6.x86_64
mod_gearman-1.4_nagios4-1.el6.x86_64
2 x mysqlhost in ha (corosync + pacemaker )
Code: Select all
Oracle Linux Server release 6.6
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
mem: 8 GB
mysql-commercial-server-5.7.9-1.1.el6.x86_64
Nagios is checking about:
1500 hosts
13000 services.
Our final numbers will be double of this ones.
We have checked our network, database, and io, and all of them are working ok.
Thanks,
Regards.
Re: Slow performance in nagios xi
Posted: Mon Apr 18, 2016 9:11 am
by tmcdonald
A few questions/points:
- What Nagios XI version is this?
- What is the output of ipcs -q?
- Have you implemented a RAM disk, offloaded the DB (assuming so), or done any other performance tweaks?
Re: Slow performance in nagios xi
Posted: Mon Apr 18, 2016 9:19 am
by igarcia
Hi,
The output of the command:
Code: Select all
ipcs -q
------ Colas de mensajes -----
key msqid propietario perms bytes utilizados mensajes
0x06000002 327680 nagios 600 121422848 118577
The info of the core server:
Code: Select all
....
rrdcache
ramdisk
offset mysql
Nagiosxi Version: 5.2.7
---
The df output
Code: Select all
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/SystemGroup-lvol1
12G 3,7G 7,5G 33% /
tmpfs 3,9G 0 3,9G 0% /dev/shm
/dev/sda1 477M 80M 373M 18% /boot
/dev/mapper/SystemGroup-lvol3
4,9G 161M 4,5G 4% /tmp
/dev/mapper/SystemGroup-lvol2
9,8G 156M 9,2G 2% /var/log
/dev/mapper/SystemGroup-lvol4
15G 123M 14G 1% /var/spool
/dev/mapper/DataGroup-lvol1
40G 21G 18G 55% /APPS
tmpfs 4,0G 34M 4,0G 1% /var/nagiosramdisk
Re: Slow performance in nagios xi
Posted: Mon Apr 18, 2016 12:16 pm
by tmcdonald
One last thing to check:
/usr/loca/nagios/bin/ndo2db --version
You are likely on 2.0.0 and there is a later version that should fix this issue. If you are comfortable recompiling yourself, you can download the 2.1b2 version here:
https://github.com/NagiosEnterprises/nd ... tils-2.1b2
and do a simple
./configure make make install loop to update. Otherwise let me know and I can provide step-by-step instructions.
Re: Slow performance in nagios xi
Posted: Mon Apr 18, 2016 2:15 pm
by igarcia
Ok, one question.
We will have problems with the autoupgrade after compile ndo?
Thanks,
Regards.
Re: Slow performance in nagios xi
Posted: Mon Apr 18, 2016 5:02 pm
by tmcdonald
I'm not sure I understand the question. These changes should survive an upgrade to XI if that is what you are asking.
Re: Slow performance in nagios xi
Posted: Tue Apr 19, 2016 3:33 am
by igarcia
Hi,
We have compile the suggested version and make some test. Sadly those test where unsatisfactory, so we have return to the previous version. I include some logs and issues.
NDOMOD log showing the version in use
Code: Select all
Apr 19 09:45:58 xxxxxxxxx nagios: ndomod: NDOMOD 2.1.0b2 (12-02-2015) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
Apr 19 09:45:58 xxxxxxxxx nagios: ndomod: Successfully connected to data sink. 0 queued items to flush.
Apr 19 09:45:58 xxxxxxxxx nagios: ndomod registered for process data
Apr 19 09:45:58 xxxxxxxxx nagios: ndomod registered for log data'
Apr 19 09:45:58 cnagios: ndomod registered for system command data'
Delay between Core and NDOMOD. Keeps in an invalid amount of time
Code: Select all
Apr 19 10:05:30 xxxxxxxxx nagios: SERVICE ALERT: yyyyyyy;ESX NET;OK;SOFT;2;CHECK_ESX3.PL OK - net receive=279.00 KBps, send=738.00 KBps, all 2 NICs are connected
An additional problem that appears is that has disappear all of our statistics
Referencing our last question, we would like to know the impact, in future upgrades from the official nagiosxi package, if we use such a self compiled module.
Re: Slow performance in nagios xi
Posted: Tue Apr 19, 2016 4:24 pm
by ssax
Please PM one of us your profile so that we may look through your configuration. You can download it by going to Admin > System Config > System Profile and click the Download Profile button in the top right corner.
Thank you
Re: Slow performance in nagios xi
Posted: Wed Apr 20, 2016 5:45 am
by igarcia
ssax wrote:Please PM one of us your profile so that we may look through your configuration. You can download it by going to Admin > System Config > System Profile and click the Download Profile button in the top right corner.
Thank you
I have sent the pm.
Thanks.
Re: Slow performance in nagios xi
Posted: Wed Apr 20, 2016 4:46 pm
by tmcdonald
It looks like you forgot the attachment - please re-send the profile making sure to attach it in your reply.
Update: File received and shared with team.