nagios/adagios slow

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
stefanotoni
Posts: 9
Joined: Tue Oct 27, 2015 6:40 am

nagios/adagios slow

Post by stefanotoni »

Hello, I am using a nagios core installation with adagios frontend for the configuration
The Software in installed in a virtual machine with 16GB ram
The problem is that the GUI is often very slow. It seem adagios GUI more slow redarging the nagios GUI

Nagios 3.5.1
Adagios 1.6.1

I notice a very strange peak load cyclic, h24x7day (see tha attach LOAD.png)

Below the top:
top - 12:10:04 up 28 days, 23:50, 1 user, load average: 0.95, 2.32, 3.07
Tasks: 197 total, 1 running, 194 sleeping, 0 stopped, 2 zombie
Cpu(s): 23.2%us, 12.4%sy, 0.0%ni, 64.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16467308k total, 15891644k used, 575664k free, 244156k buffers
Swap: 8191996k total, 123564k used, 8068432k free, 4119744k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2958 nagios 20 0 1269m 292m 2444 S 2.3 1.8 2293:46 nagios
2886 root 20 0 1412m 22m 3376 S 0.3 0.1 70:55.31 collectd
1 root 20 0 21456 1464 1120 S 0.0 0.0 1:31.57 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 2:12.81 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 1:58.34 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 0:01.63 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 2:14.00 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/1

Who can help me?
Thank you
Stefano
Attachments
LOAD.png
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: nagios/adagios slow

Post by mcapra »

We'd have to look at a few Nagios logs as well as the full Nagios configuration set to give a definitive answer from the Nagios Core end of things. Are you able to share those things?

We can't offer much regarding Adagios since we don't maintain it.
Former Nagios employee
https://www.mcapra.com/
stefanotoni
Posts: 9
Joined: Tue Oct 27, 2015 6:40 am

Re: nagios/adagios slow

Post by stefanotoni »

Hello what do you need?
- which logs do you need? /var/log/nagios?
- full Nagios configuration: all the object, command, service, hosts config file?also nagios.conf...

Thank you
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: nagios/adagios slow

Post by tmcdonald »

Before we get too far into the configs, would it be possible for you to upgrade to the latest Core? 3.5.1 is somewhat old, and the latest is 4.3.0. There have been many performance improvements since then.

If that is not possible, we would need to look for checks that are scheduled to run at the frequency you are seeing the spikes, or find a pattern where multiple checks running more frequently line up their check times periodically and cause the spikes.

It might also be related to some settings in your nagios.cfg relating to check spreading, so that would be handy to send over.
Former Nagios employee
stefanotoni
Posts: 9
Joined: Tue Oct 27, 2015 6:40 am

Re: nagios/adagios slow

Post by stefanotoni »

In attach the:
- nagios.conf
- ps -elf (during peak). I see a lot of nagios process

[root@nethmonitor nagios]# ps -elf |grep nagios.cfg |more
1 S nagios 2958 1 5 80 0 - 325064 hrtime Feb21 ? 1-20:51:07 /usr/sbin/nagios -d /etc/nagios/nagios.cfg
1 S nagios 18306 2958 0 80 0 - 325065 pipe_w 10:51 ? 00:00:00 /usr/sbin/nagios -d /etc/nagios/nagios.cfg
1 S nagios 18309 2958 0 80 0 - 325065 pipe_w 10:51 ? 00:00:00 /usr/sbin/nagios -d /etc/nagios/nagios.cfg
1 S nagios 18312 2958 0 80 0 - 325065 pipe_w 10:51 ? 00:00:00 /usr/sbin/nagios -d /etc/nagios/nagios.cfg
1 S nagios 18314 2958 0 80 0 - 325065 pipe_w 10:51 ? 00:00:00 /usr/sbin/nagios -d /etc/nagios/nagios.cfg

[root@nethmonitor nagios]# ps -elf |grep nagios.cfg |wc -l
70
[root@nethmonitor nagios]# ps -elf |grep nagios.cfg |wc -l
80
[root@nethmonitor nagios]# ps -elf |grep nagios.cfg |wc -l
79

is normaly?

Thank you
Attachments
ps-elf.txt
(16.62 KiB) Downloaded 351 times
nagios.conf.txt
(45.27 KiB) Downloaded 363 times
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: nagios/adagios slow

Post by scottwilkerson »

If you cannot upgrade to 4.3.0, at a bare minimum I would suggest changing the following nagios.cfg option


change:

Code: Select all

use_large_installation_tweaks=0
to

Code: Select all

use_large_installation_tweaks=1
Finally, if it is Adagios that is causing the problem, you would need to reach out to the developer of Adagios separately as we do not maintain that project.
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
stefanotoni
Posts: 9
Joined: Tue Oct 27, 2015 6:40 am

Re: nagios/adagios slow

Post by stefanotoni »

The change (use_large_installation_tweaks=1) no solve the problem

However I realy don't understand the reason of the load peak:
- why the load grow..
- why after reaching a value 4/6 the load decrease. There is something that limit the peak??

Someone can give me some information?

Thank you
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: nagios/adagios slow

Post by scottwilkerson »

What version of Nagios are you running? I remember this cycling of load from some early 4.x versions of Core

Also, how many CPU Cores does your system have?
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
stefanotoni
Posts: 9
Joined: Tue Oct 27, 2015 6:40 am

Re: nagios/adagios slow

Post by stefanotoni »

Nagios 3.5.1

Produttore
VMware, Inc.
Modello
VMware Virtual Platform
Modello CPU
2 x Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz

[root@nethmonitor ~]# more /proc/cpuinfo

Code: Select all

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
stepping        : 1
microcode       : 184549406
cpu MHz         : 3399.997
cache size      : 20480 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 20
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_t
sc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx
hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm
bogomips        : 6799.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
stepping        : 1
microcode       : 184549406
cpu MHz         : 3399.997
cache size      : 20480 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 20
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_t
sc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx
hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm
bogomips        : 6799.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

[root@nethmonitor ~]# top

Code: Select all

top - 11:31:13 up 42 days, 22:11,  1 user,  load average: 2.16, 2.66, 1.99
Tasks: 272 total,  12 running, 258 sleeping,   0 stopped,   2 zombie
Cpu(s): 18.7%us, 15.2%sy,  0.0%ni, 65.8%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  16467308k total, 15883924k used,   583384k free,   235196k buffers
Swap:  8191996k total,    73996k used,  8118000k free,  2807084k cached

I notice that consider 16,46GB ram, used 15,88 GB ram. Swap used 73MB
Is possibile that the memory in not enough and the system swap to the disk (loading the server resource?)

Also, I see the graph attach (disk pending operations) that have frequently disk access. Is possibile that is combine with the load peak?Do you the reason of this disk access about nagios?

Thank you
Attachments
DISK_PENDING.png
Last edited by mcapra on Wed Apr 05, 2017 2:35 pm, edited 1 time in total.
Reason: please use [code] tags for technical output
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: nagios/adagios slow

Post by mcapra »

How many hosts/services is this machine handling?
stefanotoni wrote: I notice that consider 16,46GB ram, used 15,88 GB ram. Swap used 73MB
Is possibile that the memory in not enough and the system swap to the disk (loading the server resource?)
That's very possible. Have you tried adding more RAM to this machine?
Former Nagios employee
https://www.mcapra.com/
Locked