Page 1 of 2
Naigosxi preformance issue
Posted: Tue Mar 06, 2018 1:29 am
by strmon
Hi Support,
Need your help in troubleshooting NagiosXI performance. Please let me know what details you need from system from understanding.
My configuration are as follows:
Server:
OS: centos 6 64 bit
Mem: 12 GB
CPU : 6 (vcpu)
NagiosXI: 5.4.3
host in monitoring: 98
Services : 1980
Do let me know what other details you need from my side to troubleshoot the issue.
Re: Naigosxi preformance issue
Posted: Tue Mar 06, 2018 11:43 am
by scottwilkerson
Can you describe the problem you are experiencing (be as detailed as possible)?
If we can get a detailed description of the problem, as well as have you PM myself or another tech you system profile zip from
Admin -> System Profile
That would be a good start
Re: Naigosxi preformance issue
Posted: Wed Mar 07, 2018 2:14 am
by strmon
Now and then we receive error like,
Performance Grapher : not running
Database Maintenance : delay
Also, system become close to freeze and its hard to even ssh. We recently upgraded system memory from 8 to 12 Gb, still no luck.
In order to troubleshoot, i've preformed steps shared in Nagiosxi performance guide, after that i found it bit better bit still not preformance graph is not working.
As requested, i've attached profile zip, have a look and suggest.
Re: Naigosxi preformance issue
Posted: Wed Mar 07, 2018 9:46 am
by scottwilkerson
Just looking at the system log I an seeing errors for over 60 SNMP traps/sec and the traps look identical
How to do you have these hosts configured to send traps? At this pace you could be receiving thousands a minute, which could really slow down a single system.
In your npcd.log I see this before your restart
Code: Select all
[03-07-2018 11:35:30] NPCD: WARN: MAX load reached: load 683.550000/40.000000 at i=1
[03-07-2018 11:35:45] NPCD: WARN: MAX load reached: load 741.590000/40.000000 at i=1
[03-07-2018 11:36:00] NPCD: WARN: MAX load reached: load 831.490000/40.000000 at i=1
[03-07-2018 11:36:38] NPCD: WARN: MAX load reached: load 745.150000/40.000000 at i=1
[03-07-2018 11:36:53] NPCD: WARN: MAX load reached: load 676.530000/40.000000 at i=1
[03-07-2018 11:37:08] NPCD: WARN: MAX load reached: load 611.300000/40.000000 at i=1
[03-07-2018 11:37:23] NPCD: WARN: MAX load reached: load 550.440000/40.000000 at i=1
[03-07-2018 11:37:54] NPCD: WARN: MAX load reached: load 465.090000/40.000000 at i=1
[03-07-2018 11:38:21] NPCD: WARN: MAX load reached: load 393.590000/40.000000 at i=1
[03-07-2018 11:38:49] NPCD: WARN: MAX load reached: load 341.270000/40.000000 at i=1
[03-07-2018 11:39:04] NPCD: WARN: MAX load reached: load 303.270000/40.000000 at i=1
[03-07-2018 11:39:19] NPCD: WARN: MAX load reached: load 307.120000/40.000000 at i=1
[03-07-2018 11:39:34] NPCD: WARN: MAX load reached: load 333.760000/40.000000 at i=1
[03-07-2018 11:39:49] NPCD: WARN: MAX load reached: load 336.360000/40.000000 at i=1
[03-07-2018 11:40:14] NPCD: WARN: MAX load reached: load 357.650000/40.000000 at i=1
This is why you machine is slow, the load is sometimes reaching 1000. you need to determine what is causing the load, it could be the traps, but likely more than that and hard to decipher because you just restarted 5 minutes before creating the profile.
Being you just restarted
Re: Naigosxi preformance issue
Posted: Mon Mar 12, 2018 12:48 am
by strmon
I will look into that snmp issue. Meanwhile attaching fresh profile after weekended run. Have a look and suggest.
Re: Naigosxi preformance issue
Posted: Mon Mar 12, 2018 1:34 pm
by cdienger
The logs show:
ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may need to be tuned. See README.
Tuning is covered in
https://support.nagios.com/kb/article.php?id=139 . Please follow the guide and let us know if you have any further issues.
Re: Naigosxi preformance issue
Posted: Mon Mar 12, 2018 1:38 pm
by scottwilkerson
also in your /etc/my.cnf you likely want to add
under
because we are seeing you hit a limit in many php log files.
Re: Naigosxi preformance issue
Posted: Thu Mar 15, 2018 1:13 am
by strmon
I have followed the tuning guide and instruction provided on link. But still performance is low. Still getting error for Performance Grapher and Db maintenance ..!
Attaching fresh profile file, have a look.
Re: Naigosxi preformance issue
Posted: Thu Mar 15, 2018 8:46 am
by scottwilkerson
you have errors that show your system has run out of memory and shut processes down.
I would start by rebooting the server
Re: Naigosxi preformance issue
Posted: Mon Mar 19, 2018 3:36 am
by strmon
PFA , fresh profile file after reboot.