NagiosXI performance issue

[anoop]

Hi Team,
Thanks for your reply.. Waiting for my resources to integrate Mod-Gearman.. and i will update you soon after configuring the same... and one more thing.. we have seen

ps -aux | grep httpd, we find so many process running backend like nearly 20 processes... is this will impact the performance and even MRTG consuming some utilization..

We given polling interval as 10 minutes now and performance is little bit better than before but average host/service check latency is going like 60 to 70seconds, but came to know that it should have 10 to 15 sec an average...

Thanks in advance.

[slansing]

It is common for that many apache forks to be running because of the XI interface. That is totally fine. Can you send us a screenshot of Admin > Monitoring Engine Status.

[anoop]

Hi team ,


Thank you very much. We did not integrate mod-gearman till now but we are planning to install it. is there any kind of limitation in modgeraman regarding active/passive check. I am attaching 2-3 more screenshot of httpd service , can you please check and let us know is it ok or any kind of spiking from httpd and ndo2db services on nagiosxi server. some time mrtg utilizing more cpu....

what is the reason behind "Monitoring Engine Event Queue" automatically stopped

[sreinhardt]

Are your network host and services, mostly implemented via the network switch and router wizard? If so, I think we need to walk you through splitting up your configurations, prior to implementing mod_gearman. If my idea is correct, your spikes are happening, because the network checks are not spaced out, but instead done within one set of checks and scheduled at the same time. Please confirm if this sounds about right for your setup, and we can begin walking you through that.

[anoop]

HI Team,

We configured Network Switches and firewalls using Network Switch/Router Wizard for Ping and Network Utilization and we configured 150 switches with 25 services each and in total it became "3750" services and 150 firewalls with one service each and gross it counts 3900 service checks with a polling interval of "6" minutes and retry interval with "2" minutes....

And network device is scheduled at same time for service checks..

Suggest us with the better solution..

[slansing]

Okay this is what we will have to do, open:

/etc/mrtg/mrtg.cfg

Copy the starting lines of the config file from the top down to the blank line before the first interface name IE "### Interface 1 >> etc etc" The lines you copied should look similar to this:

Code: Select all

######################################################################
# Multi Router Traffic Grapher -- Example Configuration File
######################################################################
# This file is for use with mrtg-2.0
#
# Note:
#
# * Keywords must start at the begin of a line.
#
# * Lines which follow a keyword line which do start 
#   with a blank are appended to the keyword line
#
# * Empty Lines are ignored
#
# * Lines starting with a # sign are comments.

# Where should the logfiles, and webpages be created?

# Minimal mrtg.cfg
#--------------------

HtmlDir: /var/www/mrtg
ImageDir: /var/www/mrtg
LogFormat: rrdtool
LogDir: /var/lib/mrtg
ThreshDir: /var/lib/mrtg

#Target[r1]: 2:[email protected]
#MaxBytes[r1]: 1250000
#Title[r1]: Traffic Analysis
#PageTop[r1]: <H1>Stats for our Ethernet</H1>

WorkDir: /var/lib/mrtg

Now place these lines in a new mrtg.cfg file, such as /etc/mrtg/mrtg2.cfg. At the top, like they were in the original mrtg.cfg

Now go back to the original mrtg.cfg and add the following line under the WorkDir line:

Code: Select all

Include: /etc/mrtg/mrtg2.cfg

Or use the directory and .cfg name you defined.

Now move the about half way through that original config file that you added the above line in, make sure you are on a line that starts with a new interface, it should say something like:

Code: Select all

### Interface 13 >> Descr: 'Port:-13-Gigabit---Level' | Name: '' | Ip: '' | Eth: '' ###

"All interfaces start after a </div> and begin with a commented title."

Once you have copied all interfaces including and below this line place them in a new mrtg config file in /etc/mrtg, for example, /etc/mrtg/mrtg2.cfg.

Now remove all of the interfaces you copied from the original mrtg.cfg file.

To verify that the changes have worked check one of the interfaces you copied manually from the XI interface, it should check properly and should return faster. Now, a side note here, you may want to use more than just two mrtg.cfg files, simple repeat the above steps but make sure that you always include extra config files in the main mrtg.cfg file as that is what nagios looks to first. And also make sure you do not use the same interface more than once in other config files.

[anoop]

HI Team,

thank you for replying, We configured Network Switches and firewalls using Network Switch/Router Wizard for Ping and Network Utilization and we configured 150 switches with 25 services each and in total it became "3750" services and 150 firewalls with one service each and gross it counts 3900 service checks with a polling interval of "6" minutes and retry interval with "2" minutes....

And network device is scheduled at

same time

for all service checks..

Suggest us with the better solution..

[abrist]

"same time" for all service checks..

This is to be expected after adding them. It may take a few check cycles for the checks to get scheduled in a balanced way . . .
Has it resolved to a more balanced schedule at this time?

[anoop]

Are your network host and services, mostly implemented via the network switch and router wizard? If so, I think we need to walk you through splitting up your configurations, prior to implementing mod_gearman. If my idea is correct, your spikes are happening, because the network checks are not spaced out, but instead done within one set of checks and scheduled at the same time. Please confirm if this sounds about right for your setup, and we can begin walking you through that.

HI Sreinhardt,

thank you for replying, We configured Network Switches and firewalls using Network Switch/Router Wizard for Ping and Network Utilization and we configured 150 switches with 25 services each and in total it became "3750" services and 150 firewalls with one service each and gross it counts 3900 service checks with a polling interval of "6" minutes and retry interval with "2" minutes....

Let us know how to configure Mod_Gearman according to the scenario given above.... like how many workers needed and we are planning to maintain one worker at present for all network devices. So, do we need to configure any additional plugins apart from the Mod_Gearman Worker..???

And in total, we are going to add 30000 service checks in future. How many workers we need and suggest with the hardware configuration.. Today, we faced monitoring engine stopped for 4 times and graphs are also not generated for some time.. Explain us how to resolve this issue...

Thanks in advance...

[abrist]

How many services/hosts are you currently monitoring, and what is the expected maximum quantity of checks?