pnp4nagios/mod gearman/nagios 4.1

[yesilyurtav]

Hello,

We are testing pnp4nagios, mod_gearman and nagios 4.1. We use pnp4nagios version 0.6, mod gearman neb module version 1.4, worker 2.3 , and nagios 4.1 core.

Everything looks fine, pnp4nagios can successfully read performance data from gearmand, but it creates graphs, (rrd and xml) for every service which has performance data. We don't need graphs for every service, for some services, we have performance data, but don't need graphs for them.

It might be not problem for 100-200 checks, but we have 40k service checks. This problem triggers performance problem, because we need to increase prefork of pnp processes, minumum 400-500 prefork we need, but if pnp doesn't create graphs for every service which has performance data, maybe only 100 will be enough.

I couldn't find any reply within pnp4nagios documentation. srv-pnp is only the link for graphs.

Is there any idea ? how can I declare in pnp only some services I need graphs ?

Regards.

[jdalrymple]

process_perf_data *: This directive is used to determine whether or not the processing of performance data is enabled for this service. Values: 0 = disable performance data processing, 1 = enable performance data processing.

[yesilyurtav]

No, It doesn't work, I tried. Because, pnp reads every service check result, and extract text after pipe . I disabled with this parameters, but pnp still continue to create graphs.

Regards.

[jdalrymple]

I'm not a pnp4nagios wizard so you'll have to bear with my ignorance.

I'm only familiar with synchronous, bulk, and npcd mode - all of these rely on the process_perfdata commands directives and commands (run by the nagios daemon) and if they're not executed perfdata is not processed. From what I'm reading you must be using gearman to process perfdata and that skirts the process_perfdata process?

My recommendation, use npcd mode. We have many clients processing hundreds of thousands of services' perfdata without npcd performance issues on XI systems. Most don't disable perfdata for performance reasons - it's simply not necessary. If you have to though, you then have that option.

[yesilyurtav]

Hello,

I red in pnp website, npcd mod is not available for nagios 4. we are usinf nagios 4.

I fixed the problem, pnp workers were notenough to handle 40k service checks, because worker was only 1. I increased it now to 300, and I see most graphs, but pnp servers' load is 300 !!!

Regards.

[rkennedy]

What are the specs of the server running nagios? Additionally, are you running a ramdisk at all?

[yesilyurtav]

server is a vm, 3 core intel xeon 3.3. Ghz, and 8 Gb ram. We have tmpfs partition but we don't use it fpr pnp or nagios.

I realised that, even pnp runs or not, after sometime, generally 24h , 1 day, nagios stops due to out of memory error. Before that, we had 3 Gb mem, but now 8 Gb, still have out of memory error with > 40k service check.

Istn't is enough to check <40k services with 8gb ram on nagios 4.1.1 ? as a neb module, only mod_gearman runs.

Regards.

[rkennedy]

There are quite a lot of factors that go into how many checks 8GB of ram can handle. I am actually quite surprised things are running smooth with 8GB as that isn't very much for XI. Are these 40k service checks passive or active?

Can you post the output of -

Code: Select all

top|head -5

How many workers are you using with mod_gearman, can you run the above command on them as well and post the output?

[yesilyurtav]

We have 3 servers for workers, and each server has maximum 500 worker threads. 40k checks are active checks. 70 % percentage of these checks, use a perl script to handle data via snmp. top shows me top 5 comands are gearmand and nagios deamon. 8 Gig is not enough for 40k service cheks in nagios xi ?

Regards.

[tmcdonald]

8 Gig is not enough for 40k service cheks in nagios xi ?

Are we talking about Nagios XI or Nagios Core? You posted this in the Core section and have been discussing Core until now. 40K for a single XI server is well beyond what we recommend you run, but Core might be able to handle the load since there is less running in the background.