Tuning logstash pipeline

[sacom01]

Dear Nagios Team,

My NagiosLS recieve >200k message per minute now.
I want a number to tuning pipeline worker.
And how to config batch_size, batch_delay, max_inflight in nagios log server?

Config in my system:
{:timestamp=>"2021-04-12T13:50:10.742000+0700", :message=>"Starting pipeline", :id=>"main", :pipeline_workers=>16, :batch_size=>125, :batch_delay=>5, :max_inflight=>2000, :level=>:info}
My system run on 2 server: 16 CPUs, 512Gb Ram.
JVM logstash and elasticsearch: 64Gb.


Thanks Nagios team.

[cdienger]

The pipeline options can be configured in /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:

Code: Select all

  DEFAULT_SETTINGS = {
    :default_pipeline_workers => LogStash::Config::CpuCoreStrategy.maximum,
    :pipeline_batch_size => 225,
    :pipeline_batch_delay => 5, # in milliseconds
    :flush_interval => 5, # in seconds
    :flush_timeout_interval => 60, # in seconds
    :debug_config => false,
    :allow_env => false
  }