Log Server randomly crashes

[CameronWP]

Hi: We have been having an intermittent issue where the logstash process keeps running but when you check the status it says (exited) and no logs are being added to the indexes. The log is throwing this issue:

{:timestamp=>"2018-08-24T02:46:35.489000-0400", :message=>"Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.", "exception"=>#<NoMethodError: undefined method `>=' for nil:NilClass>, "backtrace"=>["(eval):7667:in `initialize'", "org/jruby/RubyArray.javain `each'", "(eval):7665:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):7685:in `initialize'", "org/jruby/RubyArray.javain `each'", "(eval):7679:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval)in `filter_func'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:267:in `filter_batch'", "org/jruby/RubyArray.javain `each'", "org/jruby/RubyEnumerable.java:852:in `inject'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:265:in `filter_batch'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:223:in `worker_loop'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}

I am not sure where to start with this, as I don't see why it would work for periods of weeks and then randomly stop working complaining about a filter. Thanks!

[cdienger]

You could be running into the problem described in https://support.nagios.com/kb/article/n ... g-576.html. I would follow the steps to increase the memory allocated to the process as well increase the file limit.

[CameronWP]

Thanks for the reply. I had made the change from that article some time ago. Are there different values I should be using other than the ones that are in that article?

Thanks!

[cdienger]

The process crashing is usually due to a resource issue. What does the drive space, memory, connection count look like? Is there anything in the elasticsearch logs(/var/log/elasticsearch) around the time you see logstash crash?

[CameronWP]

Drive space:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos_logserver-root 122619392 2627000 119992392 3% /
devtmpfs 32870444 0 32870444 0% /dev
tmpfs 32882860 0 32882860 0% /dev/shm
tmpfs 32882860 1361412 31521448 5% /run
tmpfs 32882860 0 32882860 0% /sys/fs/cgroup
/dev/sda1 503040 215964 287076 43% /boot
/dev/mapper/centos_logserver-home 5230592 33104 5197488 1% /home
wpddsvr1:/data/col1/LogSnaps 8145197568 4518415872 3626781696 56% /mnt/LogSnaps
/dev/mapper/centos-NagLogs 5858285568 3417171952 2441113616 59% /NagLogs
tmpfs 6576572 0 6576572 0% /run/user/0
tmpfs 6576572 0 6576572 0% /run/user/1000

Memory and CPU:

1m, 5m, 15m Load: 0.78, 0.68, 0.69
Memory (Used/Free): 60.89 GB / 1.83 GB
Swap (Used/Free): 1.18 GB / 12.50 GB
Total Memory: 62.72 GB
Total Swap: 12.50 GB
CPU User/Sys: 2% / 0%
CPU Idle: 96%
CPU Vendor: Intel
CPU Model: Xeon
Total Cores: 32

I will see if I can correlate the crash with the Elasticsearch log.

[cdienger]

Sounds good. Let us know if you find anything.