Page 1 of 1

Scheduled jobs/alerts are not running.

Posted: Tue Dec 28, 2021 1:30 pm
by NaveenR
Hello,

Alerts have stopped running as scheduled. we can trigger them manually it works. I have restarted elasticsearch and logstash services i get below errors in logs. And i have reset all jobs in command subsystem they all are in Waiting stage.


{:timestamp=>"2021-12-27T15:47:40.004000-0500", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2021-12-27T15:50:23.960000-0500", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"multiline-syslogerr\", port=>5544, codec=><LogStash::Codecs::Multiline pattern=>\"^\\\\s\", negate=>true, what=>\"previous\", charset=>\"UTF-8\", multiline_tag=>\"multiline\", max_lines=>500, max_bytes=>10485760>, host=>\"0.0.0.0\", data_timeout=>-1, mode=>\"server\", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>>\n Error: Address already in use", :level=>:error}
{:timestamp=>"2021-12-27T15:50:23.960000-0500", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"multiline_SysOutlog\", port=>5544, codec=><LogStash::Codecs::Multiline pattern=>\"^\\\\s\", negate=>true, what=>\"previous\", charset=>\"UTF-8\", multiline_tag=>\"multiline\", max_lines=>500, max_bytes=>10485760>, host=>\"0.0.0.0\", data_timeout=>-1, mode=>\"server\", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>>\n Error: Address already in use", :level=>:error}
{:timestamp=>"2021-12-27T15:50:24.441000-0500", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2021-12-27T15:50:24.475000-0500", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2021-12-27T15:50:24.481000-0500", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2021-12-27T15:50:24.487000-0500", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2021-12-27T15:50:24.524000-0500", :message=>"Pipeline aborted due to error", :exception=>"NoMethodError", :error=>"undefined method `close' for nil:NilClass", :backtrace=>["/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:83:in `add_patterns_from_file'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:372:in `add_patterns_from_files'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:368:in `add_patterns_from_files'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:263:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in `register'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:182:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:182:in `start_workers'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:136:in `run'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}
{:timestamp=>"2021-12-27T15:50:24.962000-0500", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"multiline-syslogerr\", port=>5544, codec=><LogStash::Codecs::Multiline pattern=>\"^\\\\s\", negate=>true, what=>\"previous\", charset=>\"UTF-8\", multiline_tag=>\"multiline\", max_lines=>500, max_bytes=>10485760>, host=>\"0.0.0.0\", data_timeout=>-1, mode=>\"server\", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>>\n Error: Address already in use", :level=>:error}
{:timestamp=>"2021-12-27T15:50:24.971000-0500", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"multiline_SysOutlog\", port=>5544, codec=><LogStash::Codecs::Multiline pattern=>\"^\\\\s\", negate=>true, what=>\"previous\", charset=>\"UTF-8\", multiline_tag=>\"multiline\", max_lines=>500, max_bytes=>10485760>, host=>\"0.0.0.0\", data_timeout=>-1, mode=>\"server\", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>>\n Error: Address already in use", :level=>:error}

Re: Scheduled jobs/alerts are not running.

Posted: Mon Jan 03, 2022 1:46 pm
by NaveenR
Nagios user account is expired. We have created new password, it is working now.