unrecoverable plugin error

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
User avatar
benhank
Posts: 1264
Joined: Tue Apr 12, 2011 12:29 pm

unrecoverable plugin error

Post by benhank »

both nodes in my cluster are reporting this error. what does it mean and how do I fix it?

Code: Select all

{:timestamp=>"2021-12-02T11:53:27.334000-0500", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: <LogStash::Inputs::Tcp type=>\"snmptrap\", port=>161, codec=><LogStash::Codecs::Line charset=>\"UTF-8\", delimiter=>\"\\n\">, host=>\"0.0.0.0\", data_timeout=>-1, mode=>\"server\", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>>\n  Error: Permission denied", :level=>:error
Proudly running:
NagiosXI 5.4.12 2 node Prod Env 2500 hosts, 13,000 services
Nagiosxi 5.5.7(test env) 2500 hosts, 13,000 services
Nagios Logserver 2 node Prod Env 500 objects sending
Nagios Network Analyser
Nagios Fusion
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: unrecoverable plugin error

Post by pbroste »

Hello @benhank

Thanks for reaching out, want to get a System Profile from you so we can see what is going on.

To send System Profile:

Code: Select all

/usr/local/nagioslogserver/scripts/profile.sh
The System Profile is found in '/tmp/system-profile.tar.gz'

Depending on the Profile size, may need to use the split command to size down the compressed Profile.

Code: Select all

split -b 40M profile.tar.gz part
Please send each 'part[x]' in a separate PM.
    • Make sure that System date/time/timezone are all synced up
    • [list]
    • Code: Select all

      date
      ls -l /etc/localtime
      php -r 'echo date("D M j G:i:s T Y")."\n";'
      grep "date.timezone =" /etc/php.ini
      grep date.timezone /etc/php.ini
      php -r "echo date_default_timezone_get();" #timezone
      php -r " echo date('Y-m-d H:i:s');"        #date
      php -r " echo gmdate('Y-m-d H:i:s');"      #time
  • Please send over System information:
  • [list]
  • Code: Select all

    sudo cat /etc/os-release
     sudo uname -a
    sudo hostnamectl
    cat /etc/os-release
    uname -a
     hostnamectl
    php -v
    yum repolist
[/list][/list]


Thanks,
Perry
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: unrecoverable plugin error

Post by pbroste »

Hello @benhank

Thanks for following up and sending over the System Profile. And thanks for pinging me on this, oppps on my part I typed out a response yesterday and being a tab junkie I went ahead and cleared them before hitting the send.

To follow up, we would like to have you add the rules to allow UPD traffic on port 161 and 162 to your firewall iptables rules. We don't see these listed in the Profile results.

API commands to verify:

Code: Select all

curl 'localhost:9200/_cluster/state?pretty'
curl 'localhost:9200/_cat/indices?pretty'
curl 'localhost:9200/_cluster/health/*?level=shards&pretty'
Let us know how things look,
Perry
User avatar
benhank
Posts: 1264
Joined: Tue Apr 12, 2011 12:29 pm

Re: unrecoverable plugin error

Post by benhank »

The firewall is disabled on all three servers that this is happening on.
Also This is a fresh cluster, newly built and not ingesting data yet.
You do not have the required permissions to view the files attached to this post.
Proudly running:
NagiosXI 5.4.12 2 node Prod Env 2500 hosts, 13,000 services
Nagiosxi 5.5.7(test env) 2500 hosts, 13,000 services
Nagios Logserver 2 node Prod Env 500 objects sending
Nagios Network Analyser
Nagios Fusion
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: unrecoverable plugin error

Post by pbroste »

Hello @benhank

Thanks for following up, the message; "A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"snmptrap\..." From what I see it appears that there is a plugin associated with snmptrap that is failing. Let's make sure that the logstash service looks okay, and should look similar:

Code: Select all

systemctl status logstash

Code: Select all

● logstash.service - LSB: Logstash
   Loaded: loaded (/etc/rc.d/init.d/logstash; generated)
   Active: active (running) since Mon 2021-12-06 06:27:47 CST; 3 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 43 (limit: 101107)
   Memory: 638.5M
   CGroup: /system.slice/logstash.service
           ├─1005 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log  -w 4 nagios
           └─1030 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
The /usr/local/nagioslogserver/.... owner: nagios and associated with nagios group:

Code: Select all

ls -laR /usr/local/nagioslogserver/ | grep -Ei '^d' | less
Also, take a look to see if there is anything logged in the system logs, or we will need to try to figure out which 'plugin' the error message is referring to.

Code: Select all

grep -Eir 'snmptrap' /var/log/
Please let us know how things look,
Perry
User avatar
pbroste
Posts: 1288
Joined: Tue Jun 01, 2021 1:27 pm

Re: unrecoverable plugin error

Post by pbroste »

Also, check for a plugin directory:

Code: Select all

find /usr/local/ -type d -name plugin
Thanks
Perry
Locked