Page 1 of 1
unrecoverable plugin error
Posted: Thu Dec 02, 2021 12:02 pm
by benhank
both nodes in my cluster are reporting this error. what does it mean and how do I fix it?
Code: Select all
{:timestamp=>"2021-12-02T11:53:27.334000-0500", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"snmptrap\", port=>161, codec=><LogStash::Codecs::Line charset=>\"UTF-8\", delimiter=>\"\\n\">, host=>\"0.0.0.0\", data_timeout=>-1, mode=>\"server\", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>>\n Error: Permission denied", :level=>:error
Re: unrecoverable plugin error
Posted: Fri Dec 03, 2021 10:36 am
by pbroste
Hello
@benhank
Thanks for reaching out, want to get a System Profile from you so we can see what is going on.
To send System Profile:
Code: Select all
/usr/local/nagioslogserver/scripts/profile.sh
The System Profile is found in '/tmp/system-profile.tar.gz'
Depending on the Profile size, may need to use the split command to size down the compressed Profile.
Please send each
'part[x]' in a separate PM.
- Make sure that System date/time/timezone are all synced up
[list]Code: Select all
date
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
grep "date.timezone =" /etc/php.ini
grep date.timezone /etc/php.ini
php -r "echo date_default_timezone_get();" #timezone
php -r " echo date('Y-m-d H:i:s');" #date
php -r " echo gmdate('Y-m-d H:i:s');" #time
- Please send over System information:
[list]Code: Select all
sudo cat /etc/os-release
sudo uname -a
sudo hostnamectl
cat /etc/os-release
uname -a
hostnamectl
php -v
yum repolist
[/list][/list]
Thanks,
Perry
Re: unrecoverable plugin error
Posted: Wed Dec 08, 2021 3:33 pm
by pbroste
Hello
@benhank
Thanks for following up and sending over the System Profile. And thanks for pinging me on this, oppps on my part I typed out a response yesterday and being a tab junkie I went ahead and cleared them before hitting the send.
To follow up, we would like to have you add the rules to allow UPD traffic on port 161 and 162 to your firewall iptables rules. We don't see these listed in the Profile results.
API commands to verify:
Code: Select all
curl 'localhost:9200/_cluster/state?pretty'
curl 'localhost:9200/_cat/indices?pretty'
curl 'localhost:9200/_cluster/health/*?level=shards&pretty'
Let us know how things look,
Perry
Re: unrecoverable plugin error
Posted: Thu Dec 09, 2021 11:17 am
by benhank
The firewall is disabled on all three servers that this is happening on.
Also This is a fresh cluster, newly built and not ingesting data yet.
Re: unrecoverable plugin error
Posted: Thu Dec 09, 2021 5:22 pm
by pbroste
Hello
@benhank
Thanks for following up, the message; "A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"snmptrap\..." From what I see it appears that there is a plugin associated with snmptrap that is failing. Let's make sure that the logstash service looks okay, and should look similar:
Code: Select all
● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash; generated)
Active: active (running) since Mon 2021-12-06 06:27:47 CST; 3 days ago
Docs: man:systemd-sysv-generator(8)
Tasks: 43 (limit: 101107)
Memory: 638.5M
CGroup: /system.slice/logstash.service
├─1005 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4 nagios
└─1030 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
The /usr/local/nagioslogserver/.... owner: nagios and associated with nagios group:
Code: Select all
ls -laR /usr/local/nagioslogserver/ | grep -Ei '^d' | less
Also, take a look to see if there is anything logged in the system logs, or we will need to try to figure out which 'plugin' the error message is referring to.
Please let us know how things look,
Perry
Re: unrecoverable plugin error
Posted: Thu Dec 09, 2021 5:32 pm
by pbroste
Also, check for a plugin directory:
Code: Select all
find /usr/local/ -type d -name plugin
Thanks
Perry